Introduction to Multimodal Analysis: Second Edition 9781350069145, 9781350069138, 9781350069176, 9781350069169

Citation preview

i

Introduction to Multimodal Analysis

ii

ALSO AVAILABLE FROM BLOOMSBURY Multimodal Discourse, by Gunther Kress and Theo van Leeuwen The Bloomsbury Companion to Contemporary Peircean Semiotics, edited by Tony Jappy The Visual Narrative Reader, edited by Neil Cohn Understanding Media Semiotics, by Marcel Danesi

iii

Introduction to Multimodal Analysis Second Edition Per Ledin and David Machin

iv

BLOOMSBURY ACADEMIC Bloomsbury Publishing Plc 50 Bedford Square, London, WC1B 3DP, UK 1385 Broadway, New York, NY 10018, USA   BLOOMSBURY, BLOOMSBURY ACADEMIC and the Diana logo are trademarks of Bloomsbury Publishing Plc   First published in Great Britain 2007 This edition published 2020   Copyright © Per Ledin and David Machin, 2020   David Machin and Per Ledin have asserted their right under the Copyright, Designs and Patents Act, 1988, to be identified as Authors of this work.   Cover design: Olivia D’Cruz   All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage or retrieval system, without prior permission in writing from the publishers.   Bloomsbury Publishing Plc does not have any control over, or responsibility for, any third-party websites referred to or in this book. All internet addresses given in this book were correct at the time of going to press. The author and publisher regret any inconvenience caused if addresses have changed or sites have ceased to exist, but can accept no responsibility for any such changes.   A catalogue record for this book is available from the British Library.   A catalog record for this book is available from the Library of Congress.   ISBN: HB: 978-1-3500-6914-5 PB: 978-1-3500-6913-8 ePDF: 978-1-3500-6916-9 eBook: 978-1-3500-6915-2   Typeset by Newgen KnowledgeWorks Pvt. Ltd., Chennai, India To find out more about our authors and books visit www.bloomsbury.com and sign up for our newsletters.

v

Contents

List of Images  viii Preface to the Second Edition  xi



Introduction  1 What is the specific approach taken in this book?  5 The social uses of design features  7 Outline of the chapters  10

1 What is multimodal analysis?  13 The idea of choice in communication  13 Semiotic choice  15 Materials and affordances  16 Discourse  18 Discourses and social practices  20 The shift to multimodal communication  21 Technologization  21 Integrated design  22 A new commodified communication setting moral standards  24 Coding of semiotic resources  31 Identifying codes  33

2 Pictures and images  37 A semiotic approach: Denotation and connotation  38 Carriers of connotation  42 Representing participants  48 Actions in images  55

vi

vi

Contents

3 Modality: Experiencing the world through images  61 The origins of modality as a linguistic concept  63 Modality markers and scales  66 Kinds of visual modality  74 Perspective in images  80 Engagement and frontal perspective  81 Non-frontal perspective  82 Angles  83 Distance  84

4 The meaning of colour in visual design  87 Patterns in colour as a semiotic resource  88 Colour as a semiotic system  89 The value of colours  96 Semiotics of colour  98 The dimensions of colour  98

5 The meaning of typography  111 Typeface and design  114 Typography as a semiotic system  117 Inventory of typographic meaning potential  124

6 Textures and materiality  139 Texture as a semiotic system  140 The dimensions of texture  149 Materials  157

7 Composition and page layout  167 Some basic principles of salience  170 Framing  178 Images and composition: Coordination and hierarchies  186

vii

Contents

8 Diagrams and flow charts  193 Classification  197 Composition and orientation  205 Composition and grouping  208 Causality  210 Temporality  212 Bibliography  219 Image Index  225 Index  235

vii

viii

Images

.1 0 1.1 1.2 1.3 1.4

Fairtrade coffee packaging  2 Plastic surgery website   14 Interior of Heritage Bicycles Café  17 Bounce healthy snack package and product  23 Diagrams used by Swedish School Inspectorate to compare good and bad preschools  27 2.1 Unikum Quality Development in preschool   40 2.2 Gears flow chart template   45 2.3 Jurnalul National (Romanian Newspaper)  53 2.4 Pyramid to show strategic plan for university   54 3.1 Viva La Papa chips  64 3.2 Modality scale for articulation of detail  66 3.3 IKEA catalogue  67 3.4 Toothpaste tube. Riddells Creek Organic  68 3.5 (a, b and c) Modalities of representations of heart  76 3.6 Closer magazine, Issue 699, 2016  79 4.1 Sustainability Scorecard 2011/12  94 4.2 Starbucks café  96 4.3 Starbucks colour palette  102

ix

Images

5.1 The line in Figure 5.1 seems passive and timeless compared with that in Figure 5.2  112 5.2 Whereas the one in Figure 5.2 seems proud and strong  113 5.3 The thicker line in Figure 5.3 might seem proud but also more solid and immovable  113 5.4 The one in Figure 5.4 in comparison seems dynamic and changing  113 5.5 The line in Figure 5.5 seems severe and unwelcoming  114 5.6 The line in Figure 5.6, in comparison, seems softer, warm or gentle  114 5.7 Rambo film poster  118 5.8 Cinderella film poster  119 5.9 Children’s sports camp website  123 5.10 Bargain Booze storefront  128 5.11 Sandro storefront  128 5.12 Fit Kitchen food package  133 6.1 High street gym, Nordic Wellness  141 6.2 CrossFit ‘box’  141 6.3 Natural and industrial textures in the interior of the café   146 6.4 Textures used in integrated design with ‘lived’ surfaces to evoke a past   147 6.5 Interior of Atrium building with clean industrial ‘core’ look   148 6.6 (a) and (b) Baby rattles  150

ix

x

x

Images

.7 GNC drinks bottles  152 6 6.8 Fairview cheese package   155 7.1 BNP poster  168 7.2 L’Oréal advertisement    172 7.3 (a) Organic Protein™ Plant Based Protein Powder. (b) SYNTHA-6® ISOLATE protein powder  173 7.4 Oxfam poster    175 7.5 Wax kit packaging, Sally Hansen  183 7.6 Page spread of ‘Vision 2016’  187 7.7 University web page for quality development   190 8.1 Different diagrams representing the same organization  194 8.2 Preschool chart showing how play can be broken down and evaluated   197 8.3 Pathway flow chart for Fairtrade shopping   204 8.4 Diagram showing output for a school, Lillåns Södra Skola  213 8.5 Template for SMART goals   213 8.6 Strategic diagram for Newcastle University  215

xi

Preface to the Second Edition

It is over a decade since the first edition of Introduction to Multimodal Analysis was published. Feedback from those who used the book told us that it was user-friendly and great for teaching. It took some of the groundbreaking tools and theories developed by Kress and van Leeuwen and presented in easy access form as a step to reading the original material. There were several motivations for writing the second edition. At a basic level, the examples used in the book and the points of reference used in the commentary and analysis date over ten years. It needed updating. Multimodality has also moved on a lot over a decade. The work of Kress and van Leeuwen, particularly their classic book Reading Images published in 1996, had a huge impact on the study of visual communication. It showed that we could break down visual communication into its components to allow more detailed forms of analysis, to see how it hangs together and works. But critical reflection on this model was only really beginning at the time of writing and it remains the base reference for many teachers and researchers, particularly in linguistics. Yet over the past decade, multimodality has developed into a range of subfields each with its own priorities, model and forms of data. And communication has changed a lot over this time. Not only has it become more multimodal, but also these modes have been used in very specifically new ways. Communication has become much more heavily codified and systematically deployed. My own work and research using multimodality has also changed and grown. And over the last five years this has been done working as a co-author with Per. It made sense that writing this book was now our joint project. So this book has a new section on theory and what multimodality is to account for these changes and to show what kind of multimodality this book is. Some of the older chapters remain, such as on images, modality, colour, typeface and composition. They have fresh examples. They retain the same basic toolkit approach as before. But they are slightly modified to reflect

xii

xii

Preface to the Second Edition

developments in theory and shifts in communication. And new chapters have been added on texture and materials and on diagrams. These have been added due to what we perceive as useful developments in multimodality and also through the need to provide tools which allow us to work on the ways that communication now takes place. We hope that the analytical tool kits in this book can be an inspiration to explore the fascinating visual world which we have made, and which we inhabit, and a take a critical perspective on it. They are designed to help facilitate research which allows students to investigate and take a stance on all kinds of multimodal communication. In doing so the tool kits will need to be combined in different ways. What we encounter as data for an analysis, such as a medical web site, the design of a café or the promotion of ecological food, employ all sorts of semiotic resources that are combined for certain purposes in certain contexts and for certain interests. Looking into the multimodal communication going on is, we hope to make clear, exciting.

1

Introduction When you see a particular product on a supermarket shelf, sat among a range of others, you will find information on that packaging that tells you what it contains, such as coffee, tea, biscuits and toothpaste. But that package will also communicate ideas and values about the product which can suggest that the product is ‘fun’, ‘fresh’, ‘traditional’, suitable for men or women, ‘sensual’, ‘healthy’ and so on. This is communicated not only in the language on the package but also through how the package looks, its shape, its texture, the use of typeface and colour. Such ideas and values may in fact have very little to do with the actual qualities of the product but are associations which are loaded onto the product through design. A  ‘traditional’ loaf of bread may likely have been made in a factory through industrial processes. But the look and feel of the packaging may communicate otherwise. Such design choices will have been carefully made and tested on focus groups. So specific people from an identified market segment will be asked if the carton for a new brand of yoghurt suggests ‘health’ and ‘vitality’. Should they not do so correctly, the product would be much less likely to end up in a shopping basket. This book is about how ideas like this are communicated on and through visual design. In this book we also want to encourage a critical awareness as to how seemingly innocent designs want to persuade us that we have certain needs, that we can change ourselves and the planet by consumption, for example. In the case of the coffee package in Figure 0.1, we can see how design choices have been made to tell specific consumers something about the associations, the ideas and values of a Fairtrade product. So here a particular kind of image, font and colour has been chosen to allow the consumer to make this distinction very quickly. Were we to ask most people what they thought of such a design, they may come up with terms such as ‘rustic’, ‘honest’ or ‘stylish’. If pressed, they

2

2

Introduction to Multimodal Analysis

Figure 0.1  Fairtrade coffee packaging. Level Ground Trading.

may say that the designs use a friendly, positive, personalized approach to presenting the people the product claims to help. They might say the colours are ‘softer’ or ‘simple’. So buying the product would mean that we take a stance, perhaps to fight inequalities or to contribute to sustainability, that we stand up for environmental concerns and make this planet better. Or at least the design communicates this. But what is it in the designs which allows the product to communicate ideas and values such as ‘rustic’ or ‘simple’ or even ‘saving the planet’? How does a designer do this? What choices do they make? If we asked the same question about any form of visual design such as an advert, web page or personally designed social media feed, a person may use words like ‘serious’, ‘bold’, ‘romantic’ or ‘moody’. But what is it that communicates such things and can we point to these, describe and analyse them? The aim of this book is to provide a set of tools to help us answer this question. In fact, when we use terms like ‘rustic’, ‘soft’ or ‘lively’ to describe a design, we are not describing the composition itself but rather the effect created by the elements and features in the design and how they have been arranged. And by doing so we overlook how such meanings are produced. Many of us have a quite undeveloped ability to describe and understand what we see in visual design. And this is interesting given the amount of time, money and expertise which goes into producing designs which have such effect. And since the availability of cheap software from the end of the last century, visual design has become a much more routine part of all media and documents we encounter each day and at much more sophisticated levels.

3

Introduction

Visual designers themselves often refer to design decisions as aesthetic choices. They talk about having a ‘feel’ for designs. But when we look at groups of products or adverts of websites which deal with the same or different topics, we find patterns in the similarities and differences. Designers can rely on viewers or consumers who will come and pick up and examine a product package to understand certain design choices. In the case of the coffee package in Figure 0.1, they will immediately recognize ideas and values of Fairtrade in the design in a way that would not work if a blank package simply carried the words ‘This is Fairtrade’. And across the media and communications industries, there has been an incredible increase in understanding of this process. This means that designers can communicate ideas and values in design much more precisely. For websites or any content which is driven by advertising, this shift in design culture is particularly important. For example, a news site will carefully think through visual design, in fact even building its entire identity as an outlet from this point, to address a specific market segment which it can then deliver to advertisers. This means that when such readers come onto their pages they feel correctly engaged not only by the content but also by how it is presented. In other words, the meaning of what is read should not be seen as independent from the design. Design is not mere dressing but part of the meaning of how the content is experienced. Design here will mean the choices in colours and fonts. It will mean making choices in the kinds of images that are used and how they are used, integrated with text, graphics, lists and data. It will mean how these are assembled together, aligned and placed in space. Designers of contemporary newspapers have told the authors that the aim was to create a more feminine and less formal manner of address to readers which suggested ‘engagement’ rather than ‘information’. Readers were to be ‘addressed’ as well ‘informed’. Such a form of address was accomplished through very specific uses of typeface, spacing, alignment, use of ‘engaging’ images and graphics. This attention to design and what it can be used to tell us about our identities and those who produce the design can be seen on university websites, or even the routine documents which now pass through our places of work. But for the most part accounts of such process, by visual designers and by the people they address, are done through adjectives and aesthetic terms. While we are aware of the effect on us when we are addressed, whether it is a university web page, a news website or packet of coffee, we tend to not have the tools to be able to identify how these work. And

3

4

4

Introduction to Multimodal Analysis

this is the topic of this book. This means asking, for example, what are the characteristics of a design that looks ‘honest’? But more specifically it asks what characteristics of typefaces, colours or textures can be used to build up a design whole which can communicate ‘honesty’, or ‘romantic’, ‘fun’ or ‘serious’. And how can we document and understand the importance of the way that such elements are positioned in relation to each other in designs? In the case of the Fairtrade package, as with other food packaging, decisions have been made as regards how much of the person we see and how detailed this image appears. Does this play a role in the way that the images relate to the sections containing text and information below and the brand to the top right? Throughout the chapters in this book we provide toolkits which allow us to take a systematic approach to all these questions. Rather than explaining designs in terms of adjectives, we can describe what it is in them that allows these meanings to be produced. Of course when we see web pages, documents or coffee packages we do not attend to things like typefaces, colours or the nature of the images in themselves. These are encountered as parts of material objects and come in typical configurations of features and qualities. And we have been tutored throughout our lives to understand the kinds of meanings carried by such typical material forms. When entering a supermarket or opening a web page, we already have a sense of what will be communicated and in what way. We ‘know’ that the resources harnessed for designs tend to come in established configurations and do not need to work this out from the colour or font or images in each individual case. And the material wholes we encounter are already established parts of ways of doing things in society, such as selling and buying food, and of attracting students to study at a university. Such things will come with established associations and be oriented to specific kinds of social relations between people. In this case the finer level design features should be seen as ways of allowing that material whole to successfully communicate specific ideas and values in the context of that particular form of communication. We look along the supermarket shelf at the coffee. We immediately recognize the Fairtrade brands communicating ethical shopping and ideas about honesty and authenticity. And we recognize other coffee products using different designs to communicate other kinds of associations – for example, of Italian chic, using elegant fonts, saturated gold and blue colours and a shiny surface.

5

Introduction

What is the specific approach taken in this book? Introduction to Multimodal Analysis takes a social semiotic approach to communication. This is an approach which allows us to break visual designs down into their basic components and understand how these work together. It is one which emphasizes how we tend to share resources for communication in societies. This is in the form of the meaning of words and grammar and also in the case of visual design. This is why the designer can try to predict what kinds of designs will be understood by shoppers as ‘authentic’ or ‘honest’. Although just as wrong choices and misunderstandings can be a quality of language, so can be the case in visual design. And given that we can all now design documents, web pages and social media accounts with cheap and easy-to-use software, we may find more unevenness in the use of semiotic resources. Multimodal analysis is different to former kinds of semiotics such as that of Roland Barthes (1977) which tended to look at individual signs and their simple, more direct meanings in terms of what they connote or signify – for example, the colour red could be said to connote danger. A  multimodal approach would be interested in the way that signs are used in combination, adding up to a whole. Red can mean different things when combined with other kinds of semiotic resources to make more subtle and complex meanings. In this sense the meaning of signs is treated as a potential rather than as something fixed. So the meaning of the sign is realized in context through its combination with other elements. A  bright red used on a Fairtrade coffee package would be unlikely to mean ‘danger’, as we see in Figure  0.1. Multimodal analysis would also want to ask more carefully what kind of shade, purity or concentration of red we were dealing with. And it would want to observe how and why that red had been deployed in the particular design. For example, was it used to create coherence across the design to link an element in an image to a logo? Such an approach would be not only mindful of the different meaning potentials carried by red but also open to how this interacted with the meaning potentials of all other elements. Here context is important. Context concerns how these features and qualities are used in combination, and how they are harnessed in specific domains and serve the interest of some, but not other, groups.

5

6

6

Introduction to Multimodal Analysis

Multimodality was a term which emerged particularly from the work of Kress and Van Leeuwen (1996, 2001). These scholars were highly influenced by both linguistics and earlier forms of semiotics. Through this combination they were able to offer something that promised to be highly distinctive. And since this time multimodality has grown massively into a field in its own right. Kress and Van Leeuwen’s work has been particularly influential in language-based disciplines where the study of the visual has been largely neglected. In the field of linguistics the focus of meaning making has, of course, been language. So, for example, the study of an advertisement might reveal the use of linguistic devices such as directives (commands) and neologisms (rhyming), which are used to sell products. But of course, advertisements, with the exception of radio, tend to also carry some kind of visual communication. Here meaning is created by the way the language interacts with the visual elements, as we see in Figure  0.1. In such a case it would appear crazy to hope to capture the way that the advertisement worked by language analysis alone. Kress and Van Leeuwen were concerned to stress the importance of moving away from what they called a monomodal approach, that is, carrying out analysis solely on the basis of the mode of language. This was, of course, a challenge to how academic work tends to get done where departments tend to specialize in different modes, such as linguistics, photography, music, film studies and art. To many outside of language-based disciplines, this recommendation to consider the visual in our analysis may seem rather odd. In fields such as media and communication, cultural studies, art history, film or photographic studies, the visual has long been considered to be the central focus for study, and many excellent books brim with ideas and methods for its analysis. And perhaps one criticism that could be levelled at those based in linguistics who work with multimodality is that they tend not to engage with such literature. But what multimodality brings that is different, and highly useful for all researchers of the visual, is the idea that all forms of communication need to be approached with the same attention to detail as has been established in linguistics. While linguists are often unaware of much academic work in the visual, the opposite lack of engagement in linguistics, in more systematic procedures for analysing language, is true of many media, cultural studies and film researchers. Those who work with more structured and systematic approaches to language may feel that much of the work they produce is rather impressionistic and carried out without any real toolkit.

7

Introduction

Just as linguistics offers a more or less precise, systematic methodology for describing and analysing language, Kress and Van Leeuwen, and the multimodal scholars inspired by their work, seek to develop the same kind of systematic toolkit for analysing visual communication. This would equip the researcher with greater powers to describe and document what we see rather than simply referring to the effects of visual elements through adjectives like ‘authentic’ or ‘rustic’. In their earlier work Kress and Van Leeuwen were of the view that models and concepts from linguistics could be easily transported to be used in visual analysis. They used terms like ‘visual syntax’ and ‘visual grammar’ and looked for things like the visual equivalent of verbs. For example, a photograph or drawing of a woman hitting a man would be said to be the visual version of the sentence ‘the woman hits the man’. But these authors quickly moved away from this idea. Yet other kinds of concepts, drawn from models of language, have proved durable. We look at these in Chapter 1 and use them to build toolkits throughout the book, placing these in what we view as a model which represents the direction multimodality has developed since Kress and Van Leeuwen’s earlier work.

The social uses of design features Multimodality has developed into many sub-fields (see Jewitt et al., 2016), and some of these have remained more committed to the idea of modelling a grammar of visual communication in the fashion of linguistics. This book does not provide a detailed account of these different fields but it is helpful to understand one thing that lies at the basis of multimodal analysis: Hallidayan linguistics. Multimodality is greatly influenced by the work of Halliday (1978, 1985) who was interested in the social use of language. This differed from other theories of language which were focused on accounting for a more rigid, or formal, grammar. Instead Halliday approached language as a system of choices, or alternatives. So it is not so much something fixed in grammar, but more fluid and used by people creatively in contexts, like a set of tools. And there was emphasis on how language is shaped through its cultural, historical and social uses. In this sense it not only is about neutral communication but also carries the meanings and values of societies in particular moments in time. We can see how this idea could apply to the way we have been talking about visual communication earlier in this introduction, where the visual

7

8

8

Introduction to Multimodal Analysis

designer of the coffee packages makes choices as regards colour, typeface and type of image in order to meet their communicative aim in this particular context. Such choices have meaning due to their potential in a particular social and cultural moment. A monochrome picture of a peasant may have the potential to mean ‘authentic’, ‘honest’, ‘hard working’, in twenty-firstcentury northern Europe. But this may not have been the case 100 years ago, or in other parts of the world in the present. So Halliday’s linguistics has a highly social and functional side to it. Halliday’s linguistics also has a highly systemic side to it. He sees language as an overall system of choices which comprise the tools we apply in contexts according to our needs. And this larger system is made up of layers of smaller subsystems which build into the whole. The smaller subsystems are things like verbs, or words for indicating levels of certainty such as ‘It is the case’ or ‘It may be the case’. Those in linguistics who work with Halliday’s ideas seek to create entire models of these systems which they call ‘systemic networks’. This drive for identifying systems in communication can be seen in the work of scholars who are more interested in modelling underlying grammar forms of communication such as visual design than looking at how it is used in specific instances of communication, driven by specific interests and in specific times and places, for answering research questions, as we are in this book. These systemic-oriented multimodal scholars are also highly influenced by Halliday’s notions of ‘language metafunctions’. These are simply the three functions on which language is based. So, language has to communicate ideas (the ideational metafunction), it has to signal our relationships to ideas and persons (the interpersonal metafunction) and marshal these into a structured whole (the textual metafunction) (Halliday and Hasan, 1989). For the systemic-oriented multimodal researcher the idea is that the analyst looks for the underlying grammatical systems which realize these metafunctions. The main drive here is to model the systems of visual communication. However, for some, these notions of systems and metafunctions simply become problematic when applied to other forms of communication beyond language (Ledin and Machin, 2018). While ambitious readers of this book may want to explore these differences in approaches to multimodality, what is necessary at this point is simply to grasp the subtle difference in emphasis between wanting to map systems and wanting to look at the social uses of communication. The work of Kress and Van Leeuwen has also been heavily driven by the social part of Halliday. Here visual communication can tell us something about the contexts where it was produced, about social relations, about

9

Introduction

ideology and the kinds of motivated ideas that are being shared. In the example of our coffee packaging in Figure 0.1, we can see that the smiling peasants here have been stylized so that they fuse nicely into the design as a whole, aligning with the kinds of ‘creative’ fonts and modernistic flat panels of colour. Fairtrade products are seen by some to be rather odd in some ways (Cliath, 2007; Low and Davenport, 2005). Here, structural, global economic inequalities, built into a system of international regulation that favours corporations and powerful economies, can be addressed by middle-class consumers, through an act of shopping. The peasant appears as appreciative, attractive, out of context from their actual situation of labour and part of a chic modernist design. In this case the semiotic choices must be understood in this particular socio-historical moment. And when these four packages sit on the shelf, the peasants from different parts of the world, from Peru and Tanzania, all strike similar poses and are placed and framed in the same way on the package. Despite their massively unique positions in the global economies and colonial history, their difference is hidden in this design. A single distant ethnic other identity is presented for the consumer where difference is marked only by type of hat. This social part of the work of Kress and Van Leeuwen was also influenced by the work of the Marxist linguist Voloshinov (1986), which stressed that language must never be studied in abstract or in an unhistorical manner and that it has a material basis. Signs or multimodal texts are part of the material (and social) world. The meaning of words is part of a struggle over definition of reality where the powerful in society will seek control over this process. Signs are never fixed but have affordances which are always realized in communicative interactions, which will carry traces of the power relations underlying them. The same idea, as we see in the case of the coffee packaging, can be applied to choices in visual communication. We return to some of these concepts in the following chapter, but here it is important to note that the kind of multimodality presented in this book is very much a social one. As you will see, there is an element of ‘system’ involved in that we are interested in documenting and presenting available choices for creating visual communication, which in turn provide us with the tools to carry out much more detailed analysis. But here we present these rather as ‘inventories’ than as any form of system. And it will be clear that this is in the first place a social form of analysis. We present these inventories so that we can draw out the meanings and ideologies in the instances of visual communication that we seek to analyse. In the next chapter we provide a clear account of how we must understand these inventories of choices as

9

10

10

Introduction to Multimodal Analysis

part of whole instance of communication that are already embedded in parts of everyday life and how it is meaningful to us, rather than as more abstract systems. The Fairtrade packages seen in Figure 0.1 are not encountered by the shopper through individual choices rooted in broader abstract systems of grammar but as material objects already meaningful in the world as part of moral statements against corporate exploitation and cynicism and as part of particular expressions of taste and agency.

Outline of the chapters Chapter  1: This chapter deals with the theoretical basis of the book. It explains how we must view communication as based on choices deployed in contexts for specific communicative purposes. But such choices are part of routinized patterns and configurations which are already fused into patterns of behaviour and social organization. In other words, multimodal text analysis must include awareness of contexts and typical patterns of use. We explain also how there have been huge shifts in the nature of communication that we must grasp in order to understand contemporary designs. At heart this is about the increase in control and attention to the detail in semiotic materials and choices. This relates to the drive to control communication and also to regulate and treat all things as commodities, even ideas and feelings. And we see a new form of design emerging as a result, one which relies less on running text and more on symbolism. Chapter 2: This chapter deals with iconography. Here we look in greater detail at different kinds of visual elements that can transport meaning, at the ‘hidden meanings’ of pictures and images. We must familiarize ourselves, when looking at compositions, with the process of identifying these in order to understand the kinds of discourses that they connote. We look at objects, settings, persons and photogenia. Chapter  3: This chapter deals with modality. Modality relates to how we are positioned to experience the world as it is represented in visual communication. It is a concept that can be used to think about how real, or not, a representation claims to be. In a visual composition we can ask which elements are real, which are less than real and which are more than real. This analysis can reveal how certain things can be concealed or even enhanced in a composition. We look at different types of modality. A flow chart claims a different kind of truth than a photograph, for example. We end the chapter looking at how images can position the viewer.

11

Introduction

Chapter 4: This chapter is about colour. Most of us are used to thinking about the way that individual colours have particular meaning – red for danger and so on. Interior design programmes tell us that blues can be cold, purples are sensual and yellow is happy. But such observations are impressionistic and not systematic. Also there is far more to colour than hue. Colours can seem flat and be of equal intensity across a single surface, as we might find in cartoons. Or we may see all the subtle nuances of colour that we find when light and shadow play on objects in the real world. These and many other qualities have meaning potential. In this chapter we present an inventory of colour. We also look at the different kinds of communicative work that colours can do. Chapter 5: This chapter deals with the meaning of typography. It provides an inventory of the meaning potential of different kinds of letter shapes and dimensions. Again this draws on metaphorical association. Typographic form can also be used to fulfil different kinds of communicative function. Chapter 6: This chapter deals with textures and materials. Designers now attend much more to textures, such as in food packaging, interior design and transport. Textures are also now used in a way that more carefully links with the use of other semiotic resources such as colour. Here we show that textures can be broken down into a set of qualities that allow us to more carefully document and analyse how they can be used to communicate. The meaning of textures is deeply related to materials. The chapter ends with a look at how we can ascribe some common meanings to different materials. Chapter 7: This chapter deals with visual composition. Semiotic materials tend to be used in different kinds of typical configurations. So they must be analysed as part of these configurations and in contexts. Nevertheless, we show in this chapter that there are some patterns in composition which we can observe running across such configurations. Here we look at creating salience, at the use of borders and framings as part of processes of separation, linking and classification. We also look at text alignment and the placing of images in design. Chapter 8: In this chapter we look at diagrams. Here we are interested in how graphic shapes can be used to classify and to represent processes, persons and social and work relationships. We look at how this is a case of what we call ‘integrated design’, where what people do and why may not be clearly represented in running text but may be symbolized at different levels. This may be a useful tool for those who wish to shape how certain process or power relationships are represented. Such diagrams, simply produced using software, now appear in many forms of routine everyday communication. Here we explain how to break them down and understand how they create meaning.

11

12

13

1 What is multimodal analysis? This chapter presents the basic theoretical framework for the chapters that follow. Here we explain the nature of the approach and tools this book presents. What we present is a theory of visual communication which is linked to how society and power operate. It is about the way that visual materials come to house our ideas and values and shape and structure how we act and interact. We also place this into some of the shifts that have taken place in visual communication, particularly in the second decade of the twenty-first century. This is part of the key to revealing how the elements and features of visual communication have become used in new and more integrated configurations.

The idea of choice in communication As we began to explain in the introduction to this book, the approach we use is inspired by a theory of communication in linguistics pioneered by Halliday (1978). Halliday himself called this a ‘social semiotic’ theory of communication. You may have come across the term ‘semiotics’ (Barthes, 1977) in other forms of visual analysis where it has been used to look at things like symbolism in advertisements. So we might look at the webpage for the plastic surgery company seen in Figure 1.1 and say that the body we see as the main element on the page connotes sleek, glowing, beauty. Barthes was interested in the way that such connotations could then be associated with the product or service being advertised. In this case plastic surgery is not about medical procedures, grasping at lost youth, body management or alteration. The surgery itself is omitted and the fantasy of the outcome foregrounded. The semiotics of Barthes may also then look at the meanings of other elements on the design, such as the abstract drawing of the figure to

14

14

Introduction to Multimodal Analysis

Figure 1.1  Plastic surgery website, California Surgical institute (http://www. californiasurgicalinstitute.com/).

the top left, connoting something artistic, or ‘fine art’. But these are just two parts of the whole design. Social semiotics is different from this however, as it wants a more systematic form of analysis. We need to look at the whole of the composition. And this is very much influenced by the tradition of linguistics where we do not try to grasp the meaning of a sentence only by attending to the connotations of one single word in it. We would need to understand how it was part of a whole, how its meaning was realized through its placement with other words and tenses and where it was positioned in that sentence. In the case of the plastic surgery webpage in Figure 1.1, it is clear that the fonts are important. They are formed by quite fine lines emphasizing lightness rather than wider lines to suggest weight and substantiality. They appear rather tall and elegant rather than short and squat. The colours emphasize brightness rather than darkness. There is a lot of empty space on the page, suggesting ease and room to breathe and simply the luxury of space. A  state-run hospital may place less emphasis on space and luxury. While many state-run hospitals are also now heavily branded, foregrounded in some images may be machines used for medical procedures and more use of space to show these in order to indicate ‘resources’. The private clinic here does not represent these at all, but rather luxury, ease, lightness and optimism. The plastic surgery clinic also uses a particular kind of level of realism in its representation of the body and of the setting, which is an empty space.

15

What is Multimodal Analysis?

Semiotic choices The notion of choice is important in how a social semiotic approach deals with what is taking place here. The designer of the webpage (seen in Figure 1.1) had a choice as regards the thickness of the typeface, the kinds of colours (tones, saturation and purity levels), levels of brightness, where to put the borders on the page, how thick and what shape they should be and where to locate the cut-lines and the words. Such choices have been made here with a sense of the way that people will understand such a combination in this particular socio-historical present and in regard to what would be the target group of the clinic. Social semiotics is a theory of language and communication based on the idea of choices between available options. So in order to carry our an analysis of the plastic surgery clinic webpage we would be interested in describing the choices and meaning potentials of semiotic resources such as fonts, colours and uses of borders. We could, for example, ask if the font was curved, angular, narrow or wide. Or what meanings dilute rather than saturated colours create. Choosing the thinner typefaces or the dilute colours on the webpage in Figure 1.1 can provide messages about a kind of experience of plastic surgery – certainly not one that foregrounds surgical instruments, cutting of actual flesh, removal of unwanted body parts. By a ‘system of choices’ we mean that in, for example, language, when, as speakers, we want to describe a person, thing, place or idea, we have a range of options we can choose from. Our choice will depend on our needs in any particular context. For example, if we describe a person there are choices relating not only to size and shape but also to things like religion and ethnicity. Such words, in one sense, do not really ‘describe’ that person in any neutral sense but allow us to use culturally evolved terms to ‘evoke’ things about them from a particular point of view. And, of course, the choices we make will reflect what we want to achieve in that context – such as persuade someone to like or dislike a person. In Halliday’s social semiotic model of language, which was a huge inspiration for multimodality, the analyst would be interested in making an inventory of the kinds of choices available to speakers. So, for example, we could create an inventory of the choices available for indicating levels of certainty. I am doing that I will do that

15

16

16

Introduction to Multimodal Analysis

I must do that I may do that I would do that I could do that Making such an inventory, and understanding the meaning potential for each, allows us to better look for the instances of choices made in, say, speech or a text. If a politician comments that ‘we must act for the good of the nation’, it is not the same level of commitment as saying ‘we will act for the good of the nation’. As such, politicians often hedge their promises and commitments carefully. This notion of choice was taken up in multimodality to enable a more systematic and predictive form of analysis. This approach based on creating inventories of choices was different than some earlier forms of visual analysis that had been based more on open forms of analysis of individual elements in visual communication. In media and communication and cultural studies, there had been great reliance on the approach of Roland Barthes (1977), for example, which we began to discuss in the introduction, who looked at things like the meaning of individual elements in advertisements rather than working from an inventory of meaning potentials. In multimodality, we would be interested in all of the design choices found on an advertisement. This would include fonts, colours, the layout of the design, its shape and texture. For example, Van Leeuwen (2005) has shown that fonts can be described as regards a limited range of qualities such as curvature versus angularity, narrow versus wide, heavy versus light. Each quality can communicate quite specific ideas. Lighter and more vertically oriented fonts might be used to communicate a diet food, for example.

Materials and affordances In multimodality, as well as an interest in the smaller level choices, such as fonts, there is also an interest in the form of communication that we think of as the ‘semiotic materials’ (Ledin and Machin, 2018). When we encounter and interact with instances of communication, such as the plastic surgery website, or say the café interior pictured in Figure 1.2, we certainly do not attend to them as regards the individual smaller choices described above, even though these may be part of our experience of them. Rather, we

17

What is Multimodal Analysis?

Figure 1.2  Interior of Heritage Bicycles Café. Image from ‘Food & Wine – Hybrid Coffee Shops’ by Michael Salvatore, Heritage Bicycles Blog, 4 December 2014.

encounter them as whole things which we can think of as semiotic materials. This is important for the process of analysis that this book allows. It allows us to be mindful of the reason the instance of communication is used and also how it is in a sense already meaningful to us. When we enter a trendy café in Stockholm or London to buy a Fairtrade cappuccino and organic croissant, like the Fairtrade coffee package we discussed in the introduction, we already know about things like cafés, global inequality, chic style, climate change and so on. And we know that shopping for ethical products can be a reasonable way to express moral and political opinions  – even though many analysts would question whether this can actually ever bring about any significant changes.

17

18

18

Introduction to Multimodal Analysis

If, to begin with, we take the case of the Fairtrade food package seen in Figure 0.1, this is a form of communication that has evolved to do specific things as processes of commodification, standardization and consumerism have progressed since the end of the nineteenth century. Things like monuments, Facebook, cafés and food packages are all semiotic materials which have evolved to do very different kinds of things. This is because semiotic materials themselves also come loaded with ideas and assumptions, sometimes called affordances (Gibson, 1979), and shape communication and social behaviour. So packaging shapes how we communicate about food, and also here about ethical behaviour and fashion. The café in Figure 1.2 is loaded with assumptions about style and ethics, and it also shapes how we behave. As we see, it has rough wooden benches and tables, communicating something authentic and unprocessed, as does the stripped back industrial look of other parts of the café. But the tables are also designed so that people can sit and work alone at their computer or in small groups. They are for flexible urban living. In the case of the package and the café, these build into our lives meanings and values regarding how we understand and relate to the world and other people in it. In this sense semiotic materials shape social organization and social interactions. They allow different things to be accomplished, and each shapes what can be done in any instance. A stone monument is good for showing something is deeply important. A webpage would not do the trick, nor would a café. But if you wanted to share information with people around the world, the stone monument would be less useful.

Discourse In semiotics it is common to talk about the way that signs, or semiotic choices can connote particular ideas about the world, in other words, the different meanings they can transport. For example, a flag can connote ideas associated with nationalism and patriotism. A  flag like the British Union Jack might connote unity and strength to some. A  person might display it in their garden or on their T-shirt, to connote a nation that had a large, proud empire, which had a favourable outcome in two large-scale wars in the twentieth century. Of course, to others the same Union Jack might instead connote racism and intolerance as its use is often associated with far right groups or at least with conservative views of nationhood and

19

What is Multimodal Analysis?

retrospective imagined national pride rather than a more open-minded cosmopolitanism and internationalism. In either case, in the use of the flag as with the use of all national flags, there is a transport of meaning. Therefore, this material object is able to connote complex ideas about the nature of the world and the people in it – whether they are to be considered citizens under that flag, and exactly what associations that belonging brings with it. Following Foucault (1977, 1980) we think of these complex ideas as ‘discourses’. Put simply, this means chunks of knowledge or ideas about the way the world works and the way people in that world behave. In media and communications studies, the term ‘discourse’ is often used for a confusing range of phenomena. For example, we might read about ‘television discourse’ or ‘news discourse’. It is hard to imagine what ‘discourse’ could have in common in these cases. Discourse is also sometimes used to mean talk itself or a piece of writing or text. But here we will use it to refer to socially constructed knowledge about the world. Union Jack is one semiotic material that is able to connote this glorious past. But more complex combinations of materials can be used to the same effect and to introduce more meanings, such as national monuments, buildings, advertisements for products which are made in that country. Hobsbawm and Ranger (1992) have described how national war monuments were constructed around Britain after the First World War as a way of promoting the idea of nation and national interest at a time when, in countries to the east, workers’ movements were drawing the attention of the working classes to move against the ruling classes. The British working classes had been treated appallingly during and after the war, about which many were angry. They were much closer to revolution than is generally depicted in conventional British history (Miliband, 1973; Mahon 1976). So, using monuments the British government was promoting a particular discourse about the nature of British society through the use of semiotic materials deployed in public squares in cities, towns and villages. These would carry religious imagery, perhaps a cross, and then inscriptions that might mention bravery, sacrifice and sons of the nation that gave in the name of King and country. The statues would often appear in the form of classical sculpture pointing to high ideals and values. Materials would be bronze, suggesting hand forged, or stone, durable like mountains. Clearly in such cases plastic or a hollow tin structure would have different connotations. In this case the slaughter of millions of young men and civilians in the name

19

20

20

Introduction to Multimodal Analysis

of colonial profiteering becomes represented as something about eternity, shared interest, high ideals, dignity and Christian values. In this book we present toolkits in each chapter which can be used to reveal the discourses which are communicated. So how does a food package or a café communicate discourse? What kinds of discourses are these and whose interests do they serve? We saw in the introduction that the coffee package communicated discourses about happy, authentic, peasants along with chic design – a discourse where we can solve the inequalities of global capitalism through acts of shopping.

Discourses and social practices How we represent the world in semiotic materials is also related to social practices (Fairclough, 1992). It is not simply that things like food packages and adverts carry representations. Rather these form our material world and are part of the way that we act in it. The Fairtrade coffee package shapes how we consume and how we act towards others, how we think about ourselves. The café structures our day, gives it particular kinds of meaning. And both lay out different kinds of social relations. In regard to the packaging, the consumer is benevolent and moral. In the café a media worker checks their email in a chic, industrial stripped back space designed for such individual, fast activities. As they do this, they can make moral shopping decisions too. In this sense discourses do not exist only in language or in visual representations but are coded into all of material culture. That means all of the objects in the setting where you now find yourself are so coded. All of the examples we look at throughout this book therefore carry traces of discourses. These not only are abstract ideas about the world but also form the basis for what we do, how we plan and organize and the structures we have created in order to do so. We live these discourses out when we act and think. Semiotic materials house our ideas, values, identities and templates for social interactions. But of course, just like language, all semiotic materials are loaded with the power interests of the times that they evolved (Voloshinov, 1986). Just like the Union Jack flag and the café, these are not neutral. In this book we look at examples of food packaging, school quality improvement documents, design of rooms and flow charts. All of these shape how we think, behave and interact with others.

21

What is Multimodal Analysis?

The shift to multimodal communication Kress and Van Leeuwen (2001) argued that the way we are communicating is changing. Many people seem to think that communication is becoming more visual. But perhaps what is really the case is that the previously segregated role of the different semiotic modes has changed. Formerly we used the different modes in isolation. There was a preference for ‘monomodality’. So writing came without illustrations, art used only the visual on a flat canvas. These became institutionalized into formal disciplines based on the single modes, literary studies and art criticism. But this has changed. The changes in the roles of the different modes can be seen in schoolbooks, on magazine pages, in advertisements and even on the bills that come through your door. If you are able to compare a current electricity bill to one from fifteen years ago you will see a change from boring functional typeface to the use of logos, colour, more interesting fonts, text boxes and frames. Many of these compositional elements are designed to help you to read and understand your bill, whereas previously this was left in the hands of the wording alone. They are also intended to create a different mood, to address you in a friendlier manner. Of course, when you fail to pay, later letters may use different visual cues to indicate a more serious tone. These changes have become possible not only because of people thinking about the way they can use the different modes but also because technology has made it simpler. One person can now sit at a computer and with very little effort search and paste in different combinations and arrangements of text, image and illustration. In the places where we work, it is common to find routine documents carrying colours, graphics and images.

Technologization This attention to precision in design can be captured through Fairclough’s (1992) notion of ‘technologization’. This is important for how we think about visual communication in this book. Towards the end of the twentieth century, he observed a shift in the use and control of language which he called the ‘technologization of discourse’. This referred to the increase

21

22

22

Introduction to Multimodal Analysis

in the control and shaping of language in order to fulfil economic and political objectives. This leads to a standardization and codification of the resources of communication. For example, teachers must codify learning and assessment, not on the basis of pedagogical theory or expertise, but driven by policy in order to maximize outputs (Hopmann, 2008). This codified language then takes over from that formerly used by professionals to carry out and plan their work. Technologization was part of the increase in ‘commodification’ or ‘marketization’ throughout society where all parts of social life become ‘organized in terms of production, distribution and consumption’ (Fairclough, 1992: 207). It has also been shown that this technologization of discourse has also taken place in visual communication, where, since the 1990s and along with the rise of software, there has been a huge increase in the control and shaping of visual design in order to fulfil economic, political and institutional objectives (Machin and Polzer, 2015). Media outlets, corporations and public institutions invest care into coding ideas and values into visual content. The management of the institutions where we work will invest time and money to create documents which engage us visually in very specific ways.

Integrated design In fact it is not only that communication has become more multimodal. Something else has taken place which is of high importance for how we carry out analysis in this book. In communication the roles formerly taken by writing have often been taken over by other semiotic resources. In the example of the ‘Bounce’ package in Figure  1.3, we are told about the effects of the ‘energy’ in the product not in language but by the picture of jumping figure and also by the rounded, bouncy-looking font. In fact ‘high-energy’ products are those that contain a higher number of joules of energy. This can mean sugar. This may come from ‘natural’ sources, say concentrated grape juice. But it is still sugar. Often products which relate to health, fitness, caring for the planet make claims not through language, but through images, fonts, colours and textures. The Bounce package is slightly rough in texture, suggesting something more natural. The form of the product itself is rougher as if handmade. There is lots of bright, clean, empty space on the package. Many of the fonts are very slim. While the product itself looks roughly made, the packaging is very different. It is more chic and modern.

23

What is Multimodal Analysis?

Figure 1.3  Bounce healthy snack package and product. Bounce Foods Ltd.

In fact, the nutritional content of Bounce is very much like that of a traditional snack such as a Mars bar. So high in sugar and fat. But the nature of this product is communicated carefully through design. The language is formed by isolated words and chunks of text. It is not explained to us how the whole thing works together – how we get energy and also ‘defence’ (or even what this is) and what relation the antioxidants have to this. This has been referred to in multimodality as a form of ‘integrated design’ or ‘new writing’ (Van Leeuwen, 2008; Ledin and Machin, 2018). These concepts emphasize that there is an integration which changes the way that both language and other semiotic resources are used. Important here is the way that the kinds of causalities, identities and connections formerly made in running text are now made using combinations of chunks of text, bullet lists, combined with images, graphics, colours and other semiotic resources. We will be seeing this throughout the examples in this book. A  café may use colour or texture to link a stripped back industrial section to one that looks old and wooden. We may find a monochrome photograph of a peasant, also coded with the same colour saturation and textures. We are not told in language how the parts are interrelated, rather this linking is accomplished through the integrated design.

23

24

24

Introduction to Multimodal Analysis

And this is all highly technologized since great care and expertise can go into harnessing different semiotic resources for codification. The huge advantage of this kind of communication is that it can easily suppress, gloss over or replace complexities and contradictions. The peasant Fairtrade worker can be linked to shopping or a trendy café where you do your email and eat organic croissants each morning. Official documents and news reports may use colour, graphic elements, flow charts and images where actual important details are missing. Technologization and integrated design may be about increased control and management of semiotic materials and semiotic resources. But this is not to say that the documents produced are in any way clearer or improved. This is simply different, a new form of communication facilitated by digital technologies.

A new commodified communication setting moral standards In his account of technologization, Fairclough (1992) argued that a process called ‘commodification’ was also part of this social change. This refers to the way that all things in our lives now become coded as commodities which can be used to generate profit. In the case of the Bounce snack we can see such an example where the notion of healthiness becomes coded in a way where it can be packaged up and sold to people. In such a case the discourses of health formulated by such packaging can colonize and shape how we come to think and act in regards to taking care of our bodies. But as we showed above, this coding, in the case of the Bounce packaging, involves a process whereby exactly how this product is healthy, what it means by ‘energy’, how it works, how the different ingredients and qualities interrelate are not clear. The design of the package with the fonts, the colours and shape and texture communicates all this at a symbolic level and creates a coherent whole. This increase in commodification and codification has also been shaped and driven by the current, dominant model of governing society called neo-liberalism (Saad-Filio and Johnston, 2005). This is a complex and contested term, but it seeks to account for the way that all things in society become increasingly run along market principles. For example, schools in the 1970s tended to be organized around the idea of explorative learning and imagination. School rooms shifted away from rigid rows of desks and authoritarianism to open plan- and student-centred learning. But at the

25

What is Multimodal Analysis?

time of writing, the meaning of teaching and learning has again changed. Individual teachers and schools become subject to close performance management. Learning becomes broken down and coded into clear targets and criteria for quality improvement. Schools are under pressure to show how grades improve each year or suffer punishment, even if they operate in disadvantaged areas, where there are many challenges facing teachers and students. In this case we find commodification and codifications, driven by neo-liberal ideas about society. Forms of codifications take place to allow all processes to be measured and compared. Education, like product production, can therefore be quality measured and performances can be assessed. This commodification and codification is done differently in different social domains. But what runs throughout is that this is carried and justified most often by claiming high moral standards. The branding of a coffee as Fairtrade or sustainable, or a snack as healthy and non-GMO (non-genetically modified organisms), states that by consuming we can act ethically and morally. We are helping to fight inequality and saving the planet, however complex and structural such problems in fact are. A new quality performance system of a preschool will claim that quality improves and that the children learn more and better things. Where formerly children’s development was primarily assessed by the teacher, they must now be managed to show that they are meeting national goals. It would be hard to oppose such a thing as the nurturing of children is a moral and ethical issue. One author has access to an app which allows him to see how his five-year-old’s son’s play in a woodland becomes coded into many parts relating to scientific discovery, social competence, emotional development. On one level it is comical, but it is also challenging, since it suggests that a person is not taking the right moral stance. It is integrated designs which allow this moral dimension to be coded. One result of this is what something we, using Habermas (1984), call ‘communicative rationality’ ceases to take place. We can see this to some extent in the case of the design of the Bounce snack bar. How the different ingredients work together, how this gives energy and provides a ‘defence boost’ is not explained in the kind of terms we might expect in communicative rationality. Linking and coherence here is provided by the types of font that are used, the bright florescent colours which have white running through them. Communicative rationality means that we seek a mutual understanding of something together, such as discussing different priorities or problems. In doing so we believe the other to be truthful and

25

26

26

Introduction to Multimodal Analysis

use a language that is comprehensible and can contribute to an agreement achieved through communication. On food packages like Bounce this is not the case. In one sense the idea of ‘communicative rationality’, we should point out, is somewhat naïve (Calhoun, 1992). Of course running text can also be persuasive, manipulative, metaphorical and symbolic. The aim of critical discourse analysis has precisely been to reveal such processes where they are less obvious in texts. What Habermas means is that we should be alarmed when communicative rationality is set aside and when we cease to strive for a communication where the priorities and understandings of people involved can be clearly articulated. Here we use this concept to help us to begin to characterize and discuss what is taking place in these integrated designs, because they tend to conceal, obscure and abstract everyday experiences using a new form of visual language. Shortly we show we can more meaningfully complete this account with the help of other concepts. We can see more clearly how communicative rationality breaks down, looking the example of a document produced by the Swedish School Inspectorate shown in Figure 1.4. In this document the Swedish School Inspectorate reported that half of the preschools (ages 3 months to 5 years) where it carried out evaluations failed to delivering proper teaching. In Figure 1.4, we see that three levels of quality have been presented and represented through a diagram, an integrated design. The quality improves as we move down the levels. In each case we find a summary of what is taking place and a graphic to illustrate. In the first case we are told the school does little in terms of steering development and children’s own interests govern how they are stimulated and challenged. This fits with an earlier model promoted in Sweden called ‘free play’. In the second case the children are stimulated and challenged by teachers but without any clear goals (which are provided by at a national level dealing with intellectual, social and emotional development). In the third and best case, stimulation and challenging is done clearly in the context ‘of the goal the preschool is seeking’. In simple terms, where the author’s son and his best friend spend an hour throwing fallen leaves at each other during their day at preschool, this must be directed in a way which creates stimulation and challenges to fulfil national goals and targets. In the diagrams, the goals are symbolized to the right, with green dots of different sizes. As we see the colour gets more saturated as we move down the page, which indicates increasing quality. In fact, in the top diagram there is just a question mark in the middle of the pale dots. The caption

27

What is Multimodal Analysis? Barnets villkor för utveckling och lärande illustrerat med tre figurer.

Barn Barnets intresse

?

?

Förskollärare Figur 1. Barnet intresserar sig för något och stimuleras och utmanas genom lek, miljö och material samt andra barn. Förskollärarsen stimulerar och utmanar inte lärandet. Det är osäkert i vilken riktning barnets utveckling går.

Barn Barnets intresse

Stimulera och utmana lärandet

Málatt sträva mot?

Förskollärare Figur 2. Barnet intresserar sig för något och stimuleras och utmanas genom lek, miljö och material samt andra barn. Förskollärarsen stimulerar och utmanar inte lärandet generellt utifrån barnets intresse dock inte medvetet mot de mål som förskolan ska sträva efter. Det är osäkert om barnets utveckling sker i riktning mot målen.

Barn Barnets intresse

Stimulera och utmana lärandet

Málatt sträva mot

Förskollärare Figur 3. Barnet intresserar sig för något och stimuleras och utmanas genom lek, miljö och material samt andra barn. Förskollärarsen undervisar, det vill säga, stimulerar och utmanar utifrån barnets intresse men med riktning mot de mål som förskolan ska sträva efter. Barnet ges möjlighet att utvecklas i riktning mot målen.

Figure 1.4  Diagrams used by Swedish School Inspectorate to compare good and bad preschools. From ‛Förskolans pedagogiska uppdrag’, Skolinspektionen 2016, p. 13. Available at: https://www.skolinspektionen.se/globalassets/ publikationssok/granskningsrapporter/kvalitetsgranskningar/2016/forskolanped-uppdrag/rapport-forskolans-pedagogiska-uppdrag.pdf.

starts as:  ‘The children take interest in something and are stimulated and challenged through play, settings and materials’ (note that the circle with blue dots is labelled ‘the interest of the child’). Here the older model of free play is shown to be misjudged by the Swedish Schools Inspectorate, because there is no clear goal alignment  – the caption ends:  ‘It is unclear which direction the development of the child takes.’ And we see that the diagram represents this failure to meet the goal, as the pale green colour suggests. The third level portrays successful teaching, as seen from the strong and

27

28

28

Introduction to Multimodal Analysis

saturated green. Here the learning activities align exactly with goals. This is also stated in bold typeface in the caption, where the development unfolds ‘in direction towards the goals’. We also see on the diagram that the children (‘barn’) and the preschool teachers (‘förskolelärare’) are represented by same size arrows. When things are working well, these must have a two-way arrow between them and be of the same shade of colour. How these are the same and what the two-way arrow represents is not clear. The process of what takes place in a preschool has now been codified and commodified and presented through graphic shapes and shades and hues of colours. Causalities are represented through arrow shapes. But what managing and challenging children to meet the goals actually means as a process, and what the result is for the children being quality developed, is not made clear. But the result is incredible levels of codifications of mundane yet fun acts like throwing autumn leaves at your friend. Teachers must show how they can take these observations further to demonstrate continued increases in quality. We might accept that it could be valuable for there to be some set goals in a preschool, for example, to learn to share toys, to recognize numbers, to take some responsibilities such as for washing one’s hands. Habermas (1984) would call the use of such goals an ‘instrumental rationality’. It is a rationality based on reaching a particular outcome. A  good example of instrumental rationality would be where a doctor would make a diagnosis and take measures that are apt for that diagnosis, such as prescribing a medicine. Habermas sees instrumental rationality as part of modern societies. But this must interplay with communicative rationality and with the priorities and perceptions we have in our everyday lives. Communicative rationality means that we seek a mutual understanding of something together, such as discussing different priorities or problems. In doing so we believe the other to be truthful and use a language that is comprehensible and can contribute to an agreement achieved through communication. In the case of the Bounce package and the preschool diagram, what is being measured exactly, what is true, desirable, a good course of action to take, and so on tends to be set aside. We meet a communication that abstracts and fragments what people actually do in actual contexts, including their motives for doing so. If this form of communication and its instrumental rationality no longer interplay with communicative rationality, Habermas would say that this instrumental and visual language enforces ‘violent abstractions’. We argue that this is the case in the diagrams from the Swedish School Inspectorate.

29

What is Multimodal Analysis?

And all forms of classifications, Bernstein (2000) reminds us, are about power. They are about the power to define how things are, in terms of the parts of education or attaining health. Bernstein’s interest lay specifically in how classifications are re-contextualized in pedagogical communication. The diagram in Figure 1.4 is one example, where new classifications, laden with power and ideology, determine what forms of communications are valid, what behaviour counts as appropriate and why. This means that new classifications are imposed upon teachers. Formerly the teachers were in control of the pedagogical communication, which relied on the professional judgement of teachers. But now ways of communicating and teaching are specified in detail, and there is huge pressure on teachers to adapt to this. In the case of the Bounce bar healthy snack and other foods such as those labelled as sustainable, organic or Fairtrade, we are clearly dealing with sales and marketing where the point is obviously to make profits. In media and communication studies it has long been established that this means that consumption is linked to certain ideals: we want to look good, to be or show that we are successful, belong to a certain group, signal our identity and lifestyle, and all this is to a large extent achieved through acts of consumption. What has changed the last decade is that consumption takes on moral values, and today’s marketing often has political dimensions, such as saving the planet or helping poor people in remote parts of the world. The opposite kinds of consumer choices can be seen as immoral shopping. And in their designs such packages use images in monochrome, typefaces, colours and textures to communicate about what is natural, unmediated, environmentally friendly, all of which can be combined with signals that such things are also about being stylish, chic and modern. This new and moral consumer culture certainly transforms traditional social relations. This is achieved, Bauman (2007:  11) argues, ‘through the annexation and colonization by consumer markets of the space stretching between us’. When a branded and moral space becomes the way in which we relate to each other, make priorities and understandings of world, consumer culture has taken over the role of what once was politics, to reach a mutual understanding and take social action and vote in order to change the world. Returning to Habermas we can say that the communicative rationality, our striving to make sense of ourselves and the world in a comprehensible language, including taking part in elections, risks to be perverted and in the hands of big business. Through acts of consumption we make distinctions and invest in high moral standards as a commodity that, at least for those

29

30

30

Introduction to Multimodal Analysis

with the money needed, always can be bought. What multimodal analysis allows is for us to be able to carry out analysis of instances of communication to show how and where such processes are taking place.

Summary textbox of concepts for what is multimodality Semiotic choice: communication is done by making choices from a pre-existing repertoire which are assembled together into semiotic materials. Choices are made in order to shape the meaning of the particular instance of semiotic material. Semiotic material:  these are the actual objects and things which the semiotic choices form. Semiotic materials shape social organization and social interactions. A monument shapes activities and interactions differently than a letter or a piece of music. Material affordances:  semiotic materials each fulfil different communicative aims and have evolved to do so. These lay out how communication can take place and what can, and cannot, be done. Discourses:  these are ideas about how and why thinks occur and can reflect particular power interests in society. Semiotic materials and semiotic choices can be deployed in order to shape how ideas and forms of social relations are embedded into our everyday environments. Social practices: these are the doings and forms of organization in society which are based on discourses. Semiotic materials can play a role in shaping how we act what can and what cannot be done and the ideas that this carries. Technologization: all forms of communication are becoming more and more coded and used in systematic ways. This has been driven by the marketization and commodification, as well as new technologies, with the need to communicate more accurately and code more thoroughly for specific target audiences. Integrated design:  semiotic resources are now used together in much more systematic ways. Writing too has changed and is often interrelated with images, diagrams and charts. This requires us to have new critical skills since we may be presented with designs where things like causalities, motivations, processes and the roles of participants are much less clear.

31

What is Multimodal Analysis?

Communicative rationality: to what extent do these new integrated designs lose what we would take to be rational communicative characteristics? What if we are continually communicated to not through more clearly formed arguments and explanation but through integrated designs where causalities, motivations, processes and the roles of participants are much more abstracted?

Coding of semiotic resources In this book we look at how we make meaning multimodally by the use of different semiotic resources in order to understand how we communicate through semiotic materials. What we want to clarify here is how this coding or meaningfulness of semiotic resources takes place and also how we identify such codes in order to produce the kinds of inventories that we provide throughout the chapters of the book. The coding of semiotic resources is based on three principles. These are: distinctions, contextualizations and configurations.

Distinctions All codes presuppose distinctions. Colours are meaningful for us. Yellow can mean bright and optimistic due to its metaphorical associations with sunshine and summertime. Red might mean emotional due to associations with blood and reddened faces. But in itself colour is a continuous spectrum and part of the electromagnetic spectrum of the universe. Here what is visible light for humans comprises a very limited range of wavelengths. For making meaning, this visible spectrum must somehow be divided into parts. We have to make distinctions. Naming colours in language imposes categories, and different languages have very different numbers of colour words and do this slightly differently. Colours may also be distinguished through other kinds of metaphorical associations. A pale, diluted yellow may have a very different meaning to a rich, saturated yellow. Here the coding is based on the distinction between associations of more intense versus more muted experiences, for example, of pale versus rich sunlight, of a saturated or diluted fruit drink. We can also think of shapes. With contemporary digital technologies, graphic shapes are prefabricated. In Microsoft Word, there is a drop-down

31

32

32

Introduction to Multimodal Analysis

menu for ‘shapes’ where all sorts of graphic devices are sorted into groups. Just as we divide colour up in language for particular purposes to make it possible to talk about colour, this software also makes distinctions and gives us preset choices to create diagrams or flow charts.

Contextualizations On this general level, we get basic distinctions that make codings possible. We could use experiential metaphors and say, for example, that red has basic associations. When a person becomes tired or exerted, the face can become reddened as the heart speeds up. When we are wounded, of course, we bleed. Thus, red can be associated with energy. We could say that these levels of energy may be coded by levels of saturation versus dilution. But such descriptions do not say much about why a children’s toy or a bank logo has a certain shade of red (and is not, for example, blue or brown). The exact meanings of red, or any colour, simply have to be established in relation to different domains and semiotic materials. Similarly, we can perhaps say that rectangular graphic shapes bring associations to something definitive and closed, but to uncover the exact meanings and uses of such shapes in diagrams we must look at them in context. The principle of contextualization informs us how to pin down more exact codings. Certain distinctions become relevant in certain (but not other) social contexts. A  café like Starbucks may use red as part of the colour palette used in its cafés. This may be more of an earthy-plum-red. This allows something of the warmth and emotional meanings of red to be used in ways which integrate the surfaces it covers into other parts of the design and other surfaces and objects. These may carry other colours like earthy-brown-charcoal. These colours are all hybrid and ‘creative’ yet are designed to sit together and create harmony in the fashion of the ‘integrated designs’ we described above. The bright saturated red on a baby rattle maybe about energy, but alongside other pure primary colours, it is coded for the simplicity and innocence of childhood. If we take graphic shapes, these are certainly tied to different contexts and domains. In flow charts shapes have specific meanings: a rectangular shape codes ‘process’, if the edges are rounded ‘start or stop process’, and if shaped as a diamond the meaning is ‘make yes or no decision’.

33

What is Multimodal Analysis?

Configurations What we also see from the account of the café, the chart made with the graphic shapes and the baby rattle is that things like colour and shapes, which are coded for meaning, come in different configurations. The colour used in Starbucks is designed to coordinate with the materials, textures, furnishings and layout of the café. The colours are earthy, vibrant and creative, and work with the natural textures of the tables and floors, the polished steel of the coffee machines, lighting and ventilation pipes and the softer sitting areas. The café is not about colour in itself and its meaning must be understood with regard to the configuration. In a bar chart, the main graphic shape is rectangular. But obviously the bar chart is not about rectangular shapes in themselves. The meaning of the chart cannot be found only at this level. In order to understand this and bring out the codings, we must use the principle of configuration. Bar charts often map numbers of things on the x-axis and time on the y axis. So the size of the rectangular bars is the point and makes comparisons over the years possible. So the configuration in this domain of bar charts is to combine graphics (shapes and lines with arrows) with numbers in an exact composition with an x- and y-axis. We would not simply approach this from the meaning of the shapes themselves. Shapes may be used in the pattern in a carpet in a café, or as part of a baby rattle. So semiotic resources come to us in configurations, as co-articulated in different ways in different domains. These are configurations that have become conventionalized in specific domains and contexts.

Identifying codes When carrying out social semiotic research, we must somehow find out which codings are operative in a certain domain. Doing this is based in two methodological principles: make an inventory and use the commutation test.

Inventories This means sampling data systematically from the domain you wish to research. For example, we want to research food packaging to look at foods

33

34

34

Introduction to Multimodal Analysis

which are healthy. Here we can collect a sample of healthy foods and those same brands or types of food which are not marked as ‘healthy’. We should create an inventory that allows us to feel that we have exhausted the patterns that exist. We then look the sample and see that surprisingly healthy foods seem to use more angular fonts and brighter hybrid colours. So we have established some simple distinctions. Of course, researching these packages would have to take things like materials, shape and pictures into account as these are important and usual parts of these particular configurations. All these contribute to the meanings of food packaging. In pursuing the analysis, we would discover more distinctions and be able to refine and underpin our interpretations. In order to identify and understand more about the distinctions we would use the commutation test.

The commutation test This is a process of changing something in the multimodal configuration, for example, the colour or the typeface. What would happen with the meaning of a food package if we changed the typeface? If healthy food packaging had an angular typeface and the regular product a more rounded typeface we could change this. If the meaning does change, it would strongly indicate that we had found an important distinction. It may relate to healthy products appearing more technical, precise and modern. We may find other distinctions, for example, where the healthy product had a much slimmer typeface, different colours, more space on the design. And we would need to look across the configuration to look for such distinctions. And we would need to understand the whole in relation to specific contexts where these configurations have become routine parts of shaping social practices. In summary, we can never start our analysis from one semiotic resource, such as colour or typography. Nor must we see a semiotic resource as something coded in itself, something isolated. All codings presuppose contextualizations and configurations.

Summary text box for coding Distinctions: making distinctions allows us to point to choices, for example, by dividing the colour spectrum into actual colours or

35

What is Multimodal Analysis?

distinguish thick or thin strokes as for types. Coding is based on such distinctions. Contextualizations: the exact meanings distinctions have will always be determined in context. For example, the colour red may mean very different things when placed in the context of different semiotic choices, as part of different semiotic materials and in relation to different social practices. Configurations: semiotic resources are used together and come out in configurations. The design of a café or a diagram is simply multimodal. Such configurations are for the most part well established in society and relate to social practices. Inventories: to establish distinctions and codes we must make an inventory of the semiotic resource or semiotic material we want to investigate. Commutation test: in the inventory we will test what distinctions and meaning that are operative by changing something in the multimodal configuration: the typeface, the colour, the graphic shapes, and so on.

Student activities The aim here is simply to become more familiar with applying the concepts we have looked at in this chapter. It is important that we can use these concepts not simply in a theoretical fashion but in a way that allows us to look at objects and instances of communication in the world. The task is to choose on object (semiotic material) such as a food package, a children’s toy, a university webpage, an Instagram account. You will then use the concepts listed below to talk about and help understand how this communicates and creates meaning for us: Semiotic choices: List at least four semiotic choices that have gone into this design. Using the section on ‘distinctions’, how do you know that these are semiotic choices? Discourse: In the case of two of these semiotic choices say what broader ideas are communicated by them. This could just be something like ‘solidity’, ‘fun’, ‘order’, and so on. Ideology: We can also ask here whose interests these discourses serve in terms of ideology. If this is a food package, for example, how are you being encouraged to think about a particular product? What would the ideology be if this was a social media account?

35

36

36

Introduction to Multimodal Analysis

Discursive practices: How does this object, this semiotic material, shape the way that we act or interact with others? Does it foster any specific kinds of social relations? For example, a Fairtrade coffee package may suggest certain kinds of ways of relating to people. Integrated design: Explain how different semiotic resources are used to communicate different ideas on your chosen object. Point out where the link or causality between them is not overtly explained. For example, on a university web page there may be a text which explains its commitment to sustainability alongside a picture of some smiling students. Yet it is not explained how these students relate to the issue of sustainability. Configurations: Semiotic resources are always used together and are found in configurations which we easily recognize and which relate to established social practices. Explain how this is the case for your chosen example.

37

2 Pictures and images The documents, web pages and advertisements we analyse throughout this book in terms of typography, colours, texture and composition also tend to contain pictures, images and icons of different kinds. This may be a photograph as we see in Figure  5.9 of the children’s camp website, or the peasant workers on the coffee packages in Figure  0.1. In these different cases, we see that the ways the persons are represented are very different. Decisions have been made about how to represent people to communicate specific meanings about them. We can ask why, in the case of the coffee, we do not see a larger group of workers on each package. Why just one? In the case of a Fairtrade product, as in Figure 0.1, it is thought important that the consumer feels they have a personalized relationship with the producer/worker. Having a close-up of a single, smiling, appreciative worker is one way that this can be done. We can also see that a decision has been made not to show the peasants actually working in a field or the plant where they process and package the product, but in a more decontextualized way with a blank background. On the one hand, this allows the complex production processes to be concealed. But it also has the effect of foregrounding the personal aspect of the peasant rather than the act of labour or the actual setting. In this case the decision of who to represent and how is clearly important for shaping the meaning of the Fairtrade product. On other Fairtrade packages, we may indeed see some aspects of setting. In the case of the photographs of peasant workers on the wall of Starbucks in Figure 4.2 we do see groups and we do see actual labour. Such photographs also typically show objects such as a wicker basket or rural settings. We certainly do not see machines used for processing, transport, branding work or bookkeeping. We do not see the newish German sports car one of the workers has been able to buy through reaping the benefits of Fairtrade. This is all part of the

38

38

Introduction to Multimodal Analysis

way that Fairtrade must be presented to consumers as related to authentic honest forms of manual labour somehow cut off from the modern world. It is through such representations that a packet of coffee can become part of a discourse about taking a moral stance against impersonal corporate capitalism through the social practice of shopping, or of visiting your local Starbucks for a croissant. Persons can also be represented on designs through drawings and icons, such as the jumping figure on the Bounce package in Figure 1.3 or the classical type figure to the top left of the plastic surgery web page in Figure  1.1. Here we are not so much dealing with the representation of persons but of instances of iconography that are also used to add meaning to a design. Clearly, were we to use the classic figure on the Bounce package it would change its meaning. And were we to use a photograph of a real person jumping in the air the meaning would be different again. In Figure 2.4 we see students and staff represented in a flow chart of how the university operates through 3D drawings. Here too we can ask what role this plays. How would this change the meaning were we to find photographs of actual staff and students going about their daily activities? In this chapter it is such decisions  – how people are represented, with what kinds of objects and in which kinds of settings – that are of interest to us. These can all be used in visual designs to communicate broader discourses and to place them in social practices. They can be used to give new inflections to semiotic materials already based in set social practices.

A semiotic approach: Denotation and connotation Roland Barthes (1973, 1977) was interested in the kinds of elements and features we considered above, such as how the Fairtrade workers were represented in each case. When we see a close-shot of a smiling peasant, it brings a particular set of associations for us, which the designer of the package can rely upon. A wicker basket also has associations for us. For Barthes it is important to think about how these associations came about. They are not simply given but have become developed in cultures over time. Such established associations can, of course, appear neutral, but they are not. In Western cultures it is common on food packaging and in cafes to see images of peasant workers used to communicate discourses of

39

Pictures and Images

honest authentic labour. Yet such workers find themselves at the bad end of a world where the global economic system favours the corporations of the Global North, where their own country may be crippled by International Monetary Fund structural adjustment loans which require their government to undertake massive austerity measures and to privatize public services. Barthes would want to urge us to critically assess such representations in order to draw out the kinds of ideologies that they represent. For critics of Fairtrade, such images, we might say, help us to feel that we can address these huge global issues by acts of shopping (Low and Devenport, 2005) rather than fighting to change the system. We often hear about the world’s richest 1 per cent. But in fact that 1 per cent is us and comprises the people who can afford to buy Fairtrade coffee. We begin by looking at the way that Barthes distinguishes between the literal and hidden meaning of images. This involves a simple process of observation and analysis. But it is important as it calls for us to give great care in how we describe and document what we see. Barthes (1973) offered two concepts ‘denotation’ and ‘connotation’ to describe the literal and hidden meaning of images. Below we deal with these in turn.

Denotation Some images cannot really be said to communicate a general or abstract idea. They show particular events, particular people, places and things. They ‘document’. Or in semiotic terminology, they denote. A photograph of a family member or of a house simply denotes these things. It represents a particular person and a particular place, respectively. We might say that these photographs document or denote reality. A news photograph might denote or document a political meeting or a riot. In the images in Figure 2.1, we see children in a forest, and two children and a teacher sat on a multi-coloured mat. So, asking what an image denotes is asking: who and/or what is depicted here? Of course, we never really see any image in this kind of innocent way. Images usually mean something to us. Nor are they neutral recordings of reality. In a wedding photograph you may choose to include only your friends who were dressed particularly fashionably in order to conceal that your family were basically a pretty odd bunch of people. Anyone looking at this photograph will see not just people at a wedding but rather fashionable people at a wedding. Denotation, then, still involves choices in representation.

39

40

40

Introduction to Multimodal Analysis

Figure 2.1  Unikum Quality Development in preschool (https://www.unikum. net/).

There is no neutral documentation. But denotation, for Barthes (1977), is one way to think about the first level of meaning in a photograph. To understand what we see in any image we need to know something about what we are looking at. There can be no completely naïve viewing; we are always making meaning rather than just seeing. But Barthes saw it as useful to think of this everyday knowledge as the literal rather than the symbolic message of the image.

Connotation While some images might be intended to denote and document, others will still depict particular people, places, things and events, but ‘denotation’ is not their primary or only purpose. They depict concrete people, places, things and events to get across general or abstract ideas. These are used to connote ideas and concepts. So asking what an image connotes is asking: What ideas and values are communicated through what is represented and through the way in which it is represented? Or, from the point of view of the imagemaker: How do I get general or abstract ideas across? How do I get across what events, places and things mean? What concrete signifier can I use to get a particular abstract idea across? This is Barthes’s second layer of meaning. When analysing an image at this level we can ask what the cultural associations of elements in the image are. It is these associations that connote

41

Pictures and Images

particular discourses, models of the world, scripts of likely sequences of activity, kinds of people and kinds of problems and solutions. Now we can think about the connotations of the children seen in the forest and the children sat on the mat. Figure 2.1 is a screenshot slide taken from a video presentation for an app called Unikum. The app forms a digital system which is used in schools in Sweden as part of a quality development process. Here all aspects of a preschool child’s day, including social interactions with other children, have to be measured, recorded and steps taken to ‘push this further’. In this system all kinds of activities and behaviours can be coded against specific targets. In fact, much research shows that such quality assessment is not driven by accepted pedagogical theory, is even damaging for education and eats up the time of teachers and devalues their own professional judgement (Ball, 2003). Rather this is part of a political drive, where all public services become operated on marketized principles (Ledin and Machin, 2016). Here the images that have been chosen bring with them important connotations so placing this quality development system within a specific set of discourses. On this slide we see an activity presented in the system as ‘forestmaths’. This has important meanings. In Sweden there is a culture of taking schoolchildren out into the abundant forests, both for activities and to simply ‘be’ in nature  – this has associations of ‘simplicity’, ‘cleanness’ and ‘equality’. This outdoor pedagogy is still important and connected to ideas of promoting children’s natural creativity and fostering sound values in a sound body. In the Unikum system, all activities must be governed by quality assessment/development based against a set of given national targets. On this slide we see a circle at the centre which provides the four stages of how the national targets can be achieved. There is a cycle of quality development: planning, documentation, following up and reflection. Here the circle itself is used as a kind of icon which suggests a cycle, and a whole integrated thing, where the segments, depicted as arrows, moving clockwise are the stages in the cycle. Here photographs have been used as ‘evidence’ of how the cycle works. So the two-year-old looking at an insect under a stone is ‘doing forest maths’. They can also be assessed on how they then engage with others about their discovery, how they are developing feelings and social competences. The images in Figure 2.1 help to naturalize and frame the cycle. The ‘forest’ activity brings with it cherished Swedish connotations of simplicity, innocence and

41

42

42

Introduction to Multimodal Analysis

equality. We see the children out in the forest, then sat in the floor on a carpet, with a teacher and then two teachers sat together happily reflecting. One of the authors, at the time of writing, has children in preschool where teachers speak of the workload created by such a system and the absurdity of coding all things in terms of targets. Yet here those promoting the use of the system use established connotations in order to re-contextualize its meaning. We make more observations about these images shortly.

Three further points These are general rules of which we should be mindful when considering denotation and connotation: ●●

●●

●●

The more abstract the image, the more overt and foregrounded its connotative communicative purpose. Barthes said that there was no innocent image free of connotation. This may be true, but there are degrees of this in terms of image use. Whether the communicative purpose of an image is primarily denotative or connotative depends to some extent on the context in which the image is used. A photograph of a child walking in a forest, taken by a parent may in the first place be used to document a pleasant day out. Later it may be used on a quality assessment PowerPoint slide to connote ‘children in nature’ and that ‘learning is natural and simple for children’. What an image connotes may, in some contexts, be a matter of rather free association. But where image-makers need to get a specific idea across, they will rely on established connotators, carriers of connotations, which they feel confident their target audiences will understand (whether consciously or not).

Carriers of connotation Drawing on Barthes we can consider a number of elements and styles of representation that are particular carriers of connotation. The people we see in representations are of course of high importance. In all of the examples in this book the people who are depicted have been chosen and represented in ways intended to represent specific discourses. First, however, we want

43

Pictures and Images

to look at the roles of objects and settings in images and what these can be used to connote.

The meaning of objects in images Simply the choice of objects placed into or onto any visual design can signal a discourse. For example, in Figure  2.1 we find the picture of the teacher talking to the children sat on a mat which appears to have bright rainbow stripes. In fact, what the picture shows is a teacher carrying out a performance management session with the children. It is based on reaching targets and systematizing all activities, even while out in the forest looking under stones and throwing leaves at each other. All such activities must be measured and improved, in regards to learning and social interaction. But the rainbow carpet, and of course the act of sitting on the floor, suggests comfort, informality and even fun. Had the photograph showed the teacher talking to the two children from behind a dark wooden desk and all were sat on chairs this scene would have communicated something very different. As we explained in Chapter 1, the commutation test, changing something in the multimodal configuration, is an important way to uncover meanings. We can, for example, ponder how the use of a dark carpet would have altered the meaning here. The mat here, therefore, is part of how this performance management becomes re-contextualized through a discourse of fun and informality. In the photograph to the left, we see a child’s hands holding stones. The use of stones also helps to bring associations of naturalness. The children could have been counting the candies or some collectable type of small plastic toys that are the latest fashion. Stones, like the forest itself, point to simplicity and nature. In short, the rationalism and target-driven nature of the performance system becomes softened by the use of nature. In Figure  4.2 we can see connotations brought by different kinds of objects. In the case of Starbucks the photograph placed in the middle shows a woman holding a basket, which does not look entirely completed. It is not clear if she is making baskets or if she is using it as part of her work. What is important here is that baskets represent the kind of pre-technological, honest, authentic type of work activities that form part of the Fairtrade discourse. Critics observe not that Fairtrade cannot depict people carrying out more technological or modern types of production processes (Varul, 2008). It would not have worked, for example, had the women been holding

43

44

44

Introduction to Multimodal Analysis

a laptop which she used to manage the books of her business. Running a Fairtrade company may of course require many kinds of skills related to business knowhow, stock-taking, sales and the use of advanced farming techniques. But these do not appear in the discourse as it is represented to the Western shopper. On the photograph to the left of the woman with the basket in Figure  4.2 we see simply hands. They appear to be sorting some kind of agricultural product. Here the hands themselves are used to connote the authenticity of hand-made and machine-free. This also relates to the meaning of honest labour. Of interest here is that manual labour in this case does not carry the connotations of tedium, repetition, powerlessness (ask anyone who has done this kind of work) but something honest and pre-modern, in a way that it can be used to attract consumers who want to help, or shop in a way that makes a stand against faceless corporate capitalism. We also see the use of hands used slightly differently on the Fairtrade coffee package in Figure  0.1. Above the brand name to the top right of the package we see an icon depicting two hands that overlap each other. Here hands are used to represent the interaction between the producer and perhaps the brand and the consumer. Rather than depicting the actual economic relationship between the two sides, the relationship is symbolized by the interlocking hands. Here too hands suggest something personal and authentic, as we reach out to someone. In Figure  2.2 we see the use of a different kind of iconography in comparison to the cycle we saw in Figure 2.1. Here we find cogs suggesting something machine-like with interlocking parts. This figure is a template which could be used by anyone wanting to present some kind of process in an institution. So here they may want to show how different departments and levels of management work together like a machine. And cogs are part of machines which are built to achieve a clear and specific purpose. What is of interest here is that by using this diagram the user would not be obliged to include how people actually interact when they take on different work tasks. The interaction is simply symbolized in this case. Even something like a school could be represented in this way, where teachers are cogs, working with management, parents and children. What such diagrams can conceal is conflicts, inequalities and who really has the power. It can simply omit an important cog such as economics and the politics of privatization. And it invites a view that things can be rationalized.

45

Pictures and Images

Put text here > Download this awesome diagram Your text here > Download this awesome diagram

Your text here > Download this awesome diagram Put text here > Download this awesome diagram

Your Logo

Figure 2.2  Gears flow chart template, ©The Slide Team. Available at: https:// www.slideteam.net/1013-business-ppt-diagram-4-stages-gears-businessprocess-flow-chart-powerpoint-template.html.

Of course, we would be more likely to find a school and the relationship of its parts represented through something natural like a tree which has a different set of associations. Here processes and participants could be represented as individual branches. But again, this can be used to conceal actual processes and power.

Settings Barthes also drew attention to how settings can be used to communicate wider meanings. As with objects, different kinds of settings can have more or less shared associations among people in a particular society, although as with objects these may be fairly culturally specific. Let’s return to Figure 2.1 and the two children depicted in a forest. The use of nature here, while highly resonant in Swedish culture, is typical of how nature can be used to connote ‘natural’, ‘simplicity’, ‘freedom’ ‘tradition’ and ‘innocence’. Food packaging for dairy products or bread may use images of idyllic countryside scenes to bring such values to

45

46

46

Introduction to Multimodal Analysis

products which are in fact produced entirely by industrial methods. In this case though, we need to be mindful that there are different kinds of nature. The forest we see in Figure 2.1 is clearly one which has been slightly cleared by use. A photograph of a remote forest with mists hanging over it would not be right in this case. Here we would rather have the meaning of ‘remoteness and freedom’, ‘wilderness’ or ‘adventure’. Such a forest could be used for packaging to draw on these meanings, for example, for a mineral water. We may see a brand of car speeding along an empty track in that kind of forest, or other landscape that can connote adventure and freedom. Should the forest appear almost like a jungle then we shift to the exotic and idyllic, often seen in Fairtrade branding where the customer is offered up not products made by poor people just down the road from them, but in remote beautiful travel locations free from the complexities of the modern world. The meaning of Figure 2.1 would have been very different had the quality development been depicted through two children in a busy urban setting, say a large road intersection, even if we were suggesting that counting and discovery could relate to traffic and homeless people. This would have lost the romantic element brought by the forest. Of course, urban settings can also be used to suggest ‘modernity’, ‘romance’, ‘high culture’, ‘exotic’, ‘the thrills of the big city’, ‘sophistication’. But such meanings may be less useful for selling a kind of performance management for children. Another kind of rural setting appears in the images of the peasants in the photographs on the wall of Starbucks in Figure  4.2. We have already observed in this chapter that Fairtrade marketing tends to not show workers in settings comprising more modern forms of industrial machinery nor technology. It would not work to see a peasant worker sat at a laptop doing their accounts, nor calling on an iPhone to arrange for a crop to be collected. This business end of the process is deleted for the consumer. Rather we tend to see grass, huts, baskets, straw hats, all pointing to pre-technology, simplicity and a rural idyll. There is one kind of setting that is also highly important in the photographs that we now see in designs. In the image showing the peasant worker on the coffee package in Figure  0.1, there is no actual setting to speak of. These persons could be anywhere. Such decontextualized images tend to be more symbolic than documentary. They show not so much a particular moment in time, or a particular event, but symbolize an idea or a concept. In such cases we are drawn to the role played by the participant and the objects which we can see. In the case of the workers

47

Pictures and Images

here, therefore, we are not offered a document bearing witness to a worker carrying out their work under a specific set of conditions, perhaps giving an indication of the effort required, such as the extent of the labour required for carrying the product. There is just a blank space, carrying the sandy brown colour which connotes the warmth perhaps of an exotic setting. Machin and Thornborrow (2003) suggest that this decontextualization has the effect of removing certain aspects of reality, allowing props to work in a way that they could not in the real world. They liken this to the way that children’s fairy tales work. They draw on the psychologist Bruno Bettelheim (1979), who argued that fairy tales quickly indicate that they do not take place in the real world. This allows different rules to apply. It means that certain everyday problems are removed. So young children can go on long journeys alone and save kingdoms. The decontextualized world of images of employees or students seen on university websites and brochures advertisements has the same effect. The everyday details and issues are removed, allowing the participants to embark on a kind of enchanted journey. Their magic amulets in such cases are the branded products, the courses on offer or the institution as a whole. Similarly, the decontextualized images on the Fairtrade coffee packaging remove the actual details of the context which allows us to think in abstractions about appreciative honest, authentic workers from exotic places. It more easily allows the idealization that the act of shopping can bring about change.

Visual depictions of people The images of people we see in designs may have been through many stages of editing and restyling. The photographer may have decided on a close-up. An editor or page designer may have decided that further cropping was required in order to show only one person rather than a group, or to give the impression that the person is closer to the frame, or to exclude individuals who might confuse the meaning they wish to convey. But the end product will have been designed or chosen, at some point in the process, to communicate particular ideas about the participants and a particular attitude towards them. In this section we look at some ways to focus on how people are represented and how they are represented as acting which can connote ideas and values about these persons.

47

48

48

Introduction to Multimodal Analysis

Representing participants In this section we look at the semiotic resources for depicting kinds of participants. This draws on the work of Van Leeuwen (1996) and Machin and Mayr (2012). This tells us more specifically what kinds of people are represented.

Individuals and groups In images, people can be shown as individuals or en groupe. This can make a difference to the way that the people and the events in which they are involved are represented. ‘Individualization’ is realized linguistically by singularity, for example, ‘the woman’ as opposed to ‘women’. This has the effect of drawing us close to specific people and therefore humanizing them. All tabloid journalists know that a story telling the experiences of one person will be far more compelling than statistics that speak of many. Visually, individualization is realized by shots that show only one person. So, for example, the workers on the Fairtrade coffee packing in Figure 0.1 are shown as individuals, and not in a larger group. In this case, this can be important for personalizing the relationship between the worker and the consumer. The designer may have felt that a picture which depicted a larger group of workers would not have the same effect. In Figure 4.2 showing the Starbucks café we see that the photographs of workers on the wall depict both individuals and groups. In this sense, we see more about the people as a group, yet in the centre nevertheless we see the person, holding the basket, who is more individualized. In one image we see a line of people walking along carrying objects. It is not particularly clear who these people are nor what they are doing. But here, there is no individualization and we are not encouraged to align with their feelings and interests in the fashion of the coffee packaging. Perhaps in this case it may be less comfortable to create a more individual relationship with someone engaged in collective labour carrying objects along a roadside. In language, ‘collectivization’ can be realized by plurality or by means of mass nouns or nouns denoting a group of people (e.g. clan, militia, terrorists). We can immediately see the ideological effect of this where instead of individuals who might have specific motivations we are dealing with anonymous groups.

49

Pictures and Images

Visually, collectivization is realized by images that show groups or crowds. The members of the groups or crowds can be ‘homogenized’ to different degrees. They can all be shown wearing the same clothes, performing the same actions or striking the same poses. In a photograph showing a large group, a kind of individualization as well as collectivization can be created as the people depicted strike different poses. This has become typical on publicity shots for TV shows and also on institution websites where a group of employees or students are shown orchestrated into interesting configurations and a range of poses. Collectivization can also be achieved by focus on the generic features of a group of people so that they are turned into types. For example, a news photograph of Muslim people in London might foreground those individuals wearing traditional clothing. If we look at news photographs it can be useful to compare how different kinds of participants are individualized or collectivized. In the news coverage of a conflict we might see a group of militia shown as group, homogenized and striking the same poses. In contrast, the ‘victims’ of the conflict may be represented in the form of a child shown as an individual, which emphasizes our identification with him. We may also see a shot of a ‘peacekeeping’ soldier, shown standing alone in medium-shot looking thoughtful. In Figure 2.4 we see generic images of teaching staff and students used in a strategic diagram. Their genericity is also communicated through the way they all strike ‘active’ poses. There are suggestions of individuality through the colour of clothing and slight differences in forms of the figure. But a close look shows that there are a few different figures arranged to suggest individuality. Individualization and collectivization can occur at the same time for the people depicted. What is interesting is to examine in what ways they are individualized and in what ways they are collectivized. On a university website a student may be individualized and seen looking out at the viewer, but at the same time collectivized to show that they are a typical student at the university. In Figure  2.4 the pyramid shows how everything will get better at the university if everyone follows this strategic plan as it is shown here. We can see that for the teachers ‘passion, talent, commitment and leadership’ are listed. So here all teachers must have all these. All must lead. This is how the goals will be met. Often in such institutions there can be staff shortages, temporary contracts and many other structural issues that hamper quality.

49

50

50

Introduction to Multimodal Analysis

Yet such diagrams tend to devolve success to the responsibility of staff – here represented as generic types in these action poses.

Distance This is the association of physical proximity and intimacy. In images as in real life, distance signifies social relations. We ‘keep our distance’ from some people we do not like and ‘get close to’ people we see as part of our circle of friends or intimates. In pictures, distance translates as ‘size of frame’ (close shot, medium shot, long shot, etc.). This is simply how close to the viewer a person is represented in an image. In early silent movies, action originally all happened on the same plane. The breakthrough came with the use of close shots which allowed the viewer to identify with individuals. Directors could cut to close-ups of faces to reveal reactions and emotions. This meant that viewers got to see the characters as individuals with feelings. Those who remained in the middle distance seemed more remote. Once the close shot was used, those actors who did remain in the distance appeared as generic characters. This association of closeness with individualization and intimacy of feelings reflects everyday life. When we allow people very close to us, this means that we have some degree of intimacy with them. This varies between cultures, but generally we feel uncomfortable if strangers get too close. So a closer shot suggests intimacy, whereas a longer shot is much more impersonal. If we look at the image of the worker in the Fairtrade coffee packaging in Figure 0.1 he is quite close to the viewer, although this effect could of course have been created through cropping. Nevertheless, this suggests intimacy. It draws our attention to him as an individual. In the Starbucks’ image in Figure  4.2, showing the workers walking along, the people remain in the middle distance. Therefore, this suggests an impersonal relationship. They are not individuals but represent people in general. Advertisements, for example, for a kind of face-cream may use extreme close-ups. This can symbolically represent the intimacy of the physical contact that the advert speaks of, although at the same time framing might be used to keep the model at a distance so that the person looks at us from behind a boundary. In a photograph, individualization can also take place where, for example, we see a group of eight students sat studying in a classroom and one of those

51

Pictures and Images

looks directly at the viewer, creating a kind of social interaction with them. In such a case, the viewer is encouraged to consider the state of mind of that particular person. Close shots can also suggest claustrophobia or a threat if the kinds of people represented are otherwise of a sort that we might not welcome. For example, it is common to see photographs from the Middle East in Western newspapers where we see groups of men holding guns or crowds of protesters in extreme close-up. This takes us too close to the energy of the moment, suggesting the need to pull back from the madness of the situation. Of course, such a physical reaction does not help us to consider the reasons for their behaviour, nor the wider social and political context, which could indeed make their protests seem quite understandable.

Categorization Visual representations of people can also categorize them, regardless of whether they are also ‘individualized’ or ‘collectivized’. Visual categorization is either ‘cultural’ or ‘biological’ or a combination of the two. Cultural categorization is realized through standard attributes of dress, hairstyle, body adornment and so on. The Fairtrade workers seen in Figures  0.1 and 4.2 are categorized through clothing and the objects that they carry. Looking across Fairtrade designs we see that many workers wear a hat that allows them to be marked as being from Latin America, Africa or Asia. Here clothing clearly signifies ethnic types. On television programmes we may see people in reality programmes represented as cultural types. This may be in a show which looks at how working-class people shop at cheap outlets (there is such a programme called Ullared in Sweden). The people depicted are often shown in ways that ridicules them and foregrounds that they have bad taste in terms of the things that they buy and the clothing that they wear and the objects that they value (Eriksson, 2015). In news coverage of conflicts, we might see US or allied soldiers categorized culturally by clean military uniform and the advanced technology that they carry and wear on their helmet. This emphasizes the sophistication and authority of the soldiers. In contrast, the militia might be shown wearing part-civilian clothing. This categorizes them as non-official soldiers. Biological categorization is achieved through stereotyped physical characteristics. Such categorization may be used to invoke both positive and negative connotations. In the first case, images might depict soldiers

51

52

52

Introduction to Multimodal Analysis

as Action Man-type stereotypes or women as Barbie-type stereotypes of female attractiveness. We might see American soldiers in movies or news images displaying just such a biological categorization of square jaw and muscular build. Soldiers shown writing home in the classic war photograph style, meanwhile, may be more youthful and of slimmer build to foreground vulnerability. In stock images found used for advertisements or other kinds of promotional material, models may be generic types, such as a pretty teacher or student, an office worker or a traveller. This may involve biological categorization. They may be attractive but indistinctive. Specific traits may allow them to be used to represent someone ‘alternative’ or ‘edgy’. And it may involve cultural categorization, as they wear particular kinds of clothing or carry certain objects. In the second case, there are racist stereotypes. In political cartoons, biological categorization is used to represent ethnic stereotypes. Van Leeuwen (2001) shows examples of the representation of black people with exaggerated lips. During Bush’s ‘war against terrorism’, pro-Bush cartoonists would represent Middle Eastern people with large noses and long, thin faces. But we can also see such categorization, for example, in images which seek to represent multiculturalism. A university or local school may wish to foreground its ethnic diversity as a means of signalling its openness and fairness. While the actual population of the institution may be white, the photograph on the website will foreground the Asian and Black students. Here we might want to ask why the institution needs to foreground a discourse that has an ethnically diverse population when it does not.

Anonymization This can be where people are represented but their identity is somehow unclear or uncertain. We can see this in different ways in the photographs of Fairtrade workers on the Starbucks wall. We see a row of people walking along, all slightly out of focus. In another image, we see only hands. In each case these anonymous individuals symbolize honest, slow, manual work. In the photograph in our university strategy document in Figure 2.4, we see people only represented through blurred motion trails, as if they are zipping about in the building. Here the persons represent ‘focused activity’ and ‘motion’ rather than specific persons. Such persons here can be used to signify the idea of ‘dynamic’ and ‘reaching targets’ which is an important part of how public institutions must now represent themselves.

53

Pictures and Images

We see another form of anonymization in Figure 2.1 where we see the children in the forest. We do not see their faces. Here too these represent generic types and become carriers of meaning related to childhood, imagination and nature.

None representation Finally, it is crucial if someone is not represented in an image. Consider first how this works in language. The Palestinian camp was attacked today.

Here there are no actors or agents. In this case it is the attackers who are excluded. The attack just happened. Therefore, no responsibility or blame is assigned. The same can occur visually. We may find a news headline or news voice-over telling us that ‘US soldiers confront Militia’. Yet visually we see no such confrontation. In the images of the Fairtrade workers in Figures 0.1 and 4.2 we could say that there is a lack of representation of those people who might carry out processing, transport and the management of the product being sold. We only see the workers who manually gather and sort the product. A news story about the evictions of a Roma camp by contractors using bulldozers and dogs, a camp which included women and children, is

Figure  2.3  Jurnalul National (Romanian Newspaper), 9 August 2012. INTACT Media Group.

53

54

54

Introduction to Multimodal Analysis

Figure  2.4  Pyramid to show strategic plan for university. From ‘Strategy 2020: Building Success’ summary document © Edinburgh Napir University.

accompanied by an image, seen in Figure 2.3, of four adult Roma being led away calmly by a single middle-aged, smartly but casually dressed, police offer. Here we can clearly see the re-contextualization of a highly aggressive act against families into something calm involving a small number of adults (Breazu and Machin, 2018). In such an image there is clearly lack of representation of key actors in this situation. And those who are represented are categorized both biologically and culturally. The police officer is smart but casually dressed in a white short-sleeved shirt. It would have been different had he been wearing riot gear or even a protective helmet. The Roma all wear clothing that may be stereotypically attributed to them. The police office is also biologically categorized as he appears middle-aged and approachable. Certainly not muscular and intimidating.

Summary text box for the representation of participants Individuals and groups: people can be depicted as individuals or as a group. If they are depicted as a group they can be ‘homogenized’,

55

Pictures and Images

made to look like and/or act or pose like each other to different degrees, creating a ‘they’re all the same’ or ‘you can’t tell them apart’ impression. Distance: this is the association of physical proximity and intimacy, coded by ‘size of frame’ (close shot, medium shot, long shot, etc.). Categorization:  people can be represented with attributes that connote stereotyped characteristics. These attributes can be cultural (for instance, in the case of representing Muslims with traditional clothing) or biological (for instance, in racist representation of Arab people with exaggerated noses and thin faces). Anonymization:  here we see people but key aspects of their identity are concealed. Non-representation: this can create anonymity. It can be a way of concealing responsibility for actions or can remove the role of some participants.

Actions in images Here we analyse action in images. We are interested in who does what and what gets done. In Critical Discourse Analysis it has been shown that a more careful analysis of action in texts or speech can reveal less obvious messages about who has agency, what kind of agency they have and who does not. For example, in this sentence we can see how the actions of a Fairtrade worker are represented: The peasant harvested the coffee beans

We can see that they are the agent here in regard to carrying out the process of harvesting. Reading a larger text on Fairtrade promotional material we may find that this is all they do. They do not do any of the other tasks of processing, transporting, planning, packaging, marketing and so on. So while the producers of Fairtrade products may have many skills and be involved at many levels, these are not included in the text. In a text we might find that a branded Fairtrade company says that ‘we work with producers on farms to bring you . . .’ . In this case the notion of ‘work with’ is pretty abstract and could mean many things. But what we can

55

56

56

Introduction to Multimodal Analysis

see here is that the company here becomes the agent. It could be written that the producers work with company. But here it may be important to foreground the helping aspect of the Fairtrade process. The point here is that language analysis of actions allows us to draw out who has agency, who is the doer and who is not. The same kinds of questions can be asked of images. Of course, still images are captured moments (Sontag, 2004). So we can only guess at what is actually going on. We can, as Peirce (1984) suggests, base our guesses on ‘indexical signs’. So if we see a picture of a person holding a tool standing next to a car with its hood open, we might guess they are fixing the car. We do not see the events taking place but the objects, scene and poses suggest they are. In video we can see actions and events taking place. Below we present an inventory we can use for the analysis of indexical actions in images or actual actions found in video.

Emotional processes This is where a person is shown to have a particular facial expression which indexes their mood. This may also be indexed through pose. The workers on the Fairtrade package in Figure 0.1 appear happy and have open poses as if relaxed. On such packages it is rare to see such workers doing anything else. The action here of smiling is foregrounded to connote the positive effects of the Fairtrade relationship. Some Fairtrade workers as well as smiling, stare directly into the camera and strike a confident pose. In this case we are given a sense of their sense of agency and self-management. It is typical in news and sports reports to select images of persons where facial expressions and poses reflect the angle of a story, even though the image will be from an entirely unrelated moment. Where a football manager or politician is under heavy criticism we may see them looking downwards, frowning and with rounded shoulders.

Mental processes This is similar to emotional processes. But here the image, using gesture and posture, is used to communicate inner thoughts. And setting too can offer cues. Typically here we find images where people are depicted looking off frame into the distance. A politician may be represented looking upwards and off frame to suggest they are thinking about the future and possibility.

57

Pictures and Images

A  woman may be depicted sat in a chair in a waiting room looking downwards at her hands. How we interpret her state of mind may relate to objects in the room, such as if we see medical charts on the wall. The image of the child’s hands holding the stones in Figure 2.1 is used to suggest mental process related to being inquisitive, discovering and learning. In a former time it may have simply been used to show a child dreamily enjoying playing out of doors rather than participating in a quality development exercise. When analysing a set of images we can ask, for example, if one set of participants is represented more often through mental processes than through carrying out more concrete tasks. Being given access to the mental processes of participants can be one way of bringing the viewer closer to them, of aligning them with their concerns and interests.

Verbal processes Here we see people who appear to be talking. In Figure 2.1 we can see the images where the teacher talks to the student and to her manager. Here communication itself is seen as an important part of the quality improvement process and is clearly foregrounded here. Showing people communicating also helps to provide more personal dimension to what is in fact a fairly abstract system of recording and measuring all kinds of behaviour in a database, whose content may have had very little input from teachers. In the Fairtrade images on the wall in Starbucks, seen in Figure 4.2, in contrast, the participants are not seen as carrying out verbal processes. There is something silent about these images. Here what is foregrounded may not be so much what these people do, but the generic types they are. Nor here is it clear what kind of states of mind the participants are in. But certainly what they have to say is not relevant. They are generic types and engaged in honest work. Perhaps here conversation would complicate the simplicity of this message and would signal organizational issues and negotiations.

Material processes This is simply doing something in the world that has a material result or consequence. Again, here we look for the elements and features in the image that index such material processes. So, in an image, we might see someone mending a bike, a teacher teaching students or soldiers protecting civilians. Here we can simply ask who is the agent of such processes, or who is the

57

58

58

Introduction to Multimodal Analysis

‘doer’. In a textbook we might simply ask a question relating to how men and women are represented in regard to material processes. And then we would have to consider how these material processes are represented in other terms. For example, in IKEA kitchen adverts, at the time of writing, we see both men and women preparing food. The men are shown somewhat comically as performing as artist-chefs. We are encouraged to laugh at them. Yet women are shown as serious and effective. Here there is not comedy. We would have to ask why this is the case. It is also important to ask where material actions are absent, where we might expect them to be represented. A website for a hospital may foreground verbal processes where we see receptionists, doctors and nurses talking to people, but no images of people being treated. The same may be the case on a university website. We see lots of students smiling, sitting in airy spaces, but the normal everyday material processes of teaching and studying are absent. In both cases we can relate this to how public institutions are required to market themselves to customers as offering a good service. Communication and openness here suggest good customer service. And the persons given the task to create the websites may in fact have no knowledge or experience of teaching or of treating patients.

Text box summary of actions in images Emotional processes:  this is where moods are indexed by facial expression. Mental processes: this is where we are encouraged to think about the internal state of the person as coded by facial expression and posture. Verbal processes:  this is where people are depicted as communicating. Material processes: this is where it is indexed that concrete actions with outcomes are being undertaken.

Student activity In this activity, we will ourselves create some photographs to experiment using categories for creating intimacy or distance and for aligning viewers alongside or against those depicted in images. You will use your mobile phone camera to take the following kinds of photographs of classmates:

59

Pictures and Images

Create a photograph which both individualizes and collectivizes participants at the same time. For example, this could show a group where four people strike the same poses or appear to be doing the same thing, where just one person looks knowingly at the camera. Create three photographs which suggest a generic type of person. Of course, the easiest for you could be a student studying (see how the university website does this). But in each of the three photographs, you will use angles to suggest three different kinds of relations with the viewer. Create your own band promotional photograph for an album sleeve, website or social media account. You will have to choose a genre. Do some research on the kinds of individualization and collectivization that usually characterize photographs for this genre of music. Think about postures, and these communicate different moods and mental processes. How does pose play a role here? Do band members strike similar or very different poses? Are the poses rigid, relaxed, open, closed and so on?

59

60

61

3 Modality: Experiencing the world through images When we look at images, they can show things to us. They can show us what a person or place looks like. They can also communicate ideas about things. A photograph of a person may show them in ways that suggest something about them. They are seen laughing, sat working on a laptop, or jogging along a beach. In the last chapter we looked at the ways that photographs and images can communicate in these ways. But images can also be created to shape how we engage with and experience the worlds they represent. We can think about this in the fashion of how an airport designer might create narrow corridors with lower ceilings to give people the sense that these are not places for stopping, but for moving along. Then they can create much wider, taller spaces to communicate the opposite. Each shapes how we experience and engage with the world. Images can also shape our experiences of the world. This can be in relation to the kinds of truth claims they are making and also how they are set up to allow the viewer to engage with the things that they depict. Images are often subtly, or not so subtly, composed or changed in order to conceal or enhance certain elements. Often in analysis we might want to say that an image looks ‘unrealistic’ or ‘stylized’, or that certain aspects are ‘exaggerated’, but we tend to lack the detailed vocabulary for saying exactly why this is so. Images can be naturalistic in that they simply record what is out there in the world. A photograph in your family album will be such an image. What you see in it is pretty much what you would have seen had you been there when it was taken. For example, you might see a picture of a family member at a wedding. You can see them in their cheap rented suit, with other people in the background chatting. You can see that the weather was awful that day.

62

62

Introduction to Multimodal Analysis

This photograph is a record of what happened at that moment in that place. Of course, you could have selected to take one scene over another in order to give a particular impression of events, for example, taking only photographs of smiling people to create a happy impression. But a naturalistic image would be a recording of the moment you chose to take the photograph. This use of the photograph as document has had a massive impact on our societies. Photographs have documented the reality of wars, poverty, famines and cruelty, often bringing these to the wider attention of the public. Think of the example of Nick Ut’s famous photograph of a naked child in Vietnam fleeing after a US Napalm attack. Such photographs bear witness. They are compelling evidence that informs us about actual moments in time and give us the chance to act upon them. Many academics have commented on the way that photographs have this compelling claim to represent truth, to be witnesses (Barthes, 1977; Sontag, 2004). And the same commentators have shown how this affordance to document allows arbitrary selected moments to suggest to us that they represent complex events and process which run over time. So Nick Ut’s photograph represents not only what happened on that specific time and place, but even more broadly the American war in Vietnam as a whole. An image of an individual smiling student on a university website can represent all students at the university. We can say that these images that document, that bear witness to events, claim to represent the world as we would have seen it had we been there. Here we can use the term modality to account for how real a representation should be taken to be, or what kind of experience it conveys. This has also to do with how we are positioned as viewers, as this has to do with what kind of perception and perspective we can take. Looking at someone’s face directly, or seeing something from the side and below, certainly gives different types of access to what is represented. Other kinds of photographs do not show the world exactly as it would have been had we been there. Images have always been manipulated, and the camera, through use of focus, exposure and choice of shot, can offer versions of reality. Photographers and graphic designers have been able to modify and add to photographs after they have been taken. We now have software on our telephones to allow us to quickly modify photographs in terms of colour, brightness, tonal range and so on. With little effort the image of our family member could be changed to fade out the background, to brighten or richen the colours. Where photographs like this do not appear to be naturalistic, where they do not represent the

63

Modality: Experiencing the World through Images

world as it was had we been there, we would say that they were of low modality. Such photographs do not document or bear witness. The term ‘modality’ therefore refers to the way we communicate as how true or as how real a representation should be taken (i.e. not how true or how real it really is). Being aware of how to analyse this in communication allows us access to the ideology of a representation. Asking what is hidden, changed, lessened in importance or what is enhanced, added, given increased salience can tell us about the view of the world that is being created for us. This is important in all kinds of images, news photographs, advertisements, promotional photographs, social media images and so on. This chapter provides a toolkit for analysing a range of features in images.

The origins of modality as a linguistic concept Modality is a way of analysing images that have been inspired by linguistic analysis, allowing us to reveal what is offered to us as certain and what is concealed. One very important issue for us in communication is the reliability of what people tell us. Is what people are telling us certain, truthful? This is important as we need to know how to then act on what people say. In linguistics, Halliday (1985) revealed that language provides us with resources to express kinds and levels of truth. This is centred on a specific grammatical system called the modal auxiliaries. These are verbs such as ‘may’, ‘will’ and ‘must’ and adjectives such as ‘possible’, ‘probable’ and ‘certain’. We use them all the time when we speak to show certainty, as in ‘I will have a beer tonight’, as opposed to ‘I may have a beer tonight’. These terms allow degrees of truth, or more precisely of probability, to be expressed. Consider the following two statements: It is certain that he is in Cardiff. It is possible that he might be in Cardiff. The second of these statements has lower modality than the first  – there is less certainty. A  representation that expresses high modality claims to represent closely what we would expect to find in the real world. One use of this observation is that it allows us to look at which aspects of a written or spoken representation are offered to us as certain and which are distanced from the real world, avoided or edited out. Looking at what is reduced in

63

64

64

Introduction to Multimodal Analysis

Figure 3.1  Viva La Papa chips.

modality, taken out, left in or emphasized, therefore, allows us to reveal something about the ideology of that representation. This is clear in the following statement: It is likely that the attack was carried out by an Islamic extremist group

We cannot be certain how true this statement can be taken to be. Ideologically, the events are dismissed as yet another instance of a generic extremist group

65

Modality: Experiencing the World through Images

rather than specific actions done for particular reasons. These modality markers, such as ‘likely’, ‘might’ and ‘possible’, provide scales of certainty that we can use as guides as to the nature of the certainty of a message. Kress and Van Leeuwen (1996) suggest that modality is also communicated through broader ways that we evaluate the statements of others. For example, a politician might say: Some people seem to think that globalization is a bad thing

In this case what people think is made to seem less certain by the use of ‘seem to think’. They only think it is the case which suggests mere opinion. The political might continue: But others know that it is an opportunity

In this case the certainty and level of modality is raised as the other people simply ‘know’. Here we are dealing with fact. For Kress and Van Leeuwen (1996), modality is ‘interpersonal’. It is not about expressing absolute truths but about aligning listeners with some truths and distancing them from others. As we will see, this is a key point in the way that we can think about modality in images being a resource for representing things, places, people and ideas as if they are not quite real but rather as vague notions or as fantasies. It was Kress and Hodge (1988) in their book Social Semiotics who first pointed out that modality could also be expressed non-verbally. They thought that the same kinds of test could be applied to photographs or any visual representation. But in place of words such as ‘possible’ or ‘might’, Kress and Van Leeuwen (1996) suggest, there are other techniques whereby modality can be reduced and reality can be avoided or changed. For example, in the case of photographs we can look at the detail of the subjects of the image and of the setting. Has this been reduced or sharpened? We might think of this effect as being like the use of words such as ‘might’ or ‘certain’. We can ask whether these details are different in the photograph than they would have been if we had been there. If they are different, then we can ask why. We can ask what the meaning potential of this change is. Drawing on this linguistic basis, Kress and Van Leeuwen (1996) developed eight visual modality markers that we examine below. These are scales that run from high to low modality, much like the scales from ‘certain’ to ‘uncertain’ with ‘probable’ in between. With this type of scaling, it is possible to talk about modality as being ‘high’ (‘certain’) or ‘low’ (‘uncertain’), with ‘probable’ in the middle of this scale.

65

66

66

Introduction to Multimodal Analysis

Modality markers and scales Degrees of the articulation of detail This is whether or not we are able to see the details on the surface of the elements in a photograph, whether its form is represented as it would have been had we been there. It is a scale that runs from maximum detail to maximum abstraction. The difference of the two extremes might be found between a wedding photograph of a family member and a matchstick drawing of the same person. In this case, there is a difference in scale between the representation of the particular with all its detail and the representation of the generic. The fully detailed photograph shows an individual person, whereas the matchstick person could be just about anyone. The difference between the specific details and the generic is like the difference between saying that there is a specific politically motivated group who carried out an attack and saying that there was a generic extremist organization. In each case, the specific is of higher modality. The scale between full articulation of detail, high modality and abstraction, low modality can be illustrated as in Figure 3.2. As in language we have the scale running from ‘this is’ through ‘this might be’ to ‘this may not be’. In photographs and compositions we often find subtle, but nevertheless significant, variations in degree of articulation of detail. In Figure  3.3 we see a spread from the IKEA catalogue. The person sat on the sofa is slightly out of focus. The detail of the skin has been reduced so that we cannot see any flaws or blemishes. The same lack of detail is also characteristic of the clothing. This is not how we would have seen the person had we been there. Here modality is lowered only slightly. However, it is significant nonetheless. This reduction of detail is often seen in advertisements, making the model seem without imperfections. This idealizes the subject of the photograph. The children seen on the children’s camp website in Figure 5.9 are in contrast shown in higher modality.

Maximum articulation of detail

Minimum articulation of detail

Figure 3.2  Modality scale for articulation of detail.

67

Modality: Experiencing the World through Images

Figure  3.3  IKEA catalogue. Image from ‘The Colors of IKEA 2016’ by Maurice Svay, Blog de Maurice Svay, 17 August 2015.

Slight reductions in articulation of detail can also make a scene slightly dreamlike. We can see this in the image of the torso of the body on the plastic surgery website in Figure  1.1. We also see this on the picture of sustainable development in the bottom left photograph in Figure 7.7. Here the globe appears slightly out of focus, and lighting and exposure have been used to create a hazy effect over all the elements. This reduction in modality is typical of advertisements and one way that advertisements can represent an idea or concept rather than indicate that they are documenting a moment in time. We can see more dramatic reductions in modality. The leaves on the toothpaste tube in Figure  3.4 are represented as a sketch to symbolize something natural. When placing representations of things such as plants, fruit, people or animals on products, different choices in modality can be harnesses. If we look at Figure  7.3 showing the two protein powder containers, we can see that the organic product to the left carries an image of a flower. This is of slightly higher modality than the leaves on the toothpaste, but is still not in the form of a naturalistic flower. In both cases we can say that the leaves and flower are idealized since they are simplified.

67

68

68

Introduction to Multimodal Analysis

So we can ask what qualities the representations foreground (such as the colour). And here the style of the representation can be important. For example, is the artwork more impressionistic and whimsical, or more modernist and clean? We also see high modality, as for the scale of articulation of detail, in the photographs used for quality development in the preschool in Figure 2.1. Here the high modality can be important for grounding the more abstract nature of the system that is being promoted. In Unikum’s material promoting and explaining their system, they sometimes use lower modality images of happy people to symbolize positive evaluation of their system. But they also use these higher modality images as a way to show parents and teachers how it is grounded in everyday familiar practices, such as going out into the forest to play. We can see the removal of details in the example of the Closer magazine cover in Figure 3.6. Here the skin and colour of the models appear smoothed

Figure 3.4  Toothpaste tube. Riddells Creek Organic.

69

Modality: Experiencing the World through Images

of wrinkles and shadow. Here too there is a sense of lowering of truth claims. This is a magazine that sells simplifications and a surface level analysis. Here, of course, the use of colour, typeface and composition plays an important role.

Degrees of articulation of the background This is to do with how well we can make out the details of a background in an image. Backgrounds might be ranging from a blank space, via lightly sketched in- or out-of-focus backgrounds, to maximally sharp and detailed backgrounds. On the quality development slide in Figure 2.1 we can see the background clearly, at least where we see the children in the forest to the top left. We can see what kind of forest this is, which is important for the message of the slide which seeks to place the quality development system into a naturalistic, safe and familiar setting. The setting also helps to anchor the system into natural and every day of the school. Photographs which show exact details of locations anchor the events in a particular moment of time or can be used to connote that this is taking place. Here the children are anchored into a moment of quality development. On the Rambo poster in Figure 5.7 we see something different. Here the background is less distinctive. We can make out trees and attack by helicopters, but it is not possible to make out exactly what kind of location this is. Clearly the background is a montage of settings from the film. At the bottom we see a boat. But nothing here is articulated in detail. In contrast the character Rambo and his gun are articulated in exaggerated detail. In such cases representation can become ‘more than real’ and idealized. In Figure  0.1 on the Fairtrade coffee packaging we can see how the workers appear against the sandy brown background. In one sense this is completely decontextualized. Here the blank background communicates a sense of a warm exotic setting which is identical across the range of products from different locations. These workers are entirely decontextualized from the variety of contexts in which they work and live. And researchers have noted that this is one way that we romanticize these people from around the world into a kind of single honest and exotic ethnic type (Varul, 2008). This is different from the articulation of background in Figure  3.1 for the Viva La Papa chips. Here we can see a setting which appears

69

70

70

Introduction to Multimodal Analysis

to be comprised of trees, but this is not entirely clear. In this case the representation is enough to suggest something exotic. In contrast the two workers are represented in vivid detail. We see all the lines of their faces. In such cases high articulation of detail can signal higher levels of truth. As with the workers on the coffee packages it is the truth of the honest worker which is of salience here. In the children’s camp website in Figure 5.9 we find that the background is slightly out of focus. But this is not to the extent that we cannot see that there are trees and that this is set in an open space, perhaps a sports field. In this sense we would not say that the photograph is decontextualized, rather than reduction in detail helps to foreground the children. Figure  3.3 showing the pages of an IKEA catalogue is different as the setting here appears overly articulated. While there is clear focus on some of the items in the room, we can see details throughout in a way that is not common in standard photography. In this case it is the participant who is slightly out of focus.

Degrees of articulation of depth This is the scale running from deep perspective to its complete absence. In naturalistic modality we would see depth as in everyday vision. Imagine a tree in full three dimensions with the complexity of depth of its leaves. Low modality would be where this was reduced. For example, in a children’s picture book, we could find a tree represented by two layers only, by a circle representing all the leaves being laid over the trunk. In between would be isometric perspective. This is where we see three-dimensional objects drawn flat on a page in a way that maintains their original proportions rather than reducing dimensions that are further away from us, as we would see in the real world. With this perspective we know there is depth because of our experience of looking at three-dimensional objects, not because a threedimensional object is represented. Architectural drawings use isometric perspective, as do older computer video games such as Mario or more contemporary proto-games for apps. In these cases depth is difficult to judge. It is often used therefore for visual illusions such as the eternal waterfall in the drawing of Maurits C. Escher (1961). Kress and Van Leeuwen (2006) suggest that depth can be more than real, for example, with the use of fish-eye lenses or where an artist exaggerates the convergence of two vertical lines to increase a sense of distance.

71

Modality: Experiencing the World through Images

Degrees of illumination – articulation of light and shadow This scale is to do with how much the lighting in the image appears as it would have been had we been there. This is a scale that ranges from zero articulation to the maximum number of degrees of depth or intensity of shade. We can see a clear case of reduced articulation of light and shadow in the case of the Wax Kit in Figure 7.5. We see that the woman’s face reveals no shadows. To the left we see that the product itself is saturated in bright light. Here brightness and cleanness are important meanings for this product. But in both cases illumination departs from naturalism. The brightness here also brings a sensory feel of softness due to the slight diffused effect. Stock images often have this quality which can be used to bring optimism and sensuality to advertisements and promotions. The lack of shadow along with the presence of bright light gives the effect of optimism that can, of course, be transferred to products and promotions. Darkness and pools of shadow can have the opposite effect, creating moodiness and a sense of concealment. The people in the Oxfam poster in Figure 7.4 are both lit from the front and with backlighting. This is typically used to create a sense of optimism and also for communicating truth and transparency. News readers will often have both front and backlighting. We can imagine the difference were there dark shadows behind them.

Degrees of articulation of tone This is whether the colours in the image have a range of differences in tones, of levels of gradation of brightness, or whether we find only simple polarities of dark and bright at the other end of the scale. A simple illustration of this can be found on the leaves shown on the toothpaste tube in Figure  3.4. In comparison, the flower on the organic protein in Figure 7.3 shows more articulation of tone variation. In the first case the simplicity is part of what is being communicated. In the case of photographs, in the university vision document in Figure 7.6 we find that the tone on the faces of the four people is naturalistic. Tonal variations on their skin are as we would have seen them had we been present. We see a very different case in the image of Rambo in Figure 5.7.

71

72

72

Introduction to Multimodal Analysis

Here we see extremes of light and dark which here increase the sense of drama. In Figure 0.1, we can see that the faces of the workers on the coffee packages use monochrome to create contrasting tones which has the effect of exaggerating lines and wrinkles. Such an effect can also connote a heightened level of truth and documentation.

Degrees of colour modulation This is the scale from flat, unmodulated colour to the representation of all the fine nuances of a given colour. When we look at colour in everyday situations, such as on the clothing you are wearing, there is no point at which there is the ‘true’ colour. There will be a range of colours based on the way the contours and folds catch light and shadow. This is colour modulation. We can see levels of modulation that we might expect in a naturalistic setting in Figure  7.6 showing the University vision. Light falls on the contours of people’s faces and their clothes. On the Closer magazine cover in Figure 3.6 in contrast, we see something different. Here the faces of the models appear flatter and without the contours created by how light falls. Here there is sense of simplification. On the Wax Kit in Figure  7.5 we can see a similar effect, although more subtly done. This is typical of how modulation is used in promotion images and advertising to create a sense of certainty and simplicity as well as genericity. Children’s toys and illustrations often use unmodulated primary colours. We can see the opposite of this in the case of the Rambo film poster in Figure 5.7 where modulation is exaggerated which suggests grittiness and complexity. In the case of the workers on the Fairtrade coffee we also see an exaggeration produced to raise the lines on the faces. Here this can signify that we are dealing with complexity and gritty reality. Of course, in this case the background is removed so complexity of circumstances is only signified.

Degrees of colour saturation This is how full and rich colours appear, ranging from black and white to maximally saturated colours. Here we can find that advertisements or webpages carry saturated colours to bring emotional warmth and energy. The warm sandy brown colour on the Fairtrade packaging in Figure  0.1

73

Modality: Experiencing the World through Images

brings this association to the workers seen in the image. The setting here is represented through these meanings. On the children’s camp website we can see that saturated yellow has been added to an otherwise naturalistic image to add extra emotional energy. On the Battle for Britain poster in Figure 7.1 we can see that the family has worn the colours of the BNP logo, but these are represented as richly saturated. The IKEA colours in Figure 3.3 appear more rich and vibrant than when we actually go to the store. At the other end of the scale there would be a black-and-white image where all of the colour had been drained out. Naturalistic modality lies in between. One of the effects of increased saturation of colours is a sensory one. Kress and Van Leeuwen (1996:  170) refer to this as the ‘modality of the senses’. The colours in the BNP poster in Figure 7.1 connote an emotional intensity, although this combines with the simple, flat colours to indicate a lack of depth to these emotions. Advertisers can use this sensory meaning potential to increase the sensuousness or emotional engagement of products. Often single elements, such as lips and hair, will be saturated. Children’s toys often use saturated colours, making them ‘more than real’.

Degrees of articulation of colour differentiation This is about how many colours there are. It is the scale from maximally diverse, as we might find in naturalistic modality, to monochrome. We can see that on the children’s camp website in Figure 5.9 the colour palette of the photograph has been enhanced to create a sense of fun and energy. In contrast the images on the Fairtrade coffee in Figure 0.1 carry a single colour. In some cases such a use of monochrome, as in black and white, can connote timelessness or seriousness. Monochromes are associated with older images, news photographs and art. But the monochrome can allow the hue and quality of this colour to dominate the meaning of the image. In this case it is the sandy brown. On the Rambo poster in Figure 5.7 we find a very limited colour palette, a burnt-brown and a white-yellow. But unlike the Fairtrade package here we find different tones of those two colours. This suggests less fun or variety in the contents of the film, or at least the variety is in degrees of intensity around the matter of violence and explosions. We could imagine the same poster done in a larger colour palette including reds, yellows and blues.

73

74

74

Introduction to Multimodal Analysis

Summary of eight modality scales Each of the following scales can be applied to assess the modality configuration for an image. Where there is reduction we have abstraction and where there is increase we have exaggeration. Degrees of the articulation of detail: a scale from the simplest line drawing to the sharpest and most finely grained photograph. Degrees of articulation of the background:  ranging from a blank background, via lightly sketched in- or out-of-focus backgrounds, to maximally sharp and detailed backgrounds. Degrees of articulation of depth: ranging from the absence of any depth to maximally deep perspective, with other possibilities (e.g. simple overlapping) in between. Degrees of articulation of light and shadow:  ranging from zero articulation to the maximum number of degrees of ‘depth’ of shade, with other options in between. Degrees of articulation of tone:  ranging from just two shades of tonal gradation, black and white (or a light and dark version of another colour), to maximum tonal gradation. Degrees of colour modulation: ranging from flat, unmodulated colour to the representation of all the fine nuances of a given colour. Degrees of colour saturation:  ranging from black and white to maximally saturated colours. Degrees of colour differentiation: ranging from maximal differentiation of colours to a limited colour palette and monochrome.

Kinds of visual modality Kress and Van Leeuwen (1996) show that it is not the case that modality always decreases as scale markers indicate lower levels, for example, as articulation is reduced. If this were so, simple line drawings would always have low modality. Sometimes such abstraction can be of high modality, in other words making a high truth claim. This is true of scientific drawings. A diagram representing the trajectory of the Earth around the sun would be thought of as having a high truth value. In such a diagram it is important that there is abstraction so that we can show the essence of how things work. We would similarly think about a diagram of the way the human heart works

75

Modality: Experiencing the World through Images

as making a high truth claim even though it did not look exactly like a heart if we saw one, as in the image at the top in Figure 3.5. So visual truth claims can come in different forms. If we saw a photograph of a man assembling a flat-pack IKEA table, we might, having consulted our eight modality markers, conclude that it was high modality, that it was as it would have been had we been there. But if we applied the same criteria when we look at the IKEA instructions which show a simple cartoon man assembling a cartoon table, we would have to say there was reduced articulation of detail, reduced background detail and so on. If we were examining this as a claim to naturalistic representation then it would be low modality. In this case the representation does not claim to be naturalistic but at the same time makes a high truth claim about how a person would assemble their table. And if it did indeed help us to assemble our table correctly, we would agree about its truth claim. The photograph and the diagram are clearly very different kinds of representations. But both could be said to be true despite this difference. We cannot say that one is more real than the other. The photograph represents the process of assembling the table as if we had been there in the room. It represents a specific instance of table assembly. The diagram, in contrast, represents the process of assembling the table as accurately as possible, but generically. The first we call naturalistic modality or the naturalistic coding orientation. The second truth we call scientific modality or the scientific coding orientation. In addition we have a third kind of truth which we call sensory and relates to how we relate to the world through our feelings. The same set of modality markers can be applied, but different points on the scales will mean different things. It was Kress and Van Leeuwen (1996) who distinguished these different coding orientations of visual modality, each established by particular social groups. The logic of this is that what is considered as real depends on how reality is defined by social groups who have the power to do so in different contexts. When we approach modality in this way, we are moving away from the linguistic view of modality, as coded by auxiliary verbs in clauses, to a focus on how visual resources are configured to make different claims about how the world should be experienced. This allows us to make distinctions between the truth claims of the three hearts in Figure 3.5. The top image is a diagram, obviously claiming to represent the underlying principles for how the heart works and drawing on a technical and abstract coding orientation. The bottom left image is about emotions and sensory modality, where the heart symbolizes true feelings, drawn as it is with

75

76

76

Introduction to Multimodal Analysis

Aorta

(a)

Pulmonary arteries Vena cava from upper body

Pulmonary veins

Right atrium

Left atrium

Vena cava from lower body Right ventricle

Left ventricle

Septum (dividing wall)

Heart : Internal structure

(b)

(c)

Figure 3.5  (a, b and c) Modalities of representations of heart.

77

Modality: Experiencing the World through Images

thin and curved strokes and filled with saturated red and pink colours. The bottom right image is more naturalistic, as it is a stylized drawing of how an actual heart looks. But it is not a high naturalistic modality: it is a drawing without background and where colours are used to distinguish parts, which means that there is also a technical modality present, where the functioning of the heart is part of the representation. But these colours are unmodulated and saturated, which, arguably, gives a touch of sensory modality – we can think of this image as apt for a children’s book. So it is of note that coding orientations might be combined in one image, just as we can look at the world (naturalistic modality) in an emotional way (sensory modality).

Naturalistic modality: The truth of our eyes Here photographs can carry massive weight as truth, even though they capture only arbitrary moments. News photographs cannot capture complex processes or events that happen gradually – they capture moments that must somehow reflect an event or something memorable. Yet we still see them as truthful, as evidence. We have seen the use of naturalistic modality as a form of truth throughout this chapter, for example, where a complex system of quality development in a preschool uses the truth of photographs of children playing in a forest. We have not included an example of this above, but an actual photograph of a heart would be naturalistic modality. This may be good for showing someone the truth of what a heart looks like. But it may be less useful for other modality coding orientations.

Sensory modality: The truth of our feelings The second kind of truth is the sensory modality. This truth criterion is based on the effect of pleasure (or un-pleasure) created by visuals. This is the truth of feelings. For example, the paintings of the Impressionists claim the sensory truth of what is out there. These painters, drawing on ideas in philosophy and psychology, questioned the idea of an objective reality out there independent of our own perceptions. The process of perception, rather, is one whereby we can experience the world only through our limited physiology, which would then be filtered through our cultural and emotional baggage. We might say that the impressionist painters painted the sensory truth of what they witnessed. This would be more truthful than naturalistic representations as it did not pretend that vision could be objective. We can

77

78

78

Introduction to Multimodal Analysis

think of toys as offering children not the truth of naturalism but the sensory experience of the thing. A toy car might have massive wheels or exaggerated yellows. These are representing the essence of ‘wheelness’ or of ‘yellowness’. We have already seen in our analysis that sensory modality can be an exaggeration of some of the modality levels in the case of naturalistic modality. So when colour saturation is increased, say in an advert where a woman’s clothing is a rich red, while naturalistic modality is lowered due to exaggeration, we can say that what is being offered is a sensory truth. We can see the sensory truth above in the drawing of a heart. This is the truth of emotions. Should we send our loved one a photograph of a real heart, of even a detailed sketch this would not be the right kind of truth.

Abstract or technical modality: The truth of the intellect The third coding orientation is abstract or technical modality. This relates to the truth of what an object is like generically or what the underlying principles of how something functions. Individual difference is not important here, but the general principles, not visible for the eye. This truth claim relies on these principles having been discovered by the human intellect, by logical reasoning giving knowledge of something. We see this in the diagrams for representing how IKEA tables should be assembled. We also see this kind of truth in the form of flow charts which claim to represent the truth of the technical details as regards how something, or a process, works. We can see this kind of diagram of the heart to the left in Figure 3.5. The truth of how something works can be important. If we were instructing a medical student in open-heart-surgery we would not want to give them only a photograph. The drawing of the love heart would also not help. The drawing on the right-hand side may help in this case, but here too, as we said, colours have been used to create a sensory effect and ‘fun’ look for a children’s textbook. We often find the use of different coding orientations in different kinds of documents. On the plastic surgery website in Figure 1.1, we find a sensory modality where the light is saturated and there are restricted shadows. Skin details have reduced articulation. But behind the torso we see a grid pattern and there are a few ‘cut here’ lines added. A food product packaging may show images of food which have saturated colours as in the example of Fit Kitchen in Figure  5.12. But this can also

79

Modality: Experiencing the World through Images

be accompanied by elements which signal technical modality such as the lines which point to the nutritional content. In other ways the label seeks to connote naturalness with its texture and giving a sense of unmediated access to the food.

Figure 3.6  Closer magazine, Issue 699, 2016. American Media Inc.

79

80

80

Introduction to Multimodal Analysis

Summary of coding orientations Naturalistic modality and the naturalistic truth criterion:  the more an image resembles the way we would see that something if we saw it in reality, from a specific viewpoint and under specific conditions of illumination, the higher its naturalistic modality, relying on the truth of our eyes. Sensory modality and the sensory truth criterion: this is based on the effect of pleasure (or un-pleasure) created by visuals:  the more the visual qualities of an image (or other form of visual) stir our feelings, the higher its sensory modality. It is the truth of feelings. Abstract or technical modality and the abstract truth criterion: the more an image shows the deeper ‘essence’ of what it depicts, or the more it represents the general schema underlying superficially different instances, the higher its abstract modality. It is the truth of the intellect, of knowledge.

Perspective in images The following section relates to our bodily experiences of the world and other humans in it. It focuses on how we relate to others, specifically how this is coded visually. The tools presented here are to help us to think about how viewers are included and excluded and how different kinds of perspectives can code different levels of engagement. In photographs we can meet people face-toface, and they might look at us or look in another direction. We might see them from the side, or face their backs. We might be positioned higher or lower than the participants in the image, and the distance to them can be very different. All this tells us something about how we are supposed to engage with image and its subject. If we look at an image such as the IKEA catalogue in Figure 3.3, or a Fairtrade worker in a photograph on the wall of Starbucks seen in Figure 4.2 what kind of perspective is the viewer given? Asking this question can also tell us something about how the viewer is encouraged to align alongside or against what they see. This type of modality draws on our bodily experiences of the world. Being face-to-face with someone at a close

81

Modality: Experiencing the World through Images

distance would facilitate, often even demand, some kind of interaction. Being in, for example, a narrow corridor with all office doors closed would entail a completely different bodily experience, where interaction is not possible. Such experiences are commonly used as a basis for visual communication.

Engagement and frontal perspective We start with the fact that we as humans engage with others with our front. The front is our face, our eyes and sensory organs, and displays who we are, and we look into the eyes of friends to see how they are or feel. This is crucial for how depicted persons can engage us.

Direct address In pictures, as in real life, the depicted people can look at the viewer so that there is symbolic ‘contact’ or ‘interaction’ between the viewer and the people depicted. On the advert for L’Oréal in Figure 7.2 the model looks out at the viewer. In movie posters a central character might look out at the viewer. This has two functions. On the one hand, it creates a visual form of address – the viewer is acknowledged. On the other, it produces an image act. The image is used to do something to the viewer. This is what we can think of as a ‘demand image’. It asks something of you in an imaginary relationship. The kind of interaction suggested, or the exact mood of address, will be determined by other factors. There may be a kind of smile that invites us in or allows us to share in the joy of a moment. There might be an angry frown. In this case we may relate with them in terms of them being a danger or superior to us. Or the person might have a bodily posture that suggests welcome through, say, open arms, or aggression through clenched fists being placed on their hips. All of these allow the image to define its relationship to the viewer. In real life we know what happens when someone smiles at us. We must smile back or offend them. When we see a smile in an image we know what is being communicated, although we know naturally that there will not be the same kind of immediate consequences if we do not respond. In this imaginary relationship we will recognize that there is a particular kind of demand – that we, as viewers, can accept or deny.

81

82

82

Introduction to Multimodal Analysis

Looking at the viewer, or the right to do so, in itself suggests power. Hartley (1982) has shown that in news broadcasts, trade union officials are not allowed to speak to the camera, whereas politicians, journalists and presenters are. In this way the trade unionists are not allowed to address the viewer, to enter into an imaginary relationship with them. So a worthwhile question to ask is what kinds of people are allowed to look out of the frame and engage with us and what kinds of people are not?

Potential address In a picture we might meet someone face-to-face, where the depicted person is not looking at the viewer, so that this kind of ‘contact’ or ‘direct address’ is absent. Still, when we, as viewers, are facing someone, especially if we meet a close-up of a face, there is a potential address. We are basically invited to engage with this person. How this invitation should be assessed depends on several factors. In advertising and promotional imagery, and also in many photographs in women’s lifestyle magazines, we often see the person represented looking off frame. In these instances we may find a caption. In a women’s lifestyle magazine such as Cosmopolitan we might see a woman looking off to the right of the frame, slightly downwards, basically expressionless. The caption might say, ‘Are you worried about your relationship/work colleagues/skin condition?’ In this case we are invited into the thoughts of the person represented, so their gaze off frame comes with a direct address in language, often a question.

Non-frontal perspective When we do not approach, or see, people face-to-face, but from the side or from behind, or at a long distance, there is no address. Non-frontal perspectives signal that we are some kind of observers, as if given a snapshot of a reality we are not part of, possibly as a bystander or at time as an eavesdropper. Here, no potential for engagement or interaction with the people depicted are coded. We are, as viewers, not called upon for a response. The images are rather offering information available for scrutiny. The workers seen on the coffee packaging in Figure 0.1 do not look out at the viewer, unlike the

83

Modality: Experiencing the World through Images

woman in the Fairtrade photograph on the wall of Starbucks. These workers do not engage with us.

Angles The angle from which we view a person can suggest different relations between the people represented and the viewer. These are to do with the physical association of being removed from a scene, a moment in time and our physical associations of height and power.

Horizontal and oblique angles In the case of a frontal perspective we are engaged by the person represented. In the case of non-frontal perspective we become voyeurs on the scene. But our involvement in a scene can also be changed through viewing positions, through the angle of interaction around the horizontal plane, which have meaning potentials.

Vertical angle This is to do with power and the association of height and superiority/ inferiority. If you look up at someone, this has the metaphorical association of them having higher status than you. It also means that they are physically in a stronger position than you. We associate size with power and status. In social life we might stoop down to speak to small children if we do not wish to intimidate them in an attempt to come down to their level. However, when we look down at someone this gives us the sense of power and them the sense of vulnerability, as a small child might look up at us. These kinds of associations influence the way we assess the relative power of the person depicted. If someone is shown at the same level in a photograph, then equality is implied. But once our viewing position is raised or lowered, our status relationship is changed. We can see the use of angle of interaction to create perspective in the case of the students in Figure 7.7. Here our perspective is looking up them. They feel empowered, especially as we see them brightly lit and all smiling. The photograph on the children’s camp website in Figure 5.9 appears to be taken from an angle it suggests equality with the children, or to give us a perspective which aligns with them.

83

84

84

Introduction to Multimodal Analysis

Oblique angles These are where the camera will be tilted or ‘cantered’ so that the person appears at an angle rather than being positioned vertically in the frame. These are used to give an unsettling effect or to suggest tension. Oblique angles are used less frequently in news photography but are often found in cartoons and movies, although newspaper supplements have begun to carry more ‘artistic’ photographs to connote design sophistication. Such images can also, due to the playfulness of angle, connote energy and movement.

Distance This is the association of physical proximity and intimacy. In images as in real life, distance signifies social relations. We ‘keep our distance’ from some people we do not like and ‘get close to’ people we see as part of our circle of friends or intimates. In pictures, distance translates as ‘size of frame’ (close shot, medium shot, long shot, etc.). This is simply how close to the viewer a person is represented in an image. In early silent movies action originally all happened on the same plane. The breakthrough came with the use of close shots which allowed the viewer to identify with individuals. Directors could cut to close-ups of faces to reveal reactions and emotions. This meant that viewers got to see the characters as individuals with feelings. Those who remained in the middle distance seemed more remote. Once the close shot was used those actors who did remain in the distance appeared as generic characters. This association of closeness with individualization and intimacy of feelings reflects everyday life. When we allow people very close to us this means that we have some degree of intimacy with them. This varies between cultures, but generally we feel uncomfortable if strangers get too close. A closer shot suggests intimacy, whereas a longer shot is much more impersonal. The distance has to be thought of as working differently in frontal versus non-frontal perspectives. We can see the use of closer shots in the case of the L’Oréal advert in Figure  7.2 and on the coffee packaging in Figure  0.1, where we see the workers in intimate close-up. We see a more extreme close-up in Figure 7.5 showing the package for a facial wax kit, although here it is the quality of the skin that is important rather than aligning us with the feelings of the participant since she is heavily cropped and decontextualized.

85

Modality: Experiencing the World through Images

We see others represented with greater distance. In the quality development images in Figure 2.1 we see the children in the middle distance. We see the close-up of the hands, which is the activity to which the viewer is given greater proximity. We see the teachers talking in middle distance. These teachers are not represented as especially remote but they are not so close as we are encouraged to imagine the experience of that individual teacher. Close shots can also suggest claustrophobia or a threat if the kinds of people represented are otherwise of a sort that we might not welcome. For example, it is common to see photographs from the Middle East in Western newspapers where we see groups of men holding guns or crowds of protesters in extreme close-up. This takes us too close to the energy of the moment, suggesting the need to pull back from the madness of the situation. Of course, such a physical reaction does not help us to consider the reasons for their behaviour.

Summary of positioning the viewer Frontal and non-frontal perspective: in pictures, as in real life, we can meet the depicted people face-to-face. If a person looks at the viewer we have direct address, a symbolic ‘contact’ or ‘interaction’. If a person that we see face-to-face looks off frame, we have a potential address:  we are invited to share the thoughts, feelings, fun and so on of the person. Direct and potential address can take many different forms. It can be pleading, seductive, arrogant, friendly and so on. If we do not meet the depicted people face-to-face we have no address. As viewers we then are coded as observers. Horizontal angle: in pictures as in real life, becoming involved with people means, literally and figuratively, ‘confronting’ them, coming ‘face-to-face’ with them. The side-on view is more detached, although combined with closeness it can, depending on the circumstances, index togetherness. In pictures this translates as the (horizontal) angle: frontality and profile (and the various in-between possibilities). Vertical angle: there is also the vertical angle. You can either ‘look down on’ or ‘look up to’ people to various degrees. Distance: in pictures as in real life, distance signifies social relations. We ‘keep our distance’ from people we do not want to ‘be in

85

86

86

Introduction to Multimodal Analysis

touch with’ and ‘get close to’ people we see as part of our circle of friends or intimates. In pictures, distance translates as ‘size of frame’ (close shot, medium shot, long shot, etc.).

Student activity The following suggested activities are designed to encourage you to think about the way that all kinds of semiotic materials around you, in shopping, instructional materials, and play objects, involve decisions in regard to modality. 1. Search an online grocery shopping site. Look for products which use representations of nature, animals or persons on packaging. For your analysis choose one specific thing, such as flowers, cows, fruit and so on. Look for the subtle different kinds and levels of modalities used in different cases. Explain how these modality choices help to shape the meaning of the product. 2. Look at educational resources for children online, such as for learning about history, science, the environment. Find examples of the different kinds of modalities used to represent different topics and learning areas  – naturalistic, sensory, abstract. Show where we find combinations of these used in the same learning material. Explain the meaning of these differences in each case. How is this ideological? 3. Do a Google search for toys. Or simply bring some toys with you to the class, from your younger siblings, or those of your own your parents never gave away. The idea here is to look at how different types of toys represent objects from the world in different kinds of modality. You could compare how two brands represent persons. If they are not naturalistic, in what ways is modality changed? Which aspects/parts are abstracted or represented in a sensory way?

87

4 The meaning of colour in visual design If we look at the plastic surgery webpage in Figure 1.1, we find that it has a very different use of colour than the webpage for the children’s camp in Figure 5.9. We also see great contrasts in the use of colour between the two film posters for Rambo and Cinderella in Figures 5.7 and 5.8. Generally, if we asked the people around us about these colour differences, they might say that one is more ‘calming’, ‘soft’ or ‘pretty’. They might say the camp webpage is more ‘fun’, ‘childish’ and ‘energetic’. The Rambo poster might be described as ‘gritty’. But these adjectives tell us more about the effect of these colours rather than about what we are seeing. What is it that makes colour ‘fun’ or ‘soft’? In fact we have a limited vocabulary for talking about colour. This lack of vocabulary is even more evident when we come to think about other levels of qualities of the colours themselves. Why is it that in the case of the plastic surgery webpage the colours are more muted, whereas on the Cinderella poster they are more saturated? This is more difficult to answer as we are less familiar with thinking of the symbolic values of different qualities of colours such as levels of saturation. A graphic designer commented to one of the authors that in a newspaper redesign they had changed the white they used to make it a ‘purer, flatter, brighter white’ and that they used some more muted hybrid colours across the design to create links across the page. So in this case the designer used the same colours but with different qualities. Why could this be? Clearly, there are important symbolic meanings of both different hues of colours and their more subtle qualities, including things like levels of saturation and purity. And here the designer suggests that colours are used also as a device for structuring compositions. Colour is therefore another important semiotic resource that can be studied systematically.

88

88

Introduction to Multimodal Analysis

In the world we never encounter colour as a thing in itself but as a quality of a semiotic material which already has meanings as part of social practices – in other words in terms of what we do with it, how it shapes and organizes behaviour. Even art which uses just colour, such as the paintings of Mondrian, uses geometrical shapes, canvas, and is given meaning by the discourses we use to give meaning to art as a social practice and as an industry. In Chapter 1 we explained in detail how the coding of meanings must also be seen as being produced in particular multimodal configurations that are used in specific contexts. So we cannot understand how colours mean in isolation from these. But we can nevertheless think more closely at the meaning potentials of colours. While colour use might seem more random or based on aesthetic decisions, it is used by designers deliberately, although they themselves will usually speak in terms of artistic and instinctual reasons for use rather than of colour as a semiotic resource which has more or less predictable meaning potentials. In this chapter, we are interested in exactly how we can more accurately account for the regularities and patterns in the use of colours and of colour qualities in designs. This chapter draws on Kress and Van Leeuwen’s (2002) work on colour to discuss the concepts of hue, saturation, modulation, luminosity and other qualities of colours to establish some rules for its use and analysis.

Patterns in colour as a semiotic resource Writers such as John Gage in Colour and Culture and Colour and Meaning (1993, 1999) have written lengthy volumes on the uses of colours throughout history, looking at their different symbolic values and meanings for cultures. Umberto Eco (1985), attempting to write about the ways that cultures allow us to perceive colours, realized how little people seem to know about colour as a form of communication. Gage (1999) says that language itself has been used to label only a tiny fraction of the millions of colour sensations that most of us are able to see. In Colour and Meaning he looks in detail at the many different theories of colour that humans have devised, across cultures and through history. Basically artists, psychologists, philosophers and theologians have never really come up with anything that allows prediction of meaning, although in certain times and places colours have

89

The Meaning of Colour in Visual Design

had very strict and defined symbolic meanings, as happened during times of heraldry. Kress and Van Leeuwen (2002) set out to establish just these kinds of regularities that can be found in the use of colour. Are there systems of choices available in colour of which we can create an inventory? If so, how can we describe this system?

Colour as a semiotic system Kress and Van Leeuwen (1996) draw on Halliday’s (1978) view that language always fulfils three communicative functions. We considered how these work in Chapter 1 of this book. This is the view that language must communicate ideas, it must communicate the personal stance of how people relate to those ideas and it must create coherence among its parts. Visual communication is very different than language and it would be an error to assume that this model can simply be transferred to apply to something like colour (Ledin and Machin, 2018). But nevertheless we can draw in these as a device to help us to take a first step into thinking about how colour can be used in communication.

How colour communicates ideas Colour can denote specific people, places and things as well as classes of these. Colours of flags can denote nations; corporations use colours to denote their identities. BMW makes sure its dark blue is different from the blue of Ford. On maps, colours identify water, land and forests. Here there is an iconic element in the choice of colour. In Medieval times the iconic colour symbolism was largely fixed. On knights’ outfits, white was used to symbolize cleanness of body, red to remind of the task of killing and brown to remind of mortality and the ground (Gage, 1993: 84). In the Middle Ages, green was used to express the unity of Godhead. Blue was used to symbolize coldness of heart. We find colour used to convey ideas in everyday contexts. Uniforms can signify rank. In Britain, nurses would wear darker blues to show higher status. In religious paintings such as those by Botticelli, blue was used for purity and divine light. In politics, red is used to connote leftwing ideologies. Gage says there is a common theme of white representing truth and purity and black its opposite. This is to do with metaphorical values of clarity and obscurity.

89

90

90

Introduction to Multimodal Analysis

Looking at images across the chapters in the book we can see how colours communicate ideas. The colours on the Bounce healthy snack package in Figure 1.3 suggest something about optimism and energy. There is a white shade running through the three colours. Other chocolate bars may use darker colours to communicate more about the qualities of the chocolate. On the children’s sports camp webpage in Figure 5.9, we can see brighter tones and also a wider colour palette communicating something of the idea of fun and playfulness. We also see bright colours and a wider colour palette on the Cinderella movie poster in Figure 5.8. But compared to the sports camp webpage these colours are much richer and suggest the idea of emotional intensity or richness. Other designs use one single colour, such as the Fairtrade coffee package in Figure  0.1. The image is a muted, monochrome, sandy brown. There is something measured and earthy about this design pointing to exotic settings and natural qualities, but rendered in a tasteful way. We see this more monochrome effect too on the Rambo film poster in Figure 5.7. Here too there is something earthy, yet we find the use of intense brightness and darkness, suggesting emotional highs and lows. In the inventory of meaning potentials which follow shortly, we explain these interpretations more carefully and systematically.

How colour communicates attitudes Bright colours can be used on documents in order to draw attention to them. Red might be used on warning signs. Interiors of buildings today are coloured to convey calming effects. Newspaper designers are very interested in the way that colours can be used to hold readers’ attention. We might paint a restaurant or café in particular colours to communicate ‘welcome’ or ‘sophistication’. Looking at the two storefronts in Figures 5.4 and 5.5 we can see that Bargain Booze is less formal and somehow warmer and brighter in its mood with its saturated reds, yellows and whites. In comparison, the Sandro storefront is much more serious and restrained with its darker colours and limited colour palette. This then leads to the brighter interior with its warmer natural shades. On the Rambo film poster in Figure  5.7 we see that the burnt brown colour is impure yet quite saturated and with massive ranges in tone. As well as communicating the idea of the military and the battleground, this could be seen as conveying a mood of emotional intensity of extreme highs and

91

The Meaning of Colour in Visual Design

lows. So the meaning of colour here lies not only in the hue but also at the level of its qualities. On the children’s camp webpage in Figure 5.9 we might say that the wider colour palette of primary colours is typically used for communicating the idea of childhood. We could imagine the difference were this colour palette used for the Rambo film poster or if the monochrome was used for the children’s camp. Of course, it is not really possible to distinguish moods from the way colour communicates ideas. But it is a useful question to ask in order to help us to think about how colours are being deployed for communicative purposes.

How colour creates coherence Colour can also be used to create links, coherence and contrasts on designs. It can be used to create classifications. And in this way colour can be a part of how a semiotic material hangs together as a whole. In a textbook or on a document, all the main headings might be in blue. This helps to create coherence. The same colour might be used for headings with the same status or to indicate the same kind of classification of some kind. On an advertisement we might find that the fonts used to describe a product are the same colour as parts of the product itself. Simply this helps to create linking across a design which can coordinate or differentiate between parts. A webpage like the children’s sports camp in Figure 5.9 may use colours like green and blue to create coherence across the parts. So the menu buttons may take colours from the photograph. Designers will say that this can help viewers to experience a design as more coherent, although the type and degree of coherence here can suggest more or less formality. And importantly what is coordinated, differentiated, placed in hierarchies using colour can be a matter of the interests of the designer. How two things coded as being related through the use of colour are in fact related does not need to be explained but is rather symbolized here. If we look at the Fairtrade coffee package in Figure 0.1 we can see that the ‘Level ground’ in the logo uses the same colour as the names for the product type, such as ‘Tanzania’. In other words, here colour can create links and coherence between parts. It can be used to code things as being of the same order. How things are connected, or of the same order, may never be clearly specified. And, as we show later in this chapter, things that are clearly different and even clashing can be coded by colour as being of a related order.

91

92

92

Introduction to Multimodal Analysis

On the Fairtrade coffee package we also see that the picture of the worker is coded as the same colour as the background with the muted sandy brown colour. These are the same across the product types, for Peru, Tanzania, Columbia and so on. So these workers and settings are coded as the same, in this sense. They are all benefiting from Fairtrade here. In such cases, the criticism could be made that these workers live in very different local conditions and in societies with very unique economic situations and in regard to things like the severity of IMF (International Monetary Fund) restructuring loans and to the World Bank. Yet these differences and complexities become flattened out for the consumer (Low and Davenport, 2005). But here colour plays a role in concealing these and helps to communicate our consumption, buying this coffee, as an act tied to high moral standards. On the Bounce healthy snack package in Figure 1.3 we can also see how colour is used to link what are in some ways an odd hodgepodge of things, of ingredients, qualities and branding. We have the three colour bands across the package which are bright and optimistic. These have a white shade running through them (the bone-white and the green-white). This helps to create not only difference but also coherence. And we find that font colour is used to create links across these bands. Black is used in the circle around the word Bounce and in ‘Energy Ball’, ‘Defence Boost’ and ‘High Fibre’. The font used for the brand ‘Bounce’ is white and therefore links with the words ‘Spirulina’ and ‘Ginseng’ which are also white. The font for ‘High in Antioxidant Vitamin E’ uses the colour from the green band above. These links help to create coherence across an odd mixture of ingredients, dietary notions and effects. Bounce is in fact a product which is high in sugar and fat and not unlike a standard chocolate snack like a Mars bar. Researchers have shown that it can be the case that products which are otherwise unhealthy can use certain ingredients for ‘clean washing’, and that in such cases a ‘hodgepodge’ of things can be used to suggest a product that ticks many ‘good for you’ boxes (Claith, 2007; Schneider and Davis, 2010). In the case of Bounce the use of colour plays an important role in creating an integrated coherent whole. On the university sustainability card in Figure 4.1 we can also see how colour, as well as other semiotic resources, has been used to create integration. If we look at the contents of the document we see a range of issues, such as recycling, new cycle parking places, volunteering and biodiversity. While each item has a different colour they are linked through the same kinds of colour qualities. For the most part all of these are hybrid colours like the yellow-green-white for recycling and grey-purple for volunteering. They are

93

The Meaning of Colour in Visual Design

relatively dilute, pastel, colours, but with saturated colours used for headings and diagrams. It is useful to look at how this works in a little more detail to show the work that can be done with colour. In fact, sustainability is a highly complex and abstract notion. A  look at the documents which describe its meaning shows that it can include environmental issues, sustainable jobs, a strong economy and social equality. These are things which on many levels appear contradictory. Can we indeed have a strong and growing global economy and at the same time take care of the planet? But for public institutions, such as universities, it can be necessary for them to signal to governments how they are raising their sustainability profile. So a university whose primary purpose might be education also has to invest time and resources into meeting required targets for things like sustainability. Looking at the sustainability scorecard in Figure 4.1 we can look at how one university is signalling that it is working with sustainability. But colour here is used to gloss over many contradictions and ambiguities. We see that recycling has increased. Over 3 years there has been a 2 per cent reduction in carbon emissions (represented also by the shading on the icon for a foot which is clearly much larger than 2%). On another panel, we see that in one building at the university, the use of electricity reduced immediately after a ‘go green’ week. In both cases it is hard to know how to interpret such information. Is this single building and single instance indicative of anything more sustained? Further down we see a number of volunteers mentioned. We are not told what they volunteer for, although we see an icon of a person holding a leaf. And on another panel we find information on biodiversity, being told that 100 people found over 300 species. It is not clear why it is relevant that it is 100 people, or if the number 300 is high or not. And given most people have jobs in the university such as teaching, security, catering we might wonder who was engaged in this activity. Were they not doing their own jobs? And we might ask how it is useful for a university to count biodiversity on its small urban campus. Finally, on another panel we are told there has been an increase in cycle parking spaces. One of the authors worked at this university for several years. Its city centre location makes it highly dangerous for cyclists – during the time he worked at the university he gave up cycling during a period where a number of staff were involved in road accidents. As in most British cities, provisions for safe cycling are virtually absent. This is not to criticize the university for creating cycle parking spaces, but to raise the question as to

93

94

94

Introduction to Multimodal Analysis

Figure 4.1  Sustainability Scorecard 2011/12. © University of Leicester.

how much this can actually be beneficial in isolation if not coupled to wider shifts in city planning. And, overall, we might ask how such institutions can, as well as offering education and doing research, solve such serious structural issues. The point we wish to make here, however, is that colour

95

The Meaning of Colour in Visual Design

hue and colour qualities are used to help create a sense that these things are somehow part of a coherent bigger picture, that they are of a related order. Moving to another example, we can see that colour is important for creating coherence in interior design, as we see in the image of the café in Figures 6.3 and 6.4. In such cafes, the colours of the surfaces, walls, furniture, menus and cups will have been carefully thought through. In Figure 4.3 we see the set colour palette of Starbucks, consisting of hybrid earthy colours which are carefully designed to work alongside each other. Figure 4.2 shows how this palette is used in an actual café. These will be combined with, and coordinate well with, natural textures of wood surfaces and polished metal. As seen in the photograph even the pictures of the Fairtrade workers will be integrated into the colour scheme, here through the use of monochrome. As described in Chapter 1 this integration results in a form of communication which changes the way that both language and other semiotic resources are used. Important here is the way that the kinds of causalities, identities and connections formerly made in running text are now made using combinations of chunks of text, bullet lists, combined with images, graphics, colours and other semiotic resources. On the sustainability card it was not made clear in language how all the parts worked together. The connections and logic were rather symbolized through colour, typeface and iconography. How our act of consumption of a coffee and croissant helps to address global economic inequality may not be described in clear text in a café, but the connections may be symbolized through things like colour and textures. As we said in Chapter  1, branded spaces are now deeply moral, and consumer culture has to a large extent taken over the role of traditional politics. Consumption is about making the right choices, because we are judged by our acts of consumption. We all want to save the planet. So buying a coffee or a croissant today would come out as a sort of political action, and integrated designs are used to communicate these values. This means, in Habermas’s (1987) words, that communicative rationality changes and potentially gets undermined. Communicative rationality describes our possibility to discuss, explain priorities and reach understanding in a language or design that is comprehensible and apt for these purposes and for taking political action. Nowadays we get scorecards for sustainability where very different categories are randomly measured on a Fairtrade coffee, asking us to take a stance and act politically, but using integrated designs that do not in fact explain what the matter exactly is about or why. When a branded, moral space, like a Fairtrade café, becomes the way in which we relate to each other, communicate priorities and

95

96

96

Introduction to Multimodal Analysis

Figure 4.2  Starbucks café. Richard Levine/Alamy Stock Photo.

understandings of world, it is clear that consumer culture has taken over the role of what once was politics. Returning to Habermas we can say that the communicative rationality, our striving to make sense of ourselves and the world in a comprehensible language, including taking part in elections, risks to be perverted and in the hands of big business. Through acts of consumption we make distinctions and invest in high moral standards as a commodity that, at least for those people who have the money, can be bought. The Fairtrade coffee in Figure 0.1, the design of the café seen in Figure 6.3 and the sustainability scorecard in Figure 4.1 make it very clear that what once were public and political spaces, where people would gather, discuss and take social action to change society, now are branded spaces. Politics is carried out within brand cultures, and we find a commodity activism that is deeply moral. In this branded space moral values are everywhere, and we judge each other and feel guilty by acts of consumption.

The value of colours Gage (1999) shows that from around the Middle Ages there have been attempts to define colour systems. Some writers tried to show how colours

97

The Meaning of Colour in Visual Design

related to each other or how certain colours affected the way we saw them when they were positioned together. Others began to distinguish colours in terms of qualities other than hue. Later palette systems of colour were developed which could show how basic colours could be used in combination to create unlimited new colours. Kress and Van Leeuwen (2002) see this as the move from a lexical approach to colour to a grammatical approach. This is because it begins to deal with the way meaning is produced through combinations. As well as attempts to place colours into systems, others such as Goethe (1970) and Kandinsky (1977) looked at the meaning of colours in terms of effect. This is what we might think of as the interpersonal level of colour. Here different colours could be thought of as stimulating different kinds of emotional effect. Painters began using colour to express the characters that they painted. Colours such as blue and grey were associated with modesty and calm. This was different to earlier symbolic use of colour where it was the association that was important. It is this kind of effect that can still be found in interior design books and television programmes, although there is no agreement or consistency as to the specific effect of different colours. However, this will not stop a design consultant claiming authoritative knowledge when they produce reasons for the colours in your local shopping arcade or a psychologist claiming that you should paint your prison pink as it will reduce aggressive behaviour (Walker, 1991). But much of this kind of discussion rests on the meaning of hue, so of red, blue and green, rather than on other qualities of colours. We tend to lack any specific vocabulary for other values of colours. It seems that much of the meaning of colour hues is drawn from association. So yellow is often described as sunny, green as being sickly. But much of this work remains unable to describe colour effect systematically. For example, designers at Poynter Institute in the United States, which specializes in newspaper and magazine design, after extensive research into colour seem to say little more than: ‘Colour can evoke emotions and create moods that enhance meaning. We respond to colour in physiological ways.’ They conclude that colour can increase page coherence and that people say they prefer pages that contain more colour and have the impression that they give more information (Poynter Institute, 1991). They also show how certain combinations of colours work well together. But again this is not done in any systematic fashion. This is what Kress and Van Leeuwen take a step towards.

97

98

98

Introduction to Multimodal Analysis

Semiotics of colour Kress and Van Leeuwen (2002) start from an observation made by Kandinsky (1977) that colour has two direct kinds of value. One of these is direct value or the effect that the colour has on the viewer. The second is the associative value. So we might associate blue with water, yellow with sunlight and so on. What we can do as semioticians is to analyse the sets of affordances or meaning potentials a colour can carry in terms of these two values. Kress and Van Leeuwen distinguish two kinds of affordances in colour or two sources for making meaning: ●●

●●

Association: this is to do with the kinds of cultural associations a colour might have. This might mean blue with water. But this association might also be with blue as purity as we often associate water with purity. Or blue has been associated with science, truth and knowledge. Blue has also been associated with royalty as in royal blue, which has its history in the cost of certain dyes and pigments. To establish this there is usually need to make reference to other elements in the composition. Features:  this is the attempt to describe the distinctive features of colour from which they draw on (Jakobson and Halle, 1956). Here they look at a set of scales that runs from light-dark, saturated to desaturated and so on. But they see these not merely as sets of values but as meaning potentials. And instances of colour can be analysed as being combinations of these scales.

The dimensions of colour In this section we discuss colour meaning on the basis of a wider set of dimensions that all combine to give a rich array of possible combinations.

Brightness The meaning potential of brightness rests on the fundamental experiences we have with light and dark. There is probably no culture that does not have a mass of symbolic meanings and values based on this distinction. Much of this meaning potential comes from the association of clarity and obscurity. The same metaphorical associations can be found in language where truth is associated with light, lies with darkness. God has also often been associated

99

The Meaning of Colour in Visual Design

with light and dark with evil. Bright can be associated with moods such as happy and dark with depression and sadness. As we will see, how this meaning potential is realized will depend on the combination with other colour values and with other semiotic modes. Looking at the university sustainability card in Figure  4.1 we can see that brighter colours have been chosen for this design. It therefore appears optimistic, rather than serious and heavy. The brighter colours here can also help to communicate truth and transparency, when of course, as we noted above, what is taking place here is far from transparent. The plastic surgery webpage in Figure  1.1 also emphasizes brightness. Again this can relate to optimism, and also to transparency and truth which would be highly important for the way that visitors to the site perceive the services on offer. It may be less desirable for such a site to use much darker colours and shadows. Certainly, such a site would not want to communicate a sense of concealment or obscurity. The Rambo movie poster in Figure  5.7 might be said to have overall brightness. But looking closer we see that it uses a combination of intense brightness and also darkness. Here there is sense of emotional highs and lows, of concealment and struggle perhaps. Certainly there is a sense of emotional intensity. Such a use of bright and dark would certainly not be suitable for the surgery site. The children’s sports camp webpage in Figure 5.9 has an overall sense of brightness. But we can see that in the photograph, yellow highlights have been added to help with that brightness. Clearly this is of a different order to the kind of airy brightness that we find on the surgery webpage. But nevertheless, in this case, we are dealing with optimism and happiness. The photographs we see behind the serving counter in the café in Figure 4.2 are all darker, slightly more muted and sombre. In such a case, darkness is not intended to communicate depression or concealment. And in this case the darkness is part of the monochrome style of the images which can be associated with timelessness. These images are not therefore in the style of bright optimistic stock images, but something more serious and discerning. And this coordinates well with the rather two-tone interior of the café with the authenticity of is wooden surfaces and polished metal.

Saturation This is the scale that runs from most intensely saturated colour to the most diluted versions of the same colour (either towards pale and pastel or

99

100

100

Introduction to Multimodal Analysis

towards the dull and dark). Its meaning potential seems to lie in its ability to express emotional ‘temperature’. Less saturated colours are more toned down, subtle, gentle, even peaceful or possibly moody. More saturated colours are emotionally intense, bold and engaging. They can be not only direct, adventurous but also brooding. This value may have metaphorical associations from our sense of dilution and concentration and simply of intensity or weakness of feeling. Since highly saturated colours give a sense of emotional intensity, advertisers often use these to increase the sensory visual experience of a product. Children’s products are often designed with rich saturated primary colours to suggest emotional exuberance and engaging. If we look at the Cinderella film poster in Figure  5.8 we find entirely saturated colours. On one level, this suggests a film aimed at children. But it also communicates that it is an emotionally rich film, vibrant perhaps, even adventurous. Certainly, we have no sense here that this will be about muted moods or weakness of feeling. The Rambo film poster in Figure 5.7 is very different for many reasons as regards colour, but there is also saturation on the actor’s skin and face and in the dirty colours in the background. Again, we are dealing with emotional intensity and adventure. But clearly other features such as the extremes of light and dark, hues and the dirty look of colours (which we come to shortly) communicate something very different to Cinderella. On the storefront of Bargain Booze in Figure 5.10 we see rich saturated reds, yellow and white. Here there is a sense of emotional engagement that we do not see on the Sandro storefront which is more about muted subtle moods, particularly in the interior. The university sustainability scorecard in Figure  4.1 uses saturation as well as more muted shades. The muted more pastel colours are used for the main panels. So it is subtle and less engaging, more suitable perhaps for the meanings of sustainability. But then saturated colours are used to add some emotional excitement and engagement such as the red on the ‘Green Week’ panel, for the green segment on the earth in the ‘Recycling’ panel and also for the headings such as ‘travel’ and ‘getting involved’. Importantly here the different parts are classified as being of the same order due to this use of colour. So volunteering is classified as the same as recycling and as biodiversity. This helps this odd collection of things to be presented as a coherent whole, when it could be argued that they are not. Who is doing this volunteering and for what? What is the significance of the amount of people who found species on campus? Is the number 100 important here?

101

The Meaning of Colour in Visual Design

The children’s sports camp website in Figure 5.9 carries the kind of colour scheme associated with children, yet on the whole these are less saturated. This may suggest that the camp is for slightly older children, although as we note in Chapter 5 in this design, much ‘fun’ is communicated in the kinds of fonts that are used. The page does have some highly saturated yellow highlights which have been edited into the photograph. The yellow itself brings brightness and warmth to the scene, with suggestions of summer. But this saturation also helps this to bring a little emotional energy. On the Fairtrade coffee package in Figure 0.1, we see that the colour used for the section with the image not only has warmth due to its hue but also has a muted dilute effect. Sometimes these Fairtrade brands use highly saturated colours to suggest something like ‘world culture’ and ‘ethnic vibrance’. But here the warm diluted hue is almost soothing. This is not the same kind of message as the emotionally direct Bargain Booze, but one which is more subtle and thoughtful.

Purity This is the scale that runs from ‘purity’ to ‘hybridity’. A pure colour might be a pure red or blue, such as those seen on a colour palette. Pure colours are also seen on children’s toys which often use primary colours. Kress and Van Leeuwen (2002) say that purity and hybridity already suggest something of the meaning potential of this aspect of colour. They give the example of the pure, bright reds, blues and yellows of the modernist paintings of Mondrian. This kind of colour scheme with its certainty and flatness of colour is a key signifier of the ideology of modernity, reflected also in the buildings of architects such as Le Corbusier with straight lines and functional spaces. At the other end of the scale, a colour scheme of diluted impure, uncertain colours could become a key signifier of postmodernism. In this case uncertainty is emphasized over the certainty of modernism. In architecture, postmodern buildings may reflect a pastiche of style and even have typically internal features located on the outside. If we look at the Cinderella film poster in Figure 5.8 we find pure colours, such as the red of the prince’s trousers. There is sense in such films that there is a lack of ambiguity in what will take place. In contrast, the colours on the Rambo film poster in Figure 5.7 are impure. They are all merged and flecked with other colours. We could imagine the same poster with pure clean colours as we found on Cinderella. Clearly this would transform our

101

102

102

Introduction to Multimodal Analysis

Figure 4.3  Starbucks colour palette.

expectations. The Rambo poster suggests a dirty, contaminated world. There may be a lack of clear truth, through the moral complexity of war. Rambo may kill everyone by the end of the movie. The bad people may pay for their actions with their lives, but Rambo is himself troubled and lives in a troubled world. The Bargain Booze storefront in Figure 5.10 uses pure reds, yellows and white. There is simplicity in this emotionally engaging saturated palette. Looking at the Sandro store, to the interior we see the use of one neutral type of colour for the walls and ceiling. In such stores this may well be a hybrid type colour, some kind of variety of off-white. Such colours can suggest something slightly more complex. But here this is done also with a sense of measure, given the use of just the one colour throughout. And such colours here interact with the softening authenticity of the wooden floor. If we look at the Starbucks café in Figure 4.2, we find hybrid colours which are used in strict combinations. We see a kind of yellow orange in the middle, a grey purple sort of plum colour to the right and to the left of that a similar colour with more brown in it. All are blends of green, blue and red to different degrees. All have something slightly earthy in them. And they are designed to sit well together. Starbucks also has a dress code book where it describes what kinds of colours and textures of clothing employees may wear with their Starbucks apron. This allows an ‘individual look’ but one which must be coordinated with this palette. In the café interior we may find such colours in large flat uncluttered panels in the fashion of modern art. But the hybridity suggests subtle kinds of creativity, something slightly alternative, as well as having a greyish earthiness to them which suggest something natural and unpurified. We also see these hybrid colours on the Fairtrade coffee package in Figure 0.1. We have the sandy brown for the image where we see the peasant

103

The Meaning of Colour in Visual Design

worker. But then for the font which presents the sources for the coffee we find different colours. We see yellow orange. A sort of brown-grey-purple. Again these are hybrids. Clearly such colours would be confusing if we found them on children’s toys. But such colours can suggest both lack of purification and processing which a pure colour might suggest, and they can suggest a sense of creativity. These hybrid colours are often found, as in Starbucks, for designs which seek to communicate urban chicness. But as we see here, and on the Bounce Package in Figure 1.3, this is done using hybrid colours which have other shared properties running through them. The Bounce design has white running through the different colours. One of the authors has interviewed many designers of newspapers and websites (Machin and Polzer, 2015) who spoke specifically of the shift to more careful uses of colour. This includes attention given to purity. A pure white may be used to suggest modernism, openness, transparency. This may be combined with softer hybrid colours, some of which may have a little energy such as plum, to communicate ideas and fresh perspectives. This may not actually involve any fresh perspectives in the content at all. But this was part of a wider shift of building titles first from a visual point of view, before adding content.

Modulation This is the scale that runs from colours that are fully modulated, that are more natural, which are textured with different shades, as colours are seen in the natural world, to flat colours with no such shades. The paintings of Cezanne characterize the former where objects have shadings to denote light reflection and perspective. The use of colours with low modulation is typical of cartoons and again the paintings of Mondrian. Here colours show none of the nuances and variation created by the play of light and texture. Flat, unmodulated colour may be experienced as simple, bold or basic. Highly modulated colour may be perceived as subtle and doing justice to the rich texture of real colour – or as overly fussy and detailed. We might think of colours that are of low modulation as being generic colours. These represent the essence of a colour or an idealized version of a colour. The use of unmodulated colours in photographs can also lower modality. Clothing, interiors and skin can all be shown to be flat and without the natural shades that light and perspective can create. Of course, this can make a world

103

104

104

Introduction to Multimodal Analysis

seem clean and certain, which is important for consumerism. We might see adverts for home interiors with babies crawling over a polished wood floor. Everything seems clean and simple, partly not only through the use of limited props and light but also through digitally flattening the colours. This connotes a world where things such as happiness and consumerism have certain links. We find these flat unmodulated colours on the Cinderella film poster in Figure 5.8. The world in this film is a simple idealized world. We often find such flattened colours for films such as romantic comedies, again where there is a sense of a simplified form of social reality and social relations. In contrast, on the Rambo film poster in Figure 5.7, we find higher modulation, even exaggerated in some parts, such as where the scars stand out on his chest. Yet we also find lower modulation such as in the gun, and with the levels of saturation the scene appears part image and part artwork. The higher modulation may seek to point to grittiness and realism, at the same time we are told this is fantasy and less than real. At the other extreme, we can see that the person holding the leaf on the sustainability scorecard in Figure  4.1 is simply a flat panel of colour, and therefore completely idealized. This may sound obvious. But were this design to use a person in naturalistic modulation, or even slightly idealized, this could complicate the way the design is able to simplify and gloss over the details of what sustainability means. On the Fairtrade coffee package in Figure 0.1, we can see high modulation on the faces of the peasant workers. Here we might say that there is a sense of the nuances of the actual people. Yet the muted, warm, sandy brown colour and the monochrome have the effect of also coding the people into the brand. So in a sense at another level the representation of actual details is reduced. Here monochrome is used for its more classic look, with the addition of the exoticizing effect of the colour.

Differentiation This is the scale that runs from monochrome, which could be black and white or just shades of the same colour, to the use of many colours. This can be used to suggest timelessness as in black and white and might be used to make images seem symbolic rather than descriptive. Low differentiation can also favour certain kinds of meanings. Where there is only one colour, the hue and other values of the colours themselves

105

The Meaning of Colour in Visual Design

carry the meaning. The monochrome effect gives much symbolic power to the single colour used. Low differentiation can mean restraint, although it can also mean classiness. This is often used in advertisements which draw on nostalgia. High differentiation can mean adventurousness, fun or energy. It can also mean unserious, gaudiness, lack of taste or restraint. We can see the use of a larger colour palette on the Cinderella poster in Figure 5.8. We find colours across the spectrum, with red, blue, yellow, white and so on. With the levels of saturation and lack of modulation of the colours, this communicates a sense of fun, energy and lack of seriousness. At the heart of this film lies an issue of massive social inequalities in society and abuse within families. But the colours suggest that this may not be the actual focus of the film. In this case a monochrome blue design, as used by Picasso in some of his more depressing paintings, or just simple black and white might suggest something different. On the Rambo film poster in Figure  5.7 we find one dominant colour across the design, a kind of yellow-burnt-brown, along with some white, which becomes sometimes more yellow and at other times more brown. Clearly the intensity of bright and dark which this creates suggests highs and lows and much energy. We can say therefore that this image symbolizes emotional intensity, highs and lows of optimism and uncertainties, but certainly that these are not playful or fun and are in some ways highly emotionally contained. We see the use of monochrome on the images of the people in the Starbucks café in Figure 4.2. We also find monochrome on the images on the Fairtrade coffee package. Here, there is sense of restraint and more classic associations. We often find images of ‘ethnic others’ represented with a large bright saturated colour palette to suggest the vitality and energy of human diversity. The workers here become coded into this kind of mood. As we have seen, Starbucks monitors its staff for the colour of their clothing. So too must the peasant workers be coded into the product and brand. The plastic surgery clinic website in Figure 1.1 uses a restricted colour palette. We see different levels of saturation of the blue, which is also used for the font of the menu. Some extra colour is added in the contact use box above, also linking with the logo to the left. But for such a site it may be less desirable to have a larger colour palette to suggest fun and energy. The key here is restraint and gentleness.

105

106

106

Introduction to Multimodal Analysis

Luminosity This is the scale from luminous colour, which looks as though light is shining through it (e.g. coloured glass), to its opposite. Luminosity has a long history of being associated with the unworldly glow of magic and supernatural beings or objects. Today, it is increasingly applied in architecture. Luminosity will be found on film posters to show other-worldliness, such as for the science fiction Alien series. We will also find it on the Harry Potter or Lord of the Rings posters to connote magic.

Fluorescence This is where colours appear to glow. The fluorescent look brings a vibrant energy from within. We can see this on the Bounce health snack in Figure 1.3. While colour here – the bone-white and the grass green – suggests naturalness, the florescence suggests vitality and energy. We often find this kind of effect on health-related food products and also those related to cleaning. Here glowing white can suggest cleanliness, simplicity. Such an effect may be avoided on a product such as a twenty-year-old whisky.

Hue This is the scale from blue to red. On the blue side there are associations of cold, calm, distance and backgrounding. On the red side there are associations of warmth, energy, salience, foregrounding. Yellow and orange can be optimistic and energetic. We also find colours which are more neutral such as greys and beige and those which associate more with nature such as greens and browns. The Cinderella film poster uses warm reds, royal crimson, warm purple and gentle violet as well as yellow. The large colour palette is lively and fun, but it is also very warm. Other cartoon film posters may use much more yellows, greens and blues for foreground fun and craziness. The Rambo film poster uses burnt brown and yellow. These are the colours of dirt and the military of weapons firing, explosions and burning. The plastic surgery webpage in Figure  1.1 uses blue for calmness and distance. The Bargain Booze storefront in Figure  5.10 uses yellow for optimism and brightness and red for energy and warmth – it is an everyday

107

The Meaning of Colour in Visual Design

friendly sort of store. An upmarket wine or craft beer shop may require a different colour scheme. The Bounce package in Figure 1.3 has a mixture of natural and optimistic hues. At the top we have the green yellow, a combination of the natural green and the bright optimism of yellow. The middle layer is also green, grassgreen colour, which appears to contain some white. At the bottom is a whitegrey or bone-white. This is a white that also appears a little more ‘earthy’. All of these hues are a little fluorescent. So here we have hybrid colours that are able to suggest not only chic creativity but also nature and optimism. The fluorescent look brings a vibrant energy from within.

Summary of colour dimensions for Cinderella poster ●●

●●

●●

●●

●●

●●

Hue: range from the warmth of red to the coldness of blue. Use of reds and lilac for warmth and femininity and whites for order and cleanliness. Brightness:  truth as opposed to darkness. Very bright and optimistic. Saturation: exuberance as opposed to tenderness and subtlety. Highly saturated and exuberant. Purity:  modernism and certainty. Complete purity signifying modernity and certainty of social values. Modulation:  are there different shades as in real colours? No, flat generic colours. This is a simplified world. Differentiation: full colour to monochrome, energy to restraint. Full colour range, energetic, not classy, lack of restraint.

Summary of colour dimensions for Rambo poster ●●

●●

Hue:  range from the warmth of red to the coldness of blue. Vibrant yellow-brown colour to show military/excitement. Brightness: truth as opposed to darkness. Both extremes, deep shadow and extremes of bright yellow and white.

107

108

108

Introduction to Multimodal Analysis

●●

●●

●●

●●

Saturation: exuberance as opposed to tenderness and subtlety. Lots of saturation for emotional intensity. Purity:  modernism and certainty. Very impure, contaminated, lots of uncertainty. Modulation: are there different shades as in real colours? Both exaggerated for sensory feel and also reduced for idealization. Differentiation: full colour to monochrome, energy to restraint. Monochrome giving seriousness.

Summary of colour dimensions and meaning potentials ●● ●● ●● ●● ●● ●● ●● ●●

Hue: range from the warmth of red to the coldness of blue. Brightness: truth as opposed to darkness. Saturation: exuberance as opposed to tenderness and subtlety. Purity: modernism and certainty. Modulation: are there different shades as in real colours? Differentiation: is there a full range of colours or monochrome? Luminosity: is the colour opaque or does light shine through it? Fluorescence: is the colour glowing with vitality?

Student activity Here the aim is to simply increase awareness of the ways that colour is deliberately used in different settings. It is also to familiarize yourself with documenting semiotic choices and creating colour profiles. Often it is this level of description that is the most challenging for those starting with this kind of visual analysis. All of these can be presented as mini-research projects to the rest of the class. 1. Visit a supermarket and make photographs or look at an online store. Choose a product where there are variations for different market platforms, such as gender, age groups. Make photographs. This could be cereals, shampoo or yoghurts, for example. Create a colour profile for two of these to make a comparison. Do these colours tend to run across products which appear to be targeted at that particular group?

109

The Meaning of Colour in Visual Design

2. Go to different cafes and restaurants and take photographs of their menus. Explain how different types of places, such as fast food, bistros, ethnic-themed, use colour. 3. Choose two webpages or social media platforms in order to make a comparison. Here we are looking for how colour is used in the two cases across the designs. The aim is to create an account of how colour is used to communicate specific ideas and moods, and also in terms of composition, how colour is used to create linking across the parts, to differentiate. Here think not only of hue but also of other colour qualities.

109

110

111

5 The meaning of typography The two movie posters we see in Figures 5.7 and 5.8 are obviously intended to give potential viewers a promise of a certain kind of film. At one level, of course, it does not require any deeper form of analysis to see that one of the films is likely to be about macho violence and the other about princesses, fairies and romance. But, part of how these posters communicate, to which we, as viewers may generally give little thought, is through the fonts they use. And changing the fonts can completely change the meaning, even when we nevertheless still see an image of a muscular man with a gun, or a princess and dancing animals. Looking at the two film posters, there is an obvious difference between the ways that ‘Rambo’ and ‘Cinderella’ are written. If we asked people to describe how they are different, they might reply that one looks ‘big’, ‘tough’ and ‘macho’, and the other ‘softer’, ‘pretty’ and ‘gentle’. Clearly these typefaces have been chosen as an important part of communicating what kind of films these are, and to give ideas about the nature of the participants. We can imagine the difference were we to switch the two fonts so that ‘Cinderella’ was written CINDERELLA with a worn grainy look. Or if ‘Rambo’ was written Rambo in a bright saturated colour. This would communicate something very different. As viewers, we may, in this case, assume that they were parody movies. Perhaps that Cinderella is a comedy film where she rejects the patriarchal and social class systems typically represented in such stories. Maybe Rambo becomes uncomfortable with his role as isolated all-out-warrior. Although in such cases there would likely be other cues in the design to tell us about this. Rambo might look annoyed at being dirty, Cinderella might look out of the frame defiantly. But the qualities of these fonts can communicate quite clear ideas and associations. As we explained when talking about coding in Chapter 1, semiotic resources come out in configurations in different contexts. Typography is obviously tied to writing and, depending on context, it will always be linked to other

112

112

Introduction to Multimodal Analysis

resources, as in a newspaper headline, a logo or a film poster where it might interact with the look of a hero. Our examples will point to such configurations, but our focus in the chapter lies in looking more closely at how such typefaces can communicate. There are clear differences between the two fonts used for Cinderella and Rambo which we can describe. One takes up much more space than the other. One is much fatter than the other. One is much curvier than the other. These are choices which have been made by the designer, in order to communicate specific meanings. If we changed just one of these qualities, say if the Rambo typeface was made slimmer, it could change the meaning. In this chapter we provide an inventory of typeface qualities, looking at the meaning potentials for each. These potentials draw heavily on metaphorical association and transport of meaning. The basis of this transport of meaning of letters can be clearly found in simple physical and cultural associations. Consider the meaning potential of the lines in Figures 5.1 to 5.6. These have been described by McCloud (1994: 125) and Horn (1998: 147). In the case of the vertical line, the association is with walking tall and with status. This line is important or proud. Of course, by association height could also be negative. Tall could also mean stiffness, pompousness or arrogance. The slightly thicker vertical line could mean strength and solidity. We are used to the idea that in the real world thicker, wider things are generally far more difficult to break or to push over. But this thickness, as well as being positively associated with strength, could also be used negatively to suggest inertia, dogma or traditional, for example. The horizontal line is lower, suggesting lack of status and lack of assertiveness, perhaps even passivity since it simply moves along at the same level. Since we read horizontally, we can imagine its continuity and timelessness, although again we could have the opposite association where this means subtle, understated and steady.

Figure  5.1  The line in Figure  5.1 seems passive and timeless compared with that in Figure 5.2.

113

The Meaning of Typography

Figure 5.2  Whereas the one in Figure 5.2 seems proud and strong.

Figure 5.3  The thicker line in Figure 5.3 might seem proud but also more solid and immovable.

Figure  5.4  The one in Figure  5.4 in comparison seems dynamic and changing.

The diagonal line suggests movement  – imagine a sprinter running. The angularity suggests something dynamic, perhaps by association with something falling. Cars often have sloping backs rather than being upright and square, to give the impression of speed and power. The jagged and soft lines have associations of pain causing and softness, respectively. McCloud notes that these kinds of associations are fundamental in the art of cartooning where jagged lines simply placed around a person can

113

114

114

Introduction to Multimodal Analysis

Figure 5.5  The line in Figure 5.5 seems severe and unwelcoming.

Figure  5.6  The line in Figure  5.6, in comparison, seems softer, warm or gentle.

mean irritation or anger, whereas softer wavy lines can mean dreaminess or contentment. It is such simple associations which lie at the heart of the meaning potential of the lines that make up letters. These can be systematically described in an inventory.

Typeface and design Typefaces have always been used to convey different kinds of meaning. Graphic designers have long been aware that bolder fonts can be used to add emphasis to a word or piece of text. Curved fonts have been used as they seem more gentle and feminine. But in more recent times the variety and sophistication of fonts have vastly increased (Lupton, 2004). Advertisers now think carefully about the font used for their brand, since this can help connote core values. Movie posters, toys, newspapers all spend much on development of typefaces. As much as the semantic meaning of the word, the font itself is an important vehicle of communication.

115

The Meaning of Typography

Van Leeuwen (2006) suggests: This move towards a new role for typography is not restricted to the work of professional designers, but affects all writers. The time of the relative uniformity of handwriting and, especially, typewriting, is over, and the basic tools of the typographer are now available to every word processor user. The problem is, despite the programmatic announcement of the new typographers, we do not yet have that ‘complex grammar’. We do not yet have the concepts and techniques we need to be able to analyse and evaluate the communicative work done by typography today.

This more systematic approach to typography has begun to take place in market research. The Poynter Institute in the United States has been hugely influential in this field. Of typeface, Poynter Institute graphic designer Ann Van Wagener (2003) says: The headline style a paper chooses for a story should reflect the tone and spirit of the story. A story about the legislature probably wouldn’t have the same tone as a shoot-out.

She gives the following example:

Waves of death Waves of Death

Serif down style

Serif up style

WAVES OF DEATH

Serif all caps

WAVES OF DEATH

Sans Serif all caps

Waves of death

Sans Serif down style

115

116

116

Introduction to Multimodal Analysis

She explains that the first two typefaces are good for news headlines as they are solid and emotionally neutral. She says the problem with the third is that the caps lead to the headline being smaller, reducing the impact. Also the serif style allows more white to show which means the headline takes on a more ‘quiet personality’. She sees this as better for a feature story. Number four, she says, has impact and drama and will grab readers’ attention and is therefore good for breaking news. The last one is still bold and good for a follow-up story. These observations indicate the attention that is now given in page design to typography, to the meaning potentials of size, boldness, serifs and fullness of letters. The massive growth in attention to the systematic use of typeface in design can be thought of in terms of the notion of ‘technologization’ that we discussed in Chapter 1. There has simply been an increase in the control and management of all forms of communication in order to fulfil very specific aims, in order to design the message more accurately and to target specific viewers or market groups. For the design of newspapers, web pages and food packaging extensive market research will go into choosing the right font. The authors of this book have carried out research involving the redesigns of leading newspapers around the world, revealing the level of attention given to things like font style and how it is used on the page (Machin and Polzer, 2015). And, since it is now so easy to do, there are different levels of amateurs who may in fact find themselves choosing the font for their own blog or, since they know how to use the software, the website at the small business where they work. This may turn out okay provided they look around at other similar websites first. Of course, then again, it may not. What is of note, however, in the account of the news headlines above is that the explanations move quickly onto affect rather than actual detailed description. So one has a ‘quiet personality’, for example. But what is it here that gives this meaning? Throughout this book there is emphasis on the need for careful description and the creation of inventories of meaning potential. It is only through such an approach that we can develop a systematic toolkit both for analysis of existing texts and to allow us to be more precise about the way we make communicative choices. We want to be able to say, for example, what it is that allows a font to be ‘quiet’. We would not, for example, say that the Rambo font on the poster in Figure 5.7 is quiet. But why not? Where is quietness or ‘loudness’ in a font? What follows in this section is an attempt to provide such an inventory.

117

The Meaning of Typography

Typography as a semiotic system Kress and Van Leeuwen (1996) draw on Halliday’s (1978) view that language always fulfils three communicative functions. We have already looked at these in earlier sections. Language must communicate ideas, it must communicate the personal stance of how people relate to those ideas and it must create coherence among its parts. While visual communication is very different than language we can draw in these as a device to help us to take a first step into thinking about what a font is being used to do.

How typeface communicates ideas This is the ability of a semiotic system to represent what is going on in the world. In language this can be done by words that represent qualities and actions. So, we can say ‘the person is strong’. Here we communicate a particular idea about a person. But this can also be done by font. For example, on the Rambo poster in Figure 5.7 the typeface is used to illustrate the idea of strength and power. We certainly do not get the idea that he is either subtle or gentle. The fonts for the brand name on the Bounce healthy snack package in Figure 1.3 are rounded, appear soft and of a more medium weight. That they sit unevenly on the line, communicating ‘fun’, ‘energy’. This unevenness and roundness would not be suitable for Rambo. We can see looking at the web page for the plastic surgery clinic in Figure 1.1 that the fonts are quite fine. They are also rounded rather than angular and appear therefore gentle. They are also quite horizontally spread rather than tall. The language tells about ‘Tummy tucks’ which is a rather gentle and reassuring way of describing a specific kind of surgery. The font choice here helps to convey ideas of gentleness and of something discrete. In Chapter  1 of this book we looked at how these semiotic resources can communicate discourses. The fonts used on the Rambo and Cinderella posters communicate discourses about gender roles. Such fonts which seek to address people in a gendered way can be found on shampoo bottles, on food packaging, in advertisements for cars. How we encounter such everyday materials is therefore coded with certain discourses about femininity and masculinity. And these font choices may come in typical configurations with colours, textures and forms. Gender qualities therefore become built into our material world in ways in which it may be less easy to challenge. People may complain that clothing for little girls is pink. But they may also grow up

117

118

118

Introduction to Multimodal Analysis

Figure  5.7  Rambo film poster. ‘Rambo’ directed by Siddharth Anand. Capital Ventures, Original Entertainment, Impact Films and Siddharth Anand Pictures 2018.

in a society where fonts, colour qualities, textures and forms seek to tutor them in particular ideas, values and discourses.

How typeface communicates attitudes It is not so easy to distinguish between the way a font communications attitudes from how it communicates ideas. But it is, nevertheless, a useful

119

The Meaning of Typography

Figure 5.8  Cinderella film poster. ‘Cinderella’ directed by Clyde Geronimi, Hamilton Luske and Wilfred Jackson. Walt Disney Productions 1950.

exercise to think about this. In language we can indicate our attitude towards something. This means that we can create demands, persuade and so on. In speech this can be through the words that we choose and through the tone of our voice. A lecturer might say that ‘You will not use mobile phones during class’, suggesting a different attitude than if they stated ‘I would rather you did not use mobile phones in class’.

119

120

120

Introduction to Multimodal Analysis

Typography can also do something related. For example, a large bold font can be used to represent emphasis. The font used for Rambo is big and bold for emphasis on forcefulness. Advertisements often use joined-up writing to suggest informal address, to give a personal touch. A sign in a shop might say ‘Sale’ in writing that looks like it has been splashed on hastily with a paintbrush. This may communicate urgency. The font on a solicitor’s plaque in contrast will be formal and permanent. We might say this is addressing you through the weight of its formality and own importance. In this sense typography can be used to express attitudes such as serious, official or fun. The uneven, curved (and balloon-like) font on the Bounce packaging seen in Figure 1.3 would clearly be less desirable on an official institution document which reported on a management decision to close a department and make redundancies. In this case it may be felt that the management appeared to take a frivolous approach to such a grave matter. The font used for the Cinderella film poster would also likely be found as inappropriate.

How typefaces create coherence Typefaces are of course encountered as part of semiotic materials and never alone in isolation. These materials, such as a food package or a movie poster, have meaning to us as part of specific social practices, of shopping, staying healthy, of watching a movie with a loved one. And here we can analyse typefaces as the inner level where that material is tuned to communicate to more specific groups and carry certain discourses. But typeface can also be used to help that inner level hang together, to form something coherent, the design as a whole. And it can be used to create links and coordinations, or hierarchies and differences across that design. In Chapter  1 we talked about this coherence in regard to ‘integrated design’. This relates to ‘technologization’, where there has been an increase in the codification of semiotic resources to facilitate greater control over communication. It also relates to how the roles formerly taken by writing have become taken over by other semiotic resources. The result is a form of communication which changes the way that both language and other semiotic resources are used. Important here is the way that the kinds of causalities, identities and connections formerly made in running text are now made using combinations of chunks of text, bullet lists, combined with images, graphics, colours and other semiotic resources. Typography is just such a semiotic resource.

121

The Meaning of Typography

On the Bounce package in Figure 1.3 we can see that the word ‘Bounce’, the product brand name, has the largest font. This creates a hierarchy of salience over the other words. It could be bad for product recognition were some of the other words more salient, such as ‘high fibre’, although in other cases, such as the Fairtrade coffee package in Figure 0.1, the ‘Level Ground’ font is much smaller than that used to identify the different places where the product is sourced, such as ‘Tanzania’. Here the logo, therefore, allows the worker and the place to have salience on the package. The logo, the brand, takes a facilitator role. This may not be explained clearly in language but is symbolized by these size differences. And of course the relationship between the brand and the workers in the different countries is symbolized by the icon of the two hands touching and also by the image of the happy person. As we explained in Chapter 1, our acts of consumption today take on moral values. Here the semiotic configuration of typography, logo, image and icon set up the possibility to invest in our moral standards. We sense that this coffee is hand-picked, natural, where working conditions are good, and that we also might help the workers to get a happy life, as they have to cope with living in a poor country. Returning to the Bounce packaging, we can think more about the way that fonts can be used to communicate ‘sameness’ or ‘difference’ between things on a design. On the packet we see that the fonts at the bottom for ‘No artificial preservatives’, ‘High fibre’, ‘Gluten free’ are the same. This is more formal than the ‘fun’ font used for ‘Bounce’. This is a slim modern font, which is slightly angular and technical, suggesting ‘clear information’. So here we are told that the ‘fun’ product contains these more ‘serious’ qualities. This is not communicated in the language itself. But the sameness suggests they are on the same level, the same kind of thing. We are not told in running text what the health effects of Bounce are. But these help to connote healthiness. The font for ‘high in antioxidant vitamin E’ is the same as for the other ‘good features’, although this is of a different colour and larger in size, giving it greater salience. Then below this in an even smaller version of the same font we find ‘which contributes to the protection of cells from oxidative stress’. What we see is that the ‘good for you’ qualities are coded as the same, or classified as the same, in regard to font. In fact the research literature on food labelling informs about the ambiguous nature of how many of these ‘healthy’ or ‘beneficial’ ingredients or qualities work. In fact, Bounce is a high-sugar and high-fat product, not unlike the standard confectionary Mars bar. In this case such ingredients, it has been suggested, often serve as

121

122

122

Introduction to Multimodal Analysis

a kind of clean washing (Claith, 2007; Schuldt, 2013). It has been observed that they can become a hodgepodge of ‘good’ things, but in fact may offer little as regards a real alternative to standard snacks (ibid). Importantly, the design here helps to manage this hodgepodge. As we said, typeface does not work independently of things like colour, layout, borders and images, which we will deal with in other chapters, and once again we have a semiotic configuration that is ideologically potent because of its moral character. There is no big difference between Bounce and Mars, both are high in sugar and calories. But the integrated design of the Bounce bar evokes something healthy and environment-friendly, and this clean washing is done to try to persuade us that we can buy and eat it with a clean conscience. In Figure 5.9 we see a beautifully designed web page for a children’s sports camp. We see the menu across the top where all the choices are realized in the same font. Font here again plays a role in classifying these as being of the same order: ‘About us’, ‘Fun photos’, ‘Registration and fees’. All are realized in a handwritten style font resembling something written quickly with a thick marker pen. ‘Fun photos’ and ‘Registration and fees’ are in fact quite different things. But coding them in the same font classifies them at the same level of points of entry into the website. The same fonts also appear below, but slightly larger for ‘registration’ and ‘sports camps’. So, the main list of what takes place at this camp and what information is available is communicated as informal, spontaneous. This would be fitting for a children’s camp. The use of this font at the top and lower down also creates what designers often refer to as ‘rhythm’ on the page. We find a different font used for sections labelled ‘total sport camp’ and ‘sign up’. This is not a handwritten style as the menu choices but a printed one. Printed fonts as we saw in the example of the Bounce packet, used for the list of ingredients and qualities, can be more formal than handwritten styles. But here there is still sense of informality and fun as the letters are uneven in size and alignment. So, we find the same theme of fun maintained but less of the spontaneity and impermanence of the handwritten fonts. We then find a more measured font used for the main text body and for the more direct actions such as ‘view pricing’ and ‘pay online’. Here the font suggests the more serious or information parts of the web page. Overall on this web page the fonts help to bring different ideas and attitudes and also different kinds of relationship across the design. In many contemporary high-end designs, it is common to find three fonts which work together well, as we find on the Sport Camp web page, used to create subtle links across a design.

123

The Meaning of Typography

Figure  5.9  Children’s sports camp website. © 2009 Scott Durzo’s Total Sports Camp. All Rights Reserved.

In fact the sameness symbolized by fonts on any design maybe highly contestable. A  news site may have ‘international’, ‘national’, ‘culture’, ‘business’ along the top. On the one hand we might question why the world is divided into these specific components and not others. Several decades ago ‘business’ may not have appeared at all. And how can we set culture apart from other aspects of our society? Yet all appear of the same order and why this is so is something we can question. In Figure 4.1 we see a diagram called a ‘sustainability scorecard’ where a university seeks to present rather mixed jumble of fairly abstract things as being of the same order to show they have some kind of coherent set of process to show how they address the need to be ‘sustainable’. Here font is one resource used to code and classify this jumble of things as being of the same order, and to give a sense that there is indeed something coherent taking place, where a closer consideration shows this is clearly not the case.

123

124

124

Introduction to Multimodal Analysis

In sum, typography can be used as a way of classification of elements and features on a design. It can indicate links between elements by using the same typeface. It can indicate difference using different fonts in order. Headings might be of a different typeface than subheadings, for example. This can create a hierarchy of informational value. Or simply very different fonts can be used to carry different ideas such as more fun or formal information such as we saw in the web page above, or on the Bounce package.

Summary: The three communicative functions of typography Using the example of typography, we can say that the three functions are to: ●●

●●

●●

represent ideas – for example, a wide bold font could suggest durability; represent attitudes  – an uneven font could suggest fun, playfulness, informality; give coherence  – the same font can be used throughout a document to signify something is of the same order, of a different order or to create hierarchies.

Inventory of typographic meaning potential Below are listed features of letterforms described by Van Leeuwen (2006) that we can use to create typographic meaning. He suggests that together they can create a kind of ‘typographic profile’. But we have to remember that the meaning in such a typographic profile is only meaning potential that will be actualized when the letterforms are (1) combined with other features (colour, dimensionality, texture, etc.) and (2) used in a specific context.

Weight This is simply to do with how bold or ‘heavy’ a typeface appears. This can be the difference between bold and regular versions, for example, between

125

The Meaning of Typography

this and this. The difference is not about oppositions but a continuum going from maximally bold to maximally lighter. Increased weight is often used to increase salience. Today I saw a sign that said, ‘Not for passenger use’. Here, weight is given to the ‘Not’ for emphasis. If we looked at a document and saw certain words given such salience we would assume that these were the most important ideas contained in that document. This could be seen as communicating moods and attitudes. But weight can also be used metaphorically to communicate ideas. Bold can be made to mean ‘assertive’ or ‘solid’ or ‘substantial’, by the association with taking up space and weight. This could also mean ‘daring’ as the font ‘shouts out’ or is ‘bold’. But this kind of confidence, boldness or heaviness can mean overbearing, domineering. The opposite of bold, where typefaces are much slimmer, can give the sense of being more reserved, subtle or gentle. But this can also mean timid or insubstantial. Ann Van Wagener (2003) of the Poynter Institute points out that bolder typefaces might be used for stronger headlines, whereas a follow-up might be a little more reserved if it is to appear slightly more reflective. If we look at the Rambo film poster in Figure 5.7 the main font is bold and appears heavy, suggesting strength, endurance. In a movie poster for a film like Cinderella in Figure 5.8 we would not expect to find such emphasis. Rather, we might find much less weight to the letters, indicating sensitivity, reserve or thoughtfulness. The fonts on the Bounce healthy snack package in Figure 1.3 are slightly heavier for the brand name ‘Bounce’ than they are for the healthy qualities listed below, such as ‘gluten free’. Here ‘Bounce’ is given greater salience, and also there is a suggestion of something slightly substantial, perhaps relating to the nature of the energy boost. Other health products may use extremely delicate looking lightweight fonts to suggest slimming qualities or lightness of the body and well-being. This may be reversed where the brand is given less weight than other product information. In the case of the Fairtrade coffee packaging in Figure 0.1 the brand ‘Level Ground’, a play on words to suggest both levelling the ground of trade and also ground coffee, is much less bold than the types of coffee. The brand name sits quite discretely behind the type of coffee and the picture of the worker. In the case of a Fairtrade product it may be useful that the brand does not appear at the forefront. Few of the fonts on the sports camp website in Figure  5.9 are bold and heavy. Both the fun, handwritten style fonts and those used for more

125

126

126

Introduction to Multimodal Analysis

formal information are quite light. The font here then certainly does not communicate things like ‘bold’ or ‘daring’. Here we are dealing more with gentle, but safe, fun. Finally we see on the plastic surgery website in Figure 1.1 that the words ‘tummy tuck’ and ‘abdominoplasty’ use fonts that are certainly not bold but very fine and light. Here we find the opposite of ‘bold’, ‘assertive’ and ‘daring’, but ‘discrete’, ‘reserved’, ‘subtle’.

Expansion Typefaces may be condensed or spread out. This is the continuum between maximally narrow and maximally expanded. This can be shown here by the difference between Arial Narrow and Arial rounded MT bold. The metaphoric association of this feature draws on our experience of space. Highly condensed typefaces take up little space. This can have the meaning that they are precise, economical, humble even. Wide typefaces take up space. They spread themselves around confidently, even arrogantly. But depending on the context this meaning might be reversed. Narrow typefaces might seem cramped, restricted or crowded. This might make them seem cowardly. Wide typefaces may also be seen as spacious, expansive, with room to move. Looking at the font on the Rambo film poster in Figure  5.7 we find expansion. This is clearly not lacking in confidence as it expands out taking up space on the page. Certainly, here Rambo is not suggested to be cowardly or withdrawn. The font used for the Cinderella movie poster, as well has having less weight than Rambo, is also slightly less expanded. The word ‘Cinderella’ does not take up the whole breadth of the page. Here perhaps we can say that this quality communicates something more humble in her character. Placing words where there is more extensive space on either side, or above and below, can have a number of effects. It can create a sense of ‘room to breathe’, ‘room to think’. Or it can suggest the luxury of space. A name of an institution may sit alone in the top corner of the cover of a brochure. On a shop front the name of the store may sit in the middle of a strip of colour with lots of space on either side. This can suggest exclusivity, rather than where the name is crammed in, or busy with other information. We can see this in the case of the Sandro storefront in Figure 5.11. This would not, of course, create the same effect were the font we find on the Bargain Booze storefront used instead. Bargain Booze takes up space. There is little sense of something discrete here.

127

The Meaning of Typography

On the Fairtrade coffee package in Figure 0.1 we see a deliberate economy of use of space. While the font used for the product type, naming the source of the beans, is highly expanded, the brand name takes up little space and emphasizes narrowness. Here we can see the humbleness of the brand name, both having less weight and also taking up little space – something important for a brand that suggests it is helping to facilitate, to foreground the rights of the producers, rather than its own identity. We can see that the font used for the brand name on the Bounce packaging in Figure 1.3 is quite expanded. It does not take up the whole of the width of the package, however. And such decisions can be based in the need to create a sense of confidence (here the ‘bounce’ is confident) and also avoids being too expansive. Such health-related products might want to avoid any suggestion of arrogance, since they often seek to carry associations of being in harmony with the environment or as having other more subtle health effects, here listed below the brand name. And simply they may want to suggest substantiality but not being too filling. Of course, in such cases, it is other typographical features along with the design as a whole which communicates these meanings. But we can begin to think here about the part expansion can play. Looking at the children’s camp website in Figure 5.9 we can see that none of the fonts are particularly expanded. None take up space in the fashion of Rambo. In regard to both the printed typeface, for example, used for ‘About us’ and the handwritten one for ‘Summer fun’, both are quite condensed and use space very economically. Both suggest ‘humble’, ‘accessible’, ‘accommodating’, rather than ‘arrogant’, ‘over-bearing’. Would it be offputting for parents were the camp to use Rambo type fonts throughout?

Slope This is the difference between type that is more like handwriting and print-like typefaces. The meaning potential is between writing and printing. In the first case we have the personal, the informal, the handcrafted, the organic. Advertisers will often use such typefaces for ‘traditional’ products such as whisky or bread. Part of the brand message of such products, whatever the reality of their manufacture, is precisely that a human touch and care have gone into their making. Joined-up writing can also be used, as in the case of the camp website, simply to make a product or service less formal. We would not expect the same typeface necessarily for an object that was to be treasured and would endure for years or for a formal service offered by, say, a lawyer.

127

128

128

Introduction to Multimodal Analysis

Figure 5.10  Bargain Booze storefront. Image from ‘Spend less, drink better’ by Matt Walls, Matt Walls Wine Blog, 1 April 2014.

Figure 5.11  Sandro storefront.

129

The Meaning of Typography

Slope can also mean dynamic and full of energy, mainly when used with straight lines. We can imagine the packaging on a toy racing car using such a typeface. We can see this used in the storefront font for Bargain Booze. It is also used for lifestyle/fashion magazines to suggest ‘energy’, ‘going places’. In both cases this brings a suggestion of energy and movement, like the diagonal line shown earlier in this chapter. So here the slight slope can mean energy. The brand can draw on this association to make it seem livelier. In the case of the use of print style, we have the impersonal, the technical, the new, the mass produced. In an advert it might be important to connote the impersonal or technical, if there is scientific information, however absurd, say in the case of hair product that puts vitamins into your hair. We see the difference on the Bounce package in Figure 1.3, where ‘Defence Boost’ is handwritten and informal, whereas the health benefits ingredients below are in print style. This is more formal and claiming to be factual. The following examples imitate advertisement slogans to illustrate the meaning potentials of handwriting and print. Here we use Lucida Sans Typewriter with no slope and Lucida Handwriting. Example 1 Handmade by our experts from home-grown ingredients Handmade by our experts from home-grown ingredients Here we can see that the first line works much better. But if we reverse the effect:

Example 2 Engineered standards

in

German

laboratories

to

precision

Engineered in German laboratories to precision standards In the first example, the personal touch of handwriting adds to the idea of traditional and handmade. In the second case, this same touch does not sit well with the promise of precision and science.

Curvature Typefaces can be highly angular or have a lot of curvature. For example, we can compare FELIX TITLING with Cooper. The first emphasizes angles and straight lines in comparison with the second which emphasizes roundness and curvature. Many typefaces mix straight and curved, of course.

129

130

130

Introduction to Multimodal Analysis

The significance of these may be based on experiential and cultural associations with essentially round or angular objects. Roundness can come to signify ‘smooth’, ‘soft’, ‘gentle’, ‘natural’, ‘organic’, ‘subjective’, ‘emotional’, ‘maternal’. Roundness might also mean fluidity and ease. In contrast, angularity tends to be associated with ‘harsh’, ‘technical’, ‘masculine’, ‘abrasive’, objectivity, rationality and so on. Both may be either positively or negatively valued. If we look at the Bounce healthy snack package in Figure  1.3 we find the brand name uses a font with greater roundness. This, of course, appears round and bouncy. Also we find rounder fonts for the words like ginseng and spirulina. Here we get a sense of reassurance of nurturing and caring and there is also a sense of something more natural. Yet further down on the package we find that ‘HIGH IN ANTIOXIDANT VITAMIN E’ and ‘NO ARTIFICIAL PRESERVATIVES’ use more angular rational fonts to suggest more neutral information or facts. In such cases, of course, it may be less clear what effects ingredients or qualities have. Researchers point to the clean washing nature of such ingredients which can give the impression that a product is ‘good for you’ even though it is very high in sugar and fats (Schult, 2013). It is font here that is in part able to communicate some of the ideas about such exotic ingredients. The Bounce font itself is highly rounded to metaphorically suggest soft bouncy qualities. In this case there are references to soft bouncy balls or balloons. Yet in fact it is the high sugar content, measured in joules, which allows such products to signal up that they are high in energy. In this case the balloon/ball representation of the properties or effects of the product must be seen as part of integrated design. Effects, processes, agents and causalities are now more often represented to us in these kinds of symbolic forms. The properties of the product and what it can do for the body are communicated through references to fun and springy objects like a ball. A sense of caring and nurturing can be communicated through softer curved fonts and then some ‘facts’ through more angular rational fonts. In contrast, on the Fit kitchen package in Figure  5.12 we see a food product packaged as healthy but carrying a different discourse. All of the fonts are angular and avoid curvature. Here, there is less of a sense of food which can nurture but something direct, logical. At the time of writing, there was a phase of clean foods and functional foods. Here food must be stripped back and serve a clear purpose for the body. We can see on the Cinderella poster that roundness has been used to suggest softness and gentleness, as is her character in the film. The Rambo

131

The Meaning of Typography

poster in Figure 5.7 would have worked very differently had the font been realized with much curvature. This would have suggested a more caring Rambo, or someone who was more sensitive and creative even. Instead we find sharper edges and angles on the corners of the ‘A’ and ‘M’, for example. In both cases the use of fonts relates to hegemonic discourses of masculinity and femininity. In the case of Cinderella the character has little agency to change her own fate or that of others and is a passive victim. Only fate and magic allow here to meet the prince and end her abuse. Although she has many friends, in other words, she is socially successful and nurturing. Rambo in a sense is also trapped. Yet he has power over others and agency to shape events but certainly not through social skills or nurturing. What is of interest for us here is how these become coded into designs through fonts. We see this too across product advertising and packaging where products aimed at men may tend to have more angular fonts and those aimed at women more curved fonts. As such these discourses become coded into, and housed by, all sorts of semiotic materials which form parts of routine social practices such as shopping or washing out hair. Gender roles, how we relate to poor people in far-off parts of the world, can be coded into routine everyday objects. On the children’s sports camp web page in Figure 5.9 we find that more playful handwritten fonts and printed fonts tend to be condensed and humble. There is something accessible and safe being communicated here. But if we look at the printed fonts used for headings giving information such as in the case of ‘sign up’ we see angularity rather than curvature. In this case we see how a font can carry different kinds of meanings at the same time. So the ‘sign up’ font is both humble and also more rational and technical, than the handwritten parts. As we shall see, there are other important messages in this font too. But what we are seeing is that the fonts are communicating a sense of fun and safeness, targeted at parents, and at the same time building information giving and ‘truth’ into this design.

Connectivity In typefaces, letters can be connected to each other or can be separated by space. This can mean that they are joined up as in handwriting or that they have features, meaning that they almost touch each other. Or letters can be clearly separate and self-contained. This can mean that they can appear quite far apart and remote from each other.

131

132

132

Introduction to Multimodal Analysis

Extreme disconnection can suggest ‘atomization’ or ‘fragmentation’, while connection can suggest ‘wholeness’ or ‘integration’. But values may be reversed, with disconnection signifying individuality of the elements as a whole and connection its opposite. A movie poster for a mysterious alien or something supernatural may use disconnection to suggest some kind of fragmentation of what is known. A  comedy film about a family may use letters which are closely pressed together to simply suggest their unity. Internally disconnected letterforms, meanwhile, have a sense of not being ‘buttoned up’, which may be negatively valued, as ‘unfinished’ or ‘sloppy’, or positively, as, say, ‘easy-going’. But not being buttoned up can suggest some kind of freedom, room for more open types of thinking, or creativity. The word ‘alien’ or ‘mystery’ on a film poster can use this lack of buttoned up to mean ‘unknown’. We may see fonts which are highly disconnected for a modern art museum, but perhaps not for the sign marking a police station. In the latter case this openness and creativity which might be useful for the museum would be less appropriate. We can see the comparable use of connection and disconnection on the fonts used for the two storefronts. Sandro has clear disconnection. This can suggest open thinking and creativity, pointing to style and the latest chic, or even the luxury of space to suggest exclusivity. Bargain Booze has little disconnection as there is a unity and cohesion in its message. It is cheap alcohol. The sloping fonts also point to the directness of the service offered. In the case of the font used for Cinderella we can see very slight disconnection. This can suggest that she is open, there are possibilities for her. But she is certainly not atomized. A widely spaced out C I N D E R E L L A might suggest a version of the film where she descends into inner contemplation and distress about her lack of agency and whether to take the cynical way out by taking up the offer of patriarchal system and shack up a prince she knows little about. The children’s camp website font has mainly extreme connection. For example, if we look at the font used for ‘Sign up’, these letters are pressed together. This is clearly a sense of unity and buttoned-up. As parents who may send our children to the camp we may not want suggestions of room to breathe or atomization here but safety, order, predictability.

Orientation This is to do with how tall or flat a typeface appears. Letters can be stretched vertically or can be flattened. For example, the Bauhaus 93 appears more horizontal than the vertically oriented Gloucester MT Condensed.

133

The Meaning of Typography

Figure 5.12  Fit Kitchen food package. Fit Kitchen Foods.

The meaning potential of horizontality and verticality could be based on our own experience of up and down, high and low, and gravity and walking upright. Up can mean ‘lofty’, ‘high status’, ‘lightness’, ‘aspiration’, ‘emotionally up’, ‘walking tall’, elegant  – although the slimness of this horizontal orientation could mean instability or even pompousness. In contrast, horizontal orientation could mean not only ‘emotionally down’, ‘heaviness’, ‘stasis’ but also stability or down-to-earth. If we compare the fonts used for Rambo and Cinderella, we can clearly see the flatness of Rambo, used for strength, heaviness, durability. For Cinderella there is lightness and elegance. Perhaps here too aspiration. This sense of elegance and loftiness can be seen on the font used for the ‘Level Ground’ logo on the Fairtrade coffee package in Figure 0.1 appearing aspirational and lofty. But the product origins as in ‘Tanzania’ use more horizontal fonts which are more down-to-earth. In the case of the Bargain Booze storefront font there is little aspirations. Certainly here down-toearth is a key meaning.

Regularity Many typefaces have deliberate irregularities or an apparently random distribution of specific features. Regularity and irregularity have their

133

134

134

Introduction to Multimodal Analysis

metaphorical potential. Irregularity can mean creative or artistic or spontaneous or chaotic, out of control, madness. Regularity, in contrast, can mean conformity, restraint, order. This effect can also be achieved by using a number of font sizes, weights, orientations in different words. If we imagine the following title in a women’s magazine fashion feature: what’s

sexy

check out these hot trends

Now… … Here the change simply in font size conveys the interpersonal attitude of irreverence, playfulness and creativity. We see this irregularity across the children’s sports camp web page in Figure 5.9. Looking at the font used for ‘Total Sport Camp’, used across the design, suggests fun and playfulness. The irregularity on the Bounce font also points to energy rather than to restraint. Such irregularities can be exaggerated to be used to suggest innovation, creativity or madness. For example: my Name is aD aM

Flourishes Typography has developed a wide range of flourishes and additions which also carry meaning potential. One common flourish is the serif which is the flat bit at the top of the letter or the ‘feet’ at the bottom, as in TIMES NEW ROMAN. This tends to be associated with tradition and therefore authority. The sign which advertises a sale in your local clothes store will not contain serifs, whereas the plaque outside your lawyer’s office will. Flourishes may be rounded and expansive and include large loops or circles for the dots on the letter ‘i’. Flourishes may even include other iconographic imagery. This can be seen in the examples of the fonts STENCIL and MATISSE. Typefaces can now be custom-made to include imagery. Stencil could be used on military toys to indicate weapons and ammunition cases. Matisse could be used on packing for painting materials. Or, of course, it could be used to connote artistry on any sort of product.

135

The Meaning of Typography

The absence of any kind of flourish is also important. Simple, clean, minimalistic letters can themselves have metaphorical meaning. This can be through their uncomplicated, uncluttered forms. In the early twentieth century, Bauhaus designers constructed letterforms from basic geometric forms, therefore rejecting historical forms that had tended to serifs and flourishes. They approached letters in terms of their abstract form or their essence. This in itself carries metaphorical potential, as does the sense of cutting-edge design that moves away from tradition and the establishment. We see these clean modern fonts used for the ingredients and qualities on the Bounce packaging. We conclude by showing how the typographic meaning potential is used in the Rambo and Cinderella film posters: Typographic features

Rambo

Cinderella

Weight

Heavy, bold, traditional, immobile

Light, mobility and flexibility

Expansion

Broad, confident, solid

Slim, accommodating

Curvature

Angular, masculine and logic

Curvature, associations to femininity, flexibility emotions

Connectivity

Closer, coherence, buttoned-up

Spacing, openness, possibility

Orientation

Horizontal, stable, down-to-earth

Vertical, aspirational, elegant

Regularity

Regular, serious, even

Variety in height suggests something fun

Flourishes

Appears like stencil print from ammunition crate, looks worn

Elegant swirls

135

136

136

Introduction to Multimodal Analysis

Summary of typographic meaning potential Weight:  bold can mean substantial, stable, daring as opposed to insubstantial and timid. But it can also have negative meanings such as overbearing as opposed to subtle. Expansion: this is the range from narrow to wide. Wide typefaces take up space. This could have good or bad associations. Narrow typefaces could be seen as cramped or unassuming. Slope:  the difference is between writing and print. This has associations with the organic against the mechanical, the informal against the formal, the handcrafted against the mass produced. Curvature: this is the difference between angularity and curvature. Angles are associated with harsh and technical, curves with softness and the organic. Connectivity: letters can touch or be spaced apart. Disconnection can mean fragmented or atomization. Connection can mean intimacy or unity. Orientation:  typefaces may be either oriented towards the horizontal dimension or be flattened. Tall letters can mean not only lightness, loftiness, aspiration but also arrogance. Squat letters can mean not only heaviness or even inertia but also stability. Regularity:  many typefaces have deliberate irregularities or an apparently random distribution of specific features. This can be the difference between formality and order and anarchy, chaos or playfulness. Flourishes:  typography has developed a wide range of flourishes and additions which also carry meaning potential. These may be rounded and expansive, include large loops or circles for the dots on the letter ‘i’. They may even include other iconographic imagery.

Student activity The idea here is simply to look across a range of semiotic materials and make comparisons in order to show how different patterns tend to be found.

137

The Meaning of Typography

1. Go to your nearest high street or shopping area. Take photographs of shop signs. These can be those used for the name of the shop, those found in the window or those within the store. Choose a genre of shop, such as for clothing. You will then do a typographical profile of two so that you can compare. What different meanings do the two communicate? This should be presented in class. 2. Choose two website home pages that deal with a specific topic. This could be food or fitness, for example. Make a comparative typographical profile for the two. Are there similarities or differences? How do these relate to the contents of the two websites? 3. Use Photoshop or InDesign. Create a mock design for the front page of a magazine. Research which kinds of fonts are usually associated with the topic of your choice. What features of type do they tend not to use? Create two headings for the magazine, one with appropriate and one with an inappropriate font for the titles. Explain why this is the case.

137

138

139

6 Textures and materiality In design, much thought now goes into texture. Throughout this book we are interested in how all communication is now increasingly designed not only to be functional but also to have affect in order to engage us on an emotional level. This can be through the use of colour, graphics and images carefully thought through for a target audience. Or it can be through the textures of food packages, the paper used for the documents and leaflets you are given on different occasions. It can be the texture of the tables or floor in a restaurant, the kinds of plates and serviettes that sit on the table. It can be the textures of the materials in interior design, the soft grey concrete and polished steel of a new entrance building to your university. Such textures can not only be touched but also be seen and imagined as tactile experiences. When we walk into a building we need not touch the surfaces to experience them as having textures. We can take the shop interior of Sandro in Figure 5.11 as an example. Here, we can imagine the textures of the floors. We can see that in a Starbucks café the paper serviettes may be a rough texture, whereas the coffee cups are extra glossy and smooth. These help to create our experience of these environments. And in this case, we intuitively know it would not be quite right were the serviettes smooth and glossy and the cups rough and uneven. Textures can therefore be used in designs that we may never actually get to touch. Web pages can be given textures. You can see this in Figure 5.9 for the children’s camp where there is a 3D effect and where papers appear to be curling on the notepad and paper appears to have a slightly rougher surface. Textures can be used in a billboard advertisement for an IKEA kitchen where we can see the textures of the work surfaces and steel cooking equipment. We can also see the textures of the leaves of the lush green plants and herbs which help to bring a more natural, personal and creative feel to the product.

140

140

Introduction to Multimodal Analysis

So today textures are increasingly coded and an important part of different multimodal configurations, as we will show in this chapter. Still, and similarly to the other semiotics resources we look at in this book, there tends to be a lack of descriptive language in everyday use to describe the qualities of textures. We tend to use adjectives such as ‘hard’ and ‘soft’ or ‘rough’ and’ smooth’. And in fact these in themselves are useful analytical tools. But we tend to think about these in terms of affect and in regard to our experience of them rather than being aware of the components that can make up textures and the ways that these can be assembled to make meaning. Why in the case of the Starbucks serviette and cup would it be problematic to reverse the textures of rough and smooth? The answer lies in first breaking down textures into the way that they can be used to communicate ideas, attitudes and as part of integrated designs. And it lies in breaking textures down into its components.

Texture as a semiotic system Much of multimodality has been influenced by Halliday’s (1978) view that language always fulfils three communicative functions (Jewitt et al., 2016), applying these to all forms of visual communication. This is the view that we must understand language and all semiotic systems, like colour or textures in terms of how they communicate ideas, how they communicate attitudes and moods in relation to those ideas and how they must have the ability to create coherence among their parts. We discuss some of the shortcomings of these assumptions in Chapter 1. A form of communication such as texture is very different than language and it would be an error to assume that concepts designed to understand language model can simply be transferred to it (Ledin and Machin, 2018). But nevertheless, we can draw on these as a device to help us to take a first step into thinking about how texture can be used in communication.

How textures communicate ideas If we look at the two images of gyms in Figures 6.1 and 6.2 we can see a high street gym and a CrossFit gym, or ‘box’, as it is called. If we look at the floor, the walls and the equipment, the CrossFit box has a lot of surfaces which appear slightly worn and uneven. The paint on the wall has peeled and some

141

Textures and Materiality

Figure 6.1  High street gym, Nordic Wellness (nordicwellness.se).

Figure 6.2  CrossFit ‘box’ (authors’ photograph).

of the brick has been revealed. The equipment also looks worn, colour is uneven on the metal and plates. This is not the case for the high street gym. Here the machines and walls have even, clean surfaces. Parts of the gym machines are glossy and shiny.

141

142

142

Introduction to Multimodal Analysis

At a simple level, a casual observer may say that this is about the CrossFit box appearing more ‘rough and ready’, ‘gritty’, compared with the relative luxury look, or comfort, of the high street gym. But for the level of analysis we are interested in here, we need to be clear about what it is that is communicating these meanings, these forms of affect. But we can think a little more carefully about how meaning is being created here at the level of texture. The uneven surfaces in the CrossFit box relate to something more natural compared to those which are even. CrossFit is about natural, raw fitness and strength as opposed to the more measured and leisure-oriented training done in a gym with its rows of machines. The regular clean surfaces suggest something more controlled and processed. We can often see this meaning of irregularity on the floors and wooden surfaces in cafes such as Starbucks which can bring meanings of naturalness. A  food package such as the Bounce snack in Figure  1.3 has an irregular form when it is picked up, as does the actual snack itself once the package is opened. This irregularity differs from more standard types of high-fat and sugar chocolate bars such as Mars, which is packaged with more regular surfaces. Here then the irregularity suggests unprocessed, uncontrolled. In Starbucks, as Figure  4.2 makes clear, we find a range of textures communicating ideas about the experience of being in the café. We have the rougher textures of the napkins and the slightly uneven wooden surfaces. Here there is a combination of the natural and organic which can bring a sense of authenticity and warmth. This could be contrasted to a fast food burger café where there were predominantly even polished plastic surfaces. In Starbucks we also find the highly glossy surfaces of the coffee cups. Again such gloss would look out of place in a fast food café. It suggests refinement, attention to detail. We also have the bright polished metal coffee machines and industrial looking matching lighting. Such objects, with their clean, level, uncluttered metal surfaces, or as they sit in space, communicate modernity, forward thinking and certainty. And, of course, they allow the authentic natural surfaces to combine with urban chic.

How textures communicate attitudes If we look at the two kinds of gym in Figures 6.1 and 6.2, we find very different kinds of surfaces. The high street gym has lots of smooth, level and softer surfaces, while the CrossFit box has lots of uneven, worn and harder surfaces. The first relates to a more comfortable and assisted type of workout and the

143

Textures and Materiality

second something raw and direct. We can also say that this communicates types of attitudes about the kinds of environments that have been created and also about the identities that play out there. The CrossFit box with its uneven walls and rough surfaces suggests an attitude of ‘no nonsense’, of no polish or surface gloss. We can see in the image that the people doing CrossFit do not wear top-of-the-range sportswear in coordinated colours. They too must appear functional, relatively ‘unplanned’ and certainly not groomed. CrossFit itself is about raw power, the natural body. It is about exertion and commitment as an attitude. The bare metal and concrete is also about this rawness. In contrast, the high street gym, with its rubber mats and handgrips and plastic-cased machines, does not suggest such a raw, go-getting attitude uncluttered by artifice (although of course it is simply a different kind of artifice). We may find photographs on the wall of people training with a maxim urging us to train hard. But in CrossFit this attitude is built into the textures. In contemporary branding there is a drive to go back in time, to evoke the past, which is what Bauman (2017) calls ‘retropia’. So a snack bar might be labelled ‘raw’ or even have a cave painting on it, with ‘natural’ textures, such as being wrapped in a simple and seemingly worn paper, supporting this message. It pretends not being processed or manufactured, but somehow comes from the wilderness. Stripping off contemporary culture is certainly important for CrossFit. It was launched as being based on the ‘caveman model’, and we can sense this attitude from the rough textures and halfnaked bodies. If we look at the uneven brick wall of the café in Figure 6.4, we get the same sense of going back in time, to a distant past, and certainly we often see art galleries being located in former industrial buildings where traces from the past are deliberately kept and integrated into the overall designs. There is a difference in mood if we compare the glossy flat-surfaced plastic benches and chairs in a fast food restaurant to those in a café such as Starbucks in Figure 4.2. Such textures help to communicate the mood for what takes place in the setting. The plastics are hard and shiny, processed and modern. They do not accommodate to the body. This is about practicality, about just eating and leaving. In Starbucks we may also find harder surfaces such as wood, but these are slightly curved accommodating to the body. And the wood may have slight relief and unevenness, signposting something less processed and modern. There may be sofas with soft, giving, leather surfaces. Here the textures suggest that the environment accommodates to us, and

143

144

144

Introduction to Multimodal Analysis

that there is something personal and organic rather than simply being processed. These textures will be matched with industrial polished steel and highly modern colour schemes, as discussed in Chapter 4. So authenticity, stripping culture off and modernity, being urban chic, are combined. Comparing the two web pages in Figure  1.1 for the plastic surgery clinic and in Figure  5.9 for the children’s camp we can also see textures communicating different moods. The camp website uses a 3D look where we see the pages of the notepad curling outwards. The sticky labels used for the headings curl a little and are given some drop shadow. There is less order here, a sense of less attention to regulation. Were this website for highly official information from a regulating authority, for example, they would be much less likely to use such an effect. This also combines with the slightly rougher or ‘pasty’ textures used for the coloured paper used for the menu. Here the colour suggests more dilute, yet brighter moods, but the textures suggest something slightly natural and unprocessed. The mood here is again less formal. The clinic web page does not have notepads curling outwards or slightly rougher looking paper surfaces. This would not be so advisable for a surgery site. Here the mood must be somewhat formal and organized. And we see this in the layout of the page, use of typeface (see Chapters 5 and 7) and the iconography of the grid. But the use of colour and textures softens the mood. The skin in the model does not appear stark and vulnerable but warm and soft. The bra appears to be a soft cotton, rather than, say, a shiny synthetic fabric.

Textures and composition The use of textures in composition should never be underestimated. We tend to be more familiar with how colours can be used to create coherence, for example, whether they create good coordination or whether they clash. But in contemporary design it is clear that texture is also given much consideration in regard to how they relate to each other, and also how they become part of multimodal configurations and relate to other semiotic resources like colour and iconography. In the case of Starbucks café, the slightly uneven and slightly irregular textures of the natural wooden surfaces suggesting authenticity can coordinate with the slightly rougher weave of the aprons worn by the staff. This can also coordinate with the earthiness which runs through the hybrid

145

Textures and Materiality

colours used across the design in the cafe, the charcoal and the grey-purple plum. And this can also coordinate with the rough textures used in the napkins and the textured look of the photographs of the peasants on the wall. These more natural ‘authentic’ textures can also align well with the clean, stripped back modernist industrial look of bare metal hanging lamps and coffee machines. These too can suggest a kind of authenticity not only of chic Italian style but also of the industrial era. It is stripped back architecture. This is about ‘core’ elements here, rather than dressing, decoration or accessories. It is about the bare iron, brick and wood. These are to give a sense of lack of being contrived (although of course they are just that). This can be combined with the utilitarian look of the metal lights and exposed girders and brick and pipes. In industrial designs, there was of course an original aim that form should meet the function of the building. Here function itself, as is characteristic of integrated design, becomes also created for affect. Here authenticity can come not only from the wood and natural colours but also from associations of authentic honest labour and history (glossing over the possibly gruelling and dangerous working conditions). During the times of the Industrial Revolution it is unlikely people would want their homes to appear so utilitarian. We can see all these integrated features in the design of the café in Figure 6.3. We also find that the plates – here food was served in primitive enamel metal cans – have this historical look. The menu was handwritten on the wall on the glass of an old cabinet door. On the wall hang impressionist paintings from a local artist which can be purchased. Textures, materials and colours here combine to form this chic café experience. Looking at a further example of interior design we see the use of texture in a new university building where one of the authors works. If we look at older university entrance halls from the nineteenth century we tend to find marble and stone. We may also find darker heavy-looking wood, for doors, staircases and panelling. Such surfaces are hard, appear durable, solid, unyielding and speak of the power and authority of the institution. The building seen in Figure  6.5 is very different to this, however, and also a sort of entrance to the university. It is an atrium building and typical of those being erected by institutions at the time of writing. It is a building designed to communicate the buzzwords of the era, such as ‘innovation’, ‘creativity’, ‘flexibility’, ‘transparency’, ‘uniqueness’, ‘braveness’, which emerged in universities as they moved away from a model of education based on civic values, related to humanistic notions of knowledge, to one where they take their place in the market and serve the economy. Here

145

146

146

Introduction to Multimodal Analysis

Figure 6.3  Natural and industrial textures in the interior of the café (authors’ photograph).

ideas taken from the corporate world become implemented in universities. We see this throughout public institutions in society. This means abstract and largely meaningless concepts like ‘innovation’ and ‘creativity’ replace those of traditions of knowledge. Here the shape of the design is important. Externally the building does not appear to have a front. It is multi-level and asymmetrical. It follows the curvature of the road. Inside it forms one open space, but is multi-level as can be seen in Figure 6.5. It combines sharp and ‘surprising’ angles with the curves of furniture. On the whole it forms a large open space, combing different sub-spaces where students can sit and socialize. Here textures play a huge role in binding the different sections together. Bare concrete is a key surface throughout for walls and floors.

147

Textures and Materiality

Figure 6.4  Textures used in integrated design with ‘lived’ surfaces to evoke a past (authors’ photograph).

Along with this we find the bare metal and glass. Emphasis is on empty spaces. We also find much bare wood. This is not of the slightly uneven form we find in the café in Figure 6.3, or worn and torn by time as in the café with brick walls in Figure 6.4. Nor is it highly polished solid-looking wood as found in an older entrance hall. Here we find a soft matt surface, matching that of the concrete. The result is a combination of an installation, the modernity of the stripped back industrial chic. Throughout textures are hard and solid. But unlike the older entrance halls they have slight roughness which is entirely regulated. This has a softening effect. And then throughout in matching colours we find soft seating. We can compare this kind of design to the CrossFit box see in Figure 6.2. We find the same emptiness. We find the same bareness of the metals, the

147

148

148

Introduction to Multimodal Analysis

Figure  6.5  Interior of Atrium building with clean industrial ‘core’ look (authors’ photograph).

lack of clutter. But at another level the textures are entirely different. The flatness suggests something pure, certainty as in modernist paintings of Mondrian. This coordinates with the way that colour is also used to create large empty place planes. In fact many users complain that with its openplan design it is by no means a user-friendly place. Semiotic resources may be used to communicate creativity, certainty, yet accommodating. Yet little, other than socializing, can be done in what is usually a busy, very noisy and chaotic place. Finally, at a simple level, textures can help to provide levels of integration in smaller scale designs such as a food package. In the case of the Bounce package in Figure  1.3 we find a texture which is slightly rough, although entirely consistent and even across this roughness. This communicates both a sense of naturalness and also of processing, cleanliness and order. The package itself also has an uneven shape as compared to the sleek form of a Mars bar package. All these help to give the product meaning. In Chapters 4 and 5 we have discussed how such ‘healthy’ products are in fact not unlike unhealthy regular snacks such as a Mars bar. Both are high in sugar and fat. Looking at the list of ingredients and qualities we find a mixture of things that tell us that this snack is good for us. We find ‘high

149

Textures and Materiality

in antioxidant vitamin E’, ‘defence boost’, ‘energy’, ‘high fibre’, ‘spirulina’, ‘ginseng’, ‘gluten free’. Researchers have noted that such ingredients can have a ‘clean washing’ effect on products that are otherwise not very healthy. And it has also been noted that such healthy ingredients can comprise a rather incoherent hodgepodge of things. Products like this represent a massive growth in products marketed as healthy as public interest in eating well increases. Here texture is one way that whole is given coherence, despite this ‘hodgepodge’ of qualities. When held in the hand it is experienced as more natural and less designed. We see this too in the deliberate irregular texture of the snack itself. Yet at another level such products seek to remind us that they are not some kind of ‘hippy-type’ product. Along with hybrid chic colours, brightness and at panes of colour, these regulated ‘rough’ textures suggest modernity and forward thinking.

The dimensions of texture As with all the list of dimensions and qualities in this book, it is artificial to present these as individually as isolated things. We also experience textures in contexts as part of material wholes such as being qualities of a café, or a food package we pick up. We do not experience texture in isolation, and also in combination with other semiotic resources and in the context of the wider social practices of which they are a part. We cannot really say that a rigid or rough surface means a specific thing. But this is an important step to help to focus our ability to describe and document this aspect of semiotic materials. Here we draw on Djonov and Van Leeuwen (2011), Abousnnouga and Machin (2013), Ledin and Machin (2018) and Aiello and Dickinson (2014).

Rigidity Surfaces may be resistant or they can give more or less to the touch. A surface that gives to the touch can appear more accommodating, or comfortable. The floor of an entrance hall may be marble which is unyielding to the touch. Another may have a cork or rubber floor which gives very slightly. If we look at the two pictures of pram rattles in Figure 6.6 we can see that one is rigid and the other soft and giving. In fact, the rigid rattle is an antique and the change in textures here is related to discourse of childhood. The harder

149

150

150

Introduction to Multimodal Analysis

(a)

(b)

Figure 6.6  (a) and (b) Baby rattles. Images from PicUK and DhGate honesty centre.

one allows the child to explore the different shapes, form and size. However, the newer softer one appears to prioritize softness and cuteness over exploration. In Western societies it has been argued that we have developed highly idealized and romanticized models of childhood related to innocence and protection (Aries, 1962). In former eras, and even presently in societies across the world, children are not understood through such discourses. Here

151

Textures and Materiality

children must play a productive role in the family and are not seen in this sense of protection. A soft rattle here may be useless as an explorative toy and even unpleasant to put in the mouth, as babies do. But clearly here the appearance of cute humanized animals and these soft textures is paramount over functionality. As described above, we can see that the high street gym contains many less resistant surfaces such as soft flooring, seating on the equipment and soft rubber grips for lifting the weights. In the CrossFit case, in contrast, the rigidity and harshness are important for communicating the toughness and lack of compromise of the regime. Food packages too can be more or less rigid. The drink bottle seen in Figure 6.7 is quite rigid compared to, say, a bottle for an organic fruit drink, such as innocent. This kind of sports drink may want to communicate strength, power, commitment. A rounded soft bottle would not serve such purposes.

Relief Parts of surfaces can extend below or above a horizontal plane. This can suggest the difference between something which is natural and uneven, with imperfections, or something which is artificially flat. Relief can signal ‘authenticity’, ‘simplicity’, something worn over time, a lived surface. The flat and smooth surface might, on the one hand, be considered un-authentic, dull and pre-fabricated, but could, on the other hand, come out as pure, practical, efficient and easy to use. In Figures 6.3 and 6.4 we can see the use of relief in the wooden surfaces and the rough brick walls in the two cafes. While on the one hand such interiors have incredibly high levels of precision and integration in design with the careful coordination of colours, textures and materials the relief here suggests something worn, lived in and personal rather than pre-fabricated. This relief can suggest, as in original industrial designs, that the form should meet the function of the building, so we get a utilitarian look. In the case of the GNC bottle in Figure  6.7 we see that the surface of the bottle has relief in the form of a raised angular-type structure. It is common to find raised areas on a range of drinks bottles. This is often found on craft beers. Such raised areas forming the name of the beer or a type of logo can help to communicate handmade and authenticity. But in the case of GNC the raised form is more angular. We consider this more in a later section in this chapter, but in this case the raised area relates more to

151

152

152

Introduction to Multimodal Analysis

Figure 6.7  GNC drinks bottles. General Nutrition Centers, Inc.

creating a surface designed to be engaged with, for an active person. And the almost exaggerated smoothness of the surface also points to processing and technology. Such sports drinks market themselves by dressing up sugary water in scientific terminology. The packaging texture here also plays a role in this.

Regularity Regular textures are predictable and can mean homogenous, lack of surprise and consistency. Or they can be irregular and mean different, playful, creative or inconsistent. Irregularity can also suggest something handmade as opposed to the regular being manufactured and standardized. We would not find irregularity on medical products where regular surfaces communicate predictability, standardization and science. We can see that the children’s camp web page has irregularity where some of the pages of the notepad peel upwards and the stick-it notes used for the click-menu have raised corners and drop shadow. Along with the use of colour and fonts this is one way that the website communicates fun and playfulness, rather than order. The Bounce package seen in Figure 1.3 is irregular as is the snack itself. While the package is clearly highly manufactured and much development will have gone into its look, the irregularity here suggests something more

153

Textures and Materiality

handmade, something less standardized, with less predictability. Such packages also often come along with call-outs which emphasize the moral calling of the small-company manufacturer, also to shift away from the sense of this being a standardized product made by anonymous industrial processes. We see deliberate lack of relief in the polished metal of Starbucks and the café in Figures  4.2 and 6.3. Here the meaning relates precisely to the functional utilitarianism of the industrial look. It is stripped back and without dressing. In this sense, it integrates well with the other raw and stripped back surfaces such as the rough wooden floors. Of course the food on offer in such cafés may also have claims to being stripped back of processing and standardization in the form of organics, Fairtrade, clean eating.

Naturalness Textures can also communicate naturalness or artificiality. This is partly because materials may have their origin in nature or be manufactured and artificial. Naturalness can be associated to what is authentic, organic and environment friendly. Textures which are less naturalistic can suggest technological progress, high competence, the predictable and also the impersonal and artificial. Of course, we have already seen these meanings in the cafes in Figures 6.3 and 6.4. We have the extensive use of wooden surfaces to suggest naturalness and lack of processing. Alongside these sit the uses of bare metal lights and pipes. Here is the technological but in a stripped back sense, foregrounding functionality and utilitarianism and simplicity. Paper can be textured to foreground its naturalness or made glossy to shift away from this. We see this on food packaging or in a management document where the pages are slightly glossy, meaning somehow more important, more official, or luxury and high status. Some snacks which seek to promote themselves as healthy or natural use layers in the packaging so that the inner is plastic or foil (which itself can connote freshness) and the outer is rough paper. But while Bounce uses slight relief and irregularity to communicate naturalness and something unprocessed, the design here, with its hybrid colours and modern fonts, also points to something contemporary, forward-thinking and chic. And it has been important for manufacturers of products which claim naturalness or healthiness to consider that some market segments will be put off by

153

154

154

Introduction to Multimodal Analysis

products which appear too earthy. Fairtrade products have had to work hard to disassociate themselves from hippy type associations. We can see that the Fit Kitchen package in Figure  5.12 uses a more natural textured paper to indicate its unprocessed nature. Here, in fact, this combines with other indicators that there is also something technical here with the angular font and the lines which provide information on content. But these are fine and minimal and again point to this idea of simplicity, lack of gloss and things being stripped back to the core.

Viscosity Viscosity is about how sticky different surfaces are. This can have positive and negative meanings, depending on context. Sticky can mean dirty and unwashed – not very useful for food packaging. But it can also be used to communicate comfort and grip achieved through rubber-like or foam textures. A type of canned energy drink may use viscosity, in the form of polystyrene layer, to communicate its practical use during fitness. Literally it suggests ‘grip’. This may serve a similar purpose to the uneven surface of the GNC drinks bottle in Figure 6.7. Given the way the drink is used, such an addition may be of little practical help but it is one way that ‘active’ can be loaded onto the meaning of the drink – one way that the drink can be written in to a particular social practice. We would be surprised to find such a grip on a fruit juice bottle branded as a ‘simple, organic food’. Here other textures will be used to help us to understand which social practices are relevant in this instance. A branded detergent may have a small viscous section to help grip, or on a trigger section where spray action is involved, although this may play more of a role of suggesting utility than actually being of much practical use. One of the authors recently did a fitness class, with a dance aerobic component, attended mainly by women. We used kettle bells that had a sticky viscose cover over most of the surface. The other author has trained at a competitive level with kettle bells which were raw bare iron, with rougher ‘unfinished’ edges from the casting unremoved. While we could say that to some extent the viscose rubber here brings a sense of softness and comfort the ‘grip’ aspect can also add an element of ‘being active’. In the hall where we did the class there were photographs of attractive couples working out with mantras about go-getting and no-compromise. It is clear in this case how such contemporary fitness regimes, as we saw with CrossFit above, become coded with meanings about self-betterment and self-management. But the

155

Textures and Materiality

Figure 6.8  Fairview cheese package. Image from ‘Fairview Ripe and Ready Cheese’ by Jane Says Design, Behance, 23 October 2013.

textures help to bring slightly different accents to the communication of these ideas and attitudes.

Liquidity Surfaces may be more or less wet or dry. Wetness can relate to life and vitality or to rot and decay. Liquid is a prerequisite for humans and living organisms. Surfaces which are wet can mean purity and freshness. Dryness might connote ageing, a lack of vitality. But dryness can also signal comfort, cleanliness and order, as when we do the laundry or wash the car and then dry it to get a smooth and perfect surface. The dryness of breakfast cereal packaging can also be important to communicate crispness and freshness. The packaging for washing products may have a surface that appears to have liquidity connecting it to the shampoo within. In Figure 6.7 we see that the GNC bottle has been designed in a way to emphasize liquidity. Much investment has gone into developing plastics and bottles which have this effect. For a drink such as GNC, this effect helps to communicate vitality as well as the order and certainty of the science which

155

156

156

Introduction to Multimodal Analysis

gives the drink its properties. The same effect can be found in brands of organic fruit drinks and smoothies. Liquidity can also be found in the inner surface of the cheese packaging in Figure 6.8. The glossiness of the surface can suggest ‘life’ and freshness, while the outside may be dry and slightly lacking in vitality to suggest ageing. We can see this meaning of dryness used to suggest a different kind of freshness in the paper used for crackers. Here the type of paper can communicate lack of dampness. The wooden packages that contain cheeses may have almost powdery surfaces to suggest ageing. But a wooden tub used to contain an organic shea butter face cream, while having unevenness will be given a highly glossy, liquid surface to suggest the revitalizing and protecting qualities of the product.

Summary of textures Resistance: surfaces may be resistant or they can give more to the touch. This can be related to meanings such as stable, reliable, tough and so on vs. accommodating, comfortable, interactive and so on. Relief: parts of surfaces can extend above a horizontal plane and thus have relief. This can carry meanings such as being natural and having authenticity but of being imperfect. A flat surface can evoke meanings of being un-authentic and dull, and also signal something practical and efficient. Regularity:  this is the difference between being predictable or not. Regular textures can mean homogenous and consistent, whereas irregular textures can mean not only playful, creative, handmade but also inconsistent. Naturalness:  here the coding distinction is between naturalness and artificiality. The former can bring associations to what is authentic, organic or environment friendly, whereas the latter can suggest, for example, technological process or the impersonal. Viscosity:  this has to do with how sticky surfaces carry both positive and negative meanings. Sticky not only can mean dirty and unwashed but also can communicate comfort and grip. Nonsticky might communicate comfort as well, but then because it is clean and smooth.

157

Textures and Materiality

Liquidity: this distinction goes between wet and dry. Wetness can bring associations to life and vitality or to rot and decay, whereas dryness might not only connote a lack of vitality but also signal comfort and cleanliness.

Materials In this next section we look at the way that materials, that is, the substances deployed as we make things, can also have certain meaning potentials. We do not present this as an exhaustive inventory, but it is important to include as the meaning of textures is highly related to these. And, again, we are less familiar with thinking about how materials can come with not only functional properties but also potentials to create meanings. Ideally, we would need to make an inventory of material for any specific kind of material object, since meaning as with all semiotic resources can be very localized. But here we want to list some of the common materials deployed for their meaning potential. Materials can communicate ideas. For example, the wooden surfaces used in one café communicate different ideas than the plastic surfaces in another. And marble again would suggest something else, as would metal. And in the last case this would be different were it bronze or steel. Such materials could all work functionally in that they are sufficiently hard and durable. Yet each suggests something different. I would be confused if I took my son to preschool and find a marble floor. In domestic kitchens we can now buy steel-fronted cabinets which go along with open-plan metal-framed storage units. These look like the professional kitchens we have seen on television. But this look is also about utility and functionality and the stripped back industrial look. Kitchens around thirty years ago tended to be fitted and dominated by plastics and wood-chip with plastic covers. At this time kitchens were first commodified and moved away from being primarily functional spaces. People would buy the entire kitchen which would take up every centimetre of the space; thirty years later steel is used to communicate a new kind of functionality. Materials can also communicate attitudes. It may be less appropriate to make a war monument from plastic. Plastic has the association of cheapness, informality and industrial manufacture. An expensive Highland whisky could not be sold in a plastic bottle. If a government built a war monument

157

158

158

Introduction to Multimodal Analysis

commemorating the deaths of hundreds of thousands of young men who fought for the ‘good of the nation’ to take control of oil fields somewhere at the other side of the world that was plastic, this would seem odd, even if that plastic would last for centuries. We need stone that brings with it connotations of ancient mountains, or bronze which is associated with ancient history and hand-made processes. These materials bring a different attitude to the process of commemoration. To the same end, Starbucks and the café in Figures 4.2 and 6.3 would want neither plastic chairs nor a large bronze-clad table. Although in both cases, of course, this could depend on the kinds of surfaces the objects were given. Materials can also be used in integrated designs. We have discussed this already in the case of woods and metals in the case of the cafes. We saw this in the design of the new university building in Figure 6.5. Here there is no room for plastics, granite or bronze. We have the bare flat empty concrete surfaces connoting modernity and the stripped back industrial look of the bare metal and bare pine. Below we list some of the meaning potentials for materials. Clearly materials serve functions. Paper or plastic may not function so well for a bridge as concrete or steel. But here we want to draw attention to some of the ways that materials can be deployed for the meanings that they can carry.

Wood As we have seen in many of the examples used throughout this chapter in reference to textures, wood has associations of naturalness, the personal, the unprocessed. In food packaging it can be used for cheeses, to create a box for an exclusive box of whisky. Here wood can suggest something artisan. It can be used for cosmetics, although here a polished surface relates more to naturalness than to artisan. In each case, textures and types of wood will be important. Wood such as dark oak can be used to communicate tradition. A new university may have one council room furnished with darker wood panels. A top-of-the-range car may have wooden panels in the dashboard or doors to suggest a kind of luxury hinting at the same formal kinds of meaning. The new university atrium building in Figure 6.5 contains much wood, but here we find lighter woods like pine with smooth softer surfaces and the meaning here is not related so much to tradition, but to bring warmth and naturalness to the stripped back core and the ‘innovation’ communicate by the angles and interlocking levels and spaces. Here there is no fuss nor frills but the

159

Textures and Materiality

open spaces to think and create. Wood may be used on a children’s playground in order to soften the look of plastic and polished steel play equipment.

Glass In packaging glass is often deployed due to its associations with pre-plastic and pre-mass production packaging. Its meaning can relate to these associations as something older or traditional. Craft beers may tend to come in bottles to help bring a sense of being handmade. A fruit smoothie drink may use a glass bottle for these meanings, although newer types of glass are used to create brightness and a subtle sparkle in the liquid contents. We may find that some products that formerly used glass jars such as honey switch to squat plastic tubs to help to modernize them and make them suitable for an urban-chic market. Glass can also give a product a sense of something created to endure longer and can communicate substantiality through weight. And since it is transparent, glass can be used to allow us to see the contents, which can be another way to communicate honesty. The same glass can be used in buildings to communicate transparency. Where one of the authors once lived in Wales, the old government building with red brick and small, high windows was replaced by one with walls comprised almost entirely of glass. This can help also to suggest that the boundaries between the government and the people have been reduced. Of course, at the very same time the process of governance may become more obscure and distant from the public. Increasingly in public buildings such as universities or in high street banks we see the use of glass for partitions. This can relate again to transparency, at least the display of it. But it also relates to the shifting of professional roles. Formerly a professor would sit in an enclosed office to carry out periods of longer term, slow, study. Now their work must be oriented more to outputs and meeting institutional targets. And their job has become much more integrated into administrative processes. The glass here can be a part of how such discourses are realized in materials. In open-plan buildings such as the university atrium in Figure 6.5 glass can also help to create a sense of space, brightness and lack of shadows and clutter. Here the light, airiness and transparency create clear space for creativity and innovation. Again, these ideas and values may be communicated in design at the very same time that universities become compelled to deliver outputs and market-ready students so that there is little room for actual progress in ideas.

159

160

160

Introduction to Multimodal Analysis

Concrete At a functional level this material can be cheap and functional and much of its meaning potential comes from this. In architecture concrete is associated with a ‘brutalist’ style, particularly in the post-Second World War period, exemplified in some of the designs of Corbusier. It was used in Europe for the construction of cheap public housing and around the world in public and corporate buildings. See photographs of the City Hall in Boston, or the library at the University of Chicago. Often these buildings reveal how people are to flow around the building or expose the elements which comprise the structure, such as for accommodation, staircases, walkways. Such buildings foregrounded function rather than prettiness. They were honest and unpretentious. They also have associations of socialist utopianism and high ideas about social equality and at the time communicated ideas about progress and forward thinking. In the designs of the cafes in Figures 4.2 and 6.2 we do not find concrete. In these cases the stripped back industrial look with wood, bare brick and metal appears more related to the nineteenth century, with authentic labour and utility. In the University Atrium building in Figure  6.5 we do find concrete which recalls this forward thinking, utopian brutalism. We still find the stripped back industrial metal look, but modernism and progress play a greater role here.

Brick Brick comes in many colours and shapes. And its meaning can very much depend on the kind of building in which it is used. But one clear association of its more deliberate deployment in building designs is the nineteenthcentury industrial look. And as we saw in Figure  6.4 this raw, reclaimed brick look can be used specifically to suggest something stripped back, ‘core’ or authentic. A YouTube video series where people tell painful stories about how society makes them feel bad about their bodies may use a brick wall as a setting. We might use the same brick wall for two sisters who are promoting their organic snack bars. Such walls are clearly marked as reclaimed and are clean and ‘rough’. Brick walls that are less clean and reclaimed can be used to connote ‘crime’ and ‘poverty’. A news reader presenting a peak viewing bulletin may not want to sit in front of a reclaimed bare brick wall. This may rather suit a more casual breakfast time broadcast which gives a sense of being presented from an urban cafe. Neither would want to be shown reading in front of the dirty urban brick wall.

161

Textures and Materiality

Paper The meaning potential of paper depends very much on the textures that we looked at in the previous section. The paper used for the menu in an organic restaurant may be grainy, rough and uneven. Your graduation certificate may also feel rougher and be slightly grainy as in a sheet of parchment, to suggest something important. But it also might be very regular and also carry some glossy elements. Paper is also used carefully in food packaging. Paper can give an impression of more fragility and tends to be associated with something more handmade or traditional, where formerly goods were hand-wrapped in paper. We may find bread in paper and find it more traditional than the loaves packaged entirely in a plastic bag. A bar of soap in an organic shop may be wrapped in a stiffer form of paper. The Bounce snack in Figure 1.3, despite its highly modern chic design, uses paper with a slightly rough texture.

Carton/card Carton is often used for utility. Parcels come in carton as it is light and protective. Yet it can also be used to suggest added quality. If you are handed a document it will feel different if it has a stiff card cover. A menu might use a stiffer more substantial card to suggest something more substantial compared to a takeaway restaurant which used deliberately thin glossy paper. In food packages, carton can be used to suggest higher quality. Some deep freeze foods, such as pizzas, come in clear plastic-sealed covers, whereas others are also placed within a box. Carton can also have different thicknesses, which can also signal quality, through weight, durability and solidity. Carton can be also used to communicate ‘no frills’ in a specific way when it is used in a form which looks bare, as is the case of the brown, rough packaging in IKEA stores.

Plastics Here we find a wide variety of forms and the meanings they help to communicate will depend upon textures and shapes. And one of its advantages, of course, is that it can be moulded into different shapes. As we saw in Figure 6.7 the plastic GNC sports drink looks tall and carries an angular design to suggest technical properties and also aspiration. The plastic here is also tougher and thicker to suggest something more substantial.

161

162

162

Introduction to Multimodal Analysis

Plastic can communicate modernity and processing and can suggest cheapness, such as a beer in a plastic bottle. An organic chicken may require an added band of card or paper to suggest honesty and quality. Glossy plastic may be useful for a saver range, but given texture for something slightly more expensive. But even very cheap clear plastic can be used to communicate expense and quality. For example, Apple hardware sells each individual item, the cable, a connector, all wrapped in their own tiny plastic bag. In the university offices where we sit there is very little plastic. Desks and shelves are lined with lighter steel supports. The exception is the flooring which is a light flecked plastic. Functionally this is cheaper and durable. We could imagine the difference, however, were the floor to be uneven dark wood as in Starbucks, or concrete that you might find in a loft space.

Metal Of course, there are a huge range of metals and their meaning also depends on texture. But as a whole, some metals like iron can mean older, tradition, rustic and hand forged. Bronze too can have associations of history but relates more to ancient history and grandeur due to its cost. Bronze would be suitable for a commemorative plaque or for a war monument. Here there is association of timelessness and hand forging. Iron would be less appropriate and would bring more associations of nineteenth-century industry. Other metals like steel, aluminium represent modernity. You could not really have an aluminium statue of a soldier. It would appear somehow futuristic. It would perhaps serve to commemorate an accident involving astronauts. The metals we find in the cafes in Figures 6.3 and 6.4 and the university atrium building in Figure 6.5 tend to suggest both stripped back industrial utilitarianism and use brighter clean polished surfaces which can also foreground modernity and the future. The shelf supports in our offices are bright and polished with a matt effect. We can imagine the difference were these darker-looking functional industrial steel. The brightness here is more suited for the purposes of optimism and forward-looking attitude of an educational setting. In regard to food packaging, metal, like plastic, can be used to make lightweight and thin yet highly resistant containers. But metal can also bring a sense of quality, given it being durable. A metal lid on a bottle appears more high status than a plastic one. And the metal can be covered with lacquers or paint. Some coffees such as Lavazza produce options in metal cans with a re-sealable lid. Foils can also be used to suggest quality.

163

Textures and Materiality

Stone There are many different kinds of stone, with a massive range of textures. But a few of them have come to have more regularly used associations. Slate can mean ancient in a ‘people and landscape’ type way. It may be common to use slate in buildings which seek to communicate some kind of connection to the local land or heart of the people, such as a new government building in a newly independent region. Here slate can suggest a link to the ancient landscape in a more ‘local culture’ and ‘of the people way’ since slate, rather than other stones such as granite, marble or limestone, would have been used traditionally in small-scale building. It was also used in nineteenth-century education for writing with chalk and for the construction of blackboards. This brings some associations of hands-on learning and old industrial period learning. Slate structures may combine with large amounts of glass and exciting angles in the construction to also communicate openness and forward thinking as opposed to older dogma. Marble is associated in particular with classical architecture, with the Roman Empire or ancient Greece. It is used to communicate higher values of truth, knowledge and civilization. It is used along with symmetrical designs with straight hard edges and grand pillars. It can be used for elements in buildings where we see the patterns in the grains or the fossils it contains clearly used as part of the look. It can be given a matt look which harks to the weathered dry look of ancient classical sculpture as it appears today, such as the Washington monument, or highly polished so that it reflects the light which can be associated with exclusivity – as a material it has always been expensive to extract and manage. Marble is used in court houses or for staircases and floors in exclusive hotels. Marble is now used domestically, for example, for kitchen surfaces, but is often combined here with more stripped back minimalist looks. Marble has been used in some countries in commemorative monuments, given its potential for signifying high ideals, prestige and empire. Granite comes in many colours and can be smooth and polished or left as rough and grainy. Granite is mainly known for its associations of hardness and durability. In the United States it is highly used for counter tops. It can therefore connote solidity, reliability. It does not create the same sense of high ideals as marble which suggests also something softer and giving. Granite is often used for stone tiling and can be found in contemporary bathrooms where black tiles can cover walls and floors. It can also be used in contemporary designs. It can bring a kind of simplicity, feel of stone

163

164

164

Introduction to Multimodal Analysis

masonry and craft. But it also relates to the uses of brick and wood and natural materials.

Summary of materials Wood:  here associations to naturalness are often evoked, to the personal and unprocessed. Glass:  this material can, not least in packaging, relate to associations with something older and traditional, that is, premass production. Glass can also create a sense of space or interaction since it is transparent. Concrete:  here associations to mass production are part of the meaning potential that can encompass both what is cheap and dull and what is modernism and progress. Brick:  the meanings here are dependent on the construction or building in which it is used. Since this material has been part of building designs since long, it suggests not only authenticity but also something rough, even poverty. Paper: the meanings will largely depend on texture, where a grainy paper with relief can suggest naturalness, whereas a regular and bleached paper rather gives associations that is efficiency and uniformity. Carton/card:  this material is often used for utility and can give associations to what is practical and simple. It comes with different texture designs, so it can also signal something substantial or of higher quality. Plastics: here we find a wide variety of forms and meanings that will be dependent on its shape and the multimodal configuration. Associations can go to what is technical and durable, to modernity or to what is cheap. Metal:  once again there is a huge range of this material with different textures. Possible meanings include older and rustic, or modernity and progress, or durability and high quality. Stone:  here there are some regularly used associations which include something ancient and local, bound to a place and its history and people.

165

Textures and Materiality

Student activity Materials and textures are really things to which we rarely pay attention in terms of design. As with other suggested activities across the chapters in this book one of the main aims here is to get you more familiar with noticing and describing these. It is from this very important first step that we can then engage in the next stage of interpretation. So even if at first the meaning of these semiotic choices is not so clear, it is crucial that you are patient and carry out this level of observation. 1. On your campus there may be different kinds of buildings, old or new, those designed for specific purpose, such as a sports hall/gym, library, computer centre, security building, restaurant. Go into these places and make notes on textures and materials. Choose two different buildings and document and describe the textures and material that you find, in furniture, flooring, fittings and walls. Are there any differences? Some of these may relate to highly functional matters. But, of course, materials and textures can also tell us ideas about the meaning of what takes place in a setting. 2. In the university café or restaurant, check out the textures and material used for the packaging of snacks and drinks bottles. Take photographs of these. What kinds of textures do you find? How are these used to add meanings to the different products? 3. Do some research on web pages. Often textures are implied in designs, such as glossy, slightly rough or uneven. Hunt around and find as many examples of this as you can. Use the check list of textures in the chapter. Explain in each case the meanings that the texture brings to the product, service or ideas carried in the website.

165

166

167

7 Composition and page layout In most of the chapters in this book, we are interested in the contents or qualities of designs. So the chapters offer tools for things like colour, typography and images, and point to how such semiotic resources are combined in typical configurations, for example, colour used together with typography and writing, and deployed in different contexts. In this chapter we are interested in how different qualities and features are used together in overall designs. How are they placed in relation to each other to form a whole? How can the way that they are organized also be important in regard to how meaning is created? A  composition has an overall wholeness within which the parts are related to each other in different ways: elements can be foregrounded and backgrounded, laid out as conveying an ideal situation or a realistic one and so on. So analysing composition means finding out the relations between elements and points to how the interrelatedness between the elements makes meaning. If we look at Figure 7.1 we see a poster from the United Kingdom which seeks to motivate people to vote against the EU. This is interesting as it dates from a few years prior to the Brexit Referendum and can be seen as part of how people from particular social class contexts were mobilized to reject the EU. The poster has been designed to look ‘undesigned’. But it is clearly very well thought through. At the bottom we see one element of this composition, a happy working-class family (easily recognizable as such to the target group) with the man clearly positioned as the core. To the top right we see another element, an aircraft, a Spitfire, which has been a symbol of British resilience since the Second World War. And to the right we are told this is the New ‘Battle for Britain’, which is a written element. In between we find different sections of text and the BNP (British National Party – a farright party) logo, done in the colours of the British Union Flag.

168

168

Introduction to Multimodal Analysis

Figure 7.1  BNP poster. Image from ‘51 Powerful Propaganda Posters and the People Behind’ by Claire Stokoe, Smashing Magazine, 13 June 2010.

While it appears somewhat cheap and primitive, thought has clearly gone into composition. Why, for example, has the designer not put the aircraft cut-out overlapping or next to the family? What would the effect be in such

169

Composition and Page Layout

a case? And what if the aircraft was not placed in its own circular border, but a cut-out flying over the family? And why is it that the ‘BNP’ is allowed to overlap the family  – note that their clothes are red to coordinate with the logo. In this design, of course, it is the BNP logo that is the most important aspect of the design, placed close to and overlapping the family, although on another level the family seem be the most salient feature as they take up slightly more space on the design. Had the aircraft been placed in this position it would have appeared somehow as if the family were to go to war or be under attack. But this effect is avoided by having the aircraft placed further away and isolated by both the blue border which carries the text about the battle and the circle in which it sits. It is also important that the list of BNP beliefs is in a different section than the ‘Battle for Britain’ section as this helps also to show that this text is of a slightly different nature. Clearly in the case of this BNP poster we find some typical positionings on the page. Semiotic materials have well-trodden forms due to the way they have evolved to do specific kinds of things. Such patterns relate to the ‘canon of use’ we discussed in Chapter 1. Designs have an outer level which is what they are as a semiotic material. This could be a food package, a monument or a web page. They have an inner level which is formed by the semiotic resources tailoring them to a specific instance of communication. This may be a food package for breakfast cereal targeted at women who want to eat organic food. But the packages, like the form of a newspaper or the use of photograph, comes in typical forms. And this means that they will tend to carry typical kinds of design features and design patterns. Food packaging, a website or a monument may seek to be ‘innovative’, but they tend to nevertheless follow canons of use. Composition then is the typical forms of configurations that we tend to find in regard to how semiotic resources such as typeface, colour and positioning on a design set up relationships between elements. In this sense the poster follows certain patterns for such semiotic materials with the image at the bottom and a kind of call-out at the top with the clarifications in the middle. And analysing all designs should be done in the context of the typical canon of use. What we are interested in this chapter are some more general principles of composition that are common across different kinds of semiotic materials. What is taking place on this design is that different parts of the composition are connected or distanced from each other and are given different levels of salience. And the images have been placed and deployed in very specific ways as regards the other content. In

169

170

170

Introduction to Multimodal Analysis

this chapter we present a toolkit for analysing the way that these aspects of design can be used to create meaning: ●●

●●

●●

Salience. This is how certain elements might be made to stand out, to have the viewer’s attention drawn to them. Framing. The use of framing devices connects, relates, groups or separates elements in the design. Images on the page. The use of images together on a page or in relation to text.

Some basic principles of salience Salience is about where features in compositions can be given ranking in relation to others. This may be a way which features will be given the highest symbolic value in the composition. These are the ones that credit more attention than others. Salience can be achieved through size, colour, foregrounding, overlap, repetition and so on. But we should not assume that it should be a single salient feature since they can be given salience in different ways and to different degrees. Below we account for principles of salience.

Potent cultural symbols Potent cultural symbols can have salience on a design. In Figure  7.1, the aircraft has particular cultural significance in Britain, being a symbol for the defeat of Nazi Germany and what is represented as a struggle against the odds. Like the British Union Flag, this can be colonized by the far-right as part of a discourse of white Britishness and anti-internationalism and racism. The family in Figure 7.1 are also a potent cultural symbol here and relate to conservative ideas about family structure. The white, heterosexual family here, shown huddled around the father, represents an ideal of ‘British people’. And, of course, here the flag is a potent cultural symbol. For some people flags are about national pride, national customs and so on. When flags are waved at sporting events they serve as part of discourses which harness national mythologies relating to things like victories in war. For other people, such as the present authors, flags signify discourses of parochialism and of imagined differences between people which are usually harnessed for more right-wing political purposes.

171

Composition and Page Layout

In Figure  7.4 we can say the woman on the hair product package is a cultural symbol of beauty. The laptop seen in Figure 7.7 is also an important cultural symbol. Here we see a particularly lightweight laptop placed in a slightly untidy pile of work. Such a laptop suggests independence, mobility, rather than the dull and abstract calculation and bureaucracy which in fact comprises the processes of auditing and goal steering in institutions. In Figure  4.1, promoting sustainability, we see at the bottom left, the delicate leaves of the fern are used as a cultural symbol for the fragility and beauty of nature. In fact the whole issue of ‘sustainability’ here is highly complex and abstract and relates to EU legislation. It can include a broad range of things such as sustainable energy, protecting the climate and also sustainable knowledge and social equality and gender issues. Such policies become things that institutions must demonstrate that they are working with and which teachers must build into classes. Yet here the vagueness and abstraction are captured by the single cultural symbol.

Size This can simply be the element ranging from the biggest in the composition to the smallest. In Figure 7.1 we see that the family are the most salient single element in regard to size, followed by the BNP logo. So here as regards size the BNP foregrounds the family. This issue is at core about the British family. Clearly in this case it would have changed the meaning had the aircraft been the most salient in terms of size. If we look at the Fairtrade coffee packages in Figure 0.1 we see that the image of the worker is given the greatest salience as regards size. The place where the product is sourced is also given much salience. Yet the logo is much smaller. This is not the case with all packaging. But in the case of Fairtrade products it may be better for the brand to show that this is ‘about’ the workers. And the location should be salient to foreground the distant and exotic settings. Part of the Fairtrade discourse involves the non-modern and remoteness from corporate production. If we compare the two film posters in Figures  5.7 and 5.8 we see that Rambo himself fills almost the whole of the space. The character Rambo acting alone, with his muscles and large gun, is what this is chiefly about rather than plot. In other actions films, we may see a group standing in close proximity to each other to suggest a team. We see something slightly different in the Cinderella poster. Here while the name Cinderella tells us

171

172

172

Introduction to Multimodal Analysis

Figure 7.2  L’Oréal advertisement (https://www.maleskin.co.uk/loreal-men-expert).

the film is about her, she does not fill the scene. Such an image would have suggested that the story is more about her internal states perhaps. Rather in the film, Cinderella is a more generic attractive young woman saved from poverty and oppression by quickly marrying a rich man. It is common to find that products take up much less space in the composition than other elements. An advert in a magazine may contain a large picture of a model smiling at the viewer and a smaller picture of the product below. We can see this in the L’Oréal advertisement in Figure 7.2. In older advertisements, we would most likely find a product shown much larger with a smaller illustration. What is emphasized in the newer advert is not so much the use value of the product but the lifestyle association, connoted through the photograph of the model or other potent cultural symbol. Often in promotional literature for banks the lifestyle photograph, say of a handsome professional woman staring confidently out at the viewer, a typical potent cultural symbol, will be bigger than the information given for the account itself, at least on the cover of a pamphlet. Central to branding is the process of loading products with values that will make them meaningful to particular niche market groups. A  bank’s financial product may be much like that of a competitor. What is important is that it signifies core values that speak to the target group. So when you see a promotion for a particular bank account or a particular insurance policy, you will see the photograph, given salience through size, first, since it is the core values of the product that the promotion wishes you to see before anything else (Figure 7.3).

173

Composition and Page Layout

(a)

(b)

Figure  7.3  (a) Organic Protein™ Plant Based Protein Powder. Orgain, Inc. (b) SYNTHA-6® ISOLATE protein powder. BSN Supplements.

Colour Colour can be used to show relations between elements, for example, to create a more level feeling where parts are of equal salience, where colours are of similar levels of brightness, saturation and so on. If we look at the use of colour on the sustainability scorecard in Figure 4.1 here colour has been used in a sense to avoid salience. This helps to create a sense that the different parts, which are in fact of a highly different nature – the recycling, the counting of species  – are of the same nature. Some darker or more saturated colours are used to raise some of the titles, but again these are at the same level across the document. In the analysis of designs, it is just as important to point to what elements are the same as it is to show those who are standing out and have salience. We see this also on the university goals web page in Figure  7.7. Here are a number of ‘goals’ for increasing quality. All public institutions must now have visions and goals and strategies. And these must often include highly abstract buzzwords that come from high political levels and the EU. Here we see, for example, sustainability and internationalization. In fact a

173

174

174

Introduction to Multimodal Analysis

little investigation reveals how abstract and vague these are. Sustainability emphasizes the climate and suggests a careful use of resources. This notion does not go hand in hand with the need for internationalization and global travel, where for example academics take flights for conferences around the world.But here these are presented as easily sitting alongside each other. We have more to say about these shortly. But here we see that the green colour found on the plants for sustainability is also found on the chart above where ‘quality improvement’ is being measured. The blue used for the land mass in internationalization is the same as the blue above on the student’s sweater. So here colour is used to level and gloss over differences. An obvious use of colour to create salience is the bright saturated orange on the L’Oréal advertisement in Figure 7.2. Here the actor Gerard Butler is given the greatest salience through size. But colour, used on the product and the colour box, creates salience on a different level. So on the one level the identity of the model is the most salient thing in such designs – this may be a famous person or an attractive model – but on another it is the product that stands out. On the Fairtrade coffee packages in Figure 0.1 we also see these different levels of salience. While the image of the worker is given salience due to size it is the logo and the source of the product which are given salience through the use of the white colour. The design gives more space to the workers, placing itself modestly by taking up less space. But at another level it takes prominence. On the Rambo poster in Figure  5.7 we find something different. Of course, the model himself has cultural salience, not only due to his extreme physical form and the massive gun, but as Rambo as a character himself has become iconic as a particular notion of masculinity. But here too colour plays a role in this design at two levels. The bright yellow-white explosions are clearly salient. But Rambo himself, framed darkly against all of these explosions, becomes clearly salient. On the ‘Fit Kitchen’ ready meal package seen in Figure 5.12 we see a white panel at the top of the label to allow the Fit Kitchen brand to have salience. The black slim font stands out against the brighter white. And together this panel with the shape of the font helps to foreground the idea of simplicity and clean eating.

Tone This can be simply the use of brightness to attract the eye. We see this in a simple form on the packaging for the wax kit in Figure 7.5. We can see that

175

Composition and Page Layout

Figure 7.4  Oxfam poster (https://oxfamphilippines.wordpress.com/ climate-action-now/).

there are highlights, particularly on the woman’s lips, nose and ears. This helps not only to make her appear vibrant but also to draw attention. And the highlights on the lips also add sensuality. We can also see the use of tone on the L’Oréal composition where the product is almost shining. But so too do we find highlights on the face and arm of the model. On one level, this links the product to the model, but at another gives them both salience at a similar level.

175

176

176

Introduction to Multimodal Analysis

We can see this use of tone to create evenness of salience in the Oxfam poster in Figure 7.4. Here each person appears in a brightly toned frame and has highlights on their face. This sense of equality is one important part of the meaning of this poster. Tone can also be seen where one particular element is highlighted through directional lighting. Often in promotional photographs the photographer will direct light specifically onto the product itself. While the rest of the elements and set might be well lit, this may create a very slight aura on the product itself. In Figure  7.7 which shows the university quality development areas, we can see highlights particularly on the plant, the globe and the faces of the students. While the computer and the map are brightly lit there are no such highlights. In this case what is given salience are the natural elements and the human positive reactions. The use of elements of nature and natural materials in design as a whole is associated with bringing a sense of authenticity.

Focus In the news photograph of the Roma camp evictions in Figure 2.3 it is the police officer who is in focus. And this comes out in relation to the Roma, who are not clearly in focus. This has other meanings which we dealt with in the chapter on modality. This brings salience here to the act of policing rather than to the experience of the Roma. And in this case the policing is shown to be something calm and moderate, also communicated by the policeman’s clothing (soft hat rather than riot helmet) and the relaxed pose. But had the Roma been slightly more out of focus along with the background this would have had the effect of giving heightened salience to the policeman, either to his actions or his feelings. Another photograph may have reversed this to show the Roma man in full focus, with the rest of the scene and the policeman slightly out of focus (Breazu and Machin, 2018). On the university vision page in Figure 7.6 we can see that the backgrounds are possible to discern but are out of focus, whereas each of the four people is in full focus. Given they stand in the foreground it is clear that they are the salient element in the photographs, but the reduction of salience of the background also serves to foreground them, their feelings and ideas further. And here in each case the accompanying text describes some of their idea and feelings.

177

Composition and Page Layout

On the BNP poster in Figure 7.1 we can see that the family are slightly out of focus. The BNP logo in contrast appears highly vivid. Here the family are salient as indicated by size, but at the level of focus they are placed slightly in the background. On advertisements we can look for the way that focus can draw attention to parts of the body or face, or to other objects. In the L’Oréal advert, it is the model and not the product which appears in focus. Again at this level the emphasis is on the ideas carried by the model rather than the actual qualities of the product.

Foregrounding While elements in design can be given salience due to size, colour or tone, for example, there may be other elements which sit in the foreground. In the L’Oréal advertisement in Figure 7.2 the product is foregrounded, sitting over the orange box and at the ‘front’ of the composition. On the coffee packaging in Figure 0.1 the workers are salient in terms of size, but the logo to the top right is foregrounded. On movie posters or sports photographs the stars or main protagonists will usually be placed at the front of a group. But the name of the film, as we see in the case of the Rambo poster in Figure 5.7, can be nevertheless foregrounded over these persons. On the Cinderella film poster in Figure 5.8, Cinderella and the prince are foregrounded over the soldiers and the castle. Right in the foreground, however, we see some of the small animals which are Cinderella’s friends. But their small size in this case makes it clear that on another level they are no so salient. In other designs, objects may be larger and foregrounded but be out of focus, to also lower salience. This shows how the different principles of salience may be more or less important in each composition. Different principles may operate to draw out different aspects of the composition. We may also find, as in the Oxfam poster in Figure  7.4, that there is no foregrounding. We see the same in the vision document in Figure 7.6. In both cases all participants and texts remain at the same level. In both the cases equality is an important meaning, even in the case of the vision document where the images in fact show the vice chancellor, a professor, an administrator and a student. The point here is to show that the vision is ‘our’ vision, rather than something coming from the management.

177

178

178

Introduction to Multimodal Analysis

Overlapping This principle is like foregrounding since it has the effect of placing elements in front of others. For example, in Figure  7.1, the BNP logo overlaps the picture of the family, which at another level is a salient cultural symbol in the design. On the Fairtrade coffee package in Figure 0.1, the photograph is overlapped by the box at the bottom which shows the source of the coffee. The salient image therefore appears contained in the foregrounded brand logo and the other product information. On the children’s camp website in Figure  5.9, the photograph of the children, which is the largest single element on the page, is overlapped by the ‘summer fun’ information. This particular group of people, therefore, is shifted slightly into the background. In a sense it helps to signal that the camp is not about this specific group of people. It is, rather, an indicative group. The photograph is also overlapped by the small pictures of different kinds of balls. We might, therefore, say the ‘fun’ and ‘activities’ connotations these bring are given salience. No overlap at all can also be used to give meaning. Again this can suggest evenness and equality. But it can also suggest order, formality and tidiness. On the Bounce packaging in Figure 1.3 none of the elements, such as the icon of the jumping person or the elements in each colour band, overlap others.

Summary of principles of salience ●● ●● ●● ●● ●● ●● ●●

Potent cultural symbols Size Colour Tone Focus Foregrounding Overlapping

Framing This has to do with elements being represented as separate units or as related. The degree of flow around a page and certainty of connection between

179

Composition and Page Layout

elements can be indicated by framing. Frames can show boundaries and the absence of them can show natural connections. The symbolic value of frames can be found in the way that exhibits in a museum might have frames around collections of artefacts. Those placed within a frame are defined as being somehow the same, as having a common identity and also as being of a different order from those in other frames. So in one frame in an exhibition on exotic cultures we might have a spear, a cooking implement and an item of jewellery. Of course, these are very different kinds of objects, but they are displayed as being the same through being placed within the same frame. Obviously this is ideological and has the effect of making cultures appear easily definable and simple. We find frames all around us. Buildings can have boundary walls around them. These walls might be very small, to suggest that the building is part of the surroundings, or they might be very large to show separateness. The same kinds of principles of framing apply in composition. As in the case of frames containing cultural artefacts and walls around buildings, frames can create connections or emphasize differences between elements. Frames can be created by lines or by spaces, such as the edge of photographs, or by icons such as, say, rows of paw prints in an item on animals or celluloid film reel for an item on movies. Images themselves can also be used as frames. Sometimes this can lead to a more subtle separation of elements. Page designers can use these devices to separate or create more natural links between elements. There is also much meaning potential in frame size, shape and regularity. Frame size itself can be used to create hierarchies of salience. But tall, narrow frames can also have the meaning of claustrophobia and restriction and tension (Eisner, 1985). Jagged lines can be used to mean explosive energy or danger, softer lines for comfort, softness and dreaminess. Van Leeuwen (2005) suggests that it is possible to create an inventory of terms to describe different kinds of framing and connectivity. Each of the categories below describes the semiotic potential of types of framing/linking. As with previous semiotic meaning potentials, these are not fixed. Meaning depends on the use of other semiotic tools. Van Leeuwen lists six categories: ●●

●●

●●

●●

●●

●●

Segregation Separation Integration Overlap Rhyme Contrast

179

180

180

Introduction to Multimodal Analysis

Segregation Here the elements are separated and occupy different domains and are therefore of a different order. There will be some kind of frame of differing thickness or other sort of clear boundary. We see this in the case of the wax packaging in Figure  7.5. There is a clear boundary between the woman’s face and the product information which sits in the coloured section above. She therefore is classified as being apart. Given this is a wax not used for the face, this could help the design to nevertheless still carry the picture of the attractive woman. Had the waxing information been placed over her photograph, this may have been confusing. We can also see here that the boundary has a curved form. For a product targeting male customers this may be less likely. And on a few that we saw there was even the use of a thick straight boundary which had a metallic appearance between a male model and a beauty product. We see clear boundaries on the Bounce packaging in Figure  1.3. The icon of the jumping person, the sketched circle and the brand name sit in a different section from the information about its contents and qualities. On the next level down we find Spirulina and Ginseng:  defence boost; on the next level a mixture or comments on texture and contents and the statement. High in antioxidant vitamin E as well as three other qualities are listed below a thin black boundary. As we have commented elsewhere in the book, such products, marketed as healthy, are in fact very similar in content to traditional non-healthy snacks. And it has been said that the good qualities they claim, for example, providing defence, antioxidants, gluten free, are a ‘hodgepodge’ (Schneider and Davis, 2010) of things which are used to communicate that they are good for you. In other chapters we have looked at the role of colour, typeface and texture also telling consumers that this product is healthy. But here the boundaries help to provide a sense that these are separate and distinctive things. Note that at the bottom there are also borders between the three qualities of ‘no artificial ingredients’, ‘gluten free’, ‘high fibre’. Again this helps to create a sense that there is some kind of order here. Of importance here though is that the boundaries are not particularly thick. The different colour panels are used to create the different sections and a fine black line for the bottom section. Had these been thick, heavy lines it would have suggested that these were of a hugely different nature.

181

Composition and Page Layout

Separation Here the elements are separated by empty space. There is no physical border. In this case the elements on each side of the space have not only some similarity but also some difference since they are kept apart. This is found in the case of a news headline and the main text that it introduces. There is normally a gap between the two but not a boundary. Where text follows on immediately without a space we tend to assume that it is part of the same thing. In books such as this one some section headings must be separated from the text below, whereas some are not. This has the effect of creating different levels of connection between the heading and the text that follows. Where there is a gap of a few lines it indicates that the heading speaks more broadly about all the text that follows. Such headings indicate broader themes that are to follow. Where the heading is followed immediately by text and there is no separation, we assume a more direct connection. On the vision document in Figure  7.6 we see this use of separation to create extensive space between the headings and chunks of text and between the different groupings of images and their texts. This tells us that these are all in some way part of the same thing. But while on the one hand there are no boundaries, on the other the extensive use of separation creates a distance between them. This use of separation is now common in document design. We see this in Figure 2.1 where the images of the quality development in the preschool sit in space around the circle which carries the labels for each stage. Arrows are used to help create an indication that these are examples of each particular stage. But the use of space here helps to diminish the level of the claim that this is exactly what it is. On the Oxfam poster in Figure 7.4 we see the use of separation by the spaces between each of the frames. Each of these people is somehow apart from the others, in their own space, yet there are no significant boundaries between them. This is suitable for the message of the advert which suggests that we can all be part of this community. This would have felt very different had the spacing been much greater, such as the vision document. Here the links would have felt much weaker. As with the visual document there is a sense of community being suggested here, and also of equality, since there is an emphasis on images of the same size. But the separation plays an important role in the difference in meaning. But important in both designs

181

182

182

Introduction to Multimodal Analysis

is that the degrees of separation are even between the images. Therefore, it is suggested that there are equal relations between the participants. Had some been closer or further away this would have symbolically suggested different kinds of relationships.

Integration Here elements occupy the same space. Images can be placed in the space occupied by text or text can be placed over images. The Rambo poster in Figure 5.7 is an illustration of this. We see that the name of the film sits over the person and the scene. There is no boundary, nor separation. In a brochure such as the vision document in Figure 7.6 the statement made by a person could be placed in text over their photograph, therefore making them into one single entity. On the plastic surgery website in Figure  1.1 we can see that the ‘tummy tuck’ text appears integrated with the image of the woman’s body which is also integrated with the graphic pattern which sits behind here. Here the torso and the surgical procedure become part of the same world, where surgery becomes the natural way to the perfect body. It would have been different had the body and the text been segregated by a boundary as in Figure 7.5 for the waxing product. In cases of integration the elements operate as one. On the plastic surgery web page we see that the logo of the clinic is itself segregated and sits in a border. The ‘contact us’ information then sits also apart within its own frame.

Overlap Here elements creep onto the space that has been segregated or separated for other elements: rather than integration we have ‘bleeding’ of meaning. The element that has the ability to leave its frame is shown as being unrestricted by the conventions of the composition. Eisner (1985) suggests that where elements break frame it can therefore give them the impression of strength. In newspapers, text and image have traditionally appeared with frames or with separation – there is no overlap of elements. This can indicate some degree of formality. Where they have done so, in the case of a newspaper, such a breaking of boundaries between formal textual information and, say, an intruding image was formerly associated with a more tabloid title. In Figure  3.6 showing the cover of the magazine Closer, we see this kind of overlapping. The titles, different images and text boxes all overlap each

183

Composition and Page Layout

Figure 7.5  Wax kit packaging, Sally Hansen.

other, creating no clear sense of hierarchy, although size is used for salience. This has changed as we have moved deeper into the twenty-first century. Images, texts and drop boxes can overlap each other to create different page rhythms. Newspapers and other kinds of documents will allow images to overlap across borders. To avoid a tabloid ‘fun’ look, this will be done more carefully using patterns like symmetry or asymmetry. This not only can make a composition seem lively and less formal, since boundaries are not being obeyed, but also allows the meaning of elements to leave the confines of the frame. Here the suggestion can be made that the meaning of the two is somehow linked or shared. Often in advertisements a product may overlap a border into another frame, perhaps where the model is placed.

183

184

184

Introduction to Multimodal Analysis

On the sustainability scorecard in Figure 4.1 we can see that the different sections, for recycling and species diversity, overlap each other slightly. In this case these highly different things are given more coordination of meaning. They allow the associations of one element to bleed into another frame. In some advertisements we might not want the product to overlap into the image. For example, in the L’Oréal advertisement the meaning would have been different had it overlapped the model. While this may be usual for women’s cosmetic adverts, it is less usual for men’s products.

Rhyme Elements within frames or in different frames may be linked in some way. Connectivity can be done through any common quality such as shape, colour, posture, texture and so on. And rhyme can be created through different kinds of semiotic materials that carry meaning potential that are able to relate them. A simple form of rhyming would be where a product logo might be red, which rhymes with the clothing of the model who lies in a different frame. In Figure 5.12 we can see that the Fit Kitchen packaging uses the same font for the logo as for the technical information provided in the section bordered below. On the Bounce packaging in Figure 1.3, we also find same fonts used in different sizes across the different bands of colour. But here the qualities of the three colours also create some rhyming. They are all similar kinds of hybrid colours carrying shades of white. In the chapters on colour and texture we showed how cafes like Starbucks can use such colour schemes to create harmony and rhyme across elements in their interiors. Such colours may also rhyme with the more natural and bare textures used in flooring and surfaces. The sustainability scorecard in Figure 4.1 uses colour and font and style of iconography to create rhyming across the different panels. This plays a huge role in communicating that these different things have common core qualities. But how installing more cycle stands is related to biodiversity and volunteering as an overall sustainability plan is not clear. And closer inspection of the information reveals that it is problematic. Why is it important how many staff participated in collecting species to measure biodiversity? Should staff not be doing their jobs as teachers and researchers, or as administrators supporting the teachers and researchers? And how do we know the meaning of such a count of species? Here the rhyming across the panels helps to create

185

Composition and Page Layout

coherence and the coordination of the different thing. In another design we may have also seen a photograph of some glowing leaves or smiling students as we see in the university vision document in Figure 7.6. How the different elements actually related in terms of causalities, actions, agents would not be specified. But the rhyming devices would symbolize the connectivity. In a lifestyle magazine, the same colours might create connectivity between a text, a photograph and even an advertisement on the same page. While these may all be placed within their own frames or separated, they are nevertheless given connectivity. This is one way in which connectivity can be created between the world represented in feature items and the world of advertising. This can create a seamless link between real-world issues and consumerism.

Contrast Elements in different frames or panels may be differentiated through different characteristics and semiotic resources. Importantly contrast and rhyme can be happening at the same time. This is important as regards the ways that designs symbolically signal the similarities, coordinations and differences. At a basic level we can see contrasts in Figure 5.9 of the children’s camp website, where we see different font colours, as in the case of ‘registration’ and ‘sports camps’. The font is the same, which relates the two, but colour suggests contrast. We see the same kind of thing on the sustainability scorecard. Here while rhyming is accomplished through fonts and colour qualities, difference is communicated by colour hue and by frame sizes. So just as how the different items are the same is only communicated symbolically, so also how they are different is communicated at this level.

Summary of semiotic resources for creating framing and connectivity ●● ●●

Segregation: use of physical frames to create difference. Separation:  separation by space rather than by frames. Here there is difference but not to the same degree as in segregation. Space can be bridged.

185

186

186

Introduction to Multimodal Analysis

●● ●●

●●

●●

Integration: this is where elements occupy the same space. Overlap: this is where elements are not constrained by frames and spaces. Their meaning can also bleed into other spaces. Rhyme: colour, posture, size and so on can be used to create links between elements. Contrast: colour, posture, size and so on can be used to indicate difference.

Images and composition: Coordination and hierarchies In Chapter 2 we looked at tools for analysing the contents of images. This involves how we can look at the meaning of objects, settings and how persons are represented. But also important is how photographs are placed and used as part of designs. Here we look at how they can be used to create forms of hierarchies and rhyming on a page and also specifically at how they can be used as a way to provide coordination, or links between issues across images or between images and texts that may conceal contradictions and conflicting matters. This relates to many of the points made above in regard to framing. But there are some additional observations we need to make about the photograph.

Photographs, hierarchies and rhyming Photograph sizes and placing can be used to give elements different degrees of salience and relationships to other elements on the design. Same-size frames can be used to give the impression that very different elements are of the same order. In Figure 7.6 showing the university vision, we can see that this double page spread represents different kinds of people involved in university life: management, a professor, an administrator and a student. These are used to symbolically represent ‘all levels’. There appears to be a hierarchy from right to left as we begin with management. But in other ways equality is symbolized through the same sizes of photographs and the same size chunks of text. On the quality improvement web page in Figure  7.7, we also see same size and shape images used to classify hugely different and abstract issues

Figure 7.6  Page spread of ‘Vision 2016’, Örebro University Magazine, 2012, pp. 20–1.

187

188

188

Introduction to Multimodal Analysis

as somehow forming a coherent group. In such designs the rhyming is also performed by the table type layout so that each element has the same parts: an equal-sized text and image. In contemporary document design we are frequently confronted by elements which are ‘boxed’ in the same way to suggest sameness, relationships between them or contrast. If we look at the two lower text boxes they tell us (bottom left) ‘At Örebro University sustainable development will shape the whole working environment’ and (bottom right) ‘Internationalization is a way for Örebro University to increase quality across its whole working environment’. Opening the two documents which can be accessed in each case shows how abstract the two ideas are. But these have become required at a political level, originating in the EU, to show quality increases. So staff are required to show how both are involved at all levels of work, such as teaching. In many ways, the two are hugely contradictory. Sustainability suggests measure and environmental caution. Internationalization is related to globalization. The spacing of the images helps to avoid the suggestion that they are some kind of group and to background the actual sense that they do represent everyone in the university. And since they are represented in individual frames and not together in one frame, it allows them to remain as individuals. Other images in the same document show different kinds of teaching staff in one image standing close together to suggest a close community and sameness. In other designs we may find image sizes used to suggest difference or hierarchies. But this can be used to suggest ‘difference’ and ‘variety’ where there is in fact none. On the Oxfam poster in Figure  7.4 we can see that some of the photographs are slightly bigger than others. The message of the overall poster is of equality and sharedness. But having different sizes of photographs suggests there is some uniqueness. But this is also done in the context of other qualities and features which also create rhyming, such as colour hue and brightness. And across this design the degrees of separation are even, suggesting even distances between the people.

Photographs and coordination This quality of the photograph relies on its affordance to give the impression that it neutrally represents reality, that they are captured moments in time. Even highly symbolic images such as we see in Figure 7.7 for the quality development web page comes out as if they are somehow naturally related to the concepts and issues to which they are

189

Composition and Page Layout

assigned. Viewers do not, for example, look at this page and think ‘who are those young people’, or ‘what is that small transparent globe doing among those ferns’. Such images, in visual designs, can be deployed to play an important part in what is called ‘coordination’. Coordination is simply the presenting of two things as being one or as closely aligned. The quality improvement web page deploys the same size and shape of images to classify hugely different and abstract issues to suggest that they are somehow a coherent group. Similarly, the university vision document in Figure 7.6, which also includes targets and strategies to fulfil the vision, is in many ways highly abstract. At ground level of teaching and researching they are pretty meaningless. Such visions are part of the way institutions must self-brand, be ‘ambitious’ and act like corporations. But this configuration of images symbolically represents how each level of the university experiences the vision. The people we see in the photographs become coordinated with the meaning of the vision. The use of separation and large sections of white space also serves an important role in loosening how direct this kind of coordination is represented. Also important in coordination here is that the images can suggest that they show participants or processes not clearly identified in the text. So a text about students in general benefiting from a situation or policy can be placed in proximity to a photograph of smiling students. These are clearly not actually the students who are benefiting but symbolizes students in general. The problem is of course that documents can be formed not of running texts but of chunks of texts, often carrying buzzwords about innovation and quality improvement. The participants, such as who carries out all the work, and the beneficiaries may be only ever symbolized such as in this image of the students. The complex notion of sustainability and what a single university can do to save the planet (as well as educate students so that they all get jobs) becomes encapsulated in a photograph of a delicate leaf and transparent globe. In the end what gets produced are more abstract documents of the like seen in the case of the sustainability scorecard in Figure 4.1.

189

190

190

Introduction to Multimodal Analysis

Figure  7.7  University web page for quality development. Orebro U n i ve r s i t y   h o m e p a g e :   h tt p s : / / w w w. o ru . s e / o m - u n i ve r s i t e t e t / vision-strategi-och-regelverk/.

Summary of image and composition Hierarchies: sizes and placing can be used to give images different degrees of salience and relationships to other elements on the design, which create hierarchies in the composition. Coordination: this is simply the presenting of two things as being one or as closely aligned, for example, using the same size and shape images.

191

Composition and Page Layout

Student Activity 1. Choose two advertisements of the same product, but aimed at different types of people, such as age or gender. Describe the different forms of salience that you find. Do these create differences in what gets foregrounded and backgrounded? Critically reflect on the notion of the term ‘eye catching’ often used by people to talk about designs. Why is this term in fact not at all helpful? 2. Carry out research online to find and collect web pages, book pages, documents, advertisements and so on which can be used to illustrate examples of uses of framing and borders in designs: segregation, separation, integration. Explain how these work to create meaning in each case. And in each case explain how this would affect the meaning if the border choice was changed – so, for example, if in place of separation we find integration. You can use Photoshop to edit documents to experiment with how such changes influence meaning. 3. Take a set of photographs of a number of your fellow students. These will comprise:  a portrait, a full body pose, one where they are doing an action, such as jumping, one where they are laughing, one where they wear an expression of ‘wonder’. Using Photoshop or simply by printing them out and pasting them onto a sheet of A4 you will create two very different designs. You will show how the same photographs can be sized and organized to suggest (1) order, equality, formality, separation; (2) fun, informal, integration.

191

192

193

8 Diagrams and flow charts Throughout the chapters in this book we have been showing how semiotic resources are mobilized together to create meanings. Depending on context, different semiotic resources are used. An idea about the nature of events in a place can be communicated by the quality of a colour placed on a document, by a photograph, by the textures used on food packaging. And different things, persons and process can be classified as sharing properties or as being different due to how they are placed in compositions or by using generic types. So identities, ideas and relationships may never be specified in language, but are communicated by other semiotic resources and their deployment. In Chapter 1 of this book we explained how this is part of a wider shift in communication to a form of ‘integrated design’. Here ideas, attitudes, identities, relationships and causalities can be found symbolized rather than clearly accounted for in running text. In this chapter, we look further into how forms of classifications and processes are represented in different types of diagrams. It has become commonplace to find diagrams such as flow charts, line graphs and so on in documents and media, used to illustrate how things work in their essence. In this type of visual communication, the most important semiotic resources employed are language, specifically chunks of written language; graphics, such as shapes, arrows and lines, and composition, where these elements are distributed in space, on a paper or a screen, using different software. A simple example to illustrate this can be seen in Figure  8.1. Let’s say that these diagrams are used by the management to show how a university is organized. As we see, chunks of writing are used to list three groups  – manager, academics and administration. But with the use of graphics and composition we get three different diagrams that point to very different organizational principles. In the top taxonomy-type diagram, we see the university

194

194

Introduction to Multimodal Analysis

Manager

Academic staff

Admin staff

Services

Manager

Academics

Academics

Manager

Admin

Admin

Figure 8.1  Different diagrams representing the same organization (created by authors).

represented as a hierarchy of relations. The management is shown at the top and the different kinds of employees below. In such cases we can consider how the sub-levels are represented. Are they parallel? In this case we see that academics, administrators and services are placed equally at the same level. We could also ask if the boxes are the same or different size which might also suggest sameness, difference or some kind of ranking. In this case they are the same. In terms of analysing such diagrams we may want to ask if actual differences are somehow concealed, for example. One of the authors was recently at a meeting where academics had been placed at the bottom of a taxonomy beneath administration, suggesting clearly that their work was controlled by administrative priorities. The two diagrams below also show the structure of the university but emphasize something different. The diagram to the left shows management, academics and administrators as three same-size circles which all overlap in the middle and which overlap each other in slightly different, but equal, ways. This kind of diagram emphasizes a goal-oriented structure, rather than hierarchical. Here there is a sense that management is part of the collaborative process. But in this diagram where collaboration and working together specifically take place is not specified only symbolized by the overlapping areas.

195

Diagrams and Flow Charts

Of course, while we might find such a diagram in a document, the actual university may be increasingly hierarchical. And in many work environments, a shift to increasingly top-down and micro-managed forms of organization leads to a corresponding set of staging of equality and consensus in such documents. In this diagram we also see that services are no longer represented. One of the authors worked at a university where services did become removed from all documentation once they were outsourced to a company many miles away. The bottom right diagram represents the three participants equally through writing, in the same font and size. What is represented graphically here seems to be a kind of directionality, since we find arrows moving clockwise. While the first diagram suggested a hierarchy, this one suggests an ongoing cycle. And it is one that is entirely balanced where the parts are equal. The arrows at each stage are the same size, colour and thickness. Here we see that in fact while the academics respond to the management, the management in turn relates to the academics through the admin. Of course, such a diagram may be used simply to suggest that the university is equal, and an ongoing cycle where we all work as one part of the cycle. But again, in analysis we would want to ask why a management may want to represent an organization in this manner. What is clear from these examples is that diagrams are strong semiotic tools for the representation of general principles and processes, for example, modelling the functioning of an organization or a technical manufacturing process or learning outcomes in schools. In short, diagrams abstract what actual people and things do in actual contexts. Obviously, different individuals of a university staff will have different relationships with each other and with managers and administrators, but the point of a diagram is to de-contextualize, to represent principled abstractions and omit what people do in actual time and space. So diagrams are strong semiotic tools also in the sense that they serve certain – but not other – interests. They map the world in ways that necessarily de-contextualize, that omit people, processes, settings and relationships that exist in actual time and space. In Figure 2.1, for example, we looked at a PowerPoint slide which showed how all aspects of child development in a preschool could be improved. This is not represented in running text saying who does what and why, nor how the different parts are related. This is abstracted and represented as a kind of flow chart. At the centre of the diagram is a circle which carries the words ‘national goals’. Around this we find four stages: ‘plan’, ‘document’, ‘follow-up’, ‘reflect’. These are shown on the outer part of the circle orbiting the words national goals. And we can see that this is represented as a cycle with no finishing point

195

196

196

Introduction to Multimodal Analysis

since each stage points to the next, as is indicated by the arrow form of the box in which is sits. This is a never-ending process of quality development. Such quality development systems are related to very specific trends in society, as we also discussed when analysing the diagram of the Swedish School Inspectorate in Figure 1.4. All things, particularly in public institutions like schools, hospitals and churches, must be shown to be improving. And this must be a measurable thing. Charts and graphs can then be made to allow comparisons to be made. Managers and policy makers who know very little about teaching and learning can then make judgements as to where further quality improvements need to be made. For the actual teachers in the case of the school it has been shown to be a highly depressing process since their professional judgement is no longer what must guide their practice but these national goals. And such goals tend to no longer come from priorities defined by teachers themselves but by policy makers. In Figure 8.2, we see a related diagram from a preschool in Sweden that one of the author’s children attends. The school has a ‘quality wall’ where different goals and strategies are pinned and given graphic shapes separated in space. The heading at the top states ‘today we “only” played’. The diagram below has play (LEK) at the centre, represented in a ‘cloud’-type form. Around this we see the target areas for quality improvement which must be used as the drivers for the four-stage process seen in Figure 2.1. Bottom left we see ‘motor skills’ (‘motorik’). To the middle left ‘intellect’ (‘intellekt’). Other goals for quality development that may be less traditionally found in the preschool are found at the top right ‘social competence’ (‘social kompetens’) and at the very bottom ‘feelings’ (‘känslor’). Social competence includes things like ‘teamwork’ (‘samarbete’) and ‘taking initiative’ (‘initiativtagande’). Feelings include things like ‘sympathy’ (‘sympati’), ‘love’ (‘kärlek’). Of note is that this diagram classifies motor skills and feelings, knowledge and social competence as being of the same order. All are carried in the same font and placed in an oval frame and are positioned in an ordered way circling play. How much these things are different or the same, or how appropriate it is that they are subject to quality development, is excluded in part by this form of classification. And each target area has a box containing the lists of examples. So we find that ‘play’ sits at the centre and is therefore what unites all of these satellite things which orbit around it. Of course, what is really at the centre of this is the political drive to take control away from professional teachers. It also allows them to set up a system where responsibility for good schools is not based on the government providing a sound infrastructure,

197

Diagrams and Flow Charts

Figure  8.2  Preschool chart showing how play can be broken down and evaluated (authors’ photograph).

sufficient staffing, good working conditions, but on the staff themselves to show they are meeting targets. In Chapter 1 of this book we referred to the process of technologization in society. This is a process whereby communication and processes become increasingly codified in order for them to be more closely managed. This relates to the increasing level of commodification in society – where all things are seen as products to be marketed at specific target groups. And it relates to the kinds of social organization where all things are treated under market principles. So, whether it is a product being made in factory or a four-yearold child in preschool, we must maximize outputs. Workers, whether in a factory, or teachers in a preschool must clearly show how they are delivering quality. Work must be codified and measured, and here diagrams are a key tool. And the interest in this chapter, as runs through this book, is in how diagrams and flow charts can be used to shape how reality is presented.

Classification In what follows we present a set of tools for analysing how diagrams communicate. As we have shown writing, graphics and composition are the

197

198

198

Introduction to Multimodal Analysis

major semiotic resources mobilized in diagrams, so we will emphasize these. Obviously other semiotic resources are also deployed, such as colour and typography, but they are not defining features of diagrams. A core feature of diagrams is classification. If we go back to Figure 8.1, we find writing used for a classification including the entities managers, academics and administrators. With the use of graphics and composition these abstracted groups enter in to different principled relations:  as a hierarchy, as a goal-oriented structure and as an ongoing cycle. We can create classifications and categories out of all sorts of things. We might make a list of different kinds of flowers, sports cars or a shopping list. What is important here is that such lists are largely socially agreed upon, that these things share qualities in common. They are what are called paradigms. One paradigm here is that of flowers. We would not expect to find a sports car included in this category or paradigm. Such paradigms are culturally agreed upon. But they are always constructed and presented in someone’s specific interest. A shopping list may have the interest of making sure that we have all the things we need to eat in the house over the coming week. But a list of ethnicities or racial types may serve a different kind of interest such as the apartheid system in South Africa (Bowker and Star, 1999).

Classification and writing In diagrams, classifications are typically rendered by chunks of writing. In the list of components of play in Figure 8.2, we find a huge range of things classifying play as related to feelings, social skills, ideas, motor skills and so on. This clearly reflects the interests of a particular ideology, of what is called neo-liberalism, where the state is no longer itself responsible for the quality of schools but devolves this to managers who are to measure quality through performance measures. In Figure 2.1 the processes of: plan, document, follow-up, reflect become a paradigm:  the category of the things involved in meeting the national goals. Of course, we might contest these. When I see my five-year-old son playing with stones in the forest, pretending they are treasure, or dinosaur bones, I  may want to suggest a different set of categories to identify the components. In a former time this may have been related to play as a form of distraction, as pleasure, as unstructured imaginary exploration. But these things sound less like part of a paradigm of quality development. So when we are analysing diagrams we can ask what kinds of things are presented as

199

Diagrams and Flow Charts

a category or paradigm and how transparent this is. We can consider how such classifications include and exclude and shape the structure which is represented to support specific political interests. Paradigms can be seen as mostly comprising processes, things or people. Linguistically these are coded by verbs, names and nouns. And a characteristic of integrated designs such as flow charts and diagrams is the way that writing is also used to classify things as being of the same order due to the grammatical form chosen. Below we look at representing processes and people in a little more detail.

Processes Flow charts and diagrams often represent some kind of process. This can be represented graphically, which we come to shortly, or often through specific kinds of verb forms. We often find the use of imperatives and infinitives. Again these can classify things as being of the same order, when they are not, and are also useful for shaping how processes appear, or how they do not appear.

Imperatives These code and issue a command. This is like saying to someone ‘bring me a drink’. You do not ask them in terms of ‘could you bring me a drink’. You command them. Imperatives are often used in advertisements such as ‘buy one now!’. It can suggest something immediate with no messing around. In Figure 2.1 we can see that the four processes involved in meeting the national goals are coded as imperatives, such as ‘Plan your activities’ (‘planera er verksamhet!’), ‘Follow up your work’ (‘följ upp ert arbete!’). And all end in exclamation marks for the ‘do-it’ emphasis. This verb form is also important for flow charts as it does not explicitly show who does the action, just that it should be carried out. And for a chart which claims to represent the core of how things are done in a workplace, this may be problematic.

Infinitives Infinitives are the root form of verbs. They do not need to include who is doing the verb or who is affected by the doing. If I ask what you usually do

199

200

200

Introduction to Multimodal Analysis

in the evening you may answer ‘watch a film, meet friends’. These are general activities. In Figure  8.6 we see a strategic plan for the Australian university of Newcastle. We can see that the goals are at the top in terms of things like ‘leaders in education’, ‘research excellence’, ‘diversity enrichment’. These goals each sit at the top of a column and here we meet nominalizations. There is a direction flow upwards to the goals where each column shows the steps in the university to achieving the goals. This is the strategic plan. The goals running along the top will relate to ‘metrics’ or things the university must show numerically to the government that they are improving. Looking down the columns we see the use of infinitives to code what gets done to achieve the goals. So we find ‘Understand students needs and expectations’ as a step to being leaders in education. We find ‘Develop centres of research excellence’ as part of the goal of research excellence. For the goal diversity enrichment we find ‘Develop entry pathways and support mechanisms’. It does not state who will do these things. And given this is a strategic plan one might expect for it to be clear who does what. In fact it is often the case that all academic staff must show that they are meeting all of the targets, which can create huge pressures on workloads and distract from core activities such as teaching and doing research. This has been one standard criticism of these kinds of goal-based systems (Power, 1999).

People People can be included in the writing on flow charts, or they can be excluded. So we can look for how people are represented; are they shown as individuals, groups or generic types? The kinds of classifications of persons offered can invite a simplified view of the world and of processes. On the university diagram in Figure  8.6 we find two main collective groups in the strategic steps. We find ‘students’ and ‘research graduates’. These are represented as one singular group in each case. Any teacher knows that students differ greatly across a cohort and across different courses. But here we are not encouraged to think about the students as individuals. This follows the pattern of the representation of quality development in Figure 2.1. This is a one-size-fits-all model. But for the most part this diagram foregrounds processes rather than persons. There is also the collective ‘we’ found in the mission statement at the top of the design: ‘We are a university of distinction’ and ‘We are responsive, dynamic and strong’. But it does not specify if this is all employees. So does this include admin staff and cleaners, for example?

201

Diagrams and Flow Charts

In language many actors are missing in this flow chart. Teaching staff are not present. This is interesting, given it is the staff who must create the satisfied students, do the research and create the community engagement. And it is not clear who in the community will benefit from the engagement since this remains at a generic collective level ‘community’. Nor is it specified who benefit from the nominalized ‘diversity enrichment’. We do find a list of ‘stakeholders’ to the top left: students, community, employers, funders, professional bodies. And this classification, this paradigm, is interesting given that formerly the main actors in an institution aimed at creating the highest level of education in society, would be the teachers and the students. This relationship is entirely absent from this kind of list which foregrounds not knowledge or education, but the university’s instrumental role in relation to employers and the community. But how these different, varied and possibly clashing interests can all be met is not explained in the flow chart. For example, do the needs of employers and businesses clash with things like community engagement? And missing from this list are the indigenous groups who benefit from ‘diversity enrichment’. So while having the appearance of something systematic, a strategy, there is vagueness and confusion at many levels.

Visual classifications Of course, people, things and process can also be classified visually. In Figure 2.4 we see a more contemporary style strategic diagram for a university. This time it is done in 3D in the form of a pyramid. Here we find lists of processes presented as nominalizations such as simply ‘engagement’ and ‘participation’. We are not even told what the students here must engage with or participate in. But in this case we do find two sets of participants: ‘staff ’ and ‘students’. And these are represented visually in a 3D graphic. So here the people who are missing from the language which tells us about the processes through nominalizations are represented visually. We are not told ‘these are the students who are engaging in a specific activity’, but they represent the overall participants here. There is an attempt to suggest individualization as the figures appear to be unique. A  closer look, however, reveals that there are a limited number of figures who are arranged to conceal this and create a sense of individualness. But these are clearly generic types represented through 3D cartoon characters. All appear to strike vibrant, ‘ready’, poses – so ready to

201

202

202

Introduction to Multimodal Analysis

do ‘engagement’ and ‘participation’. And we see the same amount of staff as students (11). These figures help to positively evaluate the strategic plan. Here it is of note that actual photographs of staff and students are not used, although in other cases such a strategic plan might be accompanied by images of smiling students as we saw in Figure  7.7. On one level the viewer will know that these are not the actual students referred to in the text. But such images have the affordance that they do document actual moments in time and therefore can carry such meanings to an overall design. A university strategy diagram may carry images of a new atrium building filled with light as well as images of smiling students. Here, just as the 3D images of the staff, these persons come to represent all staff in the university so the atrium building represents the setting, glossing over the huge variety of workplaces which comprise a university.

Graphic shapes In flow charts and diagrams the shapes that are deployed bring specific meanings to the classifications used. And these can come built into the templates of the software used to create such designs. For example, curvature can mean something more ‘organic’ or ‘soft’ and can be used to suggest something less fixed or rigid. So in Figure  8.6 we can see that the components represented as the stages leading to the fulfilment of the goals are rounded in shape. Angularity, in contrast, can suggest something ‘definitive’, ‘fixed or ‘closed’. So in Figure 8.6 we can see that the goals are represented in rectangular boxes which suggest that these are the more fixed components of the whole strategic plan. So too we see that ‘performance culture’ is in such a box. The suggestion here may be that these are the root foundations upon which everything is built. The use of graphic shapes to symbolize types of classifications can be seen in the logic of the diagram in Figure 8.6. All the components in the strategy which sit in the five columns are shown in oval graphic shapes. These are classified as being of the same order. So ‘community based learning’ is the same as ‘satisfied students’. Yet these are coded as being different to the goals at the top which are placed in squared graphic shapes. Here ‘leaders in education’ is coded as the same as ‘research excellence’. What is of interest to us in such designs is the way that actual differences and complexities are overlooked by such classifications. We see this in the way that each goal has

203

Diagrams and Flow Charts

its own column which has the same form. So in each case it is the same basic strategic steps which lead to its fulfilment. And these are each classified into ‘drivers’ and ‘outcomes’ in the arrows to the left. Here it is interesting to see what is classified as a driver or an outcome. ‘Develop centres of research excellence’ seems to be more of an aim or ambition than a driver. And at the base of diagram is not what might be expected in terms of good infrastructure, such as good resources, staffing, working conditions, good staff–student numbers. Instead we find ‘performance culture’ and a list of mindsets. The infrastructural things such as ‘student support services’ appear to arise from having this culture and mindset. Workers in organizations subject to these kinds of strategic systems can experience that management documents lack engagement with fundamental practical issues of getting things done. Graphic elements can also be iconic. In Figure 2.2 we saw that they can be in the form of cogs to suggest interlocking parts working together towards an outcome. Others may take the form of cell-like octagons which can also suggest ‘building blocks’ in the fashion of chemistry. Elements can also take the form of houses to suggest building or of trees to suggest something which grows or flourishes. They can take the form of a human to suggest the parts of a single agent. In Figure 2.4 we see the form of a pyramid. As a design form this is useful to show the gradual focus into a more narrow focus which is at the top. But pyramids also have connotations of classical civilizations. In Figure  8.4 we see the form of a picture-book drawing of a school used.

Size and weight of graphic elements Graphic elements can also be classified or give salience by the size and weight. It may be in the interest of a designer to represent things which are very different as being of the same order. In the three diagrams in Figure 8.1 all emphasize that there is sameness in this regard between management, academics and administrators. In another diagram one of these could have been larger to suggest importance. In Figure 8.2 all of the components which orbit play are of the same size, depending simply on word length, even though the list of things in each category varies. There is no sense that one is more important than the others. In Figure 2.1 we see the decision to make the photographs roughly of equal size to the circle at the centre which carries the components. We can imagine the difference here were the circle to be much larger with very small orbiting

203

204

Figure  8.3  Pathway flow chart for Fairtrade shopping. Image from ‘Want to Learn More About Fair Trade? Here’s a Primer’ by Fair Trade USA, GOOD Worldwide Inc. (www.good.is), 6 November 2012.

205

Diagrams and Flow Charts

photographs of children and teachers. Here we may get a sense of them being overwhelmed. Here there is more of a sense of balance of proportion. On the pyramid in Figure 2.4 the images of staff and students take up less of space than each of the three stages which lead to the fulfilment of the vision. We can see the importance of size on the Fairtrade diagram in Figure 8.3. The effects of buying Fairtrade:  ‘protecting the environment’, ‘improving lives’, ‘quality products’ are represented in the same size – rounded graphic shapes. So too are the icons used to represent a range of things like ‘health care’, ‘environmental protection’ and ‘democracy’. These are in fact hugely distinctive things yet here classified as the same. Research into Fairtrade shows its complex and highly uneven nature across the world (Low and Davenport, 2005). And it shows how Fairtrade often takes place in countries devastated by IMF loan repayments and resulting austerity measures which typically result in deterioration of services such as water, health and education (Stubbs, 2016). Such a diagram, therefore, serves to grossly simplify the nature of global trade and where solutions may lie. And critics of Fairtrade argue that it gives the impression that we can make substantial changes in the world, such as bringing about democracy, saving the environment, through acts of shopping in the face of the structure of the global economy organized to favour large global corporations. As we discussed in Chapter 1, communication is today commodified and acts of consumption loaded with moral values.

Composition and orientation This relates simply to how experiential metaphors give meaning to spatial composition. So does the flow chart or diagram have a sense of building upwards, or of moving forwards? Is there a sense of fixed rigidity and immovability or of flexibility? Is there a sense of cyclical or organic processes? Is there a sense of core components where others link or follow? Here we look at how a few of the typical types work, although these often work in combinations. And the point of analysis is not to label here but to observe the work that the design does to create meaning. The point is that composition is something coded.

Centre-margin In Figure 8.2 we see how play is represented at the centre of its components. Of course, where an item is put in the centre this is not neutral. One of

205

206

206

Introduction to Multimodal Analysis

the authors has been at leadership training courses where ‘innovation’ or ‘collaboration’ was put at the centre and then things like ‘solutions’, ‘ideas’, ‘transformation’, ‘smart working’ orbited these. What was ignored were things like staffing levels, high teaching loads, budgeting issues, excessive bureaucracy, which provided the biggest hindrances to things like ‘ideas’ or actual new ways of doing things. So we could argue that ‘time’ or ‘money’ should have been at the centre. In the same way putting play, here shown in a dreamy ‘cloud-like graphic shape, is used to legitimize the codification of processes which is taking place in the ‘quality development’ at the preschool.

Bottom-up Figures  8.4 and 8.6 have vertical orientation. Here an upwards direction carries meanings of aspiration, building, growth, drawing on experiential metaphors. It also allows the diagrams to suggest that what lies at the bottom is the base or foundations. So in Figure  8.6 this is performance culture. Irrespective of resources or the quality of staff, it is these from which everything else is built. In Figure 8.4 we also see that the goals, here a set of values, emanate out of the top of the school, again using vertical directionality metaphorically to suggest rising of standards and improvements. Other preschool diagrams might use the iconography of a tree to suggest growth and flourishing. In Figure  2.4 the shape of the pyramid is used to show building to the focused point of the vision.

Left–right These diagrams often simply represent development over time, particularly in Western cultures which are based on left to right reading flows. A bar chart is a good example. Left to right gives a sense of sequentiality, development or causality, even where it is not overtly explained. And as often in diagrams, time and cause are fused. Going from a to b means (abstracted) time must unfold, but the arrows also claim means–end causality. A is a means for reaching b. Of course this fusing can be used to gloss over that the two may not be the same. For example, a model to show teachers should use student feedback may have boxes to the right in a sequence like ‘defining better learning goals’, ‘design new forms of learning experienced’, ‘development of new forms of assessment oriented to goals’. Such a model can suggest directionality and natural progression of things. It also suggests that the parts can easily flow

207

Diagrams and Flow Charts

and are in themselves unproblematic. The sequentiality over time can also at the same time suggest that there is causality. But such a representation can gloss over issues that prevent such a causality from taking place. Students may not provide this kind of information when asked to provide feedback, which can often be mainly negative. The students may be of hugely varied abilities, which makes simple goal setting and leaning experiences much more complex. As we have looked at earlier in this chapter and in Chapter 1 in regard to Figure 1.4 the Skolinspektionen the goals used in teaching can come from policy and not from pedagogy. Goals in a sense will therefore distort what is taking place rather than provide an improvement. Yet this kind of diagram can represent these as an unproblematic flow.

Cycles Figure 2.1 shows circularity as does the third of the diagrams in Figure 8.1. In the process of work there is therefore no end point. This is a constant cycle of planning documenting, following up reflecting and planning again. The use of the cycle here also helps to suggest something natural, something that takes on its own momentum. An organization might represent its financial year as a cycle, where there are various meetings and budgeting points. Again, this can suggest something established and organic rather than representing this as a left-to-right set of stages.

Pathways Figure 8.3 is typical or a form of orientation where rather than causality we are shown rather a pathway or journey. Causality here is created in part by the signpost ‘your purchase means’. But the dotted line here invites us to follow and see what the purchase does and the changes it can bring about in the world. In such cases we can ask how such a pathway directs us in some ways and not in others. As we noted above such a representation simplifies and idealizes the complex and varied phenomenon that is Fairtrade.

Networks These formations can suggest taxonomies, links and pathways. They can suggest that they show the interactional relationship of parts in a system. These suggest that they represent more complex forms of connectivity. These

207

208

208

Introduction to Multimodal Analysis

can be used, as we saw in the first diagram in Figure 8.1, to suggest pathways, hierarchies and levels. They can show multiple links between different parts. So ‘services’ might be depicted as having lines leading to all parts, but academics only to some parts. These kinds of networks of course suggest patterns of connectivity which can be used to gloss over actual hierarchies. We can question what kinds of connectivities are foregrounded and what are not. Taxonomies can be combined with bottom-up diagrams to suggest a more dynamic and interactive process of reaching goals.

Composition and grouping Important in diagrams are that semiotic resources are combined to group elements, to show that they are of the same or a different order and communicate symbolical meanings. Here colour, typography, graphic shapes, iconography, space and so on are combined so that we encounter them in configurations.

Framing This relates to how elements are separated on grounds of sameness and difference. Typically this is done through combinations of space, frame and colour. The detailed meanings of these are dealt with in different chapters of this book. But here we can look specifically at how this works as part of creating meaning on flow charts and diagrams. In Figure 8.6 we can see that framing here is done in several ways. Each goal has its own column which is distinguished from others by colour. So each of these is presented as different from the others. As we have already suggested, this can be problematic where there may be contradictions between the different columns. We can also see different degrees of framing on this flow chart. Each of the components sits in its own clear frame. But between the stages running vertically in each column we find other frames which serve to classify. Between ‘Internal process’ and ‘stakeholders’ we find only dotted lines creating a boundary. This suggests a weaker boundary so that the components in each section may be related. This is contrasted to the framing of the things in ‘internal processes’ from those in ‘learning & growth’ where we also find a white space. So things like the performance culture and mindsets are represented as something further apart in terms of

209

Diagrams and Flow Charts

classification. This appears to put the attitudes and culture of staff somehow as being apart from the working environment. We see that these are shown to drive the working environment rather than the other way around. We can ask how this might benefit the management. In Figure 2.4 we can see that classification is done by colours which create boundaries between the stages, even though, as we consider below when we look at causality, it is unclear how these can be shown as distinctive stages. Of importance in this design is that the staff and students each have their own face of the pyramid. We are not told in clear terms how the two sides of the pyramid interact. So on one side we find ‘control costs’ and on the other corresponding level ‘fit for purpose IT systems’ and ‘fit for purpose physical infrastructure, supporting an excellent student experience and sense of community’. Do these two parts of the strategy carry any contradictions? Placing them on different faces of the pyramid helps to gloss over such connections being made. In Figure 2.4 we also see the framing which takes place as the pyramid sits in front of the map of the world, communicating ‘international’ or ‘global’. We are not explicitly told how all this is international or how this relates to what staff or students do. But this is symbolized in the way that we saw that the images in Figure 7.7 showing quality development at another university were represented in images.

Symbolization The combination of different semiotic resources for grouping elements can also carry symbolic meanings. If we look at Figure 2.1 we can see that the four components of reaching the national goals – plan, document, follow-up, reflect – are classified as being the same since they are shown in the same type of arrow-shaped boxes within a circle. Therefore, the ‘explanation’ for the coherences here is partly communicated symbolically. Looking at Figure 8.2 we can see that play (‘lek’) has a different graphic shape than the ‘components’ which sit around it – these elements are clearly not of the same order. But these elements are represented as being of the same order, since all of them have the same font and the same shaped box. Such uniformity is also important in suggesting the ‘order’ and technical nature of this process of measuring and quality improvement. Other such diagrams may also use fonts or iconography, for example, related to play and childhood. Documents on the wall in the preschool each used different bright saturated primary colours, yet carried uniform geometric shapes. Here both order and fun are symbolized.

209

210

210

Introduction to Multimodal Analysis

On graphic representations, colour can be used for its iconic value or as an important part of creating classifications, as we mentioned above. Colour is dealt with in detail in Chapter 4 of this book where a toolkit for analysis is provided. But here we can make a few comments on its use specifically in some of our examples. In Figure 2.4 we can see that the different stages of meeting the university vision are classified as different through colour. So the things gathered in one colour on either face of the pyramid are of the same order and different from others. A look at what lies in these areas shows a quite odd collection of things, yet their place in the pyramid. Colour can have an iconic role too. Looking at Figure 8.6 we can see that the bright pastel shades help to communicate simplicity and optimism. In Figure 8.4 we see that the arrows emanating from the roof of the school are saturated primary colours connoting childhood and fun. So while each goal is of a different colour, the quality of that colour helps to classify them of being of the same order. On other diagrams a larger colour palette may be used to indicate variety. So in Figure 8.2 the components of play could be in different colours. Next to this diagram were a series of individual quality improvement targets each presented in a different bright colour. Here there is a sense of difference. But if all were of the same level of saturation and purity this would classify them as the same at a different level.

Causality Graphic representations often include elements having effects on each other. Typical expression form tends to include arrows between elements and also linguistic labels. Other diagrams may not represent causality but rather systems of relations. In all such cases we can ask how such representations seek to re-contextualize processes and things for specific interests, simply where causalities are concealed, foregrounded, replaced or added.

Directional flow Causality can comprise a set of boxes moving from left to right which have arrow forms to the right side. Other arrows may provide links backwards for things like reflection or repeat. It may take the form of a line which weaves

211

Diagrams and Flow Charts

around a little where stages are marked to suggest more of a journey. This may resemble the moves on a children’s boardgame. In all cases it is important to consider what kinds of causalities are suggested and how they are symbolized. For example, if we know that academic staff produce less research when they have heavy teaching and admin loads, we can look at how a flow chart includes or excludes such things when showing how ‘research excellence’ is achieved such as in Figure 8.6. What we see in this case is that there is no room on this diagram for the two to interact, to show where there may be overlaps or conflicts. Such flow charts and the way they divide up and codify process can then relate to how these processes become managed and administrated. So all of the goals across the top of Figure 8.6 become managed and administrated by distinctive pathways. Each may work independent of the others. We can also think about the logic of represented causalities. If we look closely at Figure  2.4 it is not clear how the things listed could be seen as successive steps in a strategy. For example, ‘control costs’ sits in the foundations for the student face of the pyramid as well as ‘environmental sustainability’. One might rather think these are in themselves goals rather than foundations. And then it is not clear how these should lead to ‘student engagement’, nor then to ‘excellent service and support’ above in ‘student aims’. And causalities can be present, but what is actually causing that to happen is not clear. And one affordance of diagrams is just this. In Figure 2.4 on the pyramid we can see that each layer has three ‘arrow head’ shapes above it indicating causality to the layer above. But if, for example, we look at the things listed in foundations for students it is not clear how ‘control costs’ has a causal relationship to any of the things in the layer above such as ‘personal responsibility for own development’. Nor does it appear to be a problem to have five things in the bottom layer and only three in the layer above. Do all five of these cause the three things above? Such symbolization of causalities has the advantage that such specifications are not necessary. Where causalities are given in pathway flow charts such as we see in Figure  8.3 it is important to think about what is included and what is excluded. We may be given alternative paths as if ‘these are the choices’. But we may feel there are other paths that are simply not included. For example, in the Fairtrade pathway, there is no inclusion of the way that ‘democracy’, ‘education’, ‘healthy’ may be deeply affected by the pressure of IMF restructuring loans. This can have a hugely destabilizing effect on countries. The role of the branding of the company who is selling the

211

212

212

Introduction to Multimodal Analysis

product is missing here. This pathway presents a clean flow from the act of shopping to transforming all sorts of things in the world. In Figure  8.2 we do not find clear causalities. And where none are presented we can ask why. It is not clear in this diagram how any of the components which orbit LEK (‘play’) interrelate or effect each other. We do not see how ‘känslor’ (‘feelings’) effect ‘learning’, for example. These are simply presented as components of play. In this diagram, arrows of the nature seen in Figure  8.5 could have been added to suggest that play somehow rather influences these things, or to show how something like how a child feels or how confident they are in motor skills could influence something like social interaction. But here the aim is to simply show that all these things are being measured independently. As such causalities and links are suppressed.

Temporality Graphic representation is often about development of some kind. Typical expression forms are arrows between elements, a timeline or a coding with verbs having duration in time. As we have seen language can be important here. Infinitives can be used to suggest ongoing processes of activity, of developing, of growing. Temporality can relate to causality. For example, we can have a basic left to right sequences which suggests processes taking place sequentially. The circle on the preschool diagram in Figure 2.1 represents a temporal process through a cycle. An arrow growing from left to right and slightly rising and increasing in weight can be used to suggest that over time things are improving. In Figure 8.2 we see temporality in the form of the dotted line which we are invited to follow. There is a sense here of a story where we make our purchase and then see the changes that happen in the world. We can also find inferred temporality which can be seen in the diagram of the school in Figure 8.4. This diagram invites a bottom-up reading, which is typical for diagrams that want to represent increased outputs and neverending success. The arrows which are the goals for the characteristics of the children at the school flow outwards and upwards:  ‘social’ (‘sociala’), ‘responsible’ (‘ansvarsfulla’), ‘full of knowledge’ (‘fulla av kunskap’), ‘communicative’ (‘kommunikativa’). Below the school we find the foundations which are ‘joy’ (‘glädje’), ‘security’ (‘trygghet’) and ‘orderliness’

213

Diagrams and Flow Charts

Figure 8.4  Diagram showing output for a school, Lillåns Södra Skola. Image from Örebro Municipality website. Available at:  https://www.orebro.se/ faktaomskolan/skolansvision.4.143ed3c11ba280c71e800094440.html.

Figure  8.5  Template for SMART goals. Image from ‘Working on Weekly Class Smart Goals’ by Stephanie Van Horn, 3rd Grade Thoughts Blog, 7 November 2014.

213

214

214

Introduction to Multimodal Analysis

(‘ordning och reda’). Here a temporality and causality can be inferred that it is through the school that children can become what are represented as the goals. Of interest here is that the goals are represented as nominalizations, so as noun forms of verb processes. So they are communicative. But what do they communicate about? They are responsible, but about what? One of the authors had a child in a school in Sweden which had these kinds of values and goals. Yet, as with many schools in the country, it had been recently privatized and was run not by a head teacher but by a professional manager who had formerly worked in an entirely unrelated kind of organization. This created many problems for the teachers as he did not really understand their jobs. And money became the centre of all things. The kinds of quality development systems we have looked at in this chapter were implemented. At the same time such schools produce these strategic plans. The diagram we see here will relate to other documents which will detail the strategies. These will be implemented by the kinds of measuring systems we saw in Figures 1.4 and 2.1. So we see that ‘social’ and ‘responsible’ (seen as social competencies and feelings in the play diagram) can be coded and quality improved. Things like ‘joy’ here shown as an adjective will too become coded, displayed and quality improved. In such cases the uses of wider colour palettes typical for representing childhood help to remind us that all this is about ‘fun’ and ‘innocence’. Another diagram designed to improve productivity and outcomes is seen in Figure 8.5. Here we see ‘smart goals’ to be used by children to organize their learning. Again we see a one-size-fits all template. So all learning can fit into this template. It also emphasizes measurement and coding since all learning must have evidence. Here the meaning of the template is also communicated in part by the word ‘smart’ and the use of its letters to provide the list of stages. Here the word smart and its letters have an iconic role in suggesting ‘breaking things down in components in a smart way’. At the time of writing, the term ‘smart’ was highly popular in management communication. This breaking down, as with the case of ‘play’ in Figure 8.2, suggests therefore that here we reveal the basic components of all learning. One of the authors has seen his eleven-year-old daughter use such templates for all parts of her schooling. The SMART goal template was developed for sale and project management in the private sector as a tool that could assess the efficiency and profitability of a business. But nowadays, and already in primary school, it has become a self-management tool for children where they set up measurable goals to organize their learning. This, together with the other examples we have used

Figure 8.6  Strategic diagram for Newcastle University. Image from the NEW DIRECTIONS Strategic Plan 2013–15, The University of Newcastle, Australia. Available at:  (https://www.newcastle.edu.au/__data/assets/pdf_file/0004/33538/NeW-DirectionsStrategicPlan.pdf).

215

216

216

Introduction to Multimodal Analysis

in this chapter, clearly illustrates the new and commodified communication we talked about in Chapter 1 and the integrated designs deployed in it. It also points to how what Habermas (1987) calls communicative intentionality risks to be undermined. The SMART management model used also in primary school certainly relies on an instrumental rationality and processes and measures all learning in the same way regardless of actual contexts and individuals. Following Habermas we would say that the model might be okay as long as it interplays with communicative rationality, with teachers and children’s perceptions of and interactions about what is important or desirable. But if it loses its connection to communicative rationality, it risks imposing merely ‘violent abstractions’ and measuring everything and everyone in ways that are basically incomprehensible and at odds with what those measured believe is true. These new forms of entrepreneurial control risk to make teachers and children ‘mere nuts and bolts on some distant production line’, as Ball (2003: 216) puts it.

Summary text box for diagrams Classification and writing:  we can ask what kinds of paradigms or categories are made in writing. Important here is to look at the extent to which these might be considered reasonable or ‘true’ paradigms and categories. We also ask what is included, excluded and consider how grammatical forms can work as classifications. Visual classifications:  people, things and processes can be classified visually. Here we can look at representation through graphic shapes, how these symbolize qualities, similarities and differences. Composition and orientation:  we find typical configurations of elements in diagrams and flow charts which come in different combinations. Here we can think about what kinds of structures, flows, directionality and connections these suggest. Composition and grouping: here we need to consider how elements are separated and grouped in terms of sameness or difference. We look at the use of framing and how sameness or difference is communicated through symbolization.

217

Diagrams and Flow Charts

Causality and temporality:  diagrams and flow charts can com­ municate how people and things are linked and how processes take place over time. Since this is done symbolically, we can ask how these are symbolized and what kinds of causalities and temporalities are foregrounded and backgrounded, present or omitted.

Student activity These suggested activities aim to increase your understanding of how flow charts are not simply tools to help show how processes happen, but that they also shape and manipulate how those appear to us. 1. On the internet you can find different templates for flow charts – networks, cycles and so on. Choose three of these. Test out putting different parts of your institution, levels of your sports club or your group of friends into the different boxes. Try different ones of these. Explain the kinds of meanings, processes and relationships they suggest in each case. Do some of them seem to work better in different cases? Do some of them conceal the nature of the actual relationships between people? 2. Do an internet search for flow charts. Do this in relation to a topic with which you are familiar. It could be sport, climate change, health care, education, fitness. Carry out an analysis of one of these. What kind of flow chart is this and how does this shape the meaning of the process represented? How are the stages or participants and the causalities represented? Are there any such things that are perhaps absent from the chart that you might expect to be included? 3. Create a simple flow chart of your own. This can be done in word or simply drawn on paper and can take the form of a simple process chart. Choose a process with which you are familiar. It could involve studying, your job, a social occasion and so on. Decide the following:  Which graphical shapes will you use and should these be the same or different for different elements? What kind of forms will you use to show causalities (dotted lines, heavy arrows, etc.)? Importantly, one of the goals in this design is to conceal or delete one key part, stage or participant in the process.

217

218

219

Bibliography

Abousnnouga, G. and Machin, D. (2013) The Language of War Monuments, London: Bloomsbury. Aiello, G. and Dickinson, G. (2014) Beyond authenticity: A visual-material analysis of locality in the global redesign of Starbucks stories. Visual Communication 13(3): 303–21. Aries, P. (1962) Centuries of Childhood, New York: Vintage Books. Ball, A. (1993) The teacher’s soul and the terrors of performativity. Journal of Education Policy 18(2): 215–28. Barthes, R. (1973) Mythologies, London: Fontana. Barthes, R. (1977) Image-Music-Text, London: Fontana. Bateman, J. A., Wildfeuer, J. and Hiippala, T. (2017) Multimodality: Foundations, Research and Analysis: A Problem-Oriented Introduction, Berlin: De Gruyter Mouton. Bauman, Z. (2007) Consuming Life, London: Polity Press. Bauman, Z. (2017) Retropia, London: Polity Press. Bernstein, B. (2000) Pedagogy, Symbolic Control, and Identity: Theory, Research, Critique, Lanham, MD: Rowman & Littlefield. Bettelheim, B. (1979) The Uses of Enchantment, Harmondsworth: Penguin. Billig, M. (1995) Banal Nationalism, London: Sage. Bourdieu, P. (1991) Language and Symbolic Power, Cambridge: Polity Press. Bowker, G. C. and Star, S. L. (1999) Sorting Things Out: Classification and Its Consequences. Cambridge, MA: MIT Press. Breazu, P. and Machin, D. (2018). A critical multimodal analysis of the Romanian press coverage of camp evictions and deportations of the Roma migrants from France. Discourse & Communication 12(4): 1–18. Calhoun, C. (ed.) (1992) Habermas and the Public Sphere. Cambridge, MA: MIT Press. Cliath, A. G. (2007) Seeing shades – Ecological and socially just labeling. Organization and Environment 20(4): 413–39. Djonov, E. and van Leeuwen, T. (2011) The semiotics of texture: From haptic to visual. Visual Communication 10(4): 541–64. Eco, U. (1985) How culture conditions the colours we see, in Blonsky, M. (ed.) On Signs, Baltimore, MD: John Hopkins, pp. 157–75. Eisner, W. (1985) Comics and Sequential Art, Tamarac, FL: Poorhouse Press.

220

220

Bibliography

Eriksson, G. (2015) Ridicule as a strategy for the recontextualisation of the working class: A multimodal analysis of class-making on Swedish reality television. Critical Discourse Studies 12(1): 20–38. Fairclough, N. (1992) Discourse and Social Change, Cambridge: Polity Press. Foucault, M. (1977) The Archeology of Knowledge, London: Tavistock. Foucault, M. (1980) Power/Knowledge: Selected Interviews & Other Writings 1972–1977, ed. Colin Gordon, New York: Pantheon Books. Freud, S. (1976) The Interpretation of Dreams, Middlesex: Penguin Books. Gage, J. (1993) Colour and Culture – Practice and Meaning from Antiquity to Abstraction, London: Thames and Hudson. Gage, J. (1999) Colour and Meaning – Art, Science and Symbolism, London: Thames and Hudson. Gibson, J. J. (1979) The Ecological Approach to Visual Perception, Boston: Houghton Mifflin. Goethe, J. W. von (1970 [1810]) Theory of Colours, Cambridge, MA: MIT Press. Habermas, J. (1984) The Theory of Communicative Action, Vol. 1: Reason and the Rationalization of Society, Cambridge: Polity Press. Halliday, M. A. K. (1961) Categories of the theory of grammar. Word 17(3): 241–92. Halliday, M. A. K. (1978) Language as a Social Semiotic System, London: Edward Arnold. Halliday, M. A. K. (1985) An Introduction to Functional Grammar, London: Edward Arnold. Halliday, M. A. K. and Hasan, R. (1989) Language, Context, and Text: Aspects of Language in a Social-Semiotic Perspective, Oxford: Oxford University Press. Hartley, J. (1982) Understanding News, London: Routledge. Hjelmslev, L. (1961) Prolegomena to a Theory of Language, Madison: University of Wisconsin Press. Hobsbawm, E. and Ranger, T. (eds) (1992) The Invention of Tradition, Cambridge: Cambridge University Press. Hodge, R. and Kress, G. (1988) Social Semiotics, Ithaca, NY: Cornell University Press. Hopmann, S. (2008) No child, no school, no state left behind: Schooling in the age of accountability. Journal of Curriculum Studies 40(4): 417–56. Horn, R. E. (1998) Visual Language: Global Communication in the 21st Century, Washington, Bainbridge Island: MacroVU. Jakobson, R. and Halle, M. (1956) Fundamentals in Language, The Hague: Mouton. Jansson, A. and Parding, K. (2011) Changed governance of public sector organizations = Challenged conditions for intra-professional relations? The International Journal of Public Sector Management 24(3): 177–86.

221

Bibliography

Jewitt, C. and Oyama, R. (2000) Visual meaning: A social semiotic approach, in Jewitt, C. and Van Leeuwen, T. (eds), Handbook of Visual Analysis, London: Sage, pp. 134–56. Jewitt, C., Bezemer, J. and O’Halloran, K. (2016) Introducing Multimodality, Milton Park, Abingdon, Oxon: Routledge. Kandinsky, W. (1977 [1914]) Concerning the Spiritual in Art, New York: Dover. Kress, G. and Hodge, B. (1979) Language as Ideology, London: Routledge. Kress, G. and Van Leeuwen, T. (1996) Reading Images: The Grammar of Visual Design, London: Routledge. Kress, G. and Van Leeuwen, T. (2001) Multimodal Discourse: The Modes and Media of Contemporary Communication, London: Arnold. Kress, G. and Van Leeuwen, T. (2002) Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication 1(3): 343–68. Lakoff, G. and Johnson, M. (1980) Metaphors We Live by, Chicago: University of Chicago Press. Ledin, P. and Machin, D. (2016) The evolution of performance management discourse in corporate strategy diagrams for public institutions. Discourse, Context & Media 13(B): 122–31. Ledin, P. and Machin, D. (2018) Doing Visual Analysis: From Theory to Practice, London: Sage. Low, W. and Davenport, E. (2005) Has the medium (roast) become the message: The ethics of marketing fair trade in the mainstream. International Marketing Review 22(5): 494–511. Lupton, E. (2004) Thinking with Type: A Critical Guide for Designers, Writers, Editors and Students, New York: Princeton Architectural Press. Machin, D. (2004) Building the world’s visual language: The increasing global importance of image banks in corporate media. Visual Communication 3(3), 316–36. Machin, D. and Thornborrow, J. (2003) Branding and discourse: The case of Cosmopolitan. Discourse and Society 14(4): 453–71. Machin, D. and Mayr, A. (2012) How to Do Critical Discourse Analysis, London: Sage. Machin, D. and Polzer, L. (2015) Visual Journalism, London: Palgrave. Mahon, J. (1976) Harry Pollitt: A Biography, London: Lawrence and Wishart. McCloud, S. (1994) Understanding Comics: The Invisible Art, New York: HarperCollins. Miliband, R. (1973) Parliamentary Socialism, London: The Merlin Press. Parding, K., Abrahamsson, L. and Jansson, A. (2012) New conditions for identities, cultures and governance of welfare sector professionals: The teaching profession. Ephemera 12(3): 294–308. Peirce, C. S. (1984) Writings of Charles S Peirce: A Chronological Edition. Vol. 2 1867–1871. Bloomington: Indiana University Press.

221

222

222

Bibliography

Power, M. (1999) The Audit Society: Rituals of Verification, Milton Keynes: Open University Press. Poynter Institute (1991) ‘Eyes on the news’, Accessed 18 April 2008, http:// poynterextra.org/cp/colorproject/color.html. Saad-Filho, A. and Johnston, D. (2005) Neoliberalism: A Critical Reader, London: Polity Press. Said, E. (1981) Covering Islam: How the Media and Experts Determine How We See the Rest of the World, New York: Pantheon. Schneider, T. and Davis, T. (2010) Fostering a hunger for health: Food the the self in ‘The Australian Women’s Weekly’. Health Sociology Review 19(3): 285–303. Schuldt, J.P (2013) Does green mean healthy? Nutrition label color affects perections of healthiness. Health Communication 28(8): 814–21. Sontag, S. (2004) Regarding the Pain of Others, New York: Farrar, Straus & Giroux. Stubbs, T. H. et al. (2016) The Impact of IMF conditionality on Government Health Expenditure: A cross-national analysis of 16 west African nations. Social Science & Medicine 174: 220–7. Van Dijk, T. A. (1991) Racism and the Press, London: Routledge. Van Dijk, T. A. (2008) Discourse and Context: A Sociocognitive Approach. Cambridge, New York: Cambridge University Press. Van Leeuwen, T. (1996) The representation of social actors, in Coulthard, C. R. and Coulthard, M. (eds), Texts and Practices, New York: Routledge, pp. 32–70. Van Leeuwen, T. (2001) Semiotics and iconography, in Van Leeuwen, T. and Jewitt, C. (eds), Handbook of Visual Analysis, London: Sage, pp. 1–9. Van Leeuwen, T. (2005) Introduction to Social Semiotics, London: Routledge. Van Leeuwen, T. (2006) Towards a semiotics of typography. Information Design Journal + Document Design 14(2): 139–55. Van Leeuwen, T. (2008a) News forms of writing, new visual competences. Visual Studies 32(2): 130–5. Van Leeuwen, T. (2008b) Discourse and Practice, London: Routledge. Van Wagener, A. (2003) ‘Today’s headline styles’, accessed 18 April 2008, http:// poynter.org/column.asp?id=47&aid=77003. Varul, M. Z. (2008) Consuming the Campesino: Fair Trade marketing between recognition and romantic commodification. Cultural Studies 22(5): 654–79. Ventola, E., Cassily, C. and Kaltenbacher, M. (2003) Perspectives in Multimodality. Document Design Companion Series, Amsterdam: John Benjamins.

223

Bibliography

Voloshinov, V. N. (1986) Marxism and the Philosophy of Language, Cambridge, MA: Harvard University Press. Wagner, K. (2015) Reading packages: Social semiotics on the shelf. Visual Communication 14(2): 193–220. Walker, M. (1991) The Power of Color, New York: Avery Publishing Group. Worth, S. (1981) Pictures can’t say ain’t, in Worth, S. and Gross, L. (eds), Studying Visual Communication, Philadelphia: University of Pennsylvania, pp. 85–108.

223

224

225

Image index

Below we account for the images in the book and specify which analytical tool is used on which page. # 0.1

Image

Analytical tool

Page

General multimodal design

1–3, 5f., 9f., 18, 37

Images: connotation

44

Images: settings

46

Images: participants

48, 50f., 53

Modality: scales

69, 72f.

Modality: perspective

82, 84

Colour: semiotic system 90f., 96 Colour: values

101f., 104

Typography: semiotic system

121

Typography: meaning potential

125, 127, 133

Composition: salience

171, 174, 177f.

226

226

Image index

# 1.1

Image

Analytical tool

Page

General multimodal design

13, 14f., 38

Modality: scales

67

Colour: values

99, 105f.

Typography: semiotic system

117

Texture: semiotic system 144 Composition: framing

182

1.2

General multimodal design

16–18

1.3

General multimodal design

22f., 38

Colour: values

103, 106f.

Typography: semiotic system

117, 120f.

Typography: meaning potential

125, 127, 129f.

Texture: semiotic system 142, 148

1.4

Texture: dimensions

152

Texture: materials

161

Composition: salience

178

Composition: framing

180, 184

General multimodal design

26–9, 196

Diagrams: composition

207, 214

227

Image index

# 2.1

2.2

2.3

2.4

Image

Analytical tool

Page

Images: denotation

39

Images: connotation

41–3

Images: settings

45–6

Images: participants

53

Images: processes

57

Modality: scales

68f.

Modality: perspective

85

General multimodal design

195f.

Diagrams: classification

198f., 203

Diagrams: composition

207, 209, 212, 214

Images: connotation

44f.

Diagrams: classification

203

Images: participants

54

Composition: salience

176

General multimodal design

38

Images: participants

49–50, 54

Diagrams: classification

201, 203, 205

Diagrams: composition

206, 209f., 211

227

228

228

Image index

#

Analytical tool

Page

3.1

Modality: scales

69–70

3.2

Modality: scales

66f.

Modality: perspective

68

3.3

Modality: scales

70, 73

3.4

Modality: scales

70, 73

Kinds of modality

75, 78

Modality: scales

68–9, 72

Composition: framing

182

3.5

4.1

Image

Colour: semiotic system 85, 92f. Colour: values

99, 100, 104

Typography: semiotic system

123

Composition: salience

171, 173

Composition: framing

184

229

Image index

# 4.2

Image

Analytical tool

Page

Images: connotation

43–4

Images: settings

46

Images: participants

48, 50f., 53

Images: processes

57

Modality: perspective

80

Colour: semiotic system 95 Colour: values

99, 102, 105

Texture: semiotic system 142f. Texture: dimensions

153

Texture: materials

158, 160

4.3

Colour: values

95

5.7

Modality: scales

69, 71f.

General multimodal design

87, 111, 116

Colour: semiotic system 90f. Colour: values

99, 100f., 104, 105

Typography: semiotic system

117

Typography: meaning potential

125f., 131

Composition: salience

171, 174, 177

Composition: framing

182

229

230

230

Image index

# 5.8

5.9

Image

Analytical tool

Page

General multimodal design

87, 111

Colour: values

100f., 104f., 105

Typography: meaning potential

125

Composition: salience

171, 177

General multimodal design

37, 139

Modality: scales

67, 70, 73

Modality: perspective

83

Colour: semiotic system 90f. Colour: values

99, 101

Typography: semiotic system

122

Typography: meaning potential

125f., 131, 134

Texture: semiotic system 144 Composition: salience

178

Composition: framing

185

5.10

Colour: values

100, 102, 106

5.11

General multimodal design

39

231

Image index

# 5.12

6.1

Image

Analytical tool

Page

Kinds of modality

78

Typography: meaning potential

130

Texture: dimensions

154

Composition: salience

174

Composition: framing

184

Texture: semiotic system 140, 142 Texture: dimensions

6.2

6.3

151, 153

Texture: semiotic system 140, 142, 147 Texture: dimensions

151, 153

Texture: materials

158, 162

Colour: semiotic system 95, 96 Texture: semiotic system 145, 147

6.4

Texture: dimensions

151, 153

Texture: materials

158, 162

Colour: semiotic system 95 Texture: semiotic system 143, 147

6.5

Texture: dimensions

151, 153

Texture: materials

160, 162

Texture: semiotic system 145–6 Texture: materials

158f., 160, 162

231

232

232

Image index

#

Analytical tool

Page

6.6

Texture: dimensions

149

6.7

Texture: dimensions

151, 154f.

Texture: materials

161

6.8

Texture: dimensions

156

7.1

Modality: scales

73

General multimodal design

167–70

Composition: salience

170f., 177f.

Composition: framing

181

Modality: perspective

81, 84

Composition: salience

172, 174, 177

Composition: framing

181

Modality: scales

67, 67f., 71

7.2

7.3

Image

233

Image index

# 7.4

7.5

7.6

7.7

8.1

8.2

Image

Analytical tool

Page

Modality: scales

71

Composition: salience

171, 176f.

Modality: scales

71f.

Modality: perspective

84

Composition: salience

174

Composition: framing

180, 182

Modality: scales

71f.

Composition: salience

176f.

Composition: framing

181f., 185

Modality: scales

67

Modality: perspective

83

Composition: salience

171, 173, 176

Diagrams: classification

202

Diagrams: composition

209

General multimodal design

193–4

Diagrams: classification

198, 203

Diagrams: composition

207f.

General multimodal design

196

Diagrams: classification

198, 203

Diagrams: composition

205, 209f., 212, 214

233

234

234

Image index

#

Analytical tool

Page

Diagrams: classification

205

Diagrams: composition

207, 211

8.4

Diagrams: composition

206, 210, 212

8.5

Diagrams: composition

212, 214

8.6

Diagrams: classification

200, 202

Diagrams: composition

206, 208, 210f.

8.3

Image

235

Index

Abousnnouga, G. 149 abstract/technical modality 78–9 abstraction 74 actions, in images 55–6 emotional processes and 56 material processes and 57–8 mental processes and 56–7 verbal processes and 57 agency and action 55–6 Aiello, G. 149 aluminium 162 angles 83 horizontal 83, 85 oblique 84, 85 vertical 83, 85 angularity 202 significance of 130, 131 anonymization, of pictures and images 52–3, 55

BNP poster 168 composition in 168–9 salience in 177, 178 bold font 124–6 Bounce package 22–3, 24, 25, 28, 29, 38 colour coherence in 92 fonts in 117, 120, 121, 125, 127, 129, 130, 134, 135 framing in 180 hue in 107 hybrid colours in 103 rhyming principle in 184 texture in 142, 148, 152–3, 161 boundaries, significance of 180 brick 160 brightness 98–9 communicating truth and transparency 99 bronze 162

background, articulation degrees of 69–70, 74 Ball, A. 216 Bargain Booze 128 colour saturation in 100 disconnection on fonts in 132, 133 fonts in 126, 129 hue in 106 pure colours in 102 Barthes, R. 5, 13, 16, 38, 39, 40, 45 Bauman, Z. 29, 143 Bernstein, B. 29 Bettelheim, B. 47 biological categorization 51–2, 54

carton/card 161, 164 categorization, of pictures and images 51–2, 55 causality 210, 214, 217 and directional flow 210–12 caveman model 143 centre-margin diagrams 205–7 children’s sports camp website 91, 123 angle of interaction in 83 background in 70 brightness in 90, 99 colour differentiation in 73 colour saturation in 73, 101 contrast principle in 185

236

236

Index detail articulation and 66 disconnection on fonts in 132 fonts in 122, 125–6, 127, 131, 132, 134 salience in 178 texture in 144, 152 choice in communication 13–14 inventory of 16 system of 15 Cinderella film poster 87, 90, 107, 119 colour differentiation in 105 colour saturation in 100 disconnection on fonts in 132 fonts in 117, 120, 125, 126, 131, 133 hue in 106 pure colours in 101 salience in 171–2, 177 typographic features in 135 unmodulated colours in 104 classification 197–8 and graphic elements size and weight 203–5 and graphic shapes 202–3 and imperatives 199 importance of 29 and infinitives 199–200 and people 200–201 and processes 199 visual 201–2, 216 and writing 198–9, 216 Closer magazine 79 overlapping principle in 182 close shots, significance of 84–5 codification 22 and commodification 24–5 coding orientations abstract/technical modality 78–9 naturalistic modality 77 sensory modality 77–8 codings identification of 33–4 of semiotic resources 31–3

collectivization 48–9 colour 31, 33, 87–8 brightness 98–9, 107 classification through 210 in coherence creation 91–6 communicating attitudes 90–1 communicating ideas 89–90 cultural associations of 98 differentiation 73, 74, 104–5, 107 dimensions of 98–107 distinctive features of 98 fluorescence 106 hue 106–7 hybrid 102–3 iconic role of 210 in integration creation 92, 95 luminosity 106 modulation 72, 74, 103–4, 107 patterns in 88–9 purity 101–3, 107 and salience 173–4 saturation 72–3, 74, 78, 99–101, 107 semiotics of 98 as semiotic system 89–96 value of 96–7 Colour and Culture (Gage) 88 Colour and Meaning (Gage) 88 commodified communication, for moral standards 24–30 communicative rationality 25–6, 28, 29, 31, 95, 216 commutation test 34, 35 composition 167–70 and framing 178–9 contrast 185, 186 integration 182, 186 overlap 182–4, 186 rhyme 184–5, 186 segregation 180, 185 separation 181–2, 185 images and 186

237

Index photographs, hierarchies, and rhyming 186–8 photographs and coordination 188–9 and salience 170 colour 173–4 focus 176–7 foregrounding 177 overlapping 178 potent cultural symbols 170–1 size 171–174 tone 174–6 and textures 144–9 composition and grouping 208, 216 framing and 208–9 symbolization and 209–10 composition and orientation 205, 216 centre-margin 205–7 cycles 207 networks 207–8 pathways 207 concrete 160, 164 configuration principle and codings 33, 35 conflicts, news coverage of 51 connectivity 184, 185, 208 in font 131–2, 136 and framing 179 connotation 13–14, 18–19, 134, 40–1, 104 carriers of 42–7 of colour 89–91 luminosity and 106 of naturalness with texture 79 object meaning in images and 43–5 rules of 42 settings and 45–7 visual depiction of people and 47 contextualization principle and codings 32, 35 contrast and framing 185, 186 coordination 188–9, 190 critical discourse analysis 26 action and 55 CrossFit gym 140–3

comparison with high street gym 143, 151 cultural categorization 51, 52, 54 cultural salience 174 cultural symbols, potent 170–1 curvature 202 in font 129–31, 136 cycle diagrams 207 decontextualized images 46–7, 69, 70, 84, 195 demand image 81 denotation 39–40 and pictures and images 39–40 rules of 42 depth, articulation of 70, 74 design culture, shift in 3 design decisions, as aesthetic choices 3 design features, social uses of 7–10 detail, articulation degrees of 66–9, 74 diagonal line 113 diagrams and flowcharts 193–7 and causality 210, 214, 217 directional flow 210–12 classification 197–8 graphic elements size and weight and 203–5 graphic shapes and 202–3 imperatives and 199 infinitives and 199–200 people and 200–201 processes and 199 writing and 198–9, 216 visual 201–2, 216 and composition and grouping 208, 216 framing 208–9 symbolization 209–10 and composition and orientation 205, 216 centre-margin 205–7 cycles 207

237

238

238

Index networks 207–8 pathways 207 and temporality 212–16 Dickinson, G. 149 directional flow 210–12 discourse 18–20, 30 monuments as 19 and social practices 20 technologization of 21–2 distance 84, 85–6 and pictures and images 50–1, 55 distinction principle and codings 31–2, 34–5 Djonov, E. 149 Eco, U. 88 Eisner, W. 182 emotional processes 56, 58 empty spaces 146–7 engagement and frontal perspective direct address and 81–2 potential address and 82 equality, sense of 176, 177, 178 Escher, M. C. 70 expansion, in font 126–7, 136 eyes, truth of. See naturalistic modality Fairclough, N. 21, 24 fairtrade packaging 2, 3, 4, 9, 10, 18, 20 action and agency in 55–6 articulation degrees of background in 69 Barthes on 38–9 categorization in 51 closer shots in 84 colour coherence in 91–2, 95 colour modulation in 72, 104 colour saturation in 72–3, 101 connotations and 43–4, 46 emotional processes and 56 fonts in 121, 125, 127, 133 hybrid colours in 102–3 individualization and 48

intimacy and 50 monochrome in 73, 90, 104, 105 non-frontal perspective in 82–3 non-representation in 53 pathway flow chart for 204, 211 salience in 171, 174, 177, 178 significance of 37–8 silence in 57 tone articulation in 72 feelings, truth of. See sensory modality Fit kitchen package 133 fonts in 130 rhyming principle in 184 salience in 174 textures in 154 flourishes, in font 134–5, 136 fluorescence 106 focus and salience 176–7 fonts angular 130 choices of 117 curved 114 disconnected 132 horizontal 133 importance of 14, 16 as important vehicle of communication 114 irregular 134 printed 122, 127, 129, 131 regular 134 rhyming principle and 184 rounder 130, 131 sameness/difference between things and 121–3 vertical 133 foregrounding and salience 177 Foucault, M. 19 framing 50, 178–9 composition and grouping and 208–9 contrast and 185, 186 integration and 182, 186 overlap and 182–4, 186

239

Index rhyme and 184–5, 186 segregation and 180, 185 separation and 181–2, 185 free play 26, 27

horizontal line 112 horizontal orientation 133 Horn, R. E. 112 hue 106–7

Gage, J. 88, 89, 96 Gears flow chart template 45 genericity 64 close shot and 84 collectivization and 49 colour 103 composition and page layout and 172 diagram and 75 and flowcharts and 201 of images 49, 50, 52, 53, 57 modulation and 72 representation of 66 and specific details compared 66 technical modality and 78 gesture and posture 56–7 glass 159 GNC drinks bottles 151–2, 154, 155, 161 Goethe, J. W. von 97 granite 163–4 graphic elements size and weight 203–5 graphic shapes and classification 202–3

ideational metafunction 8 IKEA 58 catalogue 66, 67, 70, 80 modality in 73, 78 textures in 139, 161 illumination degrees. See light and shadow, articulation of images. See also pictures and images as document 62 modification of 62 perspective in 80–1 frontal perspective and engagement 81–2 non-frontal perspective 82–5 significance of 61 imperatives 199 impressionist painters and sensory truth 77 indexical signs 56 individualization 48, 49, 201 closeness and intimacy and 50–1 individuals and groups, of pictures and images 48–50, 54–5 infinitives 199–200, 212 instrumental rationality 28 integrated design 22–4, 30 textures in 147–8 integration and framing 182, 186 intellect, truth of. See abstract/technical modality interior design, textures and 145, 146 internationalization 188 and sustainability 173–4 interpersonal metafunction 8 inventory 33–4, 35 of choices 16 iron 162

Habermas, J. 25, 26, 28, 29, 95, 96, 216 Halliday, M. A. K. 7–8, 13, 16, 63, 89, 117, 140 handwriting and print, meaning potentials of 129 Hartley, J. 82 Heritage Bicycles Café, interior of 17 hierarchies 189 rhyme and photographs and 186–8 high street gym 141, 142 comparison with CrossFit gym 143, 151 Hobsbawm, E. 19 Hodge, R. 65 horizontal angles 83, 85

239

240

240

Index irregular fonts 134 irregular textures 152 significance of 142 isometric perspective 70

Kandinsky, W. 97 Kress, G. 6, 7, 8, 9, 21, 65, 70, 73, 75, 88, 89, 97, 98, 101, 117

light and shadow, articulation of 71, 74 linguistics, of Halliday 7–8, 16 liquidity, in textures 155, 157 L’Oréal advertisement 172, 184 close shots in 84 direct address in 81 focus in 177 foregrounding in 177 overlap in 184 and salience 174 tone in 175 luminosity 106, 107

language 9 collectivization and 48 colour naming in 31 communicative functions of 89, 117 communicative rationality and 95, 96 direct address in 82 importance of 212 integration and 23 meaning making and 6 metafunctions 8 metaphorical associations in 98 modal auxiliaries and 63 non-representation in 53 social semiotic model of 16 social use of 7 as system of choice 7, 8, 15 technologization and 21–2 visual 26, 28 Ledin, P. 149 letterforms, features of 124 connectivity 131–2, 136 curvature 129–31, 136 expansion 126–7, 136 flourishes 134–5, 136 orientation 132–3, 136 regularity 133–4, 136 slope 127–9, 136 weight 124–6, 136 lifestyle magazine 82, 185

Machin, D. 47, 48, 149 McCloud, S. 112, 113 marble 163 material affordances 16–18, 30 material processes 57–8 materials and textures 157–8 brick 160 carton/card 161, 164 concrete 160, 164 glass 159 metal 162, 164 paper 161, 164 plastics 161–2, 164 stone 163–4 wood 158–9, 164 Mayr, A. 48 meaning making 31, 40 language and 6 sources for 98 mental processes 56–7, 58 metafunctions, types of 8 metal 162, 164 modal auxiliaries 63 modality 61–3; See also images; pictures and images abstract/technical 78–9 of heart representation 75–7 high 66, 67–8, 74, 75 as interpersonal 65

Jurnalul National (Romanian newspaper) 53 focus and salience in 176

241

Index as linguistic concept, origins of 63–5 low 65, 66, 67, 70, 75, 103 markers, and scales articulation degrees of background 69–70, 74 articulation degrees of detail 66–9, 74 articulation of depth 70, 74 articulation of light and shadow 71, 74 articulation of tone degrees 71–2, 74 colour differentiation 73, 74 colour modulation degrees 72, 74 colour saturation degrees 72–3, 74 naturalistic 61–2, 70, 73, 75, 77, 78 of senses 73 sensory 77–8 visual, kinds of 74–9 monochromes 73, 90, 104, 105 monomodal approach 6, 21 moral consumer culture 29 multiculturalism and categorization 52 narrow typefaces 126 naturalistic modality 61–2, 70, 73, 75, 77, 78 naturalness, in textures 153–4, 156 neo-liberalism 24–5, 198 network diagrams 207–8 Newcastle University strategic diagram 215 diagrams and flowcharts and 200, 202, 206, 208, 209, 210, 211 non-frontal perspective 82–3 angles 83–4 distance 84–5 non-representation, in pictures and images 53–4, 55 oblique angles 84, 85 open-plan buildings, glass in 159 Organic ProteinTM plant based protein powder 173, 174 orientation, in font 132–3, 136

overlapping and framing 182–4, 186 and salience 178 Oxfam poster 188 salience in 175, 177 separation principle in 181 paper 161, 164 paradigms 198–9 pathway diagrams 207 pathway flowchart, for fairtrade shopping 204 Peirce, C.S. 56 people and classification 200–201 photography 7; See also images; pictures and images of anonymization 52 categorization and 52 colour and 91, 95, 99, 101, 103 composition and page layout 169, 172, 172, 176–80, 182, 185, 186–9 connotation and 43, 44, 84 and coordination 188–9 denotation and 39–40 of diagrams and flowcharts 193, 202, 203, 205 and diagrams compared 75 as document 62 as evidence 41 hierarchies and rhyming and 186–8 idealization of 66 individuals and groups 48, 49, 50–1 modality and 61–3, 65–6, 69–71, 73, 75, 77, 78, 80, 82–5 settings and 46 textures and 145, 154, 160 as truthful 77 visual depiction of 47 pictures and images 37–9 actions in 55–8 anonymization and 52–3 categorization and 51–2

241

242

242

Index connotation and 40–2 carriers of 42–7 denotation and 39–40 distance in 50–1 individuals and groups as 48–50 non-representation and 53–4 plastics 161–2, 164 plastic surgery website 14 brightness in 99 colour differentiation in 105 colour use in 87 fonts in 117, 126 hue in 106 integration principle in 182 sensory modality and 78 texture in 144 political cartoons 52 power, significance of 29 Poynter Institute 115 pram rattles 149–151 preschool chart 196, 197 processes and classification 199 quality development, university web page for 190 Rambo poster 87, 107–8, 118 background in 69 brightness in 99 colour differentiation in 73, 105 colour modulation in 104 colour saturation in 90, 100 fonts in 117, 120, 125, 126, 131, 133 hue in 106 impure colours in 101, 102 integration principle in 182 monochrome in 90 salience in 171, 174, 177 tone articulation in 71–2 typographic features in 135 Ranger, T. 19 re-contextualization, of aggressive act 54

regularity in font 133–4, 136 in textures 152–3, 156 relief, in textures 151–2, 156 representation of attitudes 117–18, 124 of causalities 211 of colour 89, 93, 103, 104, 105 commodified communication and 26, 27, 28 composition and page layout and 170, 178, 185, 186, 188, 189 connotation and 42–3 denotation and 39 of diagrams and flowcharts 194, 195, 196, 199–203, 205–10, 212, 214 discourse and social practices and 20 generic 66, 201 graphic 210, 212 of ideas 118–20, 124 modality and 62–70, 72, 73, 75–8, 82, 83, 84, 85 of modernity 162 naturalistic 75, 77 non-representation and 53–4 of participants 48–54 pictures and images and 37–8, 44, 45, 47, 57, 58 realism in 14 textures and 149 typography and 111, 117–20, 130 retropia 143 rhyme and framing 184–5, 186 photographs, hierarchies and 186–8 Riddells Creek Organic and toothpaste tube 68 rigidity, in textures 149–51, 155–6 roundness, significance 130 salience 170 colour and 173–4

243

Index focus and 176–7 foregrounding and 177 overlapping and 178 potent cultural symbols and 170–1 size and 171–174 tone and 174–6 Sandro storefront 128 disconnection on fonts in 132 texture in 139 scientific drawings 74 segregation and framing 180, 185 semiotic choices 15–16, 30 semiotic material 17–20, 24 semiotic resources, coding of 31 configurations and 33 contextualizations and 32 distinctions and 31–2 semiotic system colour as 89–96 texture as 140–9 typography as 117–24 sensory modality 77–8 separation and framing 181–2, 185 signs colour and 90 indexical 56 meaning and significance of 5, 9, 18 significance of 9 typography and 120, 122, 125, 131, 132, 134 size and salience 171–174 slate 163 slope, in font 127–9, 136 smart goals template 213, 214, 216 social practices 20, 30 social semiotics 13, 14, 15 Social Semiotics (Kress and Hodge) 65 space, use of 127 Starbucks café image 96 anonymization in 52 brightness in 99 colour coherence in 95

colour palette 102 configuration principle in 33 connotations in 43 contextualization principle in 32 distance in 50 hybrid colours in 102, 103 individualization in 48 monochrome in 105 settings in 46 significance of 37 silence in 57 texture in 139, 142, 143, 144, 153, 158 steel 162 stock images, categorization in 52 stone 163–4 stripped back look 18, 20, 23, 130, 160, 162 textures and 145, 147, 153, 157 sustainability 92–3, 171, 189 at Örebro University 188 and salience 174 scorecard 93, 94, 95, 99, 100, 104, 123, 184, 185 Swedish School Inspectorate and communicative rationality 26, 27 symbolization and composition and grouping 209–10 systemic networks 8 technologization 21–2, 24, 30, 120, 197 television programmes and cultural categorization 51 temporality 212–16 textual metafunction 8 textures 139–40 authentic 145 communicating attitudes 142–4 communicating ideas 140–2 and composition 144–9 dimensions of 149 liquidity 155, 157 naturalness 153–4, 156

243

244

244

Index regularity 152–3, 156 relief 151–2, 156 rigidity 149–51, 155–6 viscosity 154–5, 156 in integrated design 147–8 materials 157–8 brick 160 carton/card 161, 164 concrete 160, 164 glass 159 metal 162, 164 paper 161, 164 plastics 161–2, 164 stone 163–4 wood 158–9, 164 natural and industrial 145, 146 Thornborrow, J. 47 tone and salience 174–6 tone degrees, articulation of 71–2, 74 typeface 111–14 in coherence creation 120–4 communicating attitudes 118–20 communicating ideas 117–18 and design 114–17 meaning potential inventory (see letterforms, features of) typography. See typeface Unikum Quality Development, in preschool 40, 41, 45–6 anonymization in 53 background in 69 flowchart representation in 195 high modality in 68 Van Leeuwen, T. 6, 7, 8, 9, 16, 21, 48, 52, 65, 70, 73, 75, 88, 89, 97, 98, 101, 115, 117, 124, 149, 179 Van Wagener, A. 115, 125 verbal processes 57, 58 vertical angles 83, 85

vertical line 112 vertical orientation 133, 206 viscosity, in textures 154–5, 156 visual categorization 51 visual classification 201–2, 216 visual design, importance and significance of 2 Viva La Papa chips 64, 69–70 Voloshinov, V. N. 9 war 52 discourse on 19 moral complexity of 102 potent cultural symbols and 170 textures and materials and 157, 160, 162 wax kit packaging 183 framing in 180 weight, in font 124–6, 136 wide typefaces 126 women categorization and 52 curvature and 131 individualization and 48 integration and 182 light and shadow articulation and 71 material processes and 58 mental processes and 57 non-representation and 53 object meaning in images and 43–4 overlap and 184 potent cultural symbols and 171, 172 potential address and 82 regularity and 134 segregation and 180 sensory modality and 78 tone and 175 viscosity and 154 wood 158–9, 164 writing and classification 198–9, 216