Complexity of Interaction: Studies in Multimodal Conversation Analysis [1 ed.] 3031307267, 9783031307263

Citation preview

Complexity of Interaction Studies in Multimodal Conversation Analysis Edited by Pentti Haddington · Tiina Eilittä Antti Kamunen · Laura Kohonen-Aho Iira Rautiainen · Anna Vatanen

Complexity of Interaction “The publication of this volume marks a new and much needed development of ideas that complexity is essential in the organization of human conduct. The editors have done a masterful job of bringing together researchers in discussions that explore the multimodal and multisensorial properties of human sociality and provide an illuminating analytical and conceptual framing of complexity and social interaction.” —Asta Cekaite, Professor, Department of Thematic Studies, Linköping University, Sweden “It is commonplace that social interaction is complex. This volume is the first to systematically explore sources and orders of this complexity in various interactional settings. It shows how technical mediation, multi-activity, the presence of several parties, the wealth of multimodal resources, and multi-sensorial perceptions create complicated and fragile constellations and processes in social encounters. In addition to fine-grained in-depth analyses of video recordings, the authors establish complexity as a theoretical lens that allows us to gain unprecedented insights into the fabrics of intersubjectivity and interactional organization in our current societies.” —Arnulf Deppermann, Professor, Leibniz Institute for the German Language, University of Mannheim “This book recognizes that participants in social interactions communicate in a multisensory manner, are often involved in several activities at once, and have unequal access to interpretive resources. This is a challenge for research in multimodal conversation analysis, which is based on extremely rich audiovisual recordings. The authors provide invaluable analytical tools to understanding these and other complexities while embracing the richness of the data captured in audiovisual recordings of real-life situations. A real tour de force!” —Esther González-Martínez, Professor of Sociology, University of Fribourg, Switzerland “This book attends to two important developments within conversation analysis. On the one hand, it explores the relevance of a wide array of multimodal and multisensorial resources for the ways in which people make sense of social conduct. On the other hand, it embraces the analysis of rich and multi-faceted interactional, institutional and technological environments. Building on these developments, the authors do not shy away from what they call the ‘complexity of interaction’, but instead, they treat it as an opportunity to enhance our understanding of the organisational features of everyday life. The book draws together a collection of high calibre authors who are studying social interaction in an impressive range of settings. And, as a collection, the book offers insights at the very cutting edge of conversation

analysis, which are of relevance to all those concerned with understanding social interaction in its rich and complex detail.” —Jon Hindmarsh, Professor of Work and Interaction, King’s Business School, London, UK “This volume showcases the broad range of interactional issues that can be addressed through qualitative video analysis. The chapters uncover how people exploit multimodal resources to solve practical problems in multiactivity settings, when dealing with technologies or navigating transient opportunities for participation. The reader will find exquisite accounts on how we systematically use all our senses, material ecologies, as well as language to bring about social life when gaming, singing, eating, playing, and training the military.” —Leelo Keevallik, Professor, Department of Culture and Society, Linköping University, Sweden “From the perspective of multimodality, social interaction can be truly complex, and the analyst might sometimes be tempted to look away from that complexity, just to be able to produce a coherent analysis. The authors in this book, in contrast, address this challenge head on. In their highly accessible way of teasing apart different sources of interactional complexity – the multiplicity of activities, asymmetricities, participation frameworks and interactional settings – the reader is encouraged to tackle the complexities of interaction one by one. The book is thus essential reading to all analysts of multimodality in interaction.” —Melisa Stevanovic, Assistant Professor, Faculty of Social Sciences, Social Psychology, Tampere University, Finland “Building on a growing body of work studying interaction from a multimodal perspective, and interested in the study of multi-layered character of action, the book proposes a bold and fecund move: making the complexity of interaction into a central focus of the analysis so as to identify new interactional phenomena. Such a “manifesto for complexity” is not only a great contribution to ethnomethodology and conversation analysis, but it should also be highly relevant to wider communities of researchers and practitioners attuned to the intricacies and “complexity” of human activities.” —Christian Licoppe, Professor, The Economics and Social Sciences Department, Télécom Paris

Pentti Haddington · Tiina Eilittä · Antti Kamunen · Laura Kohonen-Aho · Iira Rautiainen · Anna Vatanen Editors

Complexity of Interaction Studies in Multimodal Conversation Analysis

Editors Pentti Haddington Languages and Literature University of Oulu Oulu, Finland

Tiina Eilittä Languages and Literature University of Oulu Oulu, Finland

Antti Kamunen Languages and Literature University of Oulu Oulu, Finland

Laura Kohonen-Aho Languages and Literature University of Oulu Oulu, Finland

Iira Rautiainen Languages and Literature University of Oulu Oulu, Finland

Anna Vatanen Languages and Literature University of Oulu Oulu, Finland University of Helsinki Helsinki, Finland

ISBN 978-3-031-30726-3 ISBN 978-3-031-30727-0 https://doi.org/10.1007/978-3-031-30727-0

(eBook)

© The Editor(s) (if applicable) and The Author(s), under exclusive licence to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Cover credit: Andrea Comi/gettyimages This Palgrave Macmillan imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Acknowledgements

We thank publisher Palgrave Macmillan for taking this book into its publication programme and for believing in it from the outset. We especially thank Executive Editor Cathy Scott for the assistance, support and faith in this book project, and Project Coordinator Naveen Dass for the help and support during the typesetting process. It has been a pleasure to work with both of you. We also thank the contributing authors for their hard work and patience with the process and us editors. We could not have done this without you. This book’s theme represents a long-standing research interest of the COACT community (Complexity of [Inter]action and multimodal participation) at the University of Oulu. For more than a decade, the researchers in COACT have shared an interest in the study of multimodal interaction in diverse settings and across different languages, exploring, for example, the complexity of multi-layered action and activity. We want to thank the members of COACT community for being there and sharing their work, thoughts and ideas with us. You have inspired the work in this book.

v

vi

Acknowledgements

The first steps of this book were taken at the IIEMCA conference in Mannheim, Germany, in July 2019. At the conference, the editors of this book organised the panel called Interactional practices and complexity in diverse settings: Findings, methods and implications. The talks and discussions in the panel encouraged us to start working on an edited volume on the complexity of interaction. We thank the participants in the panel, many of whom have also contributed to this book. We are grateful to our colleagues—both contributors to the book and external colleagues—who committed to the peer review process of the chapters and provided detailed comments to them along the way. Your advice and comments have been invaluable. We are also grateful to the three scholars who reviewed the book proposal. The reviewers’ suggestions were very helpful and developed the book in important ways. We also thank Marika Helisten and Mirka Rauniomaa for reading the early drafts of the book’s introductory chapter and giving constructive feedback. We also thank Lorenza Mondada for important comments and thoughts about ‘complexity’ near the end of the project. We gratefully acknowledge the funding and support for two research projects without which this book would not exist. The two projects— iTask: Linguistic and embodied features of interactional multitasking and PeaceTalk: Talk and interaction in multinational crisis management training—have received funding from the Academy of Finland (decision numbers 287219 and 322199) and the Eudaimonia Institute at the University of Oulu. Finally, we acknowledge our debt to our colleagues around the world whose work is a constant source of inspiration to us. It is a joy and privilege to have the possibility to exchange ideas and observations about talk, language and multimodality in social interaction. We also extend our heartfelt gratitude to our families, friends and other close ones for their interest and support along the way.

Contents

1

On the Complexities of Interaction: An Introduction Tiina Eilittä, Pentti Haddington, Antti Kamunen, Laura Kohonen-Aho, Iira Rautiainen, and Anna Vatanen

Part I 2

3

4

1

Complexity That Resides in Multiactivity and Multisensoriality

Tasting vs. Eating: The Methodic and Situated Differentiation of Embodied Multisensorial Activities in Social Interaction Lorenza Mondada

29

Metagaming and Multiactivity: How Board Game Players Deal with Progressivity Emily Hofstetter and Jessica Robles

65

Embodied Noticings as Repair Initiations: On Multiactivity in Choir Rehearsals Anna Vatanen

99

vii

viii

Contents

Part II

Complexity That Resides in Asymmetries Related to Affordances, Resources and Roles

5

6

7

Intersubjective Interaction During the Word Explanation Activity in Social Virtual Reality Heidi Spets Building on Linguistically Exclusive Talk: Access, Participation, and Progressivity in a Multinational Military Staff Antti Kamunen and Pentti Haddington Nudging Questions as Devices for Prompting Courses of Action and Negotiating Deontic (A)symmetry in UN Military Observer Training Iira Rautiainen, Pentti Haddington, and Antti Kamunen

Part III 8

9

145

175

217

Complexity That Resides in the Coordination of Participation Frameworks

Playing Together on a Large Screen: Spatiality, Materiality, Temporality and the Complexity of Interaction Heike Baldauf-Quilliatre and Biagio Ursi Getting (Others) Involved with Smartphones: Participation in Showing Sequences in Multiparty Settings Iuliia Avgustis and Florence Oloff

10 The Primacy of Affective Engagement in Simultaneously Unfolding Participation Frameworks Julia Katila, Sara A. Goico, Yumei Gan, and Marjorie Harness Goodwin

255

297

347

Contents

Part IV

Complexity That Resides in the Characteristics of Interactional Settings and Environments

11

Ambulatory Openings Elliott M. Hoey

12

Openings of Interactions in Immersive Virtual Reality: Identifying and Recognising Prospective Participants Pentti Haddington, Laura Kohonen-Aho, Sylvaine Tuncer, and Heidi Spets

13 Transitions Between Interactional Spaces: Working Towards Shared Understanding in a Hybrid Workshop Setting Laura Kohonen-Aho Index

ix

389

423

457

507

Contributors

Iuliia Avgustis Faculty of Humanities, University of Oulu, Oulu, Finland Heike Baldauf-Quilliatre Université Lyon 2 Lumière, Lyon, France Tiina Eilittä Faculty of Humanities, University of Oulu, Oulu, Finland Yumei Gan Shanghai Jiao Tong University, Shanghai, China Sara A. Goico University of California, Los Angeles, CA, USA Marjorie Harness Goodwin University of California, Los Angeles, CA, USA Pentti Haddington Faculty of Humanities, University of Oulu, Oulu, Finland Elliott M. Hoey Department of Language, Literature, and Communication, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands Emily Hofstetter Linköping University, Linköping, Sweden

xi

xii

Contributors

Antti Kamunen Faculty of Humanities, University of Oulu, Oulu, Finland Julia Katila Tampere University, Tampere, Finland Laura Kohonen-Aho Faculty of Humanities, University of Oulu, Oulu, Finland Lorenza Mondada University of Basel, Basel, Switzerland Florence Oloff Leibniz-Institut für Deutsche Sprache, Mannheim, Germany Iira Rautiainen Faculty of Humanities, University of Oulu, Oulu, Finland Jessica Robles Loughborough University, Loughborough, UK Heidi Spets Faculty of Humanities, University of Oulu, Oulu, Finland Sylvaine Tuncer King’s College, London, UK Biagio Ursi Université Lyon 2 Lumière, Lyon, France Anna Vatanen Faculty of Humanities, University of Oulu, Oulu, Finland; University of Helsinki, Helsinki, Finland

List of Figures

Fig. Fig. Fig. Fig.

3.1 3.2 3.3 3.4

Fig. Fig. Fig. Fig. Fig.

3.5 4.1 4.2 4.3 4.4

Fig. 4.5 Fig. Fig. Fig. Fig. Fig. Fig.

4.6 4.7 4.8 4.9 4.10 4.11

View of the game Skull , L-R: Kat, Tara, John, Adam Attention shift to Tara All players orient to Tara Bird looking at board and preparing to play, other players talking Tina waving dice in Hal’s face Choir setting A crystal ball view from the 360° camera View from the GoPro camera Images of Example 1. In images where the singer sings, there is a speech bubble with a note; no such bubble means no singing at that moment Music of Example 1. The moments when the still images are taken have been marked with numbers Images of Example 2 Music of Example 2 Images of Example 3 Music of Example 3 Images of Example 4 Music of Example 4

75 78 78 84 87 106 107 107

112 113 115 116 117 117 120 120

xiii

xiv

Fig. Fig. Fig. Fig. Fig. Fig.

List of Figures

4.12 4.13 4.14 4.15 5.1 5.2

Fig. 5.3

Fig. 5.4 Fig. 5.5 Fig. 5.6 Fig. 5.7 Fig. 5.8 Fig. 5.9

Fig. Fig. Fig. Fig. Fig. Fig.

6.1 6.2 6.3 6.4 8.1 8.2

Fig. 11.1 Fig. 11.2

Images of Example 5 Music of Example 5 Images of Example 6 Music of Example 6 Avatars. ©Rec Room Inc. Edited view of all three streams. Participant 1 sees what is on the upper left, and their avatar can be seen on the upper right. Vice versa, participant 2 sees what is on the upper right, and their avatar can be seen on the upper left. The lower stream is from a 360-degree camera recording the participants (highlighted) in the physical space. ©Rec Room Inc. Recording set-up. Above: 360-camera. Left: participant wearing the HMD and hand-held controllers. Right: motion detection tower 3D pen held by an avatar Left: Heikki’s view, Pertti’s drawing is highlighted. Right: Pertti’s view Left: Lisa’s view. Right: Pat’s view. The drawing appears flat for Lisa, yet curved for Pat Pat’s view when he leans to his left on l. 18 Pat’s view when he leans on l. 29 Left: Pertti’s view of Heikki’s avatar body and the drawing. Right: Heikki’s real body. The area of the drawing movement is indicated with circles in both images BC provides the meaning of the acronym DIRLAUT Discussing the deployment of simulated helicopters DEP works to interpret ENG’s talk LOG follows a discussion in Finnish Initial setting and spatial organisation (group 6) Setting and spatial organisation during the gaming (group 6) Jamie approaches Ryan and Chuy directly (a) Omar and Ryan approach Gerardo from behind; (b) Tomas retracts tape measure, Gerardo turns to face Omar and Ryan; (c)–(d) Omar and Ryan near and establish proximate formation

122 123 126 127 150

151

152 155 159 162 163 163

167 185 190 198 204 259 267 396

399

List of Figures

Fig. 11.3

Fig. 13.1 Fig. 13.2

(a) Pablo directly approaches Jamie and Horacio; (b) Jamie spots Pablo approaching; (c) Pablo nears (a)–(b) Fernando walking in Ben’s direction; (c) Fernando and Ben greet each other as Fernando passes; (d) Ben takes step backward and Fernando continues onward (a) Ryan looks up at crane/panel; (b) Isidro enters setting; (c) Ryan sees Isidro, Isidro looks up at crane/ panel; (d) Ryan returns gaze to crane/panel, Isidro approaches; (e) Isidro gives suggestion to Ryan in passing; (f ) Isidro walks away (a) Ryan’s walking trajectory and destination; (b) gaze to Isidro (off camera) and slower pace while talking; (c) slower walk and slight approach; (d) return to walking trajectory Restricted approach with an abbreviated trajectory A user wearing a head-mounted display, holding two controllers, and standing next to a beacon that monitors the player’s movements Two avatars in the Rec Room virtual reality game Merged video streams: 360-degree video (bottom), the view of the participant on the left (top left frame) and the view of the participant on the right (top right frame) Workshop setup Workshop locations 1 and 2

Image 6.1

Personnel and seating arrangement in the TOC

Fig. 11.4

Fig. 11.5

Fig. 11.6

Fig. 11.7 Fig. 12.1

Fig. 12.2 Fig. 12.3

xv

403

405

408

412 414

429 430

432 462 464 181

1 On the Complexities of Interaction: An Introduction Tiina Eilittä , Pentti Haddington , Antti Kamunen , Laura Kohonen-Aho , Iira Rautiainen , and Anna Vatanen

1.1

Introduction

The studies in this volume share the view that everyday social life is deeply tied to the ways in which people talk, interact, and engage in joint activities with each other. In order to study interaction in everyday life, this volume draws on the analytic mentality of conversation analysis (CA). CA is a qualitative, bottom-up, and strictly empirical approach to T. Eilittä (B) · P. Haddington · A. Kamunen · L. Kohonen-Aho · I. Rautiainen · A. Vatanen Faculty of Humanities, University of Oulu, Oulu, Finland e-mail: [email protected] L. Kohonen-Aho · A. Vatanen University of Helsinki, Helsinki, Finland P. Haddington e-mail: [email protected] A. Kamunen e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_1

1

2

T. Eilittä et al.

the study of the organisation of social interaction. It grounds its analyses on audio and/or video recordings of interactions that have been collected in real-life (i.e., naturally occurring) situations. Analyses of the recordings are supported by detailed transcriptions of the participants’ talk (Hepburn and Bolden 2017; Jefferson 2004) and embodied conduct (Mondada 20181 ). Overall, CA aims to shed light on the orderly nature of everyday social life and reveal the systematics of how interactants coordinate their actions relative to the talk and behaviour of their co-participants as well as the contingencies of the interactional situation. One of the core pillars that CA research stands on concerns “intersubjectivity,” that is, the “joint understanding and sharing of experience between humans” (Sorjonen et al. 2021: 1; see also Enfield and Sidnell 2022; Linell 2017). Intersubjectivity is a precondition for all joint human activity (e.g., Enfield and Sidnell 2022: 12). At the same time, human interaction is the primordial site where intersubjectivity is created and maintained: Participants make sense of each other’s actions and sequences of actions, create moments of shared experience that these actions embody, build referential common ground, and establish shared understandings of the meanings of linguistic forms, in the moment of their production and relative to the context in which they are produced (Deppermann 2019; Enfield and Sidnell 2022; Peräkylä 2013; Sorjonen et al. 2021). CA is one of the rare research methodologies that analyses the accomplishment of intersubjectivity in real-life and real-time interaction. It has provided rigorous and detailed analyses of interactional practices 1

See also https://www.lorenzamondada.net/multimodal-transcription.

L. Kohonen-Aho e-mail: [email protected] I. Rautiainen e-mail: [email protected] A. Vatanen e-mail: [email protected]

1 On the Complexities of Interaction

3

and processes by which shared experiences and mutual understandings are accomplished in a joint and coordinated manner in social interaction (see Enfield and Sidnell 2022: 29). First of all, and most fundamentally, intersubjectivity is evidenced in the ways in which social participants jointly and systematically coordinate the basic organisational features of their interactional behaviour. For example, intersubjectivity is evidenced in the ways in which co-participants take and allocate turns at talk as well as time and order them with respect to the turns by co-participants (i.e., turn-taking organisation; Sacks et al. 1974). The turn-taking machinery provides participants with opportunities to indicate and check whether and how they have understood one another (e.g., Heritage 1984; Heritage and Atkinson 1984; Mondada 2007; Sacks et al. 1974: 729). Intersubjectivity is also evidenced in the ways in which shared understandings are accomplished through the organisation of single actions into sequences of actions: The sequentially next turn displays, through its design and the action it performs, an interpretation and understanding of the previous turn and its action (Schegloff 2007). As Heritage (1984: 259) notes, [b]y means of this organization a context of publicly displayed and continuously up-dated intersubjective understandings is systematically sustained . It is through this ‘turn-by-turn’ character of talk that the participants display their understandings of ‘the state of the talk’ for one another.

Interactants can also solve troubles emerging in interaction—a practice called repair organisation (Schegloff et al. 1977; Schegloff 1992)— to restore mutual understanding, for example, in situations where a response to an initiating turn indicates an ill-fitted or somehow wrong understanding of it (Haakana et al. 2021; Heritage 1984; Raymond 2019; Schegloff 1992). A large part of CA research has focused on talk as the single modality of interaction and paid less attention to the multimodal richness that is inherent to social action and interaction. Furthermore, in CA, the organisation and practices of interaction have often been studied in relatively

4

T. Eilittä et al.

orderly and static situations, such as telephone or dinner table conversations, or in dyadic institutional appointments or sessions, such as medical or therapy interactions.2 The studied settings have also generally involved participants who are physically co-present and thus mutually available for interaction and who have symmetric and uninterrupted access to the same interactional resources (e.g., the same language or the same physical environment). Noting this we do not, however, mean to consider any of the above as an instance of negligence on the part of conversation analytic research but a natural part of the emergence and development of the research and methodology of CA. The development of recording technologies and video analysis tools, as well as methodological advancements in CA, has given rise to a branch of CA called “multimodal CA.” With improved recording technologies, it has become easier to not only focus on multimodal features of interaction but also gain access to and study more complex settings and interactions in which participants are, for example, involved in several simultaneous activities or conversations, moving around or dispersed. At the same time, the studied phenomena and relevant resources taken into consideration for analysis have progressively enlarged, leading to new discoveries, for example, on mobility, multiactivity, materiality, and technology in interaction. All this has helped multimodal conversation analysts to reveal the richness, multiplicity, and sophistication of how social participants establish intersubjectivity and everyday social life in and through interaction. In line with multimodal CA and by adopting an analytic lens of complexity that focuses specifically on the use of multi-layered features and the rich interconnectivity of multimodal resources in interaction (see Sect. 1.3), this book aims to answer, for example, the following questions: How do interactants organise or engage in multiple activities simultaneously? How are sensory practices (e.g., tasting and touching) tied to the organisation of activities? How do participants manage asymmetrical access to knowledge, language, or the rich multimodal context? How do 2 Repair organisation, on the other hand, has been studied broadly in the context of so-called atypical situations, such as those involving second language speakers (e.g., Kurhila 2001; Lilja 2010) or participants with aphasia or hearing impairment (e.g., Barnes 2016; Laakso 2000; Pajo & Laakso 2020).

1 On the Complexities of Interaction

5

participants coordinate shared access and mutual availability in hybrid and distributed settings? How are participants’ assigned and assumed roles oriented to in the constitution of tasks and activities? The following section gives a brief synthesis of research in multimodal CA. Section 1.3 outlines what we mean by “complexities of interaction” and what a focus on complexity can add to existing research in CA. Section 1.4 motivates the structure of the volume and introduces the chapters in it. Finally, Sect. 1.5 concludes the introductory chapter and offers suggestions for future research on the complexities of social interaction.

1.2

Multimodal Conversation Analysis: Background

Multimodal CA3 has investigated how interactants use the rich range of multimodal affordances (Gibson 1979) and resources, such as the body or features of the environment, for producing and interpreting social actions and building shared understandings.4 It has shown how multimodal resources create emergent intersubjective fields that participants use for producing and ascribing meanings to social actions (Peräkylä 2013: 552). For example, pointing gestures and mutual gaze have been shown to contribute to the building of a shared focus of attention (e.g., C. Goodwin 2000; C. Goodwin and M.H. Goodwin 2004) and common understanding in interaction (e.g., C. Goodwin 1986; Streeck 2009). Furthermore, Charles Goodwin’s work (e.g., 2000) about the ways in which co-participants can attend to multimodal resources in the environment for interaction (e.g., language, objects and material features) to produce and interpret actions has been particularly influential. Goodwin (2000: 1490) showed how the local and intertwined array of such semiotic resources—what he called “contextual 3 An influential branch of CA research in this area confluences with workplace studies (e.g., Hutchins 1995; M.H. Goodwin 1996; Hutchins and Klausen 1996: Luff et al. 2000). 4 For example, Stivers and Sidnell (2005), Nevile (2009), Streeck et al. (2011) and Haddington et al. (2013) have influenced and contributed to the emergence and development of multimodal CA.

6

T. Eilittä et al.

configuration”—“frame, make visible, and constitute the actions of the moment.” The building and maintaining of shared understanding in interaction also involve the participants’ joint and mutual access to each other’s actions. For example, the systematic vocal and embodied ways in which people together inhabit a space and establish group formations through postures and positions of the body provide mutual access to co-participants’ conduct and the possibility to monitor and interpret each other’s behaviour (Goffman 1964, 1978). The orientation to joint and mutual access is also evidenced in the systematic ways in which co-participants can, through various embodied and verbal actions, gradually establish joint orientation in emerging interactional encounters (Mondada 2009, 2014a). Furthermore, by analysing the multimodal features of actions and activities, research in multimodal CA has shown how social actions can be produced and interpreted as “action complexes” or “complex multimodal Gestalts,” that is, packages of several modalities such as language (morphosyntactic, lexical, and prosodic features), gesture, touch, gaze, the body, and other material resources (Cekaite 2016; C. Goodwin 2006, 2007; M.H. Goodwin 2017; Mondada 2011, 2014a, 2018, 2019; Stukenbrock 2021; see also “lamination,” C. Goodwin 2013, 2018, and “multimodality,” Cekaite 2015). As an example, M.H. Goodwin et al. (2002) have shown how during a game of hopscotch, prosody, the built environment, gesture, and the body mutually elaborate one another in the accomplishment of action. The above studies have focused on how participants progress one activity at a time, inhabit the same physical space and/or rely on audible and visual cues when producing or ascribing meanings to actions. However, multimodal CA has also studied increasingly complex situations, activities, and settings, as well as extended its analytic scope beyond audible and visual cues (Broth 2009; Cekaite 2015, 2016; Haddington et al. 2013, 2014; Keating 2005; Luff et al. 2003; Mondada 2011; Nevile 2004, 2009; Nishizaka 2007, 2011). For example, it has studied multiparty situations and how they can present complex forms of participation. An example of this is called “byplay”; an interactional episode that is simultaneous yet “subordinate” to the current main activity

1 On the Complexities of Interaction

7

and involves different participants (M.H. Goodwin 1997). Relatedly, multimodal CA has also analysed the ways and practices by which coparticipants advance and coordinate multiple, intersecting or conflicting activities and action trajectories through talk and embodied action (Deppermann 2014; Haddington et al. 2014; Helisten 2019; Kamunen 2019, 2020; Kamunen and Haddington 2020; Mondada 2011, 2014b; Nevile 2009), and how they orient to and manage multiple different temporalities of events and activities (e.g., in the context of game playing or when occasioned by technologies) and adjust them to the ongoing action and its projectable trajectory (Mondada 2013). Finally, complementing recent work in embodied practices in interaction, CA has also started to explore touch as a practice for establishing intersubjectivity (e.g., Throop 2012; M.H. Goodwin 2017; Meyer 2017; Mondada et al. 2021). Touch is central in interactions where adults deal with infants (Field 2001), but tactile actions can also be important in adult-adult interactions, for example, when healthcare workers deal with patients (Nishizaka 2007, 2011). In CA, there are far fewer studies on the role of touch as a modality for establishing intersubjectivity than there are studies that focus on vocal practices of organising social interactions.

1.3

Towards the Analysis of Interaction Through the Lens of Complexity

The chapters in this volume build on and complement the research in multimodal CA outlined in Sect. 1.2. They rely on the common denominator complexity of interaction to characterise and highlight the multilayeredness of actions and activities, and to include detailed accounts of the ingredients of actions and activities (e.g., related to embodiment, multimodality or the broader ecology of interaction) and their role in how people establish shared understanding and experience in interaction. The search for complexity in CA is not self-evident, nor is it always necessary to focus on complex matters. Some strands of CA research have focused on talk in dyads, in a single activity (e.g., telephone conversations), as well as interactions in which the participants have symmetric access to resources of interaction and each other. In other words, the

8

T. Eilittä et al.

focus has been on what could be characterised as non-complex or “simple” situations. Through this work, CA has successfully revealed the orderliness and systematicity of social interaction. This book—in line with multimodal CA—takes a distinct epistemological perspective: The chapters adopt an analytic lens focusing on “complexity.” They highlight the multi-layered and multidimensional features of interaction and make visible interactional practices or organisations of social order that might otherwise escape the analyst’s attention (see Sect. 1.4). Consequently, the chapters also demonstrate that it is possible—or sometimes even necessary—to direct analytic focus on the complexities of interaction to identify and analyse practices that are used to organise interaction and achieve intersubjectivity in it. It is important to note that complexity—the multi-layeredness of action or the rich interconnectivity of resources in interaction—is neither an interactional phenomenon, a practice nor an inherent feature of any one aspect or context of social interaction. Nor does it necessarily refer to phenomena that are treated or verbalised as complex by the participants. In some respects, this presents a methodological challenge for CA: In CA, analytic claims—in this case, concerning the complexity of interaction—should capture the member’s perspective in a given interactional situation. However, members may not hearably or visibly treat complexity as complex. Consequently, analysing the complexity of interaction is a continuous and demanding balancing work. When is the analysis of an interactional detail redundant and something that distances the analyst from the member’s perspective? Is there a danger that complexity alienates the analyst from the focus on the orderly and systematic nature of interaction, that is, “order at all points” (Sacks 1984: 22)? The answer to these questions is twofold. First, participants process complex interactional features holistically, as meaningful composite Gestalts. In these cases, it is important to model, reconstruct, and parse the various layered and overlapping features and dimensions of social action and activity in order to identify, analyse, and describe them. Such analysis sheds light on the constituent elements of action that for the participants constitute holistic action. In fact, an analogy can be made to the analysis of language and grammar: When a speaker talks to a

1 On the Complexities of Interaction

9

recipient, the recipient does not parse or reconstruct the turn into its constituent linguistic parts (e.g., phrases, words, morphemes, phonemes) but interprets the speaker’s utterance holistically. Nevertheless, the analysis of the grammar of the turn requires distinct methods in order to reveal the intricacies and subtle details of the structures in question (on the analysis of linguistic complexity, see, e.g., Dahl 2004; Di Garbo et al. 2019). In a similar manner, when analysing the subtle structures of social interaction, specific methods can reveal the details and intricate features of interaction, and demonstrate the multiple layers involved, even though the participants interpret one another’s actions holistically and may not explicitly orient to the subtleties involved. However, second, participants do occasionally, if relevant for the activity, orient to or address details or particulars of complexity (e.g., when addressing the normativity or intelligibility of action). Solving such situations may also involve participants’ “additional interactional work,” that is, actions and practices that aim to solve disruptions or asymmetries in joint activity. In such cases, participants treat complexity as an oriented-to and accountable feature of action. Such situations can also occur at moments where intersubjectivity collapses and participants aim to recover and restore symmetry and mutual orientation or when participants balance between advancing activity and maintaining intersubjectivity (see Markaki et al. 2013). As “deviant” situations, they also reveal social order and make it apparent. The next section introduces the structure and the chapters of this volume. The studies in the book provide examples of what the study of complexity looks like. They also present illustrative cases of how multimodal conversation analysis can be used to study the complexities of social interaction and intersubjectivity. They showcase some of the challenges and solutions for capturing the complexity of interaction on video and present solutions for representing the interactions visually and in writing.

10

1.4

T. Eilittä et al.

Contributions to the Present Volume

The chapters in this book contribute to multimodal CA by studying social interaction in environments that involve embodied action, multiactivity, multiparty interaction, rich multimodal ecologies, and technologized and technology-mediated settings. The collection of settings studied in the chapters is wide and somewhat coincidental; they are not meant to represent a full list of possibly complex interactional situations, nor are the settings themselves the focus or starting point of the studies. Instead, the authors start from the multi-layeredness of action and activity and then explore the data through the lens of complexities that the data gives rise to. As the chapters demonstrate, the contexts and settings where complexities of interaction emerge and can be examined are diverse and cover broad territories of everyday life. The chapters study interactions in both mundane and professional settings: in-person interactions (Hoey; Avgustis and Oloff ) and video-mediated interactions (Kohonen-Aho); mundane gameplay activities at homes (Hofstetter and Robles) and in public places (Baldauf-Quilliatre and Ursi); and food-centred interactions in a professional kitchen, at a market, and in a gastronomic restaurant (Mondada). The analyses also draw on video-recorded data collected in rarely studied and specialised institutional and professional settings such as rehearsal sessions of a 50-person amateur choir (Vatanen), car-patrolling exercises with multinational military observer teams (Rautiainen, et al.) and exercises in military tactical operations centres (Kamunen and Haddington). The studied settings illustrate the salient ways in which participants, in different interactional situations, establish and maintain access to each other or the shared interactional environment (i.e., build intersubjectivity) when they interact: that is, when they are (1) co-located and physically co-present in a shared space (Hoey; Kamunen and Haddington; Avgustis and Oloff; Rautiainen, et al.; Katila, et al.; Hofstetter and Robles; Baldauf-Quilliatre and Ursi; Mondada; Vatanen), (2) co-present in a virtual space where participants have access to each other through virtual reality (Spets; Haddington, et al.), or (3) interacting in a dispersed, hybrid space where some of the participants are physically co-present and interact with others via a virtual connection (Kohonen-Aho; Katila et al.).

1 On the Complexities of Interaction

11

The chapters explore the multi-layered features and the rich interconnectivity of multimodal resources as an aspect of everyday life and activity through the lens of complexity, focusing on how complexity of interaction resides in 1. multiactivity and multisensoriality (i.e., how participants organise multiple activities and how sensory practices are tied to the organisation of activities), 2. asymmetries related to affordances, resources and roles (i.e., how asymmetrical access to language, knowledge or the multimodal context features in the organisation of activities), 3. the coordination of participation frameworks (i.e., the ways in which participants manage their access and availability to each other and the shared context of interaction), 4. the characteristics of interactional settings and environments (i.e., how the nature of the setting features in the organisation of action and activity). The following sections open up these themes and offer a summary of each chapter in the book.

1.4.1 Complexities That Reside in Multiactivity and Multisensoriality Part I investigates complexity that resides in the participants being engaged in multiple action or activity trajectories simultaneously, possibly with multiple senses. In general, the chapters analyse how actions are composed and interpreted as action complexes or complex multimodal Gestalts (see also Mondada 2014a), and how participants progress multiple courses of action as multiactivity (see also Haddington et al. 2014; Mondada 2014b). They also explore how orientation to senses emerges as central to the organisation of social action. This section begins with studies on interactions in small groups (Mondada; Hofstetter and Robles) and continues to a study on interaction in a large assembly of participants (Vatanen).

12

T. Eilittä et al.

Lorenza Mondada analyses complexity as the intertwinement of multiple temporalities and resources, and approaches intersubjectivity by examining how it is secured for actions that are achieved with the body rather than with language. The chapter studies how participants situatedly and methodically organise, manage, and disentangle complex arrays of resources and practices to produce the fine intelligibility, distinction, and recognition of actions consisting of the ingestion of food. While eating and tasting food seem very similar actions, a closer analysis reveals that they are organised in different ways and treated as doing different things. Furthermore, they support different forms of multiactivity: Eating can be embedded in talk, while tasting can exclude talk. Eating and tasting also progress different interactional trajectories. The distinction between eating and tasting emerges from the local context of the activity, and participants may shift from one to the other and orient to one or the other as normatively expected or not. In sum, the chapter demonstrates that sensorial practices in interaction, which are largely under-described in CA, are accountable and deeply intersubjective. Emily Hofstetter and Jessica Robles examine the ways in which participants engaged in playing board games manage their engagement in two different but simultaneous activities, and how they navigate and organise their actions and participation as either in or out of gameplay. The authors show what kind of talk during dual orientation to game and non-game activities supports or hinders the game’s progression, and in which situations non-game talk is considered problematic and accountable. The phenomenon of metagaming features complexity in that the players demonstrate orientations to multiple possible memberships and categorial relations, as well as to multiple simultaneously ongoing activities. Anna Vatanen shows that orientation to progressivity while engaged in multiple activities is dependent on what is jointly considered the most important activity in the situation. Vatanen’s chapter on repair-initiations—a key resource in restoring intersubjectivity—in choir rehearsals investigates the allocation of interactional resources in a multiactivity setting, where noticings of mistakes are communicated to a nearby co-participant embodiedly during the collective action of choir

1 On the Complexities of Interaction

13

singing. The chapter thus shows how choir singers display subtle orientation to fellow singers’ mistakes during joint singing—or let the mistakes just pass. The chapter also notes that the participatory affordances of the simultaneous activities play a key role in the local organisation of multiactivity.

1.4.2 Complexities That Reside in Asymmetries Related to Affordances, Resources, and Roles Part II explores how asymmetrical (limited, partial, fragmented, or conflicting) access to the same affordances and resources is oriented to and solved in and through interaction. The chapters analyse interaction in situations involving asymmetric access to visual resources such as a virtual object (Spets), to the used language and the multimodal context (Kamunen and Haddington), and to the co-participants’ roles, experience, and knowledge (Rautiainen, Haddington and Kamunen). Heidi Spets examines how participants’ asymmetric perspectives to an object of interest in an immersive virtual reality (VR) affect the progression of an activity. By focusing on a word explanation game, Spets shows how participants employ certain verbal and embodied resources without the full array of affordances of the human body to progress the word explanation sequence and complete the interactional task at hand, aiming at establishing shared understanding. Emerging practices as well as the novelty and variation of available interactional resources contribute to the complexity of VR interaction. The next two chapters in Part II study interactions in multinational crisis management training. Antti Kamunen and Pentti Haddington study interactional episodes in a tactical operations centre in which some participants do not have access to the language spoken or other resources around them that are necessary for building joint understanding. First, the analysis shows how the trainees with no access to the language used rely on available semiotic resources to interpret what the co-participants are saying. Second, the chapter demonstrates that in some situations even if the participants share the same language, interaction between them

14

T. Eilittä et al.

may not be possible due to the lack of shared access to other interactional resources, such as the multimodal context. The chapter describes the complexity of navigating an interactional environment where one’s access to and participant status in the surrounding conversations is not always stable and, at times, needs to be (re)negotiated. Iira Rautiainen, Pentti Haddington, and Antti Kamunen study the complexity of different roles in UN military observer training. The chapter examines how the assigned and assumed roles relate, on the one hand, to the asymmetric deontic rights to make a decision, and to the asymmetries in the trainees’ experience and competence (i.e., their epistemic status) in completing a given task, on the other. The chapter focuses on nudging questions, showing how they are used as devices by more experienced team members to prompt a less-experienced team leader to initiate, adjust, or advance a course of action. The analysis shows how nudging questions display their producer’s orientation to the complex asymmetries between the team members’ experiences, competences, and situated roles. By way of constituting an indirect approach, the questions deal with a social challenge and do face-saving work.

1.4.3 Complexities That Reside in the Coordination of Participation Frameworks Part III investigates complexities that reside in the organisation of participants’ involvement in multiple participation frameworks. The first two chapters explore how technologies (i.e., smartphones and large screens) are embedded in the organisation and accomplishment of activities and participation frameworks. They study situations in which a technology is embedded in a group activity and contributes to changes in the emergent participation framework (Baldauf-Quilliatre and Ursi) and where a participant uses a technological device to transform the participation framework (Avgustis and Oloff ). The final chapter explores the role of affective engagement in progressing multiple participation frameworks simultaneously (Katila, et al.). Heike Baldauf-Quilliatre and Biagio Ursi study interaction during gameplay in front of a large screen. The analysis focuses on the spatial

1 On the Complexities of Interaction

15

organisation of participants and how it is related to the dynamics of participation frameworks. They discuss spatiality, materiality, and temporality as dimensions of interactional complexity and focus specifically on the ways in which the participants jointly navigate the screen, orient to the screen as a fragile object, and address past and future events on the screen. Iuliia Avgustis and Florence Oloff investigate showing sequences in situations in which interaction centres around smartphones. Their chapter examines how speakers and their recipients coordinate multiple foci of attention while manipulating a smartphone as part of a joint activity. The situation involves multiple layers of complexity: the technological affordances of the mobile device (and especially the coparticipants’ lack of access to some of these affordances) as a means to change the structures of participation frameworks, the multiactivity ensuing from their use, and the participants’ coordination of action within multiparty settings. Julia Katila, Sara Goico, Yumei Gan, and Marjorie Goodwin study interaction in situations where participants are progressing different activities while engaged in multiple concurrent participation frameworks. By drawing on excerpts from different interactional contexts, they focus on the emergence and handling of emotions in the above situations. They show that the expression of affect (e.g., conveyed through touch, hand gestures, and prosody) and emotional engagement makes possible individual participants’ access to multiple concurrent participation frameworks.

1.4.4 Complexities That Reside in the Characteristics of Interactional Settings and Environments Part IV explores complexities that reside in interactional settings and environments. The chapters show how a setting—a construction site, virtual reality, or a hybrid workshop—may present features that restrict participants from (fully) seeing or hearing each other and how participants coordinate and adjust their actions to overcome such problems for establishing mutual availability and shared understanding of the ongoing

16

T. Eilittä et al.

activity. The chapters study complexity also through different facets of co-presence. First, openings to interactions are studied in an environment where the participants are physically co-present but mobile (Hoey). Second, openings are studied in a setting where participants are copresent simultaneously as virtual avatars and co-participants in the same physical space (Haddington, et al.). Third, the coordination of joint activity is studied in a setting where several participants are physically distributed to different locations but aim to establish co-presence with one another through technology (Kohonen-Aho). Elliott Hoey studies how participants approach one another and open conversations in construction sites including various mobile, material, and spatial configurations. Hoey addresses the practical complexities created by this material environment by showing that in settings where participants are not necessarily able to see or hear each other, they can manage the challenges in establishing and achieving a shared interactional space by moving closer to each other or repositioning themselves within the physical space. Hoey identifies and analyses three practices for approaching and initiating interaction in a construction site: direct, oblique, and restricted approaches. Pentti Haddington, Laura Kohonen-Aho, Sylvaine Tuncer, and Heidi Spets analyse interactional openings between virtual avatars in immersive virtual reality (VR). They analyse opening episodes in which the future participants, through the ways in which they talk and use their avatar bodies, orient to and pre-empt complexities involved in establishing mutual orientation in VR. The analysis focuses on practices by which virtual participants identify (as an unacquainted interactional partner) or recognise (as an acquaintance) each other in order to establish an encounter. The authors argue that the organisation of the interactional openings reflects the fragmented nature of VR as an interactional setting. In the final chapter, Laura Kohonen-Aho studies the giving and receiving of instructions in a hybrid interaction setting with multiple participants and communication technologies present. The chapter analyses challenges that a team receiving instructions via a remote connection—in contrast to a team that is co-located with the participant giving the instructions—encounters during instruction-giving. The difficulties are connected to the asymmetries between the different physical, virtual,

1 On the Complexities of Interaction

17

public, and private interactional spaces and the transitions between these spaces.

1.5

Conclusions

An important proportion of conversation analysis has focused on the study of vocal and verbal features of talk-in-interaction. Talk has been— rightly and naturally so—considered as the central resource that people draw on to build common ground and shared understanding when they interact. However, in addition to talk, participants use embodied and other multimodal features as resources for producing and interpreting social actions. Together with talk, also they are tools with which humans accomplish social practices and intersubjective understanding as parts of their everyday lives. Indeed, not nearly all CA research has focused only on the features of talk. An expanding segment of CA, which is often called multimodal CA, studies how participants, in addition to talk, use and rely on embodied and multimodal resources to interact meaningfully with each other. In particular, the work by Charles and Marjorie Goodwin, and Lorenza Mondada, has had a ground-breaking contribution to the development of multimodal CA. This is also the context from which the studies in this volume emerge. They study how social participants use the diverse array of interactional resources (e.g., vocal, verbal, multimodal, spatial, and technological) in social interaction. Furthermore, the book’s chapters use the term “complexity” to highlight the richness of the ways in which interactants build common ground and shared understanding in multimodal interaction. Complexity incorporates a broad array of phenomena that are often investigated and analysed individually, regardless of others. However, the studies in this volume suggest that examining the interplay and interconnections between resources, progressivity, and participation, and their possible internal complex features adds to what we currently know about how people establish and maintain intersubjectivity in interaction. The chapters contribute to the understanding of complexity and social interaction in four ways. First, they study how complexities reside in multiactivity and multisensoriality in interaction. The chapters explore

18

T. Eilittä et al.

how interactants orient to and advance multiple actions and activities (i.e., sequential or temporal trajectories) simultaneously by using a diverse and multi-layered array of interactional resources, including senses. Second, they study how complexities reside in participants’ asymmetric access to interactional affordances, roles, and resources. By drawing on video recordings from many understudied contexts (e.g., immersive virtual reality and multinational crisis management training), the chapters explore the ways in which asymmetrical access to the same affordances and resources is oriented to and solved in interaction. Third, the volume studies how complexity resides in the organisation of participants’ involvement in multiple simultaneous participation frameworks (e.g., across groups and technological environments). The chapters examine situations where technology-centred interaction as well as affective expressions contribute to the formation and coordination of multiple, even simultaneous, participation frameworks. Fourth, the volume studies how complexities reside in the interactional settings or environments that are particularly complex for participants in terms of maintaining co-presence and mutual access. The chapters explore how participants orient to and overcome challenges in establishing mutual availability and shared understanding in settings that are mobile, hybrid, and artificial (virtual). The above-mentioned features of complexity are not isolated or independent of each other but closely intertwined in action. While the chapters take one particular feature of complexity as their starting point, the developing analyses reveal how social action and interaction—and joint meaning-making—can involve the interplay and interconnection of several features and complexities at the same time: the intelligibility and accountability of action and activity rely on the available resources; resources, in turn, are inherent elements of the local interactional environment; and the local interactional environment is the natural site for specific actions and activities. In sum, even though CA has previously studied complex interactions and situations to some extent (see Sect. 1.2), the area remains understudied. This book provides a fresh vantage point to social interaction by deliberately attending to the intricate details that constitute and manifest as complexity of interaction. The chapters suggest that

1 On the Complexities of Interaction

19

studying social interaction systematically through the analytic of lens of multi-layered action and complexity can reveal new interactional practices and phenomena. The book also aims to provide analysts with tools and concepts that help, not only to understand, but also to make visible and analyse the complexity of interaction. In this respect, the volume is a manifesto for complexity; science often simplifies the analyses and descriptions of the focus phenomena in order to make them “more reportable.” However, the current volume represents a different approach: It embraces complexity instead of turning to simplifications for the sake of generalisability. Acknowledgments We are grateful to Marika Helisten, Mirka Rauniomaa, and the contributors to this volume their insightful feedback and critical comments on earlier drafts of this chapter. We are also indebted to Lorenza Mondada for important comments and thoughts that have shaped and specified our understanding of “complexity” and helped us to formulate some of the arguments in the chapter.

References Barnes, Scott. 2016. “Aphasia and Open Format Other-Initiation of Repair: Solving Complex Trouble in Conversation.” Research on Language and Social Interaction 49 (2): 111–127. https://doi.org/10.1080/08351813.2016.116 4399. Broth, Mathias. 2009. “Seeing Through Screens, Hearing Through Speakers: Managing Distant Studio Space in Television Control Room Interaction.” Journal of Pragmatics 41: 1998–2016. Cekaite, Asta. 2015. “The Coordination of Talk and Touch in Adults’ Directives to Children: Touch and Social Control.” Research on Language and Social Interaction 48: 152–175. Cekaite, Asta. 2016. “Touch as Social Control: Haptic Organization of Attention in Adult-Child Interactions.” Journal of Pragmatics 92: 30–42. Dahl, Östen. 2004. The Growth and Maintenance of Linguistic Complexity. Amsterdam: John Benjamins Publishing.

20

T. Eilittä et al.

Deppermann, Arnulf. 2014. “Multimodal Participation in Simultaneous Joint Projects.” In Multiactivity in Social Interaction: Beyond Multitasking, ed. by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 247–281. Amsterdam: John Benjamins Publishing. Deppermann, Arnulf. 2019. “Intersubjectivity and Other Grounds for ActionCoordination in an Environment of Restricted Interaction: Coordinating with Oncoming Traffic When Passing an Obstacle.” Language & Communication 65: 22–40. https://doi.org/10.1016/j.langcom.2018.04.005. Di Garbo, Francesca, Bruno Olsson, and Bernhard Wälchli. 2019. Grammatical Gender and Linguistic Complexity. Studies in Diversity Linguistics 26. Berlin: Language Science Press. Enfield, Nick J., and Jack Sidnell. 2022. Consequences of Language: From Primary to Enhanced Intersubjectivity. Cambridge, MA: The MIT Press. https://doi.org/10.7551/mitpress/14795.001.0001. Field, Tiffany M. 2001. Touch. Cambridge MA: MIT Press. Gibson, James J. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin. Goffman, Erving. 1964. “The Neglected Situation.” American Anthropologist, New Series 66 (6): 133–136. Goffman, Erving. 1978. “Response Cries.” Language 54 (4): 787–815. Goodwin, Charles. 1986. “Gestures as a Resource for the Organization of Mutual Orientation.” Semiotica 62: 29–49. Goodwin, Marjorie H. 1996. “Informings and Announcements in Their Environment: Prosody Within a Multi-Activity Work Setting.” In Prosody in Conversation: Interactional Studies, ed. by Elizabeth Couper-Kuhlen and Margret Selting, 436–461. Cambridge: Cambridge University Press. Goodwin, Marjorie H. 1997. “Byplay: Negotiating Evaluation in Storytelling.” In Towards a Social Science of Language: Papers in Honor of William Labov Vol. 2, ed. by Gregory R. Guy, Crawford Feagin, Deborah Schiffrin, and John Baugh, 77–102. Amsterdam: John Benjamins Publishing. Goodwin, Charles. 2000. “Action and Embodiment Within Situated Human Interaction.” Journal of Pragmatics 32 (10): 1489–1522. https://doi.org/10. 1016/S0378-2166(99)00096-X. Goodwin, Charles. 2006. “Human Sociality as Mutual Orientation in a Rich Interactive Environment: Multimodal Utterances and Pointing in Aphasia.” In Roots of Human Sociality: Culture, Cognition and Interaction, ed. by Nick J. Enfield and Stephen C. Levinson, 97–125. Oxford: Berg. Goodwin, Charles. 2007. “Environmentally Coupled Gestures”. In Gesture and the Dynamic Dimension of Language: Essays in Honor of David McNeill ,

1 On the Complexities of Interaction

21

ed. by Susan D. Duncan, Justine Cassell, and Elena Terry Levy, 195–212. Amsterdam: John Benjamins Publishing. Goodwin, Charles. 2013. “The Co-operative, Transformative Organization of Human Action and Knowledge.” Journal of Pragmatics 46 (1): 8–23. https:/ /doi.org/10.1016/j.pragma.2012.09.003. Goodwin, Marjorie H. 2017. “Haptic Sociality: The Embodied Interactive Constitution of Intimacy Through Touch.” In Intercorporeality: Emerging Socialities in Interaction, ed. by Christian Meyer, Jürgen Streeck, and J. Scott Jordan, 73–102. Oxford: Oxford University. https://doi.org/10.1093/acprof: oso/9780190210465.003.0004. Goodwin, Charles. 2018. Co-operative Action. Cambridge: Cambridge University Press. Goodwin, Charles, and Marjorie H. Goodwin. 2004. “Participation.” In A Companion to Linguistic Anthropology, ed. by Alessandro Duranti, 222–244. Oxford: Blackwell. Goodwin, Marjorie H., Charles Goodwin, and Malcah Yaeger-Dror. 2002. “Multi-modality in Girls’ Game Disputes.” Journal of Pragmatics 34 (10– 11): 1621–1649. Haakana, Markku, Salla Kurhila, Niina Lilja, and Marjo Savijärvi. 2021. “Extending Sequences of Other-Initiated Repair in Finnish Conversation.” In Intersubjectivity in Action. Studies in Language and Social Interaction, ed. by Jan Lindström, Ritva Laury, Anssi Peräkylä, and Marja-Leena Sorjonen, 231–250. Amsterdam: John Benjamins Publishing. Haddington, Pentti, Lorenza Mondada, and Maurice Nevile. 2013. “Being Mobile: Interaction on the Move.” In Interaction and Mobility: Language and the Body in Motion, ed. by Pentti Haddington, Lorenza Mondada, and Maurice Nevile, 3–61. Berlin: Walter de Gruyter. Haddington, Pentti, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile (eds). 2014. Multiactivity in Social Interaction: Beyond Multitasking. Amsterdam: John Benjamins Publishing. Helisten, Marika. 2019. “Disjunctively Positioned Problem-Noticings in Managing Multiactivity.” Research on Language and Social Interaction 52 (4): 318–342. https://doi.org/10.1080/08351813.2019.1657274. Hepburn, Alexa, and Galina Bolden. 2017. Transcribing for Social Research. Los Angeles: Sage. Heritage, John. 1984. Garfinkel and Ethnomethodology. Cambridge: Polity Press. Heritage, John, and J. Maxwell Atkinson. 1984. “Introduction.” In Structures of Social Action, ed. by J. Maxwell Atkinson and John Heritage, 1–16. Cambridge: Cambridge University Press.

22

T. Eilittä et al.

Hutchins, Edwin. 1995. Cognition in the Wild . Cambridge, MA: MIT Press. Hutchins, Edwin, and Tove Klausen. 1996. “Distributed Cognition in an Airline Cockpit.” In Cognition and Communication at Work, ed. by Yrjö Engeström and David Middleton, 15–34. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139174077.002. Jefferson, Gail. 2004. “Glossary of Transcript Symbols with an Introduction.” In Conversation Analysis: Studies from the First Generation, ed. by Gene Lerner, 13–31. Amsterdam: John Benjamins Publishing. Kamunen, Antti. 2019. “How to Disengage: Suspension, Body Torque, and Repair.” Research on Language and Social Interaction 52 (4): 406–426. https:/ /doi.org/10.1080/08351813.2019.1657287 Kamunen, Antti. 2020. Busy Embodiments: The Hierarchisation of Activities in Multiactivity Situations. Tampere: University of Oulu. Kamunen, Antti, and Pentti Haddington. 2020. “From Monitoring to Comonitoring: Projecting and Prompting Activity Transitions at the Workplace.” Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion 21: 82–122. Keating, Elizabeth. 2005. “Homo Prostheticus: Problematizing the Notions of Activity and Computermediated Interaction.” Discourse Studies 7: 527–545. Kurhila, Salla. 2001. “Correction in Talk Between Native and Non-native speaker.” Journal of Pragmatics 33: 1083–1110. Laakso, Minna S.M. 2000. “Self-Initiated Repair by Aphasic Speakers in Home and Aphasia Therapy Conversations.” Journal of Neurolinguistics 13 (4): 258–260. Lilja, Niina. 2010. Ongelmista oppimiseen. Toisen aloittamat korjausjaksot kakkoskielisessä keskustelussa [Other-Initiated Repair Sequences in Finnish Second Language Interactions]. Jyväskylä Studies in Humanities 146. Jyväskylä: University of Jyväskylä. Linell, Per. 2017. “Intersubjectivity in Dialogue.” In The Routledge Handbook of Language and Dialogue, ed. by Edda Weigand, 109–126. New York: Routledge. Luff, Paul, Jon Hindmarsh, and Christian Heath (eds). 2000. Workplace Studies: Recovering Work Practice and Informing System Design. Oxford: Cambridge University Press. Luff, Paul, Christian Heath, Hideaki Kuzuoka, Jon Hindmarsh, Keiichi Yamazaki, and Shinya Oyama. 2003. “Fractured Ecologies: Creating Environments for Collaboration.” Human-Computer Interaction 18 (1): 51–84. https://doi.org/10.1207/S15327051HCI1812_3.

1 On the Complexities of Interaction

23

Markaki, Vassiliki, Sara Merlino, Lorenza Mondada, Florence Oloff, and Véronique Traverso. 2013. “Multilingual Practices in Professional Settings: Keeping the Delicate Balance Between Progressivity and Intersubjectivity.” In Multilingualism and Diversity Management Vol. 2, ed. by Anne-Claude Berthoud, François Grin, and Georges Lüdi, 3–32. Amsterdam: John Benjamins Publishing. https://doi.org/10.1075/mdm.2.01mar. Meyer, Christian. 2017. “The Cultural Organization of Intercorporeality, Interaction, Emotion and the Senses Among the Wolof of Northwestern Senegal.” In Intercorporeality: Emerging Socialities in Interaction, ed. by Christian Meyer, Jürgen Streeck, and J. Scott Jordan, 143–171. Oxford: Oxford University Press. Mondada, Lorenza. 2007. “Multimodal Resources for Turn-taking: Pointing and the Emergence of Possible Next Speakers.” Discourse Studies 9: 194– 225. Mondada, Lorenza. 2009. “Emergent Focused Interactions in Public Places: A Systematic Analysis of the Multimodal Achievement of a Common Interactional Space.” Journal of Pragmatics 41: 1977–1997. Mondada, Lorenza. 2011. “The Organization of Concurrent Courses of Action in Surgical Demonstrations.” In Embodied Interaction: Language and Body in the Material World , ed. by Jürgen Streeck, Charles Goodwin, and Curtis LeBaron, 207–226. Cambridge: Cambridge University Press. Mondada, Lorenza. 2013. “Coordinating Mobile Action in Real Time: The Timely Organisation of Directives in Video Games.” In Interaction and Mobility: Language and the Body in Motion, ed. by Pentti Haddington, Lorenza Mondada, and Maurice Nevile, 300–342. Berlin: De Gruyter. Mondada, Lorenza. 2014a. “The Local Constitution of Multimodal Resources for Social Interaction.” Journal of Pragmatics 65: 137–156. Mondada, Lorenza. 2014b. “The Temporal Orders of Multiactivity.” In Multiactivity in Social Interaction: Beyond Multitasking, ed. by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 33–75. Amsterdam: John Benjamins Publishing. Mondada, Lorenza. 2018. “Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing Multimodality.” Research on Language and Social Interaction 51 (1): 85–106. Mondada, Lorenza. 2019. “Contemporary Issues in Conversation Analysis: Embodiment and Materiality, Multimodality and Multisensoriality in Social Interaction.” Journal of Pragmatics 145: 47–62. Mondada, Lorenza, Sofian A. Bouaouina, Laurent Camus, Guillaume Gauthier, Hanna Svensson, and Burak S. Tekin. 2021. “The Local and Filmed

24

T. Eilittä et al.

Accountability of Sensorial Practices.” Social Interaction: Video-Based Studies of Human Sociality 4 (3): 2446–3620. https://doi.org/10.7146/si.v4i3. 128160. Nevile, Maurice. 2004. Beyond the Black Box: Talk-in-interaction in the Airline Cockpit. Aldershot: Ashgate. Nevile, Maurice. 2009. “‘You Are Well Clear of Friendlies’: Diagnostic Error and Cooperative Work in an Iraq War Friendly Fire Incident.” Computer Supported Cooperative Work (CSCW) 18 (2–3): 147–173. Nishizaka, Aug. 2007. “Hand Touching Hand: Referential Practice at a Japanese Midwife House.” Human Studies 30 (3): 199–217. Nishizaka, Aug. 2011. “Touch Without Vision: Referential Practice in a Nontechnological Environment.” Journal of Pragmatics 43: 504–520. Pajo, Kati, and Minna Laakso. 2020. “Other-Initiation of Repair by Speakers with Mild to Severe Hearing Impairment.” Clinical Linguistics & Phonetics 34 (10–11): 998–1017. Peräkylä, Anssi. 2013. “Conversation Analysis in Psychotherapy.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 551–574. Chichester: Wiley-Blackwell. Raymond, Chase. 2019. “Intersubjectivity, Normativity and Grammar.” Social Psychology Quarterly 82 (2): 182–204. https://doi.org/10.1177/019027251 9850781. Sacks, Harvey. 1984. “Notes on Methodology.” In Structures of Social Action: Studies in Conversation Analysis, ed. by John Heritage and J. Maxwell Atkinson, 2–27. Cambridge: Cambridge University Press. Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson. 1974. “A Simplest Systematics for the Organization of Turn-Taking for Conversation.” Language 50: 696–735. Schegloff, Emanuel A. 1992. “Repair after Next Turn: The Last Structurally Provided Defense of Intersubjectivity in Conversation.” American Journal of Sociology 97 (5): 1295–1345. https://doi.org/10.1086/229903. Schegloff, Emanuel A. 2007. Sequence Organization in Interaction: A Primer in Conversation Analysis. Vol. 1. Cambridge: Cambridge University Press. Schegloff, Emanuel A., Gail Jefferson, and Harvey Sacks 1977. “The Preference for Self-Correction in the Organization of Repair in Conversation.” Language 53: 361–382. Sorjonen, Marja-Leena, Anssi Peräkylä, Ritva Laury, and Jan Lindström. 2021. “Intersubjectivity in Action: An Introduction.” In Intersubjectivity in Action. Studies in Language and Social Interaction, ed. by Jan Lindström, Ritva

1 On the Complexities of Interaction

25

Laury, Anssi Peräkylä, and Marja-Leena Sorjonen, 1–22. Amsterdam: John Benjamins Publishing. Stivers, Tanya, and Jack Sidnell. 2005. “Introduction: Multimodal Interaction.” Semiotica 156 (1): 1–20. Streeck, Jürgen. 2009. Gesturecraft: The Manufacture of Meaning. Amsterdam: John Benjamins Publishing. Streeck, Jürgen, Charles Goodwin, and Curtis LeBaron (eds.). 2011. Embodied Interaction: Language and Body in the Material World . Cambridge: Cambridge University Press. Stukenbrock, Anja. 2021. “Multimodal Gestalts and Their Change over Time: Is Routinization Also Grammaticalization?” Frontiers in Communication 6: 662240. https://doi.org/10.3389/fcomm.2021.662240. Throop, C. Jason. 2012. “On the Varieties of Empathic Experience: Tactility, Mental Opacity, and Pain in Yap”. Medical Anthropology Quarterly 26 (3): 408–430.

Part I Complexity That Resides in Multiactivity and Multisensoriality

2 Tasting vs. Eating: The Methodic and Situated Differentiation of Embodied Multisensorial Activities in Social Interaction Lorenza Mondada

2.1

Introduction

Ordinary language refers to ingesting food with different verbs, such as eat, guzzle, munch, and taste (in French manger, bouffer, dévorer, déguster … in Italian mangiare, inghiottire, ingozzare, trangugiare, degustare…, in German essen, fressen,…). These verbs refer semantically to different ways of absorbing food and drinks that are socially and normatively accountable, recognizable by the participants within socio-interactional activities. The corresponding actions constitute interesting phenomena for reflecting on the embodied dimension of action formation and action ascription, and the accountability of action, concerning both its intelligibility and normativity. While most categorizations of actions reflected upon in the literature refer to talk (such as requests, offers, invitations, requests for information, complaints…), continuing (often in a critical L. Mondada (B) University of Basel, Basel, Switzerland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_2

29

30

L. Mondada

way) the heritage of speech act theory, other actions, silently embodied, have been largely neglected. Even if they can be (but are not always) prospectively announced or retrospectively assessed with corresponding lexical formulations, these actions are often silently performed: they constitute an occasion to reflect on how participants locally and endogenously treat action formation and ascription of embodied conducts, and indeed organize their interactions by orienting to subtle differentiations between apparently similar actions. This chapter addresses the difference between eating (and drinking ) vs. tasting as practices methodically distinguished by the participants in social interaction, witnessable and witnessed as such. Addressing this distinction is an occasion to discuss several challenges of multimodal analysis: the identification of embodied complex multimodal Gestalts, their distinctive character and accountability, their situated details adjusting to the specificity of the context, and their systematicity across contexts, their temporal organization and embeddedness in the current activity, be in terms of multiactivity or not, and the recognizability and recognition by the co-participants. Moreover, conducts like eating vs. tasting raise specific issues related to corporeality: food practices are still poorly addressed in EMCA as far as their embodiment is concerned. They relate to the bio-sociality of bodies, that is, the physiological dimensions of the flesh and the normative and socio-cultural discipline of the body. They also incarnate multisensorial relations to materiality, which are often defined in merely private ways, although sometimes associated with socialization, culture, and social distinction. Thus, interrogating the distinction between eating vs. tasting addresses some of key contemporary issues in EMCA related to multiactivity, multimodality, and multisensoriality. Within the perspective of this book about complexity, in their apparent simplicity, practices of eating vs. tasting address issues that have been considered as complex from the perspective of the analyst— such as the transcription, annotation and analysis of multimodality, multisensoriality, and multiactivity, the local accountability and recognizability of embodied practices, their intertwined multiple resources and temporalities. This chapter reflects about how participants situatedly and methodically organize, manage, and disentangle these webs

2 Tasting vs. Eating

31

of resources and practices, how the simplest engagement in tasting as an individual practice is delicately supported and made possible by the collective achievement of a specific interactional space and participation framework, and how the multiple temporalities characterizing the multiactivity of eating and talking are organized in unproblematic and almost unnoticed way by those who engage in it. This chapter also shows how practices that have been treated as private—such as multisensorial practices involving the body and the materiality of the world—are in fact interactionally accomplished and therefore intersubjectively accountable, even when performed in a solitary way, thus showing the non-straightforwardness of dichotomies such as private/ public, and individual/social.

2.2

Embodied Action Formation and Ascription: Food Practices in Social Interaction

Food practices have been often invoked in studies of social interaction, but often ignored as involving the bodies of the participants. In the analysis of dinner conversations, talk has been mostly favored over food. When food practices have been considered, this has been often with a focus on their verbal or sounding outcomes—such as assessments and response cries (Mondada, 2009a; Wiggins, 2002, 2013, 2019) rather than the action of eating itself. Another series of approaches has been interested in the sequential positioning of the food or drinking practices, relatively to the organization of talk (such as in disagreement sequences, Mondada, 2009a, in closing environments, Laurier 2008, in slots for responses, Hoey, 2018). Even recent studies of tasting activities (Fele, 2016; Liberman, 2013, 2018) have mostly focused the social activities of assessing, categorizing, and describing taste, despite attempts to integrate the body, from global body postures to the details of movements of the hand or the mouth. The embodied, carnal, and physiological dimensions of ingesting food

32

L. Mondada

and liquids, of eating, drinking, and tasting, are still being neglected— as if the nose and the mouth, would escape multimodal descriptions in terms of sipping, sniffing, slurping, masticating, chewing, chomping, crunching, and swallowing (but see Fele, 2019; Mondada, 2018a, 2020, 2021). Recent studies of the multisensoriality of food activities (Mondada, 2021) attempt to integrate these aspects within an analysis of sensorial practices that cannot be reduced to mere physiological, internal, private sensations, and experiences. Rather, the interactional approach to sensoriality in social interaction shows how it is intersubjectively organized and accomplished by all participants, even when only one person is engaged in tasting, smelling, or touching (Mondada, 2018a, 2021). Food activities have been also addressed in relation to multiactivity (Haddington et al., 2014): early comments by Goodwin (1984: 228– 229, 232) point at problems of coordination between eating and talking, referring to the issue of how both are organized, hierarchized, and intertwined in dinner conversations (Mondada, 2009a, 2009b; Hoey, 2018). Fine-tuned ways of orchestrating multiactivity address the fact that eating/drinking activities mobilize body resources (e.g., the mouth) in ways that compete with talking activities, thus implying either exclusive or delicately embedded ways of organizing multiactivity. Multiactivity also raises the issue of action formation and action ascription: how do participants recognize more than one activity in multiactivity? More particularly, how are embodied actions made recognizable by the doers and actually recognized by the co-participants? The issues of action formation and action ascription have been widely discussed in relation to recognizable linguistic formats (Deppermann & Haugh, 2022; Levinson, 2012) but much less to embodied silent formats: how is one embodied action intersubjectively, methodically, and orderly produced in such a way to be distinguished from others? How is this action also locally recognized and ascribed as such? Activities such as eat, guzzle, and taste are routinely and common-sensically distinguished by ordinary people and have normative judgments associated to them— but little is known about how to describe them in systematic, detailed, and multimodal ways as they are bodily achieved in social interaction. This study shows the complex multi-layered temporal way in which

2 Tasting vs. Eating

33

multimodal resources are assembled to produce these distinctive actions, and how this is oriented to and recognized by the participants in situ. Tasting and eating have been discussed by pragmatist field studies, which have pinpointed the specific ecology of tasting sessions: the sample is presented in the form of a tiny bit or a few drops, inviting tasters to focus on quality rather than quantity, which often is not entirely ingested (e.g., wine is spat out). Each participant accesses their own samples. These material arrangements prompt a distinct, individual appraisal of tasting, privileging knowledge over pleasure (Perullo, 2016). By contrast, in eating, the entire dish is offered, in its complexity, shared with other table companions and enjoyed together with table conversation. Comparing music lovers and wine amateurs, Hennion (2005, 2007, 2015) defines tasting as an activity relying on a full attention on taste, rather than a dispersed involvement in the surrounding sociability. The specific engagement of the wine taster as an amateur involves a detachment, suspension, and pausing from what is happening around and a total concentration on taste. Hennion defines tasting in terms of ‘attention’ favoring the ‘stronger presence of the object being tasted’ (2007: 108), and the bracketing of everything else (‘The ordinary state is this spontaneous management of multiple relationships to one’s body, to others, to things, to events, rather than in the univocal installation of oneself into a rapport with a definitively delineated object’ 2007: 105). Converging with findings in EMCA (Mondada 2018a, 2018b, 2021), these descriptions call for further analyses: they tend to focus on the taster alone, and say little of the finely-tuned coordination between the taster and other participants—as if others would be simply put into parentheses. Previous analyses of the ‘exclusive mode’ of multiactivity (Mondada, 2014) showed that the exclusive focus on one and only activity requires interactional work; and analyses of tasting moments demonstrated that they are collectively established and supported by the co-participants, as well as actively monitored by them, in a way that configures tasting not just as an individual separated moment, but as a publicly witnessable one (Mondada, 2018a). This paper thus shows how activities of food ingestion are situated social achievements of all the participants (including those who do not eat or taste), who collectively build the intelligible and normative accountability of these practices, and

34

L. Mondada

differentiate between them, in a way that is consequential for what is made relevant and what comes next. So, the questions explored in this chapter concern several analytical layers: the issue of action formation and action ascription is developed in relation to the distinction between tasting vs. eating as it is accomplished and seen by the participants. This enables an analysis of the accountability of embodied practices in their details, characterizing the engagement of the body as flesh, involved in activities related to food and the senses. This also grounds a discussion about the intersubjectivity of these embodied sensual moments, and the relation between individual practices and their public and intersubjective dimensions as they are witnessed by others within the contextual relevances of the ongoing activity. Distinctive practices of tasting vs. eating/drinking constitute perspicuous phenomena inviting us to explore their specific multimodal organizations, and as they are oriented to by the participants and as they are discoverable by the analyst.

2.3

The Methodic Organization of Tasting vs. Eating Across Settings

The analyses focus on three very different settings in which food activities were video-recorded in situ: a workday in a professional kitchen in Spain, commercial encounters at a market in Alsace (France), and dinners in an upper-scale restaurant in Lyon (France). In these settings, tasting is not the main activity (by contrast with tasting sessions), but rather an occasioned moment within the activity, in which participants engage in tasting for specific purposes: in the restaurant kitchen, the cooks and the manager taste new dishes while creating a new menu; at the market, tasting is offered by the seller to stop passers-by at his stand and possibly engage them in buying something; at the restaurant, the customer tastes the wine suggested by the sommelier in order to accept or refuse it. Even in sequential environments in which tasting (and not just eating) is projected, and normatively expected by the previous action, like in offers to taste, the response is not always tasting. Participants have choices regarding the way they configure their ingestion of food/drink: they can

2 Tasting vs. Eating

35

format their ingestion as either ‘eating’/ ‘drinking’ or as ‘tasting’; they can configure their action as accountably doing tasting, but also as doing eating/drinking. Moreover, they can also formulate their own action as tasting or eating/drinking—although the accountability of these actions does not rely on such a formulation, but rather on embodiment. Thus, the analyses approach the distinction between these two practices, as they are locally formatted in accountable ways by the doers performing them, and as they are methodically recognized and oriented to as distinct by the co-participants. The focus of the analysis on a diversity of contexts shows its systematicity. The focus on institutional contexts further accentuates the contrast between these practices, which are however observable in unformal settings too. The fine-tuned distinction between tasting vs. eating has been systematically described on the basis of another context, gourmet food shops (Mondada, 2018a: 753, 759, 2021: chap. 8), in which buyers and sellers create together a specific interactional space and embodied participation framework for the activity of tasting cheese. In a nutshell, tasting is locally established and achieved as a sensorial activity in which the taster (the customer) engages when all the parties withdraw from talk, creating a moment of individual focus on the sample characterized by the taster’s absent gaze or gaze on the sample tasted, a particular way of vigorously chewing, and a static body posture. This individual engagement is nonetheless a public one, since the tasting experience is observable for/observed by others (the seller) witnessing the visible movements of the cheeks, the tongue, and the mouth. Thus, the tasting experience is conducted in a specific interactional arrangement that is actively achieved by all co-participants, securing a distinct moment for the taster to silently and individually taste the sample and for the others to witness it. This grounds the possibility of the intersubjective access to the tasting experience: others can access to the fact that someone is sensorially engaging in that experience and often to the judgments and emotions (such as pleasure) emerging and resulting from that experience, which are projectable, anticipable, and witnessable. In this chapter I expand on this, by demonstrating that this methodic accomplishment of the spatio-temporal conditions of tasting is observable across a diversity of tasting activities. I also show that tasting is

36

L. Mondada

accountably formatted and recognized in a way that contrasts with the practice of eating/drinking. The latter way of ingesting food and liquids are ordinary practices organized and embodied in a distinct way, within a very different interactional space and participation framework, rather characterized by the continuous engagement of the person eating or drinking in the ongoing talk and social activity. In other words, the contrast between the two is based on the fact that tasting is configured by all parties as a focus on the sensorial experience, excluding other activities, first and foremost talk; whereas eating/drinking is configured as a multiactivity in which the person eats/drinks and participates to the ongoing sociable activity in an embedded way, without withdrawing from it.

2.3.1 Tasting vs. Eating a New Dish in the Restaurant’s Kitchen The distinction between tasting and eating is sometimes not only bodily achieved but also formulated in so-many-words by the participants. For instance, these activities are explicitly mentioned in offers, invitations, and requests. The first setting observed, features regularly invitations to taste, as well as some announcements prefacing eating. The data are taken from a video-recorded week of work in the kitchen of a creative restaurant in the island of Menorca, in Spain. We focus on a session in which the two cooks (Marta and Elisa) and their manager (Leandro) taste possible dishes to be served to the customers in the upcoming season. The cooks prepare samples of the dishes and the team tastes them, critiquing them and elaborating on possible improvements. We join the action as the chef, Marta, brings the dish on the table, producing an invitation to taste (using the verb probar literally ‘try’, ‘test’, meaning ‘taste’):

2 Tasting vs. Eating

Excerpt 1 (MAOT1_8-05_pruebaprueba)

37

38

L. Mondada

1

2

3

The three participants look at the sample, which is a filling with fish that will feature as part of a dish. It is positively assessed by Leandro (9) on the basis of its visual appearance, which Marta aligns with (10). Next, he cuts a tiny bit to taste (11) and this enables him to feel the consistency, which is manifested by a response cry (12). This delays the tasting, which is urged by the cook, with a double imperative (15). He cuts another tiny sample and looks at Marta, who repeats the double imperative (17). At this point, Leandro looks down at the dish (as do the other coparticipants, Fig. 1), takes a bit in his mouth and begins to chew,

2 Tasting vs. Eating

39

looking into the void/at the dish. His chewing is continuously monitored by Marta and Elisa (Fig. 2) and lasts several seconds. After 8 seconds, Leandro does a big eyebrow, before swallowing and exploring his tongue with his mouth—completing the tasting proper. He also grimaces, turning toward Marta (19). His facial expressions are seen by the co-participants (Fig. 3), who bust into laughter (20–21), before he actually produces his verbal positive assessment (22). Tasting is here explicitly formulated in the invitation to taste; it is collectively achieved by the participants who silently either engage in tasting (Leandro) or observe the taster (Marta, Elisa). Leandro tastes by focusing on the remaining sample, immobile. Tasting is publicly displayed by him, through his manifested concentration, the vigorous and thereby visible chewing; its completion is displayed by the swallowing, and by the eyebrow, projecting a positive assessment. The co-participants observe him silently and quietly until they see tasting coming to completion—visibly achieved by Leandro turning to Marta. They respond to the positive judgment before it is uttered in so-manywords, orienting to its embodiment and treating it as intersubjectively available. This methodic accomplishment, collectively supported and publicly accountable, constitutes the complex multimodal Gestalt typical of tasting that we observed in very different settings and languages (cf. for the same methodic unfolding in another context than the ones studied here, Mondada, 2018a, 2021: chap. 8–9). In some cases, it is contrasted with another Gestalt, typical of eating (or drinking, cf. below). Both can be formulated by the participants, like in the following case. Marta has brought on the table a preparation wrapped in a cabbage and she and Elisa have just tasted it. Leandro hasn’t yet touched it and is invited to taste (1). Later on, Marta announces that she will eat the remaining of the dish (Excerpt 2b).

40

L. Mondada

Excerpt 2a (MAOT1_32-10 pruebalo/lo voy a comer)

In response to Marta inviting him to taste (1) Leandro does not only produce an affirmative token (2) but actually engages in taking the wrap with some sticks (3) encountering some problem in grasping them, signaled through a response cry, and (4) and puts it in his mouth (5, Fig. 4). He chews vigorously and visibly for about 9 seconds, observed by the co-participants (Fig. 5), and then turns to Marta, while chewing in a slower and less pronounced way, before uttering a gustatory mm (Wiggins, 2002). This is enough for both cooks, who laughs and return to the kitchen. Thus, we observe that the tasting unfolds in a similar way as in extract 1. Next, after some discussion, Marta announces what she

2 Tasting vs. Eating

41

is about do to, formulating it with the verb comer ‘eat’ (30). She then finishes the rests of the dish: Excerpt 2b (MAOT1_32-10 pruebalo/lo voy a comer)

.

42

L. Mondada

While Marta puts the remains of the dish into her mouth, Leandro continues to talk about possible variations of the dish, looking down at the (now empty) plate. Elisa looks at him as main speaker and also aligns with him (Fig. 6). During Leandro’s extended turn, Marta continues to chew. Her body moves, to and away from the table, as she grasps a napkin, cleans the table in front of her, and puts it back (Fig. 7). She eats while visibly following the discussion and displays some alignment with what is said (39, 46). She is not observed by any of the co-participants, who ignore her engagement with food. Eating is not witnessed, it might be seen but remains unnoticed, and Marta’s contribution to the discussion does not depend on it. Eating happens here at the end of the moment devoted to the dish, after tasting has happened. In most of this

2 Tasting vs. Eating

43

activity, what remains of the dish/samples is not eaten but thrown away. Eating is configured and formatted in a very distinct way than tasting– not only as far as its public dimension is concerned (others do not pay attention to it happening) but also its accountability (chewing is done in a less manifest and visible way) and sociability (eating does not exclude participating, even if in a silent way, to the ongoing discussion). The distinction between how participants configure tasting vs. eating is again observable at another moment, when Marta brings some gambas wrapped in a green rice crispy coating. Leandro asks a question about how the preparation will be served, as a single dish or as part of another dish (1–3), while Marta is already taking a prawn and putting it into her mouth (3).

44

L. Mondada

Excerpt 3a (MAOT2_20-49)

8

9

2 Tasting vs. Eating

45

10

The moment in which Marta has put her sample in the mouth corresponds to the completion of Leandro’s turn, making a suggestion (Fig. 8). At this point, Marta engages in two very different actions: she responds to the suggestion in a negative way, frowning and shaking her head (Fig. 9), and she begins to chew. Thus, in this particular sequential position, she takes the food and she disagrees with Leandro about the proposed dish, thus engaging in eating and talking, within a form of multiactivity. After having shaken her head again, she responds negatively and engages in the defense of her variant, gesticulating with the hand that still holds the sample (Fig. 10). She continues her argument (9–15), looking

46

L. Mondada

alternatively at her recipients, while at the same time continuing to chew, especially in the pauses between her turns at talk. At the end of her plea, she looks at the remaining bit (15), and pursues a response from her co-participants (17), while putting it into her mouth. Instead of responding (although Elisa minimally nods, 17), they all engage in tasting: Excerpt 3b (MAOT2_20-49)

2 Tasting vs. Eating

47

11

After ingesting the sample, Leandro looks down at the dish; Elisa bites audibly the crispy sample and chews while looking at the remaining bit in her hand, turning it and inspecting it from different perspectives, configuring her tasting as a multisensorial engagement involving vision, touch, hearing, and taste (Figs. 11–16). Marta seems to orient to the crunchy sound and looks at Elisa; she can see her visibly chewing and exploring the sample (Fig. 11). The tasting is brought to completion by Leandro, exiting his immobility and touching the remaining prawns with his sticks while swallowing. This is noticed by Marta, and later by Elisa,

48

L. Mondada

as Leandro stands up and produces his positive assessment (22) (which will be followed by some other critiques). This setting, in which the cooks and the manager discuss new dishes within their professional activity, confirms the systematicity of the multimodal Gestalt publicly recognizable as achieving tasting, as well as its contrast with eating. When a dish is brought on the table, tasting as well as eating are an option for the participants. The practice in which they engage is achieved, that is produced and recognized, as such in accountable ways and has different forms of consequentiality for the following actions. These practices can be also formulated in somany-words, announced, prefaced, and invited as such—although their accountability does not rely uniquely nor necessarily on these explicit formulations.

2.3.2 Responding to Offers to Taste in the Market The difference between tasting and eating is observable in other settings, for instance, in commercial encounters where a free sample is offered to taste to possible customers: this is observable in food shops (Mondada, 2018a, 2021) as well as in markets (Mondada, 2022a). In the latter case, passers-by are offered a sample while they walk by the stand, understood as initiating a possible transaction. In response, they can engage in tasting in very similar ways as described above; at the end of tasting, as they repristinate talking with the co-participants, they can continue to chew the remaining bits in a way that is configured rather as eating than tasting (2018a: 759). However, some passers-by in the market might also engage in consuming the sample offered to them in a way that is accountably achieved as eating rather than tasting. In this section, I show both cases, on the basis of a video-recording of a market’s stand selling Alsace’s specialties, mainly Munster cheese and local liquors. As passers-by walk along his stand, the seller repeatedly offers some cheese to taste, using this offer as a way to stop them and transform them into customers. The passers-by respond in different manners, either accepting or refusing/ignoring the offer. When accepting, they can grasp the offered cheese in different ways, that are

2 Tasting vs. Eating

49

consequential for their progression into a selling encounter or not. In this section, I demonstrate that in response to an offer to taste, participants might either taste or eat the sample offered, both responsive actions being formatted in distinctive ways. Moreover, in this local ecology, these two actions have different consequentialities: tasting is related to stopping, favoring an exclusive focus on the sample, whereas eating can be achieved en passant, in a mobile way, favoring multiactivity. While tasting might project a possible purchase, eating rather projects the closure of the transaction. We join a first case, in which a couple of passers-by is addressed by the seller offering a taste (2). One member of the couple (pb1) refuses by thanking (in German, 3) and continues to walk, whereas the other one (pb2) accepts, walking toward the stand and stopping to grasp the sample (Fig. 17): Excerpt 4 (MUST_00-38-26/CLI70)

50

L. Mondada

18

19

2 Tasting vs. Eating

51

20

The passer-by takes a sample and thanks while bringing it into his mouth (6–7). Observed by the seller, he engages in tasting by chewing in a visible vigorous way and disengages from the exchange with the seller by looking aside, at distance, for a few seconds (Fig. 18). The tasting is brought to completion by him closing his eyes while he does a gesture with the thumb and the index, manifesting a positive assessment (Fig. 19). He only looks at the seller after that, when uttering the verbal assessment (9, Fig. 20). As soon as he makes his gesture, the seller turns from him to the produces, showing them and offering a discount price: he orients to the completion of the tasting and the resulting positive assessment as making a purchase possible (Mondada, 2022b). However, this possibility seems to be oriented to by the passer-by too, who, while uttering the positive

52

L. Mondada

assessment, begins to walk again, leaving the stand. He repeats the positive assessment (11) while doing a good-bye gesture. The tasting and its completion are thus interpreted by both as progressing in the interaction, either for deciding to buy or for closing the encounter. The tasting thus represents an accountable moment in which the possible customer makes a sensorial judgment, and the seller witnesses it, anticipating the outcome, crucial for the next possible actions. The acceptation to take an offered sample can be accountably formatted in a very different way, as shown in the next excerpt: Excerpt 5 (MUST_00-28-18)

2 Tasting vs. Eating

53

54

L. Mondada

As in the previous case, the passer-by is addressed by the seller (1) with an offer to taste (2–3), multimodally formatted by extending the board and by explicitly mentioning tasting (3). The passer-by stops and takes a sample (4, Fig. 21), putting it in her mouth. While doing that, she also verbally responds, with an acceptance token, some comment (partially unclear) about the usefulness/necessity of the action, and a laughter. She also looks at the seller, while chewing, before gazing at the products he shows, responding immediately and talking about them. At the same time, she also begins again to walk (Fig. 22) and stops again (Fig. 23), after raising her eyebrows. The eyebrow is not related to what she has in her mouth but is responsive to the commercial offer. She swallows, then continuing to clean her mouth with the tongue, while the seller utters the price, showing the products. After swallowing a second time, she responds to him with an objection, prefaced as dispreferred, preparing a refusal to buy (11), while beginning to walk away. The seller responds to the objection (13–14) but she leaves anyway. Here, the passer-by responds to the invitation to taste by engaging in eating rather than tasting. She formats this in her verbal response when ingesting the sample, possibly alluding to eating as a necessity, and laughing, projecting that she is disaligning with the offer to taste, and that she is rather doing just eating. She also engages in multiple activities: she ingests the sample, while addressing and responding to the seller, looking at him and at the products he shows, while alternating walking and stopping. Eating is typically done as multiactivity, and contrasts with the previous cases of tasting, characterized by an exclusive unique focus on the tasted sample. In the dataset, the passers-by stopping to taste

2 Tasting vs. Eating

55

often engage in a prolonged commercial exchange, become customers, and end up buying the product. By contrast, the passers-by eating the sample never buy the product; rather, eating is achieved while walking away. This setting shows in an exemplary way that an invitation to taste can be responded to by aligning and doing tasting but can also be resisted to by doing eating.

2.3.3 Tasting vs. Drinking Wine in Fine Dining Tasting routinely occurs in upper-end restaurants, which constitute a perspicuous setting to observe distinct practices characterizing customers tasting vs drinking wine. The third dataset documents practices of fine dining in a gastronomic restaurant in France, in which we focus on the wine service. A couple has just asked for recommendations for a red wine and the sommelier brings a suggested bottle. He pours some wine for both of them to taste and, if good, accept it. In response one of the customers (CUS2) engages in tasting, while the sommelier (SOM) observes him, waiting for some positive or negative approval, as well as the other customer (CUS1), who watches him tasting without touching her own glass. Excerpt 6a (winch_D3T2_002318)

56

L. Mondada

2 Tasting vs. Eating

57

The customer takes the glass (9, Fig. 24), brings it to his nose, smells it (Fig. 25), then swirls it (Fig. 26), then smells again (Fig. 27), before drinking it (Fig. 28). After drinking, he holds the glass in the air and looks at it (Fig. 29). Meanwhile, his tongue explores his mouth, he clicks the tongue and the palate (12), circulating the wine in his mouth (13), then clicking the tongue again (14). These movements of the tongue are both visible and audible and demonstrate some connoisseurship, a methodic way of exploring taste and aftertaste. Finally, the customer nods (15) and this is seen by the sommelier as the completion and (positive) assessment of the wine tasting. He comes closer with the bottle, ready to serve, even before the customer verbally produces his agreement (16), and pours the wine in the wife’s glass (17). Wine tasting occurs following a specific methodic multisensorial exploration of its features; even though this can be attributed to a socialization and connoisseurship in the wine culture, the procedure is remarkably similar to the ones described above for other types of food, further demonstrating the methodic character of tasting as a careful and focused multisensorial apprehension of food. Also, like in the previous

58

L. Mondada

cases, the observer (here the sommelier, but also the other customer) witnesses the sensorial exploration of tasting, and it is possible embodied projection of the outcome (here anticipated in the movement of the sommelier pouring the wine even before the verbal assessment). A few minutes later, the same customer (CUS2) drinks the wine, in the midst of a conversation with his wife (CUS1), after they have asked the waiter what the dish they just had was made of. Excerpt 6b (winch_D3T2_28-37_drink) (5 min 20 sec later)

The wife (CUS1) points back at the fact that she had guessed that the dish was made of activated vegetal carbon (1). The husband (CUS2) minimally aligns and she produces a third turn (3), which might close the sequence or alternatively offer the occasion to expand the alignment. During the first turn he takes his glass; he begins to drink on the third, and during the silence that follows (4). As soon as he puts the glass back on the table, he contributes to the ongoing sequence with a kind of (concessive) compliment (5–6). So, his drinking happens within a sequence that is further expanded after it. Both customers maintain mutual gaze and a body posture that shows their continuous common engagement in the conversation. In this case, the wine is drunk without any special attention to it.

2 Tasting vs. Eating

59

These extracts show how, during fine dining, specific sequences are devoted to tasting, under the gaze of the sommelier—where the aligned normatively expected response is tasting, as shown here, but which can also occasion just drinking (not shown here)—whereas at other moments the same wine is simply drunk. Although drinking, like eating, constrains the actions of the participant—mostly impeding them to speak—this does not exclude specific forms of participation to the ongoing conversation (drinking being positioned precisely in a way that enables to sustain the continuation of the interaction, see Hoey, 2018).

2.4

Discussion

The three settings studied in this paper show convergent results, which document the systematicity of the complex multimodal gestalt characterizing tasting across contexts, the distinct accountable embodied action formation of tasting vs. eating, and the orientation of the participants recognizing and ascribing them as they are accomplished in an embodied silent way, and/or as they are verbally formulated. Tasting, as well as eating/drinking, are practices that involve first and foremost the body, understood in its physiology and flesh—relying on movements of the mouth crunching, chewing, masticating, and swallowing. However, the ingestion of food can be formatted in two different socially accountable types of action, tasting vs. eating. They are differently produced: these movements of the mouth tend to be more accentuated, more expanded, and slower in the former than in the latter; in tasting, they are also applied on a reduced quantity of food/drink (a sample), all this fostering a maximization of taste sensations. These specific ways of ingesting food refer to several layers of multimodality and multisensoriality. First, they relate to sensorial practices (e.g., exploring the mouth with the tongue enhances the haptic dimension of taste) that can be socialized within specific methods (e.g., swirling the wine in the glass and holding it in specific ways under the nose in order to better smell the aroma). Second, they relate to the visual aspect of the practice, which, once looked at and witnessed, becomes visible. These movements are not only available to the co-participants looking at the taster, but are

60

L. Mondada

also videoable, that is, available to the analyst producing, watching, and transcribing the video-recording (see the detailed multimodal transcripts of the movements of the tongue and mouth). This is a first dimension of accountability securing the intersubjectivity of sensorial practices. But the accountability of these practices is not just a matter of how the food is ingested and masticated; it crucially relates to the ecology and participation framework in which this is achieved and recognized. In tasting, the taster is exclusively focused on the sensorial experience, withdrawing from the participation to the ongoing conversation, and the co-participants not engaging in tasting do not talk with them and often monitor them. This collectively designs an interactional space in which tasting is achieved as an individual yet public activity. This grounds the interactive order of tasting. Eating/drinking is organized differently: it can be considered as a subordinated side-involvement (Goffman, 1963: 65) although it is not merely a self-involved, self-absorbing physical act (1963: 64–65), being rather orderly embedded in the ongoing conversation, and its sociality. It is unnoticed, and its consequentiality is not the same for the ongoing interaction (it can become the target for normative comments if it is recognizable as something else, like guzzling, or munching, or can be topicalized for other reasons, cf. Mondada, 2009a). These actions are thus configured in and through complex multimodal gestalts, the recurrency of which shows the generality across social and cultural contexts. These multimodal gestalts are ‘complex’ for the analyst reconstructing their multi-layered temporal and sequential organization. Although they are subtly assembled by the participants, they are not treated as ‘complex’ by them, who often address them holistically as unproblematic (although they can also orient to their singular details when actively engaging with practical problems, such as the precise timing of eating and talking). The ways participants orient to the intelligible, accountable, and normative aspects of eating/drinking are quite opposite to orientations toward tasting. Consequently, the individuality of tasting, which is sustained and witnessed by the co-participants, is reflexively defined by them very differently than the individuality of eating/drinking, which is rather unnoticed (they do not stare at the person eating/drinking).

2 Tasting vs. Eating

61

This shows that both are ascribed different normative dimensions. These features address issues of individual/intersubjective/public/social organization of bodily actions, in a way that is not reducible to the dichotomy between private/public or individual/social. The distinction between tasting vs eating/drinking corresponds to two patterns described by previous analyses of multiactivity (Mondada, 2014). Tasting relies on the exclusive order, de facto exiting multiactivity, by privileging and prioritizing only one activity—hence the focus on the sensory experience. Eating/drinking is characterized by the embedded order, which orderly manages to insert the progressivity of one activity (ingesting food/drink) within another (talking) (eating/drinking exclude the parallel order, which is the continuous and smooth progressivity of both activities at the same time, because eating uses the same resources as speaking, that is the mouth). Tasting typically (although not always) happens in particular sequential environments, such as in response to an offer to taste, possibly grounding assessments or decisions. In this sense, tasting is sequentially embedded in a larger interactionally organized course of action, for which it is consequential. Among the restaurant team, it leads to critiques and elaborations of the dishes, among the shopping customers to a decision to buy, among the diners to the selection of the wine. However, not everything produced in response to offers to taste constitutes tasting, and tasting might also happen serendipitously in sequential environments not designed for it. Participants can also respond to offers to taste by merely eating/drinking, thus orienting to the sociality of actions of ingesting food/drinks and to their specific formatting. Participants might constrain the interactional environments, projecting/expecting tasting rather than eating (or the opposite); in response, co-participants might either align or disalign with it. As a local emergent outcome, actions are accountably formatted and ascribed as tasting or eating/drinking. Systematic analyses of tasting vs. eating contribute to discussing action formation and ascription in relation to embodied actions. It also contributes to the study of multiactivity, multimodality and multisensoriality—for a better understanding of food consumption and sensuous

62

L. Mondada

engagements with food as an embodied social and intersubjective experience in interaction. Conventions Talk is transcribed following Jefferson’s conventions; embodied conducts are transcribed following Mondada’s multimodal conventions. Acknowledgments This paper has been written within the project “From multimodality to multisensoriality: Language, Body, and Sensoriality in Social Interaction (intSenses)” funded by the Swiss National Science Foundation (project number 100012_175969) (2018–2023).

References Deppermann, A., & Haugh, M. (Eds.). (2022). Action Ascription in Social Interaction. Cambridge: CUP. Fele, G. (2016). Il paradosso del gusto: Forme visibili dell’apprezzamento estetico. SocietàMutamentoPolitica, 7(14), 151–174. Fele, G. (2019). Olfactory objects: recognizing, describing and assessing smells during professional tasting sessions. In D. Day and J. Wagner (eds.) Objects, Bodies and Work Practice. Bristol: Multilingual Matters, pp. 250–284. Goffman, E. (1963). Behavior in public places. New York: Free Press. Goodwin, C. (1984). Notes on story structure and the organization of participation. In J.M. Atkinson and J. Heritage (eds.) Structures of Social Action: Studies in Conversation Analysis. Cambridge: CUP, pp. 225–246. Haddington, P., Keisanen, T., Mondada L., and Nevile, M. (eds.). (2014). Multiactivity in social interaction: Beyond multitasking. Amsterdam: Benjamins. Hennion, A. (2005). Pragmatics of taste. In M. Jacobs and N. Hanrahan (eds.) The Blackwell Companion to the Sociology of Culture. Oxford: Blackwell, pp. 131–144. Hennion, A. (2007). Those things that hold us together: taste and sociology. Cultural Sociology, 1(1), 97–114.

2 Tasting vs. Eating

63

Hennion, A. (2015). Paying attention: what is tasting wine about? In A. Berthoin Antal, M. Hutter and D. Stark (eds.) Moments of Valuation: Exploring Sites of Dissonance. Oxford: OUP, pp. 37–56. Hoey, E. M. (2018). Drinking for speaking: the multimodal organization of drinking in conversation. Social Interaction: Video-Based Studies of Human Sociality, 1(1), 10–7146. Laurier, E. (2008). Drinking up endings: conversational resources of the cafe. Language & Communication, 28(2), 165–181. Levinson, S. (2012). Action formation and ascription. In J. Sidnell and T. Stivers (eds.) The Handbook of Conversation Analysis. Malden: WileyBlackwell, pp. 103–130. Liberman, K. (2013). The phenomenology of coffee tasting: lessons in practical objectivity. Etnografia e ricerca qualitativa, 1, 35–58. Liberman, K. (2018). Objectivation practices. Social Interaction. Video-Based Studies of Human Sociality, 1(2). https://doi.org/10.7146/si.v1i2.110037. Mondada, L. (2009a). The methodic organization of talking and eating: assessments in dinner conversations. Food Quality and Preference, 20, 558–571. Mondada, L. (2009b). Emergent focused interactions in public places: a systematic analysis of the multimodal achievement of a common interactional space. Journal of Pragmatics, 41(10), 1977–1997. Mondada, L. (2009c). Multimodalità e multi-attività nelle conversazioni a tavola. In M. Fatigante, L. Mariottini, and M.E. Sciubba (a cura di). Lingua e società. Roma: Angeli. Mondada, L. (2014). The temporal orders of multiactivity. In P. Haddington, T. Keisanen, L. Mondada and M. Nevile (eds.) Multiactivity in Social Interaction: Beyond Multitasking. Amsterdam: Benjamins, pp. 33–75. Mondada, L. (2018a). The multimodal interactional organization of tasting: practices of tasting cheese in gourmet shops. Discourse Studies, 20(6), 743– 769. Mondada, L. (2018b). Multiple temporalities of language and body in interaction: challenges for transcribing multimodality. Research on Language and Social Interaction, 51(1), 85–106. Mondada, L. (2020). Audible sniffs: smelling-in-interaction. Research on Language and Social Interaction, 53(1), 140–163. Mondada, L. (2021). Sensing in Social Interaction. Cambridge: CUP. Mondada, L. (2022a). Appealing to the senses: approaching, sensing and interacting at the market’s stall. Discourse & Communication, 16(2), 160–199.

64

L. Mondada

Mondada, L. (2022b). Attributing the decision to buy: action ascription, local ecology, and multimodality in shop encounters. In A. Deppermann and M. Haugh (eds.) Action Formation and Action Ascription. Cambridge: CUP. Perullo, N. (2016). Taste as Experience. New York: Columbia UP. Wiggins, S. (2002). Talking with your mouth full: gustatory mmms and the embodiment of pleasure. Research on Language and Social Interaction, 35 (3), 311–336. Wiggins, S. (2013). The social life of ‘eugh’: Disgust as a form of evaluation in family mealtimes. British Journal of Social Psychology, 52(3), 489–509. Wiggins, S. (2019). Moments of pleasure: a preliminary classification of gustatory mmms and the enactment of enjoyment during infant mealtimes. Frontiers in Psychology, 10, 1404. https://doi.org/10.3389/fpsyg.2019.01404

3 Metagaming and Multiactivity: How Board Game Players Deal with Progressivity Emily Hofstetter

3.1

and Jessica Robles

Introduction

Games are ostensibly a special context for social interaction. Game researchers propose that when one enters into gameplay, they step into a virtual world—what some call the “magic circle” (Klabbers 2009)— a somewhat different mode from the rest of ordinary life that requires one to adopt a lusory attitude (Bergström 2010) in which normal rules and expectations may be suspended or transformed. Game-focused literature continues to describe aspects of gaming as more or less inside/ outside the game, based on theory or interviewee opinion, despite numerous challenges to this view (e.g., Consalvo 2009). Moreover, very E. Hofstetter (B) Linköping University, Linköping, Sweden e-mail: [email protected] J. Robles Loughborough University, Loughborough, UK e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_3

65

66

E. Hofstetter and J. Robles

little research has examined closely how players treat certain actions as part of a game or not during the course of gameplay. How do players make sense of actions as relevant (or not relevant) to the game, or to any other concurrent activities? In this paper, we present an inductive analysis of actual gameplay interaction to theorize, from players’ perspectives, a typology of activities that are “outside” the game, what have sometimes been called “metagame” activities (Bergström 2010). As an alternative to the idea of the magic circle, we propose taking a conversation analytic approach that treats game/non-game distinctions as matters of multiactivity (Haddington et al. 2014); in the course of progressing the game overall, players organize the “byplay” (M.H. Goodwin 1997) of competing activities that spontaneously arise and dissolve over the course of a game. By examining how players manage real, complex moments where multiple concurrent activities come into conflict, we develop a situated account of “metagaming” as a player’s concern, and how multiactivity is treated as morally accountable.

3.1.1 Games, Metagames, and the “Magic Circle” When engrossed in a game, a player “lives” in a different phenomenological context: for example, role-playing involves simulation of people and activities unfamiliar to one’s everyday life (Fine 1981). Engrossment in games is part of what produces games as “fun”; and although research has demonstrated many of the instrumental benefits of gameplay (e.g., learning and education, Abdul Jabbar and Felicia 2015; therapies, Navarro-Newball et al. 2014 etc.), the fact that games are fun is generally seen as a sufficient reason for playing them (Goffman 1961). While engrossed in a game, participants are sometimes said to be inside what’s called the “magic circle”: an abstract space where new worlds and rules are created. The circle boundary is theorized as fairly strict according to Salen and Zimmerman (2004), but research has shown flexibility for different kinds of games and situations (see Consalvo 2009; Moore 2011). Pargman and Jakobssen’s (2008) participants did not describe a boundary at all, instead framing games as a mundane part of everyday life. This range supports the “weak boundary”

3 Metagaming and Multiactivity

67

model proposed by Juul (2008: 62), who argued that “game scholarship should be about analyzing the conventions of this boundary, and how and when this boundary is negotiated”. In light of this call, we examine how participants themselves attribute and negotiate activity as game or not-game through interactional resources. We will question whether and how participants orient to game and not-game as thoroughly separate activities and show how the players manage their multiple engagements in situated gameplay. In considering shifting in or out of modes of play, one example of a temporary game suspension (which would pause or cross the idea of magic circle) is conversation about the game itself, often for the purpose of “understanding of the game in terms of how the game is to be played, their role in the game and the culture around the game” (Jakobsson et al. 2007: 158). In most games, talk organizes forms of expertise in the course of handling the game (including physical components such as video-game controller manipulation, sequential components such as order of gameplay, rules and allowable exceptions, etc.), but does not constitute gameplay itself. These out-of-game aspects are part of what is called the “metagame”: “the relationship between the game and outside elements,” which also includes “everything from player attitudes and play styles to social reputations and social contexts in which the game is played” (Salen and Zimmerman 2004: 481). Donaldson (2017: 440) describes metagame expertise as “the awareness of and ability to negotiate the game around the game”. Note that these descriptions of “metagame” differ, suggesting it is not always clear what constitutes “metagaming”, let alone whether metagaming is “in” or “out of ” the game and ostensible magic circle. Neuenschwander (2008) attributes the need for metagaming to the fact that explicit game rules are incomplete for actually structuring players’ actions (as Liberman 2013 also demonstrates in instances of actual play); and the metagame is rarely indicated in instructions at all. Thus, the implicit rules of a game must also be managed through actions or talk that are not themselves gameplay. The term “metagaming” is also used by board gamers. In a study of the implicit rules of gaming, based on interviews with gamers, Bergström (2010) describes several categories of implicit rules involved in gameplay, including committing to completing a game from start

68

E. Hofstetter and J. Robles

to finish, no “unacceptable whining” or “gloating” (see also Hofstetter and Robles 2019), and relevantly, “no metagaming”. Bergström specifies metagaming (p. 90) as “allowing undue outside influences”, which is claimed to always be seen by gamers as problematic. However, while a logical organization of interview data and conscious reflection from gamers, the categories do not consistently correspond to how players actually act during games. Players label actions as metagaming that Bergström categorizes differently (see Example 1 below, which Bergström’s structure calls “between game memory”), but moreover, players do not treat these rules as universally applicable or consistently defined. In other words, the “consensus exception” that Bergström notes, wherein any action can be acceptable if mutually agreed to be so, is applied extensively, rather than as an unusual exception. Furthermore, as Liberman (2013) has discussed, the methods by which players “agree” to rule alterations is not (typically) through active consensus making, but through post-hoc ratification of rule breaches. The interactional environment in which rule adherence unfolds is thus critical to investigate and account for, if we are to investigate how the rules of the game activity are managed. For example, in situ, the word “metagaming” can be used to sanction conduct deemed unacceptable—that is, to pronounce something as metagaming is to label it as morally unacceptable play, and to suggest the conduct in question should be halted or ignored. This occurs in the following example. Here, players are suggesting various targets to be “killed” or knocked out of the round; Dave is suggested (L1, 4; “coup”-ing is this game’s term for eliminating a player), and Gerry is suggested (L6). Garth dismisses the whole discussion as “metagame” (L13).

3 Metagaming and Multiactivity

69

(1) (GN_Coup_35.30)

Garth’s sanction arrives just after Gerry and Dave argue about the acceptability of being eliminated early two games in a row (L7–10). Gerry’s complaint has an ostensibly “out-of-game” premise, namely that his experience in another (just prior) playthrough of the game should have bearing on this ongoing game. Garth’s sanction also overlaps James’ speaking turn extensively (L12–14). Typically, one or other speaker would drop out in such extended overlap (Sacks et al. 1974; Jefferson 1986), but Garth continues to reinforce his sanction, even though he is simply repeating it. He thereby treats the sanction as both legitimate and important to put on record. James proposes an ostensibly in-game reason for picking Dave as the one to eliminate (strategically removing the player who is “ahead” with a lot of money), potentially motivating his repeated attempts to take a turn at talk, since his utterance will provide a way for the group to stop metagaming. The above example gives some insight into ways that players can use the term and concept of “metagaming” to do action in live gameplay. This paper considers how the member’s term compares with an analytic account of how players actively navigate and organize action as in or out of gameplay. In other words, metagaming as a members’ category serves as our inspiration for investigating evidence for or against the magic circle, in actual gameplay. When trying to map specific game

70

E. Hofstetter and J. Robles

actions that occur in natural recordings of games, Bergström’s categories do not provide much that resonates with the actual in-progress production of gameplay, as the categories are organized based on self-reflection rather than observation (see Pomerantz 2012), and since the “consensus exception” is more of a universally applied component of achieving rules in situ than an exception to rule adherence. The natural ecology of interaction in gameplay is more complex, with all its competing activities and relational demands. Therefore, we suggest taking an interactional approach that situates metagaming within the actions that participants take during gameplay, rather than based on theory or on participants’ elicited reflections. An interactional approach would provide a more systematic, ecologically valid account. By looking at how participants treat game/non-game distinctions in situ, we will re-analyze the concepts of metagaming and the magic circle from an emic perspective. In examining the multiactivity engagements that metagaming entails, we aim to contribute to our understanding of how participants both organize complex environments (board games with co-occurring socializing and background distractions) and manage the accountability of constantly shifting activity foci (see Vatanen, this volume).

3.1.2 Insights from Interactional Studies on Gaming Activities Few interactional studies to date look at games, and those that do have focused mostly on video games or sports (see e.g., Baldauf-Quilliatre and Colón de Carvajal 2020, 2021a, b, this issue; Mondada 2012; Piirainen-Marsh and Tainio 2009; Tekin and Reeves 2017; see Evans and Fitzgerald 2016 for basketball). Although board games are regularly used as a convenient source of data collection (Betz and Deppermann 2018; Kendrick and Drew 2016; Rossi 2015; Viney 2015), moments of game play are rarely analyzed (Drew and Kendrick 2018; Sutinen 2014 show some game play; and to our knowledge, only Hofstetter and Robles 2019; Hofstetter 2020a, b, 2021; Liberman 2013; Sterphone, 2022 analyze actual board game play as a phenomenon). One particularly relevant subset of interactional research has explored instances when

3 Metagaming and Multiactivity

71

participants engage in multiple activities or “multitasking” (Haddington et al. 2014). For example, Sutinen (2014) analyzed how players in a role-playing game transitioned back into gameplay, after interruption or lapse of activity (see also Helisten 2019). Through stepwise coordination, all players eventually resume joint attention on the game. The general interactional orientation toward maintaining progressivity (Stivers and Robinson 2006) is particularly relevant in complex activities such as gaming, and players seem especially sensitive to ensuring they display that they are moving the game forward (Hofstetter 2021). Players distinguish whether the actions are progressing the game as compared to some other activity; in another game-focused study, Mondada (2012) has examined how players of video games manage spatial resources to display being “in the game” and “out of the game”. Our paper expands this analysis by examining board games, rather than video games, in addressing issues of the magic circle and the participants’ category of “metagaming”; and in a focus on progressivity rather than spatiality. The spatiality of board games is also quite different than video games: it involves only non-digital objects and interlocutors; it lacks a screen and a digital “space” for manipulation of game objects; and therefore being “out of ” the game is a qualitatively different process, since addressing co-participants in the non-digital space, and gazing away from the board, do not entail non-game activity in the same way. While video game players can be said to “switch between one interactional space and another” (Mondada 2012: 239), board game players cannot—the medium itself does not afford the same spatial or temporal distinctions between “in-game” play and “out-of-game” talk. Furthermore, Mondada analyzes players’ reactions to or discussion of the video game as retrospective, “out-of-game” moments, whereas this is not the case in board game interaction, where such talk is part of the progressivity of the game itself. The video game medium provides specific rest moments for talk that does not hinder the progressivity of the game (e.g., during a replay video), whereas board game players must use moments when other players are taking their game turn to do non-game activities. This means board games have different contingencies for metagaming, and that board game structure may make metagaming more problematic or accountable.

72

E. Hofstetter and J. Robles

Finally, we have evidence that players treat potential shifts between game-specific categories and personal or situational categories as relevant for doing moral work. Sterphone (2022) has shown that in interaction over wargaming, attribution of action to game-specific categories (e.g., players’ roles) is a means of downgrading complaints about action and providing a justification for complainable action within the activity. By invoking categories that make the game relevant, rather than individual or general properties, participants can manage the trajectory of a complaint. In doing these different invocations between game and beyond-game categories, players also orient to the different activity layers available in the interaction, a practice we further investigate below.

3.2

Data and Methods

This article presents an analysis of 43.5 hours of video-recorded board game interactions among 56 different adults, 20 different gaming sessions, and 17 different games (generally involving three-four players per game), with 78 cases. Three games were collaborative (all players work together against the game), and the remaining 14 were competitive (all players compete to be a single victor). Games were collected by the first author among gaming groups and at board game cafes, where the author had been doing fieldwork for approximately two years, as well as games uploaded to the internet to board gaming specialist websites (e.g., Boardgamegeek.com). The latter were not “teaching” videos, but playthroughs of the game. Although these are potentially recorded with an audience in mind, they are unscripted, and still must deal with the natural contingencies of board game play. Transcripts are formatted according to Jefferson (2004) and Mondada (2018) conventions (with the exception that embodied conduct receives a line referent, see Appendix A). For space limitations, we present only a few examples; each situation we show below should be understood as the clearest (often shortest) example of a pattern across the data. Given the interest in “metagaming” in the board game community (as seen in Bergström 2010), both authors examined the data for instances where players orient to interaction or game moves as inside or outside

3 Metagaming and Multiactivity

73

the game (the “magic circle”), but not as themselves legitimate or illegitimate actions or moves (i.e., excluding instances where following the game rules was at issue). The data were not coded as instances of in-orout-of-game, but instead the authors inductively searched for instances where the relevance of some action or move to the game (vs. to other concurrent activities) were made relevant. These instances involved an element of multiactivity, but more precisely instances where the memberassigned status of something as multiactivity for local purposes was being negotiated. With respect to the members’ term “metagaming”, the examples here would not necessarily be explicitly labeled as metagaming in situ or in reflection (e.g., in an interview). “Metagaming” as a term in the data often occurs in moments where fairness or acceptability was at issue (see also Hofstetter and Robles 2019), such as partners giving each other preferential treatment instead of competing fairly. The analysis below instead questions how elements of the magic circle, including metagaming, are negotiated by players for relevance and acceptability at this particular moment in gameplay. The analysis draws on conversation analysis (CA) in that it uncovers the emic procedures participants use to accomplish social actions and tasks and relies on the participant orientations seen in the data alone (as displayed in their unfolding actions and turns at talk) as the determiner of the validity of findings. It should be noted that in CA, one generally avoids looking for or applying terminology that has been theorized before the data—what are called analysts’ terms—and should instead approach the data unmotivatedly and only describe actions that can be shown to be demonstrably oriented to as such by the participants themselves (see Sidnell and Stivers 2013). Our analysis is unusual in seeking instances that may address a theoretical question that already exists, concerning the magic circle. However, CA is the method that best grounds findings in emic, situated practices, and has the highest ecological validity, which thus will best answer Juul’s (2008) call to uncover how the magic circle is negotiated in actual instances of gameplay. Therefore, our analysis will not attempt to apply a priori definitions of metagaming to code behaviors. Instead, we conceptualize orientations toward the magic circle as instances of both

74

E. Hofstetter and J. Robles

identifying and negotiating the acceptability of multiactivity. We distinguish between action (with the voice or body) that itself progresses or supports progressivity of the game; and action that does not do so. The validity of this distinction is shown in whether and how participants orient to those actions as progressing the game, or not. We show that one way to conceptualize metagaming and the magic circle, emically, is to use these participant orientations to multiactivity and progressivity; and that any given action could be treated as along the spectrum of metagaming behaviors based on those orientations.

3.3

Analysis

There are a number of activities people routinely engage in during gaming that do not, in and of themselves, constitute a game-move. In this section, we show how participants navigate (and sometimes police) the boundaries of game activities (see also Vatanen, this volume). While multiactivity—accomplishing various activities simultaneously— is a recurrent feature of everyday human interaction (Haddington et al. 2014), players’ prioritizing of the game over other involvements, or vice versa, can become morally accountable (see also Kamunen 2019). Goffman (1963) suggested that participants organize their actions as addressing “main” and “side” involvements (the former claiming the majority of attention, the latter existing sustainably alongside the former without overtaking it), and as “dominate” or “subordinate” (the latter being deferred to the former). Game players are typically accountable to the game as the main and dominate involvement; however, we deviate from this point onward with Goffman by highlighting the fluidity and non-exclusiveness of involvements. While multiactivity as a concept better describes the participants’ constant navigation of multiple focuses of attention, Goffman’s involvements remind us that a game provides a participation framework (Goodwin 2007) with very specific accountable priorities.

3 Metagaming and Multiactivity

75

3.3.1 Multiactivity and Game vs. Other Involvements Before examining different orientations to the acceptability of multiple involvements, we will first show an example to demonstrate when multiactivity becomes relevant and how it impacts the progressivity of the ongoing game. At the start of the example, talk runs concurrently to the game without delaying it but also without progressing it; and at the end of the example, players maximally attend progressing the game, demonstrating no further attention to alternate involvements. During gameplay, participants navigate any number of activities simultaneously that do not require suspension of the game and are not treated as “interruptions”; gameplay is constantly layered with this kind of complexity. In Example 2, three participants are playing a game characterized by long silences; behind them in the frame (Fig. 3.1), another participant (Tara) is cooking something in the kitchen (see Baldauf-Quilliatre & Ursi, this volume, on spatiality and participation frameworks). Tara has come home long after the others ate dinner, not being part of the original expected gathering of players, and is now taking advantage of the food leftovers.

Fig. 3.1 View of the game Skull, L-R: Kat, Tara, John, Adam

76

E. Hofstetter and J. Robles

Example 2 below begins with six seconds of silence during which the game is in-play; it is Adam’s game turn. The numbers being announced are bets (L1, L20, L31). Tara, who is in the kitchen and not playing, initiates a sequence with John that is unrelated to the game. (2) (BG 160712 Skulls_12:05)

First, let us describe the sequence of events. In line 8, Tara makes an offer to John, presumably of what she is cooking. She formulates the offer in a way that anticipates refusal, preemptively opening a refusal based option, or are you:::. John’s uh:m: (L14) may indeed project a refusal, though John abandons further response. Tara shifts to professing some difficulty or uncertainty with her activity (L13), to which John responds with laughter at line 18. In overlap with Tara, Adam takes his turn at the game: four (L20). This overlap shows Adam is not attending Tara, but

3 Metagaming and Multiactivity

77

attending the game; indeed, Tara’s question (L8) specifically addressed John and not Adam or Kat, not including them in the discussion of food. Though visibly facing and attentive to the game, John responds to Tara’s comment with a counter-compliment, it smells nice (L23), and in line 26 Tara responds to this with a counter that attributes the compliment to John’s pesto (that he made). John receipts this compliment with a response-cry of appreciation (L27). Another silence follows as the game continues, in which John checks on the progress of Kat’s game turn (L28, L30). During this interaction, John’s conversation with Tara is not part of the game: those involved in the game do not treat what he says as relevant to what they are doing; and John and the other players mostly remain in bodily orientation to the game (the table, the cards, and his fellow game players) (except lines 11–22, see Figs. 3.2 and 3.3). His contributions to the conversation with Tara are maintainable without disrupting the progress of the game; his laughter (L18) does not promote continuing the sequence with Tara (and allows him to return his attention to the game), and his comment on Tara’s activity with the food is based on a sense (smell) that he can have without disrupting his visual and bodily attention to the game. Only, when Tara initially addresses John is there some micro-level negotiation of what this moment might mean to the game-in-progress and how to sustain the complex multiple engagements: as Tara is completing her utterance (L8), John begins to turn his head to look over his shoulder toward her (L11), and Adam’s eye gaze shifts toward Tara as well (Fig. 3.2). Subsequently, as John completes his uh:m: in line 14, he also shifts his shoulders and torques his upper body toward Tara, and Kat’s face direction and gaze tracks this motion and also visually orients to Tara (Fig. 3.3) (see Schegloff 1998; Kamunen 2019 on how body torque maintains multiple engagements). During this moment, there is a momentary shift of the participation framework to accommodate the non-player and the non-game activity. As soon as Tara shifts from offer to commentary (L13), one by one each participant shifts attention back to the game, in order of who must play next first Adam, whose game turn it is, and who is most responsible for making the next move, then Kat, and finally John, who shifts his body and gaze back to the game. John thereafter restricts his participation with

78

E. Hofstetter and J. Robles

Fig. 3.2 Attention shift to Tara

Fig. 3.3 All players orient to Tara

Tara in a way that lets him also attend the game (L22–27). Thus, at the start of this sequence, John manages dual involvements (Raymond and Lerner 2014)—one in the game, one out of it—and in coordination with his other game players, indicates that the game is his primary

3 Metagaming and Multiactivity

79

involvement (Schegloff 1998). The interaction comes off so that the parallel sequence between John and Tara, while garnering the attention of Kat and Adam, does not disrupt or notably slow the progress of the game. John’s contributions are likewise designed to maximize his ability to monitor the game’s progress and make a timely contribution when it is his game turn. At the end of this exchange (L28–31), the participants orient wholly to progressing the game. They barely talk except to perform game moves (L31), and their actions are directly related to progressing the game; multiactivity is no longer relevant. What we find is that, overwhelmingly, people treat instances of nongameplay that are simultaneous with gameplay unproblematically, as long as the gameplay itself can continue to progress. Although we are distinguishing as analysts between action that progresses the game, and action that does not (Vatanen, this volume), the members sustain and orient these complex multiple orientations through multiactivity. Where the multiactivity is sustainable is where both (or however many) activities can progress without hindering the progressivity of other simultaneous activities. There is no evidence in this example any magic circle is broken. The players organize Tara’s speaking turns as non-game activity, by virtue of their orientation to Tara’s speaking turns as not relevant to the game, but also as activity that is sustainable with the game and non-invasive, as they do not sanction John or Tara for interrupting, distraction, or delay. This is very much consistent with the preference for progressivity in conversation itself: as conversation analytic research has shown, participants do not sanction simultaneous behavior as “interruptive” or “not paying attention” unless there is a normative reason to do so. In order to comply with this expectation, participants will usually provide accounts when their simultaneous multiple activities might infringe upon the primary involvement. For example, so-called common sense and self-reports suggest that people find co-present use of mobile phones “rude”; however, actual observations (e.g., Ling 2008) challenge this; and analysis of mobiles in interaction demonstrates that participants tend to orient to mobile phones at interactionally-appropriate moments, for example, when one is not the currently-addressed recipient in a three-party conversation (e.g., DiDomenico et al. 2020).

80

E. Hofstetter and J. Robles

The remainder of this chapter aims to investigate when and how multiactivity during gameplay is treated as morally sanctionable and unacceptable, and how players organize the “interruptability” of the magic circle. We will suggest that the basis of this organization is maintenance of progressivity, and that players only orient to a magic circle as a thing to be maintained when progressivity is threatened. In the next sections, we will look at the different types of action that can occur during game play that ostensibly might delay progressivity. We discuss (1) in-game talk that directly progresses the game (such that the delay is relevant), (2) in-game talk that orients to but does not directly progress the game, and (3) in-game talk/activity that cannot be sustained with gameplay.

3.3.2 Metagaming and Progressivity In this examination, we will see how players still demonstrate orientation to their accountability for maintaining progressivity, and how they manage the delays and “distractions”. These results offer a different way to think about interaction around what counts as gameplay (vs. additional layers of complex activity organization) from participants’ situated actions, offering a new way to address conversations about both magic circles and metagaming, and contributing to sociological understandings of how gamers manage multiactivity in the progress of their gameplay.

3.3.2.1 Talk About the Game, During the Game that Promotes Progressivity of the Game In this section, we will show talk that occurs during game play that does not hinder progressivity, but in fact promotes it. The existence of this talk demonstrates that not all talk in game play is automatically oriented to as metagaming or outside the magic circle, contra findings that rely on players’ conscious reflections (e.g., Bergström 2010; Salen and Zimmerman 2004). In the following example, the player “Bird” is completing his move in the game Dominant Species, in which animal classes compete for survival.

3 Metagaming and Multiactivity

81

Bird is able to place cubes on three tiles. He picks a tile where Spider already has cubes. The players are competing for who has the most cubes on certain tiles, so this move increases the competition between these players. (3) (160908_DominantSpecies_pt1_14.25)

Bird announces his choice for how to play his game turn (L1) which is ratified by the other players (L2–3). He then begins to narrate the action that constitutes his game turn, of four on here: (L4–5), as he places the cubes. All of these actions so far support the progressivity of the game; announcing his choice and narrating what his choice entails makes his game turn available for other players, allows other players to comment on the move for its validity, and to keep track of the game changes. Spider then makes a moaning complaint sound (L9), drawing on conventional crying “wah” sounds (see Hofstetter, 2020a), which demonstrates his dislike of Bird’s move but also ratifies it as valid and complete. All the players have now shown orientation to the unfolding of Bird’s game turn and helped progress it forward through ratification of the move, so that the game turn can now be completed. We will address Bird’s Hi (L5) in the next section. Thus at least, some talk and action during game play supports the progressivity of the game. Such “announcements” of game events, although not-game actions themselves, are ubiquitous in the data and appear in most game turns, in every recording. The following examples focus on disruptions to progressivity.

82

E. Hofstetter and J. Robles

3.3.2.2 Talk About the Game, During the Game, that Does Not Promote Progressivity of the Game In this section, we will discuss examples of talk or action that are ostensibly “on topic”, but which do not promote progressivity or show orientation to it directly. We suggest these instances are moments of multiactivity and are treated as part of the game. First, we return to the example just seen in the prior section (Example 3). If we show the events that occur subsequently, we see that talk arises that does not itself promote progressivity. (4) (3 continued)

We see again Bird announcing his game turn, which makes it available for others to ratify and react. However, then Bird temporarily suspends the narration of his actions and shifts his footing (Goffman 1981). He addresses Spider with Hi:: (L5) and claims to be joining Spider. The extreme high pitch associated with his speech suggests Bird is embodying his character, i.e., the “bird” cubes that he is placing are greeting the “spider” cubes (see Klewitz and Couper-Kuhlen 1999 on how prosody can function to enact different voices). Spider responds with a sound conventionally associated with whining (L9), wah:. Bird repeats his greeting, and then narrates his action in a way that both continues the characterization of his action as a “greeting” and also downgrades the competitiveness of his move—he is “just joining” Spider

3 Metagaming and Multiactivity

83

(rather than taking over the tile with more cubes). Spider uses a cartoonylike wobbly voice, possibly also voicing his “character”, protesting Bird’s move (L11). Bird then returns to narrating in a normal voice as he chooses two other tiles to play on (L12–13). This prosodically unusual interlude still concerns the game, even if it does not progress the game— it narrates the events in a playful manner, helping to index and enact the playful nature of the game context (Bateson 2006). This is precisely the kind of talk-in-interaction one might expect to support the “magic circle” or lusory agreement, rather than detract from it or be problematic the way that metagaming and action “outside” the magic circle is reported to be. In situ, the players display no indication that it is a problematic exchange, but instead ratify and align with the playfulness (and competitiveness) together. Furthermore, Bird seamlessly joins talk that does not progress his game turn to talk that does progress his game turn and promote the overall progressivity of the game. The interlude above also does not suspend or hinder Bird’s bodily actions, and Bird continues placing tokens and thus progressing the game. Although the players mark the distinction between talk that narrates and talk that makes available Bird’s game moves (particularly through pitch, vocal quality) indicating a shift in footing, they do not treat either as accountable or inappropriate. In summary, this example demonstrates how ostensibly out-of-game talk (talk which does not itself produce or progress game action) is both integrated into the game activity, and treated as an acceptable, indeed fun, contribution. In the next example, from the same game, it is Bird’s turn to play. While Bird displays actions showing the ongoing progression of his game turn, the other players discuss the game itself (Fig. 3.4).

84

E. Hofstetter and J. Robles

Fig. 3.4 Bird looking at board and preparing to play, other players talking

(5) (160908_DominantSpecies_pt1_1.04.13)

Bird’s talk (L1–3) is designed as “self-talk” (Keevallik 2018), in that the vocal quality and content do not encourage responses from the other players. The non-lexical sounds in particular (L3) are not clearly articulated as actual words. The other players take Bird to be speaking in a way

3 Metagaming and Multiactivity

85

that does not require a response, given that Reptile speaks over Bird’s talk (L4). In this way, the players treat Bird’s game turn as ongoing, but not complete, and Bird’s talk as supporting the continuation of his own game turn, rather than as discussion inviting response (see Hofstetter 2020b for a fuller discussion of this kind of self-talk with co-occurring side talk). Bird also maintains his gaze at the board, excluding gaze from the other players, even when responding to Reptile (L7). Reptile’s talk, however, does not progress the game. Reptile assesses the game experience so far, and why he likes the particular game they are playing (L4–9), and Spider ratifies the assessment of the game as something that involves constantly planning. Meta-comments about the game, like Reptile’s, are also not especially common, but always involve a similar interplay of assessments, and occur at moments when they do not hinder the game’s progressivity; they are instances of concurrent multiactivity. All three players are locally managing multiactivity by orienting to dual involvements during the game, such as when Bird’s vocalization attends to the discussion of the game (L7) while bodily orienting to progressing game play (L2 onward). In this sense, the game play is the main involvement (in Goffman 1963’s terms), as its progressivity is prioritized (see also Example 5). The talk above is unquestionably about the game, since it assesses the experience of the game itself. Furthermore, it is produced at a moment when the game turns cannot progress beyond Bird. Bird’s game turn may itself be progressing, and visible as such through Bird’s displays, but it is not possible for other participants to play yet. By positioning the assessment at this moment, Reptile prevents such talk from hindering progressivity at some other point—for instance, instead of initiating this talk while other players waited for Reptile to begin and take his own game turn. Reptile has topicalized the game and made it part of the explicit discussion, which does not itself promote progressivity, but he does it in a way that is sensitive to the context of progressing game turns. Players may thus be able to accomplish a great deal of action concurrent to the game without hindering its progressivity, and without receiving sanctions from other players, as long as they are sensitive to the progression of the game and organize the action with respect to progressivity. In

86

E. Hofstetter and J. Robles

actual gameplay, then, players do not treat all co-occurring action that does not progress the game as somehow problematic, nor as irrelevant to their ongoing social activity, contrary to prior discussions of the magic circle and metagaming. There appears to be the possibility, in situ, of doing action that is not strictly gameplay without sanction or trouble, as long as it does not hinder the progressivity of the game. These multiactivity complexities are sustainable and even methods for being sociable and affiliative. We will show further examples that are even less directly associated with the game itself in the next section.

3.3.2.3 Talk That Cannot Be Sustained with an Ongoing Game In this section, we examine moments when players treat some action as unsustainable with the ongoing game. Players attempt to reorient the action to be about the game again, treating the foregoing actions as no longer sufficiently addressing the progressivity of the game (see also Sutinen 2014 on the managed return to the activity of a role-playing game). Below, Tina is taking her game turn and has been audibly debating her choices. James and Hal have been discussing whether some types of armies are better than other types of armies in this game (each “race” of armies has different special abilities). They continue to discuss as Tina places her armies on the board (L11–16). At the end of her game turn, Tina has the option to roll the dice to (hopefully) get a temporary, extra army, and she interrupts the other two players’ discussion to make this dice roll (L17) (Fig. 3.5).

3 Metagaming and Multiactivity

Fig. 3.5 Tina waving dice in Hal’s face

(6) (GN_SmallWorld_28.50)

87

88

E. Hofstetter and J. Robles

Hal and James discuss the types of game “races” and their benefits (L5–16) while Tina gazes at the board (L2), and self-talks about her moves (L4, 6). Tina announces her moves as they occur (L12), although the others do not take the opportunity to comment. Their ongoing discussion is available to Tina, who can see that they are not displaying attention to her ongoing game turn (see also Example 5). When it comes to rolling the dice, Tina requires the other players’ attention. Dice results are a witnessable event—that is, the witnessing of the event as it happens provides for its validity (e.g., the dice were rolled properly rather than “placed down’). As Tina picks up the dice, Hal’s gaze goes toward the board (L18–19), but Tina does not treat that is sufficient attention, and Hal is still verbally continuing discussion of the “races”. Tina holds the dice in front of Hal’s face (L18) and states she needs to roll the dice (L17). At these two actions, Hal ceases his talk, leaving his utterance incomplete (L16). Tina’s claim to “need” to roll the dice is a sufficient account for Hal and James to provide undivided attention to the dice rolling and witness the event. James and Tina react to the result of the dice roll in tandem, in overlap, showing the mutual coordination of attention to the dice has resulted in simultaneous reactions. The reactions also provide evidence that the roll has been witnessed and is being treated as valid. There are moments, then, when multiactivity is treated as unsustainable, and the game is given priority while other actions are suspended (see examples in Haddington et al. 2014; Vatanen, this volume). Dice rolls require witnessing to be legitimate, and so may be a site where concurrent talk is regularly unsustainable or avoided. The reverse can also happen, where an activity that is not itself play can disrupt the ongoing course of play and cause it to be suspended temporarily. In the next example, Adam completes his move, and it becomes John’s game turn. However, at that moment, a token is heard dropping to the ground (L5). The play is suspended to deal with picking up the piece.

3 Metagaming and Multiactivity

89

(7) (160716_TashKalar_pt2_22.52)

Just as Adam passes the game turn to John (L7), the token makes a sound as it drops. This is not attended until Kat observes that something dropped (L9), at which mention all players look to the ground—the noticing affects the multiactivity (Helisten 2019). Although John originally had begun showing attention to his cards and his game turn, once Kat observes the dropping, all players redirect their attention to the dropped item. It being closer to John, he picks it up and returns it to Kat’s play area (L13–15). Kat apologizes (L12), identifying the problem as the piece sticking to her arm and then dropping as she moved. She

90

E. Hofstetter and J. Robles

thereby claims responsibility not just for her piece and failure to keep it on the table, but also for disrupted the game. John describes the size of the table, which, being only just big enough to hold all their game boards and tokens, may have caused Kat to have to place her arms down on top of her tokens rather than in a free zone of table. It being John’s table, he is in a sense accounting for its size. Kat shows some orientation to mitigating this potential accountability by claiming to like the size of the table (whatever difficulty it may have caused her) (L34). John explicitly marks the return of his focus to his game turn (L39). The process of returning to the game after retrieving the piece is not straightforward (see Sutinen 2014). John’s observation of the table is complicated by his ownership of it, making it relevant for the other two to counter any potential criticism. Play could have resumed after line 23, and John’s gaze is largely focused on his cards and the board (L24, 33), but progress is interrupted by Adam and Kat’s counterings of any criticism of the table. The interruption of game progressivity is thus prolonged, step by step, beyond the retrieval moment, and it is a live issue for them to solve how and when they will return to the game. The literature definitions of the magic circle and metagaming might point to these examples as instances that show multiactivity during gameplay is unacceptable. However, as we have shown in the collection as a whole, what these examples demonstrate is that players use progressivity as basis for the appropriateness of multiactivity conduct. Example 7 reinforces that participants prioritize solving trouble that arises (the dropped game piece) over progressivity (see Kamunen 2019), while Example 6 emphasizes that players monitor each other’s multiactivity engagements enough to be able to reorient them to a key gameplay moment as necessary (suspending talk for a dice roll).

3.4

Discussion

In this paper, we have taken up Juul’s (2008) call to investigate game boundaries as they are negotiated and made relevant in actual circumstances. This has led to an emic, situated analysis of “metagaming’, in the sense of action (including embodied movement and talk) that

3 Metagaming and Multiactivity

91

occurs during game play but is not itself a game action. Metagaming has been described as events “around” the game, which we conceptualize as multiactivity. We have shown several different types of this metagaming, but all are organized around a central principle: at all times, minimize the disruption to the progressivity of the game. Where an event forces one to suspend the progressivity, make that suspension as minimal as possible. Where players fail to uphold this principle, they are vulnerable to sanction, and accountable for not supporting progressivity. Our analysis of metagaming shows a very different perspective on the concept than prior work. Although players may consciously label non-game activity as problematic in interviews, during the game itself there are few circumstances that are actually oriented to as problematic, despite the complexities arising from multiple concurrent tasks. Instead, players manage games as sites of multiactivity (Haddington et al. 2014) without difficulty, skillfully displaying their attention at crucial moments in ways that support the ongoing collaboration of playing the game. The complexity of multiactivity does not pose difficulty for the participants, even while it does impose moral requirements on players for how to properly participate. Players do engage in practices to uphold at least some distinctions between “game” and “not-game”—their multiactivity orientations are produced (Raymond and Lerner 2014), and that display of activity distinction becomes a resource for demonstrating attention and engagement. For instance, players treat bodily orientation to the board and table as attention to the game, and take advantage of this resource to actively demonstrate attention elsewhere at times, and at other times to display multiple simultaneous activity involvements (e.g., Example 1). All this raises questions about the relevance of the magic circle as an emic category. In orienting to some action as not progressing the game (or supporting the progression of the game), players treat that action as not part of the game itself; it is ostensibly metagame, outside the magic circle as the games literature defines. On the other hand, these actions are carefully interwoven into the game interaction: they could not exist in the same form were they not part of the game situation. Furthermore, since activities around the game are organized for their sustainability with play; they may actually support ongoing play. “Side talk” while a player debates their choices (Examples 5 and 6) permits the active player to

92

E. Hofstetter and J. Robles

delay their game turn and reduces pressure to hurry (Hofstetter 2020b). Disrupting or suspending progressivity to solve trouble (Example 7) allows for the game to maintain its coherency (with all pieces, all rules, etc.). Assessing the game (Example 5) and joking about game moves (Example 3 and 4) provides opportunities for affiliation. All of these layers of complexity make the game more sociable in a colloquial sense, as well as more feasible to play in a practical sense. As such, the magic circle may be less informative as a distinction between game and non-game and may be better treated as a resource for managing the local complexities of playing the game and players’ orientations to task-relevant issues. For example, using game-specific categories can be done to manage the accountability of complaints (Sterphone 2022), while labeling action as “non-game” can be a means for reasoning for its inappropriateness. This is an important alternative, because the role of progressivity in determining what is accountably “metagaming” or “out of game” in an actual game event demonstrates the way nongame norms (everyday interactional norms) permeate and facilitate game norms; categories of activity are resources for doing interactional work and facilitating reasoning about joint activity. A question to ask in future studies is whether non-habitual gamers, who have not been socialized into game norms, show the same orientations and/or how they learn such orientations to metagaming and progressivity. We may see then how board gamers police and teach the acceptable forms of metagaming and progressivity to newcomers, and how that structures a broader culture for board game play. Beyond gaming, such a reinterpretation of metagaming and the magic circle has consequences for Goffmanian notions of context (e.g., 1963). Rather than seeing context as a governing force, the practices analyzed in this paper show how participants can use “activity” as a morally organizing principle, with which to do action, hold each other accountable, and facilitate progress of the activity at hand (Evans and Fitzgerald 2016; Evans 2013). The playing of a game may necessitate a multiactivity scenario, rather than exclude it via a magic circle, in order to draw on norms that come from everyday interaction, as well as to organize the game activity as a focus of progress.

3 Metagaming and Multiactivity

93

Acknowledgements The authors of this paper would like to thank several reviewers, as well as feedback from the 2017 National Communication Association and 2018 International Communication Association attendees. This work was funded in part by the Swedish Research Council VR2016-00827.

References Abdul Jabbar, A.I., and P. Felicia. 2015. Gameplay Engagement and Learning in Game-Based Learning: A Systematic Review. Review of Educational Research 85: 740–779. https://doi.org/10.3102/0034654315577210. Baldauf-Quilliatre, Heike, and Isabel Colón de Carvajal. 2020. Encouragement in Videogame Interactions. Social Interaction. Video-Based Studies of Human Sociality 2 (2). https://doi.org/10.7146/si.v2i2.118041. ———. 2021a. Co-Constructing Presence Between Players and Non-Players in Videogame Interactions: Introduction to the Special Issue. Journal Für Medienlinguistik 4 (2): 1–13. https://doi.org/10.21248/jfml.2021.46. ———. 2021b. Spectating: How Non-Players Participate in Videogaming. Journal Für Medienlinguistik 4 (2): 123–61. https://doi.org/10.21248/jfml. 2021.33. Bateson, G. 2006. A Theory of Play and Fantasy. In The Game Design Reader: A Rules of Play Anthology, edited by K. Salen and E. Zimmerman. Boston: MIT Press. Bergström, K. 2010. The Implicit Rules of Board Games: On the Particulars of the Lusory Agreement, 86–93. Tampere, Finland. Betz, E., and A. Deppermann. 2018. Indexing Priority of Position: Eben as Response Particle in German. Research on Language and Social Interaction 51: 171–193. https://doi.org/10.1080/08351813.2018.1449449. Consalvo, M. 2009. There Is No Magic Circle. Games and Culture 4: 408–417. https://doi.org/10.1177/1555412009343575. DiDomenico, S.M., J. Raclaw, and J.S. Robles. 2020. Attending to the Mobile Text Summons: Managing Multiple Communicative Activities Across Physically Copresent and Technologically Mediated Interpersonal Interactions. Communication Research 47: 669–700. https://doi.org/10.1177/009365021 8803537.

94

E. Hofstetter and J. Robles

Donaldson, S. 2017. Mechanics and Metagame: Exploring Binary Expertise in League of Legends. Games and Culture 12: 426–444. https://doi.org/10. 1177/1555412015590063. Drew, P., and K.H. Kendrick. 2018. Searching for Trouble: Recruiting Assistance through Embodied Action. Social Interaction. Video-Based Studies of Human Sociality 1. https://doi.org/10.7146/si.v1i1.105496. Evans, B. 2013. Order on the Court : The Interactional Organization of Basketball Practice Activities. Doctoral thesis, University of Western Sydney. Evans, B., and R. Fitzgerald. 2016. ‘It’s Training Man’! Membership Categorization and the Institutional Moral Order of Basketball Training. Australian Journal of Linguistics 36: 205–223. https://doi.org/10.1080/07268602. 2015.1121531. Fine, G.A. 1981. Fantasy Games and Social Worlds: Simulation as Leisure. Simulation & Games 12: 251–279. https://doi.org/10.1177/104687818101 200301. Goffman, E. 1961. Encounters: Two Studies in the Sociology of Interaction. Indianapolis: Bobbs-Merrill Company. ———. 1963. Behaviour in Public Places: Notes on the Social Order of Gatherings. New York: The Free Press. ———. 1981. Forms of Talk. Philadelphia: University of Pennsylvania Press. Goodwin, C. 2007. Participation, Stance and Affect in the Organization of Activities. Discourse & Society 18: 53–73. https://doi.org/10.1177/095792 6507069457. Goodwin, Marjorie H. 1997. Byplay: Negotiating Evaluation in Storytelling. In Towards a Social Science of Language: Papers in Honor of William Labov. Volume 2: Social Interaction and Discourse Structures, edited by G.R. Guy, C. Feagin, D. Schiffrin, and J. Baugh, 77–102. Amsterdam: John Benjamins Publishing. Haddington, P., T. Keisanen, L. Mondada, and M. Nevile, eds. 2014. Multiactivity in Social Interaction: Beyond Multitasking. Amsterdam: John Benjamins. Helisten, M. 2019. Disjunctively Positioned Problem-Noticings in Managing Multiactivity. Research on Language and Social Interaction 52: 318–342. https://doi.org/10.1080/08351813.2019.1657274. Hofstetter, E. 2020a. Nonlexical “Moans”: Response Cries in Board Game Interactions. Research on Language and Social Interaction 53: 42–65. https:/ /doi.org/10.1080/08351813.2020.1712964.

3 Metagaming and Multiactivity

95

———. 2020b. Thinking with the Body: Embodying Thinking as a Practice in Board Games. In Discursive Psychology and Embodiment: Beyond SubjectObject Binaries, edited by S. Wiggins and K.O. Cromdal, 247–73. Palgrave Studies in Discursive Psychology. Palgrave Macmillan. https://doi.org/10. 1007/978-3-030-53709-8. ———. 2021. Achieving Preallocation: Turn Transition Practices in Board Games. Discourse Processes 58: 113–133. https://doi.org/10.1080/016 3853X.2020.1816401. Hofstetter, E., and J. Robles. 2019. Manipulation in Board Game Interactions: Being a Sporting Player. Symbolic Interaction 42: 301–320. https://doi.org/ 10.1002/symb.396. Jakobsson, P., D. Pargman, and J. Rambusch. 2007. Exploring E-Sports: A Case Study of Game-Play in Counter-Strike. In Proceedings of the 2007 DiGRA International Conference: Situated Play. from http://www.digra.org/ digital-library/forums/4-situated-play/. Jefferson, G. 1986. Notes on “Latency” in Overlap Onset. Human Studies 9: 153–183. https://doi.org/10.1007/BF00148125. ———. 2004. Glossary of Transcript Symbols with an Introduction. In Conversation Analysis: Studies from the First Generation, edited by G.H. Lerner, 13–31. Pragmatics & Beyond. Amsterdam: John Benjamins. Juul, J. 2008. The Magic Circle and the Puzzle Piece. In Conference Proceedings of the Philosophy of Computer Games, edited by S. Günzel, M. Liebe, and D. Mersch, 056–067. Potsdam: University of Potsdam Press. Kamunen, A. 2019. How to Disengage: Suspension, Body Torque, and Repair. Research on Language and Social Interaction 52: 406–426. https://doi.org/10. 1080/08351813.2019.1657287. Keevallik, L. 2018. Sequence Initiation or Self-Talk? Commenting on the Surroundings While Mucking Out a Sheep Stable. Research on Language and Social Interaction 51: 313–328. https://doi.org/10.1080/08351813.2018. 1485233. Kendrick, K.H., and P. Drew. 2016. Recruitment: Offers, Requests, and the Organization of Assistance in Interaction. Research on Language and Social Interaction 49: 1–19. https://doi.org/10.1080/08351813.2016.1126436. Klabbers, J. 2009. The Magic Circle: Principles of Gaming & Simulation. Rotterdam: Sense Publishers. Klewitz, G., and E. Couper-Kuhlen. 1999. Quote—Unquote? The Role of Prosody in the Contextualization of Reported Speech Sequences. Pragmatics 9: 459–485. https://doi.org/10.1075/prag.9.4.03kle.

96

E. Hofstetter and J. Robles

Liberman, K. 2013. More Studies in Ethnomethodology. SUNY Series in the Philosophy of the Social Sciences. Albany: State University of New York Press. Ling, R. 2008. Mobile Telephones and the Disturbance of the Public Sphere. Europe 115: 1–17. Mondada, L. 2012. Coordinating Action and Talk-in-Interaction in and Out of Video Games. In The Appropriation of Media in Everyday Life, edited by R. Ayaß and C. Gerhardt, 231–270. Amsterdam: John Benjamins. ———. 2018. Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing Multimodality. Research on Language and Social Interaction 51: 85–106. https://doi.org/10.1080/08351813.2018.1413878. Moore, C. 2011. The Magic Circle and the Mobility of Play. Convergence 17: 373–387. https://doi.org/10.1177/1354856511414350. Navarro-Newball, A.A., D. Loaiza, C. Oviedo, A. Castillo, A. Portilla, D. Linares, and G. Álvarez. 2014. Talking to Teo: Video Game Supported Speech Therapy. Entertainment Computing 5: 401–412. https://doi.org/10. 1016/j.entcom.2014.10.005. Neuenschwander, B. 2008. Playing by the Rules: Instruction and Acculturation in Role-Playing Games. E-Learning and Digital Media 5: 189–198. https:// doi.org/10.2304/elea.2008.5.2.189. Pargman, D., and P. Jakobsson. 2008. Do You Believe in Magic? Computer Games in Everyday Life. European Journal of Cultural Studies 11: 225–244. https://doi.org/10.1177/1367549407088335. Piirainen-Marsh, A., and L. Tainio. 2009. Collaborative Game-Play as a Site for Participation and Situated Learning of a Second Language. Scandinavian Journal of Educational Research 53: 167–83. https://doi.org/10.1080/003138 30902757584. Pomerantz, A. 2012. Do Participants’ Reports Enhance Conversation Analytic Claims? Explanations of One Sort or Another. Discourse Studies 14: 499– 505. https://doi.org/10.1177/1461445611434229. Raymond, G., and G.H. Lerner. 2014. A Body and Its Involvements: Adjusting Action for Dual Involvements. In Multiactivity in Social Interaction, edited by P. Haddington, T. Keisanen, L. Mondada, and M. Nevile, 227–46. Amsterdam: John Benjamins. https://doi.org/10.1075/z.187.08ray. Rossi, G. 2015. The Request System in Italian Interaction. Nijmegen: Radboud University. Sacks, H., E.A. Schegloff, and G. Jefferson. 1974. A Simplest Systematics for the Organization of Turn-Taking for Conversation. Language 50: 696. https://doi.org/10.2307/412243.

3 Metagaming and Multiactivity

97

Salen, K., and E. Zimmerman. 2004. Rules of Play: Game Design Fundamentals. London: MIT Press. Schegloff, E.A. 1998. Body Torque. Social Research 65: 535–596. Sidnell, J., and T. Stivers, eds. 2013. The Handbook of Conversation Analysis. Wiley. Sterphone, J. 2022. Complaining by category: Managing social categories and action ascription in wargame interactions. Language & Communication 84: 46–60. https://doi.org/10.1016/j.langcom.2022.03.001. Stivers, T., and J.D. Robinson. 2006. A Preference for Progressivity in Interaction. Language in Society 35: 367–392. https://doi.org/10.1017/S00474045 06060179. Sutinen, M. 2014. Negotiating Favourable Conditions for Resuming Suspended Activities. In Multiactivity in Social Interaction: Beyond Multitasking, edited by P. Haddington, T. Keisanen, L. Mondada, and M. Nevile, 137–166. Amsterdam: John Benjamins. Tekin, B.S., and S. Reeves. 2017. Ways of Spectating: Unravelling Spectator Participation in Kinect Play. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 1558–70. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3025453. 3025813. Viney, R. 2015. Everyday Interaction in Lesbian Households: Identity Work, Body Behaviour, and Action. Thesis, Loughborough University.

4 Embodied Noticings as Repair Initiations: On Multiactivity in Choir Rehearsals Anna Vatanen

4.1

Introduction: Musical Rehearsals and Multiactivity

Previous conversation analytic studies on musical rehearsals have predominantly investigated the rehearsals as instructional interaction and focused on various epistemic and deontic dimensions in the setting. The conductor’s (or teacher’s) instructions, evaluations and corrections have been studied in orchestra and choir rehearsals (Emerson 2018; Emerson et al. 2019; Merlino 2014; Parton 2014; Stoeckl and Messner 2021; Weeks 1996), ensemble music workshops (Veronesi 2014), vocal master classes (Szczepek Reed et al. 2013; Reed and Szczepek Reed 2014) and private instrument lessons (Nishizaka 2006; Stevanovic 2017, 2020; A. Vatanen (B) Faculty of Humanities, University of Oulu, Oulu, Finland e-mail: [email protected] University of Helsinki, Helsinki, Finland © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_4

99

100

A. Vatanen

Tolins 2013). However, a musical rehearsal is not just about instructional interaction but involves various other dimensions, too. Especially when a group of musicians acts as a collective, other forms of interaction occur as well. For example, Nishida and Yokomori (2018) looked at how a string quartet that operates without a conductor jointly achieves the stopping and restarting of play during their rehearsals, and Balantani (2022) studied joint decision-making within groups of two or three equal musicians when they practice together. The current study starts from the observation that there is an opportunity in choir rehearsals for multiple participation frameworks (Goffman 1981; M. H. Goodwin 1997; Goodwin and Goodwin 2004) to occur simultaneously, and the members of the choir may attend to more than one activity at the same time. The singers may participate in at least two simultaneous participation frameworks: first, sing together with the whole choir (as a member of a collective; cf. Lerner 1993) that is managed by the conductor, and second, engage (via talk or bodily conduct) in emergent, ad hoc participation frameworks with the fellow singers—mostly those seated nearby, in other words, the singer’s neighbours. The current collection involves neighbour interaction that is initiated when one singer produces an embodied noticing of something unexpected in the singing and another singer responds (or does not; cf. Kääntä 2014). These noticing sequences will be examined in this study. During the rehearsals, the conductor guides the choir using directives and assessments (see Emerson 2018; Emerson et al. 2019) as well as nonlexical turns (see Tolins 2013), exerting his/her deontic authority (see Stevanovic and Peräkylä 2012) to determine what is done and when. However, most of the time during the rehearsals the conductor conducts (silently) and the choir sings, following the conducting. The latter types of moments are the focus of the current study. During these moments, it occasionally happens that a singer sings something else than what is written in the musical score he/she is supposed to follow. In the current data, the conductor, for the most part, just lets these events pass without paying attention to them, while the singers more often visibly orient to them. The singer whose singing departs from the expectations (I will hereafter, as a shorthand, use the words “mistake” and “mistakemaker”) may, for example, frown and momentarily interrupt his/her

4 Embodied Noticings as Repair Initiations

101

singing, sometimes turning to a neighbour. Sometimes, however, it is a neighbour who visibly notices the fellow singer’s mistake. This type of behaviour forms another line of activity (a form of “byplay”; see Goffman 1981: 134; M. H. Goodwin 1997) that progresses while the conductor conducts and (the rest of ) the choir sings. Furthermore, at these moments, the singers may be involved in multiactivity (e.g., Haddington et al. 2014; Mondada 2011, 2012, 2014); in other words, they may observably attend to more than one activity at a time, coordinating and enabling the progression of these activities with various verbal and bodily means (see also Hofstetter and Robles, this volume). In this way, the situations can be regarded as complex, or multi-layered, as the participants need to coordinate their participation in multiple activities simultaneously. The activities—singing and interacting (bodily) with the neighbour—may run in parallel, without one disturbing the other in any way. At certain moments, however, one activity may have to be momentarily abandoned in order to carry out the other—for example, it is impossible to sing and speak at the same time—in which case the activities progress in a mutually exclusive fashion (“exclusive order”). The progression of the activities may also be embedded within one another (on the orders of multiactivity, see Mondada 2014). One of the activities, singing, follows a temporal and rhythmic organisation, whereas the other activity, social interaction with a neighbour, is mostly sequentially organised. Similar activity occurs also in other types of settings. In classrooms, for example, pupils may interact with each other while still (at least partly) orienting to and following the teacher’s teaching, which is the official, common agenda of the lesson. Students’ parallel activities (or byplay) that are verbal have been termed desk talk (Sahlström 1999) and side talk (Lemke 1990). Even though in principle forbidden, they have several benefits according to Lemke (1990): they are argued to maintain the students’ mutual relationships and to give them a possibility to discuss the topic of the lesson and solve problems related to it—which is what occurs in the choir rehearsals, too. In his micro-ethnographic, ethnomethodological study, Koole (2007) argues that when engaging in parallel activities, students nevertheless display in various ways an orientation to the activity led by the teacher as the central activity. Sahlström

102

A. Vatanen

(1999) shows, using conversation analysis (CA), that this type of simultaneous orientation may be visible in the students’ body posture and gaze behaviour, for example; this too happens in the current choir data. Lehtimaja (2012) further analyses the ways in which parallel conversations among students are initiated, maintained and terminated, and how orientation to the parallel conversations (one teacher-led and another among the students) is displayed. It has been argued that parallel conversations among students are often initiated when there is a transition or a (short) quiet phase in the teacher-led activity (e.g., Markee 2005). In contrast, the choir singers’ neighbour interaction analysed here occurs during the conductor-led singing activity. The analysis presented in this study is conducted using multimodal conversation analytic methods (see, e.g., Sidnell and Stivers 2013), which entail close moment-to-moment sequential analysis of interaction. The study asks: how do the participants orient to and make visible making a mistake or having a problem in singing, and how are the two activities (singing and interacting with a neighbour) organised as multiactivity? In other words, how do the participants deal with the complexity (or multilayeredness) of the situation? (See Sect. 4.5.1 for a summary and discussion on these questions.) Furthermore, how is this sequence—the (embodied) noticing/repair initiation and its potential response—organised? (See Sect. 4.5.2 for a summary on repair organisation and Sect. 4.6 for a discussion on noticings in the current context.)

4.2

Noticings of Mistakes, Embodiment and Repair Organisation

As reported above, some troubles or mistakes the choir members make in singing are identified by the participants in the form of noticings, most of them achieved with embodied means. In doing a noticing, a participant retroactively makes a feature of the setting relevant and mobilises attention to it—here, to a mistake or a problem in singing—and thus transforms a perceptual/cognitive event into a potentially interactional event (Schegloff 2007: 87, 219). Only some of the potential sources for noticings are singled out as publicly noticed by the participants

4 Embodied Noticings as Repair Initiations

103

(Keisanen 2012: 201). Keisanen, examining noticings uttered by car drivers, points out that those “noticings that make salient some trouble are used as displays of accountability and for assuming responsibility for one’s actions” (Keisanen 2012: 197). This is the case also for the noticings made by the singers in the current study: these noticings show that the singers display accountability and responsibility over their singing, including the mistakes they make. Besides being retrospective, noticings also proactively project candidates for relevant next actions (Schegloff 2007), which in the current context concern correcting the mistakes made. Contrary to most conversation analytic literature on noticings that has investigated verbal turns (e.g., Keevallik 2018; Keisanen 2012; Laanesoo and Keevallik 2017; Rauniomaa et al. 2018; Siitonen et al. 2021), the instances in the current collection are mostly accomplished with bodily means only. Kääntä (2014), studying classroom interaction, examined embodied noticings done by students. With sudden gaze shifts (see also Kidwell 2009) and facial expressions, the students publicly displayed having noticed a mistake made by the teacher. These embodied displays, most of them followed by a verbal correction initiation, conveyed the students’ knowledgeable epistemic status in relation to the trouble source (the teacher’s mistake). In a similar manner, in the current choir data, the singers’ bodily behaviours display their knowledge, conveying their awareness of the mistake they just made. Making a mistake is thus oriented to as an accountable action. Helisten (2019) investigated noticings in everyday situations involving multiactivity. In her data, the noticings, launched in an embodied manner and explicated verbally, occasioned an intervening course of action that temporarily suspended the progression of the current main activity. In the current data, to the contrary, there are various ways in which the ongoing activities are organised, as will be analysed below; however—one exception aside—the activity of joint singing by the whole choir is not interrupted or suspended. The embodied noticings examined by Oittinen (2020), on the other hand, “draw attention to technical problems that need to be fixed” as well as “function as prefaces to the interactional work required to progressively recover the interactional space” (Oittinen 2020: 4). The noticings examined here

104

A. Vatanen

similarly draw attention to “problems”, but less so function as prefaces to interactional remedial work. Instead, the remedial work—attempting to sing correctly—is most often embedded in the activity as it continues, achieved when the problematic part is sung the next time. This is due to the collective nature of the activity: progressing the joint singing, without interruptions, is oriented to as being of primary importance in the setting (more discussion on this question will follow below). Most noticings examined in the previous literature draw attention to something in a co-participant’s action or something that is (mainly) relevant for a co-participant (e.g., Helisten 2019; Kääntä 2014; Laanesoo and Keevallik 2017; Oittinen 2020; Rauniomaa et al. 2018), or to something in the environment (e.g., Keevallik 2018; Siitonen et al. 2021). Most of the noticings examined here, to the contrary, mobilise attention to a feature of one’s own actions (see also Keisanen 2012). As Kääntä (2014: 103) analyses, embodied noticings do not themselves identify the noticed feature but rather make visible that something has been noticed. In Kääntä’s classroom data (see also Helisten 2019), the verbal turns that follow the noticings then subsequently identify the trouble source. In the current choir data—involving advanced amateurs and professionals—it seems rather obvious that the embodied noticings point to some mistakes in singing as their source, yet the precise nature of the mistake remains unidentified, or at least unexplicated. The bodily responses that follow some of the noticings do not explicitly identify the trouble source either (for the participants, however, this is not an issue, as they seem to know what they are responding to). This study, for its part, thus responds to Kääntä’s (2014: 103) plea for more research on if and how embodied noticings are responded to by others. This study also adds to the existing body of literature that examines how noticings are recurrently occasioned by a trouble of some sort and are followed by a remedy of the trouble (see, e.g., Helisten 2019; Keisanen 2012; Kääntä 2014; Laanesoo and Keevallik 2017; Oittinen 2020). Moreover, it is fruitful to examine the mistakes, noticings and their responses that occur in the data in terms of repair organisation (e.g., Schegloff et al. 1977; Jefferson 1987). Both noticings and repair initiations are actions that are launched from second position and identify something in the prior as their source (Schegloff 2007: 217–219). A

4 Embodied Noticings as Repair Initiations

105

mistake a singer makes becomes a trouble source when a participant shows orientation to it, which here happens in the form of a(n embodied) noticing. The noticing itself functions as a repair initiation, and it may be produced either by “self ” (i.e., the mistake-maker) or “other” (a fellow singer); in other words, there are both self-initiations and otherinitiations of repair in the data. As the embodied noticing does not single out what exactly was problematic in the prior, it is an “open” repair initiation (see Drew 1997; Haakana 2011; Dingemanse et al. 2013). Most CA literature on repair organisation has focused on verbal interaction or on embodiment that co-occurs with the verbal. Repair sequences initiated only with embodied means have been studied much less. Seo and Koshik (2010) argue that specific gestures can be used as open repair initiations in conversation, and that they are positioned in turn transition space after the turn that is treated as problematic. Mortensen (2016) focuses on a specific gesture, cupping the hand behind the ear, and shows that it functions to initiate repair on a hearing problem. Oloff (2018) also analyses certain forms of bodily conduct as initiating repair in possible response slots and shows that participants treat them as pointing to either troubles in hearing or troubles in understanding. These three studies (Mortensen 2016; Oloff 2018; Seo and Koshik 2010), as well as some other studies on embodied repair (e.g., Martin and Sahlström 2010), have all focused on different institutional or professional settings. The current study, while also analysing a specific institutional activity, adds to this line of inquiry by addressing “specific task-related repairables” and the “connections [of embodied repair practices] to specific trouble sources”, which have been identified by Oloff (2018: 31–32) as points requiring more research. Furthermore, Lilja (2010: 79–83), addressing bodily repair, stresses the different temporalities of embodiment compared to talk and finds that bodily repair initiations should be analysed in their own right, not just by comparing them to how verbal repair sequences are organised. Analysing (potential) embodied repair initiations (e.g., their positioning) in a collective performance, such as choir singing, adds yet another layer of complexity into the analysis and will be discussed below.

106

4.3

A. Vatanen

The Data: Choir Rehearsals as an Interactional Setting

This study draws on video recordings of naturally occurring choir rehearsals as research data. The corpus consists of recordings of five rehearsals, which lasted 13 hours altogether. The recordings were made with a 360° video camera in the middle of the rehearsal space. In three rehearsals, the 360° camera was supplemented by a wide-angle GoPro camera at the back of the space (see Figs. 4.1 and 4.3). The choir singers sit in two or three semi-circular rows in front of the conductor, who mostly stands by his music stand (see Figs. 4.1, 4.2 and 4.3). The musical scores of the pieces the choir sings are used as supplementary material in the study. Each participant gave their informed written consent regarding the use of the materials in research and related publications. Identifying information such as names has been changed in the transcripts; however, consent was given to use still images from the videos in a non-anonymised form.

Fig. 4.1 Choir setting

4 Embodied Noticings as Repair Initiations

107

Fig. 4.2 A crystal ball view from the 360° camera

Fig. 4.3 View from the GoPro camera

The chosen recording equipment affords varying degrees of access to the individual singers’ voice and bodily behaviour. The singers who are seated closest to the camera—in other words, the front row—are the most visible in the camera view, whereas some singers in the back row are

108

A. Vatanen

not fully visible. Seeing the singers’ mouths is essential in many examples, as it reveals whether the singer sings the right words at the right time (concluded using the musical scores as an aid). How well the individual singers’ voices are audible depends on what else is going on in a given moment; most often the analyst does not have access to the tone/note an individual singer sings but only to each voice (e.g., the soprano) as a whole. In other words, not everything that occurs in this multi-party setting is accessible for the analyst; the analysis is made solely on the basis of what is available in the data. The recorded choir has approximately 50 singers. It is an amateur choir that sings at an advanced level. The choir belongs to a Lutheran congregation in Finland, and its repertoire consists of classical music, mostly religious. The conductor is a professional and is responsible for leading the activities in the rehearsal. He uses, for example, directives and assessments (see Emerson 2018; Emerson et al. 2019) as part of his conducting, directing his turns at various times to the choir as a whole, to one of the voices (sopranos, altos, tenors or basses) or to a smaller subset of the singers. The singers are supposed to follow the conductor’s conducting and sing accordingly, continuously aiming at better performance—as specified by the conductor (cf. Parton 2014). Occasionally, a singer may also ask the conductor a question or otherwise comment on the joint activity. In the “community of practice” (Lave and Wenger 1991; Wenger 1998) of the choir, the members are committed to common goals. In addition to the official goals and agenda of the rehearsals, there are a number of unwritten rules or practices according to which the members act—some of which will emerge in the analysis and discussions below. Singing is hearable to others and thus public. The embodied noticings are visible and thus public, too. The collection that this study draws on consists of 79 cases where someone publicly orients to a mistake or a problem, either their own or someone else’s, while (the rest of ) the choir is singing and the conductor is conducting. In other words, what happens during moments when the choir is not singing (but is, for example, supposed to listen to the conductor’s guidance in silence) is out of the scope of the analysis. Almost all of the cases in the collection (N 74, 94%) come in the form of a facial expression or a body

4 Embodied Noticings as Repair Initiations

109

movement, such as a smile, a frown, a grimace, a head shake, ducking or turning upper body towards a neighbour, yet there are also verbal turns that address a mistake or a problem (N 5, 6%). In the vast majority of the cases (N 72, 91%), the singers notice their own mistakes; 25% of these noticings receive a response from a neighbour. Only in a very small subset (N 7, 9%), the one who produces the noticing is someone else than the mistake-maker, most often the closest neighbour. Due to the participants’ spatial configuration in the analysed choir (see Figs. 4.1, 4.2 and 4.3), the noticings—regardless of whether they receive a response or not—can be thought of as being addressed to the conductor (at least), as he stands in front of the singers, who sit in rows and thus do not see each other as well as the conductor sees them. Quite similar to teachers in a classroom, the conductor is oriented to as having the highest epistemic and deontic authority in the situation (cf. Kääntä 2014). The conductor, however, almost never attends (visibly or verbally) to these mistakes or the noticings they occasioned (an exception to this is analysed in Example 6 below). In the following, the data will be analysed in detail.

4.4

Analysis: Noticings of Mistakes in Singing and Their Responses

The mistakes the choir singers make in singing occasion different noticing sequences in the data. The noticing—that is, the repair initiation—may be produced by the mistake-maker (self-initiation of repair) or a fellow singer (other-initiation of repair). Some of the noticings/ repair initiations receive a response, whereas others do not. In the analysis below, I pay attention to the resources used in these sequences, whether they are bodily, verbal or both, and analyse the ways in which the participants organise their activities as multiactivity. We will see that both the mistake-maker and the fellow singers may organise the simultaneous activity trajectories (singing with the whole choir and orienting to the mistake/problem) in a “parallel order” or in an “exclusive order” (see Mondada 2014).

110

A. Vatanen

The analysis is organised as follows (see also Table 4.1). A main analytical distinction is drawn between self-initiations of repair (noticings of own mistakes) and other-initiations of repair (noticings of others’ mistakes). Another important distinction is whether the noticing and its response are produced with embodied or verbal means; this difference is significant for the analysis of multiactivity in the situation. First, in Sect. 4.4.1.1, I introduce cases involving an embodied noticing of a mistake produced by the mistake-maker her/himself, followed by no response (category 1 in Table 4.1, N 54; 68%). In Sect. 4.4.1.2, I analyse sequences where an embodied noticing of one’s own mistake is followed by an embodied response by a fellow singer (category 2, N 13; 16%). The analysis focuses on embodied noticings, and only one example of a verbal turn addressing a trouble (category 3, N 5; 6%) is shown. In Sect. 4.4.2, I analyse cases where a noticing of a mistake is produced by a fellow singer, not the mistake-maker, that is, other-initiations of repair (category 4, N 7; 9%). The examples are further analysed in terms of the orders of multiactivity: whether the mistake-maker and the fellow singer(s) organise the two activity trajectories (joint singing and neighbour interaction) in a parallel or an exclusive order. After the analytical sections, I summarise the findings with regard to multiactivity (Sect. 4.5.1) and repair organisation (Sect. 4.5.2), and then provide a discussion of the analysed phenomena in Sect. 4.6.

4.4.1 Embodied Noticings as Self-Initiations of Repair 4.4.1.1 Embodied Noticings That Receive No Response The majority of the cases in the data (N 50, 63%) consist of an embodied noticing of one’s own mistake that receives no response. These cases are, at least seemingly, organised as private activity, as no response from the co-participants is even attempted (cf. Keisanen 2012). In addition, there are four cases (5%) where after the embodied noticing of one’s own mistake, the singer pursues a response from a co-participant but receives none (see Example 3 below). Whether the embodied noticing makes a

Sub-category

Verbal

3B

Bodily and/or verbal

Verbal

Bodily

2B

3A

Bodily

Bodily

Bodily

Noticing type

2A

Other-initiations of repair 4. Noticings of 4 others’ mistakes and their responses N 7, 9% Total 79, 100%

3. Verbal turns addressing troubles, and their responses N 5, 6%

2. Embodied noticings, response received N 13, 16%

Self-initiations of repair 1. Embodied 1A noticings, no response 1B N 54, 68%

Category

Bodily and/or verbal

Verbal

Bodily

Bodily

– (Not attempted) – (Attempted but not received) Bodily

Response type

Exclusive: stops singing when getting notified of the mistake

Exclusive: singing stops/no singing

Exclusive: stops singing upon making the mistake (or singing doesn’t start) Exclusive: stops singing to ask the question

Parallel: continues singing throughout

Exclusive: singing stops/no singing Exclusive: singing stops/no singing

Order of multiactivity for the mistake-maker

Parallel (when bodily), exclusive (when verbal)

Parallel: continues singing throughout Parallel: continues singing throughout Parallel: continues singing throughout Exclusive: singing stops/no singing

-

-

Order of multiactivity for the other(s)

Table 4.1 Noticings of mistakes in choir rehearsals and the orders of multiactivity involved

Example 3

Example 1

Example no. in the article

Total 79, 100%

7, 9%

2, 3%

3, 4%

Example 5, Example 6

–

Example 5

8 (5: Example 2 shared problem), 10% 5, Example 3, 6% Example 4

50, 63% 4, 5%

N, %

4 Embodied Noticings as Repair Initiations

111

112

A. Vatanen

response relevant or not is related to the way it is designed (cf. Keevallik 2018 on self-talk). Let us first consider a case where the activity is entirely private. In this and similar cases, the mistake is noticed visibly (mostly with a facial expression) and it is thus public, yet no attempt to involve others is made. (Whether these displays are somehow targeted at the conductor, showing the favourable ability to recognise one’s mistakes, remains unresolved with the current research methods.) With the visible noticing, the singer provides an “embodied assessment” of his/her own activity—the singing. In the following case, Matti (circled in the images in Fig. 4.4) makes a mistake while singing the framed tenor in the piece represented in the music in Fig. 4.5.1 Example 1 [Category 1A, 19.2._2:12:10]

Fig. 4.4 Images of Example 1. In images where the singer sings, there is a speech bubble with a note; no such bubble means no singing at that moment

1

The four rows are, from top to down, for sopranos, altos, tenors, and basses. All these rows are sung simultaneously, proceeding from left to right. The parts that are written between each of the vertical lines (i.e., the bar lines) last equally long (e.g., in Fig. 4.5, there are three parts that each last equally long). The examples presented here vary in the combination of voices (sopranos, altos, tenors, basses); there may be up to eight different voices sung simultaneously.

4 Embodied Noticings as Repair Initiations

113

Fig. 4.5 Music of Example 1. The moments when the still images are taken have been marked with numbers

Matti first sings fluently together with the others (Img. 1). Then, he makes a mistake (concluding from a comparison of his mouth movements and the musical score, he utters the word uns in a place where it does not appear in the musical score) and visibly notices it with his bodily activities: he raises his eyebrows (Img. 2), stops singing and starts smiling (Img. 3), and temporarily covers his mouth with his left hand (Img. 4–5). Then, he joins the choir again, still smiling (Img. 6). All this occurs during less than three bars in the music (see Fig. 4.5), and the mistake-maker quickly re-joins the shared activity of singing the current piece. Matti organises the two activities here in an exclusive order, as he temporarily abandons singing just after making the mistake, upon producing the embodied noticing. Other participants display no signs of having noticed Matti’s mistake (even though his neighbours may of course have heard it), and Matti makes no attempt to pursue a response from his neighbours, for example by gazing at them. This type of noticings of mistakes resembles the so-called response cries analysed by Goffman (1978). Without visibly orienting to anyone else, the participant publicly shows that he/she has recognised having done something unexpected: in Goffman’s cases, for example stumbling on a crowded street and uttering oops; here, making a hearable mistake in singing and frowning or smiling at that. The visible displays of having made a mistake show that mistakes are oriented to as accountable behaviour in this choir, and that the singers orient to the norm of singing according to the musical score (and the conductor’s conducting). Singing in an advanced choir, such as the one examined here, requires a considerable amount of musical skill and expertise. Producing a noticing of a mistake

114

A. Vatanen

can serve as one type of demonstration of such expertise. Furthermore, being aware of one’s mistakes is arguably the first step towards correcting them, which enables acting according to this norm—joint singing as a collective action.

4.4.1.2 Embodied Noticings That Receive a Response In the collection, there are thirteen (16%) cases where an embodied noticing produced by the mistake-maker is responded to by another participant, likewise by embodied means. In eight cases (10%), the mistake-maker continues singing throughout the episode, and thus, noticing the mistake occurs in parallel with the activity of singing; in five cases out of these eight, the problem/mistake seems to be shared, as both participants produce an embodied noticing, sometimes even at the same time. Furthermore, there are five cases (6%) where the singer stops singing upon making the mistake and visibly noticing it (or the mistake may be that he/she did not start singing together with the others at the right time), and thus, the activities progress in a mutually exclusive fashion. In all these cases, the co-participant’s response to the noticing occurs simultaneously with his/her singing and thus, his/her two activities progress in parallel. Let us first examine a case where Julia and Tuuli (circled in Img. 1, Fig. 4.6) both produce an embodied noticing of a trouble. The part the choir sings at that moment is shown in Fig. 4.7 and involves eight voices altogether. The focal singers (are supposed to) sing the framed lower alto and together use sheet music that Julia holds in her hands. Example 2 [Category 2A, 12.2._1:11:45] At first, both singers’ singing progresses smoothly (Img. 1). Then, Julia visibly notices something problematic: she raises eyebrows and moves head slightly backwards (Img. 2), while both she and her neighbour Tuuli continue singing. Directly after this, Tuuli changes the syllable, singing the syllable dis together with the higher alto (circled in Fig. 4.7)—which, in comparison with the musical score (Fig. 4.7), she does too late, as she should have sung that syllable already prior to that moment (i.e., she is

4 Embodied Noticings as Repair Initiations

115

Fig. 4.6 Images of Example 22

not temporally aligned with the other lower altos); Julia, on the contrary, sings the words as they are written in the score. When they arrive at the syllable mei, where Image 3 is taken, Julia frowns, still singing, and Tuuli starts to smile, waving her left hand below her face (Img. 3). Right after that, Tuuli abandons singing, while Julia continues until the end of the current verse (Img. 4), still frowning. When the lower sopranos start the next verse with the syllable uu (underlined in Fig. 4.7), Julia still has her eyebrows furrowed and Tuuli keeps her index finger on her chin (Img. 5). Then, Julia releases her frown (Img. 6). Here, based on the singers’ bodily behaviour, Julia was the first to notice something unexpected, displaying it with facial expressions—quite likely her neighbour Tuuli singing the previous syllable (yh) for too long. Soon after, Tuuli also visibly orients to something having gone wrong. She abandons singing, whereas Julia continues to progress both activities in parallel. Both thus produce an embodied noticing of something problematic—but it is impossible to say, with the current data, whether the oriented-to trouble was caused by Tuuli only, or whether Julia had some problems as well (e.g., singing a 2

In the images where arrows are used, blue arrows denote movements of body parts (here, hand). Yellow arrows are used for the direction of gaze (not used in this example).

116

A. Vatanen

Fig. 4.7 Music of Example 2

wrong voice, which is not accessible in the data in the way that singing wrong words most often is).

4 Embodied Noticings as Repair Initiations

117

In the next extract, there are two instances of neighbour interaction: one unsuccessful, embodied pursuit of a response, and another that succeeds in mobilising a response from another neighbour. The mistakemaker here is Pekka, the bass circled in Image 1 (Fig. 4.8), and the choir is (supposed to be) singing the music as shown in Fig. 4.9. Example 3 [Categories 1B and 2B, 19.2._2:12:30].3

Fig. 4.8 Images of Example 3

Fig. 4.9 Music of Example 3

3 In this example (and the following ones), I apply the Jeffersonian transcription system for talk (Jefferson 2004) and singing, and the Mondada system for embodied conduct (Mondada 2019). Additional symbols used here are “♫” for singing and “♪” for speech with singing-like rhythm

118

A. Vatanen

Pekka first sings fluently with the other basses (line 4, Img. 1). After the first bar, he makes the mistake of starting too early (line 5), after which he stops singing, grimaces (Img. 2) and ducks (line 6). He then turns to Jari, the neighbour on his left (line 6, Img. 3), but Jari looks at his own music, singing, and does not pay observable attention to Pekka—thus prioritising the activity of singing. Pekka withdraws his gaze from Jari back to his own music and joins the choir again (line 7, Img. 4). This occurrence is thus an instance of an embodied noticing that pursues a response from a neighbour but does not receive it. After Pekka joins the choir again, he smilingly turns his gaze to his other neighbour Timo, who sits on his right (line 9, Img. 5). Timo and/or melody. To be also noted here is that in this extract the choir (CHO) sings a doublechoir piece. Only the singing of choir 1 (CH1) is transcribed here, which is the choir that Pekka belongs to. “CH1B” (e.g., line 6) refers to the basses of choir 1. CON refers to the conductor.

4 Embodied Noticings as Repair Initiations

119

notices Pekka’s movement and, continuing singing and gazing at his music, begins to smile and raises his left hand and a finger (Img. 5). He thus acknowledges Pekka’s mistake (which he quite likely heard) and provides a rather minimal response—the raising (or even a slight wagging) of the finger. He does this while keeping his involvement in the joint singing activity intact and shows this orientation also with his body posture, not twisting his body towards the co-participant (cf. Kamunen 2019a; Sahlström 1999). He thus progresses the two activities in parallel. Having received Timo’s response, Pekka smilingly continues singing, gazing at his music (Img. 6). For Pekka, the activities here (singing and interacting with the neighbour Timo) are organised in parallel, whereas his first, unsuccessful attempt to involve a neighbour (Jari) was produced after he had abandoned singing—in other words, those activities were produced in a mutually exclusive manner. The next case is similar to the previous one in that it involves a mistake-maker (Eevi, circled in Img. 1, Fig. 4.10) who abandons singing upon making a mistake and bodily notices the mistake by raising eyebrows and shaking head (Img. 2). She then, as Pekka in the previous case, turns to her neighbour Iiris (Img. 3) and briefly grabs her own nose (Img. 4). Iiris, while continuing singing, smiles slightly in response (Img. 5), and after the choir has completed singing the piece (see line 1 and the music in Fig. 4.11; the focal singers’ alto voice is framed), turns to the mistake-maker Eevi, still smiling (Img. 6). What is different in this case is that Eevi verbally comments on her mistake after the choir has finished the singing (line 4, Img. 7).

120

A. Vatanen

Example 4 [Category 2B, 19.2._2:09:20]

Fig. 4.10

Images of Example 4

Fig. 4.11

Music of Example 4

4 Embodied Noticings as Repair Initiations

121

After the choir has finished singing the piece (Fig. 4.11 represents the end of the piece), Eevi laughingly verbalises her mistake, thereby also accounting for it: ( ) meni väärin ( ) “got it wrong” (line 4, Img. 7; unfortunately, most of her turn is inaudible). Both her neighbours join the laughter and together smile at the difficult spot in the music (line 5, Img. 7–8). Also, the conductor evaluates the piece the choir has just sung, acknowledging the singers’ difficulties he heard—korkeempi sävellaji on vaikeempi “the higher key is more difficult” (slightly before the current part in this piece, there is a similar part with a lower key)—and instructs the choir to sing it again (line 8). Thus, in this example, the mistake-maker first produced an embodied noticing of her mistake, initiating repair on it, and after the choir finished singing, she also verbally commented on her mistake. Producing the repair solution—i.e., correct singing of the part where the mistake occurred—does not occur during this fragment but only later, as the choir continues to rehearse this piece. In the next section, we will examine instances where a fellow singer notices the mistake-maker’s mistake in an embodied manner, which leads to various action trajectories among the collective.

122

A. Vatanen

4.4.2 Embodied Noticings as Other-Initiations of Repair The collection includes seven cases (9%), where it is not the mistakemaker but another singer who first attends to a mistake that has occurred. These cases are thus instances of other-initiations of repair, which have been claimed to be dispreferred in contrast to self-initiations (Schegloff et al. 1977; cf. Kurhila 2001; Haakana and Kurhila 2009). A rather similar order seems to be observable in the current data with regard to the mistakes made in singing. I will now analyse two such cases. In the first one (Example 5), there is one neighbour who produces an embodied noticing of another singer’s mistake but does not stop singing, and no others pay attention to them. In the second case (Example 6), on the contrary, several fellow singers attend to a mistake made by two singers. In this case, the conductor eventually attends to the event as well and ends up stopping the whole choir and advising the singers. Let us first examine the simpler case. Ella (circled in Img. 1, Fig. 4.12) and Viola (circled in Img. 2) sing the soprano (framed in Fig. 4.13; SOP in the transcript), first looking at the conductor (Img. 1). After a short pause in the music (line 2), the conductor gives the beginning note by singing the first syllable py and gives the rhythm with yks, kaks “one, two” (line 3), and the choir starts to sing (line 4 onwards). Example 5 [Categories 4 and 3A, 19.2._56:40]

Fig. 4.12

Images of Example 5

4 Embodied Noticings as Repair Initiations

Fig. 4.13

Music of Example 5

123

124

A. Vatanen

The altos, tenors and basses start singing first (line 4), and the sopranos start in the next bar (line 5, see Fig. 4.13). The music given to the choir has been revised by hand as to the sopranos’ part, as visible in Fig. 4.13. Ella makes the mistake of singing along to the original notes, changing to the second syllable hä too early (line 6). Viola arguably hears this and turns abruptly to Ella (Img. 2), beginning to smile, whereupon Ella stops singing (Img. 3). Viola’s bodily conduct, I argue, initiates repair on Ella’s singing concerning the part that had gone wrong. Continuing singing,

4 Embodied Noticings as Repair Initiations

125

Viola turns back to her music, and they both smile (Img. 4). Ella then briefly glances at Viola (line 7), and shortly after, Viola turns her gaze from her music to the conductor (line 9). Then, Ella turns to Viola again (line 9) and produces a verbal turn, apparently a question, gazing and pointing at her music (line 10, Img. 5). Viola turns to look at Ella’s music (line 10, Img. 6) and shakes her head in response, continuing singing (line 11, Img. 7). She thus progresses the two activities—singing and responding to Ella—in parallel (as she did when noticing Ella’s mistake on line 6). Ella says something short in response and makes a note in her music with her pencil (line 12)— quite likely with the aim of correcting her singing when they next sing the piece. The choir completes singing the piece, and both Ella and Viola look at their music (line 11, Img. 8). The conductor provides a concluding demonstrative adverb näin “like that” (line 13) and a comment semmonen oli tämä “such was this” (line 15), and announces the next song (line 15), thus not paying any attention to Ella and Viola. Ella organises the activities of singing and orienting to the mistake/ trouble in a mutually exclusive fashion, as she stops singing upon making the mistake and getting a look (cf. Kidwell 2005) from Viola. She does not resume singing during the piece, but instead verbally addresses Viola, most likely concerning the part where she made the mistake. Talking and singing are, necessarily, mutually exclusive activities, as they both exploit the vocal tract. This instance is one of the few cases in the collection where a mistake is oriented to by verbal means while the rest of the choir still sings. A neighbour may respond to the verbal turns bodily, for example with a nod or a head shake (like here). The collection also involves two cases where the response is verbal as well, and thus, both singers organise their activities—singing and attending to a mistake/trouble—as mutually exclusive, temporarily favouring the neighbour interaction over the joint singing. In the example above, Viola—the recipient of the verbal turn—progresses the two activities in parallel: singing and noticing Ella’s mistake in a bodily manner (with gaze) and singing and responding bodily to Ella’s question (with a head shake). This is possible due to the affordances of the activities: singing (from memory) can be combined with bodily interaction.

126

A. Vatanen

In the next and final example, the choir has just completed singing verse 1 (line 1) and the conductor utters eteenpäin “moving on” (line 2). The second verse is on the same page as the first, and the choir starts to sing it (line 4, see Fig. 4.15)—except for the bass Timo (circled in Img. 1, Fig. 4.14), who turns the page (line 3) and starts to sing the third verse (line 5, see Fig. 4.15). Timo does not notice his mistake until his neighbours point it out to him several bars later (line 7 onwards). Example 6 [Category 4, 12.2._1:01:20]

Fig. 4.14

Images of Example 6

4 Embodied Noticings as Repair Initiations

Fig. 4.15

Music of Example 6

127

128

A. Vatanen

Timo sings the wrong verse for five bars (lines 5, 7). His mistake is well hearable to all since at this point all others sing the melody and only he (and his neighbour who sings from the same sheet as him) sings something different (see the music in Fig. 4.15). During the second bar, the conductor briefly gazes at the sopranos’/basses’ direction (line 5), visibly noticing the hearable mistake, but this goes unnoticed by the focal singers. During the fourth bar in the verse, at least three other basses (sitting on Timo’s left) turn to gaze at Timo (lines 4–5, Img. 2), and shortly after, also a neighbour on his right (Img. 3), all of them

4 Embodied Noticings as Repair Initiations

129

at the same time continuing singing and thus progressing the activities in parallel. In the beginning of the fifth bar, Timo stops singing (line 7). At the same time, the conductor freezes his hands and briefly gazes towards the basses (line 7), simultaneously with two of Timo’s neighbours gazing at the conductor (Img. 3). Immediately after the conductor stops conducting (Img. 4), most of the singers gradually stop singing, while at least one singer continues until the end of the row of notes (line 6). During the bar in which Timo stops singing and the bar that follows, several of Timo’s neighbours point his mistake out to him in various ways, thus initiating repair. When Timo is about to stop singing, Eeli (also a bass), who sits on Timo’s right, turns to him (Img. 3). Timo turns to Eeli (Img. 5) at the same time as Timo’s neighbour on his left (Pekka) points to Timo’s music with his hand (Img. 5). Having stopped singing, Eeli verbally advises Timo, who looks at his own music, leaning forward (Img. 6). Then, he straightens up (Img. 7) and the conductor says kaikki kakkossäk- eikö vain “everyone verse two, right” (line 9, Img. 8), not directly addressing Timo but “everyone”. He then walks from his music stand to the piano (line 10). At this point, Sakari, a tenor in the front row, comments on the episode with a smiling face: sivu ei käänny vaikka niin äkkiä luulis “the page doesn’t turn even if one might think so” (line 11, Img. 9), twisting his upper body in Timo’s direction though not directly looking at him. In this turn that orients to the mistake, he uses the Finnish zero person (ø luulis, “ø might think”), thus constructing the experience as one with which anyone can identify (Laitinen 1995, 2006; Suomalainen and Varjo 2020). At the same time, Pekka pats Timo on the back (Img. 9), smiling. Timo smiles widely (Img. 10), and many of the fellow singers smile and look in Timo’s direction (Img. 11). The conductor guides the choir to now begin at bertta (letter B in the music), gives the chord by singing laa laa laa and the rhythm with ja yks kaks ja “and one two and” (line 17), and the choir begins to sing the third verse (line 18). This instance of orienting to a mistake forms the most elaborate case in my collection, and this is the only case where the singers’ noticing of a mistake is followed by the conductor stopping the whole choir and giving specific instructions on how to continue. Given that the

130

A. Vatanen

vast majority of the mistakes and noticings of them do not receive any response, and given that in most of the cases that do receive a response, the response is produced by a fellow singer, one might wonder why the conductor stopped the choir this time. This case is rather explicit in revealing that the participants orient to a preference for (or a norm of ) noticing one’s own mistakes quickly and repairing them quickly. In the previous cases, we have seen that when making a mistake and noticing it, the singers immediately stop their incorrect performance or correct their singing on the fly. On the contrary, when Timo continues singing wrong for as long as five bars, his neighbours first gaze at him (Img. 2, 3), which he does not notice but just looks at the music he holds in his hands (Img. 2–4). Some of his neighbours notice that the conductor stops conducting (Img. 3), and two neighbours make an attempt at advising him: Pekka by pointing, which Timo does not see (Img. 5), and Eeli by verbally guiding, which Timo listens to (Img. 4–6). This quite likely would have been enough to get Timo literally on the same page with others. However, having already stopped the choir after seeing several singers orient to Timo, the conductor verbally accounts for the stoppage by advising about the right verse everyone should be singing. Once the common participation framework has been verbally opened by the conductor, Sakari also joins in with his friendly remark on Timo’s mistake, uttered with a smiling voice. This example thus shows that also in the context of choir rehearsals, at least in the current data, the participants display a preference for self-initiated repair over other-initiated repair (cf. Schegloff et al. 1977; Kurhila 2001; Haakana and Kurhila 2009).

4.5

Summary of Analysis

4.5.1 Multiactivity in Choir Singing The analysis above focused on embodied noticings of a mistake made in singing yet acknowledges that verbal noticings also occur. Some of the embodied noticings by the mistake-maker—that is, self-initiations of repair—were produced simultaneously with the singing activity, whereby

4 Embodied Noticings as Repair Initiations

131

the two activities progressed in parallel (Example 2). In most cases, however, the singer stopped singing upon producing the embodied noticing of their own mistake, thus organising the two activities in an exclusive order (Examples 1, 3, 4). In the current collection, embodied (acknowledging) responses to the embodied noticings were all produced in a parallel order, while continuing singing (Examples 2–4). In some cases, a verbal response was produced later, when the joint singing had been stopped/finished (not shown here). In a few cases, a response was pursued but not received, in which case the fellow singer stayed involved with the joint singing only (Example 3). One example (Example 5) also included a case where the mistake-maker addressed a verbal turn to her neighbour, in which case the two relevant activities— singing and speaking—were, necessarily, mutually exclusive. However, in this case as well, the neighbour’s bodily response was produced in parallel with singing. The analysed instances of noticings of mistakes by fellow singers—that is, other-initiations of repair (Examples 5 and 6)— involved both bodily and verbal actions, of which the bodily ones were mostly produced in parallel with singing, whereas the verbal ones were necessarily produced having (momentarily) abandoned singing. Overall, the whole collection highlights the affordances of different activities and the ways in which singing and social interaction—either embodied or verbal—can be progressed and combined. With various practices, the participants skilfully balance their participation in the multi-layered setting, managing their orientation to the simultaneously relevant, sometimes momentarily conflicting activities (or participation frameworks). In the analysed cases, practices of multiactivity are used for simultaneous involvement in two activity trajectories: (1) singing with the choir, conducted by the conductor, and (2) noticing mistakes made by oneself or a fellow singer, and responding to the noticings. Crucial senses for attending to the two activities include (musical) hearing and (peripheral) vision, and the participants employ skilled coordination of multiple embodied and vocal resources in this work (for the role of another sense, taste, in organising simultaneous activities in specific settings, see Mondada, this volume). As progressing one activity may inhibit engaging in another, the analysis also points to the limits of attention and participation (see also Hofstetter and Robles, this volume).

132

A. Vatanen

4.5.2 Repair Organisation in the Collective Activity of Choir Singing In this study, the embodied noticings were analysed as repair initiations. More specifically, the embodied actions were analysed as open repair initiations, as they—similarly to the verbal turn “what”, for example— do not point to a specific trouble source in the prior. The timing of the embodied repair initiation tells something about the position of the trouble source: they are always produced right after the trouble source. However, in the collective activity of choir singing, it is not meaningful to determine the position of the repair initiation as it is determined in talk-in-interaction (e.g., “during the trouble source turn”, “in the next turn”, etc.; see Schegloff et al. 1977), as choir singing is not characterised by turn-taking but is a simultaneous, collective action (cf. Lerner 1993; on the questions of embodiment in repair, see Lilja 2010). In the choir data, the repair solution—singing the problematic part correctly—does not occur right after the repair initiation, but only when the problematic part is rehearsed the next time, and the singer sings the same passage again. In this case, it is a self-correction/repair that is embedded (Jefferson 1987; Kurhila 2001) in the course of the rehearsal. Furthermore, most often when the singers pay attention to another singer’s mistake by visibly noticing it and thus by initiating repair on it, they do not themselves provide the correction (the repair solution); doing so at a moment when the collective singing has already moved on would rather disturb the joint activity. An essential feature of repair organisation is that it suspends and delays the ongoing talk. In the activity of a collective, this is not as straightforward. Most often, the producer of the trouble source (the mistake-maker) stops singing upon making the mistake and producing the noticing/repair initiation—but the collective almost always continues undisturbed. There was only one example in the collection where the conductor stopped the whole choir because of one singer’s mistake (Example 6). In this case, however, several singers oriented to the mistake (and to the mistake-maker), so its occurrence had already distracted the joint singing to a large extent.

4 Embodied Noticings as Repair Initiations

133

Concerning repair in talk-in-interaction (e.g., Schegloff et al. 1977), it has been shown that it is more preferred to initiate repair on one’s own talk rather than on another’s. The current data suggest that the choir singers also orient to a preference for initiating repair on one’s own actions and doing it quickly. In the examined setting, initiating repair on someone else’s singing is a subtle matter. Pointing to deviations from norms and expectations is related to the participants’ expertise, and identity and role as choir singers. In the choir setting, the conductor is oriented to as the one with the highest deontic authority, being entitled to determine what is done when—and, for example, to guide the singers in their singing. The singers’ role, on the other hand, is mainly seen to involve singing following the musical score and the conductor’s conducting and guidance. The singers’ orientations to repairables in the singing activity is a sensitive and complex matter. The participants show delicate balancing between what they publicly orient to and what they let pass (see also Kamunen 2019b). Avoiding overt orientation to others’ “mistakes” and rather orienting to progressivity of the joint activity has also been observed in other situations and settings, for example non-pedagogic interactions between “native” and “non-native” speakers (see Kurhila 2001, 2006), conversations with a person with dementia (Pilnick et al. 2021), and everyday talk (Jefferson 2007), as well as when encountering a technical trouble in video-mediated business meetings (Oittinen 2020). In the current data, the singers similarly display orientation to the progressivity of the joint activity of singing and thus to furthering the common agenda of the rehearsal (see Hofstetter and Robles, this volume, on orientation to progressivity and multiactivity during gameplay).

134

4.6

A. Vatanen

Discussion and Conclusion: On Noticings, Response Relevance and Orientation to Norms and Progressivity

The current study started from the observation that choir singers sometimes pick up on some unexpected features of their singing, called here mistakes, by producing an embodied noticing. The noticing is most often accomplished with a facial expression and/or a body (part) movement, and it is positioned right after its source (the mistake), while the singing activity still continues, at least by the rest of the choir. The singers thus rely on specific bodily conduct, accomplished in a specific position, for performing the social action of noticing. These noticings display an orientation to and enforce a norm of singing “correctly” (according to the musical score) and show that the singers orient to deviations from this norm as accountable actions (cf. Goffman 1978). In the collection, the noticings target information that is audible: deviations from what is expected concerning timing, words (lyrics) and pitch. In other words, in the context of choir rehearsals, these matters are oriented to as repairables by the singer-participants (cf. types of repairables in previous work, e.g., Mortensen 2016).4 The analysed instances take place in an early phase in the process of the choir preparing specific pieces of music to be performed. In a rehearsal where a piece is practised for the first time, the emphasis is more on learning the right voices (correct words with correct pitch, rhythm and timing), whereas at the end of the process, the focus is on polishing interpretative nuances for example. The current collection indicates that embodied noticings occur especially when the choir is practising the basics of the pieces to be performed; during the later recorded rehearsals, these kinds of noticings did not really occur. This suggests that in the collective learning process that strives towards a better joint outcome, deviations from the basic features in the music are visibly oriented to in 4 Repair initiations produced by the conductor are out of the scope of the present paper; the data suggest that the array of repairables that the conductor orients to is wider (e.g., intensity, being in tune, interpretative nuances, etc.).

4 Embodied Noticings as Repair Initiations

135

the early phase, and these deviations—or mistakes—are then corrected in (or for) the next time. Finally, this study lends itself to reflections on the degree to which noticings and (embodied) repair initiations exhibit response relevance. What do these turns expect as their next actions, and how strong is the response pressure? As pointed out in the analysis (see also Table 4.1), most (68%) of the initiating turns do not receive a response. In 63% of all cases, the participant did not even attempt to involve anyone else in the sequence; the bodily action, in that sense, seemed private (or selforiented). In a few cases (5%), a response was pursued but not received. In the rest of the cases of self-initiations (22% of all cases), as well as in most of the other-initiations (9% of all cases), there was a response. This somewhat mixed picture seems to suggest that embodied noticings/repair initiations do not straightforwardly expect a response. Instead, the turns can be designed both as self-oriented actions and as actions addressed to a co-participant, requiring a response. It must be noted, however, that the nature of the current collective activity plays a role here. In the context of choir rehearsals, the singers orient to the progressivity of the joint action (see Hofstetter and Robles, this volume, on board games). Thus, they do not always engage a coparticipant in the action of noticing their own mistakes (see Example 1), and they may also ignore a co-participant who pursues a response from them (see Example 3). In other words, the setting and the ongoing activity can make a difference in how the participants view and use social actions in general. If a self-initiation (noticing of one’s own mistake) pursues a response from a co-participant, the relevant response is a simple acknowledgement (see, e.g., Example 3). From the point of view of the joint activity, the relevant next action after an embodied noticing/ repair initiation may also be to produce a correction at the next possible slot (i.e., the next time the piece is sung). Overall, the sequences initiated by embodied noticings indicate the multiple layers of the current interactional setting: they display the participants’ musical expertise as well as their responsibility for and orientation to the norms of collective choir singing—singing correctly according to the musical score, as well as privileging the continuity and progressivity of the collective action.

136

A. Vatanen

Acknowledgements My warmest thanks to the recorded choir for enabling this research by letting me record the rehearsals for research purposes. I also wish to thank the editor team, members of the COACT research community at the University of Oulu, as well as the anonymous reviewers for most helpful comments on earlier drafts of the article, which have greatly aided me in improving the analysis.

References Balantani, Angeliki. 2022. “Non-lexical vocalisations + “so_was” as a multimodal package in establishing joint decisions in music rehearsals.” Language & Communication 87: 147–160. Dingemanse, Mark, Francisco Torreira, and Nick J. Enfield. 2013. “Is ‘huh?’ A universal word? Conversational infrastructure and the convergent evolution of linguistic items.” PLoS ONE 8 (11): e78273. Drew, Paul. 1997. “‘Open’ class repair initiators in response to sequential sources of troubles in conversation.” Journal of Pragmatics 28: 69–101. Emerson, Kathryn. 2018. The social organisation of the choir rehearsal: How interaction between conductor and choir is used to shape the choir’s singing. PhD thesis, University of Sheffield. Emerson, Kathryn, Victoria Williamson, and Ray Wilkinson. 2019. “Once more, with feeling: Conductors’ use of assessments and directives to provide feedback in choir rehearsals.” Musicae Scientiae 23 (3): 362–382. Goffman, Erving. 1978. “Response cries.” Language 54 (4): 787–815. Goffman, Erving. 1981. Forms of talk. Philadelphia, PA: University of Pennsylvania Press. Goodwin, Marjorie H. 1997. “Byplay: Negotiating evaluation in storytelling.” In Towards a social science of language, ed. by Gregory R. Guy, Crawford Feagin, Deborah Schiffrin, and John Baugh, 77–102. Amsterdam: John Benjamins. Goodwin, Charles, and Marjorie H. Goodwin. 2004. “Participation.” In A companion to linguistic anthropology, ed. by Alessandro Duranti, 222–243. Oxford: Basil Blackwell. Haakana, Markku. 2011. “Mitä ja muut avoimet korjausaloitteet [Mitä (‘what’) and other open class repair initiators in Finnish interactions].” Virittäjä (1): 36–67.

4 Embodied Noticings as Repair Initiations

137

Haakana, Markku, and Salla Kurhila. 2009. “Other-correction in everyday interaction: some comparative aspects.” In Talk in interaction: Comparative dimensions, ed. by Markku Haakana, Minna Laakso, and Jan Lindström, 152–179. Helsinki: Suomalaisen Kirjallisuuden Seura. Haddington, Pentti, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile. 2014. “Towards multiactivity as a social and interactional phenomenon.” In Multiactivity in social interaction: Beyond multitasking, ed. by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 3–32. Amsterdam: John Benjamins. Helisten, Marika. 2019. “Disjunctively positioned problem-noticings in managing multiactivity.” Research on Language and Social Interaction 52 (4): 318–342. Jefferson, Gail. 1987. “On exposed and embedded correction in conversation.” In Talk and social organization, ed. by Graham Button, and J.R.E. Lee, 86– 100. Clevedon: Multilingual Matters. Jefferson, Gail. 2004. “Glossary of transcript symbols with an introduction.” In Conversation analysis: Studies from the first generation, ed. by Gene Lerner, 13–31. Amsterdam: John Benjamins. Jefferson, Gail. 2007. “Preliminary notes on abdicated other-correction.” Journal of Pragmatics 39: 445–461. Kamunen, Antti. 2019a. “How to disengage: Suspension, body torque, and repair.” Research on Language and Social Interaction 52 (4): 406–426. Kamunen, Antti. 2019b. Busy embodiments: The hierarchisation of activities in multiactivity situations. Doctoral dissertation, University of Oulu. Keevallik, Leelo. 2018. Sequence initiation or self-talk? Commenting on the surroundings while mucking out a sheep stable. Research on Language and Social Interaction 51 (3): 313–328. Keisanen, Tiina. 2012. ““Uh-oh, we were going there”: Environmentally occasioned noticings of trouble in in-car interaction.” Semiotica 191 (1–4): 197–222. Kidwell, Mardi. 2005. “Gaze as social control: How very young children differentiate ‘the look’ from a ‘mere look’ by their adult caregivers.” Research on Language and Social Interaction 38 (4): 417–449. Kidwell, Mardi. 2009. “Gaze shift as an interactional resource for very young children.” Discourse Processes 46: 145–160. Koole, Tom. 2007. “Parallel activities in the classroom.” Language and Education 21 (6): 487–501. Kurhila, Salla. 2001. “Correction in talk between native and non-native speaker.” Journal of Pragmatics 33: 1083–1110.

138

A. Vatanen

Kurhila, Salla. 2006. Second Language Interaction. Amsterdam: John Benjamins. Kääntä, Leila. 2014. “From noticing to initiating correction: Students’ epistemic displays in instructional interaction.” Journal of Pragmatics 66: 86–105. Laanesoo, Kirsi, and Leelo Keevallik. 2017. “Noticing breaches with nonpolar interrogatives: Estonian Kes (“who”) ascribing responsibility for problematic conduct.” Research on Language and Social Interaction 50 (3): 286–306. Laitinen, Lea. 1995. “Nollapersoona [Zero person].” Virittäjä 99 (3): 337–358. Laitinen, Lea. 2006. “Zero person in Finnish. A grammatical resource for construing human reference.” In Grammar from the human perspective: Case, space and person in Finnish, ed. by Marja-Liisa Helasvuo, and Lyle Campbell, 209–231. Amsterdam: John Benjamins. Lave, Jean, and Étienne Wenger. 1991. Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Lehtimaja, Inkeri. 2012. Puheen suuntia luokkahuoneessa. Oppilaat osallistujina yläkoulun suomi toisena kielenä -tunnilla [Directions of talk in the classroom: student participation during Finnish as a second language lessons in secondary school]. Helsinki: University of Helsinki. Lemke, Jay L. 1990. Talking science: Language, learning, and values. Norwood, NJ: Ablex Publishing. Lerner, Gene H. 1993. “Collectivities in action: Establishing the relevance of conjoined participation in conversation.” Text 13 (3): 213–245. Lilja, Niina. 2010. Ongelmista oppimiseen. Toisen aloittamat korjausjaksot kakkoskielisessä keskustelussa [Other-initiated repair sequences in Finnish second language interactions]. Jyväskylä Studies in Humanities 146. Jyväskylä: University of Jyväskylä. Markee, Numa. 2005. “The organization of off-task talk in second language classrooms.” In Applying conversation analysis, ed. by Keith Richards, and Paul Seedhouse, 197–213. Basingstoke: Palgrave Macmillan. Martin, Cathrin, and Fritjof Sahlström. 2010. “Learning as longitudinal interactional change: From other-repair to self -repair in physiotherapy treatment.” Discourse Processes 47 (8): 668–697. Merlino, Sara. 2014. “Singing in “another” language: How pronunciation matters in the organisation of choral rehearsals.” Social Semiotics 24 (4): 420–445. Mondada, Lorenza. 2011. “The organization of concurrent courses of action in surgical demonstrations.” In Embodied interaction, ed. by Charles Goodwin, Curtis LeBaron, and Jürgen Streeck, 207–226. Cambridge: Cambridge University Press.

4 Embodied Noticings as Repair Initiations

139

Mondada, Lorenza. 2012. “Talking and driving: Multi-activity in the car.” Semiotica 191 (1–4): 223–256. Mondada, Lorenza. 2014. “The temporal orders of multiactivity.” In Multiactivity in social interaction: Beyond multitasking, ed. by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 33–75. Amsterdam: John Benjamins. Mondada, Lorenza. 2019. Conventions for multimodal transcription. Retrieved from: https://www.lorenzamondada.net/multimodal-transcrip tion. Mortensen, Kristian. 2016. “The body as a resource for other-initiation of repair: Cupping the hand behind the ear.” Research on Language and Social Interaction 49 (1): 34–57. Nishida, Hiroko, and Daisuke Yokomori. 2018. “Stopping/restarting play during string quartet rehearsals: An ethnographic approach to performance analysis.” Studies in Languages and Cultures 40: 69–84. Nishizaka, Aug. 2006. “What to learn: The embodied structure of the environment.” Research on Language and Social Interaction 39 (2): 119–154. Oittinen, Tuire. 2020. “Noticing-prefaced recoveries of the interactional space in a video-mediated business meeting.” Social Interaction. Video-Based Studies of Human Sociality 3 (3): 1–24. Oloff, Florence. 2018. ““Sorry?”/“Como?”/“Was?”—Open class and embodied repair initiators in international workplace interactions.” Journal of Pragmatics 126: 29–51. Parton, Katharine. 2014. “Epistemic stance in orchestral interaction.” Social Semiotics 24 (4): 402–419. Pilnick, Alison, Rebecca O’Brien, Suzanne Beeke, Sarah Goldberg, and Rowan Harwood. 2021. “Avoiding repair, maintaining face: Responding to hard-tointerpret talk from people living with dementia in the acute hospital.” Social Science & Medicine 282: 114156. Rauniomaa, Mirka, Esko Lehtonen, and Heikki Summala. 2018. “Noticings with instructional implications in post-licence driver training.” International Journal of Applied Linguistics 28: 326–346. Reed, Darren, and Beatrice Szczepek Reed. 2014. “The emergence of learnables in music masterclasses.” Social Semiotics 24 (4): 446–467. Sahlström, Fritjof. 1999. Up the hill backwards: On interactional constraints and affordances for equity-constitution in the classrooms of the Swedish comprehensive school . Uppsala: Uppsala University. Schegloff, Emanuel A. 2007. Sequence organization in interaction: A primer in conversation analysis. Cambridge: Cambridge University Press.

140

A. Vatanen

Schegloff, Emanuel A., Gail Jefferson, and Harvey Sacks. 1977. “The preference for self-correction in the organization of repair in conversation.” Language 53 (2): 361–382. Seo, Mi-Suk, and Irene Koshik. 2010. “A conversation analytic study of gestures that engender repair in ESL conversational tutoring.” Journal of Pragmatics 42 (8): 2219–2239. Sidnell, Jack, and Tanya Stivers (eds). 2013. The Handbook of Conversation Analysis. Oxford: Wiley-Blackwell. Siitonen, Pauliina, Mirka Rauniomaa, and Tiina Keisanen. 2021. “Language and the moving body: Directive actions with the Finnish kato “look” in nature-related activities.” Frontiers in Psychology 12: 661–784. Stevanovic, Melisa. 2017. “Managing compliance in violin instruction: The case of the Finnish clitic particles -pA and -pAs in imperatives and hortatives.” In Imperative turns at talk: The design of directives in action, ed. by Marja-Leena Sorjonen, Liisa Raevaara, and Elizabeth Couper-Kuhlen, 357–380. Amsterdam: John Benjamins. Stevanovic, Melisa. 2020. “Mobilizing student compliance: On the directive use of Finnish second-person declaratives and interrogatives during violin instruction.” In Mobilizing others: Grammar and lexis within larger activities, ed. by Carmen Taleghani-Nikazm, Emma Betz, and Peter Golato, 115–145. Amsterdam: John Benjamins. Stevanovic, Melisa, and Anssi Peräkylä. 2012. “Deontic authority in interaction: The right to announce, propose and decide.” Research on Language and Social Interaction 45 (3): 297–321. Stoeckl, Hartmut and Monika Messner. 2021. “Tam pam pam pam and mi– fa–sol: constituting musical instructions through multimodal interaction in orchestra rehearsals.” Multimodal Communication 10 (3): 193–209. Suomalainen, Karita, and Mikael Varjo. 2020. “When personal is interpersonal. Organizing interaction with deictically open personal constructions in Finnish everyday conversation.” Journal of Pragmatics 168: 98–118. Szczepek Reed, Beatrice, Darren Reed, and Elizabeth Haddon. 2013. “NOW or NOT NOW: Coordinating restarts in the pursuits of learnables in vocal masterclasses.” Research on Language and Social Interaction 46 (1): 22–46. Tolins, Jackson. 2013. “Assessment and direction through nonlexical vocalizations in music instruction.” Research on Language and Social Interaction 46 (1): 47–64. Veronesi, Daniela. 2014. “Correction sequences and semiotic resources in ensemble music workshops: the case of Conduction® .” Social Semiotics 24 (4): 468–494.

4 Embodied Noticings as Repair Initiations

141

Weeks, Peter. 1996. “A rehearsal of a Beethoven passage: An analysis of correction talk.” Research on Language and Social Interaction 29 (3): 247–290. Wenger, Étienne. 1998. Communities of practice: Learning, meaning, and identity. Cambridge: Cambridge University Press.

Part II Complexity That Resides in Asymmetries Related to Affordances, Resources and Roles

5 Intersubjective Interaction During the Word Explanation Activity in Social Virtual Reality Heidi Spets

5.1

Introduction

Social virtual reality (VR) is a virtual environment where one can interact with other users in addition to interacting with the environment itself. Such environments have become a new area of interest for studies of social interaction in the wake of the recent emergence of social VR technologies. In general, immersive virtual reality refers to computergenerated environments that can be used with a head-mounted display (HMD). In such virtual environments, a user’s movements are tracked and transferred to a virtual space for providing an immersive experience (Fox et al. 2009; Mills et al. 2022). Immersive VR offers additional possibilities for movement and interaction with the environment compared to desktop VR and games, such as a sense of presence. H. Spets (B) University of Oulu, Oulu, Finland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_5

145

146

H. Spets

Online interaction differs from face-to-face interaction. Socialising online is done based on the affordances of the particular communication technology. Affordances are the perceptions a user has about the features of a particular technology and its potential uses (Hutchby 2001). In VR specifically, depending on the technology, users’ physical bodily actions might translate into the avatar’s movements only partially when using avatar-based VR. Users are creating new forms of interacting and socialising in VR by using the affordances offered by the specific VR environments. For example, one emerging practice in the social VR platform VRChat is that the users use head patting instead of hugging to, among other things, express affection. As the users cannot hug each other’s avatars, they have come to use this alternate way of expression. VR interaction is made complex by such emerging practices as well as the novelty and variation of available interactional resources. In terms of such resources, complexity is visible in how participants talk and act in VR. For example, in VR the users can use resources that they are not used to in face-to-face interaction, such as floating objects. It is difficult to anticipate how social actions are accomplished with VR affordances and how that makes interaction complex in VR. Research is needed to both find such situations and examine how the complexity presents itself. The aim of this chapter is to analyse interaction in social VR through an examination of instances of the 3D Charades word guessing game in the social VR space Rec Room.1 As the participants partake in the word explanation activity, they can face situations where they have asymmetric access to resources (Heath and Luff 1992). This can affect the mutual availability of important resources such as gestures or drawings during the gaming activity. As a result, the participants’ shared understanding of the gaming activity and the environment can be challenged as interaction becomes fragmented (see, e.g. Haddington et al., this volume; Luff et al. 2003). These challenges can affect the participants’ ability to reach the goal of the activity, a correct guess. The analyses of situated social interaction will focus on the questions of how the participants achieve a shared understanding of the explanation, resulting in a correct guess, as well as how they use multimodal (verbal, bodily) and material resources 1

Recroom.com.

5 Intersubjective Interaction in Social VR

147

to do so, and what kind of challenges they face while doing it. The analysis will be conducted using conversation analysis (CA). CA allows for a detailed analysis of the participants’ talk and embodied actions within interaction (e.g. Sidnell and Stivers 2013). Examining video-recorded interactions with these questions as a lens helps illustrate how intersubjectivity can be established, maintained and challenged in VR interaction. Using such recordings, one can also examine how participants’ use of the features of the VR environment affect progressivity, especially in situations where participants have different access to each other’s embodied actions as well as the material environment can differ. Through the examination of interactional challenges, this chapter showcases how users orient to and solve such challenges.

5.2

Achieving and Maintaining Intersubjectivity in Social Virtual Reality

Social VR environments are designed for interaction between users more than purely for attending an event, such as a virtual concert, or engaging in a gaming activity (Blackwell et al. 2019; Maloney and Freeman 2020; Maloneyet al. 2021). Examples of social VR applications are VTime, Altspace, VRChat and Rec Room. A unique feature of social VR is that users can be co-present as avatars and share the same virtual space (see, e.g. Blackwell et al. 2019; Zhang et al. 2017). Compared to many other mediated forms of interaction, such as video calls, the users are not sharing a space with their physical bodies, which can lead to asymmetries in interaction (see Heath and Luff 1992). During interaction, participants design their actions so that they are recognisable and interpretable by other co-present participants. Participants also ensure that their own understanding of others’ actions is correct (Robinson 2016). As interaction is organised into sequences of social actions (Stivers 2013), we can examine it as it unfolds turn by turn as well as how the participants display their understanding of prior

148

H. Spets

turns in their subsequent turns. These adjacent turns create intersubjectivity, the participants’ shared understanding of “what is happening right here and now” during interaction (Peräkylä 2013). Intersubjectivity is a practical, collaborative achievement that is not achieved once-and-for-all; it emerges continuously as a product of interaction that is shared between the participants (Deppermann 2019). It needs to be maintained and re-established as new actions and knowledge arise. Intersubjectivity can also be disrupted. For example, if an action is unrecognisable to other participants, it may need to be repeated, repaired or clarified before shared understanding is established. One of the resources participants use to achieve and maintain intersubjectivity is the ability to see what the others are doing (Sidnell 2014). In the context of VR, this can become a challenge since the users’ avatar actions might become only partially visible in VR or users’ access to the sensorial and contextual information in VR is reduced (Hindmarsh et al. 2006). The avatars’ bodily actions can also be independent of the users’ actions. These automated actions (“predefined” in Antonijevic 2008) can include facial expressions, or the orientation of the avatar based on motion tracking data, or they can be randomly system-generated. In Rec Room, the users’ facial expressions are mostly system-generated: if something hits an avatar’s face, for example, the avatar grimaces. Previous research on interaction in different types of virtual environments, such as virtual worlds and virtual reality, has focused on presence, co-presence and joint attention (e.g. Berger et al. 2016; Kohonen-Aho 2017; Kohonen-Aho and Vatanen 2021; Olbertz-Siitonen et al. 2020) as well as gestures and embodied conduct (e.g. Antonijevic 2008; Locher et al. 2015). Locher et al. (2015) examine novice users of VR negotiating and navigating the virtual space. Their findings show the participants developing a common perspective or focus of joint attention, i.e. establishing and maintaining intersubjectivity, in the virtual world Second Life. For example, the participants use requests for more information as a resource to negotiate a common perspective and maintain joint attention. Hindmarsh et al. (2006) pioneering study on virtual interaction illustrates a group of challenges to interaction in VR. These challenges relate to how much of the environment each participant can see and how

5 Intersubjective Interaction in Social VR

149

the access to the environment and its resources are not mutually shared between the participants. Another challenge concerns how participants can misinterpret each other’s avatar actions. The study also shows the difficulty of assessing what others are doing and tailoring one’s own actions to match them. This chapter complements current knowledge of how intersubjectivity is established, maintained and re-established in VR interaction. More specifically, it will illustrate the interactional challenges that participants navigate in the process. The chapter also contributes to research of embodiment in VR by examining participants’ embodied conduct as a part of the word explanation activity. The analysis shows how the representations of the multimodal resources are multi-layered in social VR and how that can become a challenge for the participants.

5.3

Materials and Methodology

The Rec Room virtual environment provides its users with a large virtual space full of activities such as 3D Charades, the focus in this chapter, as well as Paintball, Disc Golf and escape rooms. More importantly, Rec Room offers a platform for real-time social interaction in VR, where random and incidental encounters between previously unacquainted participants, as well as planned meetings with friends are possible. Rec Room is publicly available software (available via most VR platforms’ stores). The participants appear in the VR environment as avatars (see Fig. 5.1). In this study, the avatars were selected from Rec Room’s avatar library, and the participants had few chances to alter their avatar. In Rec Room, the participants have the following resources to control their avatars: when the players use the HMD and hand-held controllers, the system tracks and transfers the body movements to movements of the avatar body. This allows the user to move around in the virtual space, crouch, change their bodily orientation as well as control their gaze. The hand-held controllers can be used to interact with the environment (e.g. to pick up objects) and move around the space (e.g. to teleport longer distances). The participants can see each other’s avatars in Rec Room as

150

H. Spets

Fig. 5.1 Avatars. ©Rec Room Inc.

well as hear each other. The audio is directional: the exact location of a user affects how they can hear other users’ speech and other sounds in the game. It should be noted that detecting the user’s movements from the HMD and the controllers and transferring them to their avatar is not precise. Not all the user’s movements translate to the avatar nor do the movements appear similar in the avatars as they do in the physical environment. For example, a gesture made with a controller can appear different than how one would gesture with their physical hand. The avatars are also not one-to-one with the participants’ real bodies due to their simplistic appearance. For example, the avatars do not have legs, nor do they have arms connecting their hands to their body (see Fig. 5.1). The video data collected for this study is from a quasi-experimental setting where 12 participants in pairs of two engaged in the 3D Charades game in Rec Room. The participants were recorded in “situations that simulate naturally occurring interactions and situations” (Due 2015: 154). The video recordings were made at a Finnish university by students as a part of a course on interactional linguistics. The students recruited participants that were not enrolled on the course. The research participants gave their voluntary and informed consent to participate in the study. The participants were given limited instructions of what to do in the virtual space. The focus is on the 3D Charades game because interaction between the participants is an essential means to advance this game

5 Intersubjective Interaction in Social VR

151

activity. The chosen game also provides a chance to examine embodiment in VR as it encourages the use of one’s body and gestures. It should be noted that while most CA focuses on naturally occurring interaction, more and more research is being conducted in so-called quasi-experimental settings, such as VR. In the case of these materials, the participants are using VR equipment at a research site, and had they not participated in the study, they would not have interacted with each other in the VR environment. The ways in which they interact, however, depend entirely on the participants themselves with little to no input from the researchers present in the physical space. The materials consist of three streams of video and an audio recording. Each video offers a different view: two of the views are the two participants’ first-person perspectives of what is happening in VR, and one is a 360-degree video which has been recorded in the space the coparticipants inhabit physically. The materials have been edited into videos that show all the three views simultaneously in sync (see Fig. 5.2). The advantage of using materials such as these is that the researcher can see what the participants see: the VR space is “seen through the users’ eyes”.

Fig. 5.2 Edited view of all three streams. Participant 1 sees what is on the upper left, and their avatar can be seen on the upper right. Vice versa, participant 2 sees what is on the upper right, and their avatar can be seen on the upper left. The lower stream is from a 360-degree camera recording the participants (highlighted) in the physical space. ©Rec Room Inc.

152

H. Spets

The participants inhabit the same physical space as well as the same virtual space. The recording set-up is in one room (see Fig. 5.3). It includes one overhead camera recording 360-degree view video and two sets of VR equipment for the participants. One set of equipment includes an HMD, two hand-held controllers, two motion detection towers and a computer. The computer runs the VR application and records the participant’s view. The participants will be referred to with pseudonyms, and their usernames have been blurred in VR. As the 360-recordings do not show the participants’ faces, the materials have not been further anonymised.

Fig. 5.3 Recording set-up. Above: 360-camera. Left: participant wearing the HMD and hand-held controllers. Right: motion detection tower

5 Intersubjective Interaction in Social VR

153

Excerpts have been transcribed according to the conventions in Jefferson (2004) for talk and Mondada (2019b) for embodied action. Analysis will be based on conversation analysis (CA) (Sidnell and Stivers 2013). CA examines “situated action as it happens in both its systematic organization and its diversity within various social settings” (Mondada 2016: 338). CA has been widely used to study the organisation of interaction in multimodal settings: the concept of talk-ininteraction has been expanded to a more holistic approach to examining interaction (see e.g. Mondada 2019a; Mortensen 2012). Such an approach means that multimodal resources, such as gesture, talk, posture and body movement, are examined as participants use them to organise their actions (Mondada 2016).

5.4

Intersubjective Social Interaction During the Word Explanation Activity in Social Virtual Reality

The analysis started with a collection of 13 episodes from the 3D Charades game where the participants engaged in the word explanation activity. These episodes illustrate three different kinds of situations where the participants do not share similar access to objects, drawings and their avatar bodies, which affects their mutual understanding of the word explanation activity. Three excerpts were chosen to illustrate how the progression of the word explanation activity may be jeopardised and how the participants orient to and solve such situations. The findings show how the participants utilise four resources, three verbal (repair, description and requests for more information), and one embodied (facing each other), to establish and maintain progression of the gaming activity. In the following examples, the participants face a situation where access to a drawing, participants’ differing perspectives or the avatar body presents a challenge to the participants’ shared understanding of the gaming activity and the environment. First, there will be an introduction of the word explanation activity, followed by three sections of findings focusing on different challenges to intersubjectivity.

154

H. Spets

5.4.1 The Analysed Activity: Word Explanation The goal in the studied word explanation game “Charades” is to explain a word to the other participant(s). In the virtual space Rec Room, the game is transformed into a “3D Charades” game, which adds its own unique resources and challenges to playing the game and achieving its goal. During a game of 3D Charades, the participants are not only faced with the challenges of explaining a word without using said word, but also the novelty of playing the game in a VR environment with its unique virtual affordances. In the context of the 3D Charades game, the primary goal is to explain a word in such a way that the recipient can guess what it is without the speaker using the actual word. The participants can use a 3D pen (see Fig. 5.4) to draw in the air. When drawing on a paper, the surface is flat and 2D. When drawing in VR with the 3D pen, there is no flat surface to draw on: the drawings are three-dimensional. These drawings made with the 3D pen are a unique affordance of this specific VR environment. It is another layer to the multimodal resources available to the participants, adding to the complexity of interacting in the VR environment. The 3D Charades game itself imposes certain rules and guidelines on the participants: one should use the 3D pen and draw the word without speaking. However, these rules were not always followed by the participants in the recordings. The participants made their own highly situated rules that were negotiated among themselves and often used talk in addition to gestures and drawings made with the 3D pen. The actual game rules were rarely discussed, and even if they were, the participants mostly chose to talk regardless. The examined word explanation episodes have two participants: someone explaining (“explainer”) and someone guessing the word (“recipient”). The explainer performs their explanation using talk, gesture and, in the case of 3D Charades, drawings they sketch in the air with the 3D pen. The materials show that the explanations are designed to prompt guesses and that they can be built over a longer period of time with several guesses occurring during the explanation activity. Guesses can help the explainer design their explanation by informing them of the recipient’s understanding of the explanation and the recipient’s access to,

5 Intersubjective Interaction in Social VR

155

Fig. 5.4 3D pen held by an avatar

for example, the drawing or a gesture. If the explanation is not understood as the same thing by both participants, the goal of a correct guess is that much harder to reach. Excerpt 1 shows an instance of a word explanation with two participants, Heikki and Pertti. The explainer, Pertti, signals the start of their explanation of the word windmill with check this out and then starts drawing in the air as Heikki, the recipient, watches.

156

H. Spets

Excerpt 1 (Group 2; 15:23)

The structure of a word explanation can be seen in Excerpt 1. The explanation is initiated by the explainer: here, on line 1, Pertti says check this out, drawing the recipient’s attention to him. He then draws a picture based on the card he has picked. Next, the recipient can do one of two things: make a guess or a request for more information. The latter, as well as an incorrect guess, leads to another attempt at an explanation. The recipient constructs their next turn based on their understanding of both the prior turn(s) and the latest turn. On line 7, Heikki makes a guess, wind (.) mill?. It is the correct guess, as confirmed by Pertti’s yea:h you got it (l. 9). If the guess is incorrect, the explainer can use the recipient’s previous turn(s) to design their next turn accordingly. They can do so by building on their previous explanation(s), guiding the recipient’s focus to the relevant details of their explanation(s) or even starting over, among other things. The explainer can also encourage further guesses without more explanation. Turns alternate until the goal is reached, i.e. the recipient guesses the word correctly, the in-game timer runs out, or the participants give up.

5 Intersubjective Interaction in Social VR

157

The drawings the participants make can be inscriptions, a specific type of gesture (Goodwin 2007). With inscriptions, a physical mark is left in the environment by the movement of the hand(s), and the gesture is accompanied by talk. These gestures require something to inscribe in, such as sand, in the physical environment. In Rec Room, however, inscriptions can be made in the air with the 3D pen. The participants’ joint attention is focused on and around the explanation activity. The participants need to be able to see the drawing being made—and the 3D pen it is being made with—in order to understand what is being drawn.

5.4.2 Access to the Drawing as a Challenge As the drawings made in VR are three-dimensional, a drawing can look flat to one user while for another, viewing it from another direction, it might look bent and distorted. In such situations, the participants might not have the same access to a drawing, even when facing each other with the drawing between them. In Excerpt 2, Pertti and Heikki are again playing 3D Charades. It is Pertti’s turn to draw a word and explain it. He has picked a card with the word swamp. Pertti and Heikki’s avatars are standing face-to-face, and Pertti’s drawing is located between them. The focus here is on how the participants’ access to Pertti’s drawing differs.

158

H. Spets

Excerpt 2 (Group 2; 1:37)

5 Intersubjective Interaction in Social VR

159

Fig. 5.5 Left: Heikki’s view, Pertti’s drawing is highlighted. Right: Pertti’s view

The participants have different access to the drawing: Pertti sees a straight drawing, while for Heikki, it is curved (Fig. 5.5). As the drawing is a central object in the activity, this difference can affect their actions. Pertti’s get it (l. 3) is met with silence from Heikki, and he continues with a description of what he has drawn (l. 5). Heikki responds to Pertti’s description with laughter and a high-pitched what, requesting for more information (l. 8). Heikki’s response is an open class repair initiator that does not inform Pertti of the exact trouble source (Schegloff et al. 1977). Due to this, Pertti cannot design his repair attempts appropriately. Pertti responds with an attempted repair, then accounts for the quality of his drawing with his lack of drawing skills (ll. 9–12). For both participants, the situation is challenging. However, they orient to different issues: Pertti continues as if the issue is his ability to draw, while for Heikki, the challenge is more about his access to the drawing. The participants’ access to Pertti’s drawing differs, and this is becoming a challenge for the progression of the activity. If they do not reach a shared understanding, reaching the goal, a correct guess, can become more difficult. The participants are balancing progression with maintaining a shared understanding of a central part of the explanation, Pertti’s drawing. Pertti relies heavily on the drawing in his explanation, at first. The longer the explanation continues, the more descriptive his spoken turns become, and each turn is designed to draw Heikki’s attention to certain features of the drawing. Finally, Pertti states the gist of his explanation: what is Suomi, (0.2) when you minus this thing out (ll. 23–24). The central term for this explanation, swamp, is “suo” in Finnish. The participants

160

H. Spets

are native Finnish speakers, but they have been instructed to speak in English due to the course being in English. Pertti’s explanation is based on the fact that if you remove the last two letters of Suomi (‘Finland’), you have suo (‘swamp’). By this point in the explanation, Heikki no longer has any view of the drawing. Heikki’s attention has been divided between trying to drop a camera (ll. 23–24) and the explanation. He then teleports away from his initial position (opposite Pertti) (l. 27) by accident. Due to the change in position, Heikki has no access to the gesture that accompanies Pertti’s turn (ll.22–23), and the reference to the drawing is lost on him. Heikki initiates repair by repeating Pertti’s whole turn (l. 26). Pertti’s response (l. 28) gives Heikki the information he needs to reach the goal of the activity. What happens during the excerpt leads to the drawing, a central element in the explanation for the explainer, becoming of little use to the recipient. The participants reach the goal using actions performed through talk, for example, requests for more information and repair. Heikki, the recipient, has access to only one of the resources used by the explainer: talk. The drawing is available to Heikki if and only if he is face-to-face with Pertti—and even then, only partially in this case. The same can be said of Pertti’s gestures. The participants opt to orient to talk as an accessible resource, as by the end, most of Pertti’s explanation is delivered through talk in response to Heikki’s repair initiations.

5.4.3 Participants’ Differing Perspectives as a Challenge Establishing, and especially maintaining, intersubjectivity can be challenged by the participants’ perspectives. Due to their differing views of the virtual environment, finding a common perspective can become a challenge for the participants (Hindmarsh et al. 2006; Locher et al. 2015). Perspective can become an issue when a participant does not realise that what they see is not necessarily what the other sees. In the next excerpt, Pat and Lisa are playing 3D Charades. Lisa has picked a card with the word windmill and is preparing to explain the word to Pat. Before the excerpt begins, Lisa has sketched the first lines of her drawing. As in Excerpt 2, the distortions visible from the recipient’s

5 Intersubjective Interaction in Social VR

161

viewpoint affect the progression of the activity. Before the beginning of the explanation, the participants are facing each other. Right before the explanation begins, however, Lisa teleports to another position. Now, Lisa is standing with her right side to Pat. The drawing is situated in front of her, and Pat sees it from the side. Excerpt 3 (Group 1; 7:35)

162

H. Spets

In Excerpt 3, Pat’s what is this (l. 01) shows his orientation to the activity at hand. Pat is telling Lisa that he is following her explanation but is yet unable to guess based on what she has drawn so far. By continuing her drawing, Lisa shows that she orients to Pat’s turn as an indication of his orientation to her explanation. Pat tilts his head, and after a moment, repeats the movement. This time, it is more pronounced as he leans his whole body to the left (see Fig. 5.7). Leaning is a way for Pat to get a better view at Lisa’s drawing from another angle. Figures 5.6 and 5.7 illustrate how the leaning action allows Pat to have a slightly different perspective to the drawing. He does not begin guessing, however, until Lisa prompts him to imagine a () (l. 22). Lisa’s turn shows her orienting to a missing contribution from Pat. As the leaning actions that show Pat’s orientation to the guessing activity are not necessarily accessible to Lisa, she helps him by prompting him to contribute. At this point, Lisa begins to elaborate her explanation with other resources, such as talk and gesture. Pat responds with several guesses (ll. 11–12), none of which are correct. Again, he leans to the left to examine the drawing (l. 29, Fig. 5.8). Lisa orients to Pat’s turns as responses to a lacking explanation in the sense that her ability to draw is insufficient to express her ideas. She cannot identify the challenge Pat is facing since their access to the object of their mutual attention is different due to their differing perspectives. Between Excerpts 3 and 4, Lisa focuses on elaborating her drawing by both talk and additional drawing.

Fig. 5.6 Left: Lisa’s view. Right: Pat’s view. The drawing appears flat for Lisa, yet curved for Pat

5 Intersubjective Interaction in Social VR

Fig. 5.7 Pat’s view when he leans to his left on l. 18

Fig. 5.8 Pat’s view when he leans on l. 29

163

164

H. Spets

Excerpt 4 (Group 1; cont.)

Lisa’s further explanation turns have not helped Pat. They are orienting to different challenges: Lisa to her drawing and its complexity (ll. 19– 20), Pat to trying to see the drawing how Lisa sees it (l. 18, l. 22). Pat’s movements show that he is attempting to understand the picture by changing his perspective. After several attempts to look at the drawing from another angle while staying in the same position, Pat visibly begins to prepare to teleport by aiming at a new location with the controller. His aiming is interrupted by Lisa’s and- (l. 19). Pat listens to Lisa’s explanation and responds with laughter (l. 21). He then verbalises his plans to move to Lisa (l. 22), implicating that moving might help him guess the word. After Pat has repositioned himself and signalled Lisa that she can continue, Lisa complements her explanation (ll. 24–25). These two actions—change of position and perspective as well as the added explanation that also functions as a prompt for a guess—make it possible for Pat to make a correct guess, and the goal is reached.

5 Intersubjective Interaction in Social VR

165

In addition to these several repair initiations, Pat uses head tilts and leans, i.e. embodied conduct, to show that he is unable to recognise the drawing. Pat’s embodied actions do, however, show Lisa that he is attempting to construct an understanding of what he sees. Repeating the same action shows he has not been able to do so and that the process is still ongoing. As the aim of the activity is to guess what the drawing describes, the participants are required to establish joint attention and access to it. Due to the nature of VR, participants’ perspectives might be different, and their access to the object of attention might differ. Lisa does not recognise that her drawing appears different from where Pat is looking at it. Pat resolves the situation on his own by finding a better view of the drawing that would be closer to what Lisa sees—that is, facing her.

5.4.4 Access to the Avatar Bodies as a Challenge This section shows how the co-participants’ avatar bodies contribute to the asymmetry of interaction (see Antonijevic 2008; Hindmarsh et al. 2006). The participants’ access to the other participant’s avatar and their bodily actions differ and affect the symmetry of interaction and the progressivity of the activity at hand. The explainer attempts to solve the challenge by using verbal descriptions. Ultimately, the activity does not reach its goal, a correct guess. In Excerpt 5, two participants—Pertti and Heikki—are playing 3D Charades. They are about to begin the game activity with Heikki as the explainer. He picks a card with the word rib written on it. Heikki uses talk, gesture, his body and the material environment as resources to produce the word explanation. Pertti does not share his access to these resources which affects the way the co-participants’ try to establish a mutual understanding of the target word during the explanation activity.

166

H. Spets

Throughout the excerpt, Pertti and Heikki are facing each other. There is nothing between them that could restrict the view. Their positioning is deliberately established by the explainer: Heikki teleports around until he is satisfied with their positioning on the stage. Excerpt 5 (Group 2; 0:20)

Heikki initiates his explanation by pointing at the area where his ribs are in the physical environment and saying sort of like here? (l. 02). He continues by drawing over said location. In VR, however, the drawing appears as a floating object over his avatar’s side. It is reminiscent of the

5 Intersubjective Interaction in Social VR

167

Fig. 5.9 Left: Pertti’s view of Heikki’s avatar body and the drawing. Right: Heikki’s real body. The area of the drawing movement is indicated with circles in both images

explainer’s left hand holding the card he picked earlier as well as the avatar’s missing hands (see Fig. 5.9) as both are red. The first indication of trouble with the progression of the activity comes when Pertti does not try to guess the word. So far, Pertti has said nothing but to inform Heikki of the remaining time. Heikki reacts to this indication of trouble by requesting a confirmation that Pertti has seen what he has produced and is pointing at by saying can you see this, can you see this (ll. 08–09). This also functions as a request for a contribution from Pertti. Pertti responds this time, saying yeah I can see a (0.4) h(h)and(h)- hand >sort of looking thing %...> £...>

Turns-at-talk and embodied actions were transcribed according to the conventions listed in the Appendix.

268

H. Baldauf-Quilliatre and B. Ursi

Figure 3

02 fig touW gazW gazL touL gesM

(0.2)%£(0.3)%£(0.1)@(0.2)%(0.2)£(0.2)#@% #fig.4 --->%............%t tree.......%t heart ...>%looks at the heart--->05 ...>£turns to the heart --->05 £.................£touches heart --->@,,,,,,,,,,,,,,,,,,@

Figure 4

03 LOR fig

oh# oh #fig.5

8 Playing Together on a Large Screen

269

Figure 5

04 05 DAD fig gazW gazL

(0.6) voilà% c’est bien£# that’s it that’s good #fig.6 --->% --->£

Figure 6

MUM points verbally and with a gesture at a specific part of the screen, where a small heart is visible (l.01, see Fig. 3). Doing this, she initiates a showing sequence (Licoppe and Tuncer 2019). Immediately, LOR and WIS, the boys, orient their gaze towards that zone of the screen. Both touch the heart and thus collaborate to achieve the main aim of

270

H. Baldauf-Quilliatre and B. Ursi

the game: collecting different items (l.02, see Fig. 4). The heart is identified and confirmed as an “object to see/collect” in a collaborative way. Additionally, the showing of this item is treated as “showing for a special purpose”—the heart has to be collected by touching it. Once the boys have touched the heart, it increases in size, a change which is commented on by LOR with the vocalisation oh (l.03). Similar to English “oh”, French “oh” can be used as a surprise token (Wilkinson and Kitzinger 2006), which seems to be the case here: LOR comments on the changing heart. Meanwhile, both children continue focusing on the heart, maintaining one finger in pointing/touching gestures close to the heart’s place on the screen and gazing at that part of the screen (see Fig. 5). When the heart disappears, DAD assesses the gaming action positively and closes the instruction sequence (l.05). During DAD’s turn, LOR and WIS withdraw their gaze, their hands/fingers as well as their body from the heart’s place and orient to different parts of the screen—WIS moves to the right side and LOR to the left side (see Fig. 6). By withdrawing from one particular point on the screen, the boys display changes in participation: they still collaborate in the game by searching the hidden items, but they turn away to act individually, each searching on another side of the screen. In doing so, the young players position themselves towards multiple temporal dimensions (see 3.3) of the gaming activity: they switch from a specific game phase, i.e. the finding of a single item, to a next phase in the agenda of the game, i.e. another item needs to be found. This short extract illustrates how first- and second-row participants construct a shared interactional space on and around the tablet in two main ways, namely drawing on the screen architecture and on their bodily resources: (1) they orient bodily to the part of the screen where the item that MUM has named and pointed to is situated; (2) they offer and take space to accomplish actions in the general two-row arrangement, but also, more locally, retracting and getting back to home position (Sacks and Schegloff 2002), such as when MUM (l.02) retracts her finger as soon as both children have oriented to the heart and therefore offers them enough space to tap on it (see Fig. 4); (3) they establish a joint focus by touching the heart, so they realise the targeted gaming action; (4) they maintain the joint focus by maintaining bodily orientation even once the

8 Playing Together on a Large Screen

271

gaming action is accomplished; and (5) they withdraw and move on to the next phase (i.e. finding another target) when the gaming action is approved and positively assessed. Their joint orientation is initiated by a showing sequence, which is understood as an embodied instruction and achieved through the other two participants’ use of different bodily resources. The construction of the interactional space is closely related to participation practices, indicating knowledge, collaboration and shared understanding. The initial two-row arrangement of the family members and the side-by-side monitoring of the parents are thus temporarily suspended: MUM participates in the gaming activity, showing the hidden heart on the screen. She gets closer to the tactile surface and reaches out her arm to point to the heart, displaying changes in the participation framework as well as in the interactional space and projecting an embodied response from the young players, who recognise a target. Once the children have oriented their gazes to the target, i.e. the heart, she retracts physically from the gaming; participation framework and interactional space change again, the initial arrangement is re-established and the orientation to subsequent phases within the gaming agenda is carried on.

8.4.2 Dividing the Screen and Forming Dyads The second excerpt illustrates a different configuration of the family members in front of the screen and shows another involvement in the gaming activity. It begins right after extract 1, all four participants are back to the two-row arrangement. MUM announces the next target the players have to find: “the key” (l. 03, see also Hofstetter & Robles, this volume on “the talk about the game, during the game, that promotes progressivity of the game”). This prompt is not accompanied by a deictic gesture as in the previous example, but is temporally linked to prior actions, within the game activity, through an and preface (l. 01; Heritage and Sorjonen 1994). She is still kneeling in the second row, behind LOR (see Fig. 7).

272

H. Baldauf-Quilliatre and B. Ursi

Extract 2 (06:20–06:32) 01 MUM fig

et:# a:nd #fig.7

Figure 7

02 03 MUM bodM bodD 04 fig bodM gesM bodD gesD

(0.5) @*la clef the key @.....> *....> (0.1)@(0.6)*(0.3)@(0.2)@*# (1.4)@ #fig.8 >...@leans towards the left@,,> @...........@cutting gesture on screen---@ >...*leans forward *............*touches the screen with finger->07

8 Playing Together on a Large Screen

Figure 8

05 DAD 06 fig bodM gesM

Figure 9

regarde fais comme ça look do like this (0.2)@(1.8)@(0.1)@(0.5)@#(0.2)@(0.3) #fig.9 ,,,>@ @..................@leans forward @.....@gesture on screen--->07

273

274 07 MUM gesD gesM 08 DAD fig gesD bodM 09 fig gesD

Figure 10

Figure 11

H. Baldauf-Quilliatre and B. Ursi

hm@:*:@ -->* ->@,,,@ recule un@ peu *wissem# t` es *trop près (.) step back a bit Wissem you’re too close #fig.10 *..............*pulls W from screen-> >...@sits up straight voilà# comme ça\* okay like that #fig.11 --->*,,>>

8 Playing Together on a Large Screen

275

While producing the prompt, MUM starts leaning towards the left side of the screen, she extends her arms next to LOR’s body, allowing her to touch the screen and to remove leaves from a tree (see Fig. 8). Body arrangement matters: young players are placed between the screen and the parents. Specifically, the adult is positioned behind the child so that their visual perspectives and gestures are aligned, as shown in Fig. 9 where MUM’s and LOR’s arms and hands are parallel. MUM has to go around the body of LOR (i.e. the child in front of her) to reach the left end of the screen (see Fig. 8). She abandons this uncomfortable position rather quickly and goes back to her initial position behind her son (l.06). In the meantime, DAD leans forward on the right side of the tablet. The two parents are thus playing on two separate areas of the screen, together with their children. Two dyads are at work and two different interactional spaces emerge, each involving a front row participant and a back row participant. The management of the game is quite different in the two interactional spaces. On the one side, DAD shows to WIS the way in which the child can cut the leaves from the trees and thus reveal hidden objects. He mobilises WIS’ attention with a look-prefaced turn (Sidnell 2007), followed by an embodied instruction (l.05). On the other side, MUM plays in silence alongside LOR from her second-row position (see Fig. 9), leaning forward and reaching out her arm to touch the tactile surface. She produces a nonlexical vocalisation (l.07) that both embodies the pleasure (Wiggins 2002) of the game and acknowledges a local achievement (Schegloff 1982) in the ongoing gaming activity. Here, the achievement consists in a series of swiping gestures, through which MUM has cut leaves from trees, while leaning on the left side of the screen (see Fig. 8) and forward (see Fig. 9). After this turn, she straightens up and her body is now distanced from the screen (see Fig. 10). She monitors LOR’s gaming activities but does not intervene any more until the end of the extract. Simultaneously, DAD leans forward and operates a haptic control (Cekaite 2015) on WIS: he repositions his son’s body so that WIS kneels further away from the screen (see Fig. 11)—which is in this case treated as a (potentially dangerous and fragile) object. Lines 08–09, DAD’s turn consists in a directive followed by an address (‘step back a bit Wissem’), its related

276

H. Baldauf-Quilliatre and B. Ursi

account (‘you’re too close’) and the securing of the child’s new posture (‘okay like that’). This excerpt shows the emergence of separate interactional spaces within the gaming activity, each interactional space involving a front row participant and a back row participant. Parents play together with their children and act on different areas of the screen (see Fig. 9). In contrast to schisming in conversation (Egbert 1997), here all participants act collaboratively within the joint gaming activity, following a shared agenda. The initial two-row arrangement shifts into two “side-by-side” configurations, accompanying a change in participation frameworks: parents are no longer monitoring the actions of their children, but accomplishing on-screen gaming actions alongside the two boys. Moreover, adults’ positioning towards the gaming activity varies. They can act in different ways. In extract 2, MUM plays independently on the screen being next to the child, through a noticeable reconfiguration that results in repositioning her upper body. On the other hand, DAD provides instructions through specific touching gestures on the screen and mobilises complex haptic configurations to control the location of his first-row young player in relation to the device, slightly leaning forward.

8.4.3 The Emergence and Negotiation of Individual Spaces on the Screen In the previous sections, we have shown two different ways of organising interactional spaces and their relation to participation frameworks in the family gaming interaction: first-row and second-row participants focus together on a particular screen area (8.4.1) and all participants form two dyads and thus create two interactional spaces involving one participant from each row (8.4.2). Additionally, we have pointed out the importance of embodied resources to implement these practices: mobilising deictic gestures to point at some on-screen areas to the children and subsequently offering them space (8.4.1); illustrating and accomplishing specific types of screen-based touching gestures for specific gaming purposes, alongside the children (8.4.2). The first two extracts have also shown how the participants carry out gestures on the screen

8 Playing Together on a Large Screen

277

within the game as individual players, as well as how they navigate between several positions. The following excerpt shows these changing configurations in a short time frame, which highlights the dynamics of the participation framework as well as the spatial organisation on and around the large tablet. It also provides evidence that second-row participants can temporarily become players by departing from their home position. At the very beginning of excerpt 3, the children are playing and the adults are watching their tactile moves on the screen. Extract 3 (04:00–04:10) 01 fig bodM gesM

Figure 12a

02 MED fig bodM gesM bodD gesD 03 MUM fig gesM

(0.1)@(0.4)@(0.5)# @(0.1)@(0.3)@ #fig.12a#12b @leans forwards----@ @leans backwards-> @............@-----@,,,,,@ (touches screen/barrel)

Figure 12b

*faut# trouver@* isa@ need to find Isa #fig.13 --->@ @....> *leans forwards---* *.....> @LÀ:# here #fig.14 >.@touches the screen/barrel, eyebrows up->04

278

H. Baldauf-Quilliatre and B. Ursi

Figure 13

Figure 14

04 gesM 05 MUM fig gesM gesD

Figure 15a

(0.2)@ -->@..> *il est@ là*# it’s here #fig.15a#15b >...@--->(touches the screen/clover) .>*----------*,,,,,>(touches the screen)

Figure 15b

8 Playing Together on a Large Screen

06

(0.4) fig gesM bodM gesD bodD

@*(0.2)

@(0.5)@#

*(0.1)@(0.3)

#fig.16 --->@,,,,,,,,,,,,,,,,,,,,,,,@ @leans backwards@ >,,* *leans backwards-----------*

Figure 16

07 DAD fig bodD gesD

Figure 17

voi*là *un trèfle okay a clover

*allez les *autres*# go on (look for) the others #fig.17 *leans forwards-----------------------* *extends right arm*........*t screen->

279

280 08 gesD bodD 09 MUM fig gesD gesM

H. Baldauf-Quilliatre and B. Ursi

(0.1)*(0.1) -->* *retracts right arm, leans backwards-> @dans l` coffre au tré*@sor@# in the treasure chest #fig.18 -->* @leans forwards--------@@--@(points at screen/ treasure chest)

Figure 18

10 fig gesM bodD gesL

Figure 19

@(0.3)£(0.4)£(0.1)# *£(0.2)£(0.2)@(0.9) #fig.19 @leans backwards-----------------@ -->*leans fwds, shoulders and right arm down-> £.....£--------£,,,,,£(touches the screen/barrel)

8 Playing Together on a Large Screen

11 DAD bodD 12 13 DAD fig bodM gesD

281

*là there ->*.....> (0.8) @en *bas*# down there #fig.20 @leans forwards--->> >...*---*(touches the screen)

Figure 20

During the whole interaction, MUM kneels behind LOR, but here she is involved in the game and participates in the search for one of the three hidden items: she leans forward (see Fig. 12a) and touches a barrel on the screen (see Fig. 12b). Subsequently, the barrel rises and the hidden clover appears (see Fig. 13). Immediately after having touched the barrel, MUM retracts her arm and returns to her initial position. She thus leaves the space close to the tablet to her sons, giving them a game opportunity, but they remain focused on other parts of the screen in search of items (see Fig. 13). Subsequently, MUM’s greater involvement in the gaming activity is evident: when the clover appears, she moves forward and does a second touching gesture (see Fig. 15b) which is more sustained than the previous one (see Fig. 12b). Additionally, she embodies the surprise of finding the clover by raising her eyebrows (see Fig. 14). The pointing that accompanies the locative deictic là (Mondada and Pfänder 2016) is therefore incorporated in the realisation of the second touching gesture on the screen (see the multimodal conventions on gestures’ phases at the end of the chapter). WIS and LOR focus their attention on the clover revealed by MUM only when she touches the item (see Fig. 15a),

282

H. Baldauf-Quilliatre and B. Ursi

verbalises her finding through a full clause (il est là, l.05) and collects it. At that moment, DAD is still focused on another part of the screen, but the children gaze at the clover, which becomes bigger and pulses after MUM’s second touch (see Fig. 16). Once the gaming action has been successfully accomplished, MUM leans back again and resumes her initial position (l.06). DAD gets involved in the gaming from line 02 onward: he leans forward and works with WIS on the right part of the screen, touching barrels and searching for items (see Fig. 14). His first verbal turn l.07 offers insights on the organisation of the ongoing gaming activity: he acknowledges the clover as one of the targets of the game session (‘okay a clover’) and relaunches the trajectory of the game (‘go on (look for) the others’), in favour of game progressivity (see Hofstetter and Robles, this volume). In other words, he embeds finding the clover in the shared agenda of the game by evoking the other items the players have to look for. All the family members are thus involved in the search and rely on their initiatives, focusing on distinct parts of the screen (Fig. 17). As for the various moves of the search that lead to the discovery and collection of hidden items within the game, collaboration may emerge at any time. Indeed, in line 09, MUM mentions a specific referent, “the treasure chest”, as a possible hiding place for one of the remaining items to find. Her verbal turn is followed by a reconfiguration of MUM’s body orientation. In particular, she leans towards the screen and points at the treasure chest. Being in the second row does not allow her to play easily (i.e. to touch the items on the screen), but she can participate in the gaming activity by searching for the hidden items and revealing (possible) hiding places. LOR, who is closer to the screen and can easily touch it, looks at MUM’s hand and the target of her pointing (see Fig. 18) and touches the treasure chest (see Fig. 19), which does not contain any object, though. But there are other treasure chests on the screen. In lines 10–11, DAD initiates a rearrangement of his body in front of the tablet to reach a barrel on the left side of the screen. Simultaneously (l.11 and 13), he signals another possible target on the screen deictically. Through the rearrangement of his posture, he is indicating that he will at least try to accomplish the tapping himself. He goes around WIS (see Fig. 20), reaching the opposite side of the screen and touching the barrel.

8 Playing Together on a Large Screen

283

This (longer) excerpt shows how the adults, despite their placement in the second row, display availability for interaction, participate in the gaming activity and position themselves as players. They constantly change their posture in order to reach the tactile surface or to point to candidate hiding places as clearly as possible. Nevertheless, they do not abandon the two-row arrangement, which allows them to stand back at any moment and leave an opportunity for the children in the first row to play (cf. “space offering”). In addition, the extract gives insights about possible general configurations of the gaming activity in front of a tablet: participants can act on different areas of the screen and all contribute to the success of the game, namely finding the clover, the key and the heart. They pursue the target of the game as an “agenda” (Deppermann, Schmitt and Mondada 2010), which does not specify the very details of actions that participants have to accomplish and, in some cases, how they have to coordinate their moves in order to achieve the established goal. The same deictic form (là) may index potential targets by local pointing gestures, in various participation and spatial configurations. It may be accompanied by unilateral initiatives of the parents, who verbalise their moves, such as in the cases of MUM (see Figs. 12, 14, 15) and DAD (see Fig. 20). However, parents may also mobilise a deictic form or the mention of the target object, and then offer the children space and therefore the opportunity to play (see Fig. 18). Last but not least, the role of the tablet as a complex material and spatial resource in interaction emerges from the division of the screen as a tactile surface for individual actions, which is not defined once and for all: participants may reconfigure their own space and, generally, the interactional space of the game through noticeable body movements and rearrangements (see Fig. 20).

284

8.5

H. Baldauf-Quilliatre and B. Ursi

Discussion: Complexity and Interaction in Collaborative Tablet Gaming

Our analyses of family gaming interaction on a large tablet allow us to reflect on the complexity of interaction with regard to the use of multiple embodied resources in a complex setting with dynamic participation frameworks. Complexity is understood here as related to the mobilisation of a variety of multimodal resources and to different features, mobilised in a finely tuned temporal organisation, to accomplish joint actions and to co-construct gaming activity within a screen-centred interaction (see also introduction). In this section, we discuss the main aspects that lead to the understanding of interaction complexity as multidimensional through the analysis of our interactional data: spatiality, materiality and temporality, each dimension being characterised by multi-layeredness. A first aspect of complexity concerns spatial organisation and the physical arrangement of participants: since the beginning of the game session, adults and children have positioned themselves in a two-row arrangement, which is specific to the screen-centred interactions we have studied. Here, participants’ spatial configuration is constrained by the presence of the large tablet, which is placed against a wall, by the type of games (children’s games) and by the different heights of the participants (small children and taller adults). It produces an asymmetry between parents and children, as the former cannot easily touch the screen and accomplish gaming actions, but instead they can monitor the young players and instruct them about precise and local touching gestures, as well as about target areas on which to intervene on the screen (ex. 1 and 3). The adults in the second-row position themselves in a particular way (as spectators or instructors), whereas the children in the first row are positioned as main players and as those who may need help and support. Parents can take part in the game as players through specific touching gestures on the screen (ex. 2–3) and also operate a haptic control on children’s bodies, concerning their distance from and posture in front of the screen (ex. 2).4 4

This gives insight into the way children are categorised (Kern 2018) by adults and the room for manoeuvre that is allowed to young players with / in front of a large tablet.

8 Playing Together on a Large Screen

285

This complex arrangement is articulated on two axes (the young players and the parents are side by side, respectively) and makes it possible to constitute parent–child dyads, which operate independently and simultaneously on different sides of the tactile surface (ex. 2 and 3), as well as a united team, whose members play together to achieve the purposes of the game. Consequently, the gaming interaction is constructed as an activity that allows or even needs the formation of ephemeral groups, whose two-axes spatial organisation can be negotiated (ex. 3, see Fig. 20). Screen areas as foci of attention can be signalled in progressive, step-by-step phases (ex. 3), through verbal and bodily resources, and thus catch the attention of players who are involved in other courses of action. Incursions into the space of the other group, such as the one realised by DAD in extract 3, show the collaborative and embodied realisation of the pursued aim. Collaboration is achieved through multimodal resources: pointing gestures and deictic forms constitute primary resources, which are mobilised by adults to solicit children’s attention on the on-screen objects. The tworow arrangement represents a general infrastructure for building several participation frameworks, making them relevant for different participants and enabling a transition between established frameworks. In other words, several degrees of involvement in the gaming activity are thus possible. A second aspect of complexity is the multi-layered organisation of the screen as a technological object with regard to the participants’ interaction. On the one hand, the screen is a tactile surface on which participants can act, namely by accomplishing different types of touching gestures. These gestures are specific to touch screens and need to be acquired (Ursi and Baldauf-Quilliatre 2021). On the other hand, the screen is a system of signification, where in our particular case specific objects hide other objects. Both aspects can be addressed separately in the interaction. For instance, through haptic control (such as DAD in ex. 2), parents enact norms concerning the placement in front of a tactile surface (not too close to the screen); or through reference to objects that are visible on the screen (barrels, clover, heart), they act within a specific semiotic system. In other cases, the distinction between these aspects is not that clear: by leaning forward and pointing at or touching a particular object on

286

H. Baldauf-Quilliatre and B. Ursi

the screen, the participants address simultaneously the screen as a tactile surface (since they have to lean forward to reach it) and as a semiotic system (since they point at/touch a particular item within the gaming system). Moreover, in order to lean forward to reach the tablet, the parents go around (ex. 2) and even over (ex. 3) their children’s bodies; the screen is thus integrated into a three-dimensional space as tactile surface and as semiotic system. Incidentally, reference to objects within the virtual world of the game relies on referential semantics as well: behind the leaves of a tree there might be a heart, a barrel may hide a clover and a treasure chest may hide another precious item. The tablet represents a semiotically rich virtual space, but it is characterised by a bidimensional tactile surface as for its virtual setting. On the other hand, as it is placed against a wall, it is characterised by a three-dimensional configuration that is projected on the changeable bodily arrangements of adults and children, who can lean forward in order to point at and act on specific screen areas. A third aspect of complexity concerns the multi-level temporality of the interaction within a gaming session. The finely tuned temporal organisation of interaction in video game settings has already been highlighted, especially with regard to the sequential organisation of directives (Mondada 2013a). Our data highlight another aspect: the search for hidden items follows a progressive multidimensional development and, as for the temporal dimension, an organisation in phases. This organisation emerges from an online perspective on a specific research process: participants have to prepare for the discovery of a hidden item (e.g. by cutting the leaves from a tree) or to touch the hiding object (e.g. to open a treasure chest) and then catch the target. From a broader perspective, all participants accomplish their on-screen actions on the basis of a shared agenda, which includes the different phases of the game (i.e. the search for three hidden items), and which is also explicitly accounted for by participants. In extract 3, for instance, DAD acknowledges the finding of a clover, which is reframed as one of the three steps towards accomplishing the goal of the game. The mention of the other two items that participants have to find shows an orientation in terms of local and recognisable accomplishments within an overall game activity. In other words, DAD recalls the targets of the game search but does not explicitly

8 Playing Together on a Large Screen

287

mention either the details of the actions that participants have to accomplish or the precise, coordinated strategy they have to follow in order to achieve the established purposes. In other words, the game strategy is here strictly target-bounded and emergent: participants can look for a clover or a key; after having found a heart, a constraint on the recognisability of the next target is envisaged only for the last item in a sequence (after having found a heart and a clover, the players are expected to catch the key). Our study shows that a complexity approach to interaction needs to be multidimensional; that is, it should include all the dimensions accounted for by the participants, as well as their respective multi-layered features. We argue that this is not restricted to the study of screenbased interaction, but holds for the study of interaction in general. The multi-layered approach of spatial configuration, object mobilisation and temporal organisation represents an indispensable tool to capture the complexity of talk-in-interaction (see introduction). Participants orient to all these dimensions simultaneously, and their orientation and the recognisability of their accomplishments (initiating moves or unilateral actions) are treated as fully intelligible by all co-participants. The initial constraints (spatial, material or temporal) provide an infrastructure for the interaction, whose social actualisations are inherently emergent and turn out to be negotiable in the unfolding of interaction. At no point in our data did the interactants draw on problems or troubles with regard to a complicated setting, complicated technology or complicated game. Nor did they refer in any way to complicatedness at a linguistic level. Nevertheless, a fine-grained interactional analysis highlights the complexity of its structure and organisation. We therefore argue for a clear distinction between “complicated” (made relevant by the participants in interaction) and “complex” (according to detailed analysis). Our study shows that taking into account the multidimensional and multi-layered features of interaction, namely the mobilisation of participants’ bodily resources, the semiotic systems at work and the temporalities accounted for by participants, represents a viable endeavour for the study of complexity in talk-in-interaction. Interaction with technological objects, and even everyday interaction, is a very complex phenomenon that cannot be reduced to language complexity.

288

H. Baldauf-Quilliatre and B. Ursi

Appendix Transcript Conventions Turns-at-talk are transcribed according to the ICOR conventions, which were inspired by the Jeffersonian transcript system (Jefferson 2004) and are available at https://icar.cnrs.fr/projets/corinte/documents/2013_ Conv_ICOR_250313.pdf. [] tu v: (.)

(0.8)

Beginning and end of overlap Cut-off Prolongation of the immediately prior sound Brief interval within or between turns (< 0.2 sec) Elapsed time by tenths of seconds

.h

Inbreath

t‘ es OUI

Non-standard elision Louder sounds

/

Rising intonation of the immediately prior sound

\

Falling intonation of the immediately prior sound

As for the notation of multimodality, the transcription system designed by Lorenza Mondada (2019) has been adapted according to the needs of our analyses. bodM gazL gesD

@@ %% ££ ** … – -

– (touches the heart) – (t tree) „, – -> – -> >

In the pseudonym column, indicates the body movement (bod), gaze (gaz) and hand gesture (ges) of the transcribed embodied actions; capital letters indicate the participant doing the action (M means MUM, D means DAD, W means WIS, L means LOR) Description of embodied actions is delimited between two identical symbols (one symbol per participant) that are synchronised with correspondent stretches of talk or time indications Preparation of the body movement/gaze/hand gesture The body position is reached and maintained, the gaze/hand gesture target is reached and the gaze/gesture is maintained The description of the embodied action is provided between brackets after the notation of gesture phases, if there is not enough space Body’s/gaze’s/gesture’s retraction The movement/gaze/gesture continues across subsequent lines The movement/gaze/gesture continues after the end of the excerpt (continued)

8 Playing Together on a Large Screen

289

(continued) #

#15a#15b

The exact moment at which a screen shot has been taken is indicated with a numbered hash symbol (#1) showing its position within the turn or a time measure Hash signs (#) that are followed by the same number and a lowercase letter (a, b) indicate simultaneous screen shots

References Aarseth, Espen, and Gordon Calleja. 2015. “The World Game: The Ontology of an Indefinable Object.” In Proceedings of the 10th International Conference on the Foundations of Digital Games (FDG 2015). Pacific Grove.https:// www.um.edu.mt/library/oar/bitstream/123456789/26620/3/The_word_g ame_the_ontology_of_an_undefinable_object_2015.pdf Baldauf-Quilliatre, Heike, Isabel Colón de Carvajal. 2015. “Is the Avatar Considered as a Participant by the Players? A Conversational Analysis of Multi-player Videogames Interactions.” PsychNology Journal 13 (2–3): 127–147. Baldauf-Quilliatre, Heike, and Isabel Colón de Carvajal. 2019. “Interaktionen bei Videospiel-Sessions: Interagieren in einem hybriden Raum.” [Interactions in videogame sessions: interacting in a hybrid space]. In Interaktion und Medien. Interaktionsanalytische Zugänge zu medienvermittelter Kommunikation, ed. Konstanze Marx, and Axel Schmidt, 219–254. Heidelberg: Winter. Baldauf-Quilliatre, Heike, Isabel Colón de Carvajal. 2021. “Spectating: How Non-players Participate in Videogaming.” Journal Für Medienlinguistik 4 (2): 123–161. Baldauf-Quilliatre, Heike, and Biagio Ursi. 2022. “Constructing Local Identities in Multiparty Interactions with Tablets.” In Participation & Identity. Empirical Investigations of States and Dynamics, ed. Alexander Brock et al., 11–36. Berlin: Peter Lang. Beckner, Clay, Richard Blythe, Joan Bybee, Morten H. Christiansen, William Croft, Nick C. Ellis, John Holland, Jinyun Ke, Diane Larsen-Freeman, and Tom Schoenemann. 2009. “Language Is a Complex Adaptive System: Position Paper.” Language Learning 59: 1–26.

290

H. Baldauf-Quilliatre and B. Ursi

Broth, Mathias, and Lorenza Mondada. 2013. “Walking Away: The Embodied Achievement of Activity Closings in Mobile Interaction.” Journal of Pragmatics 47 (1): 41–58. Cekaite, Asta. 2015. “The Coordination of Talk and Touch in Adults’ Directives to Children: Touch and Social Control.” Research on Language and Social Interaction 48 (2): 152–175. Cekaite, Asta, and Lorenza Mondada, eds. 2020. Touch in Social Interaction: Touch, Language, and Body. London: Routledge. Crescenzi, Lucrezia, Carey Jewitt, and Sara Price. 2014. “The Role of Touch in Preschool Children’s Learning Using iPad versus Paper Interaction.” Australian Journal of Language and Literacy 37 (2): 86–95. Day, Dennis, and Johannes Wagner, eds. 2019. Objects, Bodies and Work Practice. Bristol: Multilingual Matters. Deppermann, Arnulf, Reinhold Schmitt, and Lorenza Mondada. 2010. “Agenda and Emergence: Contingent and Planned Activities in a Meeting.” Journal of Pragmatics 42 (6): 1700–1718. Egbert, Maria. 1997. “Schisming: The Collaborative Transformation from a Single Conversation to Multiple Conversations.” Research on Language and Social Interaction 30 (1): 1–51. Goffman, Erving. 1981. “Footing.” In Forms of talk, 124–159. Philadelphia: University of Pennsylvania Press. Goodwin, Charles, and Marjorie H. Goodwin. 2004. “Participation.” In A Companion to Linguistic Anthropology, ed. Alessandro Duranti, 222–244. Malden, MA: Blackwell. Goodwin, Charles. 2007. “Participation, Stance and Affect in the Organization of Activities.” Discourse & Society 18 (1): 53–73. Haddington, Pentti, and Mirka Rauniomaa. 2014. “Interaction between Road Users: Offering Space in Traffic.” Space and Culture 17 (2): 176–190. Hausendorf, Heiko, Lorenza Mondada, and Reinhold Schmitt (eds). 2012. Raum als interaktive Ressource [Space as Resource for Interaction]. Tübingen: Narr. Hausendorf, Heiko, and Reinhold Schmitt. 2013. “Interaktionsarchitektur und Sozialtopografie: Umrisse einer raumlinguistischen Programmatik” [Interactional Architecture and Social Topography: Sketches of a Space Linguistics Programme]. Arbeitspapiere des UFSP Sprache und Raum (SpuR) 1. https:// www.zora.uzh.ch/id/eprint/78153/. Heritage, John. 2012. “Epistemics in Action: Action Formation and Territories of Knowledge.” Research on Language and Social Interaction 45 (1): 1–29.

8 Playing Together on a Large Screen

291

Heritage, John, and Marja-Leena. Sorjonen. 1994. “Constituting and Maintaining Activities across Sequences: And -Prefacing as a Feature of Question Design.” Language in Society 23 (1): 1–29. Jakonen, Teppo, and Kreeta Niemi. 2020. “Managing Participation and TurnTaking in Children’s Digital Activities: Touch in Blocking a Peer’s Hand.” Social Interaction. Video-Based Studies of Human Sociality 3 (1). https://doi. org/10.7146/si.v3i1.120250. Jefferson, Gail. 2004. “Glossary of Transcript Symbols with an Introduction.” In Conversation Analysis: Studies from the First Generation, ed. Gene H. Lerner, 13–31. Amsterdam/Philadelphia: John Benjamins. Jewitt, Carey, and Kerstin Leder Mackley. 2019. “Methodological Dialogues across Multimodality and Sensory Ethnography: Digital Touch Communication.” Qualitative Research 19 (1): 90–110. Jucker, Andreas H., Heiko Hausendorf, Christa Dürscheid, Karina Frick, Christoph Hottiger, Wolfgang Kesselheim, Angelika Linke, Nathalie Meyer, and Antonia Steger. 2018. “Doing Space in Face-to-Face Interaction and on Interactive Multimodal Platforms.” Journal of Pragmatics 134: 85–101. Keating, Elizabeth, and Chiho Sunakawa. 2010. “Participation Cues: Coordinating Activity and Collaboration in Complex Online Gaming Worlds.” Language in Society 39 (3): 331–356. Kendon, Adam. 1990. Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press. Kern, Friederike. 2018. “Mastering the Body. Correcting Bodily Conduct in Adult-Child Interaction.” Research on Children and Social Interaction 2 (2): 213–234. Kohonen-Aho, Laura, and Anna Vatanen. 2021. “(Re-)Opening an Encounter in the Virtual World of Second Life. On Types of Joint Presence in Avatar Interaction.” Journal Für Medienlinguistik 4 (2): 14–51. Licoppe, Christian, and Julien Morel. 2013. “Interactionally Generated Encounters and the Accomplishment of Mutual Proximity in Mobile Phone Conversations.” In Interaction and Mobility: Language and the Body in Motion, ed. Pentti Haddington, Lorenza Mondada, and Maurice Nevile, 277–299. Berlin: De Gruyter. Licoppe, Christian, and Julien Figeac. 2014. “Direct Video Observation of the Uses of Smartphones on the Move: Reconceptualizing Mobile MultiActivity.” In Mobility and Locative Media. Mobile Communication in Hybrid Spaces, ed. Adriana de Souza e Silva, and Mimi Sheller, 48–64. London: Routledge.

292

H. Baldauf-Quilliatre and B. Ursi

Licoppe, Christian, and Julien Figeac. 2017. “Gaze Patterns and the Temporal Organization of Multiple Activities in Mobile Smartphone Uses.” HumanComputer Interaction 33 (5–6): 311–334. Licoppe, Christian, and Sylvaine Tuncer. 2019. “The Initiation of Showing Sequences in Video-Mediated Communication.” Gesprächsforschung Online-Zeitschrift Zur Verbalen Interaktion 20: 545–571. Livingston, Eric. 2006. “Ethnomethodological Studies of Mediated Interaction and Mundane Expertise.” The Sociological Review 54 (3): 405–425. Lund, Kristine, Pierluigi Basso Fossali, Audrey Mazur-Palandre, and Magali Ollagnier-Beldame, eds. 2022. Language is a Complex Adaptive System. Berlin: Language Science Press. Mondada, Lorenza. 2005. “La constitution de l’origo déictique comme travail interactionnel des participants: une approche praxéologique de la spatialité” [The Constitution of the Deictic Origo as Participants’ Interactional Accomplishment: A Praxeological Approach to Spatiality] Intellectica 41–42: 75–100. Mondada, Lorenza. 2007. “Interaktionsraum und Koordinierung.” [Interactional Space and Coordination]. In Koordination. Analysen zur multimodalen Interaktion, ed. Reinhold Schmitt, 55–94. Tübingen: Narr. Mondada, Lorenza. 2008. “Contributions de la linguistique interactionnelle.” In [Contributions of interactional linguistics] In Congrès Mondial de Linguistique Française – CMLF 2008, ed. Jacques Durand, Benoît Habert, and Bernard Laks, 881–897. Paris: Institut de Linguistique Française. Mondada, Lorenza. 2009. “Emergent Focused Interactions in Public Places: A Systematic Analysis of the Multimodal Achievement of a Common Interactional Space.” Journal of Pragmatics 41 (10): 1977–1997. Mondada, Lorenza. 2011. “The Interactional Production of Multiple Spatialities within a Participatory Democracy Meeting.” Social Semiotics 21 (2): 289–316. Mondada, Lorenza. 2012. “Coordinating Action and Talk-in-Interaction In and Out of Video Games.” In The Appropriation of Media in Everyday Life, ed. Ruth Ayaß, and Cornelia Gerhardt, 231–270. Amsterdam and Philadelphia: John Benjamins. Mondada, Lorenza. 2013a. “Coordinating Mobile Action in Real Time: The Timely Organisation of Directives in Video Games.” In Interaction and Mobility. Language and the Body in Motion, ed. Pentti Haddington, Lorenza Mondada, and Maurice Nevile, 300–342. Berlin: De Gruyter. Mondada, Lorenza. 2013b. “Interactional Space and the Study of Embodied Talk-in-Interaction.” In Space in Language and Linguistics: Geographical,

8 Playing Together on a Large Screen

293

Interactional, and Cognitive Perspectives, ed. Peter Auer, Martin Hilpert, Anja Stukenbrock, and Benedikt Szmrecsanyi, 247–275. Berlin: De Gruyter. Mondada, Lorenza. 2019. “Conventions for Multimodal Transcription (initial version: 2001; current version: 5.0.1, Nov. 2019).” https://www.lorenzamo ndada.net/multimodal-transcription. Mondada, Lorenza, and Stefan Pfänder. 2016. “Corpus international écologique de la langue française (CIEL-F): un corpus pour la recherche comparée sur le français parlé.” [International Ecological Corpus of the French Language (CIEL-F): A Corpus for Comparative Research on Spoken French] Corpus 15. https://doi.org/10.4000/corpus.2951. Mufwene, Salikoko S., Christophe Coupé, and François Pellegrino. 2017. Complexity in Language: A Multifaceted Phenomenon. In Complexity in Language: Developmental and Evolutionary Perspectives, ed. Salikoko S. Mufwene, Christophe Coupé, and François Pellegrino, 1–29. Cambridge: Cambridge University Press. Nevile, Maurice, Pentti Haddington, Trine Heinemann, and Mirka Rauniomaa (eds). 2014. Interacting with Objects: Language, Materiality, and Social Activity. Amsterdam/Philadelphia: John Benjamins. Oittinen, Tuire. 2020. “Coordinating Actions in and Across Interactional Spaces in Technology-Mediated Business Meetings.” JYU dissertations. http:/ /urn.fi/URN:ISBN:978-951-39-8176-1. Oittinen, Tuire, and Arja Piirainen-Marsh. 2015. Openings in TechnologyMediated Business Meetings. Journal of Pragmatics 85: 47–66. Oloff, Florence. 2019. “Das Smartphone als soziales Objekt: Eine multimodale Analyse von initialen Zeigesequenzen in Alltagsgesprächen.” [The smartphone as social object: A multimodal analysis of initial pointing sequences in mundane interactions]. In Interaktion und Medien. Interaktionsanalytische Zugänge zu medienvermittelter Kommunikation, ed. by Konstanze Marx, and Axel Schmidt, 191–218. Heidelberg: Winter. Peltonen, Peter, Esko Kurvinen, Antti Salovaara, Giulio Jacucci, Tommi Ilmonen, John Evans, Antti Oulasvirta, and Petri Saarikko. 2008. “It’s Mine, Don’t Touch! Interactions at a Large Multi-Touch Display in a City Centre.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1285–1294. https://doi.org/10.1145/1357054.1357255. Piirainen-Marsh, Arja. 2012. “Organising Participation in Video Gaming Activities.” In The Appropriation of Media in Everyday Life, ed. Ruth Ayaß, and Cornelia Gerhardt, 195–230. Amsterdam/Philadelphia: John Benjamins.

294

H. Baldauf-Quilliatre and B. Ursi

Raclaw, Joshua, Jessica S. Robles, and Stephen M. DiDomenico. 2016. “Providing Epistemic Support for Assessments through Mobile-Supported Sharing Activities.” Research on Language and Social Interaction 49 (4): 362–379. Reeves, Stuart, Christian Greiffenhagen, Martin Flintham, Steve Benford, Matt Adams, Ju Row-Farr, and Nicholas Tandavantij. 2015. “I’d Hide You: Performing Live Broadcasting in Public.” In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2573–2582. https://doi.org/10.1145/2702123.2702257. Reeves, Stuart, Christian Greiffenhagen, and Eric Laurier. 2017. “Video Gaming as Practical Accomplishment: Ethnomethodology, Conversation Analysis, and Play.” Topics in Cognitive Science 9 (2): 308–342. Routarinne, Sara, Liisa E. Tainio, and Matthew Burdelski (eds). 2020. “Human-to-Human Touch in Institutional Settings.” Social Interaction. Video-Based Studies of Human Sociality 3 (1). https://tidsskrift.dk/socialint eraction/issue/view/8754. Sacks, Harvey, and Emanuel A. Schegloff. 2002. “Home Position.” Gesture 2 (2): 133–146. Schegloff, Emanuel A. 1982. “Discourse as an Interactional Achievement: Some Uses of ‘uh huh’ and Other Things that Come Between Sentences.” In Analyzing Discourse: Text and Talk, ed. Deborah Tannen, 71–93. Washington, DC: Georgetown University Press. Sidnell, Jack. 2007. “‘Look’-Prefaced Turns in First and Second Position: Launching, Interceding and Redirecting Action.” Discourse Studies 9 (3): 387–408. Tumelius, Riikka, and Leena Kuure. 2021. “Towards a Shared Vision of Language, Language Learning, and a School Project in Emergence.” Classroom Discourse 12 (4): 344–364. Tuncer, Sylvaine, Christian Licoppe, and Pentti Haddington. 2019. “When Objects Become the Focus of Human Action and Activity: Object-Centred Sequences in Social Interaction.” Gesprächsforschung - Online-Zeitschrift Zur Verbalen Interaktion 20: 384–398. Ursi, Biagio, and Heike Baldauf-Quilliatre. 2021. “Toucher les écrans ça s’apprend ! L’étude multimodale des gestes réalisés par des enfants au cours de sessions de jeu sur grande tablette.” [Touching the Screen Is Something You Learn! A Multimodal Study of Children’s Gestures While Playing on a Large Screen]. Bulletin suisse de linguistique appliquée, Numéro spécial 2021 (1), 35–56.

8 Playing Together on a Large Screen

295

Weilenmann, Alexandra, and Gustav Lymer. 2014. “Incidental and Essential Objects in Interaction: Paper Documents in Journalistic Work.” In Interacting with Objects: Language, Materiality, and Social Activity, ed. Maurice Nevile, Pentti Haddington, Trine Heinemann, and Mirka Rauniomaa, 319–338. Amsterdam and Philadelphia: John Benjamins. Wiggins, Sally. 2002. “Talking with Your Mouth Full: Gustatory ‘Mmms’ and the Embodiment of Pleasure.” Research on Language and Social Interaction 35 (3): 311–336. Wildfeuer, Janina, and Du.šan Stamenkovi´c. 2020. “Multimodale Forschungsperspektiven auf Computerspiele.” In [Multimodal research perspectives on computer games] In Sprechende Pixel – Computerspielphilologie in Schule und Hochschule / Osnabrücker Beiträge zur Sprachtheorie (OBST) 96 , ed. Markus Engelns and Patrick Voßkamp, 7–28. Duisburg and Essen: Universitätsverlag Rhein-Ruhr. Wilkinson, Sue, and Celia Kitzinger. 2006. “Surprise as an Interactional Achievement: Reaction Tokens in Conversation.” Social Psychology Quarterly 69 (2): 150–182.

9 Getting (Others) Involved with Smartphones: Participation in Showing Sequences in Multiparty Settings Iuliia Avgustis

9.1

and Florence Oloff

Introduction: Smartphone-Based Showings as a Complex Activity

This chapter aims to explore some interactional details of a basic technology-related activity: looking at visual content such as photos, websites, or videos on a smartphone screen with others. Although such showing sequences have, by now, become an integral part of our daily routines, they remain under-investigated with respect to their role in understanding technology-based social practices. We will analyse how smartphone holders initiate and carry out a showing for more than one recipient, and how they manage contingencies linked to the affordances I. Avgustis (B) Faculty of Humanities, University of Oulu, Oulu, Finland e-mail: [email protected] F. Oloff Leibniz-Institut für Deutsche Sprache, Mannheim, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_9

297

298

I. Avgustis and F. Oloff

of the smartphone and multiparty settings. This chapter will thereby pursue three interrelated research aims: Firstly, it will provide a microanalytic study of joint smartphone use in face-to-face multiparty settings, which responds to the general lack of systematic multimodal analyses of everyday routinised technology use (Oloff 2019, 2021a). Secondly, our chapter will explore how participants exploit the specific affordances of smartphones as technological objects in showing sequences, thereby contributing to the recent interest in object-centred sequences (Tuncer et al. 2019). Thirdly, it will emphasise how participants both build on and reshape various interactional and technological affordances for accomplishing a fundamentally sequentially organised joint action, thereby demonstrating the complexity of mundane multiactivity and participation in multiparty settings (Haddington et al. 2022). While smartphones are primarily designed for individual use (as opposed to, e.g. large tablets, see Baldauf-Quilliatre and Ursi, this volume), they have also become a mundane tool for managing joint social activities. Smartphones allow users to easily share texts, pictures, gifs, music, and videos with distant (Chen et al. 2015; Ito 2005; Weilenmann et al. 2013) and co-present others. In face-to-face interaction, the sharing of audio-visual content can be a part of various technology-related activities, such as searching for information on the web (Brown et al. 2015; Suderland 2020), using map apps (Brown et al. 2013; Laurier et al. 2016), playing augmented reality (AR) games (Hellermann et al. 2017), or learning languages (Lilja and PiirainenMarsh 2019). These sharing-and-showing activities in co-presence are affected by various layers of complexity: the technological affordances of the mobile device, the multiactivity ensuing from their use, and the participants’ coordination within multiparty settings. Objects are interconnected with the social organisation of action, and their use is consequential to the progressivity of interaction (Hindmarsh and Heath 2000; Nevile et al. 2014). Technological objects such as smartphones have particular affordances (Hutchby 2001; Raudaskoski 2009) that can constrain how joint actions are performed, as their small displays are only accessible to people who are located close to the smartphone user (Sahlström et al. 2019). Screen-based technological devices

9 Getting (Others) Involved with Smartphones

299

also provide asymmetrical access to participants with regard to the intelligibility and consequentiality of the on-screen activity (Heath 1986: 165–173; Heath and Luff 2000; Nielsen 2016; for smartphones, see Mantere 2020; for large tablets, see Baldauf-Quilliatre and Ursi, this volume). While, for instance, voice-control provides “democratic” access to the device (Porcheron et al. 2017: 217), smartphones are mainly haptically controlled, with their size clearly favouring manipulation by single users. Although the device itself as an object is available to co-present participants, similar to a digital camera (Aaltonen et al. 2014), showable content on the device is not readily visible and has to be made relevant and visually accessible to the recipient (Licoppe and Tuncer 2019; Raclaw et al. 2016). Therefore, smartphone-based showings require particular “attention-organising practices” (Kidwell and Zimmerman 2007), which are coordinated through a series of multimodal action pairs (Oloff 2019). Moreover, digital content has a large variety of features concerning its type (text, images, audio/video-files), temporality (static or dynamic), and access (scroll, search, zoom and clickable), for which both the demonstration and the inspection have to be adapted (Brown et al. 2013; Oloff 2019; Raclaw et al. 2016). In face-to-face interaction, the specific affordances and “polymedia” nature of smartphones (Madianou and Miller 2012) meet another type of complexity: the inherent multiactivity related to their situated use—in settings that themselves are possibly already characterised by multiple concurrent activities. Multiactivity, defined as “multiple simultaneous courses of action that can be variously related” (Mondada 2012a: 226; see also, Haddington et al. 2014), has been mostly studied in professional contexts (Heath and Luff 1996; Goodwin 1996; Mondada 2011) or mobile settings (Keisanen et al. 2014; Laurier 2005; Mondada 2012a), but more rarely in immobile mundane settings (Goodwin 1984; Hofstetter and Robles, this volume; Katila, Goico, Gan and Goodwin, this volume; Mondada 2009). Mobile devices can induce multiactivity by autonomously producing a summons (DiDomenico and Boase 2013; DiDomenico et al. 2020; Haddington and Rauniomaa 2011) or by being individually seized and handled (Oloff 2021b). In both cases, the user

300

I. Avgustis and F. Oloff

has to coordinate their manipulation with the already ongoing encounter, including the current sequential and topical development of the talk, as well as with their co-participants’ various involvements (Goffman 1963) and displays of (dis)engagement (Goodwin 1981). Contrary to showings in video-mediated communication, which have to respond to different technological affordances (Licoppe and Tuncer 2019: 548; for Virtual Reality, see Spets, this volume), showings in face-to-face settings can make use of supplementary embodied means for carrying out a joint viewing. Smartphones—which usually do not themselves represent the showable but instead support a digital showable—can be moved, tilted or handed over to the recipient. Participants can also point at the display, read aloud from the screen or comment on the ongoing manipulation (Brown et al. 2013, 2015; DiDomenico et al. 2020; Oloff 2019; Porcheron et al. 2016; Raclaw et al. 2016; Suderland 2020; Weilenmann and Larsson 2002). Both before and during a showing, the smartphone holder thus has to coordinate the affordances of the device and of the digital showable, their own device manipulation, their recipient’s current attentional focus and availability, and the overall course of the ongoing interaction. Multiparty settings, when there are more than two co-present participants, add additional interactional challenges to the successful unfolding of a smartphone-based showing sequence. Having more participants implies a bigger variation with respect to potential next speakers and recipients (Lerner 2003; Oloff 2012; Schegloff 1995, 2000), as well as the possibility of schisming (Egbert 1993, 1997) and, thus, an increase in monitoring activities (Goodwin 1980; Sacks 1992, 1: 673). Nonselected “third” co-participants have been shown to regularly intervene on the basis of the preference for progressivity (Stivers and Robinson 2006), illustrated by phenomena such as choral turn productions (Lerner 2002) or co-tellership (Hayashi et al. 2002). Any co-participants, from any number upwards of three, therefore, mean both potential supplementary recipients and potential supplementary competitive courses of action to manage. From early on, interactional studies have understood participation as a continuous, joint, and embodied achievement

9 Getting (Others) Involved with Smartphones

301

(Goodwin and Goodwin 2004; Goodwin 1979, 1980, 1981). The multimodal dynamics of participation have been heavily investigated within storytellings (C. Goodwin 1984, 1986; M. H. Goodwin 1997; König and Oloff 2018a), usually with a focus on gaze (Holler and Kendrick 2015; Zima 2020; Zima et al. 2019). Even if the material setting and multiactivity play a pivotal role in how these conversational activities and participation frameworks are managed (König and Oloff 2018b), research on participation in mundane multiparty settings does not usually focus on “object-centred sequences” (Tuncer et al. 2019; but see, e.g., Baldauf-Quilliatre and Ursi, this volume). Other research has shown how participants use multimodal resources in settings with a main or sustained joint focus on a specific type of object or task, often within professional settings (Deppermann 2014; Goodwin 1994, 1996; McNeill 2005; Mondada 2007, 2014). Participants can attract their coparticipant’s attention to a specific object or referent through perceptual directives, noticings, reportings (e.g. during a car ride (Goodwin and Goodwin 2012); in parent-children interaction (Keel 2011); and in video calls (Licoppe and Tuncer 2019), as well as through action descriptions, deictics, pointing gestures or changes in posture during guided visits, for example (De Stefani 2010; Mondada 2013; Stukenbrock 2018; see also, Goodwin 1980, 2003). However, no research has yet looked into the specificity of object-centred sequences in mundane multiparty settings that are based on smartphone use. The following analyses will illustrate how multiple co-present participants in everyday settings can work towards a joint focus on a smartphone screen. Based on video recordings of everyday encounters in several languages (Sect. 9.2), we have examined a collection of showings initiated by the smartphone holder in encounters with three or more participants. We will distinguish showings initially designed for one recipient (Sect. 9.3.1) versus for two or more (Sect. 9.3.2). In the former, the showing is transformed into a showing for more than one participant as a result of a co-participant’s multimodal display of involvement in the already ongoing showing, while, in the latter case, the showing is designed for more than one recipient from the start. We will

302

I. Avgustis and F. Oloff

demonstrate how these two types of organising a showing sequence are connected to specific verbal and embodied action formats, and how the co-present participants jointly manage the situated socio-material contingencies. Our exploratory reflections on smartphone-based showings in multiparty settings aim to contribute to the systematic description of sharing visual digital content as a routine social practice in face-to-face encounters (Sect. 9.4).

9.2

Data and Method

The data for this article are drawn from various video recordings of naturally occurring interactions in multiparty settings, with participants being native speakers of Russian (four encounters, collected between 2018 and 2020), Czech (two encounters, collected in 2014 and 2016, respectively), French (one encounter, collected in 2015), and German (two encounters in 2014). The data were collected within the projects “Smart Communication” (Academy of Finland, 2019–2023, grant number 323848) and “The Epistemics of Grammar” (Swiss National Science Foundation, 2014–2016, grant number 148146). Participants were well acquainted with each other and were filmed during mundane socialising activities. All participants gave their consent to being recorded for research purposes and authorised the use of screenshots that were not anonymised. Their names as well as other types of personal information have been anonymised in the transcripts. The recordings were produced using stationary audio recording devices and cameras; for the Russian data, a 360-degree camera and wearable action cameras were used as well. In our data set, we found 25 instances of self-initiated smartphonebased showings (Russian: 12; French: 1; Czech: 5; German: 7), which are cases in which the smartphone holder is the first to manipulate the device and project a showing. Other-initiated showings, such as showings preceded by a co-participant’s explicit request or by a mobilegenerated summons, have been excluded. The collection was analysed using multimodal interaction analysis (Deppermann and Streeck 2018;

9 Getting (Others) Involved with Smartphones

303

Goodwin 1981, 2000; Mondada 2016; Streeck et al. 2011). To explore the variety of how smartphone-based showings are initiated and carried out, we have chosen excerpts that involve between three and six participants per encounter in different everyday private and public settings. We have used Jefferson’s (2004) system for the transcription of audible conduct and Mondada’s (2019) multimodal conventions for the annotation of embodied actions. The original talk in Russian (Russian Cyrillic alphabet) has been transliterated according to Bolden (2004), while other languages are presented unaltered with idiomatic translations being provided.

9.3

Analysis: Smartphone-Based Showings in Multiparty Settings

In social settings with two participants, the categories of shower and showee are a priori complementary. In encounters with more than two participants, more interactional work is required for initiating a smartphone-based showing: The smartphone holder has to choose whom they wish to address as recipient of this showing, take into account the displayed availability of the targeted recipient, and design their visible and audible actions accordingly. Co-present others, for their part, can seek to engage as recipients in a showing, regardless of them having been initially addressed as possible recipients or not. We will now analyse how showings initially designed for one can be transformed into a showing for multiple participants (Sect. 9.3.1), followed by how the joint inspection of a display can be initiated and carried out for more than one recipient at the same time (Sect. 9.3.2).

304

I. Avgustis and F. Oloff

9.3.1 Transforming a Showing for One Recipient into a Showing for More Than One The initial formatting of a showing for a single recipient can be expressed in the first turn that projects the showing (e.g. by choosing singular verb forms). More frequently, however, a showing-for-one becomes visible as such as soon as the shower’s body and the mobile device are oriented to a specific co-participant, selecting them as the (momentary) single recipient of the showing. The transformation—either carried out as a second showing (ex. 1) or as an extension of the initial showing (ex. 2)—is usually responsive to a co-participant’s conduct. Current non-recipients can get involved in the showing by directly addressing the smartphone holder, and/or by moving their gaze, head or body closer to the mobile device and the shower. In the first excerpt (1a–c), a second showing is initiated due to a current non-recipient’s embodied conduct. Patrick (PAT), Markus (MAR), and Andi (AND) have talked about different car makes for a few minutes already. Andi had previously asked Patrick about the name of a specific older lorry model from East Germany. Patrick had then suggested the models W50 or L60 and announced his intention to look these up. One minute prior to the excerpt, he had taken his smartphone out of his pocket and self-initiated a web search. Since then, Patrick has been looking at his smartphone display while continuously talking with the others, now about different tramway models. At the beginning of the excerpt, this tramway sequence will be closed by Patrick finally showing the result of his web search to Andi. During this first showing, Markus will move closer to his co-participants, and Patrick will carry out a second showing for him.

9 Getting (Others) Involved with Smartphones

Excerpt 1a (PhPiz_001530_W50_German_part1) 01 PAT

das

pat 02 03 PAT

>>looks SP-------+...MAR->l.08 (0.6) nachwende; -Wende (.) ja? really? (.) [nee:; ] [no ] ich-] [+ [well I] >+..looks AND-> (0.4) (0.4)+*

04 05 MAR 06 07 AND 08 PAT pat 09 10 PAT pat mar and 11 PAT pat mar and fig 12 AND 13 PAT pat mar 14 AND and fig 15 and mar

is no:ch

+

ger;

>looks AND------+ >looks PAT------* +*ich glaub nich dass

die *#vorwende sind; -Wende +...looks SP------------------>l.24 *...looks AND--------------*,,,gaze down-> >looks in front-------> #fig.1 [really? ] [+hier, ] *(0.2) das is-= -= +...moves SP twd AND-> >looks down*...looks to the right-> ja vor der-# =na ich war =well I was actually (there) before the->l.21 #fig.2 (0.3--)*(0.5) -> >t/right*...looks AND->

305

306

Figure 1

I. Avgustis and F. Oloff

Figure 2

While Patrick is reformulating his opinion about the time period a specific tram type—previously mentioned by Andi—has been in service, he stops looking at his smartphone (SP in the multimodal transcript) and gazes up to Markus (l.01), then, a bit later, to Andi (l.08). As both Markus and Andi resist agreeing with him (see l.05, 07, 12; see also their brief mutual gaze at l.11, fig. 1), Patrick formulates his counterposition a last time and simultaneously looks back at his smartphone, on which the results of his previous online search seem to be already visible (l.11, fig. 1). Without pausing, he starts moving his smartphone up towards Andi. He initiates a showing sequence with the deictic hier ‘here’ (cf. Stukenbrock 2018) and an incomplete presentative structure (das is ‘that’s’, l.13). This minimal verbal framing of the upcoming showing indicates that Patrick is linking back to a prior topic: Andi’s previous enquiry about different lorry types and Patrick’s own offer to look these up. This showing is, therefore, exclusively designed for Andi, as can be also seen in the orientation of the display chosen by Patrick (l.14, fig. 2). Andi, who, during the initiation of this action, is looking in front of him and seeks to continue the previous sequence (l.14), now reorients his gaze towards the smartphone and suspends his turn shortly afterwards. He moves his head closer to the display (l.15–16, fig. 3), breathes in and starts lifting his left hand (l.16) in order to simultaneously point at the display and comment on what he is seeing (l.18–19, fig. 4).

9 Getting (Others) Involved with Smartphones

307

Excerpt 1b (PhPiz_001530_W50_German_part2) 16 AND and mar fig 17 mar 18 AND and mar mar 19 AND 20 21 PAT mar mar and and fig 22

Figure 3

.h(f)*:

--> >AND---*..PAT/AND-> #fig.3 (0.4)*(0.2) *...shifts weight to left leg-> genau *dIE: mein ich. (.) exactly these (are what) I mean (.) ->PAT/AND->l.21 >...one step to the left-> de- den hier; ja, thi- this (one) here yes (0.3) *#mhm:; _mh, *.looks SP->l.28 *shifts weight t/left leg* >looks SP--#fig.4 (0.6)

Figure 4

Markus had already opted out of the previous sequence (about the trams): Before Patrick started moving the smartphone towards Andi, Markus had already withdrawn his gaze from his co-participants (l.11) and had begun looking to the right, towards other participants (l.13, fig. 2). Consequently, Markus misses the initiation of the showing (l.13, fig. 2) and finds himself excluded from it once he turns his head and gaze back to Andi and Patrick (l.15–16, fig. 3). He gradually moves closer

308

I. Avgustis and F. Oloff

towards the dyadic formation: He first shifts his weight to the left leg (l.17) and takes a lateral step towards his co-participants (l.18, figs. 3–4). He shortly glances at Patrick’s face (l.18), but, as Patrick is still focused on the display, Markus shifts his gaze to the smartphone as well (l.21, fig. 4). He prepares, then simultaneously takes another step, and slightly bends to the left (l.22, figs. 4–5). In the meantime, however, the showing sequence for Andi has come to a close as Patrick is already withdrawing the phone (l.22). Also, Andi is formulating a general closing assessment while looking at Patrick’s face (geil, ‘awesome’, l.23, fig. 5). Although Patrick is continuously glancing at his phone’s display, he seems to have perceived Markus’ movement in his direction, as he stops retracting the phone and then moves it towards Markus (l.23–24, figs. 5–6).

9 Getting (Others) Involved with Smartphones

Excerpt 1c (PhPiz_001530_W50_German_part3) 22

mar pat 23 AND 24 PAT pat pat pat and and fig

Figure 5 25 26 MAR 27 AND and mar pat fig 28 29 PAT and mar

*+(0.6) *..one step to t/left, leans t/SP-> +,,,retracts SP .h:+#: ge[il. ] awe[some ] [yes ]

>,,,+...SP to MAR--------------------> >looks SP+..MAR----+...looks SP----->> +...leans t/left--> ----------#fig.5

#fig.6

Figure 6 (0.3) #MHM:._mhm:, (.) gen[au.] exact[ly ] [.h:] (.) so_n ding ma:l* [.h:] (.)(driving) such a thing once >leans fwd------> >looks SP-------------------------------------* >SP up----------+,,, #fig.7 * :. (0.2) durch [a- (.) durch ] afrika, er: (0.2) across [a- (.) across] africa, [mhm:._mh, ] >...steps fwd to table->> *...looks PAT----->

309

310

I. Avgustis and F. Oloff

30

(0.7)

31 AND

.h::: [aber die sind wahrscheinlich-] .h::: [but they are probably] [.ts mit dem darfst du sogar] fahren; [.ts you are even allowed to] drive this'

32 MAR

Figure 7

Rather than being a simple extension of the previous showing (for Andi), this is a second showing designed for a new recipient: Patrick shortly glances at Markus’ face and extends his arm so that the display meets Markus’ body movement halfway (fig. 6). He also reformulates an announcement of what can be seen on screen, with the more explicit formulation of the object (the W50 lorry, l.24) being formatted for an unknowing recipient. Markus further adjusts his body and head orientation to the incoming display (figs. 6–7), then acknowledges the object on screen (l.26). Andi, despite having already seen and assessed the picture of the car, concurrently reorients his gaze and body back to the smartphone (l.24–26). By positioning his head next to Markus’ (fig. 7), Andi is visibly looking at the display together with him. The orientation to a second showing and joint inspection is further evidenced by Patrick’s and Andi’s step-by-step withdrawal. They both start withdrawing only when Markus has audibly responded to the showing (MHM:;_mhm:, l.26), and before Markus orients himself to a sequence closing with a genau, meaning ‘exactly’. All participants will link their following talk to what they have been jointly looking at (l.27–32).

9 Getting (Others) Involved with Smartphones

311

In excerpt 1, the initial non-recipient, Markus, triggers a second showing sequence by moving closer to his co-participants and visibly leaning his body and head towards the device. His torqued posture and continuous looking at the device (fig. 5) function as a request for access to the screen, as Patrick quickly transforms the retraction of the phone into a new showing. During the first showing, Patrick is bending sidewards to his recipient, Andi, so that their heads are approximately at the same height, allowing them to simultaneously look at the display (fig. 4). A similar side-by-side positioning of participants’ heads can be observed during the second showing between Andi and Markus (fig. 7). By looking at the display a second time, Andi clearly displays his orientation towards a joint examination rather than participating in the second showing as a bystander (Goffman 1981). Moreover, the smartphone holder can adjust their body orientation either with respect to an explicit “joint looking” (here, during the first showing), or with respect to a more explicit “showing posture” (here, during the second showing). In both cases, however, the smartphone holder is gazing at the display as well (cf. Patrick’s continuous gaze on the display from l.11 until the end of the excerpt, with a short interruption in l.24): Similar to the Fformation (Kendon 1990)—but with a specific joint orientation of the upper bodies towards a display—this constellation could be termed a showing F -formation (cf. the “instrumental” F-formation suggested by McNeill 2005, 13). In the second excerpt (2a–b), four participants will become the recipients of a showing, this time by gradually gathering around the mobile device. Elizaveta (ELI), Petr (PET), and Nikolai (NIK) are sitting on the sofa (fig. 8), Kira (KIR) and Olga (OLG) are sitting on the opposite side of the table (not visible in most of the selected figures). While this spatial arrangement of the participants is not complex per se, it is potentially challenging for both providing and obtaining access to the small display on a smartphone. Before the excerpt begins, Petr has been telling a story about two men who were injured by an airgun. Petr now

312

I. Avgustis and F. Oloff

adds another juicy fact: The sister of one of the protagonists is a porn actress (l.01–04). In the beginning of this turn, he uses the plural form of the imperative ‘look’ (smotrite) in order to attract his co-participants’ attention (cf. Oloff 2019 for a similar use of this imperative in German; and Mondada 2012b). Petr further projects a showing by pointing at the phone display (l.01, fig. 8), then he begins to search for a video of the actress (l.03, fig. 9). Despite the projected showing being possibly designed for several recipients, and three of them having responded to Petr’s last turn, (l.07–10, 12), a sort of topical digression is introduced by Nikolai, who is holding a book and complaining about the impossibility of automated searches in physical books (l.11–13, figs. 8–9). Due to this schisming, the showing will first be carried out for one participant only (Nikolai), and then the others will join one after another.

9 Getting (Others) Involved with Smartphones

Excerpt 2a (190526_005522_Actress_Russian_part1)

Figure 8 01 PET pet fig 02 03 pet fig 04 05 06 PET 07 NIK 08 KIR 09 ELI 10 KIR 11 NIK 12 KIR 13 NIK 14 KIR 15 NIK

Figure 9 smotrite +a vto- vtoroj#chuvak look(-Pl) and the se- second dude +points at screen---> #fig.8 kotoromu otstrelili jajtso travmatom, who got shot in the balls by an airgun, u nego .hh +e:: sestra:# his .hh e::r siste:r -->+searches for video----> #fig.9 izvestnaja porno aktrisa Lulu Foks. is the famous porn actress Lulu Fox. (0.8) iz Penzy.= from = =nicho sebe.= =no way.= =ob etom vse [ves' nash tsentr ob etom ]& =about this all [all our centre about this]& [iz Pen(h)z(h)y hha hn ] [from Pen(h)z(h)a hha hn ] &ves' nash tsentr znaet,= &all our centre knows,= =et zha[lko zdes' net e]togo: m& =it is sa[d there is not t]hi:s uhm& [teper' blagoDaria Pete.] [now thanks to Petr.] &>bystrogo opredelenija znaete po slovu< &>quick search you know by word< [mgm. ] [mhm. ] [slovo a tam ] [flesher, ] [a word and there is] [a flasher, ]

313

314

I. Avgustis and F. Oloff

16 PET 17

(0.6)

Figure 10 18 PET 19 ELI nik fig 20 21 PET pet nik 22 PET pet nik fig

[vot ona. ] [here she is.]

Figure 11

!

[budete?# ] [will you(-Pl)?] [a:: ]%:h Kol',# [o:: ] :h Kol'(NIK), %looks SP--> #fig.10 #fig.11 zajdi v elektronn[uju versiju etogo:::: ]= go to the electro[nic version of thi::::s]= [+budesh' Lulu Foks iz Penzy% [will you(-Sg) Lulu Fox from Penza -->+...turns head to NIK...> >looks SP------------------------------------% %smotret'?#]= watch ]= >moves SP closer to NIK-> %,,,looks away from SP--> #fig.12

Figure 12

While Petr is still looking for the video clip, Nikolai continues flicking through the book and formulates a complaint about lacking a “quick word search” (l.11, 13, 15). As Nikolai finishes his multi-unit turn, Petr has found a list of videos of the actress in question and announces it

9 Getting (Others) Involved with Smartphones

315

by saying vot ona, ‘here she is’ (l.16). Petr then starts a new turn by asking “will you (watch)” (l.18), which is addressed to multiple participants (budete being the second-person plural), but his turn is overlapped by Elizaveta’s answer to Nikolai’s previous turn (l.19–20, 23, 25, 27– 28). Note that Petr is not looking up from his phone and therefore does not select specific recipients through gaze (1.18, fig. 10). Except for Nikolai—who now starts looking at Petr’s phone (l.19, fig. 11)— the others do not respond to Petr’s attempt to initiate a showing sequence. Because of the concurrent line of talk and a missing display of interest from other participants, the showing—that initially seems to have been addressed to all co-participants—is now, more specifically, being addressed to one recipient by means of morphosyntax and gaze: When Petr reinitiates his previously abandoned turn (l.21–22), he looks at Nikolai, uses the second-person singular of the verb ‘be’, budesh (fig. 12), and adds a description of the showable. Either due to his involvement in the sequence with Elizaveta (l.20, 23) or due to the possibly delicate nature of the visual content (Goffman 1955), Nikolai, however, withdraws his gaze from Petr’s smartphone again (l.22, fig. 12) and even suggests watching the video later (l.26). In the absence of his selected recipient’s visual attention, Petr now looks in Kira’s direction while laughing (l.24), orienting to a co-participant who is currently available as a potential recipient.

316

I. Avgustis and F. Oloff

Excerpt 2b (190526_005522_Actress_Russian_part2) 23 ELI 24 PET pet 25 ELI 26 NIK pet 27 ELI 28 ELI 29 NIK 30 PET pet fig

=etoj [knigi i:: ] =this [book a::nd ] [+KHI hihi+hihi] [ KHI hihihihi ] -->+looks SP+looks KIR---> [u tebja budet >po poiskoviku

] budesh' iskat'= you will search= [davaj +po- popozzhe chut' chut'.] [let's (do it) a bit lalater.] -->+looks SP--> =zajdi v elektronnuju versiju =go to the electronic version kni[gi #fig.13

Figure 13 31 nik pet eli fig

%poxozha ona# na penzjachku?# does she look like a resident of Penza? %looks SP-->> -->+plays video------> ------> #fig.14 #fig.15

9 Getting (Others) Involved with Smartphones

Figure 14

317

Figure 15

Petr’s attempt to involve Kira as an alternative recipient fails, as no response is forthcoming from the other side of the table. Therefore, Petr again slightly leans towards Nikolai (figs. 13–14), starts the video, and uses the singular imperative form smotri, meaning ‘look’, in order to engage Nikolai in the showing (l.30). This second personalised initiation of the showing is met with success, as Nikolai turns his gaze back to the smartphone (1.31, fig. 14). Notice also that this initiation is precision-timed with the closing of the competitive sequence, in which Nikolai acknowledges Elizaveta’s previous instruction about e-books (l. 29), as well as the resolution of the schisming. Consequently, Petr’s following polar question about the actress’s appearance (l.31, while simultaneously starting the video) successfully secures Nikolai’s and the others’ attention. Elizaveta begins to lean towards Petr and the display, and the three participants now form a recognisable showing F-formation (fig. 15). As Nikolai and Elizaveta are providing first responses to the video clip (l.33, 36, 38, 40), also displaying their involvement verbally, Kira requests visual access to the screen (l.35).

318

I. Avgustis and F. Oloff

Excerpt 2c (190526_005522_Actress_Russian_part3) 32 33 NIK

(0.9)

34 35 KIR

(0.2) dajte ja tozhe posmotrju.= let me also watch.= =nu nu ja zh, =well well I am, (0.2) [slushaj nu vse shljuxi na shljux poxo]zhi, [listen well all whores are simi]lar, [nam s Olej to[zhe interesno] UHUH huhuh] [me and Olga are al[so interested] UHUH huhuh] [ne jasno ] [not clear ] --> -kak by [kakaja ] raznitsa . so [what's the ] difference. [(govorju), ] [(I'm saying),] ---> (0.3) da net, smotri. well no, look(-Sg). (0.2) net zhe sm- >da net ne mozhet byt' iz Penzy< no loo- >well no she can't be from Penza< (0.9) vs pojd mte smotret'.* &nu that's it let's go watch. &stands up and goes--> *stands up and goes-> (2.2)&(0.9) (0.2) -->&............---> -->> -+#nu sudja*& po +govo*ru-to# da ha hah well judging from her dialect yes ha hah +moves SP down--+ ---->*..........*leans on the sofa-->> -->&lies on the sofa-->> #fig.16 #fig.17

36 NIK 37 38 ELI 39 KIR 40 NIK eli 41 ELI 42 NIK eli 43 44 NIK 45 46 NIK 47 48 KIR kir olg 49

kir eli 50 ELI pet olg kir fig

9 Getting (Others) Involved with Smartphones

Figure 16

Figure 17

319

'

As Kira’s first directive (‘let me also watch’, l.35) does not obtain any response, she upgrades her request by stating that both her and Olga are also interested (l.39). This still does not lead to Petr sharing the screen with her or Olga, as Elizaveta and Nikolai are still looking at the screen and responding to the showing (l.38, 40–46). This extended commenting activity by the already-established recipients and the continuous non-responses to further potential recipients can be linked to the specific nature of the concerned showable, which is a video file. A dynamic visual object requires watching it fully or at least partially in order to be able to comment on it. Indeed, it is only in l.46 that Nikolai actually responds to Petr’s initial question (l.31). As a consequence, Petr could not have complied with Kira’s request without interrupting the already ongoing showing. With the previous verbal requests having failed, Kira stands up while simultaneously inviting Olga to do the same (‘that’s it, let’s go and watch’, l. 48). Olga complies as soon as Kira has finished her turn. Petr moves the phone to a lower position and Elizaveta leans back (l.49–50, figs. 16–17), bodily adjusting to the arrival of the two new showees. Kira and Olga, however, engage differently in the showing: While Kira stretches out on top of the sofa, close to Petr’s head, in order to get a good visual access (figs. 16–17, l.50), Olga leans in from the side (fig. 17), adopting a more unstable and distant position (Schegloff 1998). The time span of their recipientship differs accordingly, as Kira stays in this position until the showing sequence is finished, while Olga returns to her place shortly after having joined. Excerpt 2 illustrates the different challenges both showers and showees face in more complex multiparty settings, as the potential recipients can more easily engage in diverging courses of action. Indeed, the showing fully unfolds only when the competitive course of action has

320

I. Avgustis and F. Oloff

come to a close. Here, Petr accumulates response-mobilising features (Stivers and Rossano 2010) by exploiting various action formats (imperatives, announcements, requests, polar questions) and embodied resources (pointing, gaze). He also adopts the strategy of focusing on a single recipient first (Nikolai), then adjusts the phone and his body’s position when other co-participants join. A video file as a showable is both possibly challenging—as it requires longer attention from the recipients—and convenient—as new recipients can join during a longer time span. Consequently, the showing of a video does not necessarily have to be carried out for a second time for “incoming” recipients. The video file’s temporality is also a double-edged feature for the recipients, as has been shown by Kira’s repeated attempts to obtain access to the display. The fact that she got up (instead of waiting for a first showing to be finished) illustrates both an orientation to the fleeting nature of the showable and to a preference for a joint looking (cf. ex. 1). While both excerpts in this section illustrate how a showing for one can be extended to more recipients, they show a diversity of practices related, among others, to the following aspects: • The showing can be introduced with a presenting gestalt (ex. 1), coordinating a presentative construction and the presentation of the display, thereby linking back to the previously established relevance of the showable; or it can be introduced with a complex description coupled with the imperative of a perception verb (ex. 2), simultaneously establishing the relevance of the showable and of an upcoming showing. • The co-participants’ already ongoing joint (ex. 1) or competitive courses of action (ex. 2) can delay the beginning of a showing. • The co-participants’ positions with respect to the shower enable different ways to engage, each requiring more or less effort and time (shifting gaze, leaning closer, getting up, taking a step). Vice versa, the shower needs different embodied practices and temporalities to adjust the position of the display and their body to a designed or emergent recipient.

9 Getting (Others) Involved with Smartphones

321

• Finally, the affordance of the digital showable (static versus dynamic) can facilitate repeated (ex. 1) or continuous showings (ex. 2; cf. Mondada 2012b). These different complexities (multiple affordances of the device and the showable, multiactivity, multiparty interaction) lead to a considerable variation in the verbal and embodied action formats used during a selfinitiated showing. This illustrates that smartphone-based joint activities do not consist in independently or newly shaped (e.g. by technology) actions, but unfold as specific object-centred sequences that are deeply embedded in the respective ecology of a given setting. Despite the differences, these examples both demonstrate the response relevance set up by the introduction of digital showables and the participants’ preference for their joint inspection and assessment.

9.3.2 Showings Designed for More Than One Recipient In this section, we will turn to showing sequences that are initially designed for more than one recipient. This design for more than one recipient can be traced back to the grammatical format of a possible announcement, such as using an imperative form or a pronoun in the second-person plural (cf. ex. 3), to gaze orientation and/or to the moving of the phone display to several participants. The display can be immediately oriented to more than one participant (ex. 3) or be shown to several participants in a row by means of a “sweep” (ex. 4). Excerpt 4 will also illustrate how the timing of the responses to the initiation of a showing is consequential for the order in which the display will be shown. Biagio (BIA) has invited his two fellow students Stefan (STE) and Sandra (SAN) for dinner. Prior to excerpt 3, Biagio has questioned Stefan about his studies, his family and home. Stefan has just been talking about his spouse and now mentions their daughter for the first time (l.02). Biagio incrementally builds on this turn by adding more information about the daughter (Stoenica and Pekarek Doehler 2020), thereby displaying his previous knowledge (l.06, 11). At first sight, Stefan’s

322

I. Avgustis and F. Oloff

response to Biagio’s topic-proffering requests for confirmation seems to be rather minimal and delayed. At the end of the first proffer, however, Stefan starts reaching into his left trouser pocket (l.06, cf. Figure 18). While delivering a more consistent response, he takes out his smartphone, thus preparing a showing that he will explicitly announce in a subsequent turn (l. 21). Both the announcement (the use of the secondperson plural pronoun vous ‘you’) and the alternating gaze to Biagio and Sandra (l.13) indicate that Stefan is formatting the upcoming showing for more than one recipient.

9 Getting (Others) Involved with Smartphones

Excerpt 3a (RepAr_000950_Katie_French_Part1) 01 02 STE 03 04 BIA 05 06 BIA bia ste ste 07 ste 08 STE 09 10 STE 11 BIA 12

Figure 18

(0.4) qui est la mère de ma (.) deuxième fille. who is the mother of my (.) second daughter (0.6) okay (0.8) +.h qui *est encore très jeu*ne. .h who is still very young +....looks STE--------------------> >>head t/BIA*..looks BIA----------> *...-> (.) >..left Hand to pocket...->l.13 oui:, yes (0.3) [>oui oui oui< ] [yes yes yes ] [seulement quel]ques mois. [only a f]ew months (0.3)

323

324

I. Avgustis and F. Oloff

13 STE ste ste san fig 14 STE ste ste fig 15 bia 16 STE 17 BIA 18 SAN ste san bia

Figure 19

.h (.) :# mai:nes_#euh:* .h (.) er eleven er eleven weeks er >-----*...takes SP out, seizes w/both hands-* >looks BIA-----------------*...looks SAN----* STE--------------------------> >BIA-#fig.18 #fig.19 * mercredi. #*demain. on wednesday tomorrow *--holds SP--*taps/unlocks *,,looks SP->l.32 #fig.20 +(0.4) +...looks SAN-> *el[le va] fai[re. sh[e will] b[e [right ]

:. + [ oh:: wow *starts tapping on SP-->l.26 >looks STE--------->looks SAN-------------------+,,,

Figure 20

While Stefan is specifying his daughter’s age (l.13–14), he takes his smartphone out of his pocket, positions it in his hands (figs. 18–19), and unlocks the screen with his right thumb (fig. 20). These preparative steps are precisely timed with respect to the incremental development of his turn (‘eleven weeks’, ‘on Wednesday’, ‘tomorrow’). His concurrent gaze, first to Biagio, then to Sandra and finally to his phone, illustrates that these bits of information are designed for both a knowing (Biagio) and an unknowing (Sandra) recipient (Goodwin 1979) and, at the same time, project the relevance of his phone for an upcoming action. He then starts tapping on the display (l.16). Biagio’s confirming response claims independent knowledge about the daughter’s age (voilà ‘right’, l.17; see also his previously initiated gaze to Sandra, l.15), while Sandra treats it as news (‘oh wow’, l.18; Heritage

9 Getting (Others) Involved with Smartphones

325

1984). Despite their different epistemic status, Stefan now explicitly announces a showing for both of them, using a plural pronoun (‘I will show you the latest pictures’, l. 21, 25). Excerpt 3b (RepAr_000950_Katie_French_Part2) 19 20 BIA 21 STE bia san fig 22

san bia 23 STE 24 25 STE 26

bia ste san 27 STE ste bia san 28

ste bia san 29 STE ste fig

Figure 21

(.) [+

vous mon# + [ yes ] I will show you(-PL) +..looks SP STE-+..looks STE-----+looks down/SP-> >looks down--------> #fig.21 (0.6) (1.2-)+(1.6) (0.3) ----> >STE>looks SP/STE+,,looks down->l.26 .h:: (0.9) les derniè:res (0.5) photos, the latest (0.5) pictures +(0.8)*(0.4-----) (0.4) +...looks SP_STE------> >-----*last tap, positions SP in rHand-> -> >gaze alternates+ tie, elle +*ka+tie. >left thumb hovers a/display---*taps +..turns head to STE-----------+STE+..SP-> ----> >looks SP*(0.2)+(0.2) (1.1) *...turns SP slightly t/right-> +...smiles-->> -> >looks STE-* *# *taps *taps #fig.22

Figure 22

326

I. Avgustis and F. Oloff

30 ste

31 BIA ste san 32 ste ste fig 33 BIA bia san 34 ste fig

Figure 23

*(0.3) *...->

>moves SP/display twd SAN->> moves head/torso fwd-> *(0.2------)*#(0.4) *..looks SAN*,,SP-----> *..smiles-> #fig.23 +..head follows SP_STE -> (.)*# >--* #fig.24

Figure 24

!

Both Biagio and Sandra have kept looking at Stefan and thus displayed their involvement (figs. 18–21). While Stefan is retrieving his daughter’s pictures, he continuously looks down at his phone, but keeps on adding informative pieces about what will be seen, such as ‘pictures’ (l.25) and his daughter’s name (l.27), and about the possible technical problem he seems to encounter (l.29), thereby securing his recipients’ attention. Both recipients keep on monitoring, their gaze alternating between a middistance gaze, Stefan and his phone (fig. 22). Finally, Stefan seems to have found a photo he wishes to show, as he then starts moving the phone in Sandra’s direction (l.30–34), also briefly looking at her (fig. 23). Shortly afterwards, Sandra starts bending forwards (l.31–34, figs. 23–24). While Stefan seems to initiate the showing more explicitly for Sandra, one can notice that the screen is turned halfway to Biagio as well. Biagio aligns to Stefan’s projected showing early by producing several laughter particles (l.31, 33), continuously smiling and following the movement of the phone with his head (figs. 23–24) so that all participants are now jointly looking at the photo (fig. 24).

9 Getting (Others) Involved with Smartphones

327

While the recipients’ consecutive responses to the showable cannot be shown for reasons of space, this excerpt emphasises how the initiation of a showing is entwined with the material affordances: Here, the showable is not yet available upon its announcement and its location on the phone requires a visual disengagement with the recipients. Stefan handles this dual involvement by first securing his recipients’ gaze, then producing bits of information about the upcoming showable and the retrieval process so as to constantly address them. Thus, despite the delay, the showing proper needs no further preparation (l.31–32). Although the monitoring of a recipient’s focus of attention can be done sequentially only (here, Stefan is starting with Sandra), the smartphone can be held in a way so as to simultaneously address more than one showee from the start, exploiting both the mobile device and the participants’ spatial arrangement for framing the showing as a joint activity. In excerpt 4, the verbal announcement is not explicitly formatted for more than one recipient (cf. ex. 3). Instead, it is designed in a way that offers an opportunity to all co-participants to get involved in the showing. Also, the ensuing showing is carried out in a clearly visible order, as the visual content is shown in a sweeping movement, first to one, then to the other recipient. Tina (TIN), Dana (DAN), and Mihail (MIH) are sitting in a café. Prior to the excerpt, Dana has been telling a story and Tina has been concurrently preparing her phone (lying on the table in front of her) for taking a picture. No one has responded to Dana’s story and, during the ensuing lapse, Tina takes a selfie from below (l.01, fig. 25). She then initiates a showing sequence by formulating an assessment of this picture (‘this will be the best photo’, l.02).

328

I. Avgustis and F. Oloff

Excerpt 4 (181222_014456_Nostrils1_Russian)

Figure 25 01

tin tin dan mih fig 02 TIN dan tin

03

mih tin tin dan dan fig

%#(0.3-----)% >>looks SP-> %takes a pic%leans back-> >>looks table-> >>looks table-> #fig.25 eto budet ±luch%shaja fotogra%fija s-, this will be the best photo l-, ->±...looks TIN-> ->%.............%lifts SP and leans towards others-> *(0.6)#(0.1)@(0.2)%±(0.3)%+#(1.0)@ ->*looks SP->@looks DAN-----------@ ->%......%SP DAN-> ->±looks SP------> +...leans t/SP-> #fig.26 #fig.27

9 Getting (Others) Involved with Smartphones

Figure 26 04

mih tin tin dan dan fig 05 MIH mih tin dan fig

Figure 27 @(0.3) %(0.2)%#(0.3)@%(0.1) >-looks SP--> @looks MIH----------@looks SP> >SP DAN%.....%SP MIH%....-> >--looks SP-> >leans t/SP-> #fig.28 .ts e::gh, .ts e::w,

tin

->%phone herself%,,lowers SP, puts it on table-> ->±....-> #fig.29 (0.2) ± (1.2) % (0.2) * (0.8) ± (0.6) ->±looks away-------------±looks table-> ->%turns on the camera and poses-> ->*looks away-> ja predlagaju ±posle togo kak my schas osvobodimsja, I suggest after we get free now, ->±looks MIH---> (0.6) pojti:: (0.3) @ mozhet byt' po Etazham.@ (0.6) to go:: (0.3) maybe to ((place name)) ->@looks up (to pose)-------------@

Figure 28

Figure 29

06

dan tin mih 07 MIH dan 08

329

330

I. Avgustis and F. Oloff

When Tina positively assesses her selfie (l.02), she adds a final hearable s-, which could be the beginning of the imperative ‘look’ (either smotri — second-person singular—or smotrite—second-person plural), but this lexical item is left incomplete. Tina mentions the type of showable ( fotografija, ‘picture’, and the phone’s shutter sound could be previously heard, l.01), but she does not specify what is to be seen on screen, that a showing has been projected, and for how many recipients (cf. ex. 3, l.21). Moreover, Tina looks down, not selecting any potential recipient through gaze. The announcement is thus unspecific, both with respect to the projected next action and its possible recipients. As a result, neither Dana nor Mihail provide an audible response to Tina’s turn (l.03–04). Dana, however, has started looking at Tina in the middle of her assessment (l.02). She can thus perceive when Tina begins to lift her smartphone and change her grip, putting it into her left hand (fig. 26). Mihail starts gazing at Tina’s smartphone shortly afterwards (l.03, fig. 26). When Tina is lifting the phone, she first looks at Dana, who sits further from her, and orients the screen of the phone visibly to her (fig. 27). At the same time, Dana leans closer to the phone, then smiles and nods slightly in response to the picture. After showing the picture to Dana for more than one second, Tina turns her gaze towards Mihail, who is already looking at the screen of the smartphone. Tina starts turning the phone towards him 0.3 seconds later, making a “sweep” (l.03–04, figs. 27–28), and holds her phone towards him for a shorter time (0.3 seconds). This might be explained by the fact that before turning the screen towards him, she can notice that he is already looking at the screen, and that he, therefore, already had time to inspect the picture. Even though Mihail had started looking at Tina’s smartphone just after she completed her initial assessment, he provides his (non-lexical and possibly joking) assessment only after the picture is fully shown to him (l.05, fig. 29). This illustrates that gazing at a display, inspecting visual content on the display and assessing it are distinct and sequentially organised actions, which respond both to perceptual needs and to their formatting as responsive actions (cf. Oloff 2019). Both recipients have provided a response to the showing and the showable, following the order of the screen demonstration. Tina then turns the display back towards herself (l.05), which indicates that long assessments and expanded topic

9 Getting (Others) Involved with Smartphones

331

talk are not expected. She puts the phone back on the table and adopts a new selfie pose, and Dana and Mihail subsequently withdraw their gaze (l.06). Mihail then initiates a new sequence and topic, making a suggestion for where the participants could go next (l.07–08). Here, the announcement of the showing is unspecific with respect to one or more possible recipients. This under-specification is, however, not problematic, as the showing sequence is initiated during a lapse, at a moment in which all co-participants are potentially available to respond to it. The order in which Tina shows the picture seems to be related to her perception of her co-participants’ embodied responses, that is, the display is turned towards the first co-participant she perceives looking at the screen, which is first Dana and then Mihail. Although it can be assumed that the recipients’ verbal and embodied responses to the announced showing are of prime importance for the direction of such a sweeping movement, the exact factors affecting it have to be studied more thoroughly. The two excerpts of this section illustrate that a showing for more than one recipient can be introduced in at least two different ways: • The verbal announcement can explicitly address several recipients by means of plural forms of pronouns and by building on a previously established recipiency through topical talk, for instance (ex. 3). Alternatively, the announcement can take on the form of an “unspecific” noticing (such as an assessment) that newly introduces a showable, but without specifying a particular recipient or number of recipients (ex. 4). • Accordingly, the device can be oriented to both showees simultaneously from the start (ex. 3) or adjusted in its orientation to more than one showee later, by means of a sweeping movement, for example (ex. 4). • The way the showable is linked to a previous joint course of action (ex. 3) or not (ex. 4) also impacts how the shower prepares the joint entry into the showing. If the showable is not yet available during its announcement, the shower has to retrieve it while simultaneously accounting for the device manipulation and managing their recipients’ attention (ex. 3). If the showable is already available upon initiation,

332

I. Avgustis and F. Oloff

the shower can proceed with the showing without further delay and verbal framing (ex. 4). While these excerpts show, again, the variability of how these specific object-centred sequences can unfold in a multiparty setting (Sect. 9.3.1), they also allow us to formulate some more general observations regarding the interactive management of several showees at a time. The small size of the device (and thus of the showable) is consequential to the way a showing for several can be carried out: Even if the participants are sitting close to each other, both their bodies and the device have to be brought closer. The shower is, therefore, involved in interrelated multiactivity (Mondada 2014), as they constantly monitor their recipients’ focus of attention, their bodily orientation, and the adequate orientation of the phone to their recipients. As both the acknowledgement of the showable and the monitoring are necessarily carried out on a one-by-one basis, a smartphone-based showing, therefore, oscillates between moments of truly joint action (such as looking at the device) and sequentially organised, individual actions (such as responding to the showable), even if the showing has been globally projected as a joint course of action for more than one showee.

9.4

Conclusion

In this chapter, we focused on self-initiated showings of digital content (that is, initiated by smartphone holders without a co-participant’s request) in settings with at least three participants. Co-present participants do not simply react to the presence of any potentially showable object (a presence that becomes available, for example, by an individual’s specific manipulation of the phone). Instead, they respond to an action through which the smartphone holder transforms the smartphone into a publicly available and relevant object (Licoppe and Tuncer 2019; Oloff 2019). We focused, firstly, on showings that were initially designed and/ or carried out for one recipient but were then transformed into showings for more recipients (Sect. 9.3.1), then, secondly, on showings that have been designed from the start as a showing for multiple recipients

9 Getting (Others) Involved with Smartphones

333

(Sect. 9.3.2). We were interested in the different verbal and embodied action formats smartphone holders were using for getting co-present others involved in a showing, and in how addressed and unaddressed co-participants could respond to these. In the chosen settings, the participants had to deal not only with the inherent complexity of multiparty interaction, but also with the complexity ensuing from the use of a technological device in the form of a smartphone. In smartphone-based showing sequences in multiparty settings, various pre-existing interactional affordances meet the material affordances of the technological device and the interactional affordances of the object-centred sequence itself. The management of these different layers of affordances then leads to an overall similar and fundamentally sequential course of action (glossed as a showing sequence), which, nonetheless, unfolds throughout a variety of embodied and verbal actions and temporalities (Mondada 2016). The showing itself unfolds sequentially alongside basic interactional tasks (Oloff 2019). These include, from the shower’s perspective, the introduction of a new sequence and selection of addressed recipients, the introduction of the mobile device as a relevant object for this joint activity, the presentation of the showable on-screen for inspection, and the projection of possible types of responses to the showable. The showees have to adequately respond to these different initial actions by orienting their attention first to the shower, then to the device and to the showable, and finally to the task of inspecting and responding to the showable. In return, the shower will monitor the showees’ adequate responses to the respective task at hand by observing their bodily orientation and production of fitted turns-at-talk. These basic interactional tasks have to thus be accomplished by both building on and reshaping the following interactional and techno-material affordances: • Complex interactional affordances (multiparty and multiactivity): – Targeted showees (specific/single recipient, all co-participants, any co-participant) and their current focus of attention and availability. – Current sequential and topical environment (lapse, previous and ongoing talk, schism).

334

I. Avgustis and F. Oloff

– Newness of the showable with respect to the previous conversation and the recipients’ general knowledge about it. • Complex technological affordances (of the mobile device and the digital showable): – Availability of the showable (already available on screen versus not yet). – Manipulation of the device requiring gaze withdrawal from coparticipants. – Retrieval and noticing of a showable (targeted search or random discovery, self-initiated production when taking pictures, for example). – Size of the screen and of the showable, also in relation to the distance of the showees to the screen and the distance of the shower to the showees. – Type of showable (static/picture or dynamic/video, one versus several files). As the basic interactional tasks within a showing sequence are interwoven with a given set of affordances, the participants have to exploit a variety of multimodal resources for accomplishing these. This has been illustrated by the different ways in which these showing-related tasks in our chosen examples unfolded. The announcement can be more or less specific with respect to possible selected recipients, such as using singular or plural forms of verbs and pronouns or directing the gaze or the smartphone display to a specific recipient, for instance. Multimodally unspecific announcements are lacking specific grammatical forms or directed embodied actions, and are, therefore, designed for “anyone interested”. In these cases, the first perceivable audible or visible response (from the smartphone holder’s perspective) seems to define the order of the showing. Consequently, showings initially designed for one can be extended or carried out another time for supplementary showees. The timing of these transformations also hinges on the sequential progress of the already ongoing inspection: The showers usually monitor each recipient for their individual responses to the inspection, and the next

9 Getting (Others) Involved with Smartphones

335

recipient will only be addressed after the previous recipient has provided a response to the showable. Despite this principle of “one response per recipient”, at the same time, our analyses seem to reveal a preference for joint looking rather than for separate inspection. If the spatial arrangement allows for it, smartphone holders seem to orient to looking at the screen together with their recipients, despite them already knowing the visual content. This could bring up the question if the notion of “showing” is actually an adequate gloss for this activity (see also, Licoppe and Tuncer 2019, 567–568). More generally, participants seem to prefer a joint inspection, visible in second inspections carried out by previous recipients (ex. 1), centrally held phones, fast transitions from one recipient to the other, and continuous gazing at the phone display. This leads to the systematic establishment of different embodied showing Fformations, with the shower and as many showees as spatially possible having a joint focus of attention on the display. Smartphone-based showing sequences are, therefore, collaborative achievements and by no means designed by the smartphone holder alone, even if they initially and individually handle the central object. The variety of multimodal actions through which participants involve others or get themselves involved in a smartphone-based showing reflects the inherent complexity of social interaction, even in everyday settings. For the participants in a showing, this complexity relates to the management of multiple—both sequential and simultaneous—temporalities with respect to various interactional and technological affordances. For the analyst, the situated organisation of multiparty showing sequences involves a different type of complexity, as their inherent diversity presents a challenge for precisely transcribing, analysing and collection-building (Mondada 2016, 2018). In order to understand showings as a recognisably similar course of action that is, at the same time, embedded in a large array of sequential, material, and embodied features, future investigations will have to tackle some of these various actions and features separately. Further research is needed in order to verify the validity of our exploratory and cross-linguistic analyses, especially regarding the different (specific or unspecific) action formats used for initiating a showing, their connection to previous sequences and topics, as well as their sequential implicativeness. With respect to the selection of

336

I. Avgustis and F. Oloff

particular recipients or numbers of recipients, languages have different syntactic, morphological, and lexical resources at their disposal for specifying these, which might call for more language-specific studies of these sequences. Moreover, the specific semiotic characteristics of different types of showables (with respect to their audio-visual and fundamentally two-dimensional form, their temporality, and their status as “already known” or new) call for more thorough analyses of smartphone-based showings as well, preferably using additional recordings of the device screen (Avgustis and Oloff, forthcoming). A full understanding of the particular affordances of digital showables and of their impact on the organisation of joint action would require a comparison with “mediated” showables, such as during video calls, for example (Licoppe and Tuncer 2019), and with “three-dimensional” showables in copresence (Mondada 2012b). More generally, a detailed consideration of the tension between an assumed preference for one recipient at a time and a preference for joint looking will further contribute to the understanding of the complexity and the orderliness of object-centred sequences (Tuncer et al. 2019).

References Aaltonen, Tarja, Ilkka Arminen, and Sanna Raudaskoski. 2014. “Photo Sharing as a Joint Activity between an Aphasic Speaker and Others.” In Interacting with Objects: Language, Materiality, and Social Activity, edited by Maurice Nevile, Pentti Haddington, Trine Heineman, and Mirka Rauniomaa, 125– 44. Amsterdam/Philadelphia: John Benjamins. https://doi.org/10.1075/z. 186.06aal Avgustis, Iuliia, and Florence Oloff. In press. “Collecting and Analysing MultiSource Video Data: Grasping the Opacity of Smartphone Use in Face-toFace Encounters.” In Ethnomethodological Conversation Analysis in Motion: Emerging Methods and New Technologies, edited by Pentti Haddington, Tiina Eilittä, Antti Kamunen, Laura Kohonen-Aho, Tuire Oittinen, Iira Rautiainen, and Anna Vatanen. London: Routledge.

9 Getting (Others) Involved with Smartphones

337

Bolden, Galina. 2004. The Quote and Beyond: Defining Boundaries of Reported Speech in Conversational Russian. Journal of Pragmatics 36 (6): 1071–1118. https://doi.org/10.1016/j.pragma.2003.10.015. Brown, Barry, Moira McGregor, and Eric Laurier. 2013. “iPhone in Vivo: Video Analysis of Mobile Device Use.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1031–40. New York: Association for Computing Machinery. https://doi.org/10.1145/2470654. 2466132 Brown, Barry, Moira McGregor, and Donald McMillan. 2015. “Searchable Objects: Search in Everyday Conversation.” In CSCW ’15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, 508–17. https://doi.org/10.1145/2675133.2675206 Chen, Ying-Yu, Frank Bentley, Christian Holz, and Cheng Xu. 2015. “Sharing (and Discussing) the Moment: The Conversations That Occur Around Shared Mobile Media.” In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, 264– 73. MobileHCI ’15. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2785830.2785868 De Stefani, Elwys. 2010. “Reference as an Interactively and Multimodally Accomplished Practice. Organizing Spatial Reorientation in Guided Tours.” In Spoken Communication, edited by Massimo Pettorino, Antonella Giannini, Isabella Chiari, and Francesca M. Dovetto, 137–70. Newcastle: Cambridge Scholars Publishing. Deppermann, Arnulf. 2014. “Multimodal Participation in Simultaneous Joint Projects: Interpersonal and Intrapersonal Coordination in Paramedic Emergency Drills.” In Multiactivity in Social Interaction: Beyond Multitasking, edited by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 247–81. Amsterdam/Philadelphia: John Benjamins. https:/ /doi.org/10.1075/z.187.09dep Deppermann, Arnulf, and Jürgen Streeck, eds. 2018. Time in Embodied Interaction. Synchronicity and Sequentiality of Multimodal Resources. Amsterdam/ Philadelphia: John Benjamins. https://doi.org/10.1075/pbns.293 DiDomenico, Stephen M., and Jeffrey Boase. 2013. “Bringing Mobiles into the Conversation: Applying a Conversation Analytic Approach to the Study of Mobiles in Co-Present Interaction.” In Discourse 2.0: Language and New Media, edited by Deborah Tannen, and Anne Marie Trester, 119–31. Washington, DC: Georgetown University Press.

338

I. Avgustis and F. Oloff

DiDomenico, Stephen M., Joshua Raclaw, and Jessica S. Robles. 2020. Attending to the Mobile Text Summons: Managing Multiple Communicative Activities Across Copresent and Technologically Mediated Interpersonal Interactions. Communication Research 47 (5): 669–700. https://doi.org/10. 1177/0093650218803537. Egbert, Maria. 1993. Schisming: The Transformation from a Single Conversation to Multiple Conversations. PhD Dissertation in Applied Linguistics. Los Angeles: UCLA. ———. 1997. “Schisming: The Collaborative Transformation from a Single Conversation to Multiple Conversations.” Research on Language and Social Interaction 30 (1):1–51. https://doi.org/10.1207/s15327973rlsi3001_1 Goffman, Erving. 1955. On Face-Work. Psychiatry 18 (3): 213–231. https:// doi.org/10.1080/00332747.1955.11023008. ———. 1963. Behavior in Public Places: Notes on the Social Organization of Gatherings. New York: Free Press of Glencoe. ———. 1981. Forms of Talk. Philadelphia: University of Pennsylvania Press. Goodwin, Charles. 1979. The Interactive Construction of a Sentence in Natural Conversation. In Everyday Language: Studies in Ethnomethodology, edited by George Psathas, 97–121. New York: Irvington Publishers. ———. 1980. Restarts, Pauses, and the Achievement of a State of Mutual Gaze at Turn-Beginning. Sociological Inquiry 50: 272–302. https://doi.org/ 10.1111/j.1475-682X.1980.tb00023.x. ———. 1981. Conversational Organization. Interaction between Speakers and Hearers. New York: Academic Press. ———. 1984. “Notes on Story Structure and the Organization of Participation.” In Structures of Social Action: Studies in Conversational Analysis, edited by J. Maxwell Atkinson, and John Heritage, 225–46. Cambridge: Cambridge University Press. ———. 1986. “Audience Diversity, Participation and Interpretation.” Text 6 (3):283–316. https://doi.org/10.1515/text.1.1986.6.3.283 ———. 1994. “Professional Vision.” American Anthropologist 96 (3):606–33. https://doi.org/10.1525/aa.1994.96.3.02a00100 ———. 1996. “Transparent Vision.” In Interaction and Grammar, edited by Elinor Ochs, Emanuel Schegloff, and Sandra Thompson, 370–404. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO978 0511620874.008 ———. 2000. “Action and Embodiment within Situated Human Interaction.” Journal of Pragmatics 32:1489–1522. https://doi.org/10.1016/S03782166(99)00096-X

9 Getting (Others) Involved with Smartphones

339

———. 2003. “Pointing as Situated Practice.” In Pointing: Where Language, Culture and Cognition Meet, edited by Sotaro Kita, 217–41. Mahwah, NJ: Lawrence Erlbaum. Goodwin, Charles, and Marjorie Harness Goodwin. 2004. Participation. In A Companion to Linguistic Anthropology, edited by Alessandro Duranti, 222– 244. Oxford: Basil Blackwell. Goodwin, Marjorie Harness. 1980. Processes of Mutual Monitoring Implicated in the Production of Description Sequences. Sociological Inquiry 50 (3–4): 303–317. https://doi.org/10.1111/j.1475-682X.1980.tb00024.x. ———. 1996. “Informings and Announcements in Their Environment: Prosody within a Multi-Activity Work Setting.” In Prosody in Conversation: Interactional Studies. Cambridge University Press. https://doi.org/10.1017/ CBO9780511597862.013 ———. 1997. “Byplay: Negotiating Evaluation in Storytelling.” In Towards a Social Science of Language: Papers in Honor of William Labov 2: Social Interaction and Discourse Structures, edited by Gregory Guy, Crawford Feagin, Deborah Schiffrin, and John Baugh, 77–102. Amsterdam/Philadelphia: John Benjamins. https://doi.org/10.1075/cilt.128.08goo Goodwin, Marjorie Harness, and Charles Goodwin. 2012. Car Talk: Integrating Texts, Bodies, and Changing Landscapes. Semiotica 191 (1/4): 257–286. https://doi.org/10.1515/sem-2012-0063. Haddington, Pentti, Tiina Eilittä, Antti Kamunen, Laura Kohonen-Aho, Tuire Oittinen, Iira Rautiainen, and Anna Vatanen-Nissinen (2022[tbc]). “Introduction. [title to be adapted]” In Complexity and intersubjectivity: Shared understanding in multi-layered social interaction, edited by Pentti Haddington, Tiina Eilittä, Antti Kamunen, Laura Kohonen-Aho, Tuire Oittinen, Iira Rautiainen, and Anna Vatanen-Nissinen. Amsterdam: John Benjamins. Haddington, Pentti, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 2014. “Towards Multiactivity as a Social and Interactional Phenomenon.” In Multiactivity in Social Interaction: Beyond Multitasking, edited by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 3– 32. Amsterdam/Philadelphia: John Benjamins. https://doi.org/10.1075/z. 187.01had Haddington, Pentti, and Mirka Rauniomaa. 2011. Technologies, Multitasking, and Driving: Attending to and Preparing for a Mobile Phone Conversation in a Car. Human Communication Research 37 (2): 223–254. https://doi.org/ 10.1111/j.1468-2958.2010.01400.x.

340

I. Avgustis and F. Oloff

Hayashi, Makoto, Junko Mori, and Tomoyo Takagi. 2002. Contingent Achievement of Co-Tellership in a Japanese Conversation. In The Language of Turn and Sequence, ed. Cecilia Ford, Barbara Fox, and Sandra Thompson, 81–122. New York: Oxford University Press. Heath, Christian. 1986. Body Movement and Speech in Medical Interaction. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO978 0511628221. Heath, Christian, and Paul Luff. 1996. “Convergent Activities: Line Control and Passenger Information on the London Underground.” In Cognition and Communication at Work. Cambridge: Cambridge University Press. https:// doi.org/10.1017/CBO9781139174077.005 ———. 2000. Technology in Action. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511489839. Hellermann, John, Steven L. Thorne, and Peter Fodor. 2017. Mobile Reading as Social and Embodied Practice. Classroom Discourse 8 (2): 99–121. https:/ /doi.org/10.1080/19463014.2017.1328703. Heritage, John. 1984. “A Change-of-State Token and Aspects of Its Sequential Placement.” In Structures of Social Action, edited by J. Maxwell Atkinson and John Heritage, 299–345. Cambridge: Cambridge University Press. https:// doi.org/10.1017/CBO9780511665868.020 Hindmarsh, Jon, and Christian Heath. 2000. Sharing the Tools of the Trade: The Interactional Constitution of Workplace Objects. Journal of Contemporary Ethnography 29 (5): 523–526. https://doi.org/10.1177/089124100129 023990. Holler, Judith, and Kobin H. Kendrick. 2015. “Unaddressed Participants’ Gaze in Multi-Person Interaction: Optimizing Recipiency.” In Frontiers in Psychology 6 (Article 98):1–14. https://doi.org/10.3389/fpsyg.2015.00098 Hutchby, Ian. 2001. Conversation and Technology: From the Telephone to the Internet. Cambridge: Polity Press. Ito, Mizuko. 2005. “Mobile Phones, Japanese Youth, and the Re-Placement of Social Contact.” In Mobile Communications: Re-Negotiation of the Social Sphere, edited by Richard Seyler Ling, and Per Pedersen, 131–48. London: Springer. https://doi.org/10.1007/1-84628-248-9_9 Jefferson, Gail. 2004. “Glossary of Transcript Symbols with an Introduction.” In Conversation Analysis: Studies from the First Generation, edited by Gene H. Lerner, 13–34. Amsterdam, Philadelphia: John Benjamins. https://doi. org/10.1075/pbns.125.02jef

9 Getting (Others) Involved with Smartphones

341

Keel, Sara. 2011. The Parents’ Questioning Repeats in Response to Young Children’s Evaluative Turns. Gesprächsforschung - Online-Zeitschrift Zur Verbalen Interaktion 12: 52–94. Keisanen, Tiina, Mirka Rauniomaa, and Pentti Haddington. 2014. “Suspending Action: From Simultaneous to Consecutive Ordering of Multiple Courses of Action.” In Multiactivity in Social Interaction: Beyond Multitasking, edited by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 109–34. Amsterdam/Philadelphia: John Benjamins. https://doi.org/10.1075/z.187.04kei Kendon, Adam. 1990. Conducting Interaction. Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press. Kidwell, Mardi, and Don H. Zimmerman. 2007. “Joint Attention as Action.” In Journal of Pragmatics, Special Issue: Diversity and Continuity in Conversation Analysis, 39 (3):592–611. https://doi.org/10.1016/j.pragma.2006. 07.012 König, Katharina, and Florence Oloff. 2018a. “Ansätze zu einer Multimodalen Erzählanalyse. Einführung in das Themenheft” [Approaches to a Multimodal Narrative Analysis. An Introduction to the Special Issue]. Gesprächsforschung—Online-Zeitschrift Zur Verbalen Interaktion 19:207–41. ———. 2018b. “Die Multimodalität Alltagspraktischen Erzählens” [The Multimodality of Everyday Storytellings]. Zeitschrift für Literaturwissenschaft und Linguistik 48:277–307. https://doi.org/10.1007/s41244-018-0093-7. Laurier, Eric. 2005. “Searching for a Parking Space.” Intellectica 2005/2–3, 41–42:101–15. https://doi.org/10.3406/intel.2005.1723 Laurier, Eric, Barry Brown, and Moira McGregor. 2016. Mediated Pedestrian Mobility: Walking and the Map App. Mobilities 11 (1): 117–134. https:// doi.org/10.1080/17450101.2015.1099900. Lerner, Gene H. 2002. Turn-Sharing: The Coral Co-Production of Talk-inInteraction. In The Language of Turn and Sequence, ed. Cecilia Ford, Barbara Fox, and Sandra Thompson, 225–256. New York: Oxford University Press. ———. 2003. Selecting Next Speaker: The Context-Sensitive Operation of a Context-Free Organization. Language in Society 32: 177–201. https://doi. org/10.1017/S004740450332202X. Licoppe, Christian, and Sylvaine Tuncer. 2019. “The Initiation of Showing Sequences in Video-Mediated Communication.” Gesprächsforschung: OnlineZeitschrift Zur Verbalen Interaktion 20, 545–71. Lilja, Niina, and Arja Piirainen-Marsh. 2019. “Making Sense of Interactional Trouble Through Mobile-Supported Sharing Activities.” In Teaching and Testing L2 Interactional Competence: Bridging Theory and Practice, edited by

342

I. Avgustis and F. Oloff

M. Rafael Salaberry and Silvia Kunitz, 260–88. London: Routledge. https:/ /doi.org/10.4324/9781315177021 Madianou, Mirca, and Daniel Miller. 2012. “Polymedia: Towards a New Theory of Digital Media in Interpersonal Communication.” In the International Journal of Cultural Studies 16 (2):169–87. https://doi.org/10.1177/ 1367877912452486 Mantere, Eerik. 2020. “What Smartphones, Ethnomethodology, and Bystander Inaccessibility Can Teach Us about Better Design?” In Human Computer Interaction and Emerging Technologies: Adjunct Proceedings from the INTERACT 2019 Workshops, edited by Fernando Loizides, Marco Winckler, Usashi Chatterjee, Jose Abdelnour-Nocera, and Antigoni Parmaxi, 91–100. Cardiff: Cardiff University Press. https://doi.org/10.18573/book3.k McNeill, David. 2005. “Gesture, Gaze, and Ground.” In International Workshop on Machine Learning for Multimodal Interaction, edited by Steve Renals and Samy Bengio, 1–14. Berlin: Springer. https://doi.org/10.1007/116774 82_1 Mondada, Lorenza. 2007. Multimodal Resources for Turn-Taking: Pointing and the Emergence of Possible Next Speakers. Discourse Studies 9 (2): 194– 225. https://doi.org/10.1177/1461445607075346. ———. 2009. The Methodical Organization of Talking and Eating: Assessments in Dinner Conversations. Food Quality and Preference 20 (8): 558–571. https://doi.org/10.1016/j.foodqual.2009.03.006. ———. 2011. The Organization of Concurrent Courses of Action in Surgical Demonstrations. In Embodied Interaction: Language and Body in the Material World , ed. Jürgen. Streeck, Charles Goodwin, and Curtis LeBaron, 207– 226. Cambridge: Cambridge University Press. ———. 2012a. Talking and Driving: Multiactivity in the Car. Semiotica 191 (1/4): 223–256. https://doi.org/10.1515/sem-2012-0062. ———. 2012b. “Organisation Multimodale de la Parole-en-interaction: Pratiques Incarnées d’Introduction des Référents” [The Multimodal Organisation of Talk-in-Interaction: Embodied Practices of Introducing Referents]. Langue Française 175 (3): 129–147. https://doi.org/10.3917/lf.175.0129. ———. 2013. Displaying, Contesting and Negotiating Epistemic Authority in Social Interaction: Descriptions and Questions in Guided Visits. Discourse Studies 15 (5): 597–626. https://doi.org/10.1177/1461445613501577. ———. 2014. “The Temporal Orders of Multiactivity: Operating and Demonstrating in the Surgical Theatre.” In Multiactivity in Social Interaction: Beyond Multitasking, edited by Pentti Haddington, Tiina Keisanen, Lorenza

9 Getting (Others) Involved with Smartphones

343

Mondada, and Maurice Nevile, 33–76. Amsterdam/Philadelphia: John Benjamins. https://doi.org/10.1075/z.187.02mon ———. 2016. ”Challenges of Multimodality: Language and the Body in Social Interaction.” Journal of Sociolinguistics 20 (3):336–66. https://doi.org/10. 1111/josl.1_12177. ———. 2018. “Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing Multimodality.” Research on Language and Social Interaction 51 (1):85–106. https://doi.org/10.1080/08351813.2018. 1413878. ———. 2019. “Conventions for Multimodal Transcription.” https://www.lor enzamondada.net/multimodal-transcription. Nevile, Maurice, Pentti Haddington, Trine Heineman, and Mirka Rauniomaa, eds. 2014. Interacting with Objects. Language, Materiality, and Social Activity. Amsterdam/Philadelphia: John Benjamins. https://doi.org/10.1075/z.186 Nielsen, Søren Beck. 2016. How Doctors Manage Consulting Computer Records While Interacting with Patients. Research on Language and Social Interaction 49 (1): 58–74. https://doi.org/10.1080/08351813.2016.112 6451. Oloff, Florence. 2012. “Withdrawal from Turns in Overlap and Participation.” In Prosody and Embodiment in Interactional Grammar, edited by Pia Bergmann, Jana Brenning, Martin Pfeiffer, and Elisabeth Reber, 207–36. Berlin/Boston: De Gruyter. https://doi.org/10.1515/9783110295108.207 ———. 2019. “Das Smartphone als Soziales Objekt: Eine Multimodale Analyse von Initialen Zeigesequenzen in Alltagsgesprächen” [Smartphones as Social Objects: a Multimodal Analysis of Initial Showing Sequences in Ordinary Conversation]. In Interaktion und Medien: Interaktionsanalytische Zugänge zu Medienvermittelter Kommunikation, edited by Konstanze Marx and Axel Schmidt, 191–218. Heidelberg: Universitätsverlag Winter. ———. 2021a. “New Technologies – New Social Conduct? A Sequential and Multimodal Approach to Smartphone Use in Face-to-Face Interaction.” Bulletin VALS-ASLA 1/2021a:13–34. ———. 2021b. “Some Systematic Aspects of Self-Initiated Mobile Device Use in Face-to-Face Encounters.” Journal Für Medienlinguistik 2 (2):195–235. https://doi.org/10.21248/jfml.2019.21. Porcheron, Martin, Joel E. Fischer, and Sarah Sharples. 2016. “Using Mobile Phones in Pub Talk.” In CSCW ’16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, 1649–61. San Francisco. https://doi.org/10.1145/2818048.2820014

344

I. Avgustis and F. Oloff

———. 2017. “‘Do Animals Have Accents?:’ Talking with Agents in MultiParty Conversation.” In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 207–19. Portland. https://doi.org/10.1145/2998181.2998298 Raclaw, Joshua, Jessica S. Robles, and Stephen M. DiDomenico. 2016. Providing Epistemic Support for Assessments Through Mobile-Supported Sharing Activities. Research on Language and Social Interaction 49 (4): 362–379. https://doi.org/10.1080/08351813.2016.1199089. Raudaskoski, Sanna. 2009. Tool and Machine: The Affordances of the Mobile Phone. Tampere: University of Tampere Press. http://urn.fi/urn:isbn:978951-44-7802-4. Sacks, Harvey. 1992. Lectures on Conversation (1964–1972). Oxford: Basil Blackwell. https://doi.org/10.1002/9781444328301. Sahlström, Fritjof, Marie Tanner, and Verneri Valasmo. 2019. Connected youth, connected classrooms. Smartphone use and student and teacher participation during plenary teaching. Learning, Culture and Social Interaction 21: 311–331. https://doi.org/10.1016/j.lcsi.2019.03.008. Schegloff, Emanuel A. 1995. Parties and Talking Together: Two Ways in Which Numbers Are Significant for Talk-in-Interaction. In Situated Order: Studies in the Social Organization of Talk and Embodied Activities, ed. Paul ten Have and George Psathas, 31–42. Washington: International Institute for Ethnomethodology and Conversation Analysis & University Press of America. ———. 1998. “Body Torque.” Social Research 65 (3):535–96. https://www. jstor.org/stable/40971262 Schegloff, Emanuel A. 2000. Overlapping Talk and the Organization of TurnTaking for Conversation. Language in Society 29 (1): 1–63. https://doi.org/ 10.1017/S0047404500001019. Stivers, Tanya, and Jeffrey Robinson. 2006. A Preference for Progressivity in Interaction. Language in Society 35 (3): 367–392. https://doi.org/10.1017/ S0047404506060179. Stivers, Tanya, and Federico Rossano. 2010. Mobilizing Response. Research on Language and Social Interaction 43 (1): 3–31. https://doi.org/10.1080/083 51810903471258. Stoenica, Ioana-Maria, and Simona Pekarek Doehler. 2020. “Relative-clause Increments and the Management of Reference: A Multimodal Analysis of French Talk-in-interaction.” In Emergent Syntax for Conversation: Clausal Patterns and the Organization of Actions, 303–29. Amsterdam: John Benjamins. https://doi.org/10.1075/slsi.32.11sto

9 Getting (Others) Involved with Smartphones

345

Streeck, Jürgen, Charles Goodwin, and Curtis LeBaron, eds. 2011. Embodied Interaction. Language and Body in the Material World . Cambridge: Cambridge University Press. Stukenbrock, Anja. 2018. “Forward-Looking: Where Do We Go with Multimodal Projections?” In Time in Embodied Interaction: Synchronicity and Sequentiality of Multimodal Resources, edited by Arnulf Deppermann, and Jürgen Streeck, 31–68. Amsterdam/Philadelphia: John Benjamins. https:// doi.org/10.1075/pbns.293.01stu Suderland, David. 2020. ”„oh Isch FIND_s Nich;“ Eine Konversationsanalytische Untersuchung Sprachlicher Bezugnahmen Auf Smartphonegestützte Suchprozesse in Alltagsgesprächen” [‘Oh I can’t find it.’ A Conversation-Analytic Investigation of Verbal Comments on Smartphonebased Search Processes in Everyday Conversations]. Journal for Media Linguistics 2 (2): 90–122. https://doi.org/10.21248/jfml.2019.17 Tuncer, Sylvaine, Christian Licoppe, and Pentti Haddington. 2019. “When Objects Become the Focus of Human Action and Activity: Object-Centred Sequences in Social Interaction.” Gesprächsforschung: Online-Zeitschrift Zur Verbalen Interaktion 20:384–98. Weilenmann, Alexandra, Thomas Hillman, and Beata Jungselius. 2013. “Instagram at the Museum: Communicating the Museum Experience through Social Photo Sharing.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13, 1843–52. Paris: ACM Press. https:/ /doi.org/10.1145/2470654.2466243 Weilenmann, Alexandra, and Catrine Larsson. 2002. “Local Use and Sharing of Mobile Phones.” In Wireless World: Social and Interactional Aspects of the Mobile Age, edited by Barry Brown and Nicola Green, 92–107. London: Springer Verlag. https://doi.org/10.1007/978-1-4471-0665-4_7 Zima, Elisabeth. 2020. Gaze and Recipient Feedback in Triadic Storytelling Activities. Discourse Processes 57 (9): 725–748. https://doi.org/10.1080/016 3853X.2020.1769428. Zima, Elisabeth, Clarissa Weiß, and Geert Brône. 2019. Gaze and Overlap Resolution in Triadic Interactions. Journal of Pragmatics 140: 49–69. https:/ /doi.org/10.1016/j.pragma.2018.11.019.

10 The Primacy of Affective Engagement in Simultaneously Unfolding Participation Frameworks Julia Katila , Sara A. Goico , Yumei Gan , and Marjorie Harness Goodwin

10.1 Introduction In this chapter we examine the multitude of ways in which people experience and express together their emotions while they engage concurrently in multiple participation frameworks (Goffman 1981). Building on J. Katila (B) Tampere University, Tampere, Finland e-mail: [email protected] S. A. Goico · M. H. Goodwin University of California, Los Angeles, CA, USA e-mail: [email protected] M. H. Goodwin e-mail: [email protected] Y. Gan Shanghai Jiao Tong University, Shanghai, China e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_10

347

348

J. Katila et al.

current interactional research on how people engage in the multilayeredness of social interaction, such as when people attend to interactions while their bodies are mobile (Haddington et al. 2013; McIlvenny et al. 2009) or engage in multiple activities at the same time (Haddington et al. 2014), our work investigates how individuals arrange their bodies so that they can attend to more than one participation framework simultaneously. In these settings, we focus on an understudied dimension of interaction—affect (c.f. Goodwin and Cekaite 2018; Ruusuvuori 2013). The terms affect and emotion, which we use interchangeably in this chapter, refer to the interactional (Ruusuvuori 2013) and intercorporeal (Merleau-Ponty 1962) aspects of sentiments emerging through forms of human sociality. Emotion is an omnipresent feature of social interaction (Goffman 1961) even if it is seldom topicalized or expressed in words (Ruusuvuori 2013: 330). The expression and sharing of emotions are often intertwined with other actions, making it a challenge to study emotion in interaction. We begin from the perspective that engaging in another person’s emotion is one of the most fundamental elements of intersubjective order (Peräkylä 2013: 552; Peräkylä et al. 2021) and that it is crucial for the organization of social relationships (M. H. Goodwin 2017; M. H. Goodwin and Cekaite 2018). We will examine in detail the connection between affect and accessibility to different participation frameworks in moments where multiple participation frameworks exist concurrently, and participants have various types of sensorial access to each other. The use of the term “participation framework” refers to forms of involvement which are collaboratively attended to by speakers and hearers in co-occurring action (C. Goodwin and M. H. Goodwin 2004: 222). We propose that affect has the ability to interweave various participation frameworks and the participants in them, as it is a form of engagement which resonates in space beyond a single participation framework. We draw on co-operative (C. Goodwin 2018; M. H. Goodwin and Cekaite 2018) and intercorporeal (M. H. Goodwin 2017; Katila 2018; Meyer et al. 2017) perspectives to analyze spontaneously unfolding emotion in interaction. We examine how connecting various participation frameworks through affect is accomplished in four different interactional ecologies: a baby’s health check-up in Finland, a mainstream

10 The Primacy of Affective Engagement

349

classroom with deaf students in Peru, a celebratory gathering of friends in the US, and a video call between migrant parents and their children in China. In these settings, we study the emergence of different forms of emotion, such as empathy, anger, joy, love, and conflict, which resonate in these interactions. Exploring emotionally salient moments in these various cultural contexts and interactional settings offers a perspicuous setting (Garfinkel 2002: 181–182) for examining the complexity of interaction in multiple simultaneously unfolding participation frameworks. By complexity of interaction, we refer to the multilayered-ness of affect in the coexisting participation frameworks. Affective expressions built of various embodied resources, such as touch, hand gestures, or prosody can play a role in more than one participation framework at the same time. Our first two extracts show how affect can become a medium for a basic interpersonal engagement across participation frameworks where shared linguistic resources between all participants are lacking. The third extract illustrates how participants of a multiparty interaction—some engaging through tactile contact, others, through aural and visual means—are still able to engage in the same affective atmosphere (Brennan 2004: 1) while inhabiting nested participation frameworks. We finally demonstrate how affective engagement can be sustained as a form of engagement even when a party is involved in competing activities. These extracts shed light on the various ways in which emotional engagement is being prioritized in moment-by-moment unfolding complex multiparty interactions, consisting of multiple overlapping interaction projects, activities, and frameworks. We illuminate how emotions are inherently social and often expressed in explicit social actions; they are, at the same time, distributed through inherently felt, experienced, and embodied means. These expressive and experienced aspects of emotion allow people to engage across different participation frameworks.

350

J. Katila et al.

10.2 Multilayered Interaction Through the Lens of Participation Frameworks We build upon prior work on multiactivity (Haddington et al. 2014), where a chief focus is on “the various interactional means for coordinating and ordering multiple activities” (ibid.: 24). As summarized by Haddington et al. (2014: 19), “a central issue and challenge for understanding and analysing ‘multiactivity’ concerns the sequential, temporal, serial or concurrent organisation of actions and phases across activities”. A major interest has been how participants manage two or more activities posing multiple simultaneous and, at times, conflicting demands for the organization of their actions (ibid.: 21). For example, Mondada (2014a) analyzes how surgeons engage simultaneously in operating on a patient while giving a demonstration about the operation to spectators. She argues that to investigate the conceptualization of time in interaction, one should attend to the praxeological, as well as sequential details of the unfolding interaction. The embodied action of doing surgery has a different temporal trajectory from that of instruction. C. Goodwin (2002) calls for examining time within multiactivity as a rich multimodal ecology of sign systems within a “multiplicity of concurrently relevant embodied temporalities” (p. 19), including gaze, body posture, and tools. Schegloff (1998: 540) examining “body torque” illustrates how posture is an important resource “for displaying multiple courses of action and the interactional differential ranking of those courses of action” (ibid.: 536). Taking a different approach from this previous work, we ask: how do participants simultaneously attend to the affective engagement of different co-participants within multiple participation frameworks? We take as our fundamental starting point the “participation framework” (C. Goodwin and M. H. Goodwin 2004: 222), rather than multiactivity. Thus, the focus is on how, in the midst of talk, participants display to one another what they are doing and how they expect others to align themselves towards the activity of the moment (ibid.: 222). Rather than a focus on multiactivity and divergent demands, our concern is with how someone affectively attends to people within two different participation frameworks simultaneously through the same action. Our aim is

10 The Primacy of Affective Engagement

351

to explore the affective engagement and experience of participants within multiparty interaction, as it is critical to the formation of intersubjective understanding.

10.3 Intercorporeal and Co-operative Perspectives of Interaction We follow previous literature which has shown that human beings are especially prone to engaging in social interactions with one another (Levinson 2006). Moreover, affect and emotion are primordial elements of this interpersonal tendency (Joaquin and Schumann 2013; Trevarthen and Aitken 2001; Tronick 1989) and, therefore, a basis for intersubjectivity. Yet for the most part, emotions and the construction of human relationships have been treated as separate from the moment-by-moment interactions in conversation analytic studies. For example, Schegloff (2010: 38) writes the following: two conceptions of the calling of conversation analysis (CA): One is centered on the organization of action in interaction, the organizations of practices for accomplishing those actions and courses of action, and the basic infrastructure for the whole domain—turns and their form and distribution; actions and their trajectories; troubles and their resolution; language as an interface with the physical, social, cultural, emotional, and other worlds that humans live in, grasp and navigate, etc. The other conception is centered on embodied actors, bringing the elements of the organization of human sociality just mentioned into being moment by moment in a particular place, with particular others, vying with or yielding to one another, etc.

In our view, affect should not be treated as separable from interlocutors and their relationships. Our work highlights the saliency of social relationships and examines the foundation of emotional and affective attunement for social relations (Gan 2020; M. H. Goodwin 2017, Goodwin and Cekaite 2018; M. H. Goodwin et al. 2012; Katila 2018; Katila and Philipsen 2019). We treat affect as a situated practice (Goodwin

352

J. Katila et al.

et al. 2012), which motivates and enables the social connection between individuals in context. We draw on various existing studies that have investigated affective, tactile, and haptic aspects of interaction (e.g., Cekaite 2016; M. H. Goodwin 2017; M. H. Goodwin and Cekaite 2018; Katila 2018; Peräkylä and Ruusuvuori 2012). In particular, we adopt their view that relevant affective engagement does not always occur when participants are vis-à-vis one another in a classic facing formation (Kendon 1990), and that expressions of emotion can extend over multiple turns or parts of turns, rather than being closely tied to a single turn. Emotion can persist beyond any single body movement or moment in time and can therefore be understood as intercorporeally transmitting affective atmosphere or mood (Brennan 2004; Throop 2014). Furthermore, we treat emotions as a form of behavior where sensing, expressive, and performative bodies intersect—they are both intercorporeal and semiotic (Katila and Raudaskoski 2020). As a consequence, we approach these topics employing an intercorporeal and co-operative perspective for interaction analysis of video-recorded encounters. The idea of intercorporeality developed by Merleau-Ponty (e.g., 1962), who drew from Husserl (1982), refers to the embodied and experienced aspects of interaction, and describes the simultaneity and reciprocity of sensing and being sensed when human beings are in each other’s co-presence. Due to these concurrent sentient and sensible aspects of bodies, perception of one another in interaction is inherently simultaneous and already embedded with meaning. The intercorporeal perspective enables us to approach affect in interaction as an embodied and experienced phenomenon. The intercorporeal co-existence of human beings also implies the individual’s creative ability to engage in semiotic and co-operative (C. Goodwin 2018) forms of meaning-making in interactions. The human body, language development, and other forms of symbolic action have developed into their current form in co-operative engagements and actions (C. Goodwin 2018). Analogous to a couple’s dance, co-operative action is co-participatory interaction. The actions of the interlocutors do not make sense alone; instead, their meaning is co-manufactured, through building on each other’s semiotic and corporeal productions both simultaneously and sequentially (C. Goodwin 2018).

10 The Primacy of Affective Engagement

353

We incorporate these intercorporeal and co-operative understandings of the human body and action in video-analysis of interaction, which aims at describing in detail the temporal order of emotional participation as it unfolds as both intercorporeal and symbolic forms of communication in multiparty interactions. The verbal transcription conventions are presented in the Appendix. The conventions were modified for our purpose from the work of Gail Jefferson (2004). The original spoken language in which the interactions were produced is provided in italics with English translations in bold. In what follows, we present four brief case studies of affective engagement from different contexts: a routine check-up at a child healthcare clinic in Finland, a classroom lesson in a mainstream classroom with deaf and hearing students in Peru, a surprise party in the United States, and a video call between a mother and her left-behind child and grandmother in China. Bringing together cases from different settings and cultures allows us to investigate the importance of emotion in interaction across contexts.

10.4 Analysis 10.4.1 Finnish Health Check-Up: Empathetic Expression Flows from One Participation Framework to Another Before language or gestures occur, newborn babies are already able to understand the “language of emotion”—to read the affective meanings of the caregiver’s embodied actions (e.g., Hertenstein 2002; Trevarthen and Aitken 2001). To exemplify the primacy of emotional engagement, we start by analyzing an interaction episode where a four-week-old baby is being given a routine check-up in a Finnish child healthcare center in order to test if her bodily functions have developed “normally” after her birth. The extract is part of an already existing dataset owned by Tampere University collected at Finnish children’s healthcare clinics (for a detailed description of data, see Homanen 2013).

354

J. Katila et al.

As the episode begins, Nurse 1 (NU1) is palpating the baby’s (BAB) soft spot, i.e., “fontanelle”, while Nurse 2 (NU2) and the mother (MOM) are closely following the activity. In Extract 1, we witness how the nurse’s “motherese” intonation communicates empathy to the baby in the wake of potentially uncomfortable touching, and the baby remains calm. Furthermore, we discuss how this special tone of voice also colors with positive affect the simultaneously occurring visually and aurally orchestrated participation framework between the adult participants.

10 The Primacy of Affective Engagement

Extract 1

355

356

J. Katila et al.

10 The Primacy of Affective Engagement

357

In Image 1.1, NU1 controls BAB’s body through touch (Cekaite 2016), which enables her to palpate BAB’s soft spot. Co-occurring with this haptic action NU1 incorporates so-called “infant-directed” or “motherese” (Fernald 1985; Stern 2002[1977]) tone of voice, which can be characterized by a higher and wider pitch range, repetitive word structures, and an exaggerated intonation (Fernald 1985: 181). Moreover, the

358

J. Katila et al.

tone of the voice is often accompanied by exaggerated facial expressions (Stern 2002[1977]:25–29). This voice register initiated by NU1 in lines 01 and 04, and pictured in Fig. 1, introduces the saliency of emotion in the participation framework with BAB.

As observable in Fig. 1, NU1’s pitch contour shifts up and down dramatically during her utterance: starting at 290 Hz on the words ‘a little if it is’, reaching as high as 425 Hz on the word ‘the soft spot’, going down to 245 Hz on the word ‘here’, and then back up to 370 Hz on the words ‘shall we try’. This tone can be qualitatively heard as baby talk. Moreover, NU1 uses repetitive structure (saying ‘shall we try’ two times), and accompanies the talk with an exaggerated facial expression—eyes wide, and narrowing cheeks merging towards pouting lips (Fig. 1). Laminating the haptic action—which may feel uncomfortable for BAB’s body, as an unhealed soft spot is touched—with this empathetic voice and facial expression, NU1 is creating a multisensorially radiating affective space which simultaneously communicates an apology for the possible suffering resulting from the touch. Moreover, NU2 co-participates in NU1’s action with an aligning tone of the voice (‘yeah?’ in line 03 and Fig. 1). In line 07, N1 continues with a motherese expression by praising the child. Upon the word ‘SO::’, her voice shifts from 342 to 478 Hz.

10 The Primacy of Affective Engagement

359

When listening to the audio of the extract, it is possible to hear that NU1’s voice embodies the “qualia” of gentleness and empathy which is recognizable and meaningful beyond being part of the conceptual meaning of the words. According to Harkness (2015: 573), qualia refers to a signal which materializes phenomenally in action as sensuous quality. Thus, the tone of the motherese voice aurally “touches” the baby through resonating sound waves, and enables the transmission of affect and an affective engagement between the baby and the nurse. BAB coparticipates in NU1’s action by not crying or explicitly resisting NU1’s movement; she is showing that she is, at least at a basic level, letting her body be investigated. Moreover, BAB accompanies NU1’s verbal action with the vocal productions khäh (lines 02 and 05), äääh (line 06), and ähh (08) which co-occur with N1’s affective expression in interactionally relevant places (see Sierra 2017). In the next move, and immediately after NU1’s utterance (line 09), MOM starts asking a question about BAB’s soft spot (lines 10). At this point (Image 1.3), NU1’s full body is still attending to BAB while the palpation has transformed into gently caressing the baby’s face (Image 1.3). MOM continues her turn at talk, “environmentally coupling” (Goodwin 2018: 221–242) the words ‘I have felt it (.) it is there in the head that’ (line 11) with touching her own head. The ambiguity of MOM’s use of the word ‘there’ invites the listeners’ visual attention;

360

J. Katila et al.

NU1 and NU2 turn their gaze towards MOM to witness the selftouch gesture. While in haptic formation with BAB, NU1 is applying a “body torque” (Schegloff 1998) to engage in a visually coordinated participation framework with MOM and NU2 (Katila 2018: 51–59). By maintaining her lower body towards BAB, NU1 is indicating a more temporally extended and fundamental commitment towards the affective participation framework with BAB (Image 1.4, see Schegloff 1998:536). NU1 then responds to MOM with an adult-targeted tone of voice ‘YES when they are still like that [.hhyeah]’ (line 09), while she is already twisting her body back to BAB and palpating the soft spot again (Image 1.4). With regard to the pitch contour, we find less variation in the adult-targeted speech in comparison with the motherese. As presented in Fig. 3, the pitch contour in the adult-targeted voice is said in a lower pitch tone in comparison with the previous motherese tone, and there are less dramatic pitch shifts. Moreover, the verbal and facial expression of the nurse is not exaggerated and affectively laden, as in the motherese expression.

Overlapping with the nurse’s vocal production (line 12), the mother formulates two responses (‘okeyh good’ and ‘oh good?’, line 13) through which she displays that she is already treating the nurse’s response as indicating that everything is all right with the baby’s soft spot, even

10 The Primacy of Affective Engagement

361

though NU1 has not actually made this explicit yet (she does this in her following utterance). During her next verbal action ‘.HHYEAH’ (line 14), NU1 re-initiates an aural engagement with BAB while continuing to manipulate her head (Image 1.6). Re-evoking the affective aspect of the participation framework with BAB, the nurse voices the following utterance by returning to the motherese voice: ‘it is just indeed like it is SUPP(HH)OSED to be’ (line 14).

Through this utterance, NU1 is explicitly confirming that the BAB’s soft spot has healed normally. However, by presenting her body posture and facial expression directly to BAB, as well as engaging with the baby through touch and aural action, it is observable that NU1 prioritizes the affective engagement with BAB over the participation framework with the adults. By these means, MOM is multisensorially—through her breathy and emotionally salient tone of voice, facial expression with exaggerated lip movements and touch—painting BAB’s body with an empathetic affect. The aurally, visually and haptically elicited space is, moreover, co-chorused by NU2, who, in line 15, produces motheresetoned ‘yeah?’ Next, in line 16, NU1 still continues giving more information about the baby’s soft spot (‘it needs to be open for a long time as otherwise those brains cannot grow’) which is joined in by NU2 with

362

J. Katila et al.

a motherese ‘yea-h’ (line 17) and MOM’s silent ‘ºyea-hº’ (line 18). This indicates that the mother, too, is treating her response to NU1’s action as relevant while constructing herself as a co-participant in NU1’s affective action. The baby, for her part, responds to the affective action by remaining calm and not resisting the institutional task. Due to these multisensorial aspects of her action, NU1 is able to prioritize expressing an empathetic value towards the baby while still participating in the concurrent participation framework with the adults. Moreover, through the deployment of the “motherese” voice quality, the affective atmosphere (Brennan 2004) of the framework between NU1 and BAB extends into the framework with the adults, and also communicates to the other participants—especially the mother who may be worried about her baby’s development—that everything is all right.

10.4.2 Peruvian Mainstream Classroom: The Availability of Emotion in the Face of Sensorial Asymmetries We explore further the primacy and sharedness of emotional meaning across participation frameworks by taking as our next example interaction where there are “sensorial asymmetries” among the participants (Kusters 2017: 285), that is to say different experiences of being deaf and hearing. Although it is impossible to know exactly what sensory information another individual has access to (e.g., deaf individuals have varying hearing levels), the deaf child in Extract 2 is profoundly deaf and therefore has limited access to aural information. In Extract 2, we show that recognizing and attuning to the emotional style of another person’s utterance can provide a resource for engagement and participation in a classroom context where a deaf child does not have access to the language of instruction. Extract 2 takes place in a mainstream classroom with 25 hearing students and three deaf students in Iquitos, Peru, in 2014. The presence of deaf students in this general education classroom is part of an international movement to educate students with disabilities in general education classrooms (UNESCO 1994). Mainstream classrooms in Iquitos

10 The Primacy of Affective Engagement

363

differ significantly from most cases of mainstream classrooms with deaf students in the academic literature (e.g., Holmström et al. 2015; Ramsey 1997). In Iquitos, the majority of deaf children do not acquire the linguistic resources of any named language in their homes. They do not receive hearing-assistive technology to access spoken Spanish and are not exposed to the national sign language, Peruvian Sign Language. As a result, deaf children enter the education system relying on local signs that they develop over the course of their own lifetime (Goico 2020). Upon entering hearing, mainstream classrooms, these deaf children receive no language models to acquire an accessible language, nor receive support services to access the spoken Spanish used in the classroom (Goico 2019). Extract 2 occurs during a lecture on bullying in a health education class in September 2014. Over the course of the lecture, which began two minutes prior to the start of the extract, the teacher, Mr. Inga, has been using spoken Spanish without making any accommodations for the deaf students. Extract 2 starts just after Mr. Inga (ING) initiated a call and response sequence with the class, asking them for examples of how their parents commit bullying. Picking up on the student answers, Mr. Inga says in lines 1–2, ‘insulting you all, beating you all’, as he produces sweeping points to the class (Images 2.1 and 2.2). At this point, a deaf boy, Jeremy, (wearing stripes and marked as “JER”), who was copying the text from the board into his notebook, looks up at the board (Image 2.2). Jeremy then recognizes and capitalizes on Mr. Inga’s affective display to insert himself into the lecture. The signs seen in Extract 2 are part of the semiotic repertoire that has emerged within the classroom to communicate with the deaf boys. Signs and gestures are identified in all caps. A forward slash indicates a package of bodily actions that are produced together. Following the multimodal transcription style developed by Mondada (2019), the timing of Mr. Inga’s combination of speech and co-speech gestures or speech and signs is indicated using two tiers with the speech on the first tier and the gesture or sign on the second tier. The symbol “*” is used to indicate where in the spoken utterance, the gesture or sign starts and stops.

364

J. Katila et al.

Extract 2

As Mr. Inga lowers his arm after the second point, Jeremy turns his gaze from the board towards Mr. Inga (Image 2.3). Jeremy then launches into a moment of “byplay” (Goffman 1981: 134; M. H. Goodwin 1997). He does this by imitating Mr. Inga’s emotional performance. Jeremy moves his mouth as if speaking (although he makes no sound), produces a large pointing motion across his body, and puts on an angry facial expression to imitate Mr. Inga’s affect (Image 2.4a, b). This demonstrates his ability to recognize the emotion in Mr. Inga’s embodied action even with minimal access to Mr. Inga’s spoken Spanish. As Merleau-Ponty (1962: 184) describes vividly about the recognizability of emotion in gestures: “I do not see anger or a threatening attitude as a psychic fact hidden behind the gesture, I read anger in it. The gesture does not make me think of anger, it is anger itself ”. It is important to note that Jeremy’s action is not identical to Mr. Inga’s, moving his finger up rather than sweeping it across his body. This difference indicates that Jeremy did not

10 The Primacy of Affective Engagement

365

have access to the context-specific meaning of the point as referring to the students. Nevertheless, Jeremy was still able to recognize and re-perform the affective display behind Mr. Inga’s utterance. Although Mr. Inga’s manner of lecturing has not taken into consideration the sensory ecology of his student population, Mr. Inga’s affective display becomes an important resource for Jeremy to insert himself into the lecture. Capitalizing on this resource, Jeremy stops being a nonparticipant who was engaged in a distinct activity from Mr. Inga and the class and establishes himself as a manual-visual communicator in the classroom. As Duranti (2004: 455) writes, “The very act of speaking in front of others who can perceive such an act establishes the speaker as a being whose existence must be reckoned with”. In Extract 2 (continued), Mr. Inga chooses to engage with Jeremy’s byplay.

366

J. Katila et al.

Extract 2 (Continued)

In Extract 2 (continued), Mr. Inga enters into two simultaneous participation frameworks as he takes up and recycles material from Jeremy’s previous bodily act (C. Goodwin 2018: 1), while also continuing to lecture to the class. In line 5, Mr. Inga produces a verbal utterance similar to line 1, stating ‘hitting you all’. Gesturally, however, Mr. Inga no longer

10 The Primacy of Affective Engagement

367

produces the sweeping point, but mimics the gesture that Jeremy has just produced (Image 2.5). By dividing his body into two intertwined expressive fields of distinct specificity—the class in general and Jeremy in particular—Mr. Inga is thus able to meaningfully engage in two simultaneously unfolding participation frameworks. In line 6, Jeremy once again copies the pointing up gesture back to Mr. Inga (Image 2.6). Then, in Mr. Inga’s next turn, he no longer recycles Jeremy’s gesture, but instead, Jeremy’s exaggerated emotional performance. In line 7, Mr. Inga signs ‘TO.HIT’ with no spoken word translation (Image 2.7), followed by a tag question. Mr. Inga produces the movement in a large (the movement starts way out at his sides) and forceful (the hands make an audible noise when they come together) manner, laminated with a facial expression, where he scrunches up his eyebrows and purses his lips. Jeremy imitates back both ‘TO.HIT’ and the affective display, scrunching down his eyebrows and opening his mouth as he produces the movement (line 8, Image 2.8). As depicted in Images 2.9 and 2.10, this copycat practice continues. Importantly, this exchange unfolds spontaneously and so quickly that Mr. Inga and Jeremy “couldn’t have thought that fast” (Sacks 1992: 118), with their bodily performances occurring in overlap with one another. Mr. Inga will continue to pair individual signs with his spoken Spanish while he lectures for another 20 seconds. This form of communicating maintains the two simultaneous but distinct participation frameworks. On the one hand, Mr. Inga’s signs become part of the “composite utterance” (Enfield 2009) that the hearing students in the class have access to in the lecture. On the other hand, Mr. Inga’s signs are directed specifically to Jeremy. It is important to note that these two participation frameworks, while occurring simultaneously, are not equal (De Meulder et al. 2019: 895). Mr. Inga’s change in lecture style finally makes the classroom lecture accessible to Jeremy, but the deaf boys still have unequal access to the lecture in comparison to their hearing peers. The hearing students see Mr. Inga’s signs and affective display in addition to hearing his accompanying speech (along with additional auditory information, such as the sound of Mr. Inga’s hands slapping together), while Jeremy has limited access to aural information.

368

J. Katila et al.

This example shows how emotions, at the very basic level, can be recognizable in spite of language barriers, and that sharing an emotional attitude through co-gesturing or co-signing can be a source of interpersonal bonding (Goodwin et al. 2012; Katila and Philipsen 2019). It is crucial to note that while Jeremy was able to recognize the “angry” performance in Mr. Inga’s act and the two shared a moment of intercorporeal attunement through the use of an embedded participation framework, Mr. Inga’s signs and gestures have a very different meaning for Jeremy. For most of the lecture, Jeremy and the two other deaf boys did not have access to the lecture content, and never came to understand that the topic of the classroom lecture was bullying. Emotions such as those embodied in facial expressions have been shown to be to a large extent universally recognizable (Ekman 1984), and therefore, as seen here, meaningful affective engagement can occur without shared linguistic resources (see Extract 1 between caregivers and newborns). However, when all expressive modalities are only available to some participants, expressive asymmetry remains. Thus, this example highlights both the power and limitation of affective engagement in interaction.

10.4.3 US Surprise Party: Co-Occurring Visual and Haptic Affective Participation During a Hug In what follows, we will explore further the accessibility of affect across frameworks by exploring an interactional context which is especially rich in strong emotions: a surprise party held in honor of Chuck Goodwin. The occasion of the party was the presentation of a festschrift for Chuck (see Favareau 2018). Forty-nine scholars contributed articles demonstrating how Chuck’s work had been important in their own scholarship. The book was produced quite quickly, within three months, so that Chuck could see and read it before he died. Guests came from across the country and as far away as Japan and Singapore. Many people knew this was the very last time they would see Chuck; a rainbow of emotions from sadness to joy and gratefulness was salient in the occasion.

10 The Primacy of Affective Engagement

369

We will begin by illustrating how people mobilize their haptic configurations in order to achieve affective engagement in multiple participation frameworks. In Extract 3, we show that a hug used in the activity of greeting another in a supportive interchange (Goffman 1971: 62–94) occurs simultaneously with an expression of gratitude to the larger group of party attendees. In performing an embrace, the display of emotion is a situated practice (M. H. Goodwin et al. 2012), which unfolds both sequentially (e.g., Schegloff 2010) and simultaneously (C. Goodwin 2018). In 1979, C. Goodwin showed us how the shape of an utterance is dependent upon the way recipients attend to it throughout its course; a speaker can append new segments to an emerging sentence in search of appreciative hearership from recipients. In Extract 3, we witness how the trajectory of a hug, the ways in which it is initiated, executed, and subsequently dismantled, are closely attuned to the hug recipient’s affective involvement with other participants in the moment. As multiple frameworks can be in play simultaneously, the body may index haptic engagement with a recipient while at the same time engaging with others through visual and aural means. The voice provides not only propositional content, but also a mode of affective coloring (M. H. Goodwin et al. 2012) that attunes to and shapes the emotional atmosphere (Brennan 2004) of current activities. At the surprise party, the spatial configuration for activities constantly shifted as activities themselves emerged and changed. When Chuck (positioned to the right of the frame and marked as “CHK”) spotted a guest from Illinois (positioned to the left of the frame and marked as “NUM”) who he had not seen for some time—Numa—he delivered an enthusiastic greeting: Nu(h)ma! Wow. WOW! WO:W! (Image 3.1. and line 01–02), while directing his gaze towards him. Immediately following Chuck’s initial acknowledgment of Numa, he turned clockwise from facing Numa to his right to address the assembled group in his living room with an exclamation of gratitude (line 4): All I can say is-.

370

J. Katila et al.

Extract 3

From Image 3.1 to Image 3.2, Numa moves into a close haptic configuration with Chuck. The first segment of Numa’s hug (Image 3.2) is produced while putting his right hand on Chuck’s shoulder (with fingers spread apart), as Numa initiates his reciprocal verbal greeting to Chuck (‘It’s so good to see you’, lines 5–6). Numa aligns his body side by side with Chuck’s (Image 3.2). As Numa’s left hand is presently occupied with holding a camera and Chuck has pivoted his body from an F formation (Kendon 1990) facing Numa to an F formation facing the greater audience to his right, Numa delivers a half hug rather than a full hug. In line 7, Image 3.3, when Chuck continues with his talk to the group (I’ve never seen), Numa’s fingers are no longer spread widely apart. Now his grip is visibly tightened on Chuck’s shoulder as he lowers his head into closer proximity to Chuck’s body, while Chuck lowers his head slightly to his left, towards Numa and participants to his left. We see a progressive alteration in the spatial configuration of Chuck’s facing formation from his greeting vis-à-vis Numa (line 1, Image 3.1) to his

10 The Primacy of Affective Engagement

371

positioning vis-à-vis the larger assembled group (line 6, Image 3.2), to his lowered head (line 8, Image 3.3), and back to the group (line 11, Image 3.4, Extract 3 continued). The participation framework shifts from a dyadic greeting between Numa and Chuck to Chuck’s appreciative comment addressed to the group (lines 7–15). Numa’s carefully choreographed embodied actions, overlaid with laughter as he disengages from his haptic configuration with Chuck (line 11, Image 3.4, Extract 3 continued), clear the stage for Chuck’s undivided attention to his entire array of guests. Rhythm figures in the orchestration of affective participation as well. Numa’s disengagement from his hug is closely synchronized with Chuck’s talk, produced in three cadences in lines 7, 10 and 14–15 with nearly equal rhythmic beat structure: I’ve never seen A surprise pa(h)rty Like- like this.

Each phrase gets in return a reciprocal expression of appreciation from members of the group. While performing his unilateral departure (C. Goodwin 1987), Numa displays heightened engagement through his laughter. In response to Chuck’s ‘I’ve never seen’ Numa produces a small laugh (line 8). The laugh token in Chuck’s talk, ‘A surprise pa(h)rty (.)’ (line 10), engenders laughter not only from Numa (line 11), but also from the entire group (line 12).

372

J. Katila et al.

Extract 3 (Continued)

In response to the coda of Chuck’s talk, ‘Comple::tely (.) successful.’ (line 16), the assembly responds with enthusiastic clapping and cheering (line 17–18). Chuck then expresses his gratitude for the surprise party accompanied by a joyful smile (lines 19–20) with his ‘Thank you. THANK you. THANK you.’ After the party, numerous participants reported they felt that a form of electricity and excitement was generated by Chuck’s enthusiastic response to the party. The haptic configuration initiated by Numa to Chuck is assembled and dismantled in exquisite synchronization with Chuck’s expression of gratitude to the larger assembly. In an email communication to M. H. Goodwin with Numa about her analysis of this interaction, Numa reflected on the choreography of his hug with Chuck: As I touched Chuck’s shoulder, I was in fact intensely aware that he had to deal with lots of people simultaneously and that it was really important for me to engage and disengage from the hug as affectionately and as discreetly as possible so that I didn’t get in the way of all the other things he had to do in the moment. I silently congratulated myself at the time that I had done an OK job but had absolutely no idea how complex this event was in all its multimodal, laminated details!

10 The Primacy of Affective Engagement

373

Together Numa and Chuck produce an on-line, perfectly choreographed sequence of affectively colored utterances and haptic actions, closely coordinated with Chuck’s rhythmic phrases to the assembly. Numa’s laughter engenders group laughter and in coordination with Chuck’s expression of gratitude, there is group applause. Each co-present party member has a role in the co-production of the moment-by-moment unfolding of a shared emotional atmosphere, both influencing and being influenced by it. Thus, though some participants engaged through tactile contact while others engaged through aural and visual means, each participant was able to engage in the same affective participation framework.

10.4.4 Chinese Video Calls: Affective Participation Frameworks in Virtually Mediated Interaction With the first three extracts, we have shown the primacy of emotional participation, as well as the ability of emotion, to be recognized and engaged in through the senses of touch, vision, and voice. In the final section, we take as our case perhaps one of the most intimate relationships in everyday life, parent–child interactions, in order to examine how activity unfolds in an affective-based participation framework. Extract 4 is from a large corpus of video recordings of video calls between migrant parents and their left-behind children in China. In the context of rural-to-urban migration, many parents leave home and go to distant cities to find better paying jobs. This results in the phenomenon of left-behind children, who reside in rural areas and are brought up by their grandparents. This dataset involves children who are under three years old when their parents conduct video calls with them. The children are always accompanied by at least a grandparent during the calls. The video recordings are done through a combination of an external camera view (image 4.1a) and a screen capture of the grandparents’ phone (image 4.1b) (for a detailed description of data, see Gan 2020). In Extract 4, we illustrate the maintenance of affective engagement when the video-call participation competes with other activities—especially when a child is resisting participation in the chat with their parents. As shown by many existing studies, video calls involving young children

374

J. Katila et al.

are laborious because adults often need to conduct a lot of work to get the children’s attention to the video call (e.g., Busch 2018; Gan 2021). This example is extracted from the opening of a video call. We show how a young child deploys a multitude of affective means to display her unwillingness to join a video call—for example, by producing vocalization, using embodied resources to get out of the call. However, even though the child is engaged in competing activities, the affective engagement of this video call remains. Extract 4

10 The Primacy of Affective Engagement

375

The baby in this extract is one year old. After call ringing (line 01), the grandmother (who is the actual caregiver of the baby) picks up the call and the migrant mother appears on the phone screen (see image 4.1a and 4.1b). Openings of social encounters, including face-to-face and videomediated settings, often evoke people’s display of emotion to confirm their social relationships (Pillet-Shore 2012; Gan, Greiffenhagen and Licoppe 2020). In this example, as soon as Mom appears on the phone, she displays a smiling face and then the grandmother waves her hand. Both Mom and Grandmother attempt to produce a warm greeting, to connect parent with their left-behind child via embodied displays of love (Gan, Greiffenhagen and Licoppe 2020). As noted in line 03, the grandmother’s waving hand is not at all in the frame of camera views (image 4.2). She seems to wave “for” the baby, not “for” the mother. What the grandmother does probably serves as a scaffold for the child to attend the greeting routines. Her waving is working as a bridge between the mother and the child to call the child’s attention to this emotional event. Mom then greets the child using a term of endearment: ‘Sweetheart’ (line 04). Mom repeats her greeting in line 05, but there is no response from the child. What happens next is that the child mumbles an audible vocalization, ‘(.hh) ΩhhΩh::ΩhhhΩhhh’ (line 06 and image 4.3). While doing so, the baby moves her gaze away from the phone, she lowers her body, and uses embodied force to get out of her grandmother’s holding position (image 4.5). In response to the baby’s embodied resistance to participating in the video call, the grandmother continues to use her hand to hold the baby, and visibly uses a bit more force to hold the baby up. For example, in line 13, she says ‘Here here stand up stand up Lazybones’. She uses ‘here’ many times to call the baby’s attention to the phone. Then the baby crawls away in line 14 to play on the bed, abandoning the video-call participation. While the baby is not attentive to the video-call participation framework, we see that people still mobilize affect, body, and visual resources as methods for maintaining the affective engagement between the mother and the baby. After the baby crawls away, the grandmother comments on her crawling (line 15 and 16). The grandmother uses ‘Puppy’ to describe the baby, showing her vivid metaphor of the baby’s crawling. Importantly, we see that the mother smiles largely on the screen when seeing

376

J. Katila et al.

the baby crawling (image 4.7). In migrant families, the migrant mother may not have access to their children’s daily development. Video calling provides an opportunity for the parents to “see” their children. Here (line 17), the mother’s smiles show her engagement with her baby, displaying a moment of happiness and joy. Subsequently, the baby crawls on the bed and plays with pillows or other items. However, during the whole time, even though the baby is involved in competing activities, adults tend to render the baby’s playing as a scene to watch and to engage in. For example, in Extract 4 continued, the grandmother extends her hand to position the phone towards the baby, so that the mother can see her baby on screen. Extract 4 (Continued)

The grandmother’s positioning of the phone shows her attempts to engage the baby with her mother. The grandfather (who is out of frame) also joins in the conversation and speaks to the baby, saying ‘(Your mother) is call::ing you’ (line 81). Subsequently, Mom treats the baby’s inattentiveness as an opportunity to play with her; she issues a directive in line 82. The directive seems to address the grandmother, who can use haptic resources to “drag the baby over”. With this utterance, the mother

10 The Primacy of Affective Engagement

377

produces an audible smiling voice and displays a joyful and smiling face on screen (image 4.10). The online format allows for the emergence of different types of affective expressions and participation frameworks. Here, we can evidence that even when protesting participation, affective engagement does not necessarily end even when a key participant is inattentive. This example has demonstrated that affective engagement can remain even when the child abandons or actively resists her participation. On the one hand, we have seen that the child uses the actions of producing vocalization and crawling away (from image 4.3–4.6), to create a simultaneous solitary participation framework, which is different from the participation framework of chatting with mom in a video call, as a strategy to display her resistance. On the other hand, the adults (including the remote mother and the co-present grandmother) affectively engage in the child’s inattentiveness. In particular, we have seen that the mother displays her joy in watching the baby’s crawling through the smartphone screen. Recalling McLuhan’s metaphor, we argue that the smartphone acts as an “extension” of the mother’s body (McLuhan 1964). The mother’s emotional stance was transmitted to others in the room through her facial expressions displayed on the screen and her voice quality in the phone. Despite the child’s attempts to abandon the video-call participation framework, the affective engagement between the remote parents and child still remains. Such an affective engagement creates a space for parents to act as “a parent”, i.e., caring for their baby’s routine behaviors and development even when they are only virtually interacting with their children. The management of these simultaneous participation frameworks (e.g., the child’s crawling and the video-call framework) in this special setting (i.e., conducting video calls with migrant families), in turn, demonstrates the significant challenges in maintaining intimate relationships among long-distance family members; simultaneously parents and grandparents need to put a great amount of effort in managing concurrent participation frameworks.

378

J. Katila et al.

10.5 Discussion Across a range of settings, our analysis of interactions from diverse cultural contexts has exemplified some of the creative ways in which the participants of interaction experience and express emotions in multiparty participation frameworks (Goffman 1981). Emotion has to a large extent remained an understudied dimension among studies focusing on multilayered interactional settings (e.g., Haddington et al. 2013, 2014; Mondada 2014a, b). In this chapter, we started from the notion that emotion and affect and engaging in another person’s emotion are pivotal features of the intersubjective order (Peräkylä 2013: 552). As such, our study provides a novel contribution to understanding complexity of interaction, and especially the complexity of affect in the establishment of moment-by-moment intersubjective understanding. More specifically, we examined the relationship between affect and engagement in different participation frameworks in moments where multiple participation frameworks unfold at the same time. Our analysis finds that participants can engage emotionally in more than one participation framework simultaneously. This is achieved by the participants composing their bodies into spontaneously unfolding multisensorial fields of affective expression. While the modality of touch may be fully available only to some participants because of their positions within a close facing formation, other participants can make use of intonation, gaze, and facial expression. Touch creates a dyadic framework with the party with whom one is intertwined, and a framework through which intimacy is conveyed. One’s gaze and facial expression can be directed at a multiparty framework or restricted to a dyadic unit. Depending upon the amplitude of one’s voice, the voice can either index a close aural huddle or it can transmit into a wider participation framework. The multimodality and multisensoriality of affective expression thus enables access to other participation frameworks. We suggest that such multilayered and complex affective expressions are a pervasive feature of the organization of participation frameworks. The diversity of our cases allowed us to analyze these creative ways in which affect was made accessible across participation frameworks, and

10 The Primacy of Affective Engagement

379

how emotional engagement was being prioritized in multiparty interactions. In the first two extracts, we showed how affect can become a resource for interpersonal engagement across participation frameworks where linguistic resources are not shared between all participants. In Extract 1, an empathetic motherese tone of voice and facial expression addressed to the baby enabled an affective engagement with a newborn baby, while the content of the talk was targeted to the adult participants. In the second case, a teacher, who was not a user of a sign language, was able to engage affectively with a deaf boy through a sequence of emotionally laden gestures, while at the same time the emotion embedded in the teacher’s utterance played a role in an ongoing participation framework with the group of hearing students. In the third case, we demonstrated that participants of a multiparty interaction were able to engage in the same affective atmosphere while participating in nested participation frameworks. This was possible as the recipient of a surprise party engaged with one attendee through a hug, while simultaneously opening his body so that he could secure an aural and visual field of affective experience with the larger group of attendees. Finally, we illustrated how affective engagement can remain active even when a participant is resisting her involvement in a participation framework. The child’s vocalization and crawling away from the phone to play with pillows shows resistance to engaging in the call with mom. Despite the child’s inattentiveness, mom display joy at her child’s engagement in play. While from an analyst’s point of view, the ways in which the participants can affectively engage in more than one participation framework at the same time are extremely complex, we did not find that the participants themselves oriented to the situations as such. This is because affective engagement is not only something people “produce” using their bodies; it is also something they feel and experience. While we marvel at the rapidity with which participants can orient to changing configurations of involvement, and deploy an extremely rich ecology of semiotic resources, the intercorporeal aspect of affective engagement between the participants goes beyond semiotic resources targeted at different participants. Our analysis provides evidence that affective engagement can oscillate across participation frameworks in multiparty interactions. It

380

J. Katila et al.

is fundamentally because of this intercorporeal aspect of emotion that affective engagement has the ability to unite participants across various participation frameworks and the participants in them.

Appendix The Transcription Conventions Used in the Conversations (0.5)

Numbers in brackets indicate a time gap in tenths of a second. (.) A dot enclosed in brackets indicates a micropause of less than two-tenths of a second. = An equals sign indicates an absolute contiguity between utterances. () Single parentheses indicate an unclear utterance or another sound. .hh This indicates upward inbreath. The more repetition of “h”, the longer the breath is. hhh The letter “h” repeated with no preceding dot represents exhalation. : Colons indicate a stretching of a sound. . A full stop indicates a falling tone. , A comma indicates a continuing tone. ↑↓ Upward and downward arrows mark the overall rise or fall in pitch across a phrase. °° Hollow dots indicate a speech produced with a silent voice. Under Underlining indicates the speaker’s emphasis. @@ The “at” symbol indicates speech produced with a smiley tone of voice. (()) Double parentheses indicate the analyst’s comment. Color gray indicates information about non-vocal moves. [ is used to indicate overlap. ! indicates animated or emphatic tone.

10 The Primacy of Affective Engagement

381

Transcription Symbols for Extract 2 Signs/gestures are in all CAPS. / A forward slash indicates a package of bodily actions that are produced together. The timing of speech and gestures/signs is indicated using two tiers and the symbol ** to refer to when the gesture/signs start and stop in the speech. PT indicates a point.

References Brennan, Teresa. 2004. The transmission of affect. Ithaca: Cornell University Press. Busch, Gillian Roslyn. 2018. ‘Happy birthday Grandpa’: Using videosupported technologies in family communication. Research on Children and Social Interaction 2 (1): 74–97. Cekaite, Asta. 2016. Touch as social control: Tactile organization of attention in adult-child interactions. Journal of Pragmatics 92: 30–42. De Meulder, Maartje, Annelies Kusters, Erin Moriarty, and Joseph J. Murray. 2019. Describe, don’t prescribe. The practice and politics of translanguaging in the context of deaf signers. Journal of Multilingual and Multicultural Development 40: 892–906. Duranti, Alessandro, ed. 2004. Agency in language. In A companion to linguistic anthropology, 451–473. Malden: Blackwell Publishing. Ekman, Paul. 1984. Expression and the nature of emotion. Approaches to Emotion 3: 19–344. Enfield, Nick J. 2009. Composite utterances. In The Anatomy of Meaning: Speech, Gesture, and Composite Utterances, 1–22. Cambridge: Cambridge University Press. Favareau, Donald, ed. 2018. Co-operative engagements in intertwined semiosis: Essays in Honour of Charles Goodwin. Tartu: University of Tartu Press. Fernald, Anne. 1985. Four-month-old infants prefer to listen to motherese. Infant Behavior and Development 8: 181–195.

382

J. Katila et al.

Gan, Yumei. 2020. Choreographing affective relationships across distances: Multigenerational engagement in video calls between migrant parents and their left-behind children in China ( Doctoral Dissertation). Hong Kong, China: The Chinese University of Hong Kong. Gan, Yumei. 2021. ‘Awww, she is feeding you’: Young children’s (dis) engagement in video calls with their migrant parents. Research on Children and Social Interaction 5 (1): 80–102. Gan, Yumei, Christian Greiffenhagen, and Christian Licoppe. 2020. Orchestrated openings in video calls: Getting young left-behind children to greet their migrant parents. Journal of Pragmatics 170: 364–380. Garfinkel, Harold. 2002. Ethnomethodology’s program: Working out Durkheim’s Aphorism. Lanham, MD: Rowman and Littlefield. Goffman, Erving. 1961. Encounters: Two studies in the sociology of interaction. Indianapolis (Ind.): Bobbs-Merrill. Goffman, Erving. 1971. Relations in public: Microstudies of the public order. New York, NY: Harper & Row. Goffman, Erving. 1981. Forms of talk. Philadelphia: University of Pennsylvania Press. Goico, Sara A. 2019. The impact of ‘inclusive’ education on the language of deaf youth in Iquitos, Peru. Sign Language Studies 19 (3): 348–374. Goico, Sara A. 2020. A linguistic ethnography approach to the study of deaf youth and local signs in Iquitos, Peru. Sign Language Studies 20 (4): 619– 643. Goodwin, Charles. 1979. The interactive construction of a sentence in natural conversation. In Everyday language: Studies in ethnomethodology, ed. George Psathas, 97–121. New York: Irvington Publishers. ———. 1987. Unilateral departure. In Talk and social organisation, ed. by Graham Button and John R. E. Lee, 206–216. Clevedon, England: Multilingual Matters. ———. 2002. Time in action. Current Anthropology 43 (Supplement August– October 2002): S19–S35. ———. 2018. Co-operative action. Cambridge: Cambridge University Press. Goodwin, Charles, and Marjorie H. Goodwin. 2004. Participation. In A Companion to linguistic anthropology, ed. Alessandro Duranti, 222–243. Oxford: Basil Blackwell. Goodwin, Marjorie H. 1997. “Byplay: Negotiating evaluation in storytelling.” In Toward a social science of language: Papers in Honor of William Labov.

10 The Primacy of Affective Engagement

383

Vol. 2: Social interaction and discourse structures, ed. Gregory R. Guy, Crawford Feagin, Deborah Schiffrin, and John Baugh, 77–102. Amsterdam, John Benjamins. ———. 2017. Haptic sociality: The embodied interactive construction of intimacy through touch. In Intercorporeality: Emerging socialities in interaction, ed. Christian Meyer, Jürgen, Streeck and J. Scott Jordan, 73–102. Oxford: Oxford University Press. Goodwin, Marjorie H., and Asta Cekaite. 2018. Embodied family choreography: Practices of control, care, and mundane creativity. Oxford and New York: Routledge. Goodwin, Marjorie H., Asta Cekaite, and Charles Goodwin. 2012. Emotion as stance. In Emotion in interaction, ed. Marja-Leena Sorjonen and Anssi Peräkylä, 16–41. Oxford: Oxford University Press. Haddington, Pentti, Lorenza Mondada, and Maurice Nevile, eds. 2013. Interaction and mobility: Language and the body in motion. Berlin: De Gruyter. Haddington, Pentti, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile. 2014. Towards multiactivity as a social and interactional phenomenon. In Multiactivity in social interaction: Beyond multitasking, ed. Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 3–32. Amsterdam: John Benjamins. Harkness, Nicholas. 2015. The pragmatics of qualia in practice. Annual Review of Anthropology 44: 573–589. Hertenstein, Matthew J. 2002. Touch: Its communicative functions in infancy. Human Development 45: 70–94. Holmström-Gupta, Ingela, Bagga Sangeeta, and Rickard Jonsson. 2015. Communicating and hand(ling) technologies. Everyday life in educational settings where pupils with cochlear implants are mainstreamed. Journal of Linguistic Anthropology 25 (3): 256–284. Homanen, Riikka. 2013. Doing pregnancy, the unborn, and the maternity healthcare institution, Academic Dissertation. Tampere University Press: Tampere. Husserl, Edmund. 1982. General introduction to a pure phenomenology, trans. F. Kersten. The Hague: Martinus Nijhoff. Jefferson, Gail. 2004. Glossary of transcript symbols with an Introduction. In Conversation analysis: Studies from the first generation, ed. Gene H. Lerner, 13–23. Philadelphia: John Benjamins. Joaquin, Anna Dina L., and John H. Schumann, eds. 2013. Exploring the interactional instinct. Oxford: Oxford University Press.

384

J. Katila et al.

Katila, Julia. 2018. Tactile intercorporeality in a group of mothers and their children: a micro study of practices for intimacy and participation, Academic Dissertation. Tampere: Tampere University Press. Katila, Julia, and Johanne S. Philipsen. 2019. The intercorporeality of closing a curtain: Sharing similar past experiences in interaction. Pragmatics & Cognition 26: 2/3: 167–196. Katila, Julia, and Sanna Raudaskoski. 2020. Interaction analysis as an embodied and interactive process: Multimodal, co-operative, and intercorporeal ways of seeing video data as complementary professional visions. Human Studies 43: 445–470. Kendon, Adam. 1990. Conducting interaction: Patterns of behavior in focused encounters. Cambridge, New York, Port Chester, Melbourne & Sydney: Cambridge University Press. Kusters, Annelies. 2017. Gesture-based customer interactions: Deaf and hearing Mumbaikars’ multimodal and metrolingual practices. International Journal of Multilingualism 14 (3): 283–302. Levinson, Stephen C. 2006. On the Human “Interaction Engine”. In Roots of human sociality: Culture, cognition and interaction, ed. Nick J. Enfield, and Stephen C. Levinson, 39–69. Oxford & New York: Berg. McLuhan, Marshall. 1964. Understanding media: The extensions of man. New York, NY: McGraw-Hill Education. McIlvenny, Paul, Broth Mathias, and Pentti Haddington. 2009. Communicating place, space and mobility. Journal of Pragmatics 41: 1879–1886. Merleau-Ponty, Maurice. 1962. Phenomenology of perception. London: Routledge. Meyer, Christian, Jürgen Streeck, and J. Scott Jordan, eds. 2017. Introduction. In Intercorporeality: Emerging socialities in interaction, xiv–xlix. Oxford: Oxford University Press. Mondada, Lorenza. 2014a. The local constitution of multimodal resources for social interaction. Journal of Pragmatics 65: 137–156. Mondada, Lorenza. 2014b. The temporal orders of multiactivity: operating and demonstrating in the surgical theatre. In Multiactivity in social interaction: Beyond multitasking, ed. Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 33–75. Amsterdam: John Benjamins. Mondada, Lorenza. 2019. Conventions for transcribing multimodality. Version 5.0.1. Available from https://www.lorenzamondada.net/multimodal-transc ription.

10 The Primacy of Affective Engagement

385

Peräkylä, Anssi. 2013. Conversation analysis in psychotherapy. In Handbook of conversation analysis ed. by Jack Sidnell, and Tanya Stivers, 551–574. West Sussex: John Wiley & Sons Incorporated. Peräkylä, Anssi, Liisa Voutilainen, Melisa Stevanovic, Pentti Henttonen, Mikko Kahri, Maari Kivioja, Emmi Koskinen, Mikko Sams, and Niklas Ravaja. 2021. Emotion, psychopsysiology and intersubjectivity. In Intersubjectivity in action: Studies in language and social interaction, ed. Jan Lindström, et al., 303–327. Amsterdam/Philadelphia: John Benjamins Publishing Company. Peräkylä, Anssi, and Johanna Ruusuvuori. 2012. Facial expression and interactional regulation of emotion. In Emotion in Interaction, ed. Marja-Leena Sorjonen and Anssi Peräkylä, 64–91. Oxford: Oxford University Press. Pillet-Shore, Danielle. 2012. Greeting: Displaying stance through prosodic recipient design. Research on Language & Social Interaction 45 (4): 375–398. Ramsey, Claire L. 1997. Deaf children in public schools: Placement, context, and consequences, vol. 3. Washington, D.C: Gallaudet University Press. Ruusuvuori, Johanna. 2013. Emotion, affect and conversation analysis. In Handbook of conversation analysis ed. Jack Sidnell, and Tanya Stivers, 330–349. John Wiley & Sons Incorporated, West Sussex. Sacks, Harvey. 1992. Lectures on Conve##rsation, vol. 1. Oxford: Blackwell. Schegloff, Emanuel A. 2010. Commentary on Stivers and Rossano: Mobilizing response. Research on Language and Social Interaction 43 (1): 38–48. Schegloff, Emanuel A. 1998. Body Torque. Social Research 65 (3): 535–596. Sierra, Sylvia. 2017. “Buffy sings to Cody”: A multimodal analysis of mother and pre-lingual-infant question-response sequences. Journal of Pragmatics 110: 50–62. Stern, Daniel. N. 2002[1977]. The first relationship: Infant and Mother. Cambridge and London: Harvard University Press. Throop, C. Jason. 2014. Moral moods. Ethos 42 (1): 65–83. Trevarthen, Colwyn, and Kenneth J. Aitken. 2001. Infant intersubjectivity: Research, theory, and clinical applications. Journal of Child Psychiatry 42 (1): 3–48. Tronick, Ed.Z. 1989. Emotions and emotional communication in infants. American Psychologist 44 (2): 112–119. United Nations Educational, Scientific and Cultural Organization. [UNESCO] 1994. The Salamanca Statement and Framework for Action on Special Needs Education. World Conference on Special Needs Education: Access and Quality. Salamanca, Spain.

Part IV Complexity That Resides in the Characteristics of Interactional Settings and Environments

11 Ambulatory Openings Elliott M. Hoey

11.1 Background The practical task of opening an interaction has been a long-standing interest in conversation analytic work since pioneering investigations of telephone openings by Schegloff (1968, 1986) and has attracted renewed interest in recent years (e.g., Pillet-Shore 2018, Haddington et al., this volume). Mobile practices in and for social interaction have similarly come to prominence (e.g., Haddington et al. 2013), building on early ethnomethodological studies of walking and navigation (Ryave and Schenkein 1974; Psathas 1976). This chapter is situated in these two streams of work and contributes an account of ambulatory openings in a complex workplace setting, the construction site (see below). E. M. Hoey (B) Department of Language, Literature, and Communication, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_11

389

390

E. M. Hoey

11.1.1 Openings The fundamental problem for would-be interactants is how to enter into mutually ratified participation in a shared activity, or how to transition from a state of non-interaction into a state of “focused interaction” (Goffman 1963). This process in telephone openings appears as a regular sequence of actions, notably starting with a summons-answer sequence (Schegloff 1968). But with mobile participants within one another’s perceptual reach, a lot more interactional work goes on before a first turn is uttered. Prior research on mobile openings (e.g., Mondada 2009, 2014; Mortensen and Hazel 2014) has revealed a regular sequential organization, whereby participants manage the partially overlapping practical matters of displaying (un)availability for interaction, projecting interaction, and coordinating the establishment of a shared interactional space. The problem of availability for interaction is a matter of “sizing up” another person with respect to their preparedness or inclination to enter into focused interaction. Practices used at this stage are largely to do with persons’ “body idiom” (Goffman 1959, 1963; Heath 1986), or what can be apperceived regarding what each person is attending to, whether they are stationary or mobile, what objects they are wielding or acting upon, with whom they are engaged, etc. Studies of walking, for instance, have argued for the constitutive accountability of navigational practices in partitioning shared social spaces into legible streams of movement and pockets of activity, and in rendering mobile persons into vehicular units of various sizes (Goffman 1971; Ryave and Schenkein 1974; Livingston 1987; Watson 2005). Studies of visual and gaze-related practices have shown that would-be interactants at this pre-beginning phase of an interaction are routinely engaged in modulating their accessibility to the naked senses (Goffman 1963; Kendon and Ferber 1973), and in identifying and categorizing potential participants (Mondada 2009; De Stefani and Mondada 2018; Harjunpää et al. 2018; Haddington et al., this volume). In addition to practices of walking and gaze, the management of availability also involves gaining admission (Pillet-Shore 2018), for instance with a “ticket” that supplies the right to approach someone for interaction (Sacks 1992) or even physically gaining admission past a

11 Ambulatory Openings

391

door or other barrier to entry (Tuncer and Licoppe 2018; McLauchlan and Noble 2019). A next practical issue for those who are mutually available for interaction is projecting the occurrence of interaction. This often first requires establishing mutual acknowledgment, such as through “catching their eye” (Goffman 1963; Goodwin 1981; Duranti 1997; Clark and Pinch 2010), or, in the absence of reciprocal orientations, the use of a summons (Schegloff 1968, 2007). With the establishment of mutual acknowledgment, mobile participants routinely prepare to move into an interaction. Bodily-material practices at this stage include disengaging from prior/ current involvements (Mortensen and Hazel 2014) and coordinating movement to closer proximity (Kendon and Ferber 1973; GonzálezMartínez et al. 2017; De Stefani and Mondada 2018). And when participants are within some normatively regulated distance, they may project an opening turn through pre-beginning devices (Schegloff 1996) like uhm (Mondada 2009). A final set of practices regularly observed before and during an opening turn-at-talk are directed toward the configuration of an “interactional space” (Mondada 2009). The notion of an interactional space draws attention to the spatial distribution and configuration of participants’ bodies relative to one another and to their material environment, to the collaborative emergence of those arrangements, and to the ongoing constitution of a dynamic “here and now” through participants’ orchestration of multimodal resources. The emergence of this space typically involves the (re-)configuration of participants’ bodies into reciprocal or vis-à-vis orientations (Goodwin 1981, 2000; Heath 1986; Kendon 1990; Schegloff 1998). Such configurations display availability and preparedness for interaction and may be accompanied by practices like putting away items (Mortensen and Hazel 2014) or adopting a “waiting” posture (Svinhufvud 2018). Investigations of mobility and openings have largely analyzed the beginnings of new or first time interactions. By contrast, the ambulatory openings under examination in this chapter are not openings in the strict sense; the majority of cases are not participants’ first encounters of the day (and hence do not include things like greetings). Rather, they are re-openings or re-initiations after an interregnum of non-interaction

392

E. M. Hoey

like a lapse in conversation (Hoey 2020). Such re-openings are common to certain workplaces, where interaction is optional or occasioned rather than serving as the main remit of an encounter. Such settings have been described as ongoing states of incipient talk (Schegloff and Sacks 1973) or open states of talk (Goffman 1981)—formulations which capture the optionality of talk for the recognizable progression of the participants’ activities. Openings in these settings have been less studied, though recent work as addressed initiating talk in open offices (Salvadori 2016) and during laborious tasks like mucking out a sheep stable (Keevallik 2018). This article extends research on re-openings in settings where interaction is occasioned and sporadic by examining mobile practices for re-opening interactions in construction work activities.

11.1.2 Complexity of Interaction in Construction Work Construction work may be considered descriptively complex because it characteristically involves multiactivity (Haddington et al. 2014), material objects (Nevile et al. 2014; Day and Wagner 2019), and mobile participants (Haddington et al. 2013). It thus offers a perspicuous setting for the examination of openings due to the practical complexities they introduce to that task. One sort of practical complexity emerges from the fact that an interaction may be started by any worker at any time for any reason. This is in part provided for by the workers’ accessibility. Workers are accessible in that they conduct their work largely out in the open, naked to the perception of others (see also Herrle 2015). This contrasts with other workplace settings that limit accessibility, for example, where office workers are inside offices and/or behind closed doors (Tuncer and Licoppe 2018). The visibility of others and their activities allows for decisions to be made about how and when to initiate interactions. In addition to being accessible, construction workers are addressable, “each having the right to initiate and the duty to accept an encounter with the other” (Goffman 1963: 131). There is an institutionally provided ticket to interaction, in other words. For the data under

11 Ambulatory Openings

393

examination (see below), there were about 100–200 workers were on site on any given workday. They were organized into various crews (for ironwork, cement, carpentry, etc.) and managed by a host of foremen, inspectors, and managers. Any of these workers could address another, regardless of whether they were on the same crew or whether they even knew one another. Openings were always possible given the manifold interdependencies of construction work tasks, the omnirelevance of alerting others to physical danger, and the continuous need to circulate updates and maintain awareness of ongoing work (see Heath et al. 2002). Another practical complexity introduced to the work of openings is tied to the material environment. The construction site is a continuously changing landscape in various phases of destruction, modification, and development. It is the self-same background against which work takes place and the very object of that work. The large size of jobsite under examination (over 25,000 m2 ) meant that there were numerous areas for work activities to occur. Its size also virtually ensured the frequent movement of workers around the site. As workers navigated their surroundings, they would find little in terms of stable “architectures for interaction” (Hausendorf and Schmidt 2016). That is, on many construction sites the features of the material environment only weakly organize the landscape: there are few natural boundaries like curbs and pathways to channel movement or designate places to stop and few “places for” particular interactions. Such weakly structured environments do not pre-specify where interactions should occur, the number of parties involved, how they are to arrange their bodies, and what they are to talk about. Contrast this to heavily structured settings like a movie box office, which ordinarily restricts interaction to two parties, restricts the possible configurations of interactional space, and restricts the relevant range of activities (Jucker et al. 2018). Construction work activities thus present a distinct arrangement of practical problems for the accomplishment of openings. Workers’ mobility, accessibility, and addressability, compounded by the weakly structured character of the material setting, make for a relatively high degree of indeterminacy for potential interactants. Therefore, a constitutive feature of such work is managing a situation where anyone could address anyone else from any direction at any time for any reason. This

394

E. M. Hoey

chapter will argue that a partial solution to this problem is embodied by how one participant approaches another when beginning an interaction. Central to this chapter, then, are the methodical procedures used by ambulatory participants in the course of opening an interaction and the workers’ orientations to the details of a body in motion.

11.1.3 Data The data for this chapter are approximately 100 hours of video recordings of various work activities on a building construction site in California in 2018. The project was a large residential building in the initial stages of development, around the time when the foundation is laid and columns are erected. The recordings were collected for research purposes with the informed consent of the participants, whose identifying information has been changed in the transcripts. The predominant languages used on site were American English, Mexican Spanish, and Latin American Spanish. Transcription of verbal and vocal conduct follows Jefferson (2004) and indications of body-visual behavior follows Mondada (2018). The analysis below is based on a collection of 80 openings involving an ambulatory party and a (generally) stationary party, identified in 10+ hours of recordings of construction work activities. Openings are utterances that provide for the start of an interaction, where the speaker and addressee(s) were just previously not engaged. The typical opening in my collection is a sequence-initiating action which occurs after a lapse in focused interaction, is directed at a co-present addressee, and concerns the speaker’s current or pending work activity. A few cases (n = 14) diverged from this typical instance: openings that were unrelated to work tasks (n = 7); openings mediated by walkie-talkie where the participants were distant but still mutually visible (n = 3); actions that did not oblige a response and only contingently initiated interaction (i.e., not designed as sequence-initiating actions; n = 2); and openings that constituted the participants’ first encounter of the day (i.e., not occurring after a lapse; n = 2). These cases were analyzed using conversation analytic methods (Hoey and Kendrick 2018).

11 Ambulatory Openings

395

11.2 Analysis The analysis documents three ways that an ambulatory party approaches another participant for interaction: in direct, oblique, and restricted approaches.

11.2.1 Direct Approach A direct approach is perhaps the most straightforward ambulatory method for creating some interactional space. With this method, a walking party sets and follows a trajectory which terminates at the addressee. This has the recognizable character of going to the addressee for the purpose of interaction and even of having “sought them out”. In this way, a direct approach serves as a kind of progressively emergent summons; as the ambulatory party comes nearer, it becomes increasingly recognizable that their approach is “for” the (eventual) addressee. In Excerpt 1, Jamie walks in a direct trajectory toward Ryan and Chuy, who are conducting a relatively stationary task of rolling out a ream of tarp over gravel (Fig. 11.1a). Jamie is five paces away when Ryan spots him approaching. Excerpt 1 (R1_010623)

396

E. M. Hoey

Fig. 11.1

Jamie approaches Ryan and Chuy directly

What Ryan would be able to see in Jamie’s approach is that Jamie is bodily oriented to him through gaze, head, torso, and trunk. This restricts the set of possible endpoints of Jamie’s trajectory. Ryan could also see that Jamie is aiming to pass neither behind them, thereby respecting the transactional space (Kendon 1990) where he and Chuy are working, nor in front of them, given the concrete wall that prevents such passage. The projectable endpoint of his path is thus seeably located at Ryan and Chuy. Ryan’s look toward Jamie also establishes mutual gaze and co-presence (Goffman 1963), furnishing an incipient interactional space. That is to say, Jamie’s direct approach works much like a summons (Schegloff 1968) to the extent that the projectable initiation of interaction is “given off ” (Goffman 1959) by the ambulatory party. Jamie’s approach visibly alerts Ryan to the likelihood of an upcoming interaction in the same way that the sound of footfalls or a jingling bell might audibly alert a shopkeeper to the entrance of a customer. Moreover, Jamie’s approach occasions from Ryan a display of availability in the form of gazing back, thereby furnishing an “answer” of sorts to Jamie’s “summons”.1 Jamie orients to Ryan’s availability for talk by asking, when two and a 1

There is other evidence for viewing direct approaches as akin to summonses. For one, like canonical summonses (“hey” + address term, a door knock, telephone ring, etc.), they are directed to establishing the approached party’s availability for talk. Additionally, they can occasion displays of availability like an eyebrow flash or “what?”. And conversely, they can occasion displays of unavailability like “not right now” or, in a service encounter, “I’ll be right with you” (see Harjunpää et al. 2018), or, to a street vendor, “no thank you” (see Llewellyn and Burrow 2008).

11 Ambulatory Openings

397

half steps away, “hey why do they put this?” (line 2, Fig. 11.1b) referring to the tarp. Note the absence of a dedicated summons here.2 Its absence orients to an already-established speakership and recipiency, part of which derives from the recognizability of Jamie’s direct approach. The temporal placement of Jamie’s turn also orients to the parties’ increasing proximity and their emergent L-shaped formation (Ciolek and Kendon 1980). He positions his question such that by the time it ends the parties have stabilized an interactional space (Fig. 11.1c; Mondada 2009). By starting to speak before reaching them, and by coordinating the end of his question with the end of his walking trajectory, Jamie shows himself not only as addressing his talk to them, but as having approached them for that purpose (cf. Sorjonen and Raevaara 2014 on a similar phenomenon in kiosk interactions). A direct approach like this regularly has a practical motivation. Put simply, some activities are ordinarily or more easily done when interactants are near one another. This is seen in this example. The stabilization of an L-shaped formation results in having the tarp as their common center of focus. This, in part, is how Jamie’s use of the deictic “this” (line 2) comes to indexically refer to that tarp. Referring to some object is easier when it is already in common ground. The next case shows a similar practical motivation: making a direct approach provides for the transfer of an object (cf. Tuncer and Haddington 2020). The transcript begins as Gerardo and Tomas measure the distance between scaffolds—Tomas holding one end of measuring tape and noting the length while Gerardo holds the other end in place (Fig. 11.2a). While they are engaged in this measuring task, Omar and Ryan approach Gerardo from behind. They are seeking him out specifically because Ryan was looking for his ratchet tool, and Omar believes that Gerardo has it.

2

The turn-initial “hey” is not a summons turn, but a through-produced summons that is immediately followed by an initiating action. While it is not solely directed to securing the availability of its recipients, Jamie’s turn-initial summons does orient to Ryan and Chuy’s ongoing work task by flagging his coming talk as in some way divergent from that task.

398

E. M. Hoey

Excerpt 2 (R2_012730)

11 Ambulatory Openings

399

Fig. 11.2 (a) Omar and Ryan approach Gerardo from behind; (b) Tomas retracts tape measure, Gerardo turns to face Omar and Ryan; (c)–(d) Omar and Ryan near and establish proximate formation

While Omar and Ryan approach, Tomas turns around to face Gerardo and retracts the measuring tape (Fig. 11.2a–b). Tomas’ actions visibly and haptically indicate the completion of the measuring task and make relevant a release of Gerardo’s end of the measuring tape.

400

E. M. Hoey

The visible completion of their measurement task appears to occasion Omar’s summons turn, OYE PAPA! “Hey dude!” (line 2). Omar’s verbal summons treats Gerardo as having been unavailable for interaction and its placement at this moment of task completion orients to his incipient availability. Gerardo responds by immediately dropping the measuring tape and turning around, an embodied display of disengaging from one interaction to attend to another. Compared to the prior case, the opening turn here is a verbal summons. A first practical task for Gerardo, having heard this summons, is to ascertain its intended referent. It is possible that papa “dude” is not addressed to him. In solving this problem, Gerardo may use the acoustic resources unavoidably given off in the shouted summons, which indicate the summoner’s location, distance, and perhaps identity. These tell him where to look and potentially whom to look for. Upon turning around, Gerardo will see Omar and Ryan approaching him roughly eleven paces away. Their direct approach can serve to disambiguate potential addressees by indicating through body orientations whom the summons was for. Moreover, their bodily orientations to Gerardo reveal an understanding of the summoner’s practical task in this sequential environment—that is, summoner’s should make themselves conspicuous and locatable, rather than hidden from the summonsrecipient (see Ayaß 2020 on doing “waiting”).3 Seeing one another, they become co-present as mutually ratified interactants. Both parties now share an understanding of papa “dude” as referring to Gerardo, and of his turning around to face them as the conditionally relevant response to the summons. Another practical task of the summons-recipient is to begin generating possible reasons for the summons. Again, a direct approach can indicate some answers here. Gerardo will see that Omar and Ryan continue to approach him without stopping. Their movement prefigures the formation of a close interactional space, and thereby, implicates the range 3 Compare to the characteristically teenage prank “ding dong ditch”, where one party rings the doorbell of a house (ding dong) and then promptly flees (ditch). The impish enjoyment of this trades on the expectation that summoning party remains present and that their unexpected absence will generate accounting procedures by the summoned party, procedures which may result in frustration, befuddlement, anger, etc.

11 Ambulatory Openings

401

activities that are ordinarily transacted in close range. It is not the case, for instance, that Gerardo is being summoned to walk over to them, or that they will conduct their conversation at a distance (see Excerpts 6–7). And indeed, as the parties establish a close formation (Figs. 11.2c– d), Omar inquires after Ryan’s ratchet (lines 4–8), which is eventually returned to him (not shown). These cases show that participants following a direct trajectory to a stationary party visibly project the formation of an interactional space and thus an interaction. The walker’s bodily orientation serves to designate a “natural endpoint” at the stationary party, and their progressive advance toward that endpoint can contribute to the emergent visibility of the coming interaction. The placement of an opening turn prior to reaching the addressee is a recurrent feature of such approaches. It’s one way that an ambulatory party can show themselves as approaching their addressee for the purpose of starting an encounter. The direct approaches shown here are practiced solutions to the manifold complexities of opening interactions on the construction site in that the ambulatory party can be seen as walking to the stationary party for the purpose of interaction-in-close-proximity, which affords certain kinds of interactions, such as object exchange. The recognizability of a direct approach, and in particular its projection of interaction, is itself usable by participants. It provides a systematic and embodied basis for “anticipatory openings”, which I turn to next.

11.2.1.1 Anticipatory Openings It was argued above that a direct approach projects an interaction by reference to the imminent formation of an interactional space. The approach may serve as a kind of progressively emergent summons to the extent that it can alert an approached party to the possibility of a coming interaction and thereby occasion some display of availability. It was also suggested that a summons warrants the generation of accounts for the coming interaction. These features underlie the operation of anticipatory openings (Hoey, forthcoming). In such openings, an individual sees that they are being

402

E. M. Hoey

approached, sees who is approaching them, and guesses at the substance of what is to come.4 The approached party issues the first utterance, the design of which anticipates the approaching party’s reason-for-theinteraction. Participants can anticipate, for instance, the kind of action that an approaching party is likely to produce, as in the next example. Here, Jamie and Horacio are aligning a large rebar “form” (Fig. 11.3a). They’ve been working on this task for hours, but have made little progress. Their supervisor Pablo is briskly walking toward them, and when about 20 feet away, he is spotted by Jamie, who gazes in his direction (Fig. 11.3b). Excerpt 3 (JL_043510)

4

The terminological allusion to anticipatory completions (Lerner 1996) is deliberate. “Anticipatory completions […] interchange the opportunities to produce initiating and responding actions within a sequence” (1996: 308). Similarly, anticipatory openings interchange opportunities for producing a responding to an opening turn.

11 Ambulatory Openings

403

Fig. 11.3 (a) Pablo directly approaches Jamie and Horacio; (b) Jamie spots Pablo approaching; (c) Pablo nears

Jamie’s gaze to Pablo establishes an incipient interactional space for the encounter that was projected by Pablo’s direct approach. In accounting for Pablo’s approach, Jamie could relevantly use the membership categorization device “supervisor-supervisee”. In this way, he could see Pablo as “my supervisor”, whose category-bound activities include delegating tasks to supervisees, monitoring their progress, watching for errors, etc. In this case, Pablo would be able to see Jamie and Horacio engaged in a task that he had assigned them hours ago, he could see the form and assess its alignment, he could see their lack of progress in the task, and he could begin to identify mistakes that they have made. Jamie, in seeing what is available to Pablo at a glance, can thus anticipate that a complaint may be in the offing. Before Pablo takes the opportunity to initiate the interaction, Jamie produces the anticipatory opening “get outta here please” (line 3). This directive is an effort to preempt the looming interaction (see Mondada 2022). It confirms the analysis of this scene as one in which Jamie sees Pablo as coming to them (rather than walking elsewhere) for the purpose of interaction (rather than, say, observation). Jamie’s anticipatory opening embodies his analysis of what Pablo was coming there to do. In directing him to “get outta here please”, Jamie orients to Pablo’s reason-for-the-interaction by undermining the relevance of an encounter at all. Jamie’s diagnosis turns out to be correct: as the interaction proceeds their exchange is marked by antagonism. In response to Jamie’s directive, Pablo gives a prompt and unmitigated refusal (line 4). And he does this while continuing to approach, which embodies his defiance of Jamie’s directive. Pablo then does what he had presumably come there to do: he

404

E. M. Hoey

issues a harsh complaint about their lack of progress (line 6), one which Jamie forcefully rejects (line 8). Anticipatory openings such as this show that participants understand the imminent formation of an interactional space as providing for interaction. A direct approach can serve as a summons and the identifiability of the approacher can provide for the potential substance of what is to come. This case also shows the accounting procedures that participants use in formulating a reason-for-the-coming-interaction. In the construction workplace context, such procedures rely on methods of categorizing the approaching party by reference to things like activity progress, occupational roles, and time of day. The production of an anticipatory opening interchanges the ordinary roles whereby the approaching party initiates interaction and the approached party responds. This reversal of roles is what provides for the possibility of preempting an anticipated action, as in Excerpt 3.

11.2.2 Oblique Approach We also find approaches where an ambulatory party walks in the general direction of a stationary party, but then continues past them. Such an oblique approach does not terminate at the addressee, but recognizably leads elsewhere. Physical proximity of one person to another can scaffold an embodied participation framework and provide for a focused encounter (e.g., Goffman 1963, 1971). But of course talk need not emerge. The ambulatory party may simply acknowledge the other with a glance, nod, or smile, or may even pass without a word. With an oblique approach, an ambulatory party can project minimal or no interaction to come. Part of the recognizability of this inheres in the walker’s maintenance of a steady pace as they physically near the addressee, the retention of a bodily orientation that is directed past the the addressee, and an engagement in limited mutual gaze. These features are present in the case below. Here, Fernando is traversing the jobsite. His path is indicated in Fig. 11.4a. His trajectory leads him in the direction of Ben, who stands still while gazing downward at a document.

11 Ambulatory Openings

405

Excerpt 4 (F_010320)

Fig. 11.4 (a)–(b) Fernando walking in Ben’s direction; (c) Fernando and Ben greet each other as Fernando passes; (d) Ben takes step backward and Fernando continues onward

406

E. M. Hoey

As Fernando walks by Ben, they greet one another (lines 2–3). According to Goffman (1971), such “passing greetings” occur when “two acquaintances pass close by each other on their separate daily rounds in consequence of what is seen as the routine intersecting of their activities” (1971: 75). The significance of such greetings for Goffman is in their use as a “maintenance ritual” for a relationship, serving to instantiate the routine expectedness of the encounter. Such a description, while not inaccurate, loses something of the phenomenon itself: the practical work of accomplishing passing and greeting, in this case via an oblique trajectory of movement. Both participants organize this moment as one of passing greetings through their bodily behavior and placement of talk. Fernando recognizably conducts himself such that he is visibly not approaching Ben for interaction or otherwise implicating any sort of action from him. He maintains a steady walking pace and keeps his head/gaze oriented in his direction of movement (Fig. 11.4b). This “away-oriented walking” (González-Martínez et al. 2017) visibly projects continuing along his trajectory rather than stopping. Fernando is not likely to be seen, for example, as “searching for someone”, which might involve scanning the landscape and a slower, less direct pace—conduct which may be seen as potentially searching for Ben. Complementarily, Ben does not orient to Fernando’s approach as implicating an interaction. He does not gaze up from his document nor reposition himself in anticipation of Fernando’s arrival (see Mortensen and Hazel 2014). Rather, Ben observes “civil inattention” (Goffman 1963) and is therefore seeable as keeping to himself rather than receiving or welcoming Fernando. When Fernando is about two paces away from Ben, both participants issue greetings in overlap (lines 2–3). These are recognizably passing greetings rather than greetings prior to more sustained engagement. For one, they are positioned relatively “late” in the course of Fernando’s approach. In other cases in my collection, like Excerpt 2, it is common for a greeting or summons to appear prior to coming within close proximity. And so the co-occurrence of their greetings right as Fernando passes before Ben gives evidence that these are to be heard as greetings and no more than that. Additional evidence for this is seen in their bodily conduct. As Fernando continues walking past Ben, Ben “makes

11 Ambulatory Openings

407

way” for Fernando by stepping backward to allow him to pass with greater clearance (Figs. 11.5c–d). This demonstrates his understanding of Fernando’s trajectory as aimed past him rather than at him, and reflexively constitutes their interactional space as a transient one. While the participants in the case above were in alignment regarding the obliqueness of Fernando’s approach, there are other cases where an oblique approach is less recognizable as such. That is to say, the emergence of an oblique approach as compared to a direct approach can be interactionally contingent. In the next case, there is an initial ambiguity as to whether the ambulatory party’s approach is direct or oblique. It begins with Ryan gazing upward (Fig. 11.5a). He is working with the crane operator, who is seated in a tower high in the sky. The two are coordinating to remove a panel with the crane—Ryan giving directions from the ground and monitoring as the crane operator fulfils those instructions (see Urbanik 2021). As this is going on, another worker, Isidro, enters the scene (Fig. 11.5b). Excerpt 5 (R_024530a)

408

E. M. Hoey

Fig. 11.5 (a) Ryan looks up at crane/panel; (b) Isidro enters setting; (c) Ryan sees Isidro, Isidro looks up at crane/panel; (d) Ryan returns gaze to crane/panel, Isidro approaches; (e) Isidro gives suggestion to Ryan in passing; (f) Isidro walks away

After directing the crane operator by walkie-talkie to “keep trolleyin out” (i.e., laterally move the panel away from the deictic center, line 2), Ryan looks over to Isidro, who is around five feet away (Fig. 11.5c). He would see at least two things at a glance. First, he would see that Isidro is carrying a floor scraper. This object is a material indication of Isidro’s activity. In this context, the floor scraper associates Isidro with some other activity in some other part of the jobsite. Isidro can thus be seen as “transporting” the tool elsewhere, which prefigures some other destination for him. Second, Ryan would also see Isidro gazing up at the crane/panel (Fig. 11.5c). This establishes a common focus; both have now seen the other gazing at the same thing. Their shared interest in this object introduces the possibility of talk about it. Moreover, Isidro’s gaze

11 Ambulatory Openings

409

to the crane/panel invokes his membership as “someone who also has expertise in this kind of task”. There is thus the possibility of interacting with respect to what it is that they both see. Ryan orients to this possibility by keeping his gaze on Isidro, perhaps monitoring for any indication that talk is forthcoming. Isidro, however, does not gaze to Ryan—something that would suggest forthcoming talk—but instead gazes downward and continues walking in a trajectory that runs obliquely to Ryan (Fig. 11.5d). Seeing this, Ryan gazes back up to the crane/panel (Fig. 11.5d). He evidently takes Isidro’s movement to mean that no interaction is forthcoming, and that whatever it was that Isidro saw, it did not warrant any discussion. As it turns out, however, Isidro does have something to say. As he approaches Ryan, he maintains an oblique trajectory (i.e., he doesn’t directly approach or turn to face Ryan). Similarly, Ryan doesn’t treat Isidro’s approach as implicating interaction; he keeps his gaze/head upward. But as Isidro passes before Ryan (Fig. 11.5e), he turns his head slightly toward him and suggests that he “loosen it down a little bit” (line 4). Like the greetings examined above, this suggestion is given “on the move” (cf. González-Martínez et al. 2017). Isidro does not slow his pace when making this suggestion, but walks away (cf. Broth and Mondada 2013). By maintaining his trajectory, Isidro forgoes stabilizing an interactional space with Ryan. His manner of walking does not provide for the possibility of sequence expansion (Schegloff 2007), for example, through resistance to his suggestion. And indeed, Ryan treats the suggestion as valid in accepting and implementing it (line 5). This case shows how an oblique approach may not be recognizable as such until the parties are in relatively close proximity. While there is not a precise distance at which an ambulatory party is seeable as walking “to me” or “past me”, there are practices of gaze and locomotion by which participants can treat one another as possible and likely interactants. An oblique approach is designedly transitory—both in the walker’s conveyance past the addressee and in the projection of a brief interaction. With it, the ambulatory party can prefigure non-convergence by maintaining their pace, trajectory, and body orientation as they near the stationary party. The away-oriented movement of the speaker and the careful placement of talk at the moment of passing imparts to these

410

E. M. Hoey

exchanges a “drive by” character and disfavors sequence expansion. As for the practical complexities of opening interactions on the construction site, oblique approaches examined above show the ambulatory party’s destination as “elsewhere”, which implicates a relatively brief and even incidental interaction with the stationary party. Another kind of non-convergent approach is examined in the next subsection—a restricted approach. But whereas with an oblique approach the participants come into close proximity, with a restricted approach some distance is maintained.

11.2.3 Restricted Approach With a restricted approach, an ambulatory party walks in the direction of a stationary party but does not come into close proximity. In effect, some space is maintained and interaction gets done at a distance. This resembles oblique approaches in that it is non-convergent and routinely projects a brief interaction. Restricted approaches are distinct, however, in that interaction at a distance presents rather different practical problems for initiating interaction. People near to one another are potential interactants by dint of their mutual accessibility. Proximity acts as a sort of pilot light for engagement. And so those who are distant from but still visible to one another have the problem of showing themselves as interactants where ordinarily they would not be seen that way. Two ambulatory practices for a restricted approach are shown: walking slowly toward the addressee, and stopping before reaching them. These are solutions to the practical problem of talking with someone from afar and indicating that, after this exchange, the speaker will go elsewhere. In Excerpt 6, the ambulatory party, Ryan, walks slowly toward a stationary party, Isidro (who is not shown in figure below, but whose conduct is captured by another camera and therefore available for analysis). The context for this case is that Ryan has been alternating between two activities: his primary activity is filling a bucket with concrete before sending it via crane to Isidro (who empties it and then sends it back to Ryan), and, when not doing that, his secondary activity is unrolling a tarp over gravel (see Excerpt 1). Just before this case, Ryan had sent the

11 Ambulatory Openings

411

bucket to Isidro and is now walking back to where he was unrolling the tarp (Fig. 11.6a). While headed in that direction, however, he sees something puzzling: the crane operator was transporting the bucket to Isidro by going in a counter clockwise direction, when it would have been more expedient to go clockwise. The crane’s puzzling movement occasions the exchange that emerges. Excerpt 6 (R_013500/I_014245)

412

E. M. Hoey

Fig. 11.6 (a) Ryan’s walking trajectory and destination; (b) gaze to Isidro (off camera) and slower pace while talking; (c) slower walk and slight approach; (d) return to walking trajectory

While returning to his secondary activity of unrolling the tarp, Ryan gazes to Isidro, who at this time is in conversation with others about 25 feet away (not shown). As he gives a shouted summons (line 2), his gaze/head are oriented toward Isidro but his legs/torso are oriented back to his working area (Fig. 11.6a). Like in Excerpt 2, Ryan’s summons unavoidably provides acoustic resources that Isidro, the summonsrecipient, could use in locating and identifying the summoner. Ryan, the summoner, complementarily makes himself locatable by continuing to gaze in Isidro’s direction, supplying himself as “the person to look for and attend to for the next turn”. In turning to face Ryan, Isidro will see that Ryan is not only distant but also that he is moving in a non-convergent trajectory. That is, in contrast to Excerpt 2, where a convergent trajectory presaged an interaction to be conducted at close range, in this case, the distance and non-convergence foretell of an interaction to be conducted at a distance.

11 Ambulatory Openings

413

With Isidro having shown his availability for interaction, Ryan produces the action that his summons was prefatory to, asking “HOW COME HE WENT AROUND THAT WAY” (line 4). As he shouts this question, he noticeably slows his walking pace (dotted arrow, Fig. 11.6b). That is, he restricts his approach toward Isidro by slightly retarding his walking tempo. In this way, he shows himself as not expressly approaching Isidro. And instead, he displays a double orientation: to continuing along his trajectory back to where he was working, but also to engaging in talk with Isidro. With this double orientation, he retains his ultimate destination and treats the interaction with Isidro as a waystation of sorts. Ryan also retains this double orientation in the repair sequence that follows. Perhaps because audibility is impaired by the distance, Isidro initiates repair with “HUH?” (line 6). This makes relevant an expansion of what was projected to be a short exchange. The contingent occurrence of this hearing trouble occasions a deviation from Ryan’s trajectory. He deviates by turning and walking toward Isidro as he supplies a repair solution (line 7; Rasmussen 2014). At the same time, he maintains a slow pace, which shows a commitment to his prior trajectory of movement. And indeed, the repair sequence is coterminous with this deviation. As it concludes (lines 10–11), Ryan resumes his prior trajectory (Fig. 11.6d; cf. Floyd et al. 2016). Another method for restricting an approach is by walking in the direction of the addressee but stopping before reaching them. Such an abbreviated trajectory can be seen in the following excerpt. Ben, evidently having been searching for Gustavo, spots him from around 20 feet away and walks in his direction while shouting a summons (lines 1–2, Fig. 11.7a–b).

414

E. M. Hoey

Excerpt 7 (R_020146)

In response to Ben’s summons, Gustavo promptly answers “YES” (line 3). We do not have video documentation of Gustavo, but he presumably turns and looks to Ben upon producing his answer. Gustavo at this time would see that Ben is far away but walking toward him in a convergent trajectory. However, as the sequence continues with a question and answer (lines 5–6), Ben gradually slows and comes to a stop

Fig. 11.7

Restricted approach with an abbreviated trajectory

11 Ambulatory Openings

415

(Fig. 11.7c). That is, he restricts his approach by abbreviating its trajectory. This produces a distant and stable interactional space. It indicates that Ben will be coming no closer and suggests that his destination lies elsewhere. In this case, after the question–answer sequence concludes (lines 7–8), Ben turns around and heads back in the direction he had come from (Fig. 11.7d). This suggests that such an abbreviated trajectory prefigures a return or heading back after ending the encounter. And in fact, in all five cases of abbreviated approaches in my collection, this turns out to be so. Restricted approaches are characterized by the retention of space between participants. As these examples show, such an approach not only projects non-convergence, but also gives an indication of the ambulatory party’s destination. In Excerpt 6, Ryan followed an overall orthogonal trajectory while engaging in talk, maintaining that path by walking slowly. And in Excerpt 7, even though Ben’s trajectory was aimed at Gustavo, his abbreviated approach showed that he would come no closer, and in fact that his destination was “back” to where he had come from. The restricted approaches shown here address the practical complexities of opening interactions on the construction site by showing the ambulatory party as coming to the stationary party for the purpose of a brief interaction.

11.3 Discussion This chapter systematically explored ambulatory openings in construction work activities. Three practices were examined with respect to how an ambulatory party approaches another in the process of initiating an interaction: direct, oblique, and restricted approaches. With a direct approach, an ambulatory party sets and follows a trajectory that terminates at an addressee. With an oblique approach, the trajectory comes near the addressee but continues past them, terminating elsewhere. And with a restricted approach, the ambulatory party walks in the direction of the addressee, but abbreviates or otherwise modulates their movement such that interaction gets done at a distance.

416

E. M. Hoey

These practices demonstrate how participants use the dynamic formation of interactional space as a resource for indicating “the kind of interaction this is/will be”. This addresses the practical complexities of openings on a construction site by narrowing the range of possibilities for an upcoming interaction. The projection of (non-)convergence is one way in which ambulatory practices can do this. Non-convergent trajectories (i.e., restricted and oblique approaches) disfavor the establishment of stable interactional spaces, and therefore, implicate brief interactions. Conversely, the convergent trajectories seen with direct approaches provide for the possibility of sustained interaction and also for a range of activities that are only or ordinarily done at close range. The findings additionally suggest that ambulatory practices in openings involve considerations of “intent” or “forethought”. We can contrast direct and restricted approaches on the one hand, to oblique approaches on the other. With direct and restricted approaches, there is something of a targeted character in that the ambulatory party may be seen as having come to the stationary party with something “in mind”. The commonsense reasoning here is that a person who has traversed the jobsite to speak to you has some reason for doing so, and that reason is not to be found in whatever immediately preceded their approach. We see this in operation with anticipatory openings (Excerpt 3). In that case, the stationary party accounted for the ambulatory party’s direct approach not by reference to the immediate interaction, but by reference to the lack of progress on their task over the course of several hours of working. In contrast to the targeted character of direct and restricted approaches, oblique approaches have a “drive by” quality. The parties come into proximity by happenstance rather than by design. By dint of the interaction’s incidental character, participants can reason as to the likely reason for the encounter—for example, to give “passing greetings” (Excerpt 4) or to make a suggestion based on something just seen (Excerpt 5). What emerges from the analysis is the constitutive accountability of walking practices for the interactions that they precede and/or accompany. This was especially observable in the practice of anticipatory openings. These openings trade on the recognizability of the “intentional” character of a direct approach and the work it does in projecting imminent interaction. A would-be addressee can see that they are being

11 Ambulatory Openings

417

approached directly, can categorize the approaching party, and can produce an opening utterance in anticipation of what the approaching party might say. In this way, the projectable formation of an interactional space can be treated as a summons. Anticipatory openings also show how participants’ categorization practices are tied to the production of likely reasons-for-the-coming-encounter. The ambulatory openings examined here contribute to studies of walking, openings, and interactional space. In line with this body of work (e.g., Mondada 2009; Clark and Pinch 2010; González-Martínez et al. 2017), this chapter illustrates the sequential and categorial work undertaken by participants in organizing their mobile activities. It builds on these studies by offering an account of mobile practices related to initiating interaction in a complex work setting. Acknowledgements Prior versions of this analysis were presented in 2021 at the Center for Applied Linguistics (University of Neuchâtel, Switzerland) and in 2020 at the European Conference on Conversation Analysis (Radboud University, The Netherlands). Funding Support for this research came from a Rubicon grant from the Netherlands Organization for Scientific Research (446-17-010) and from the Research Fund for Excellent Junior Researchers at the University of Basel (3GK1427).

References Ayaß, Ruth. 2020. “Doing waiting: An ethnomethodological analysis.” Journal of Contemporary Ethnography 49 (4): 419–455. Broth, Mathias, and Lorenza Mondada. 2013. “Walking away: The embodied achievement of activity closings in mobile interaction.” Journal of Pragmatics 47 (1): 41–58. Ciolek, T. Matthew, and Adam Kendon. 1980. “Environment and the spatial arrangement of conversational encounters.” Sociological Inquiry 50 (3–4): 237–271.

418

E. M. Hoey

Clark, Colin, and Trevor Pinch. 2010. “Some major organisational consequences of some ‘minor’, organised conduct: Evidence from video analysis of pre-verbal service encounters.” In Organisation, interaction and practice: Studies of ethnomethodology and conversation analysis, ed. by Nick Llewellyn and Jon Hindmarsh, pp. 140–171. Cambridge: Cambridge University Press. Day, Denis, and Johannes Wagner, eds. 2019. Objects, bodies and work practice. Bristol: Multilingual Matters. De Stefani, Elwys, and Lorenza Mondada. 2018. “Encounters in public space: How acquainted versus unacquainted persons establish social and spatial arrangements.” Research on Language and Social Interaction 51 (3): 248–270. Duranti, Alessandro. 1997. “Universal and culture-specific properties of greetings.” Journal of Linguistic Anthropology 7 (1): 63–97. Floyd, Simeon, Elisabeth Manrique, Giovanni Rossi, and Francisco Torreira. 2016. “Timing of visual bodily behavior in repair sequences: Evidence from three languages.” Discourse Processes 53 (3): 175–204. Goffman, Erving. 1959. The presentation of self in everyday life. Garden City, NY: Doubleday. Goffman, Erving. 1963. Behavior in public places: Notes on the social organization of gatherings. New York: The Free Press. Goffman, Erving. 1971. Relations in public: Microstudies of the public order. New York: Basic Books. Goffman, Erving. 1981. Forms of talk. Philadelphia: University of Pennsylvania Press. González-Martínez, Esther, Adrian Bangerter, and Kim Lê Van. 2017. “Passingby “Ça va?” checks in clinic corridors.” Semiotica 215: 1–42. Goodwin, Charles. 1981. Conversational organization: Interaction between speakers and hearers. New York: Academic Press. Goodwin, Charles. 2000. “Action and embodiment within situated human interaction.” Journal of Pragmatics 32 (10): 1489–1522. Haddington, Pentti, Lorenza Mondada, and Maurice Nevile, eds. 2013. Interaction and mobility: Language and the body in motion. Berlin: Walter de gruyter. Haddington, Pentti, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, eds. 2014. Multiactivity in social interaction: Beyond multitasking. Amsterdam: John Benjamins Publishing. Harjunpää, Katariina, Lorenza Mondada, and Kimmo Svinhufvud. 2018. “The coordinated entry into service encounters in food shops: Managing interactional space, availability, and service during openings.” Research on Language and Social Interaction 51 (3): 271–291.

11 Ambulatory Openings

419

Hausendorf, Heiko, and Reinhold Schmitt. 2016. “Interaktionsarchitektur und Sozialtopografie. Basiskonzepte einer interaktionistischen Raumanalyse.” In Interaktionsarchitektur, Sozialtopographie und Interaktionsraum, ed. by Heiko Hausendorf, R. Schmitt, and W. Kesselheim, vol. 1, pp. 27–54. Narr. Heath, Christian. 1986. Body movement and speech in medical interaction. Cambridge: Cambridge University Press. Heath, Christian, Marcus Sanchez Svensson, Jon Hindmarsh, Paul Luff, and Dirk Vom Lehn. 2002. “Configuring awareness.” Computer Supported Cooperative Work 11 (3): 317–347. Herrle, Matthias. 2015. “Availability stances in classroom openings.” In Aufmerksamkeit, ed. by Sabine Reh, Kathrin Berdelmann, and Jörg Dinkelaker, pp. 265–283. Wiesbaden: Springer. Hoey, Elliottt M. 2020. When conversation lapses: The public accountability of silent copresence. Oxford: Oxford University Press. Hoey, Elliott M. Forthcoming. “Anticipatory initiations: The use of a presumed reason-for-the-interaction in face-to-face openings.” Journal of Pragmatics. Hoey, Elliott M., and Kobin H. Kendrick. 2018. “Conversation analysis.” In Research methods in psycholinguistics and the neurobiology of language: A practical guide, ed. by A.M.B. de Groot and Peter Hagoort, pp. 151–173. Oxford: Wiley-Blackwell. Jefferson, Gail. 2004. “Glossary of transcript symbols with an introduction.” In Conversation analysis: Studies from the first generation, ed. by G.H. Lerner, pp. 13–34. Amsterdam: John Benjamins Publishing. Jucker, Andreas H., Heiko Hausendorf, Christa Dürscheid, Karina Frick, Christoph Hottiger, Wolfgang Kesselheim, Angelika Linke, Nathalie Meyer, and Antonia Steger. 2018. “Doing space in face-to-face interaction and on interactive multimodal platforms.” Journal of Pragmatics 134: 85–101. Keevallik, Leelo. 2018. “Sequence initiation or self-talk? Commenting on the surroundings while mucking out a sheep stable.” Research on Language and Social Interaction 51 (3): 313–328. Kendon, Adam. 1990. Conducting interaction: Patterns of behavior in focused encounters. Cambrdige: Cambridge University Press. Kendon, Adam, and Andrew Ferber. 1973. “A description of some human greetings.” Comparative Ecology and Behaviour of Primates 591 (668): 12. Lerner, Gene H. 1996. “Finding ‘face’ in the preference structures of talk-ininteraction.” Social Psychology Quarterly 59 (4): 303–321. Livingston, Eric. 1987. Making sense of ethnomethodology. London: Taylor and Francis.

420

E. M. Hoey

Llewellyn, Nick, and Robin Burrow. 2008. “Streetwise sales and the social order of city streets.” The British Journal of Sociology 59 (3): 561–583. McLauchlan, Anna, and Allison F. Noble. 2019. “Methods of entering where access is restricted.” Human Studies 43: 129–148. Mondada, Lorenza. 2009. “Emergent focused interactions in public places: A systematic analysis of the multimodal achievement of a common interactional space.” Journal of Pragmatics 41 (10): 1977–1997. Mondada, Lorenza. 2014. “Bodies in action: Multimodal analysis of walking and talking.” Language and Dialogue 4 (3): 357–403. Mondada, Lorenza. 2018. “Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality.” Research on Language and Social Interaction 51 (1): 85–106. Mondada, Lorenza. 2022. “Adjusting step-by-step trajectories in public space: The micro-sequentiality of approaching and refusing to be approached.” Gesprächsforschung 23: 36–65. Mortensen, Kristian, and Spencer Hazel. 2014. “Moving into interaction— Social practices for initiating encounters at a help desk.” Journal of Pragmatics 62: 46–67. Nevile, Maurice, Pentti Haddington, Trine Heinemann, and Mirka Rauniomaa, eds. 2014. Interacting with objects: Language, materiality, and social activity. Amsterdam: John Benjamins Publishing. Pillet-Shore, Danielle. 2018. “How to begin.” Research on Language and Social Interaction 51 (3): 213–231. Psathas, George. 1976. “Mobility, orientation, and navigation: Conceptual and theoretical considerations.” Journal of Visual Impairment and Blindness 70 (9): 385–391. Rasmussen, Gitte. 2014. “Inclined to better understanding—The coordination of talk and ‘leaning forward’ in doing repair.” Journal of Pragmatics 65: 30–45. Ryave, A. Lincoln, and James N. Schenkein. 1974. “Notes on the art of walking.” In Ethnomethodology, ed. by R. Turner, pp. 265–274. Baltimore: Penguin. Sacks, Harvey. 1992. Lectures on conversation (2 vols). Oxford: Blackwell. Salvadori, Francesca A. 2016. Open office interaction: Initiating talk at work. Unpublished PhD diss., King’s College London. Schegloff, Emanuel A. 1968. “Sequencing in conversational openings.” American Anthropologist 70 (6): 1075–1095. Schegloff, Emanuel A. 1986. “The routine as achievement.” Human Studies 9 (2–3): 111–151.

11 Ambulatory Openings

421

Schegloff, Emanuel A. 1996. “Turn organization: One intersection of grammar and interaction.” In Grammar and interaction, ed. by Elinor Ochs, Emanuel A. Schegloff, and Sandra A. Thompson, pp. 52–133. Cambridge: Cambridge University Press. Schegloff, Emanuel A. 1998. “Body torque.” Social Research 65 (3): 535–596. Schegloff, Emanuel A. 2007. Sequence organization in interaction. Cambridge: Cambridge University Press. Schegloff, Emanuel A., and Harvey Sacks. 1973. “Opening up closings.” Semiotica 8 (4): 289–327. Sorjonen, Marja-Leena, and Liisa Raevaara. 2014. “On the grammatical form of requests at the convenience store: Requesting as embodied action.” In Requesting in social interaction, ed. by Paul Drew and Elizabeth CouperKuhlen, pp. 243–268. Amsterdam: John Benjamins Publishing. Svinhufvud, Kimmo. 2018. “Waiting for the customer: Multimodal analysis of waiting in service encounters.” Journal of Pragmatics 129: 48–75. Tuncer, Sylvaine, and Pentti Haddington. 2020. “Object transfers: An embodied resource to progress joint activities and build relative agency.” Language in Society 49 (1): 61–87. Tuncer, Sylvaine, and Christian Licoppe. 2018. “Open door environments as interactional resources to initiate unscheduled encounters in office organizations.” Culture and Organization 24 (1): 11–30. Urbanik, Paweł. 2021. “Directives in the construction site: Grammatical design and work phases in second language interactions with crane operators.” Journal of Pragmatics 178: 43–67. Watson, Rod. 2005. “The visibility arrangements of urban public space: Conceptual resources and methodological issues in analysing pedestrian movements.” Communication and Cognition 38 (3–4): 201–226.

12 Openings of Interactions in Immersive Virtual Reality: Identifying and Recognising Prospective Participants Pentti Haddington , Laura Kohonen-Aho , Sylvaine Tuncer , and Heidi Spets

12.1 Introduction This chapter explores openings of interactions between co-present avatars in a public immersive virtual reality (VR) environment (see also Spets this volume). We focus on how interactional resources in the virtual environment, which may be similar or different from the ones in “offline” P. Haddington (B) · L. Kohonen-Aho · H. Spets Faculty of Humanities, University of Oulu, Oulu, Finland e-mail: [email protected] L. Kohonen-Aho e-mail: [email protected] H. Spets e-mail: [email protected] S. Tuncer King’s College, London, UK e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_12

423

424

P. Haddington et al.

settings,1 are oriented to and used by interactants in the organisation of openings (see also Hoey this volume). We show that in immersive VR particular accomplishments are necessary in interactional openings to maintain, or even create, intersubjectivity. The term “intersubjectivity” refers to a state of mutual understanding necessary for participants to make sense of each other’s actions and build moments of shared experience (e.g., when opening an interaction) that these actions embody (Peräkylä 2013: 553). Previous work on conversational openings in faceto-face situations and public spaces, and what happens immediately before them in “pre-openings”, has shown how a shared understanding of an imminent interaction is established, moment by moment, through verbal and embodied behaviour (e.g., talk, gazing practices, gestures, body movement) (see, e.g., Mondada 2009; De Stefani and Mondada 2018). Openings of interactions are coordinated multimodal achievements that display the prospective participants’ intersubjective orientation to each other’s verbal and embodied actions and the semiotic environment (Mondada 2009; De Stefani and Mondada 2018; Hoey this volume). However, as Sorjonen et al. (2021: 5,19) note, sometimes coparticipants have partial or asymmetric access to interactional resources required for establishing and maintaining intersubjectivity, which may lead to communicative breakdowns. We can find examples in situations where prospective participants do not share the same language (a tourist asking a local for directions) or, as is shown in this chapter, when prospective participants are uncertain about whether they hear each other. In such situations, interactional work is required to recover or manage the emerging communicative breakdown. It is in such situations that the complex web of interactional resources and intersubjective assumptions, which are usually tacit and taken for granted, become visible. In this paper, we analyse the shape and design of openings of interactions in an immersive virtual environment. We identify and analyse practices that are used in openings of interactions in VR and show 1 In this chapter, the terms “physical” and “offline” are used to refer to in-person interactions and to contrast with “virtual” and “online” interactions in VR.

12 Openings of Interaction in Immersive Virtual Reality

425

how they are motivated by the multimodal context and the interactional resources in it. We also show that the interactional resources in VR may be fragmented or misleading, which in different ways becomes evident in the design and organisation of interactional openings (see, e.g., Spagnolli 2002; Hindmarsh et al. 2006). Further, we build on De Stefani and Mondada’s (2018) analysis of how individuals who are either familiar with or strangers to each other initiate and open an interaction. We focus on two situations: openings of interactions that involve acquainted individuals and openings between unacquainted individuals. We analyse, first, interactional openings between acquainted participants. We analyse a “matching sequence”, that is, a sequence in which the participants interact to match an avatar with the person behind it, and thus to recognise each other. Second, we analyse the challenges unacquainted participants may face when they open an interaction in VR. We show that the challenges relate to the fragmentation of interactional resources in the VR (e.g., can a remote participant hear or not). More generally, we show how the organisation of openings of interactions in immersive VR is contingent on the virtual setting and the interactional resources in it.

12.2 Openings of Interactions and Interaction in Virtual Reality The openings of interactions have been studied broadly. The analysis has focused on how future participants establish the conditions for an encounter and/or open a conversation (see, e.g., Goffman 1963; Schegloff 1968, 1979; Mondada 2009; Haddington and Rauniomaa 2011; Mortensen and Hazel 2014; Oittinen and Piirainen-Marsh 2015; Rintel 2015; Licoppe 2017; De Stefani and Mondada 2018; PilletShore 2018; see also Hoey this volume). Importantly for this study, Mondada (2009) makes a distinction between “pre-openings”, the phase preceding the interaction proper, and “openings”, the phase that begins when the first words of the interaction are uttered (e.g., greetings). In the pre-opening phase, participants use a complex array of multimodal resources—for example, body movement, gaze behaviour, movement in

426

P. Haddington et al.

space (Mondada 2009) and the material features of the built environment (Tuncer and Licoppe 2018)—to jointly anticipate the possibility of and build the context for the imminent interaction. Furthermore, De Stefani and Mondada (2018) show that in the pre-opening phase, approaching another individual for the purpose of opening an encounter involves mutual categorisation of the prospective participants. On the one hand, they (2018: 251) analyse openings of interactions in which future participants who do not know each other “identify” each other as potential interactional partners. The identification of an unknown future interactant is often accomplished with descriptions and membership category terms that are relevant for the encounter. On the other hand, they (ibid.) analyse situations in which future participants who know each other also “recognise” each other, obliging them to decide whether to engage in interaction or ignore the other—or even hide. Recognition of a familiar person can be displayed, for example, with change-of-state tokens, such as “oh” (Heritage 1984). In sum, the above studies demonstrate that openings of physical interactions are made intelligible to prospective participants through a complex set of multimodal achievements in the pre-opening phase, for example, the mutual orientation of their bodies and gaze and the convergent mobile trajectories in space. They also show that it is this minute and emergent multimodal coordination between the prospective participants in the pre-opening phase that enables the later occurrence of the actual opening in form of the first words spoken (Mondada 2009). In this chapter, we show that the distinction between pre-openings and openings for interactions in VR is not clear. In physical encounters, the transition from the pre-beginning to the opening proper is also gradual (Mondada 2009; De Stefani and Mondada 2018), but it seems that in immersive VR these phases can be more intertwined and entangled and their boundaries even fuzzier. Furthermore, identifying and recognising a prospective participant can require interactional work that specifically confirms that “we are indeed opening an interaction” or “we indeed know each other”. We claim that these aspects of opening an interaction in VR are influenced by the VR environment as a setting for social interaction. The VR setting thus contributes to how prospective participants establish mutual orientation and intersubjectivity in and through interaction.

12 Openings of Interaction in Immersive Virtual Reality

427

Indeed, the resources available for participants in VR, for example for opening an interaction, are partly similar but can also be different from interactions in physical contexts (see also Kohonen-Aho and Vatanen 2021; Spets this volume). Hindmarsh et al. (2006) provide evidence for this by showing how the fragmentation and partiality of interactional resources impacts the building of shared views and intersubjectivity in VR. They identify four recurring problems.2 First, they notice that virtual spaces are not accessible to the users in the same way. This leads to fragmented interaction, that is, problems in producing and ascribing meanings to the actions of others. Second, co-participants assume a common world and thereby encounter difficulties in getting a sense of how their own avatar appears to others; the way in which they see the virtual space is not congruent with how others see it. In other words, the users’ common world is disrupted. Third, the collaborative manipulation of objects in VR is challenging because the technology prevents the users from seeing the full trajectory of the action. Such disrupted trajectories of action hide the reasons for using, moving and placing objects in the virtual space. Finally, while participants assume the availability of gestures and other embodied actions, Hindmarsh et al. (2006) show that other participants do not always have access to them. Immersive VR systems have developed significantly since the study by Hindmarsh et al. (2006); the new systems make possible more realistic and user-friendly user experiences. Nevertheless, the visual environment and soundscape of VR frame and render interpretable the co-participants’ talk and actions in ways that are specific to the VR environment. In this chapter, we focus on how future participants open an interaction and establish mutual orientation in immersive VR. We show that the ways in which these episodes are organised reflect the design of the virtual setting and the interactional resources available to the participants. We also identify and analyse interactional practices that the participants use to adapt to the asymmetrical access to interactional resources in VR when opening interactions. These practices relate to 2

The problems that Hindmarsh et al. (2006) identified were related to the head-mounted display’s (HMD) narrow field of view (55 degrees) they used. The field of view in the system used in this study is approx. 110 degrees. HMDs refer to the virtual glasses that users wear for entering an immersive virtual world.

428

P. Haddington et al.

“identifying” and “recognising” a co-participant but reflect immersive VR as a site for social interaction: prospective participants talk in order to match an avatar with the person steering it and to investigate access to possible shared interactional resources. We argue that these practices reflect the complexity and multilayered-ness of immersive VR as a setting for real-time social interaction (see also Spets this volume).

12.3 Data and Method Immersive Virtual Reality (VR) is a computer-generated and visually rich environment in which “users” are immersed through a head-mounted display (HMD) and headphones (see also Spets this volume). HMDs block out any visual access to the physical world. Immersive VR solutions have usually been designed for single users, and multi-user VR environments are not as common. However, fast network connections, high-quality displays, directional and stereoscopic audio and solutions that track body movement and replicate it in virtual environments have made possible graphically realistic, embodied and real-time interaction in virtual environments (e.g., Slater and Sanches-Vives 2016; Slater 2018). The analysis of real-time talk and multimodal interaction in virtual environments requires some conceptual clarifications. First, the term “user” is employed here to refer to the individuals wearing the VR gear in the physical space (see Fig. 12.1). Users apply two hand-held controllers for navigating and exploring the virtual environment and for manipulating objects in it (Slater 2018).3 Second, users appear in the virtual environment in the form of embodied virtual characters, avatars (see Fig. 12.2). The terms “participant” or “co-participant” refer to the human-technology assemblage in which the user’s talk and multimodal actions are mediated through the VR technology as avatar actions into the virtual environment. These mediated avatar actions are seen and

3 Controlling an avatar in immersive VR is a more holistic embodied experience than in desktop computer solutions in which an avatar is controlled with a mouse, a keyboard or by using thumbs and forefingers on a hand-held controller (see, e.g., Baldauf-Quilliatre and Colón de Carvajal 2015; Berger et al. 2016; Kohonen-Aho 2017).

12 Openings of Interaction in Immersive Virtual Reality

429

Fig. 12.1 A user wearing a head-mounted display, holding two controllers, and standing next to a beacon that monitors the player’s movements

heard by co-present participants (i.e., other human-technology assemblages) in the virtual environment through their HMDs and headphones and can be used as resources for interaction and joint activity. The materials analysed in this chapter are video recordings of interactions between co-participants in a VR online video game called Rec Room (https://recroom.com). Rec Room is a room-scale and multiplayer game environment. It is a virtual recreational centre where users can play, for example, dodgeball, paintball, 3D charades or disc golf. Rec Room provides co-participants different possibilities for interaction. First, they can engage in real-time conversations. Talking to co-participants in Rec Room is possible through an integrated microphone on the HMD. The system also provides users the option to switch the microphone off and not talk with co-participants. Hearing co-participants’ talk and sounds in the virtual environment is possible through a set of earphones that are connected to the HMD. Users can control their access to voices and sounds in the physical and virtual environment by adjusting the volume or wearing only one of the earphones, for example. Second, the users can see avatars and their body movements. The avatars have a partial body consisting of a head, an upper body and two hands (Fig. 12.2); they do

430

P. Haddington et al.

not have arms, legs, feet or neck. Micro-details of embodied actions (e.g., steps, minor body movements and trajectories and details of gestures) are not rendered in the game environment. The avatars’ facial expressions are automated interpretations produced by the game based on the user’s conduct. For example, if an object hits an avatar’s head, the avatar may grimace. Third, moving in Rec Room is possible in two ways. First, a teleportation feature can be utilised for moving longer distances: the hand-held controller is used to select a spot within the visible virtual environment, and with a press of a button on the controller, the user moves to the selected spot. Second, the game’s menu can be used for selecting and moving to a specific place in the game, for example, the main lobby or a specific game in Rec Room. In the physical space, users can move

Fig. 12.2

Two avatars in the Rec Room virtual reality game

12 Openings of Interaction in Immersive Virtual Reality

431

around within a small area defined by the VR equipment. Approaching the boundaries of the area generates a grid on the HMD warning the user to not cross the boundary. In this study, the HMDs in the rig were hooked up to a desktop computer with a cable, which also constrained the users’ movements.4 Finally, Rec Room provides the possibility for random encounters with other co-participants. They can also team up by doing a fist bump, which makes following another user easier. The research materials are video recordings of interactions between participants who are co-located both in the physical environment (i.e., they are in the same lab) and the virtual environment. The recordings also include encounters with online, remote participants in Rec Room. The video projection of the virtual environment as seen by the recorded users was captured from two computers that ran the Rec Room game. The same computers and Open Broadcaster Software (https://obsproject. com/) were used for recording the audio: the users’ talk from the microphones in the HMD and the in-game stereoscopic sound. Additionally, a 360-degree video camera and high-quality wireless microphones were used to capture the users’ bodily conduct and talk in the physical space (Keskinarkaus et al. 2014).5 This was necessary for obtaining evidence of the participants’ actions in the physical space. The materials were recorded as part of a university course on interaction analysis. Twelve volunteers (i.e., six pairs), who were not students in the course, were invited to interact in Rec Room and be video recorded for research purposes. Apart from one, the volunteers were non-native English speakers, with different language backgrounds. The language in the recordings is English as a lingua franca. All participants gave their informed consent for using the data in research and related publications, with personal information (names and faces) changed from the transcripts and other representations. Each recording situation lasted between 30 and 60 minutes. In sum, with three streams from each situation, the total amount about audio and video recordings is about 20 hours. In the post-production process, some segments of the video and audio streams were merged (see Fig. 12.3). 4 5

Most recent HMDs are wireless. The technology that was used is already old and much better solutions are available.

432

P. Haddington et al.

Fig. 12.3 Merged video streams: 360-degree video (bottom), the view of the participant on the left (top left frame) and the view of the participant on the right (top right frame)

The paper uses conversation analysis: the analysis focuses on the details of talk and embodied actions within sequences of interaction (e.g., Goodwin 2000; Schegloff 2007; Sidnell and Stivers 2013). The data excerpts have been annotated by using the systems developed by Jefferson (2004), for talk, and Mondada (2018, 2022), for embodiment and multimodality. The analyses are supported with images.

12.4 Analysis: Opening an Interaction in Immersive Virtual Reality There is no doubt that in immersive VR environments, participants recognise familiar avatars and open interactions with them, or that they mutually identify each other as prospective participants and successfully open an interaction. However, immersive VR affords resources for interaction that are complex in the form of being partial and fragmented and thus may require interactional adjustment. The following analysis focuses on such complexity in the openings of interactions. We explore two phenomena and analyse the interactional practices involved in them.

12 Openings of Interaction in Immersive Virtual Reality

433

First, we analyse how acquainted participants who are co-present both physically in the offline space and the virtual space match an avatar with a person when opening an interaction and before engaging in a joint activity. Second, we analyse how a participant can attempt to open an interaction with an unacquainted co-participant that is co-present in the same virtual space. We analyse how co-participants use the (partial) interactional resources available in immersive VR, and adjust their multimodal actions with respect to them, when opening an interaction with a co-participant. Lastly, we analyse an instance of a mistaken recognition of a participant and misalignment between co-participants followed by repair and realignment. We show how the interactional practices used to open interactions in VR can still be vulnerable because of the limited or misleading resources in it.

12.4.1 Matching an Avatar with a Person: Opening an Interaction with an Acquainted Co-participant In VR, when a user enters a virtual environment as an avatar for the first time, they may see many other avatars around them. In such cases, recognising a co-participant as an acquaintance requires that they can associate an avatar with a person. Excerpt 1 shows how acquainted participants engage in a short interactional episode in which they match each other’s avatars with the person steering the avatar. Pat and Lisa have been exploring the Rec Room environment independently. Before the excerpt’s beginning, Matias, a member of the research team (not present in the VR), has asked them to move to the game’s lobby. At the beginning of the excerpt, Pat has been waiting for Lisa for a short while and Lisa has just arrived at the lobby. In line 1, Matias is instructing Pat and Lisa to form a team and do things together in the virtual space.

434

P. Haddington et al.

Excerpt 1 Group 1 (110–20 min clips, group 1_v2.m4v, 0:11–0:40)6

Figure 4: Lisa and Pat in a facing formation in the VR during the pre-opening phase. Pat sees Lisa (i.e., leafvr2) and Lisa sees Pat (i.e., cleafvr1)

6

The two computers used in the research were called leafvr1 and leafvr2, which are also the names that appear on top of the avatars in Rec Room.

12 Openings of Interaction in Immersive Virtual Reality

Figure 5: Pat and Lisa producing greetings by waving to each other.

435

436

P. Haddington et al.

Matias’s turn (you need to stick together, lines 1–2) is targeted to Pat and Lisa together. Even though Pat and Lisa have not seen each other’s avatars before, by implication, Matias’s turn implies that they are now in the same space. This understanding is confirmed when they turn their bodies so that their avatars face each other in line 2 (see Fig. 4). It is during this embodied pre-opening (Mondada 2009) that they “lock on” to each other and see each other’s avatars for the first time. What they see is a generic avatar given by the game to their co-present participant when they have started the game. Pat’s avatar is wearing a striped blackand-white shirt and a Viking helmet and has dark complexion. Lisa’s avatar has a lilac-white shirt and light skin. Lisa’s avatar is also wearing eyeglasses and a Viking helmet, and it has a beard. Additionally, an avatar name (leafvr1 and leafvr2) is floating on top of their co-participant’s avatar’s head. Pat and Lisa turning to face each other starts the preopening phase. What follows is a sequence where Pat and Lisa match the avatar that they see with a person and thereby confirm the connection between the person and their avatar. In lines 4–5, Pat says: So this is— (0.3) This is you. Pat’s turn is produced simultaneously with a pointing gesture directed at Lisa’s avatar (lines 4–7). The turn, and especially the personal pronoun you, and the gesture together indexically refer to and identify Lisa and pursue a confirmation from her. It assumes that if this avatar is Lisa, then Lisa will be able to relate the gesture to the turn-attalk. At the same time, Pat’s turn shows that there are not enough cues for them to fully recognise each other and more interactional work is required: the matching sequence is a practice for confirming their identities. During Pat’s turn (l. 6), Lisa begins to reciprocate Pat’s gesture by extending her arm towards Pat’s avatar. Together with the gesture, her vocal response Hmmmm (l. 8) confirms that the avatar is indeed hers. In lines 7 and 9, Pat’s and Lisa’s gestures transform into waving gestures and an exchange of embodied greetings. They continue to wave their hands during the following turns: Pat’s Okay. (l. 9) and Lisa’s °Okay.° (l. 11) are interpretable as confirming their identities, as well as moving away from the matching sequence and preparing for a transition to the next activity (see Beach 1993, 1995). Pat’s This is me. (l. 13), however, extends the matching sequence by confirming, with the pronoun me, that the avatar waving the hand and talking to Lisa is indeed his. Pat and

12 Openings of Interaction in Immersive Virtual Reality

437

Lisa stop waving in line 16, after which Pat’s > Okay. < (l. 17) concludes the opening sequence, after which Lisa proposes that they move to the next activity (lines 19–21), as per the assigned task (Fig. 5). In face-to-face encounters, the process of matching the identity of an acquaintance with their appearance (i.e., recognising them) happens during the embodied pre-opening phase, before the greeting sequence, and usually involves little if any talk. Excerpt 1 shows an opening of an interaction in immersive VR in which the prospective participants are already co-present in the physical space, know each other and know they will meet in the virtual environment. The analysis shows how recognising a prospective participant can require special interactional work: the pre-opening phase begins with Pat and Lisa turning to face each other and producing a verbal matching sequence with which they connect the (visible but unfamiliar) avatar with the (non-visible and familiar) person controlling the avatar. The practice of matching an avatar with an acquaintance—in this case through questions, verbal deictics (e.g., personal pronouns) and bodily conduct (gestures and body movement)—reflects the partiality of visual resources available to recognise an acquaintance in immersive VR.7 It is also a practice to establish intersubjectivity and mutual orientation in VR. Additionally, Excerpt 1 shows how the verbal matching sequence extends temporally into the embodied greeting sequence produced with waving gestures. This shows how the pre-opening phase and the opening proper are not distinct or can be even merged in interactional openings in VR. In the following section, we analyse two excerpts in which identifying a prospective participant, that is, opening an interaction with someone

7 Our analyses of other cases show how the avatars’ limited capacity to reproduce bodily actions (e.g., gazing practices) and their ability to quickly move around and approach other avatars by using the teleportation feature result in fewer possibilities to recognise a co-present participant. While recognitional work in Excerpt 1 is accomplished verbally, co-participants can also use visual game-specific features, such as avatar names hovering over the avatars, as resources for recognising a prospective participant. We also have cases where acquaintances engage in a matching sequence when they have already opened an encounter earlier and share the virtual space. That participants “reconfirm” each other’s identity after having been in a “continuing state of incipient talk” (Schegloff and Sacks 1973: 325) is a further indication of how the partial and fragmented resources influence the organisation of co-presence and interaction in immersive VR.

438

P. Haddington et al.

one does not know, becomes problematic, again because of the fragmented nature of the resources available for interaction in VR. We show how partial access to the interactional resources in VR become explicit and are discussed.

12.4.2 Identifying a Potential Future Interactant In the previous section, the imminent participants were familiar with each other and had interacted in the physical space. In Excerpt 2, one participant attempts to open an interaction with a remote online coparticipant with whom they are not familiar—but fails. The problem can be traced to the fragmented nature of the interactional resources available to the co-participants for opening an interaction (e.g., unclarity regarding whether the co-participants can hear each other), and more generally to the lack of means to resolve the problem. In the excerpt, Michael and Ian are physically co-present in the lab. They have teamed up in Rec Room and agreed to play paintball together. They have been exploring the virtual game space silently when Michael sees a new avatar called CPTdude (l. 1, see Fig. 6). CPTdude is an unacquainted and remote user, and Michael attempts to initiate an interaction with him. He applies the teleportation feature on the controller, which releases the teleport beam, and teleports to CPTdude’s avatar (line 1).

12 Openings of Interaction in Immersive Virtual Reality

Excerpt 2 Group 5 (10–20 min clips, group 5.m4v, 3:12–4:00)

Figure 6: Michael sees CPTdude’s avatar in the distance (Michael’s view).

439

440

P. Haddington et al.

Figure 7: Michael casts out the teleport beam that projects his movement closer to CPTdude’s avatar (Michael’s view).

Figure 8: Michael teleports closer to CPTdude’s avatar (Michael’s view).

12 Openings of Interaction in Immersive Virtual Reality

Figure 9: Michael says hello? and shoots at CPTdude’s avatar (Michael’s view)

441

442

P. Haddington et al.

Figure 10: Michael has teleported behind CPTdude’s avatar and turned around (Michael’s view)

Teleportation beams are visible to all co-participants in Rec Room and they can be used to project an avatar’s next action: the direction and destination of their movement to a new spot in the virtual space. Michael points the beam towards CPTdude’s avatar and adjusts the exact destination point for four seconds (l. 1). As the beam is visible to CPTdude, it could be interpretable as a pre-opening, akin to such practices as mutual gaze, movement and body orientation in offline interactional openings in public spaces (Mondada 2009). Michael then teleports and lands in front of CPTdude’s avatar so that they are facing each other. He points at CPTdude’s avatar with a paintball gun and attempts to open an interaction with hello? (l. 2, see Figs. 8 and 9). After the hello?,

12 Openings of Interaction in Immersive Virtual Reality

443

Michael shoots at CPTdude’s avatar with the gun (l. 3). The hello? and the shooting, which is a game-specific resource, function as openers that attempt to establish a connection between Michael and CPTdude and invite a response from CPTdude. CPTdude does not respond to Michael’s actions verbally, but instead lifts his hand. The hand movement could be a response to Michael’s opener, a gesture requesting Michael to cease shooting or a movement wiping the paint from the gun off the avatar’s face. However, Michael does not orient to the hand movement at all but builds his next turn on the missing verbal response to his opening. He teleports closer to CPTdude’s avatar and asks him can you hear us. (l. 4).8 , 9 , 10 Asking whether a co-participant can hear is a common practice for co-participants to open video-mediated interactions, and it tends to occur after the image of the co-participant has appeared on the screen (Licoppe 2017). Here, Michael’s turn follows the first sighting of CPTdude but also the first failed attempt to talk to him. The turn also reflects the fragmented nature of the multimodal resources available for co-participants in VR to open an interaction with someone one does not know and identifying them as a potential interactional partner. First, the teleportation beam and Michael’s teleport jump and his opening turn hello? function as pre-opening actions and resources that could be interpretable by CPTdude as projecting an interactional opening (e.g., a greeting sequence). However, CPTdude does not orient 8 By referring to us, Michael refers to him and Ian. The use of the first-person plural pronoun here is probably motivated by the fact that Michael and Ian, as a team, share an audio connection and can hear each other. This is not necessarily clear to CPTdude. Additionally, since CPTdude does not respond to Michael, he treats not hearing, and thus a missing audio connection, as a possible reason for the absent response. 9 As research on openings of interactions in face-to-face encounters has shown, nonacquaintances need a reason to engage in interaction, whereas acquaintances need a reason to not interact with each other (see De Stefani and Mondada 2018). While the excerpt does not clearly indicate why Michael begins to interact with CPTdude, his actions after the excerpt show that he is trying to figure out possible players and the teams for a game of paintball. This is a likely motivation for him to open an interaction with CPTdude. 10 In Rec Room, there are at least two reasons why an individual may not hear—or may not want to appear as a “hearing” participant—to a co-participant. They may have deliberately or accidentally muted themselves in which case their talk is not available to others. Users can mute and unmute themselves by bringing the controller near their mouth and the HMD. They may also choose not to interact in a language they do not know or do not feel comfortable using. In VR, it is also easy to just ignore others and not interact with them.

444

P. Haddington et al.

or respond to these actions. Second, Michael treats CPTdude’s silence as a possible issue of hearing, rather than, for example, ignorance or unwillingness to interact. That Michael treats the silence as a hearing problem reflects the co-participants’ limited access to multimodal resources that would mutually confirm their involvement and commitment—or lack thereof—to establish mutual orientation needed for opening an interaction (e.g., facial expressions, gestures or gaze-specific features). In other words, the co-participants lack resources to identify a co-participant as a possible interactional partner. Third, and consequently, by asking can you hear us, Michael pursues some of the interactional work that in offline situations is accomplished multimodally in pre-openings. Similarly with Excerpt 1, the boundary between pre-openings and openings becomes blurry. After this, Michael attempts to teleport closer to CPTdude’s avatar, but at the same time, CPTdude has also teleported, and they pass each other. CPTdude, thus, by moving away, seems to ignore Michael’s attempts to establish an encounter. This prompts Michael to turn around and scan the environment for CPTdude’s avatar. Michael’s and CPTdude’s relative movement, and the lack of coordination, finesse and subtlety in their mobile actions (see Mondada 2009; De Stefani and Mondada 2018), is a further indication of the challenge to accomplish spatial proximity and openings in VR. After this, Michael prepares to teleport closer to CPTdude’s avatar (l. 5) but seems to lose sight of him. At this point, Michael and Ian begin to sort out their co-participants and whether they can hear or talk to each other (lines 6–10).

12.4.3 Prospective Participants’ Misalignment Between Identification and Recognition in Interactional Openings in Virtual Reality Excerpt 3 presents an example of how the limited and fragmented access to interactional resources in VR can influence the organisation of interactional openings and how the co-participants make use of those resources that VR specifically offers for opening an interaction. Furthermore, similar to Excerpts 1 and 2, it shows how the pre-opening and

12 Openings of Interaction in Immersive Virtual Reality

445

opening phases of interactional openings can become intertwined. The excerpt also shows the prospective participants’ misaligning orientations to whether the opening involves the identification of an unacquainted co-participant or the recognition of a familiar person. Joni is waiting for Noora and Markfour4 in a lobby outside a door that takes to the paintball game. In Rec Room, co-participants who are spatially separated from each other can hear and talk to each other if they have teamed up. Therefore, although Joni cannot see the others, the three of them are able to talk to and hear each other (lines 1, 4, 7, 9). Just as Joni is informing Noora about his location (l. 1), two avatars he has never met—383 and Messi, seemingly a team—enter the same space. We focus on the opening of the interaction between 383, Messi and Joni.11 In overlap with Joni’s turn that targets Noora, 383 says can you hear me? Excerpt 3 Group 6 (10–20 min clips, group 6.m4v, 8:06–9:00)

11 The talk between 383, Messi and Joni, is presented in italics to clarify the interaction between them.

446

P. Haddington et al.

Figure 11: 383 (closer to Joni’s avatar) and Messi approach Joni’s avatar. Markfour4 is teleporting away from Joni’s avatar to the main lobby (Joni’s view).

12 Openings of Interaction in Immersive Virtual Reality

Figure 12: Joni greets 383 by saying Hello and waving his left hand. Messi prepares to teleport closer to Joni’s avatar (Joni’s view).

447

448

P. Haddington et al.

Figure 13: Joni confirms hearing Messi and waves his hand (Joni’s view).

12 Openings of Interaction in Immersive Virtual Reality

449

Similar to Excerpt 2, 383 says can you hear me? in line 2, which requests a prospective participant, after a visual connection has been established, to confirm hearing (see also Licoppe 2017). In the context of VR, it displays 383’s orientation to the limited multimodal resources available for opening an interaction and thus the need to establish the preconditions for interaction verbally. This is also how Joni treats 383’s turn. In line 5, he responds with Hello, and a confirmation that he can hear 383 (yeah we all hear you.),12 accompanying the turn with a waving gesture. After this, also Messi moves closer to Joni’s avatar and says do you hear me? if you hear me, wave your hand., checking on his part whether Joni can hear him (l. 8, 10–11).13 Messi’s turn carefully builds a sequential context for Joni to respond to the question. In VR, it is possible that a recipient hears a co-participant but is unable (e.g., because of a dysfunctional microphone) to respond verbally. As an embodied cue alone (e.g., nodding or gazing) may not be enough to confirm recipiency, Messi invites Joni to wave his hand to ratify that he can hear him. Messi thus invites Joni to use an embodied practice that is available and particularly suited for confirming recipiency during the opening moves of an interaction in immersive VR. Joni responds by saying yeah I hear you, by waving his left hand and greeting Messi with hi. (l. 15). At this point, 383, Messi and Joni have opened an interaction and established mutual orientation, but, like Excerpts 1 and 2 above, they have relied on specific verbal and embodied practices (i.e., confirming hearing and waving) to establish the preconditions for interaction. Similarly, greetings have become intertwined with the process of identifying a potential future participant. At this point, however, Messi stops waving and says oh, some other. (l. 17), which is followed by 383’s fuck. (l. 19). Messi’s change-of-state token oh (Heritage 1984) and the category term some other, as well as 383’s expletive fuck, show that they have mistaken Joni’s avatar for someone 12

Similar to Michael in Excerpt 2, Joni refers to “us” when dealing with the ability to hear. It is difficult to determine what motivates Joni’s use of the pronoun “we” here. It may communicate membership in an existing group and thus disinterest in committing to interaction beyond the opening sequence. It may also inform 383 that besides Joni, there is a bigger group—a currently invisible “auditory with”—that may be able to hear 383. 13 The indecipherable turn component in line 8 seems to be a name, thus functioning as a device for selecting a particular next speaker.

450

P. Haddington et al.

else. In other words, 383 and Messi have treated Joni as an acquaintance that they have recognised . Joni, on the other hand, has treated the situation as an emerging interaction with two strangers (i.e., he has identified non-familiar avatars as prospective participants). After this, in line 21, Messi initiates a new greeting sequence with what’s up to which Joni replies with yeah I’m good. (.) how are you. (l. 23) At this point, Noora and Markfour4 arrive and exchange greetings with 383 and Messi by using the inscriptions of the avatar names as game-specific resources for identifying them. After this, the two groups go their separate ways, and Joni, Noora and Markfour4 enter the paintball game. The organisation of the opening in Excerpt 3 provides evidence for how the complexity of interactional resources—and their fragmented and misleading nature—in VR in subtle ways contributes to the establishing of intersubjectivity. First, asking a prospective participant if they can hear is a practice that pre-empts potential “communicative breakdowns” before engaging in interaction. Furthermore, asking a prospective participant to confirm hearing by waving their hand displays orientation to the contingency of interactional resources and their operation in VR. Second, the visual appearances of avatars—and the challenge to associate and recognise a familiar person with an avatar and identify a potential and unfamiliar future participant—can mislead co-participants (see Hindmarsh et al. 2006) in ways that become visible in organisation of openings of interactions.

12.5 Conclusions In the future, immersive VR has the potential to develop into a social environment that is regularly used for real-time interaction and collaboration. As part of such development, intersubjectivity is a central concern for the future of immersive VR. Achieving a genuine intersubjective experience in immersive VR requires shared and symmetrical access to 1. the real-time temporal and sequential progression of talk and action, 2. the trajectories of multimodal action (e.g., the shape of gestures),

12 Openings of Interaction in Immersive Virtual Reality

451

3. the virtual environment and interactional resources in it (see also Hindmarsh et al. 1998; Hindmarsh et al. 2006) and 4. speakership and recipiency. As this chapter and other studies focusing on the micro-details of realtime interaction in VR have shown (Hindmarsh et al. 1998; Hindmarsh et al. 2006; Spets this volume), as an environment for multimodal interaction, immersive VR involves complexities that in subtle ways limit and fragment participants’ shared and symmetrical access to the above four aspects of interaction. This paper has addressed this question by focusing on the openings of interactions (e.g., Goffman 1963; Mondada 2009; De Stefani and Mondada 2018). It has explored how the complexity of interactional resources features in the practices of “recognising” an acquaintance as a prospective participant and “identifying” an unacquainted but potential future participant in VR. We have shown, first, how acquainted individuals can interact in order to match a co-present avatar in VR with a person they are familiar with. We have shown that such “matching sequences” involve, for example, waving and pointing gestures, the creative use of game-specific resources, particular avatar body positions or verbal deictics. Second, we have analysed how the individuals in VR attempt to initiate an interaction with remote, unacquainted participants. The analysis shows how the participants orient to the fragmented (e.g., to the sensory environment) or misleading (e.g., virtual appearances) nature of interactional resources when they aim to identify a potential future participant. We have also shown how such complexity of the interactional resources can contribute to misunderstandings between identification and recognition in immersive VR. While we do not have enough data to argue that openings are recurrently organised in this way and involve such accomplishments, we highlight two things specific to VR environments in relation to complexity and intersubjectivity. First, we have shown how participants use interactional resources in VR to organise interaction, and how the resources provide a context for participants to make sense of each other’s actions and thus to establish an intersubjective experience. We have shown how participants adjust their talk and embodied actions with

452

P. Haddington et al.

respect to the interactional resources (e.g., when some necessary resource is missing) and their co-participants’ actions in order to shift from mere co-presence to initiate an interaction in immersive VR. They can, for example, engage in a matching sequence to connect an avatar with a person; or they can ask a prospective participant to confirm hearing with a waving gesture as part of opening an interaction. Such adjustments can be practices for anticipating and avoiding possible communication breakdowns. Second, we have shown how limited or fragmented access to interactional resources in VR may be explicitly oriented to or lead to misaligning actions. For example, participants may be uncertain about whether a co-present participant is able to hear one’s talk and have limited resources to communicate that they do not hear or do not want to engage. Third, we have shown how the interactional work that in offline interactional openings is regularly split between pre-openings and opening sequences can, in the context of VR, be intertwined. This seems to be occasioned by the limited and fragmented resources that co-participants have available for opening an interaction. Finally, our observations on the organisation of openings in immersive VR connect to the notion of an action’s intelligibility (or “accountability”, see Levinson 2013; Schegloff 2007). In any social interaction, an action’s recognisability and understandability as a specific action rest firmly on the sequential and material context of the action (e.g., Robinson 2016). Our analyses prove this: the design and organisation of co-participants’ actions in openings reflect the sequential and material context (i.e., the available resources) in VR. However, our findings also show how people adjust their actions and find ways to resolve potential obstacles to intersubjectivity presented by VR as a complex visual and multimodal environment for interaction.

References Baldauf-Quilliatre, Heike, and Isabel Colón de Carvajal. 2015. “Is the Avatar Considered as a Participant by the Players? A Conversational Analysis of Multi-player Videogames Interactions.” PsychNology Journal 13: 127–147.

12 Openings of Interaction in Immersive Virtual Reality

453

Beach, Wayne. 1993. “Transitional Regularities for ‘Casual’ “Okay” Usages.” Journal of Pragmatics 19, 325–352. https://doi.org/10.1016/0378-216 6(93)90092-4 Beach, Wayne. 1995. “Conversation Analysis: “Okay” as a Clue for Understanding Consequentiality.” In The Consequentiality of Communication, ed. by Stuart J. Sigman, 121–161. New York, NY: Routledge. https://doi.org/ 10.4324/9780203811856 Berger, Manuel, Andreas H. Jucker, and Miriam A. Locher. 2016. “Interaction and Space in the Virtual World of Second Life.” Journal of Pragmatics 101: 83–100. https://doi.org/10.1016/j.pragma.2016.05.009 De Stefani, Elwys, and Lorenza Mondada. 2018. “Encounters in Public Space: How Acquainted Versus Unacquainted Persons Establish Social and Spatial Arrangements.” Research on Language and Social Interaction 51: 248–270. https://doi.org/10.1080/08351813.2018.1485230 Goffman, Erving. 1963. Behavior in Public Places: Notes on the Social Organization of Gatherings. New York: Free Press. Goodwin, Charles. 2000. “Action and Embodiment within Situated Human Interaction.” Journal of Pragmatics 32: 1489–1522. https://doi.org/10.1016/ S0378-2166(99)00096-X Haddington, Pentti, and Mirka Rauniomaa. 2011. “Technologies, Multitasking and Driving: Attending to and Preparing for a Mobile Phone Conversation in the Car.” Human Communication Research 37: 223–254. https://doi.org/ 10.1111/j.1468-2958.2010.01400.x Heritage, John. 1984. “A Change of State Token and Aspects of Its Sequential Placement.” In Structures of Social Action, ed. by J. Maxwell Atkinson, and John M. Heritage, 299–345. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511665868.020 Hindmarsh, Jon, Mike Fraser, Christian Heath, Steven Benford, and Chris Greenhalgh. 1998. “Fragmented Interaction: Establishing Mutual Orientation in Virtual Environments.” Proceedings of the 1998 ACM conference on Computer supported cooperative work (CSCW ’98), 217–226. New York, NY: Association for Computing Machinery. https://doi.org/10.1145/289 444.289496 Hindmarsh, Jon, Christian Heath, and Mike Fraser. 2006. “(Im)materiality, Virtual Reality and Interaction: Grounding the ‘Virtual’ in Studies of Technology in Action.” The Sociological Review 54 (4): 795–817. https://doi.org/ 10.1111/j.1467-954X.2006.00672.x Jefferson, Gail. 2004. “Glossary of Transcript Symbols with an Introduction.” In Conversation Analysis: Studies from the First Generation, ed. by Gene

454

P. Haddington et al.

Lerner, 13–31. Amsterdam: John Benjamins. https://doi.org/10.1075/pbn s.125 Keskinarkaus, Anja, Sami Huttunen, Antti Siipo, Jukka Holappa, Magda Laszlo, Ilkka Juuso, Eero Väyrynen, Janne Heikkilä, Matti Lehtihalmes, Tapio Seppänen, and Seppo Laukka. 2014. “More—A Multimodal Observation and Analysis System for Social Interaction Research.” Multimedia Tools and Applications June: 1–29. https://doi.org/10.1007/s11042-015-2574-9 Kohonen-Aho, Laura. 2017. Video-based Observations on the Construction of Social Presence in Virtual Worlds. Aalto University publication series, Doctoral dissertations 200/2017. http://urn.fi/URN:ISBN:978-95260-7666-9 Kohonen-Aho, Laura, and Anna Vatanen. 2021. “(Re-)Opening an Encounter in the Virtual World of Second Life: On Types of Joint Presence in Avatar Interaction.” Journal for Media Linguistics 4(2): 14–51. https://doi.org/10. 21248/jfml.2021.30 Levinson, Stephen. 2013. “Action Formation and Ascription.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 103–130. Chichester, West Sussex: Wiley-Blackwell. Licoppe, Christian. 2017. “Skype Appearances, Multiple Greetings and ‘Coucou’: The Sequential Organization of Video-mediated Conversation Openings.” Pragmatics 27: 351–386. https://doi.org/10.1075/prag.27.3. 03lic Mondada, Lorenza. 2009. “Emergent Focused Interactions in Public Places: A Systematic Analysis of the Multimodal Achievement of a Common Interactional Space.” Journal of Pragmatics 41: 1977–1997. https://doi.org/10. 1016/j.pragma.2008.09.019 Mondada, Lorenza. 2018. “Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing Multimodality.” Research on Language and Social Interaction 51(1): 85–106. Mondada, Lorenza. 2022. Conventions for Multimodal Transcription (version 6.0.1.). https://www.lorenzamondada.net/multimodal-transcrip tion (accessed on 7 February 2023). Mortensen, Kristian, and Spencer Hazel. 2014. “Moving into Interaction: Embodied Practices for Initiating Interactions at a Help Desk Counter.” Journal of Pragmatics 62: 46–67. https://doi.org/10.1016/j.pragma.2013. 11.009 Oittinen, Tuire, and Arja Piirainen-Marsh. 2015. Openings in TechnologyMediated Business Meetings. Journal of Pragmatics 85, 47–66. https://doi. org/10.1016/j.pragma.2015.06.001

12 Openings of Interaction in Immersive Virtual Reality

455

Peräkylä, Anssi. 2013. “Conversation Analysis in Psychotherapy.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 551–574. Malden, MA: Wiley-Blackwell. Pillet-Shore, Danielle. 2018. “How to Begin.” Research on Language and Social Interaction 51: 213–231. https://doi.org/10.1080/08351813.2018.1485224 Rintel, Sean. 2015. “Omnirelevance in Technologised Interaction: Couples Coping with Video Calling Distortions.” In Membership Categorization Analysis: Studies of Social Knowledge in Action, ed. by Richard Fitzgerald, and William Housley, 123–150. London, UK: Sage.https://doi.org/10.4135/978 1473917873.n1 Robinson, Jeffrey D. 2016. “Accountability in Social Interaction.” In Accountability in Social Interaction, ed. by Jeffrey D. Robinson, 1–44. Oxford: Oxford University Press. Schegloff, Emanuel A. 1968. “Sequencing in Conversational Openings.” American Anthropologist 70: 1075–1095. https://www.jstor.org/stable/669510 Schegloff, Emanuel A. 1979. “Identification and Recognition in Telephone Openings.” In Everyday Language: Studies in Ethnomethodology, ed. by George Psathas, 23–78. New York: Irvington Publishers. Schegloff, Emanuel A. 2007. Sequence Organization in Interaction: A Primer in Conversation Analysis. Cambridge: Cambridge University Press. https://doi. org/10.1017/CBO9780511791208 Schegloff, Emanuel A., and Harvey Sacks. 1973. Opening Up Closings. Semiotica 7: 289–327. https://doi.org/10.1515/semi.1973.8.4.289 Sidnell, Jack, and Tanya Stivers. 2013. The Handbook of Conversation Analysis. Malden, MA: Wiley-Blackwell. Slater, Mel. 2018. “Immersion and the Illusion of Presence in Virtual Reality.” British Journal of Psychology 109 (3): 431–433. https://doi.org/10.1111/ bjop.12305 Slater, Mel, and Maria V. Sanchez-Vives. 2016. “Enhancing Our Lives with Immersive Virtual Reality.” Frontiers in Robotics and AI 3. https://doi.org/ 10.3389/frobt.2016.00074 Sorjonen, Marja-Leena, Anssi Peräkylä, Ritva Laury, Jan Lindström. 2021. “Intersubjectivity in Action: An Introduction.” In Intersubjectivity in Action, ed. by Jan Lindström, Ritva Laury, Anssi Peräkylä, and Marja-Leena Sorjonen, 1–22. Amsterdam: John Benjamins. https://doi.org/10.1075/ pbns.326.01sor Spagnolli, Anna. 2002. “Situated Breakdown Analysis for the Evaluation of a Virtual Environment.” PsychNology Journal 1 (1): 5–17.

456

P. Haddington et al.

Tuncer, Sylvaine, and Christian Licoppe. 2018. “Open Door Environments as Interactional Resources to Initiate Unscheduled Encounters in Office Organizations.” Culture and Organization 24 (1): 11–30. https://doi.org/ 10.1080/14759551.2017.1376326

13 Transitions Between Interactional Spaces: Working Towards Shared Understanding in a Hybrid Workshop Setting Laura Kohonen-Aho

13.1 Introduction Hybrid events, where both co-located and remote participants come together to interact via technology, have recently increased especially because of the global COVID-19 pandemic. The advantage of hybrid events is their inclusiveness in comparison with co-located events, since they provide flexibility in choosing where the participants wish to attend the event (Saatçi et al. 2020). However, when the number of both co-located and remote participants increases, hybrid settings become complex interaction environments (Büyükgüzel and Balaman 2022). Especially, the emergence of communicative asymmetry between multiple participants can lead to problems in initiating joint activities and maintaining shared understanding as the participants do not share each other’s perspectives to the setting (Heath and Luff 1992). L. Kohonen-Aho (B) Faculty of Humanities, University of Oulu, Oulu, Finland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0_13

457

458

L. Kohonen-Aho

As a result, hybrid events often suffer from ‘remote attendee problems’ (Yankelovich et al. 2004), ‘primary room dominance’ (Karis et al. 2016), and ‘collocation blindness’ (Bos et al. 2006), meaning that the setting is disadvantageous to the remote participants as they can be forgotten, ignored, or blamed for the lack of attendance by the co-located participants. Despite recent research on hybrid events (e.g., Büyükgüzel and Balaman 2022; Neumayr et al. 2021; Saatçi et al. 2020), more knowledge based on observations of naturally occurring interaction is needed to understand how joint activities are accomplished in them. Thus, further examination is needed to show how the participants in hybrid settings aim to contribute to joint activities and how and why they might end up in situations where their shared understanding faces obstacles. The aim of this study is to investigate interaction in a hybrid event, especially the participants’ possibilities to engage in multiple ‘interactional spaces’ where either joint activities are accomplished or the participants can engage in local or private activities. An interactional space refers to the spatial arrangement of bodies in the material surroundings where embodied talk-in-interaction and social actions occur (Mondada 2013). Although attendance in several interactional spaces at once has proven to be difficult, hybrid settings including both co-located and remote participants—and possibilities to use different technologies—allow for participation in multiple and even simultaneous local (face-to-face) and virtual spaces as well as public and private interactional spaces (Oittinen 2018; Wasson 2006). These cooccurring interactional spaces increase the interactional complexity for the participants, yet in ways that still need to be studied. Thus, I ask the following research question: how does the possibility to engage in multiple (local, virtual, private, and public) interactional spaces influence the participants’ engagement in joint activities in a hybrid workshop? To answer this question, I analysed video recordings from a workshop where 45 participants were divided into small teams in distinct locations and connected to one another via video connection and a text-based collaboration application. I studied how the participants’ involvements in different interactional spaces influenced their engagement in one joint

13 Transitions Between Interactional Spaces

459

activity during the workshop: giving and receiving instructions for forthcoming assignments. I analysed the instructional sequences from the perspective of two workshop locations: one where Team 1 is locally copresent with a moderator (chair) giving the instructions, and another where a remote Team 2 is receiving the instructions virtually, via video. The findings reveal occasional challenges for the remote Team 2 to understand the chair’s instructions and certain key moments that enlighten the reasons behind them. These reasons relate to moments when the chair initiates a transition between an activity that occurs in his local space and the joint instructional activity in the public space. Detailed examination of these transitions provides insights into the complexity of interacting in such a hybrid setting. The analyses show that the asymmetrical participation possibilities that prevailed between the participants as well as the various interactional and technological affordances in the setting—enabling multiple activities to co-exist for the participants in different types of interactional spaces—created challenges for coordinating the transitions and attending the joint activity.

13.2 Interaction in Hybrid Settings Common examples of hybrid settings are meetings that include both co-located and remote participants interacting via video (Saatçi et al. 2020). While a video connection was explicitly developed for creating the sense of co-presence between remote participants, interaction via video is often characterised as being asymmetrical as the reciprocity of perspectives and common frames of reference—which are necessary foundations for socially organised action—can no longer be relied on (Heath and Luff 1992; Luff et al. 2003). Since the shared ecology is fractured, it leads to potential difficulties in forming situated and shared understanding of the ongoing activity (Luff et al. 2003). Especially, the appearance of embodied actions, such as gaze direction or gestures, is transformed when viewing them on a computer display. In the end, video connection provides only mutual visual access between participants, but not a shared environment and affordances for them to rely on. Hybrid settings, where

460

L. Kohonen-Aho

some participants attend remotely while the rest are co-present around one display, increase the asymmetry of participation (Saatçi et al. 2020). Previous research has investigated interactional dynamics in hybrid settings. For example, studies have revealed that co-located participants easily forget or ignore the remote participants or even blame them for the lack of attendance (Bos et al. 2006; Karis et al. 2016; Yankelovich et al. 2004). As a contrast to these findings, Saatçi and colleagues (2020) showed how co-located participants employed both inclusion and exclusion of remote participants to secure meeting continuity. Also, spatial positioning of participants in a hybrid setting has been studied, and it has been shown to be key for progressing joint activities (Büyükgüzel and Balaman 2022). As actions between co-located participants at one site of a hybrid event are common but usually unavailable to all participants in the event, these actions are often characterised as distractions or side work that is disconnected from the joint activities of the event (Tutt et al. 2007). The unavailability of local actions to all participants is commonly treated as problematic since it can lead to undesirable outcomes, such as the exclusion of remote participants (Saatçi et al. 2020). However, local actions have also been observed to act as a resource for coordinating joint activities (Tutt et al. 2007) and filling affiliative purposes between the colocated participants (Oittinen 2018). These observations show that local actions cannot be explicitly treated as beneficial or problematic. Rather, they serve various functions, depending on the possibilities or constraints for the participants to engage in different activities, as well as the participants’ skills or preferences for interaction in the technology-mediated situation.

13.3 Interactional Spaces in Mediated Multiparty Interaction Mondada (2013) describes an interactional space as the spatial arrangement of bodies in a material environment where embodied talk-ininteraction and social actions take place. However, rather than being a contextual given, participants are doing space by discursively creating

13 Transitions Between Interactional Spaces

461

and performing it in interaction (Jucker et al. 2018). An interactional space is made relevant in the activity which the participants are mutually engaging in through embodied talk (Mondada 2009, 2011, 2013). As such, it unfolds moment-by-moment through the shared activity and can be constantly re-organised. Doing space cannot be restricted to the domain of face-to-face interaction any longer (Jucker et al. 2018). For example, in video-mediated interaction an interactional space is not achieved through the spatial positioning of physically co-present human bodies but rather as having bodily orientation to the virtually represented others via computer screen. According to Jucker et al. (2018), participants in mediated situations can interact with each other in spaces where diverse possibilities for communication are available, for example, in terms of the number of participants, temporal dimensions (synchronous vs. asynchronous interaction), and semiotic systems (textual, oral, pictorial). Especially in hybrid multiparty settings, the participants have a possibility to engage in activities taking place in different, sometimes parallel, local, virtual, public, and private interactional spaces. According to Wasson (2006), participants in hybrid meetings can attend to three types of interactional spaces: (1) the overall meeting space (or public space) comprising all distributed participants connected to one another with communication technology for engaging in mutual meeting activities, (2) local spaces where physically co-located participants are situated and may initiate other than joint meeting activities, and (3) other virtual and private spaces, such as email, through which the participants may interact with other people outside the meeting. The participants constantly negotiate their participation in these spaces at each moment and turn-by-turn (Oittinen 2018). Next, after describing the data and method, I analyse interactions in a hybrid workshop to reveal how its complex ecology becomes apparent when the participants engage in joint activities.

462

L. Kohonen-Aho

13.4 Data and Methods 13.4.1 Workshop Setting The studied setting is a 3-hour workshop where members in a sparsely populated municipality in Finland were invited to collaboratively develop ideas for a forthcoming school campus. Figure 13.1 illustrates the workshop setup where 45 participants were divided into seven teams in three distinct locations. The staff members included a chair (C) and a technical specialist (TS) guiding the event in location 1, and researchers, who were present in each location to collect data and assist the teams. The author is one of the researchers in location 2. One member in each team was assigned as a secretary responsible for writing down the team’s outputs to the collaborative assignments. The role of the chair was to instruct the workshop assignments. The chair was experienced in facilitating workshops, although not distributed ones. The instructions were mediated to the dispersed locations via video connection. Even though the chair had a visual connection to all locations, the large number of remote participants displayed on his laptop screen led him to have very limited resources to interact with or even monitor them. Similarly, the remote participants had limited resources to interact with the chair since they had a visual and auditory connection to him on a big screen placed in both remote locations, but the

Fig. 13.1

Workshop setup

13 Transitions Between Interactional Spaces

463

audio from both remote locations was disabled due to the multiplicity of participants. The participants were also connected to one another with a textbased participation tool called Presemo, which is a platform developed for enabling participation in large events (Kuikkaniemi et al. 2013; Nelimarkka et al. 2014). Presemo is used with mobile devices and its basic functions include a text chat and a poll. Content in Presemo is accessible to the participants in two different ways. In a personal interface, each participant can individually send a comment to the chat or vote in a poll. In a shared interface, all comments and poll results are displayed to the participants on a big screen placed in each location. In addition, the technical specialist used a control interface to shift between the chat and poll options and to moderate the Presemo content (e.g., create poll options from the comments that the participants sent and summarise the given task instructions). The workshop assignments alternated between collaborative assignments (ideation in teams) and individual assignments (voting in Presemo). The workshop begun with an introduction where the chair outlined the workshop objectives, and the participants were introduced to each other. The participants were also familiarised with Presemo with two warm-up tasks. In the ideation phase, the teams were assigned to discuss their ideas related to the school campus, select the best ideas, and create concrete action proposals for them. Ideation was supported with a visualisation of an imaginary school week in the forthcoming campus that the participants could collaboratively modify. During the elaboration phase, the participants evaluated the action proposals of each team by discussing them one by one in Presemo.

13.4.2 Data Collection and Analysis The data for this study comprise video recordings from two workshop locations: location 1 including Team 1, the chair, and the technical specialist, and location 2 including Team 2 (see Fig. 13.2). Since there were several teams participating from location 2, there was an additional video camera capturing interaction only in Team 2 (not shown in the

464

L. Kohonen-Aho

Fig. 13.2

Workshop locations 1 and 2

figure). In addition, the data include logfiles from Presemo that reveal the exact timing and content of the participants’ inputs. The interactional and technological affordances in this workshop enabled the participants to engage in activities in three types of interactional spaces: (1) a public space when the video connection was open to give and receive the instructions, (2) local spaces where group assignments were conducted in the face-to-face teams, and (3) private virtual spaces in Presemo that included actions within its personal interface where the individual polls were answered and in the control interface where the technical specialist followed the progress of the assignments. Using multimodal conversation analysis (Mondada 2016), I examined 32 episodes where the chair gave instructions in the public interactional space and the recipients received the instructions either face-to-face with the chair (Team 1) or via video (Team 2). The instructions were usually lengthy sequences including descriptions of the forthcoming assignment as well as the tools and materials to be used for it. There were also shorter instructions, such as reminders and clarifications for conducting the assignments. I analysed each episode from the perspective of both locations to understand how different participants engaged in the joint instructional activity. While analysing the instructional sequences, I started to identify episodes where the recipients in the remote Team 2 displayed trouble

13 Transitions Between Interactional Spaces

465

understanding the instructions. This is when engagements in other interactional spaces immediately prior to or during the instructional activity were revealed in the data, guiding me to focus my analysis on moments when the chair initiates a transition between other activities and the joint activity. Having access to both locations was necessary for starting to understand how different participants oriented to these transitions. However, since the data reveals the analyst much more than was available to the participants themselves, I aimed to not alienate myself from the participants’ social realities (Arminen et al. 2016; Olbertz-Siitonen, 2015). Thus, I analysed one perspective (location) at a time to keep in mind that the participants could not make comparisons of what was constantly happening in the interconnected environments. Next, I present my findings regarding the transitions between interactional spaces and their consequences for the remote participants in this hybrid setting.

13.5 Transitions Between Interactional Spaces The remote recipients’ trouble in understanding the instructions seemed to originate in the transitions between interactional spaces. The local devices for achieving transitions between activities have been widely studied in conversational analytic literature (e.g., Broth and Keevallik 2014; Kamunen and Haddington 2020; Keisanen et al. 2017; Markaki and Mondada 2012; Råman 2018; Szymanski 1999). Even though also interactional spaces are made relevant through the activities taking place in them (Mondada 2013), I have made the choice of describing transitions between spaces in this hybrid interaction context, since the transition usually includes changing one’s orientation from one medium or modality to the next: local (face-to-face), virtual, public, and private. Before proceeding to the transition cases followed by the remote team’s trouble, I illustrate how an instructional episode commonly took place in

466

L. Kohonen-Aho

this workshop, including the recipients acting according to the instructions. Excerpt 1,1 , 2 shows how the chair (Kalle) gives instructions to all participants in the public interactional space and how the participants in both locations display shared understanding of the instructed activity. This excerpt shows how Kalle ensures that he addresses all participants during his instructions and orients equally to the local Team 1 and the remote teams, to which he speaks via his laptop. Participants in both teams display their orientation to Kalle’s ongoing instructions by remaining silent and by turning their gaze and body positions towards him or his reflection on the big screen. Excerpt 1 occurs at the beginning of the ideation phase of the workshop. Kalle has just introduced the case: the school campus. As a part of this instruction, he explains where to find the visualisation of an imaginary school week at the campus, which will be used during the ideation. At the beginning of the excerpt, Kalle initiates a sequence where he explains where this visualisation can be found.

1

All excerpts are transcribed using Jefferson’s (2004) conventions for verbal communication and Mondada’s (2022) conventions for the relevant multimodal actions. 2 In Excerpt 1, the changes in Kalle’s gaze and upper body orientation are marked among his speech in the transcript with the following letters: T1 = Team 1, FS = front screen next to him, and LT = laptop, through which he is talking to the remote locations. The figures of Team 1 are marked as #3A–E, and figures of Team 2 as #4A–E.

13 Transitions Between Interactional Spaces

467

Excerpt 1, location 1 & Location 2. Instruction: Finding the visualisation

When Excerpt 1 begins, members in both teams orient to Kalle and listen to his instructions in silence. Members in Team 1 have turned their gaze towards Kalle (Fig. 3A) and in Team 2 towards the big screen (Fig. 4A). Team 1 reacts to Kalle’s instructions as they occur moment-by-moment: the team members turn their gazes towards the visualisation on the table only after Kalle addresses them (lines 7–8,

468

L. Kohonen-Aho

Fig. 3B). Raili is the one to reach for the visualisation when Kalle refers to it by using a deictic reference (line 9, Fig. 3C). Then Mari joins the flipping activity during the pause (line 10, Fig. 3D), after which Raili grabs the visualisation and holds it in front of her (line 13, Fig. 3E). Members in Team 2 are progressing with the instructions a little bit faster than Team 1, as they start to gradually turn their gazes to the table already when Kalle mentions esimerkkipäivä ‘example day’ (line 5, Fig. 4B). Aili reaches for the visualisation (line 6, Fig. 4C) and turns it over (line 9, Fig. 4D). Then, Meri grabs the visualisation (line 11, Fig. 4E). Only after Team 2 has finished the instructed activity, Kalle addresses the remote tables (line 13), which makes it evident that he has not monitored their progress in detail through his laptop. When Kalle guides the participants to find the visualisation, his gaze towards Team 1 reveals that he orients to their activity when progressing his instructions (line 7). In addition, the way in which Kalle proceeds within his instructing turn reveals that he adjusts his speech according to the actions of Team 1. On line 10, he takes a long pause followed by the lengthening of the word alta ‘underneath’ (line 11) and another 1.0-second pause (line 12), which further imply that he is waiting Team 1 to find the visualisation before moving on. However, we see that Kalle successfully uses ‘double orientation’ (Deppermann et al. 2010) as well to advance the sequence and draw all parties’ attention. During this sequence, he has his body positioned in between the laptop and Team 1, which enables him to switch his gaze and upper body direction effortlessly between Team 1, the laptop, and the front screen. He turns his gaze to the laptop when addressing all participants with jokaiseen pöytään ‘in each table’ (line 3) and muistakin pöydistä ‘the other tables’ (lines 12–13), and to Team 1 when he addresses them and follows their actions with the visualisation (lines 7–11). He also addresses the different teams verbally as tiimin 1 pöydästä ‘team one table’ (line 7) and muistakin pöydistä ‘the other tables’ (line 13). Contrary to Excerpt 1, the next episodes under investigation relate to such instructions that Team 2 has trouble following or understanding. I identified three reasons for their troubles, all originating from the moment when Kalle transitions between his local space and the public space to either give the instructions or during the instructions. These

13 Transitions Between Interactional Spaces

469

reasons relate to Kalle’s (1) recipient design, (2) abandonment of double orientation, and (3) assumptions about synchrony between the locations, when deciding to make the transition and when formulating the instructions. I elaborate each case with data examples in the sections that follow.

13.5.1 Transition and Recipient Design In this section, I illustrate how Kalle transitions between interactional spaces two times during the same episode: first from activities in his private virtual space and then from activities in his local space to the public space to give instructions. I show the episode first from the viewpoint of Team 1 (Excerpt 2a) and then Team 2 (Excerpt 2b). These excerpts illustrate how Kalle’s orientation to his previous interactional spaces translate into his instructions and affects Team 2’s understanding of them. In both instructions, the problem for Team 2 seems to arise from Kalle’s insufficient recipient design: the recipients are not informed why the instructions are given and who they concern. Excerpt 2a3 begins when Kalle invites all participants to listen to an instruction, which is a reminder to finish the current assignment of discussing the question ‘what is the meaning of the forthcoming school campus for the municipality?’ and send at least two answers via secretary in Presemo. The teams had been instructed to use five minutes for this task. Before the excerpt begins, Kalle and the technical specialist (Teemu) have been monitoring the participants’ actions in their private virtual space: the control interface of Presemo in Teemu’s laptop. After approximately four minutes, Teemu notices that most of the teams are not yet finished although they are approaching the given time limit.4 This prompts Kalle to transition to the public space and remind all participants to send their answers in Presemo (lines 1–10, Fig. 5).

3

In Excerpt 2a, I use two camera angles from location 1 to better illustrate the same moment in Figs. 7–9. They are marked as CH1 and CH2 in the transcript. 4 Presemo log reveals that 3/7 answers have been sent just before Teemu makes a noticing of the missing answers.

470

L. Kohonen-Aho

Excerpt 2a, location 1. Instructions: A reminder and a clarification during warm-up task 1

13 Transitions Between Interactional Spaces

471

472

L. Kohonen-Aho

When Kalle is coming to the end of his reminder, Petra, the secretary of Team 1, gazes to her laptop and starts in overlap to tell the team what she has written in Presemo (lines 11–12). Thus, it becomes evident that the reminder was unnecessary for Team 1 since Petra has already sent their answers. Suddenly, Petra stops talking and turns her gaze to the shared Presemo interface in the back screen (Fig. 6). Then, during unclear discussion which begins in Team 1 (line 14), Petra slowly leans left to establish eye contact with Teemu (Fig. 7). Once she has secured Teemu’s attention; she gazes to the back screen (Fig. 8). Petra does not say anything, but Teemu aligns with her by pointing towards the screen and replying that the answers will come soon when they move on to the next phase (lines 16–17, Fig. 9). Petra acknowledges Teemu’s response and

13 Transitions Between Interactional Spaces

473

then starts to explain that the first two answers in Presemo come from Team 1 (line 18).5 Now, Kalle seems to realise that the missing content in the shared Presemo interface may be something that all participants should be aware of and decides to clarify this. Kalle enters the public interactional space again to give another instruction where he explains Teemu’s clarification to all participants (lines 20–23). Excerpt 2a illustrated how Kalle uses private information from the technical interface of Presemo as a basis for his first instruction (reminder) and local information from Team 1 as a basis for his second instruction (clarification). Next, Excerpt 2b illustrates these instructions from the viewpoint of Team 2. First, the reminder is illustrated from the point of view of Team 2 (lines 1–27).

5

These answers are checked from the Presemo log.

474

L. Kohonen-Aho

Excerpt 2b (part 1), location 2. Instruction: a reminder during warm-up task 1 ,

13 Transitions Between Interactional Spaces

475

476

L. Kohonen-Aho

Excerpt 2b begins when Soila has just sent Team 2’s answers in Presemo6 and comments on her spelling errors (line 1). When Kalle starts to speak, the team members turn their gazes to the big screen to orient to his instructions (line 3, Fig. 10). The instructions are followed by a 1.2-second pause, during which Soila orients to her laptop to return to her secretary role (line 13, Fig. 11). On line 14, Soila orients to the reminder as a new assignment with a transition marker no niin ‘alright’ (Sorjonen and Vepsäläinen 2016). Thus, she does not treat the assignment as problematic, even though she has just sent the team’s answers in Presemo. Meri, however, does, which is seen in her hesitation and siis ‘(you) mean’—prefaced formulation (line 15) that projects her understanding of the assignment while also expressing uncertainty (Sorjonen 2018). Soila answers only with a minimal response mm and by nodding, which indicates that she treats Meri’s question simply as asking for a clarification of the given instructions (line 16). Then a 0.3-second pause follows before another answer to Meri’s question comes and someone else continues with a transition marker (lines 18–19). Meri lets the problem pass even though she still seems doubtful when making a careful suggestion about summarising (line 20). During Meri’s turn, Soila orients to her laptop again. Now, Aili is the one to solve Meri’s confusion, also by 6

Timing for the answers is checked from the Presemo log.

13 Transitions Between Interactional Spaces

477

turning her laptop towards Meri (line 21, Fig. 12). Thus, Aili has noticed the team’s answers in her personal Presemo interface. Meri acknowledges Aili’s noticing and seems to start a new turn while still hesitating (line 22). However, she gets interrupted by Heikki’s question (omitted lines). Soila does not seem to pay attention to Aili’s and Meri’s conversation, as she is still gazing towards her laptop. The beginning of Excerpt 2b revealed that Kalle—who used information from his private space as a basis for his first instruction—did not design this instruction as a clear reminder which would have revealed that it did not concern those teams who had already sent their answers. Thus, the recipient design of Kalle’s turn was consequential for Team 2, since Soila interpreted Kalle’s turn as a new assignment, not a reminder of the ongoing one. Meri seemed hesitant to start a new assignment as she appeared to treat the instruction as a reminder of the ongoing assignment. Next, the continuation of Excerpt 2b (lines 28–42) shows how Soila is still determined to start a new assignment and how Kalle’s second instruction, where he repeats Teemu’s clarification to Petra, is then received in Team 2.

478

L. Kohonen-Aho

Excerpt 2b (part 2), location 2. Instruction: clarification during warm-up task 1

13 Transitions Between Interactional Spaces

479

When Excerpt 2b continues, it becomes evident that Soila has not attended to the preceding conversation between Aili and Meri, as she still claims the others’ attention to the task (line 28). The lack of an immediate answer and a 1.2-second pause may indicate that the others,

480

L. Kohonen-Aho

at least Meri and Aili, have understood that no new task needs to be conducted. This conversation does not proceed further since Kalle starts the second instruction and the team orients towards the big screen (lines 30–34, Fig. 13). When the instruction comes to an end, Jaana notices incongruence in the notion of teidän kirjoittamat vastaukset näkyy ihan kohta ‘the answers you wrote will show in a moment’ (lines 30–31). She makes it apparent that she can already see the answers, which are visible in her personal interface in her laptop (lines 35–36, Fig. 14). This is followed by a confirmation of the same noticing from Aili (line 37). Jaana expresses confusion by asking why the answers are not visible in the shared interface (big screen) yet, during which she and her teammates turn their gazes towards the big screen (line 38, Fig. 15). One of the researchers, Harri, hears Jaana’s question and provides a clarification (lines 39–40, Fig. 16). This is followed by Jaana acknowledging Harri’s answer (line 41) and Aili responding with ah::aa ‘oh,’ a change-of-state token (line 42), which reveals a change in her orientation in respect to the matter at hand (Heritage 1984; Koivisto 2016). This excerpt illustrated how Kalle’s second instruction, that originated in Petra’s inquiry is in his local interactional space, also lacked proper recipient design for Team 2 to discover the difference between the contents in the personal and shared Presemo interfaces. As Kalle formulated his instruction only as ‘the answers you wrote will show in a moment,’ Jaana and Aili were confused since the answers were visible in their laptops. Excerpts 2a–2b showed how the information that Kalle used as a basis for both instructions was unevenly available to the participants since it originated from his private and local spaces. Having his orientation to these interactional spaces led him to leave necessary contextual information out of his instructions, resulting in complications for some members in Team 2 to understand them. Due (2015) and Nevile (2009) have observed that when speakers rely on their private experience and knowledge based on such participation frameworks that are unknowable to some participants, it can lead them to produce actions that remain unintelligible for the recipients. The current example revealed how the knowledge Kalle relied on, especially in his second instruction, was simultaneously private and public, depending on the location

13 Transitions Between Interactional Spaces

481

of the recipients. Thus, complexity of this setting became apparent in the difficulty of designing instructions for both co-present and remote participants in order to create and maintain shared understanding.

13.5.2 Transition and Double Orientation As was shown in Excerpt 1, Kalle was usually successful in displaying double orientation to two interactional spaces while instructing: his local space including Team 1 and the public space including all remote participants via video connection. However, there were occasions when spontaneous utterances, initiated by members in Team 1, required Kalle to orient to them in the middle of his instructions. Thus, Kalle ended up in situations where his orientation shifted from the public space to his local space during the instructions, which (unintentionally) excluded the remote participants. In this section, I show how Team 1 invites Kalle to participate in their local interaction space during the instructions. For Team 2, these types of situations were problematic due to their limited access to the verbal and bodily-visual conduct between Kalle and Team 1. Team 2 could not hear utterances from Team 1, whereas they did hear Kalle’s responses. Excerpt 3a, from location 1, illustrates how Kalle transitions between the public space and his local space during his instructions.

482

L. Kohonen-Aho

Excerpt 3a, location 1. Instruction: action proposals

13 Transitions Between Interactional Spaces

483

Excerpt 3a begins when Kalle is finishing the previous activity of reading through the participants’ idea proposals from the shared Presemo interface and then begins to transition to the next instruction (lines 1–7). The two fillers tota ‘well’ reveal the incompleteness of his ongoing thinking process and word search (Etelämäki and Jaakola 2009). Together with the pauses in between, these fillers seem to indicate that the transition to the instruction is taking place slowly, even hesitantly. I have omitted the part where Kalle repeats what the participants have just done and continue the transcript when he starts to instruct the participants about the next assignment: the action proposals. He instructs writing down three proposals from the ideas the teams have just written in Presemo in the previous assignment (lines 9–20). Now, Raili from Team 1 complements Kalle’s turn with a notion that each team should come up with action proposals from their own written ideas, not any ideas from any team that are presented in the shared Presemo interface (line 21). Her turn is a repair initiation to specify which ideas Kalle is referring to. Kalle self-repairs with an emphasis, which is followed by an assessment of Raili’s repair initiation nimenomaan ‘precisely.’ He also provides an emphasis nonverbally by nodding towards Team 1 (line 22). Then, he concludes the instructions. Let us now examine how the absence of Raili’s turn influences Team 2’s understanding of these instructions.

484

L. Kohonen-Aho

Excerpt 3b, location 2. Instruction: Action proposals

13 Transitions Between Interactional Spaces

485

486

L. Kohonen-Aho

When Kalle approaches the end of his instructions (line 21), Jaana makes a noticing of the ideas being displayed in the shared Presemo interface and allocates her teammates’ attention to them (line 22, Fig. 17). Now, a 1.0-second pause follows, during which Raili’s turn in location 1 takes place, but it is not heard in location 2 (line 23). Kalle’s turn omistanne nimenomaan ‘your own precisely’ is then heard (line 24). From the point of view of Team 2, this turn appears as an increment or self-repair to Kalle’s previous turn, followed by possible self-talk joo ‘yes.’ Aili orients to this increment by repeating it, which indicates that she might find something problematic in it (line 25). Kalle finishes his instructions, followed by Tanja’s minimal response, and Heikki’s repetition of omista ‘(one’s)

13 Transitions Between Interactional Spaces

487

own’ (lines 26–30) which further indicates that the instruction is not entirely clear for these recipients. Others further verbalise the confusion, Jaana with siis ‘(you) mean’ which reveals her uncertainty of what has been instructed and the need for clarification (Sorjonen 2018) and Veera by asking what the team is supposed to do (lines 31–32). Again, with siis näistä ‘(you) mean these,’ Aili projects her understanding about the assignment while also expressing uncertainty as she orients towards the post-it notes where the team has written their ideas (line 33, Fig. 18). Jaana orients towards the big screen and reads aloud the instructions displayed in the shared Presemo interface: pitää tehdä kolme erilaista toimenpidettä ‘one needs to do three different actions proposals’ (line 34). Simultaneously, Tanja asks Soila what has already been written down (in Presemo), which indicates that she is starting to understand the team’s assignment (line 35). One of the researchers in location 2, Lotta, has followed the team’s discussion and offers her assistance (line 36). Excerpts 3a–3b illustrated how Raili’s turn in location 1 required Kalle to abandon his double orientation and make a transition from the public interactional space to his local interactional space. The abandonment of the double orientation is seen in Kalle’s behaviour: even though he is bodily oriented to both spaces—to his laptop and Team 1—his gaze and verbal turns are momentarily allocated only to Team 1. However, the participants in neither location treated this as a breakdown in the public interactional space per se. Especially for Team 1, there was no transition between interactional spaces: from their perspective, Kalle was giving his instructions in their local space and thus could be targeted as the recipient of their local questions and comments. For Team 1, Raili’s and Kalle’s turns played out normally in a sequential order and were appropriate for the purpose and context of their participation framework. For the analyst, however, it is observable how Kalle transitions from the public space to his local space during the instruction, which results in the confusion displayed by Team 2.

488

L. Kohonen-Aho

Although also Team 2 seemed unaware of any transition between spaces taking place, the analyst can see this transition to still have occurred from the absence of Raili’s turn for Team 2, and Team 2 then treating Kalle’s turn ‘your own’ as problematic and out of place. It confused them as they already seemed to display correct understanding of the instructions (see line 22). Rather than being local side work that would be used to privately coordinate the joint activity (Tutt et al. 2007), Kalle’s response to Raili was publicly available and as such relevant for the remote participants as well. As have been noted by Haddington and colleagues (2014), simultaneous participation in more than one activity is often difficult because of the limited possibilities for splitting attentional allocation (see, however, their notes on ‘multiactivity’). Excerpts 3a–3b showed that contrary to a co-located setting where the present activity and the emergent future activity can be made simultaneously relevant as long as they are not simultaneously engaged in by the participants (see Kamunen and Haddington 2020), communication technologies in hybrid settings afford the formation of simultaneous and overlapping activities in interactional spaces that cannot be imbricated, since these spaces emerge via different modalities: face-to-face and via video connection and include partly different participants. As a result, Kalle could be momentarily forced to shift his attention between two intersecting activities: giving instructions to all participants and replying to a repair initiation coming from Team 1, where these two activities impact each other’s temporal and sequential trajectories. Thus, the feature that created complexity here emerged when a request for preferring one space emerged, leading to the abandonment of double orientation. Utterances tied to one space can become public in other spaces as well, potentially compromising some participants’ understanding of the joint activity.

13 Transitions Between Interactional Spaces

489

13.5.3 Transition and Recipient Asynchrony The purpose of this workshop was that all participants would proceed in synchrony with the assignments despite their separate locations. Kalle aimed for securing this synchrony by giving reminders before moving on to the next assignment. However, there were occasions when he made the decision to transition to the next instructions from the basis of the readiness of his local Team 1, resulting in asynchrony between the locations. Excerpts 4a–4b are from the beginning of the workshop. Excerpt 4a follows Kalle and Team 1 during a poll assignment until a transition to the next instruction. Excerpt 4b follows Team 2 from the moment the transition occurs and continues to show how the team conducts the instructed task. Excerpt 4a begins when Kalle and Team 1 are still conducting the current poll task. Kalle is reading the poll results aloud from the back screen and most of the members in Team 1 are still answering the poll at this point; only Sanna is finished and has turned her gaze to the back screen (line 1, Fig. 19).

490

L. Kohonen-Aho

Excerpt 4a, location 1. Instruction: Introductions in the local teams

13 Transitions Between Interactional Spaces

491

One by one, the other team members finish the poll in the following order: Seppo, Jorma, Mari, Petra, Terhi, Raili, and Tuuli (lines 2–6). Kalle tells the participants where the poll results are displayed and gazes briefly to his laptop where the remote members are displayed, after which he turns his gaze to Team 1 (lines 7–11). A 6.6-second pause occurs, followed by another 2.5-second pause, during which the last team member, Kirsi, finishes the poll (line 12). This long pause indicates that Kalle is monitoring Team 1’s progression with the task. This is seen especially when Kirsi finishes and Kalle starts to orient towards the next assignment by first gazing to the front screen where he has PowerPoint slides and then by gazing to his laptop (lines 12–13) before transitioning to the next instructions (lines 14–16). Subsequently, the first part of excerpt 4b (lines 1–27) begins from the moment when Kalle transitions to the next instructions (lines 14–16 in

492

L. Kohonen-Aho

excerpt 4a) and shows how two members in Team 2 remain in their private virtual space to finish the poll. The second part of excerpt 4b (lines 28–59) shows how this has consequences to the performance of Team 2 in the following assignment, as these members have missed the instructions and later interpret them differently from their teammates. Excerpt 4b (part 1), location 2. Instruction: introductions in the local teams

13 Transitions Between Interactional Spaces

493

494

L. Kohonen-Aho

When excerpt 4b begins, Kalle starts to instruct the participants about the next assignment, which is an introduction round in the teams. He concludes the previous activity with hyvä ‘good,’ followed by a 0.5second pause, after which he manages the transition by foreshadowing the forthcoming instruction sitte tän lisäks ‘then in addition to this’ prior to the actual instruction (lines 1–3). Members in Team 2 transition to the public space for listening to the instructions, except for Heikki and Aili, who remain gazing towards their laptops. Overlapping with the instructions, Aili engages in a local activity with Heikki by referring to the poll options displayed on her laptop screens (lines 10–14, Fig. 20). It becomes evident that Heikki has not found his way in the Presemo website yet as Aili helps him to type the address (lines 17–19). Simultaneously, Kalle continues to instruct the secretaries to send the team members’ names and Christmas wishes in Presemo (lines 18–20). When Kalle comes to the end of his instructions, Aili is still helping Heikki (lines 23–24). During the omitted lines, Harri, one of the researchers in location 2, repeats Kalle’s instructions while Aili and Heikki are still discussing the Presemo website. Next, the continuation of excerpt 4b shows how Aili’s and Heikki’s orientation to their private virtual space continues and leads them to misinterpret how the introduction round should take place. The transcript continues when the team members, except for Aili and Heikki, start to display bodily orientation to one another to begin the introduction round (line 28, Fig. 21). Aili and Heikki are still orienting to their laptops, and while they do this, the instructions about the introduction round appear in Presemo,7 inserted there by the technical specialist.

7

Content and timing checked from the Presemo log.

13 Transitions Between Interactional Spaces

Excerpt 4b (part 2), location 2. Instruction: introductions in the local teams

495

496

L. Kohonen-Aho

13 Transitions Between Interactional Spaces

497

During the omitted lines (29–35), Team 2 begins their introduction round, starting from Jaana. Just before the transcript continues, Meri’s turn has ended and Soila types her answer. Apparently, Aili has noticed Soila’s typing, since she asks Soila a question that reveals that she has not attended to the instruction regarding the secretaries sending the answers (lines 36–37). Soila confirms Aili’s understanding (line 38), which is followed by Aili’s change-of-state token which reveals a change in her orientation in terms of the instructions (line 39). During a 1.2-second pause, Soila continues typing and Heikki’s answer appears in Presemo (line 40). This shows that after missing Kalle’s instructions, also Heikki

498

L. Kohonen-Aho

has attended to the written instructions in Presemo, which does not include the information about secretaries. After receiving Aili’s answer (lines 41–42), Soila summons Heikki (line 44). Heikki does not respond and during the 2.1-second pause the team members turn their gazes towards him (line 45, Fig. 22). A sequence follows where Heikki’s actions in Presemo start to become evident to the other team members. First, Aili’s question to Heikki shows that she understands why Heikki is not answering—because he sent his answer already (line 46). Heikki confirms Aili’s understanding, which is followed by Aili’s repair where she tells Heikki the correct instructions (lines 47–48). Heikki’s actions in Presemo are not yet evident to Soila, who continues to request for Heikki’s contribution to the assignment (line 49). Now, Heikki answers that he already sent it in Presemo and points to it in his laptop (line 51, Fig. 23). Soila responds to Heikki with aijaa ‘oh’ (line 52), which treats Heikki’s announcement as new information (Koivisto 2016), after which Heikki seems puzzled about Soila’s reaction: eikö sitä saanu lähettää ‘wasn’t (one) allowed to send it?’ (lines 53–55). Then, Veera explains the instructions to Heikki (lines 56–59). Excerpts 4a–4b showed how Kalle relied on his local interactional space to achieve a transition and remained unaware of the remote participants’ status with the previous assignment. It has been observed that video-mediated interactions would benefit from some preparatory work during openings to ensure everyone’s availability to the joint activity (Arminen et al. 2016). However, due to the large number of participants in this workshop, securing everyone’s readiness to move on would have been difficult and time-consuming. As a result, Aili and Heikki were no longer participating in the workshop activity in synchrony with others but were facing competing engagements between their private virtual activity and the public activity (see Rosenbaun et al. 2016). As seen in excerpt 4b, Heikki’s and Aili’s misinterpretation of the instructions was a result of not transitioning to the assignment exactly when Kalle invited them but instead, continuing the previous task. This resulted in them working in asynchrony with the other participants. Here, complexity seemed to arise from the lack of proper mutual access between locations to secure all participants’ readiness to transition, which

13 Transitions Between Interactional Spaces

499

complicated the participants’ simultaneous engagement in joint activities. However, it is notable that after finishing their individual poll task late, Heikki and Aili got on track with the ongoing assignment by relying on affordances in their personal devices—the written instructions for the introduction round. Since these instructions lacked the notion about the secretaries and since it was made possible for all the participants to start to fill in the answers, Heikki and Aili seemed to interpret the assignment as another individual task. In fact, although being in asynchrony with the other participants, Heikki and Aili transitioned smoothly from one assignment to the next. They just oriented to conducting the assignment in a different medium—and an alternative interactional space—than the others.

13.6 Discussion The purpose of this study was to investigate how participants in a hybrid workshop engage in joint activities. The technical features of the setting and the need to coordinate multiparty interaction created a complex ecology to interact in: they enabled the existence of local, virtual, private, and public interactional spaces, which became interconnected and overlapping during the joint workshop activity: giving and receiving instructions. The data collected from two workshop locations revealed the consequentiality of the existence of multiple interactional spaces during moments when the chair initiated a transition between a local activity and the joint instructional activity that occurred in different interactional spaces. Due to these transitions, the participants’ shared understanding of the joint activity faced complications, as the remote participants struggled with following and understanding the instructions. Their struggles originated in the actions of the chair (1) doing insufficient recipient design when using information from his local interactional space as a source for his instructions, (2) losing double orientation when instructing, and (3) assuming synchrony between the locations when deciding to transition to the next instructions.

500

L. Kohonen-Aho

The findings of this study demonstrate the existence of specific types of complexities in a setting involving multiple interactional spaces and transitions between them. First, orientation to information in the chair’s local space motivated him to transition to the public space and influenced the design of the instructions, that is, the way in which they remained unintelligible for the participants who did not share the chair’s local space. Second, maintaining double orientation between the local and public spaces was difficult when the participants who shared the chair’s location intervened with the instructions, forcing the chair to compromise the understanding of the remote participants when he responded to the local team. Third, the chair’s orientation to his local interactional space together with the lack of mutual access between workshop locations complicated the right moment to transition to the public space and the maintenance of the participants’ synchronous engagement in the joint activity. The observations from this study extend the notion of communicative asymmetry as the lack of reciprocal perspectives in video-mediated interaction (Heath and Luff 1992). As multiple activities can emerge and co-exist in hybrid settings due to the different possible interactional space configurations, the asymmetry does not exist only between the distributed participants, but also between all the possible interactional spaces to which they have varying access to. Thus, the asymmetry is not only interpersonal but becomes inter-spatial. As noted by Tutt and colleagues (2007), it is not unusual for the co-located participants in hybrid events to engage in local activities that remain unnoticed by the remote others. The findings from the current study indicate that the possibility to engage in local activities should be attended to since they can become consequential for the joint activities. The private and local activities of the chair formulated into his instructions, and as such influenced the remote participants’ understanding of them. Similarly, the remote participants’ local activities remained hidden from the chair, leading to asynchronous participation in the joint activity. Such hybrid multiparty settings seem to require tight coordination at relevant times and on a turn-by-turn basis when alternating between activities across interactional spaces. In face-to-face interaction, we are aware of with whom we are sharing an interactional space with from

13 Transitions Between Interactional Spaces

501

the basis of bodily orientations and gaze direction (Mondada 2013) and notice if someone’s engagement shifts into another space, such as a private virtual space with one’s smart phone. In hybrid settings, we might not even be aware of all the possible spaces our co-participants can engage in or the resources they can use for initiating different activities. The findings of this study also provide further explanations for understanding the interactional dynamics of hybrid settings, specifically reasons that lead to disadvantageous participation possibilities for the remote participants (Bos et al. 2006; Karis et al. 2016; Yankelovich et al. 2004). For example, although ‘primary room dominance’ was strongly present in the setting, the struggles in Team 2 were not entirely explainable through the dominance of location 1 during the joint activity. Rather, affordances enabling multiple interactional spaces in different types of configurations were unequally available and referable to the participants, leading to difficulties in explicating where the instructions originated from and who they concerned. Some of the occurred trouble could be prevented by increasing the symmetry of the interaction setting, by, for example, providing shared immersive virtual reality spaces for all participants to be virtually co-present (see Haddington et al. this volume, Spets this volume). However, shared focus is mainly an interactional accomplishment and as such continuously subject to change (Mondada 2013). As Büyükgüzel and Balaman (2022) demonstrate, it is essential for the participants to actively demonstrate their availability and access to one another to be able to progress joint activities in a hybrid setting. In addition to increasing this visual and auditory connection between participants, the current study implies the need to increase the participants’ sensitivity to understand the consequences that attending to different interactional spaces during or in relation to joint activities can have.

502

L. Kohonen-Aho

References Arminen, Ilkka, Christian Licoppe, and Anna Spagnolli. 2016. “Respecifying Mediated Interaction.” Research on Language and Social Interaction 49 (4): 290–309. https://doi.org/10.1080/08351813.2016.1234614. Bos, Nathan, Judith S. Olson, Ning Nan, N. Sadat Shami, Susannah Hoch, and Eric Johnston. 2006. “‘Collocation Blindness’ in Partially Distributed Groups: Is There a Downside to Being Collocated?” In Proceedings of CHI 2006 , 1313–21. Montréal, QC, Canada, April 22–27, 2006. Broth, Mathias, and Leelo Keevallik. 2014. “Getting Ready to Move as a Couple: Accomplishing Mobile Formations in a Dance Class.” Space and Culture 17 (2): 107–21. https://doi.org/10.1177/1206331213508483. Büyükgüzel, Safinaz, and Ufuk Balaman. 2022. “The Spatial Organization of Hybrid Scrum Meetings: A Multimodal Conversation Analysis Study.” Discourse & Communication, 17504813221119004. https://doi.org/ 10.1177/17504813221119004. Deppermann, Arnulf, Reinhold Schmitt, and Lorenza Mondada. 2010. “Agenda and Emergence: Contingent and Planned Activities in a Meeting.” Journal of Pragmatics 42 (6): 1700–18. https://doi.org/10.1016/j.pragma. 2009.10.006. Due, Brian. 2015. “The Social Construction of a Glasshole: Google Glass and Multiactivity in Social Interaction.” PsychNology Journal 13 (2–3): 149–78. Etelämäki, Marja, and Minna Jaakola. 2009. “Tota ja puhetilanteen todellisuus [Tota and the Reality of Speech Situation].” Virittäjä, 2, Article 2. https:// journal.fi/virittaja/article/view/4183. Haddington, Pentti, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile. 2014. “Towards Multiactivity as a Social and Interactional Phenomenon.” In Multiactivity in Social Interaction: Beyond Multitasking, edited by Pentti Haddington, Tiina Keisanen, Lorenza Mondada, and Maurice Nevile, 3–32. Amsterdam: John Benjamins. Heath, Christian, and Paul Luff. 1992. “Media Space and Communicative Asymmetries: Preliminary Observations of Video-Mediated Interaction.” Human–Computer Interaction 7 (3): 315–46. https://doi.org/10.1207/s15 327051hci0703_3. Heritage, John. 1984. “A Change-of-State Token and Aspects of Its Sequential Placement.” In Structures of Social Action: Studies in Conversation Analysis, edited by J. Maxwell Atkinson and John Heritage, 299–345. Cambridge University Press.

13 Transitions Between Interactional Spaces

503

Jefferson, Gail. 2004. “Glossary of Transcript Symbols with an Introduction.” In Conversation Analysis: Studies from the First Generation, edited by Gene Lerner, 13–31. Amsterdam/Philadelphia: John Benjamins Publishing. Jucker, Andreas H., Heiko Hausendorf, Christa Dürscheid, Karina Frick, Christoph Hottiger, Wolfgang Kesselheim, Angelika Linke, Nathalie Meyer, and Antonia Steger. 2018. “Doing Space in Face-to-Face Interaction and on Interactive Multimodal Platforms.” Journal of Pragmatics 134 (September): 85–101. https://doi.org/10.1016/j.pragma.2018.07.001. Kamunen, Antti, and Pentti Haddington. 2020. “From Monitoring to CoMonitoring: Projecting and Prompting Activity Transitions at the Workplace.” Gesprächsforschung 21: 82–122. Karis, Demetrios, Daniel Wildman, and Amir Mané. 2016. “Improving Remote Collaboration With Video Conferencing and Video Portals.” Human–Computer Interaction 31 (1): 1–58. https://doi.org/10.1080/073 70024.2014.921506. Keisanen, Tiina, Mirka Rauniomaa, and Pauliina Siitonen. 2017. “Transitions as Sites of Socialization in Family Interaction Outdoors.” Learning, Culture and Social Interaction 14 (September): 24–37. https://doi.org/10.1016/j.lcsi. 2017.05.001. Koivisto, Aino. 2016. “Receipting Information as Newsworthy vs. Responding to Redirection: Finnish News Particles aijaa and aha(a).” Journal of Pragmatics 104: 163–79. https://doi.org/10.1016/j.pragma.2016.03.002. Kuikkaniemi, Kai, Matti Nelimarkka, Petri Lievonen, and Jukka Reitmaa. 2013. “MetaGroups: A Method for Collective Innovation at Large Conferences.” In Proceedings of CO-CREATE 2013—The Boundary-Crossing Conference on Co-Design in Innovation, edited by R. Smeds and O. Irrmann, 635–46. Espoo: Aalto University Publication Series. Luff, Paul, Christian Heath, Hideaki Kuzuoka, Jon Hindmarsh, Keiichi Yamazaki, and Shinya Oyama. 2003. “Fractured Ecologies: Creating Environments for Collaboration.” Human-Computer Interaction 18 (1): 51–84. https://doi.org/10.1207/S15327051HCI1812_3. Markaki, Vassiliki, and Lorenza Mondada. 2012. “Embodied Orientations Towards Co-Participants in Multinational Meetings.” Discourse Studies 14 (1): 31–52. https://doi.org/10.1177/1461445611427210. Mondada, Lorenza. 2009. “Emergent Focused Interactions in Public Places: A Systematic Analysis of the Multimodal Achievement of a Common Interactional Space.” Journal of Pragmatics 41 (10): 1977–97. https://doi.org/10. 1016/j.pragma.2008.09.019.

504

L. Kohonen-Aho

———. 2011. “The Interactional Production of Multiple Spatialities Within a Participatory Democracy Meeting.” Social Semiotics 21 (2): 289–316. https:/ /doi.org/10.1080/10350330.2011.548650. ———. 2013. “Interactional Space and the Study of Embodied Talk-inInteraction.” In Space in Language and Linguistics, edited by Peter Auer, Martin Hilpert, Anja Stukenbrock, and Benedikt Szmrecsanyi, 247–75. Boston: De Gruyter. ———. 2016. “Challenges of Multimodality: Language and the Body in Social Interaction.” Journal of Sociolinguistics 20 (3): 336–66. https://doi.org/10. 1111/josl.1_12177. ———. 2022. “Conventions for Multimodal Transcription.” Version 6.0.1. Available from: https://www.lorenzamondada.net/multimodal-transcription. Accessed 7th Feb 2023. Nelimarkka, Matti, Kai Kuikkaniemi, and Giulio Jacucci. 2014. “A Field Trial of an Anonymous Backchannel Among Primary School Pupils.” In Proceedings of the 18th International Conference on Supporting Group Work, 238–42. GROUP ’14. New York, NY, USA: ACM. https://doi.org/10.1145/266 0398.2660399. Neumayr, Thomas, Banu Saatci, Sean Rintel, Clemens Nylandsted Klokmose, and Mirjam Augstein. 2021. What Was Hybrid? A Systematic Review of Hybrid Collaboration and Meetings Research. arXiv preprint arXiv:2111. 06172 http://arxiv.org/abs/2111.06172. Nevile, Maurice. 2009. “‘You Are Well Clear of Friendlies’: Diagnostic Error and Cooperative Work in an Iraq War Friendly Fire Incident.” Computer Supported Cooperative Work (CSCW) 18 (2–3): 147–73. https://doi.org/10. 1007/s10606-008-9089-0. Oittinen, Tuire. 2018. “Multimodal Accomplishment of Alignment and Affiliation in the Local Space of Distant Meetings.” Culture and Organization 24 (1): 31–53. https://doi.org/10.1080/14759551.2017.1386189. Olbertz-Siitonen, Margarethe. 2015. “Transmission Delay in TechnologyMediated Interaction at Work.” PsychNology Journal 13 (2–3): 203–34. Råman, Joonas. 2018. “The Organization of Transitions Between Observing and Teaching in the Budo Class.” Forum Qualitative Sozialforschung/Forum: Qualitative Social Research 19 (1). https://doi.org/10.17169/fqs-19.1.2657. Rosenbaun, Laura, Sheizaf Rafaeli, and Dennis Kurzon. 2016. “Blurring the Boundaries Between Domestic and Digital Spheres: Competing Engagements in Public Google Hangouts.” Pragmatics: Quarterly Publication of the International Pragmatics Association (IPrA) 26 (2): 291–304. https://doi.org/ 10.1075/prag.26.2.05ros.

13 Transitions Between Interactional Spaces

505

Saatçi, Banu, Kaya Akyüz, Sean Rintel, and Clemens Nylandsted Klokmose. 2020. “(Re)Configuring Hybrid Meetings: Moving from User-Centered Design to Meeting-Centered Design.” Computer Supported Cooperative Work (CSCW) 29 (6): 769–94. https://doi.org/10.1007/s10606-020-09385-x. Sorjonen, Marja-Leena. 2018. “Reformulating Prior Speaker’s Turn in Finnish: Turn-initial siis, eli(kkä), and nii(n) et(tä).” In Between Turn and Sequence: Turn-initial Particles Across Languages, edited by John Heritage and MarjaLeena Sorjonen, 251–86. Amsterdam: John Benjamins. Sorjonen, Marja-Leena, and Heidi Vepsäläinen. 2016. “The Finnish Particle No.” In NU/NÅ: A Family of Discourse Markers Across the Languages of Europe and Beyond , edited by P. Auer and Y. Maschler, 243–80. Berlin: De Gruyter. Szymanski, Margaret H. 1999. “Re-Engaging and Dis-Engaging Talk in Activity.” Language in Society 28 (1): 1–23. https://doi.org/10.1017/S00474 04599001013. Tutt, Dylan, Jon Hindmarsh, Muneeb Shaukat, and Mike Fraser. 2007. “The Distributed Work of Local Action: Interaction Amongst Virtually Collocated Research Teams.” In ECSCW 2007 , edited by Liam J. Bannon, Ina Wagner, Carl Gutwin, Richard H. R. Harper, and Kjeld Schmidt, 199– 218. London: Springer London. https://doi.org/10.1007/978-1-84800-0315_11. Wasson, Christina. 2006. “Being in Two Spaces at Once: Virtual Meetings and Their Representation.” Journal of Linguistic Anthropology 16 (1): 103–30. https://doi.org/10.1525/jlin.2006.16.1.103. Yankelovich, Nicole, William Walker, Patricia Roberts, Mike Wessler, Jonathan Kaplan, and Joe Provino. 2004. “Meeting Central: Making Distributed Meetings More Effective.” In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work—CSCW ’04, 419. Chicago, IL, USA: ACM Press. https://doi.org/10.1145/1031607.1031678.

Index

A

Accountability of embodied practices 34 Acquaintance 16, 406, 433, 437, 443, 450, 451 Action formation 29, 30, 32, 34, 59, 61 Affect 13, 15, 89, 146, 147, 150, 153, 159, 161, 165, 169, 179, 348, 349, 351, 352, 359, 368, 375, 378, 379, 469 Affective engagement 14, 349–353, 359, 361, 368, 369, 373–375, 377, 379, 380 Affordance 5, 11, 13, 15, 18, 125, 131, 146, 154, 170, 179, 208, 257, 297, 298, 300, 321, 327, 333, 334, 336, 459, 464, 499, 501

Announcement 36, 81, 178, 310, 320–322, 327, 330, 331, 334, 498 Assessment 31, 39, 48, 51, 52, 57, 61, 85, 100, 108, 112, 321, 327, 330, 331, 483 Assignment 219, 459, 462–464, 469, 476, 477, 483, 487, 489, 491, 494, 498, 499 Asymmetry, communicative 424 Asynchrony, asynchronous 461, 489, 498–500 Availability 5, 11, 15, 18, 146, 260, 261, 300, 333, 362, 390, 391, 396, 397, 401, 413, 427, 498, 501 Avatar 16, 146–150, 153, 155, 165, 167, 168, 171, 256, 423, 425, 428–430, 432–434, 436–438, 442–445, 449–451

© The Editor(s) (if applicable) and The Author(s), under exclusive licence to Springer Nature Switzerland AG 2023 P. Haddington et al. (eds.), Complexity of Interaction, https://doi.org/10.1007/978-3-031-30727-0

507

508

Index

B

Board game 12, 70–72, 92 Body torque 77, 350, 360 Byplay 6, 66, 101, 364, 365

C

Collaborative work 218, 219, 223, 226 Collective activity 132, 135 Controller (hand-held controller) 67, 149, 150, 152, 164, 428, 430, 438, 443 Co-operative action 352 Coordination 11, 15, 16, 18, 32, 33, 71, 78, 88, 131, 181, 264, 298, 373, 426, 444, 500 Co-presence 16, 18, 148, 298, 336, 437, 452, 459 Course of action 14, 61, 103, 217, 218, 220–222, 225–227, 229–234, 236, 237, 240, 241, 243, 319, 331–333, 335 Crisis management training 13, 18

D

Deontic 14, 99, 100, 109, 133, 218–220, 226, 229, 233, 236, 237, 240, 241, 244 Double orientation 413, 468, 469, 481, 487, 488, 499, 500

E

Eating 12, 30–36, 42, 43, 48, 49, 54, 55, 59–61 Embodiment, embodied actions 7, 10, 30, 32, 35, 39, 61, 105,

132, 147, 149, 151, 153, 165, 167, 267, 288, 302, 321, 334, 353, 371, 424, 427, 430, 432, 451, 459 Emotion 15, 35, 347–349, 351–353, 364, 368, 369, 373, 375, 378, 380 Engagement 12, 15, 31, 33–36, 47, 58, 62, 67, 70, 77, 90, 91, 171, 261, 348, 349, 353, 362, 371, 379, 406, 458, 465, 498–501 Epistemics 14, 99, 103, 109, 218, 222, 227, 231, 233, 237, 241, 244, 265, 325 Ethnography 181, 221, 224, 242

F

Family interaction 258 Food practices 30, 31

G

Game/gaming 6, 7, 12, 65–67, 70–77, 79–81, 83, 85, 86, 88, 90–92, 146, 151, 153, 154, 165, 255, 256, 259, 260, 262–265, 267, 270, 271, 275, 276, 281, 282, 284–286, 429, 438, 443 Gaze 5, 6, 35, 58, 77, 85, 88, 102, 115, 118, 125, 128–130, 149, 170, 178, 182, 192, 200, 206, 258, 269, 271, 288, 304, 306, 307, 310, 315, 320, 321, 324, 326, 327, 330, 334, 350, 360, 364, 369, 375, 378, 390, 396, 402–404, 408, 409, 412, 426,

Index

442, 459, 466–468, 472, 476, 480, 487, 489, 491, 498, 501 Group-formation 6

H

Hand 4, 15, 31, 45, 92, 105, 113–115, 124, 129, 130, 150, 167, 179, 193, 201, 232, 270, 275, 288, 324, 330, 349, 367, 370, 375, 376, 429, 436, 443, 449, 450, 480 Haptics 59, 275, 276, 284, 285, 352, 357, 358, 360, 368–373, 376 Head-mounted display (HMD) 145, 149, 150, 152, 427–429, 431, 443 Hybrid setting/event 5, 457–460, 465, 488, 500, 501

I

Identification 30, 426, 444, 451 Institutional interaction 242 Instruction 16, 67, 99, 129, 150, 231, 239, 270, 271, 275, 276, 317, 350, 362, 407, 459, 463–469, 473, 474, 476, 477, 480–483, 486–492, 494, 497–501 Interactional resource 4, 12–14, 17, 18, 67, 146, 171, 207, 423–425, 427, 433, 438, 450–452 Interactional space 16, 31, 35, 36, 60, 71, 103, 178, 256, 260–262, 264, 266, 270, 271, 275, 276, 283, 390, 391, 393,

509

395–397, 401, 403, 407, 415–417, 458–461, 464–466, 469, 480, 481, 487, 498–501 Intercorporeality 352 Interrogative 186, 219, 221, 223 Involvement 14, 18, 33, 60, 74, 75, 79, 85, 91, 119, 271, 281, 285, 300, 315, 326, 348, 369, 379, 391, 444, 458

L

Language choice 178, 187, 193, 205, 207–209 Lingua franca 176, 219, 431

M

Magic circle 65–67, 69–71, 73, 74, 79, 80, 83, 86, 90–92 Matching 425, 433, 436, 437, 451, 452 Materiality 4, 15, 30, 31, 266, 284 Mediated interaction 373, 443 Metagaming 12, 66–74, 80, 83, 86, 90, 92 Military observer training 14, 244 Military staff 175 Mobility 4, 182, 391, 393 Moral/morality 72, 91 Mouth 31, 32, 35, 38–40, 42, 45, 51, 54, 57, 59, 61, 108, 113, 364, 367, 443 Multiactivity 4, 10–13, 17, 30, 32, 45, 54, 61, 66, 70, 73–75, 79, 80, 82, 85, 88, 90–92, 101, 102, 109–111, 131, 133, 298, 299, 301, 332, 350, 392, 488 embedded mode 12, 32, 101

510

Index

exclusive mode 32, 33, 49, 54, 61, 110 Multilingual 176–179, 207–209 Multimodal Gestalt 6, 11, 30, 39, 48, 59, 60 Multiparty/multiparty setting 6, 15, 178, 255, 298, 300–302, 319, 332, 333, 335, 349, 351, 378, 379, 499 Multisensoriality 11, 17, 30, 32, 59, 61, 378

N

Noticing (embodied noticing) 100, 103–105, 109–114, 118, 121, 122, 125, 129–132, 134, 135, 230, 301, 334, 477, 480, 486 Nudge/nudging 14, 218, 220–227, 229–234, 236, 237, 240–244

O

Object (virtual/material) 5, 13, 33, 71, 146, 159, 165, 168, 171, 257, 262–264, 282, 285–287, 298, 299, 301, 319, 332, 333, 335, 390, 392, 397, 401, 408, 427, 430 Opening 16, 76, 374, 379, 389–391, 393, 394, 400–404, 415–417, 424–427, 432, 433, 437, 438, 443–445, 449, 451, 452, 498

P

Participation 6, 12, 17, 59, 60, 77, 101, 131, 177, 209, 255, 259,

261, 270, 283, 298, 300, 353, 362, 371, 373, 375, 390, 458–461, 463, 488, 500, 501 Participation framework 11, 14, 15, 18, 31, 35, 60, 75, 100, 130, 131, 178, 180, 209, 255, 256, 258, 262, 264, 271, 276, 277, 285, 301, 347–350, 353, 358, 360, 362, 366, 367, 369, 373, 377–380, 404, 480, 487 Player 12, 66–69, 71–75, 77, 80, 81, 83–86, 89–91, 149, 256, 261–263, 265–267, 270, 271, 283–285, 287, 429, 443 Practice-based learning 244 Pre-opening 424–426, 436, 437, 442–444, 452 Private (vs. public) 31, 61, 303, 458, 461, 465, 480, 499 Progressivity 12, 17, 61, 71, 74, 75, 79–82, 85, 86, 90–92, 133, 135, 147, 165, 169, 177, 179, 209, 244, 271, 282, 300 Project 49, 76, 103, 180, 220, 224, 226–232, 234, 257, 258, 265, 302, 304, 312, 349, 391, 401, 406, 410, 415, 442, 476, 487

Q

Question 4, 8, 9, 14, 34, 67, 73, 77, 91, 92, 104, 125, 132, 146, 175, 176, 186, 193, 206, 207, 218, 220–223, 225–235, 237, 241–244, 257, 314, 320, 367, 397, 414, 437, 451, 458, 469, 476, 477, 480, 487, 497, 498

Index

R

Recipiency 331, 397, 449, 451 Recipient design 176, 178, 221, 469, 477, 480, 499 Recognition 12, 30, 426, 433, 444, 445, 451 Repair organisation 3, 4, 102, 104, 105, 110, 132 Response pursuit 117 Role 5, 7, 11, 13, 14, 18, 92, 133, 135, 178, 208, 218–221, 224–226, 229, 234, 242–244, 283, 301, 379, 404, 462, 476 Rules 65–68, 70, 73, 108, 154, 219, 221

S

Sanction 68, 69, 85, 86, 91 Screen 14, 15, 206, 255–259, 261–264, 266, 269, 270, 275, 276, 281, 282, 284–286, 289, 297, 299, 301, 310, 317, 319, 324, 330, 334, 336, 373, 375, 377, 443, 461–463, 466–468, 472, 476, 480, 487, 489, 491, 494 Sensory asymmetry 362 Showable 299, 300, 315, 319–321, 327, 330–336 Showing/showing sequence 14–16, 51, 88, 235, 263, 269, 271, 289, 297, 298, 300–304, 306, 308, 310, 311, 315, 317, 319–321, 326, 327, 330–335, 359, 375, 410, 427 Silence/silent 32, 43, 58, 59, 75–77, 108, 159, 235, 240, 275, 362, 444, 466, 467

511

Smartphone 14, 15, 255, 262, 264, 297–304, 306, 307, 310, 311, 315, 321, 324, 330, 332–335, 377 Space 6, 10, 16, 66, 71, 106, 146, 147, 149, 151, 152, 154, 257, 260, 261, 266, 267, 271, 283, 286, 288, 327, 358, 377, 390, 396, 400, 401, 409, 410, 417, 426, 427, 431, 433, 437, 442, 459, 461, 465, 468, 469, 477, 480, 481, 487, 488, 492, 494, 501 Spatial arrangement 261, 311, 327, 335, 458, 460 Suspension 33, 67, 75, 91, 230, 232 Sustainable 79, 86 Synchrony, synchronous 288, 371, 372, 461, 469, 489, 498–500

T

Tactile screen 256, 263, 266, 271, 277, 283, 285, 286 Task 5, 13, 14, 73, 91, 176, 177, 179–181, 207, 209, 219, 221, 226, 227, 229, 232, 234, 237, 239, 240, 242, 243, 301, 333, 334, 362, 389, 392–395, 397, 399, 400, 402, 403, 409, 437, 463, 469, 474, 478–480, 489, 491, 498, 499 Taste/tasting 4, 12, 29–36, 38–40, 43, 46–49, 51, 52, 54, 55, 57, 59–61 Technology use/technologically mediated interactions 10

512

Index

Temporality/temporalities 7, 12, 15, 30, 105, 264, 265, 284, 287, 299, 320, 333, 336, 350 Touch 6, 7, 15, 47, 256, 262–264, 266, 269, 275, 282, 285, 286, 357, 361, 378 Transitions (between activities/ interactional spaces) 17, 102, 105, 182, 265, 285, 335, 390, 426, 436, 459, 465, 468, 469, 476, 481, 483, 487–489, 491, 494, 498–500

V

Video connection 150, 348, 458, 459, 462, 464, 481, 488 Virtual reality (VR) 10, 13, 15, 16, 18, 145, 148, 300, 423, 428, 430, 444, 501

W

Walking 49, 54, 55, 192, 389, 390, 395, 397, 401–403, 406, 409–411, 413, 414, 416 Workshop 15, 99, 458, 459, 461–464, 466, 489, 498–500