Extended Reality Usage During COVID 19 Pandemic (Intelligent Systems Reference Library, 216) 3030913937, 9783030913939

Citation preview

Intelligent Systems Reference Library 216

Anitha S. Pillai Giuliana Guazzaroni   Editors

Extended Reality Usage During COVID 19 Pandemic

Intelligent Systems Reference Library Volume 216

Series Editors Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland Lakhmi C. Jain, KES International, Shoreham-by-Sea, UK

The aim of this series is to publish a Reference Library, including novel advances and developments in all aspects of Intelligent Systems in an easily accessible and well structured form. The series includes reference works, handbooks, compendia, textbooks, well-structured monographs, dictionaries, and encyclopedias. It contains well integrated knowledge and current information in the field of Intelligent Systems. The series covers the theory, applications, and design methods of Intelligent Systems. Virtually all disciplines such as engineering, computer science, avionics, business, e-commerce, environment, healthcare, physics and life science are included. The list of topics spans all the areas of modern intelligent systems such as: Ambient intelligence, Computational intelligence, Social intelligence, Computational neuroscience, Artificial life, Virtual society, Cognitive systems, DNA and immunity-based systems, e-Learning and teaching, Human-centred computing and Machine ethics, Intelligent control, Intelligent data analysis, Knowledge-based paradigms, Knowledge management, Intelligent agents, Intelligent decision making, Intelligent network security, Interactive entertainment, Learning paradigms, Recommender systems, Robotics and Mechatronics including human-machine teaming, Self-organizing and adaptive systems, Soft computing including Neural systems, Fuzzy systems, Evolutionary computing and the Fusion of these paradigms, Perception and Vision, Web intelligence and Multimedia. Indexed by SCOPUS, DBLP, zbMATH, SCImago. All books published in the series are submitted for consideration in Web of Science.

More information about this series at https://link.springer.com/bookseries/8578

Anitha S. Pillai · Giuliana Guazzaroni Editors

Extended Reality Usage During COVID 19 Pandemic

Editors Anitha S. Pillai Hindustan Institute of Technology and Science Chennai, Tamil Nadu, India

Giuliana Guazzaroni University and Research Ministry of Education Tolentino, Macerata, Italy

ISSN 1868-4394 ISSN 1868-4408 (electronic) Intelligent Systems Reference Library ISBN 978-3-030-91393-9 ISBN 978-3-030-91394-6 (eBook) https://doi.org/10.1007/978-3-030-91394-6 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

COVID-19 Pandemic has created immense usage of Digital Technologies in the field of Education, E-commerce, Tourism, Industries and Healthcare sector. Teaching, learning, exams, conducting meetings, seminars, tours and consultation of doctors are all done Online using various technologies. The culmination of many converging technologies such as Natural Language Processing (NLP), Virtual Reality (VR), Augmented Reality (AR) and Artificial Intelligence (AI) has led to the development of what is known as Intelligent Virtual Environment (IVE). This book explores how AR, VR, MR (Mixed Reality) and XR (Extended Reality) has helped in better collaboration/ functioning during this pandemic especially for the Education, Gaming, E-commerce, Tourism and Healthcare Sector. The volume edited by Anitha S. Pillai and Giuliana Guazzaroni includes nine chapters to explore the massive usage of innovative digital technologies during the pandemic. Samuel S. Y. Wang explains current medical applications of XR technologies and how it has been used to combat COVID-19 and the applications of XR technologies in medicine. Prabha Susy Mathew and Anitha S. Pillai present the benefits and barriers of XR-Based Remote Learning during the pandemic along with future directions. Barbara Silvestri introduces how VR and AR are reshaping the Fashion Industry during the COVID-19 and an experience regarding online retail along with interview with experts. Giorgio Cipolletta writes about Remix Viralia—How art and education have reshaped in the era of COVID-19. According to Anitha S. Pillai, Radhika Sunil and Giuliana Guazzaroni, the pandemic has revealed both opportunities and challenges in the usage of VR, AR, XR, etc. in many different fields, e.g. tourism, marketing, sports, gaming, etc. Giuliana Guazzaroni contributes to point out some experiences regarding art galleries and museums, especially how some technologies become mainstream during the pandemic. Natural Language Processing (NLP), Virtual Reality (VR), Augmented Reality (AR), Artificial Intelligence (AI) and Intelligent Virtual Environment (IVE) are just v

vi

Preface

some of these technologies that have seen an increase due to the impossibility of travelling or visiting museums. Crypto art and NFTs, blockchain technology has been starting to enter the art and collectibles market in an increasingly prominent way, a sector that currently appears to be lively and in which there is no lack of growth prospects. Mario Gerosa presents a unique experience, framing digital reality technology applications among museums during the COVID-19 pandemic and presenting a comparative mining research. Yowei Kang and Kenneth C. C. Yang explore a complete review of museums transformation during the emergence, explaining their point of view on the new technologies used by curators using the Text mining approach. Patrizia Schettino contributes with a series of European experiences regarding museums along with a comparison of different Online guided visit models and design elements. Editors would like to express sincere gratitude to all the authors and the reviewers for their timely contributions and reviews to bring out this book useful for the community. Our sincere thanks to Prof. Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland, and Prof. Lakhmi C. Jain, KES International, Shoreham-by-Sea, UK (Volume Editors Intelligent Systems Reference Library) for giving us the opportunity to bring out the book under this series, and all this would not have been possible without the assistance of Ms. Varsha Prabakaran and Mano Priya Saravanan from the publishing division of Springer Nature. Tolentino, Italy Chennai, India

Giuliana Guazzaroni Anitha S. Pillai

Contents

1 Use of Extended Reality in Medicine During the Covid-19 Pandemic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Samuel S. Y. Wang 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Current Medical Applications of XR Technologies . . . . . . . . . . . . . . 1.3 Current Challenges Posed by Covid-19 . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Applying XR Technologies to Combat Covid-19 . . . . . . . . . . . . . . . . 1.5 Employing XR Technologies to Minimize Covid-19 Disruption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Future Challenges of XR Technologies in Medicine . . . . . . . . . . . . . 1.7 Future Applications of XR Technologies in Medicine . . . . . . . . . . . . 1.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Extended Reality Based Remote Learning Experience During Pandemic: Effectiveness and Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prabha Susy Mathew and Anitha S. Pillai 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Types of Immersive Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 XR Devices, Components, and Technologies . . . . . . . . . . . . . . . . . . . 2.3.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 Underlying Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Mechanisms of XR Technologies to Facilitate Learning . . . . . . . . . . 2.5 XR Based Teaching and Learning—Framework . . . . . . . . . . . . . . . . . 2.6 AR, VR, MR Apps for Classroom Learning . . . . . . . . . . . . . . . . . . . . 2.7 Benefits of XR Based Remote Learning. . . . . . . . . . . . . . . . . . . . . . . . 2.8 Barriers to XR Adoption in Remote Learning During Pandemic and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 2 2 3 4 6 8 9 10 11 15 15 17 19 19 19 21 21 26 28 28 32 34 35

vii

viii

Contents

3 How Virtual and Augmented Reality Are Reshaping the Fashion Industry During the Covid-19 Pandemic . . . . . . . . . . . . . . Barbara Silvestri 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Main Theories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Augmented and Virtual Reality in Fashion During the Covid-19 Pandemic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Virtual Reality and Augmented Reality for Fashion Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Virtual Reality and Augmented Reality for Online Retail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 An Exclusive Interview with the Experts: The Invrsion Company, Milan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Remix Viralia. For a Reality of the Imaginary Re-imagined. Art and Education in the Era of Covid-19 . . . . . . . . . . . . . . . . . . . . . . . . Giorgio Cipolletta 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 For an Ecosystem Approach to Schooling . . . . . . . . . . . . . . . . . . . . . . 4.2.1 #LaScuolaContinua (#SchoolContinues) . . . . . . . . . . . . . . . . 4.3 A Hybrid Non-conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 #RestaCorale (#stayChoral) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Leveraging Immersive Technologies During the COVID-19 Pandemic—Opportunities and Challenges . . . . . . . . . . . . . . . . . . . . . . . . Anitha S. Pillai, Radhika Sunil, and Giuliana Guazzaroni 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Covid-19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Virtual, Augmented and Mixed Reality . . . . . . . . . . . . . . . . . . 5.2 Uses of Immersive Technology During the Pandemic . . . . . . . . . . . . 5.2.1 Marketing Using VR and AR . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Conferences, Meetings and Planning of Events . . . . . . . . . . . 5.2.3 Retail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.4 Tourism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.5 Sports and Gaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.6 Medicine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.7 Manufacturing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.8 Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.9 Workout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

39 39 41 43 43 45 49 52 53 55 56 58 60 64 69 71 75 76 76 76 77 79 80 80 81 82 83 84 84 85 85 86

Contents

6 Virtual and Augmented Reality in Art During the Pandemic . . . . . . . Giuliana Guazzaroni 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Art and Museums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

89 89 90 93 93

7 Virtual Galleries and Museums in Search of Their Own Specific Identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Mario Gerosa References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 8 Framing Digital Reality Technology Applications Among Museums During COVID-19 Pandemic: A Comparative Text Mining Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yowei Kang and Kenneth C. C. Yang 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.1 Digital Reality Technologies in the Museum Sector . . . . . . . 8.1.2 The COVID-19 Pandemic and the Transformation of Museums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.3 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 Framing AR, MR, VR, and XR Applications in the Museum Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Framing Best Practices by Their Technology Affordances and Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.3 Framing Digital Reality Technologies by Empirical Effects on Visitors’ Museum Experiences . . . . . . . . . . . . . . . . 8.3 Research Method and Computational Data Analysis . . . . . . . . . . . . . 8.4 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Key Phrase Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Discussions and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.1 Summary of Key Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.2 Implications for Museums in the Age of COVID-19 Pandemic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.3 Study Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Guided Digital Tours and Covid-19. Comparison of Cases in Europe and Definition of Models Grounded on Data . . . . . . . . . . . . Patrizia Schettino 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Theoretical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.1 Virtual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.2 Definition of Tour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.3 Guided Tour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

109 110 110 111 112 113 113 115 116 117 117 119 120 120 120 121 121 127 127 128 128 129 130

x

Contents

9.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Flemish Masters, Different Locations . . . . . . . . . . . . . . . . . . . 9.4.2 The Courtauld Gallery, UK, London . . . . . . . . . . . . . . . . . . . . 9.4.3 Villa Giulia, “Dirigi il Direttore”, Italy, Rome . . . . . . . . . . . . 9.4.4 Gallerie Estensi, Italy, Modena . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.5 Focus on Gallerie Estensi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.6 Comparison of the Different Online Guided Visit Models and Their Design Elements . . . . . . . . . . . . . . . . . . . . . 9.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

133 133 133 134 135 135 136 138 139 140 140

Chapter 1

Use of Extended Reality in Medicine During the Covid-19 Pandemic Samuel S. Y. Wang

Abstract The ongoing Covid-19 pandemic has generated a strong impetus to digitalize the economy and aspects of our daily life. Clinical medicine traditionally being conservative has seen a limited uptake of these new technologies which has seen widespread adoption in other industries. But due to disruptive nature of the pandemic, clinical medicine has also been forced to adapt and capitalize on these new technologies. Chief amongst them is the utilization of extended reality (XR) technologies, which is an umbrella term that encompasses a spectrum of virtual reality (VR) and Augmented reality (AR) devices that blend the physical world with the digital world. VR technologies immerses users in 3D worlds while AR technologies project 3D objects into the user’s physical environment while permitting full visibility of the user’s surroundings. XR technologies can assist in infection control measures by revolutionizing clinical ward rounds. Patient’s key blood results and vitals can be projected above each patient enhancing the speed of clinical ward rounds for large number of patient’s in community isolation facilities. Examination findings can then be dictated and automatically recorded. XR technologies can also assist clinicians during the planning and execution of highly infective/risky procedures. XR can help proceduralists simulate the procedure, limiting timing spent during the actual procedure. While XR guided robots can actually perform the high risk and delicate procedures, limiting infection risk for the proceduralist. XR technologies can overcome the disruption caused clinical education due to Covid-19 pandemic infection control measures. They can help simulate patient interaction/ clinical scenarios for medical students while keeping both patient and medical students safe from infection. Covid-19 has also generated much psychosocial distress due to the isolation stemming from infection control. XR technologies can be used to help bridge the psychosocial isolation by connecting patients with their family members, hobbies or home towns. This can be especially therapeutic when counselling patients that Samuel Wang Sherng Young was principally involved in formulating the topic, performing the literature review and writing the chapter. S. S. Y. Wang (B) Alexandra Hospital, National University Hospital System, Singapore, Singapore e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_1

1

2

S. S. Y. Wang

suffer pandemic related depression/anxiety. Particularly in palliative patients XR technologies can help simulate experiences that would be physically impossible for them. Keywords Next generation clinical services · XR enhanced clinical education · XR augmented mental health · Palliative telemedicine

1.1 Introduction Extended reality (XR) technologies are an umbrella term that encompasses a spectrum of virtual reality (VR) and Augmented reality (AR) devices that blend the physical world with the digital world. VR technologies immerses users in 3D worlds while AR technologies project 3D objects into the user’s physical environment while permitting full visibility of the user’s surroundings. As high-speed broadband becomes increasingly available, the digital revolution will gain pace and XR technologies will be more common in daily use. The ongoing Covid-19 pandemic has generated a strong impetus to digitalize our economies and aspects of our daily life. XR technologies can assist with the digitalization of a traditionally conservative field such as clinical medicine. Clinical medicine has experimented with XR technologies mainly in operative/procedural and medical education settings. However wide scale adoption in the medical field has often been slow as new technologies often have to pass high safety barriers and overcome the conservative minded clinicians.

1.2 Current Medical Applications of XR Technologies XR technologies is a rapidly evolving field with the ultimate aim being to simulate reality by creating highly similar replicas of real-life scenarios or alternatively to augment our experience of reality by providing additional valuable information to users. AR systems should (a) combine real and virtual objects in a real environment; (b) run interactively and in real-time; (c) register real and virtual objects with each other [1]. VR systems focuses on a user’s immersion in a simulated environment and the user’s interaction with that environment. With the goal being replacing real stimuli and recreating experiences with a high degree realism, which in the real world would be impossible [2]. Applications of such technologies are perhaps most commonly seen in the entertainment particularly video game industry. XR technologies display much clinical potential despite making slow inroads into the medical field. XR technologies in the medical field is mostly commonly applied in medical education as potential surgical/procedural aids. XR technologies as a teaching aid has been applied in the teaching of anatomy and physiology, XR technologies enable students to interactively explore the human body. One such example is the Stanford Virtual Heart Project which utilized the Microsoft HoloLens to aid students in

1 Use of Extended Reality in Medicine During the Covid-19 Pandemic

3

visualization of normal and abnormal anatomy and in more senior years the visualization of surgical procedures. Students have reported that the technology seemed to be more engaging when learning and incorporating medical knowledge beyond traditional teaching resources such as textbooks, videos, models, and cadavers. Some evidence also shows that students displayed an improvement in performance, observed primarily on laboratory examinations [3]. XR technologies have also been applied directly in clinical work, EchoPixel is a 3D display which has gained approval from the U.S. Food and Drug Administration as a preprocedural aid for cardiologists. The 3D display was used to visualize arteries in patients with pulmonary atresia with major aortopulmonary collateral arteries with computed tomography angiography either by using the 3D display or a traditional readout. Studies have shown that cardiologists using the EchoPixel 3D display had interpretation times of 13 min compared with 22 min with comparable levels of accuracy between traditional and the 3D display [4].

1.3 Current Challenges Posed by Covid-19 The ongoing global Covid-19 pandemic caused by the SARS-CoV-2 has posed great challenges to medicine. The highly infectious nature of the virus threatens to overwhelm the healthcare capacity of many nations. No country is an exception to this, even the developed healthcare systems in the West are under critical stress trying to keep up with the Covid-19 infection rates. Traditional models of healthcare delivery have been disrupted by the need to social distance for infection control. Social distancing by limiting the movement of crowds into and out of healthcare facilities is an important part of infection control [5, 6]. Traditional outpatient clinical reviews are potentially a significant source of crowd movements into and out of healthcare facilities. Outpatient settings have been associated with instance of community acquired transmission of Covid-19 from the hospital to the community and vice versa [7]. Medical education has also been significantly disrupted by social distancing measures, due to doctors being diverted for more clinical duties and as part of infection control measures for patient and medical student safety, medical students have been banned from the wards during the Covid-19 pandemic [8]. This has generated a significant impact due to their clinical education as a large part of medical training is experiential [9] which would require one to be physically in hospital to absorb clinical learning from interacting with doctors, patients and other allied health workers. Even amongst doctors and allied health workers, they have been segregated into various teams to minimize risk of infection and subsequent need for quarantine which can significantly reduce manpower and disrupt clinical workflow [10]. This splitting of teams have disrupted regular clinical discussions/multi-disciplinary meetings which are necessary when managing complex patients [10]. Minimizing direct exposure to Covid-19 patients whilst ensuring the patients received optimal medical care is a significant clinical challenge. This problem of

4

S. S. Y. Wang

infection exposure is exacerbated if there is a shortage of personal protective equipment which was noted during the early period of the Covid-19 pandemic surge [11]. It forces clinicians to rethink the current methods of clinical examination and clinical procedures. Chief among them would be risk stratifying infection exposure risk examinations and procedures which are often any form of aerosol generating examination or procedure. Faced with these challenges medicine, a conservative minded industry is forced to adapt and utilize new technologies and workflows to overcome the challenges by Covid-19. XR technologies are one of the many new tools that can augment the delivery of healthcare and overcome the challenges posed by Covid-19.

1.4 Applying XR Technologies to Combat Covid-19 As mentioned previously, XR technologies have been harnessed to assist proceduralists in optimizing invasive procedures during Covid-19. It is not uncommon for individuals with Covid-19 to develop respiratory failure requiring intubation and mechanical ventilation. Intubation is not only invasive and dangerous to the patient but also highly infections for the proceduralists [12]. The danger to the proceduralists is the generation of high levels of aerosols during the intubation process which can expose the proceduralist to high levels of SARS-CoV-2 [12]. This places the proceduralist at great risk of developing Covid-19. XR technologies can potentially minimize the infection risk from the procedural through facilitating the intubation process. Hence minimizing the proceduralist’s exposure time and the number of aerosols generated during the procedure. Advanced XR technologies can help provide visual aid and audio cues for the proceduralists which is particularly helpful. This is especially when the proceduralist is fully gowned up in personal protective equipment (PPE), when their visual and auditory senses might be significantly impaired. XR technologies can provide visual aids which highlight important anatomical landmarks such as the epiglottis and glottis, with the audio cue to prompt the proceduralist that they are in the correct location for intubation [13]. This may aid with a smoother intubation which will limit the number of failed intubation attempts hence minimising duration of exposure to Covid-19 and also a reduced amount of aerosol generation [14]. Alternatively should XR technologies be sufficiently advanced enough, the end goal would be to develop a robotic surrogate to handle this high risk procedure in a safe and efficacious method for the patient [15, 16]. This benefit of minimizing proceduralist infection risk can also be extended to other procedures such as surgeries which would require surgeons to spend a prolonged amount of time in contact with Covid-19 patients [17]. XR technologies can also assist with the identification of patients with Covid-19 infection. New computational models are being developed which incorporate specific algorithms to help radiologists identify radiographic changes associated with Covid19 versus that of pneumonias of other origin [18, 19]. This will greatly assist in the diagnostic process when identifying and isolating patients with Covid-19 as the

1 Use of Extended Reality in Medicine During the Covid-19 Pandemic

5

current gold standard currently relies upon molecular testing which may take up to 24 h prior to receiving results [20]. Earlier identification of Covid-19 patients allows for earlier contact tracing and isolation of suspected contacts which will assist in breaking the chain of infection. Moreover, some of these algorithms have also been developed to assist with identifying Covid-19 patients who are at greater risk for deterioration hence more medical attention can be diverted towards them [21, 22]. This will help augment other clinical parameters used to highlight risky Covid-19 patients allowing for earlier intervention should Covid-19 complications arise. Digitalization of healthcare systems is an ongoing process with different institutions in different countries at different stages of digitalization. XR technologies can help catalyse this process driven by a strong impetus from the Covid-19 pandemic. The Covid-19 pandemic has generated high patient load on healthcare systems hence traditional methods of patient care have to be modified to suit the new environment. XR technologies can also facilitate the remote monitoring of many Covid-19 patients and enhance the efficiency of traditional medical ward rounds where the patient’s clinical status are reviewed and medical management plans are made. Covid-19 patients in isolation could have their vital signs monitored remotely via wearable medical non-invasive monitoring devices [23, 24]. The data can then be projected onto computer screens for patient monitoring. This would allow nurses and doctors to remotely monitor multiple patients while minimizing infection exposure risk. To facilitate medical rounds, instead of the tradition of bringing along a pen or paper to scribe or carting around a cumbersome computer on wheels, patients’ data should be able to be projected beside the patient via AR when they are reviewed. The information that would be displayed should be their recent vital signs, relevant blood tests and imaging reports. Upon reviewing the patient, doctors should then dictate their medical findings and the subsequent medical plans which should be automatically typed out by a computer system. These systems would allow doctors to fully devote their attention to clinically reviewing the patient instead of scribing information for medical record purposes. Apart from increasing efficiency of ward rounds, these XR technologies would also limit the number of items that would require decontamination because of SARS-CoV-2 exposure. XR technologies can also assist with rapid screening of temperature and symptoms in an asymptomatic population prior to entering facilities as part of infection control measures [25, 26]. A preassigned questionnaire enquiring about visitor symptoms working with a remote temperature screening tool through an algorithm can help identify visitors at risk of transmitting Covid-19 to vulnerable patients. Identifying at risk individuals prior to them entering facilities which house vulnerable individuals such as hospitals and nursing homes can assist with infection control.

6

S. S. Y. Wang

1.5 Employing XR Technologies to Minimize Covid-19 Disruption Medical education is one of the fields which XR technologies have made significant developments even prior to the Covid-19 pandemic. XR technologies have been used to help students better appreciate human anatomy and physiology [27]. During the Covid-19 pandemic, this technology has become invaluable as face-to-face lessons particularly lab lessons have been reduced or cancelled to help curb the pandemic. To ensure that education in the clinical and basic sciences are not compromised, XR technologies can be used to supplement current virtual teaching modes [28]. AR technology can be used to visualize more complex and dynamic systems such as a real-time visualization of 3D lung dynamics superimposed directly on a manikin or on a patient in the operating room [27]. Other examples include training of technically complex skills such as laparoscopy skills with AR [27]. Minimalistic invasive surgery is more demanding for surgeons in terms of concentration, focused attention and the execution of complex psychomotor skills. Hopefully with XR technologies, the disruption to medical education can be minimized and that the experience with XR technologies in medical education can be used to improve regular clinical learning once the Covid-19 pandemic is contained. Multi-disciplinary meetings a key feature of medical management of complex cases have also been greatly disrupted during the Covid-19 pandemic [29]. Due to concerns of Covid-19 infection because of prolonged contact between individuals in an enclosed environment. The consequences can be catastrophic if multiple medical personnel were to be infected by an asymptomatic individual. Hence in view of this concern, most multidisciplinary meetings have either been suspended or conducted virtually. XR technologies can help facilitate virtual multi-disciplinary meetings by allowing for clinical information to be communicated more seamlessly during medical discussions. XR technologies can allow for the patient’s physical and clinical status to be better appreciated. Detailed radiological imaging can be broadcasted alongside biochemical test results and vital signs to all participants of the meeting. This would allow for this discussion to be more interactive amongst participants despite it being a virtual meeting. A re-evaluation of traditional healthcare delivery models via physical outpatient clinic follow-ups should be performed as outpatient clinics helps generate a substantial amount of crowd movement raising potential infection control risk [30]. XR technologies can help assist with this by facilitating telehealth outpatient reviews for selected patients [31]. Therefore, cutting down crowd movements within and without the hospital. That being said the key challenge for the clinician would be identifying patient’s suitable for this type of service, ideally it should be patients who are technically savvy, clinically stable, socially well supported and are medically informed. These patients can be reviewed remotely by doctors using XR technologies to assess the patient’s clinical status [31]. Should the patient have wearable devices that can monitor basic healthcare parameters they can also be reviewed remotely [32, 33].

1 Use of Extended Reality in Medicine During the Covid-19 Pandemic

7

After evaluating the patients, doctors can then decide to continue current medications or order other investigations which does not require the patient to be physically present. The medications can then be delivered to the patient by a home delivery pharmacy service [34]. This may help to cut down clinical burden whilst promoting social distancing and infection control [35, 36]. Other clinical services that do not typically employ XR technologies that might benefit from XR technologies during the Covid-19 pandemic are palliative care and mental health services. These services typically rely heavily on face-to-face interaction between healthcare professionals and patients. The importance of the human touch and interaction is part of the therapeutic process for these services. But during to infection control and social distancing measures, these services might benefit from XR technologies that might aim to simulate such experiences as close as possible. Palliative care services were significantly affected by the Covid-19 pandemic. At the peak of the pandemic carers and family members were prohibited from visiting patients in hospices. While palliative care nurses and doctors were restricted from performing home visits to patients that were receiving palliative care at home. Such disruption to services might leave vulnerable patient’s feeling abandoned and alone during the final periods of their life. Hence XR technologies can help provide telemedicine support, patient assessment and counselling through such technologies can hopefully enable a similar care and support as traditional service delivery models [37]. During the virtual consults, patient clinical status and symptoms can be assessed and medical treatments can be recommended. More advanced XR technologies would better aid clinicians in assessing the patient and instituting treatments [38]. XR technologies can also be used to help palliative patients to fulfil certain final wishes which may be limited due to lockdown measures [39]. XR technologies can also help connect palliative patients with their extended family who are unable to visit them in person [39]. This will help provide closure for both patient and family members. It is likely that the technology would not replace face-to-face encounters, however it can offer meaningful connection and support to families and patients during these difficult times [40]. Mental health services is another area that has been heavily disrupted by Covid19. Similar to palliative care services, mental health services are heavily reliant on in person patient reviews. Telemedicine has made some inroads into mental health services however most of it has been trailed in remote and rural settings [41]. Telemedicine in mental health shows promise but has not currently been widely adopted [42]. Paradoxically the Covid-19 pandemic has actually increased the need for mental health services as members of the population become socially isolated due to social distancing measures [43]. This problem is particularly worse for Covid-19 patients who are often isolated for a prolonged period of time and may have limited direct human contact [43]. These prolonged periods of social isolation will increase anxiety and mood related symptoms [43]. A method of mitigating these mental health issues is to ensure connectivity and communication for people kept under isolation. XR technologies can help in this regard to ensure that people remain socially distant without being socially isolated. These XR technologies can also enable patients to

8

S. S. Y. Wang

seek mental health services and allow for mental health professionals to assess and treat patients through counselling and psychotherapy [44].

1.6 Future Challenges of XR Technologies in Medicine The Covid-19 pandemic has unveiled that clinical medicine has not been prepared to readily adopt XR technologies. There is still much progress to be made when trying to adopt XR clinical technologies in clinical medicine [45, 46]. Chief among them is building up the digital infrastructure and technical expertise to support the adoption of XR technologies in clinical medicine. Stable and high-speed internet/intranet connections will be critical to ensure reliability of the XR technologies that are used in clinical practice [46]. This is important as these technologies will influence clinical decision making for doctors and hence directly impact upon patient outcomes. Storage and management of the data generated from XR technologies is also important as ease of data retrieval, security/confidentiality of data and fidelity of the data is important for routine clinical application of XR technologies [46]. This ensures that the data from XR technologies can be relied upon and be easily retrieved and clinically acted upon. In summary a very holistic and well-planned pipeline must first be established to allow for XR technologies to flourish clinically. This might also require some clinical oversight/management committees to be formed to regularly review the applications of XR technologies in medicine [47]. They can help to decide if the various clinical services are ready to adopt such technologies based on the clinical need and environment suitability. This is important as such technologies are often novel and have not been field tested in sufficiently large patient cohorts under different clinical scenarios, therefore they may not be as robust to meet the varying clinical needs. Digital literacy is an important aspect to address amongst the medical community [48–50]. Due to the highly complex and subspecialized nature of medicine, clinicians may often neglect key general skills such as digital literacy which would be an important skill in the future with the digital revolution [49]. Moreover, it is a key obstacle to overcome for the adoption of XR technologies in clinical practice. Clinicians must feel safe and comfortable with XR technologies prior to adopting such technologies in their routine clinical practice [50, 51]. Becoming digitally literate which may involve increasing familiarity with the underlying principles which underpin XR technology will help clinicians understand and trust such new technologies [52]. Clinicians will also be in a better position to convince and advice their patients when adopting such technologies in the future [52]. This will support a joint partnership between patients and clinician when adopting XR technologies. A potential problem that may accompany adoption of XR technologies would be the overreliance of XR technologies by future generations of clinicians [53, 54]. This may lead to a loss of clinical acumen and basic clinical skills [54]. This may worsen clinical outcome and efficiency of clinical management, as clinicians would utilize potentially high cost XR technologies in cases which potentially may not

1 Use of Extended Reality in Medicine During the Covid-19 Pandemic

9

require them [55, 56]. This may lead to high medical costs and potentially delayed diagnosis and treatment as clinicians would not be confident of their own clinical skills when managing patients [55, 56]. Clinicians should try to view XR technologies as another tool in their management/diagnostic armamentarium when approaching patients. XR technologies should not be viewed as the immediate solution when problems are encountered especially if there are simpler/cheaper solutions available. Equitable access to XR technologies is another potential challenge, there is a risk that XR technologies may only be accessible to patients who have financial resources or possess digital literacy [57–59]. As with all new technologies it is likely that XR technologies will not be as affordable when they are first clinically used [59]. The cost of setting up the XR technologies and training staff in its application would be considerable. Hence to ensure equitable access, government agencies should put in the initial investment as part of a broader digitalization strategy for the nation [60, 61]. This will ensure that pockets of the population will not be left behind in the digital revolution for medicine [62, 63]. Public campaigns should also be run to raise awareness about the digitalization of medicine so that the public will be more aware of the newer technologies employed in medicine [64]. This can be done as part of a public health campaign done to raise awareness about XR technologies and telehealth [64]. This will help set the appropriate expectations from the public and increase the likelihood that the public would more likely to be engaged in the digital revolution. Thus, when the digital revolution in medicine is rolled out the public would likely be more receptive.

1.7 Future Applications of XR Technologies in Medicine With the progress of the digital age, it is inevitable that clinical medicine will begin to embrace XR technologies. Covid-19 has shown the importance of being prepared to adopt XR technologies in clinical medicine. There is much opportunity for clinical medicine to readily adopt XR technologies in the future with the upcoming 5th generation broadband cellular networks which are being developed worldwide [65]. This digital infrastructure would enable high speed internet connections which would facilitate the operation of various XR technologies in clinical medicine [66]. For example, the internet connectivity would be more stable and the higher bandwidth would increase the feasibility and reliability of telehealth services. The likelihood for newer XR technologies being developed would increase enabling remote monitoring and treatment of patients from their home or in isolation facilities [67, 68]. A case in point would be the increasing popularity of wearable health devices by patients. This technology would enable a steady stream of real-time clinical data which clinicians can use during their virtual consults [69, 70]. Examples of such devices have been the FitBit or similar devices which can measure heart rate, blood pressure, temperature and even perform an electrocardiogram [71, 72]. These sets of information can then be integrated into future healthcare/clinical informatics which is an emerging field of computing which manages and assesses the

10

S. S. Y. Wang

impact of healthcare information systems and technologies, providing guidance on how they can be designed, used and managed for better and more efficient healthcare outcomes [73, 74]. Data generated can then help guide both clinicians and healthcare planners to make more informed medical decisions for both for individual patients and healthcare systems. This real-time information can help clinicians and healthcare planners to readily adapt medical decisions and healthcare policy to changing clinical situations [75]. Medicine is highly complex field due to the highly dynamic nature and multiple interacting factors. Hopefully by being able to monitor the various inputs and outputs and integrating the information would a more holistic and complete picture of medical services be formed [75]. Adoption of XR technologies by clinical medicine is simply a microcosm of the ongoing digital revolution and the state of a nation’s digitalization. Successful adoption of XR technologies in medicine hinges upon a whole of nation initiative to embrace the digital revolution. This will require coordination between both public and private agencies to develop the digital infrastructure and regulatory framework [76]. Only with such support will traditionally minded medical industry would be more willing to adopt new XR technologies in routine clinical practice.

1.8 Conclusion The Covid-19 pandemic has posed both challenges and opportunities to healthcare professionals and healthcare systems. Rapid adoption of new technologies has become part of the response to the pandemic. A clear example is the development and approval of Covid-19 vaccines in a record span of a year versus the usual development time of a decade. Similarly, medicine has found a new reason to harness XR technologies which can be used in various fields. XR technologies can be used to assist in combating the Covid-19 pandemic while also mitigating the disruption caused to the delivery of regular healthcare services. The pandemic has also exposed multiple areas in clinical medicine that can benefit from XR technologies beyond the current pandemic. By seeing the robustness and potential efficiencies that can be derived from XR technologies there will be a future impetus to continue investing in the infrastructure that will support XR technologies. Ultimately the lessons learnt from the Covid-19 pandemic has served as a catalyst which will transform the future of healthcare delivery in years to come. Allowing healthcare to experience a digital renaissance which will be able to tackle both the current Covid-19 pandemic and future health crises.

1 Use of Extended Reality in Medicine During the Covid-19 Pandemic

11

References 1. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graphics Appl. 21(6), 34–47 (2001) 2. Cipresso, P., Giglioli, I.A.C., Raya, M.A., Riva, G.: The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature. Front Psychol. 9, 2086 3. Peterson, D.C., Mlynarczyk, G.S.: Analysis of traditional versus three-dimensional augmented curriculum on anatomical learning outcome measures. Anat. Sci. Educ. 9(6), 529–536 (2016) 4. Chan, F., Aguirre, S., Bauser-Heaton, H., Hanley, F., Perry, S. (eds.): Head tracked stereoscopic pre-surgical evaluation of major aortopulmonary collateral arteries in the newborns. In: Radiological Society of North America 2013 Scientific Assembly and Annual Meeting (2013) 5. Qian, M., Jiang, J.: COVID-19 and social distancing. J. Publ. Health 1–3 (2020) 6. Lewnard, J.A., Lo, N.C.: Scientific and ethical basis for social-distancing interventions against COVID-19. Lancet. Infect. Dis 20(6), 631–633 (2020) 7. Shaker, M.S., Oppenheimer, J., Grayson, M., Stukus, D., Hartog, N., Hsieh, E.W.Y., et al.: COVID-19: pandemic contingency planning for the allergy and immunology clinic. J Allergy Clin Immunol: Pract. 8(5), 1477–88.e5 (2020) 8. Schwartz, A.M., Wilson, J.M., Boden, S.D., Moore, T.J, Jr., Bradbury, T.L., Jr., Fletcher, N.D.: Managing resident workforce and education during the COVID-19 pandemic: evolving strategies and lessons learned. JBJS Open Access 5(2) (2020) 9. Dornan, T.: Osler, Flexner, apprenticeship and ‘the new medical education.’ J. R. Soc. Med. 98(3), 91–95 (2005) 10. Lee, A.K.F., Cho, R.H.W., Lau, E.H.L., Cheng, H.K., Wong, E.W.Y., Ku, P.K.M., et al.: Mitigation of head and neck cancer service disruption during COVID-19 in Hong Kong through telehealth and multi-institutional collaboration. Head Neck 42(7), 1454–1459 (2020) 11. Jain, U.: Risk of COVID-19 due to shortage of personal protective equipment. Cureus 12(6), e8837–e (2020) 12. Howard, B.E.: High-Risk Aerosol-generating procedures in COVID-19: respiratory protective equipment considerations. Otolaryngol. Head Neck Surg. 163(1), 98–103 (2020) 13. Qian, M., Nicholson, J., Tanaka, D., Dias, P., Wang, E., Qiu, L. (eds.): Augmented reality (AR) assisted laryngoscopy for endotracheal intubation training. In: Virtual, Augmented and Mixed Reality Applications and Case Studies. Springer International Publishing, Cham (2019) 14. Kaur, R., Weiss, T.T., Perez, A., Fink, J.B., Chen, R., Luo, F., et al.: Practical strategies to reduce nosocomial transmission to healthcare professionals providing respiratory care to patients with COVID-19. Crit. Care 24(1), 571 (2020) 15. Hemmerling, T.M., Taddei, R., Wehbe, M., Zaouter, C., Cyr, S., Morse, J.: First robotic tracheal intubations in humans using the Kepler intubation system. BJA: Br. J. Anaesth. 108(6), 1011– 1016 (2012) 16. Wang, X., Tao, Y., Tao, X., Chen, J., Jin, Y., Shan, Z., et al.: An original design of remote robot-assisted intubation system. Sci. Rep. 8(1), 13403 (2018) 17. Zemmar, A., Lozano, A.M., Nelson, B.J.: The rise of robots in surgical environments during COVID-19. Nat. Mach. Intell. 2(10), 566–572 (2020) 18. Bai, H.X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., et al.: Artificial intelligence augmentation of radiologist performance in distinguishing COVID-19 from Pneumonia of other origin at chest CT. Radiology 296(3), E156–E165 (2020) 19. Harmon, S.A., Sanford, T.H., Xu, S., Turkbey, E.B., Roth, H., Xu, Z., et al.: Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nat Commun. 11(1), 4080 (2020) 20. Mei, X., Lee, H.C., Diao, K.Y., Huang, M., Lin, B., Liu, C., et al.: Artificial intelligence-enabled rapid diagnosis of patients with COVID-19. Nat Med. 26(8), 1224–1228 (2020) 21. Li, M., Lei, P., Zeng, B., Li, Z., Yu, P., Fan, B., et al.: Coronavirus disease (COVID-19): spectrum of CT findings and temporal progression of the disease. Acad. Radiol. 27(5), 603–608 (2020)

12

S. S. Y. Wang

22. Suri, J.S., Puvvula, A., Biswas, M., Majhail, M., Saba, L., Faa, G., et al.: COVID-19 pathways for brain and heart injury in comorbidity patients: a role of medical imaging and artificial intelligence-based COVID severity classification: a review. Comput. Biol. Med. 124, 103960 (2020) 23. Mohammadzadeh, N., Gholamzadeh, M., Saeedi, S., Rezayi, S.: The application of wearable smart sensors for monitoring the vital signs of patients in epidemics: a systematic literature review. J. Ambient. Intell. Hum. Comput. 1–15 (2020) 24. Dias, D., Paulo Silva Cunha, J.: Wearable health devices-vital sign monitoring, systems and technologies. Sensors (Basel) 18(8), 2414 (2018) 25. Tong, Ng W.: COVID-19: protection of workers at the workplace in Singapore. Saf. Health Work (2020). https://doi.org/10.1016/j.shaw.2020.09.013 26. Pulia, M.S., O’Brien, T.P., Hou, P.C., Schuman, A., Sambursky, R.: Multi-tiered screening and diagnosis strategy for COVID-19: a model for sustainable testing capacity in response to pandemic. Ann. Med. 52(5), 207–214 (2020) 27. Kamphuis, C., Barsom, E., Schijven, M., Christoph, N.: Augmented reality in medical education? Perspect. Med. Educ. 3(4), 300–311 (2014) 28. Iwanaga, J., Loukas, M., Dumont, A.S., Tubbs, R.S.: A review of anatomy education during and after the COVID-19 pandemic: revisiting traditional and modern methods to achieve future innovation. Clin. Anat. 34(1), 108–114 (2021) 29. Wahadat, A.R., Sadeghi, A.H., Tanis, W.: Heart team meetings during COVID-19: different formats of multidisciplinary heart team meetings under the gathering restriction rules due to the coronavirus disease-2019 pandemic are discussed. Eur. Heart J. 41(20), 1872–1874 (2020) 30. Rogawski McQuade, E.T., Guertin, K.A., Becker, L., Operario, D., Gratz, J., Guan, D., et al.: Assessment of seroprevalence of SARS-CoV-2 and risk factors associated with COVID-19 infection among outpatients in Virginia. JAMA Netw. Open. 4(2), e2035234–e (2021) 31. Ford, D., Harvey, J.B., McElligott, J., King, K., Simpson, K.N., Valenta, S., et al.: Leveraging health system telehealth and informatics infrastructure to create a continuum of services for COVID-19 screening, testing, and treatment. J. Am. Med. Inform. Assoc. 27(12), 1871–1877 (2020) 32. Kaushik, A., Patel, S., Dubey, K.: Digital cardiovascular care in COVID-19 pandemic: a potential alternative? J. Card. Surg. 35(12), 3545–3550 (2020) 33. Wosik, J., Fudim, M., Cameron, B., Gellad, Z.F., Cho, A., Phinney, D., et al.: Telehealth transformation: COVID-19 and the rise of virtual care. J. Am. Med. Inform. Assoc. 27(6), 957–962 (2020) 34. Zaidi, S.T.R., Hasan, S.S.: Personal protective practices and pharmacy services delivery by community pharmacists during COVID-19 pandemic: results from a national survey. Res. Social Adm. Pharm. 17(1), 1832–1837 (2021) 35. Koonin, L.M., Hoots, B., Tsang, C.A., Leroy, Z., Farris, K., Jolly, T., et al.: Trends in the use of telehealth during the emergence of the COVID-19 pandemic—United States. MMWR Morb Mortal Wkly Rep. 69(43), 1595–1599 (2020) 36. Aaron Baum, P.J.K., Mark, D.: Schwartz. Reduced in-person and increased telehealth outpatient visits during the COVID-19 pandemic. Annal. Internal Med. 174(1), 129–319 (2021) 37. Worster, B., Swartz, K.: Telemedicine and palliative care: an increasing role in supportive oncology. Curr. Oncol. Rep. 19(6), 37 (2017) 38. Carbone, M., Freschi, C., Mascioli, S., Ferrari, V., Ferrari, M. (eds.): A wearable augmented reality platform for telemedicine. In: Augmented Reality, Virtual Reality, and Computer Graphics. Springer International Publishing, Cham (2016) 39. Wang, S.S., Teo, W.Z., Chai, Y.W.: Virtual reality as a bridge in palliative care during COVID19. J. Palliat. Med. (2020) 40. Ritchey, K.C., Foy, A., McArdel, E., Gruenewald, D.A.: Reinventing palliative care delivery in the Era of COVID-19: how telemedicine can support end of life care. Am. J. Hosp. Palliat. Med. 37(11), 992–7 (2020) 41. Edwards, M.A., Patel, A.C.: Telemedicine in the state of Maine: a model for growth driven by rural needs. Telemed. J. E Health 9(1), 25–39 (2003)

1 Use of Extended Reality in Medicine During the Covid-19 Pandemic

13

42. Newman, L., Bidargaddi, N., Schrader, G.: Service providers’ experiences of using a telehealth network 12 months after digitisation of a large Australian rural mental health service. Int. J. Med. Informatics 94, 8–20 (2016) 43. Pfefferbaum, B., North, C.S.: Mental health and the Covid-19 pandemic. N Engl J Med. (2020) 44. Gros, D.F., Morland, L.A., Greene, C.J., Acierno, R., Strachan, M., Egede, L.E., et al.: Delivery of evidence-based psychotherapy via video telehealth. J. Psychopathol. Behav. Assess. 35(4), 506–521 (2013) 45. Chen, L., Day, T.W., Tang, W., John, N.W. (eds.): Recent developments and future challenges in medical mixed reality. In: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (2017) 46. Kabalci, Y.: 5G mobile communication systems: fundamentals, challenges, and key technologies. In: Kabalci, E., Kabalci, Y. (eds.) Smart Grids and Their Communication Systems, pp. 329–359. Springer Singapore, Singapore (2019) 47. Hilty, D.M., Randhawa, K., Maheu, M.M., McKean, A.J.S., Pantera, R., Mishkind, M.C., et al.: A review of telepresence, virtual reality, and augmented reality applied to clinical care. J. Technol. Behav. Scie. 5(2), 178–205 (2020) 48. Terry, J., Davies, A., Williams, C., Tait, S., Condon, L.: Improving the digital literacy competence of nursing and midwifery students: a qualitative study of the experiences of NICE student champions. Nurse Educ. Pract. 34, 192–198 (2019) 49. Machleid, F., Kaczmarczyk, R., Johann, D., Balˇci¯unas, J., Atienza-Carbonell, B., von Maltzahn, F. et al.: Perceptions of digital health education among european medical students: mixed methods survey. J. Med. Internet Res. 22(8), e19827–e (2020) 50. Mesko, B., Gy˝orffy, Z., Kollár, J.: Digital literacy in the medical curriculum: a course with social media tools and gamification. JMIR Med Educ. 1(2), e6 (2015) 51. Al-Qirim, N.: Championing telemedicine adoption and utilization in healthcare organizations in New Zealand. Int. J. Med. Inform. 76(1), 42–54 (2007) 52. Day, K., Gu, Y. (eds.): Influencing factors for adopting personal health record (PHR). HIC (2012) 53. McAlearney, A.S., Schweikhart, S.B., Medow, M.A.: Doctors’ experience with handheld computers in clinical practice: qualitative study. BMJ 328(7449), 1162 (2004) 54. Jamil, F., Farooqi, S., Waseem, T.: Why the art of clinician is dying? Opinions & judgements: a systematic 55. Sorenson, C., Drummond, M., Bhuiyan, K.B.: Medical technology as a key driver of rising health expenditure: disentangling the relationship. Clinicoecon Outcomes Res. 5, 223–234 (2013) 56. Hermer, L.D., Brody, H.: Defensive medicine, cost containment, and reform. J. Gen. Intern. Med. 25(5), 470–473 (2010) 57. Vollbrecht, H., Arora, V., Otero, S., Carey, K., Meltzer, D., Press, V.G.: Evaluating the need to address digital literacy among hospitalized patients: cross-sectional observational study. J. Med. Internet Res. 22(6), e17519 (2020) 58. van der Vaart, R., Drossaert, C.H.C., de Heus, M., Taal, E., van de Laar, M.A.F.J.: Measuring actual ehealth literacy among patients with rheumatic diseases: a qualitative analysis of problems encountered using health 1.0 and health 2.0 applications. J. Med. Internet Res. 15(2), e27 (2013) 59. Neter, E., Brainin, E.: eHealth literacy: extending the digital divide to the realm of health information. J. Med. Internet Res. 14(1), e19 (2012) 60. Strusani, D., Houngbonon, G.V.: What COVID-19 means for digital infrastructure in emerging markets (2020) 61. McGinnis, J.M., Powers, B., Grossmann, C.: Digital infrastructure for the learning health system: the foundation for continuous improvement in health and health care: workshop series summary (2011) 62. National Academies of Sciences E, Medicine. The promises and perils of digital strategies in achieving health equity: workshop summary (2016)

14

S. S. Y. Wang

63. Wood, B.R., Young, J.D., Abdel-Massih, R.C., McCurdy, L., Vento, T.J., Dhanireddy, S., et al.: Advancing digital health equity: a policy paper of the infectious diseases society of america and the HIV medicine association. Clin. Infect. Dis. 72(6), 913–919 (2020) 64. McKee, M., van Schalkwyk, M.C., Stuckler, D.: The second information revolution: digitalization brings opportunities and concerns for public health. Eur. J. Publ. Health 29(Supplement_3), 3–6 (2019) 65. Thuemmler, C., Paulin, A., Jell, T., Lim, A.K.: Information technology—next generation: the impact of 5G on the evolution of health and care services. In: Information Technology-New Generations, pp. 811–817. Springer (2018) 66. West, D.M.: How 5G technology enables the health internet of things. Brookings Center for Technology Innovation 3, 1–20 (2016) 67. Wu, T., Wu, F., Qiu, C., Redouté, J.M., Yuce, M.R.: A rigid-flex wearable health monitoring sensor patch for IoT-connected healthcare applications. IEEE Internet Things J. 7(8), 6932– 6945 (2020) 68. Manas, M., Sinha, A., Sharma, S., Mahboob, M.R.: A novel approach for IoT based wearable health monitoring and messaging system. J. Ambient. Intell. Humaniz. Comput. 10(7), 2817– 2828 (2019) 69. Fan, Y., Xu, P., Jin, H., Ma, J., Qin, L.: Vital sign measurement in telemedicine rehabilitation based on intelligent wearable medical devices. IEEE Access 7, 54819–54823 (2019) 70. Albahri, A.S., Alwan, J.K., Taha, Z.K., Ismail, S.F., Hamid, R.A., Zaidan, A.A., et al.: IoT-based telemedicine for disease prevention and health promotion: state-of-the-art. J. Netw. Comput. Appl. 173, 102873 (2021) 71. Thomson, E.A., Nuss, K., Comstock, A., Reinwald, S., Blake, S., Pimentel, R.E., et al.: Heart rate measures from the apple watch, fitbit charge HR 2, and electrocardiogram across different exercise intensities. J. Sports Sci. 37(12), 1411–1419 (2019) 72. Benedetto, S., Caldato, C., Bazzan, E., Greenwood, D.C., Pensabene, V., Actis, P.: Assessment of the fitbit charge 2 for monitoring heart rate. PloS one 13(2), e0192691 (2018) 73. Chen, M., Hao, Y., Hwang, K., Wang, L., Wang, L.: Disease prediction by machine learning over big data from healthcare communities. IEEE Access 5, 8869–8879 (2017) 74. Beam, A.L., Kohane, I.S.: Big data and machine learning in health care. JAMA 319(13), 1317–1318 (2018) 75. Bates, D.W., Heitmueller, A., Kakad, M., Saria, S.: Why policymakers should care about “big data” in healthcare. Health Policy Technol. 7(2), 211–216 (2018) 76. Rodriguez-Villa, E., Torous, J.: Regulating digital health technologies with transparency: the case for dynamic and multi-stakeholder evaluation. BMC Med. 17(1), 1–5 (2019)

Chapter 2

Extended Reality Based Remote Learning Experience During Pandemic: Effectiveness and Barriers Prabha Susy Mathew and Anitha S. Pillai

Abstract The COVID-19 pandemic has impacted almost all sectors due to lockdown measures taken by governments across the globe to curtail the spread of virus. It has disrupted the regular teaching and learning process, as students miss face-to-face learning experience. However digital technologies have mitigated the negative impact these lockdowns would have otherwise caused. The challenges faced because of remote learning can be handled by incorporating XR related technologies to enhance the collaborative learning experience for remote learners. The XR technologies and their applications in educational institutions are still in their nascent stages but show great potential and are all set to create a paradigm shift in the way education and training is perceived. The chapter will provide an insight on how schools, universities, and enterprise learning can be made more engaging by reviewing different techniques adopted to effectively engage the remote learners using immersive technologies. It also focuses on advantages, adoption challenges and framework for using XR in education especially during these testing times. Keywords Augmented reality (AR) · COVID-19 · Extended reality (XR) · Mixed reality (MR) · Remote learning · Virtual reality (VR)

2.1 Introduction COVID-19 pandemic has impacted most of the sectors ranging from education, research, tourism, retail, manufacturing, entertainment, etc. To contain the spread of the virus, measures such as lockdown and social distancing have pushed companies to find innovative and smarter solutions to run their businesses and to thrive in the given situation by reskilling and upskilling their workforces. Extended reality (XR) P. S. Mathew (B) Bishop Cotton Women’s Christian College, Bengaluru, India A. S. Pillai School of Computing Sciences, Hindustan Institute of Technology and Science, Chennai, India e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_2

15

16

P. S. Mathew and A. S. Pillai

Fig. 2.1 XR and its applications

has emerged to solve the issue of distance impediments, boost productivity, enhance remote learning experience, reduce errors and cost in almost every industry. The global XR market has thrived during the pandemic and has been used by several sectors to boost their sales and keep their business running. Some of the sectors [1, 2] that greatly benefitted from this technology are discussed (Fig. 2.1). XR in Retail-The new norm to ensure the social distancing, allowing fewer customers at the physical store at any given point and the need to sanitize the surface has forced the retail sector to use technology to keep their business running by opening a new channel to reach their customers. Pandemic has pushed the need for a virtual ‘try-before-you-buy’ model for enhancing the immersive shopping experience for the shoppers without leaving the comfort of their home and driving sales. XR has been explored by several brands [3, 4] like Levi’s, Lenskart, Fabindia, Marks & Spencer, Gucci, Tanishq, Ikea, Louis Vuitton, Audi, OnePlus, and L’Oréal to name a few. It provides customers a virtual try-on option for a seamless online shopping experience using just their smartphone and browser. Beauty retailers such as Sephora and Ulta are turning to AR to digitally help consumers test out products to assist in buying decisions during COVID-19 pandemic. Beauty retailer Levi’s in its pursuit to recreate social experience that people were missing in pandemic complemented its AR retail strategy with digital tools like Squad, an online co-watching video app, where friends can shop together [3]. XR in Healthcare-During the pandemic healthcare organizations have used technology extensively to track, contain the spread of virus and for vaccine development. Immersive technologies are used specifically for remote learning and collaboration, augmented remote surgery, pharmacy- specific training, AR live streaming for team training, AR anatomy, treatment planning, surveillance systems etc. [5, 6]. Virti, a U.K. based AR/VR startup uses VR simulation to retrain more than 15,000 healthcare workers so that they understand the viewpoint of a COVID-19 patient being treated with them [2]. Doctors and nurses are more susceptible to exposing themselves to COVID-19 virus as they come in direct contact with the patients, but with VR-based diagnostics, direct contact with the patient can be deferred until required [7]. XR in Manufacturing-During pandemic to avoid total lockdown and ensure safety of their workforce manufacturing industry adopted technology. They used XR to

2 Extended Reality Based Remote Learning Experience During Pandemic …

17

manage operations remotely, train the workers for using new machinery, reskill and upskill their workforces, handling hazardous materials, training them for remote product repair and maintenance activity. XR has emerged to reduce errors and costs, remove distance barriers, optimize productivity, enhance learning, and shorten the learning curve across almost every industry [8, 9]. XR in Real Estate-AR/VR has seen great usage in real estate business especially during the pandemic. It offers 360° virtual tour of properties, helps the interior designers to do their work more efficiently with greater customer satisfaction, helps with visualization of unfinished projects, immersive marketing and for remote communication with tenants [10]. XR in remote learning-The global education system has hugely been disrupted due to the ongoing pandemic forcing the educational institution to instantly switch to an unplanned and sudden online learning mode. Educators across the world are grappling to find ways to effectively engage the remote learners. The sudden change of pedagogy driven by pandemic has forced teachers to look for opportunities in XR technologies to make remote learning more engaging. Adoption to Immersive technologies especially in the educational and enterprise learning environments have seen an encouraging growth trend owing to the high quality and immersive experience it provides to its remote learners thereby improving the engagement and knowledge retention rate. The educational technology platforms helped students for continuing education and learn new courses while at home. In India, leading EdTech and online training firms saw 7.5 million new users and more than one billion minutes of watch time on its online platforms. They are using XR technologies to explain complex concepts and experiments in 3D making it clear that this tech results in more engaged and satisfied students especially during the pandemic [4]. This chapter provides insight into several instances where XR technologies can be used to make learning more meaningful and outcome oriented. This chapter aims to examine the use of XR technologies for continuity of learning during pandemic and provide an in-depth understanding of its opportunities, framework, and challenges.

2.2 Types of Immersive Technologies Virtual reality (VR): replaces a user’s real environment with a simulated environment, such as a remote destination or a virtual lecture room etc. It completely detaches user from the real world. With VR users get an immersive experience and can interact with 3D objects of the virtual world. The user experiences full immersion in the virtual environment and the medium to access the 3D virtual environment is via VR headset or HMD (Head Mounted Display). VR offers Non immersive Technology as well as Semi-Immersive and Fully Immersive Technology. Augmented reality (AR): overlays computer-generated digital content on top of the user’s real world. The superimposed digital overlay can superficially interact with the user’s real environment. The real world is enhanced with digital overlays. The most

18

P. S. Mathew and A. S. Pillai

common medium to access this environment is through wearable glass, headsets, smartphones, and tablets [1]. AR can be either Marker-based Augmented Reality with either Passive Marker or Active Marker and Markerless Augmented Reality for Model based Tracking and Image based Processing. Mixed reality (MR) is also termed as a hybrid reality that can interact with both real and virtual environments, providing an intermingled environment to the user. 360° Video: It allows users to view around him/her in all directions. Holographic Reality (HR): It is a virtual imaging technology which reproduces, and real records 3D images of an object based on the principle of light interference and diffraction. With HR technology the user is directly able to see 360-degree 3D images without any wearable device [11]. Extended Reality (XR): is the umbrella term to bring together emerging technologies such as Virtual reality (VR), augmented reality (AR), Holographic reality (HR), and mixed reality (MR) [5, 11] utilizing digital immersion and overlays on the real world that users can interact with under one term. In other words, it refers to combined real-and-virtual environments generated by computer technology and wearables which covers the full spectrum of real and virtual environments. These immersive technologies use the following senses to give its users immersive experience [12]. Visual: In XR devices, it gives users an immersive experience through a computergenerated reality as seen by the users. Such visual experience is mainly offered in video games, 360 degrees cameras. Typical devices used are neurotransmitters, goggles, HMDs. Audio: In XR devices, sound can be heard just the way one would experience in a live set up from all directions. In video games, users would experience sound in a way it would happen in real life. The audio tools such as Oculus,3D audio plugins etc. are used for the audio experience. Tactile: To improve immersive experience the sense of touch is also integrated into XR devices. Gloves, backpacks, and bodysuits are used to give users a sense of touch. Haptic technology is often used in wearable devices to provide kinesthetic feedback. Taste and Smell: Much of our perception of a meal relies on inputs from taste and smell. Based on this principle, research and development activities are underway to give users gastronomic experience. Scent cartridges provide a sense of smell to the user and electrodes placed on a user’s tongue can be used to simulate sense of taste. It will offer promising solutions for people suffering from obesity or the ones with eating disorders. All these senses are considered important for new teaching and learning methodologies. Including these multiple senses in XR design will enable learners to interact with their learning content through different senses making the process of learning more engaging and effective [13].

2 Extended Reality Based Remote Learning Experience During Pandemic …

19

2.3 XR Devices, Components, and Technologies XR technologies have recently seen an increasing trend in its use due to advancement in its hardware and computing technologies. While the current XR headsets in the market are good, there is still room for improvement to deal with latency issues, designing of less bulky devices, motion tracking, power, and display efficiency etc. The convergence of the devices such as smartphone, VR headset, and AR glasses into a single XR wearable could replace all the other screens. Mobile XR has the potential to become the most ubiquitous and disruptive computing platforms.

2.3.1 Components The major components of XR technology are hardware/devices, software, content, software development kit, and underlying technologies [14]. The ease with which the user can use XR technology has an especially important role in its success or failure. Hardware and Software To create an XR environment, hardware devices such as displays, projectors are required to experience the immersive environment, position trackers, vision systems in the form of cameras and sensors for wearable devices are used. XR technologies not only require hardware but also software. XR Software is used for running of hardware as well as for creating the XR content [9]. Software used for XR can be software applications and software development kits [14]. Some of the popular software applications are Unity3D, Blender3D, Autodesk Maya, Nuke, VRML Pad, EduVenture® VR system, SteamVR, GoogleVR Blocks, Scratch, App Inventor etc. [15]. Open Simulator and Second Life are open-source software for creating and sharing virtual environments. Similarly popular SDKs available are ARKit by Apple, Snapdragon VR SDK and Google’s ARCore.

2.3.2 Devices The XR devices that are currently available require either a smartphone or a robust PC to deliver a modest level of immersive experiences. The immersive experience delivered through these devices are limited. The smartphone is not powerful enough, while the PC requires it to be tethered with a wire creating a barrier for the immersive experience. Combining smartphones, VR headsets, AR glasses into one to create Mobile XR has tremendous potential to provide immersive experience. However, technical innovations are needed to create XR devices that offer powerful XR experiences more efficiently. XR has distinct hardware and software aspects. The hardware

20

P. S. Mathew and A. S. Pillai

market is dominated by VR headsets and the software market is mainly AR, due to the increasing Mobile AR popularity. VR Devices VR devices are essential for the users to experience the virtual world. The success of VR can be attributed to devices like Google cardboard, Oculus Rift, HTC Vive, Gear VR etc. VR devices can be classified as Head-Mounted Display (HMD), GestureTracking Device, Projector & Display Wall, Holograms [14, 16, 17] AR Devices AR devices enhance the perception of the user’s real world. The enhancement of the real world is achieved by using AR devices such as heads-up displays like zSpace, Magicleap, smartphones, Holographic displays, and AR glasses [18, 19]. MR Devices Mixed reality technology blends both physical and virtual worlds and allows them to interact, where users can interact with the virtual environment in a manner they would in the real world. MR is a type of AR that is more immersive and interactive. Several MR products like Microsoft’s HoloLens, Magic Leap One, Lenovo Explorer, Oculus Quest, Samsung Odyssey, Acer Windows Mixed Reality, Dell Visor MR etc. [14] (Table 2.1). Table 2.1 XR devices Devices

Immersive experience

Head-mounted display (HMD) or headsets Tethered (wired)

No. of users Single

Hindered due to wire

Standalone (wireless)

Fully immersive

Head-up display (HUD)

Immersive

Gesture-tracking device

Single/multiple Single

Wearable device-based recognition

Fully immersive

Touch technology-based recognition

Fully immersive

Computer vision-based recognition

Fully immersive

Projector & display wall

Semi immersive

Single/multiple

Holographic display

Fully immersive

Single/multiple

Smart glasses

Immersive

Single

Contact lenses

Immersive

Single

Handheld/mobile AR

Immersive

Single/multiple

Web based AR

Immersive

Single/multiple

Optical see through Video see through

2 Extended Reality Based Remote Learning Experience During Pandemic …

21

2.3.3 Underlying Technologies There are several underlying technologies that contribute towards overall XR experience [1, 11]. Some of them are. 5G connectivity: It provides reliable and high performance, low communication delay and high-quality content to the XR devices. The problem of discomfort or nausea when using head mounted displays due to time delay can be overcome using 5G. IoT is greatly benefitted from 5G as it will boost its ability for massive connectivity. Big data and AI: AI in XR are used for real time data rendering and to improve the quality of the content created. Cloud Computing: Scalability for the XR experiences can be obtained with the help of XR based Cloud Services. Deployment of Edge computing can improve user experience as it facilitates faster computing and provides data faster to the user. It greatly reduces the tedious task of graphic rendering and computing process for AI on XR terminals. Some of the providers are Microsoft Azure Cloud Services platform, Internet of things (IoT): Using IoT and XR will provide opportunities for the real and the virtual world to connect and interact. The XR technologies are driven by AI, connected to the IoT, and provided with scalability through the cloud computing solutions. 3D reconstruction: 3D reconstruction of scenes and objects is especially useful in the field of education. Object recognition and tracking systems: It enables the overlaying of a virtual object in real environments.

2.4 Mechanisms of XR Technologies to Facilitate Learning There is a constant change in the education system to keep up with the pace with which society is changing. Learners have varied requirements now compared to earlier times. Attention span of the learners is getting shorter so newer tools and technologies must be embraced to make learning more interesting and fun activity. Immersive learning through an emerging field now allows learners to acquire new knowledge and skills by providing them with an environment that mimics real life situations. The major benefit of immersive learning is it facilitates experiential learning, engages learners’ leading to improved retention rates and helps them in experiencing and understanding abstract scientific concepts by connecting them to the real world. Increasingly, XR-based educational applications are made available in the market especially during covid times with a goal is to enhance learning experience and promote remote learning.

22

P. S. Mathew and A. S. Pillai

XR technology though in its nascent stages has a great potential to enhance student and teacher experience. Students can visit and experience locations that are too far away without having to leave their place, remotely do their labs using virtual labs and perform experiments which otherwise is a concern due to lockdowns to curb COVID19 spread. Stanford University and Technical University Denmark found learners benefited more from XR based learning than with traditional methods. Students were more engaged, and it was found that there was an increase in knowledge retention by 76% [20]. There are several ways or mechanisms that can be used to enhance instruction [21] (Fig. 2.2). 3D visualization-3D visualization is used by teachers to enhance learners’ learning experience in a more visually engaging manner [22]. With the XR displays it is possible to project 3D objects into the learner’s physical environment and can create a shared experiences where multiple learners can view these 3D objects in their respective physical space [23]. EON-XR is an immersive platform for remote and collaborative learning, as well as it allows teachers to create interactive 3D models to design immersive content for their lesson. Rather than using the images provided in the textbook it allows teachers to create 3D interactive images and videos for their subjects. Liou and Chang [24] and Maresky et al. [25] found that 3D visualizations of the human body were extremely useful for medical students, where they could gain better understanding of human anatomy through 3D models when

Storytelling Role Play

Skill Building

Research

XR Mechanisms for Teaching and Learning

3D Visualization

Game Based Learning

Laboratory

Rich digital content

Fig. 2.2 XR mechanisms in education

Social Emotional Learning

2 Extended Reality Based Remote Learning Experience During Pandemic …

23

compared to traditional learning. With HoloAnatomy application for the Microsoft HoloLens learners can interactively explore human anatomy through holographic devices where students can even virtually dissect a body to get a closer look at organs [21] 0.3D virtual worlds are considered to be very effective solution for science labs, simulated training sessions and medical experimentations, providing opportunities for individuals involved a better option to understand concepts from their place of convenience and allowing learners to communicate with their peer group. During lockdown, such systems make it possible for learners from across the world to come together and share their experiences. Although there are many video conferencing platforms such as Zoom, Google Meets, Skype, WebEx, Teams that were quickly adopted by many for conducting online classes and discussions during lockdown imposed amidst pandemic these systems heavily relied on voice, video sharing or presentations and lacked the interactive and collaborative environment that XR can offer. XR tools can offer enhanced sense of immersion which leads to lesser distraction with more focus given to tasks on hand, personalized avatars, and dynamic visualization as well as interaction with these objects in 3D [26]. MeetinVR [27], Microsoft’s Atlspace [28], Rumii, Engage, Metaverse, Grove Classroom, XR Chat are some of the XR based platforms that provides fully immersive 3D meeting spaces with ability to create personalized avatars along with collaborative tools for 3D drawing [26, 29]. Learning by doing/Skill Building-It enables one to connect the physical and virtual world to support learners. Hands-on learning is especially critical in case of military, medical or manufacturing training. The COVID-19 pandemic has not just disrupted education in schools and universities but training across multiple disciplines. Situations such as these have led to many medical professions to use XR technologies for medical training, diagnosis, and treatment. It has been observed that surgical training using such tools increases success of surgical outcomes without putting patients at risk [21, 30]. Similarly nursing students can gain confidence in handling real life scenarios better as these technologies simulate the real-life situations and allows them to practice their skill repeatedly till, they gain confidence [31]. Lerner et al. [32] in their study created an EPICSAVE (Enhanced Paramedic Vocational Training with Serious Games and Virtual Environments) to gain insights into the training and learning effectiveness of an immersive multi-user VR simulation environment with the traditional training methods. From their study it was realized that VR experience was more engaging and an effective educational approach. Pedram et al. [33] in their study investigated the use of VR for conducting mine rescuers safety training and found a positive attitude towards technology and enhanced learning by trainees. De et al. [34] analysed the experience of online training sessions with simulated clinical scenarios for 122 final year medical students doing the course in Medicine and Surgery during the COVID-19 pandemic especially when in-hospital training was not possible. A VR platform Body Interact, with 21 patient-based clinical scenarios was used to ensure continuation and timely completion of their course. Most students were satisfied with their virtual training and felt that the quality of training met their expectations.

24

P. S. Mathew and A. S. Pillai

Rich digital content—Richer content is made available to the user by Augmenting physical objects. With richer content rather than using only text to explain concepts to the can help in engaging the learners visual and auditory senses. It has been observed that learners who were exposed to 3D content that is interactive were able to grasp the concept much better compared to learners without such richer content [21]. Students pursuing architectural courses can simulate their designs and make it look more realistic in comparison to the output they would get if they used computeraided designs [31]. Gopalan et al. [35] examined the impact of AR enhanced science textbooks on Malaysian lower secondary school students. It was found that learners were more engaged and motivated for learning science. Immersive Role-play and Collaborative learning—Cellverse, a collaborative and educational VR game [31] an MIT project was used to teach molecular biology. It requires 2 users: the explorer who wears a VR headset and the navigator who uses a tablet. The explorer views from inside the simulated cell, while the navigator gets a bird’s-eye view from outside the simulated cell. Both work as a team to choose a suitable therapy to cure the cell, demonstrating the ability of both users to recall information about the functions of organelles within the cell, the relationships between organelles, and can formulate hypotheses to fix the cell and then even execute it. Another similar project was Electrostatic Playground that teaches both Electromagnetism and electrical engineering at the same time where the user can manipulate simulated charged particles and gain understanding of how they react with each other. Thus, giving a better option to understand and experience the otherwise abstract concept. Such a responsive and interactive environment provides both a collaborative and competitive system. Natasha Skult and Jouni Smed [36] presented a project aiming at elementary school children. It was a multiplayer XR collaborative story game. The objectives for the educational XR system are set by the teacher. In the collaborative mode, the learner is required to collaborate and solve the tasks where individual actions in the virtual environment is visible to all the players and in competitive mode learners are expected to solve the tasks as fast as possible making learning fun and engaging activity. Social-Emotional Learning (SEL) through Interactive agent in XR—Several XR systems have virtual agents that can guide the learners in their learning activities. Learners can interact with the virtual agent which acts as a connection between the content and the learner. Such virtual agents can actively engage the learners by giving them audio feedback either to congratulate them on doing something correct or giving them cues so that they are encouraged to try something till they understand the concept [21]. For example, ARMath has Victor -a friendly virtual agent that interacts with students, and actively engages them in the math problem by giving them questions, guiding them, and summarizing the result [37]. XR in Research—Researchers are seeing potential in XR for experiential learning especially during the pandemic. The lab environment for researchers is difficult during this time as enclosed labs may have poor ventilation and sharing XR devices can also pose a threat for disease transmission. Ratcliffe et al. in [38] have highlighted

2 Extended Reality Based Remote Learning Experience During Pandemic …

25

several opportunities and drawbacks of XR in remote research. The team of Nano Simbox VR has stated that use of virtual environments has resulted in an increase in the rate of scientific discovery [39]. XR based Laboratory learning—Laboratory is an important process in education at schools or universities. However due to pandemic students are not able to do their experiments at their campus. Virtual lab environments are crucial to bridge the gap where students can acquire similar lab skills and remotely interact with their facilitators while they are at home [31]. Although holding virtual labs was not a practice earlier, now there are many disciplines using it. Garcia et al. in [40], presented the procedures followed by Norwegian university to conduct an XR lab during covid-19 pandemic to continue teaching and research. For medical students, residents, and surgeons, XR has been used for anatomical education, surgical procedure training, trauma procedures, resuscitation training, CPR training etc. Some of the institutes using XR in teaching and learning medicine mentioned in [41] are Stanford neurological simulation and virtual reality center, Cleveland clinic-HoloAnatomy (HoloLens), Children’s Hospital of Los AngelesImmersive Education (Oculus Go), university of California-VR Anatomy (HTC Vive), University of Illinois-Enduvo VR teaching and learning platform (HTC Vive), Texas Tech University Simulation Program and San Diego State University School of Nursing HoloPatient a MR (Microsoft HoloLens) and, a number of academic hospitals, like UCLA, New York University, Mount Sinai, Stanford University and Mayo Clinic are using Surgical Theater’s visualization platform that lets the surgeon or professor to present a case and demonstrate the treatment and surgical techniques to be applied [31]. Some more VR/AR applications in medicine are Curiscope VirtualiTee to learn about human body with AR, Bravemind VR app to train therapists to treat patients with Post-Traumatic Stress Disorder (PTSD) etc. [42] presents students with opportunity to learn, practice and perfect their skill. Similarly, learning chemistry requires lab practice so in chemistry [42], EduChemVr and unimersiv let students explore the concept of atoms and molecules. In [43] authors from their study found that students who used a virtual chemistry laboratory obtained a better understanding of the concepts. At Syracuse School of Architecture, Interactive Design and Visualization Lab (IDVL) uses XR to render architectural designs as it enables the user to walk around inside a space and allows them to interact with the objects within it [31]. For understanding concept of biomolecular electrostatics several AR/VR interfaces like UnityMol, Chimera, Nanome, and Nano Simbox are available that allow students or researchers to visualize and analyze molecular systems, their interactions, and biomolecular systems such as redox proteins and enzymes [39]. Storytelling Many educational XR applications use a creative approach of storytelling to engage the learners in the learning process while making fun. The storytelling engine uses avatars, sounds and chats to engage the learners actively in their learning tasks. Kang et al. [37] in their study found that AR storytelling was an effective way to engage

26

P. S. Mathew and A. S. Pillai

children to make them actively participate in solving math problems using a known scenario making it more relatable for the learner [21]. Sound Storytelling has been used in several places for language subjects. At Yale University, sound recording of rural Indonesia is used to immerse the learner in the Bahasa Indonesia language. XR can enhance language learning by simulating realistic language environments [31]. Game based Learning Game Based learning when integrated in a remote learning environment can make the learning process more enjoyable, promotes critical thinking, increases participation, and actively engages the learner providing them a more comprehensive understanding of the concept. Subhash and Cudney [44], in their review have observed that gamified teaching and learning practices when integrated in higher education, can improve students’ attitude, participation and learning outcomes.

2.5 XR Based Teaching and Learning—Framework The pandemic has resulted in disruption of education and professional training worldwide. Measures to contain the spread of virus required measures like social distancing and work from home for both educators and students. As a result, there was a need for a sudden paradigm shift from offline learning to online learning leading to changes in how the educational content is to be delivered. XR is changing the way the process of teaching and learning was perceived before the COVID-19 pandemic. In traditional teaching and learning environments learners had to imagine several concepts as textbooks or lectures just gave information that was difficult to conceptualize. With XR it is possible to simulate complex events by superimposing it in the learner’s real world. In recent times there has been a growing number of studies presenting immersive technologies as pedagogical tools in education. Several studies have utilized these tools in teaching different subjects, the domains identified in the reviews are for University, school, industrial training, medicine and learning foreign languages. However, the success of using XR in education often relies on faculty acceptance, reduced cost of implementation, learner’s adoption and their feedback and quality of learning experience [41]. Experts in the industry feel the need to transform learning to make it effective in the new normal. But such a sudden shift requires a framework [21, 41, 42, 45] for implementing the changes in the learning system (Fig. 2.3). Instructional Strategy Design XR has all the potential to be a game changer for pedagogy, but to do so it must be carefully integrated into the teaching and learning process. One of the most important criteria for adoption of XR in education is that XR applications must match with the educator’s existing instructional objectives for the course [31, 46].

2 Extended Reality Based Remote Learning Experience During Pandemic …

27

Fig. 2.3 Framework for XR in education

Instruction Delivery design Content design can be done by analyzing the content that has to be delivered. Based on that information, a decision must be made on whether an already existing app can be used or if modification to content on the app is required or new content needs to be created using the XR tools [47]. It is also important to classify subjects into lecture based and laboratory-based subjects [48] to identify appropriate XR based mechanisms required for simulated learning. Several instructional mechanisms used in teaching and learning process have been discussed in [15, 21, 37, 41, 47–49]. Mapping Instructional Technology To integrate XR as a pedagogical tool, educators must be encouraged to have a strong involvement right from the early stage of instruction design. Evaluation of the technology and its feasibility of incorporating in the classroom is an important aspect of instructional delivery [50]. To integrate the new technology into classroom instruction Institutes must provide support to faculty by hiring outsourced developers and technology specialists, as not all teachers can create immersive content on their own due to lack of technical skills. Both faculty and learners should have access to hardware necessary for immersive learning, show willingness to adapt to new technologies, must be trained, and made comfortable to integrate XR into classroom learning to make the process more engaging and to improve learning outcome [9]. Assessment Design Assessment is a very crucial part in the teaching and learning process as it checks whether learning outcomes are met or not. It helps teachers to understand if their choice of instructional mechanism was able to deliver the desired outcome through students’ performance [21, 51]. XR Simulations used for student learning can be taken as an assessment tool to measure the performance of the student. The instructor can evaluate it based on students’ interactions with the simulation during the learning

28

P. S. Mathew and A. S. Pillai

activity based on the learning outcomes and criteria set [31]. Zweifach et al. [41] in their study found XR as a great assessment tool for medical students where students struggling to master a skill could modify and use the XR tool to practice it repeatedly in a safe environment till significant confidence is gained. Hamilton et al. [52], from their study demonstrated that the majority of those who use I-VR (Immersive virtual reality) in education yielded better learning outcomes, while a small number of studies demonstrated no significant improvement. Johnston et al. [53], on testing of I-VR experience in students found that students trained with immersive VR scored better and had good understanding of cellular processes than those who did not undergo I-VR experience. Delivery of course content and Feedback from learners The pandemic has forced everyone to conduct classes remotely. XR technologies were able to give learners interactive and situated learning opportunities from their home without discontinuation of academic activities and compromising on the quality of education delivered. Several studies have taken feedback from the learners to improve their learning experience and have found that learners prefer to use new technologies such as XR, as it facilitates active learning through interactive simulations where they can perfect their skill at their own pace, gained better understanding of the concept [30, 37, 42, 45, 48, 52].

2.6 AR, VR, MR Apps for Classroom Learning As education and training institutes around the world are looking for alternate teaching and learning processes due to pandemic, immersive learning has been seen as a preferred medium for instruction delivery. A search on the internet reveals that there are several apps available that can be used for classroom learning. Some of them are Table 2.2.

2.7 Benefits of XR Based Remote Learning. XR in education has proved to be a great tool for learners to visualize complex and abstract concepts, get an immersive experience and collaborative environment for learning. Some of the benefits identified from the review are stated (Table 2.3).

2 Extended Reality Based Remote Learning Experience During Pandemic …

29

Table 2.2 XR apps in education [9, 31, 41, 42, 54–59] AR, VR, MR APPs

Description

Virtual tours 360 cities

Explore and take a tour of anyplace in the world with a 360° view

Google expeditions

To take virtual trip to destinations

Unimersiv

Provides 7 educational VR experiences like exploring human brain, International Space Station, human anatomy, Stonehenge, the Titanic, the Acropolis of Athens, and ancient Rome

Wild eyes

For virtual field trips

Woofbert VR

Students can explore Virtual museum

Timelooper

Free app that allows students to explore t back in time in London, from medieval times to World War II

Boulevard AR

Provides high quality art, architecture, and cultural experience

Content creation and subject learning AlchemyVR

A VR studio to create immersive educational experience

Arloopa

It is an AR Visualization tool, with built in 3D models with scientific and educational objects

ThingLink

Allows teachers to create interactive images and videos

AugThat

Provides AR based content for teaching and learning

ARize

Let’s one to link directly to website from the AR experience

CoSpaces Edu

It allows students to create 3D AR and VR environment

AR flashcards-alphabet & more

AR based learning for younger students

EON reality

A tool to create a blended-learning environment for classroom education and industrial training

ENGAGE

It is a platform that allows educators and trainers to share their content in a virtual environment, engaging learners in a classroom, or an operation theatre

VictoryXR

Gamified learning for middle and high school science

Nearpod

Offers free VR-based curriculum for teachers

Curiscope

Learn and explore human body (continued)

30

P. S. Mathew and A. S. Pillai

Table 2.2 (continued) AR, VR, MR APPs

Description

Quiver

3D colouring experience and help students understand concepts in subjects such as sciences and math

ARMath

Provides AR environment to learn math

zSpace

Focuses on learner specifically STEM (science, technology, engineering and mathematics) education, medical instruction, corporate training, research, and design

GeoGebra

An interactive tool that supports (STEM) education through AR

Universe sandbox

Physics based space simulator

Cellverse

To teach molecular biology

EduChemVr

to teach Chemistry

MEL chemistry VR

Lessons are aligned with school chemistry

Titans of space

Provides educational guided tour of the Solar System

Curiscope

Learn and explore human body

FotonVR

Provides solutions of VR classroom set up and designed for school to teach science subjects

BRIOVR

Provides a platform to build create and share VR content for immersive learning environment

TheBodyVR

Gives experience that takes learners inside human cell

Classcraft and MinecraftEdu

Promotes game-based learning

Medical training 4D Anatomy

Subscription-based app to explore human anatomy

Arch virtual, ORamaVR, Proximie

VR medical training that helps students practice surgical operations

Bravemind VR

App to train therapists to treat patients with PTSD

Holopatient

An MR app to learn human anatomy

HoloAnatomy

App to learn human anatomy

Anatomyou VR

App to learn human anatomy

Touch Surgery

Allows doctors and surgeons to prepare for surgery and learn different procedures

BodyInteract, HumanSim

Training through simulated clinical scenarios

Dentsim, Simodont, VirDenT, HAP-DENT

AR based dental training simulator

Osso VR, Immersive touch, MIST VR, LapSim, ProMIS

Semi-immersive surgical simulators

VIPAR

Allows remote surgeons to give training (continued)

2 Extended Reality Based Remote Learning Experience During Pandemic …

31

Table 2.2 (continued) AR, VR, MR APPs

Description

Language training ImmerseMe

VR to simulate conversations in foreign languages

Mondly

Language learning app

Table 2.3 Benefits of XR based remote learning Benefits

References

Anytime, anywhere learning

De Ponti et al. [34], Kang et al. [37], Chard [58]

Learn by doing approach

De Ponti et al. [34], Yang et al. [21], Lerner et al. [32], Pedram et al. [33], Doolani et al. [30]

Fun learning through Gamified learning

Subhash and Cudney [44]

Safe environment to practice and perfect skills Spiege [8], Role of immersive (XR) technologies in improving healthcare competencies [57], Tzima et al. [9], Zweifach and Triola [41] Mirroring real-life situations

Graham [56], Tzima et al. [9], Zweifach and Triola [41]

Engaging remote learners

Berkman Klein Center [46], da Silva et al. [50], De Ponti et al. [34], Garcia Estrada and Prasolova-Førland [40], Yang et al. [21], Kang et al. [37]

Boosts students’ motivation and engagement

Pomerantz [31], Skult and Smed [36]

Enhances collaborative learning

Berkman Klein Center [46], Pomerantz [31], Skult and Smed [36]

Enhances knowledge retention

Hamilton et al. [52], Johnston et al. [53], Raghavan and Rao [20]

Supports kinesthetic learning

Mbaabu [12], Tzima et al. [9]

Richer digital content

Pomerantz [31], Yang et al. [21], XRC [29]

Reduced cost and risk

Spiegel [8], Role of immersive (XR) technologies in improving healthcare competencies [57], Tzima et al. [9]

Deepens learners understanding of complex problems

Su and Cheng [43], Garcia Estrada and Prasolova-Førland [40], Ratcliffe et al. [38], Pomerantz [31], Laureanti et al. [39], Bucea-Manea-Tonis et al. [42], Zweifach and Triola [41]

Learning outcome measurement

Hamilton et al. [52], Radianti et al. [51], Johnston et al. [53], Yang et al. [21], Zweifach and Triola [41], Sprigg [64]

Reduced learning curve

Spiegel [8], Role of immersive (XR) technologies in improving healthcare competencies [57], Tzima et al. [9]

32

P. S. Mathew and A. S. Pillai

2.8 Barriers to XR Adoption in Remote Learning During Pandemic and Future Directions The COVID-19 pandemic has accelerated the use of technology in the education system to continue the academic activities during the lockdown. Although there are numerous benefits of using XR based remote learning facilities there are some challenges faced by learners that need to be discussed. Remote learning is quite different from on campus learning environments, particularly interaction between teacher and students. Learners may feel isolated and miss the direct interaction with the teacher in a remote learning environment. However, with the XR system some degree of social interaction to support better learning can be achieved with virtual agents or fellow learners [17]. According to Rocsana et al. [42] learners faced two major issues while using XR in remote learning, the cost of the hardware required and technical issues such as unstable internet connection and poor-quality graphic resolution [51, 60] of the XR content which negatively impacted their learning. Often learners find using hardware devices that are tethered, bulky and unable to display 3D content to be the most uncomfortable part of remote learning. Sanika et al. [30], from their study, realized that AR training applications that are wireless, cheap and have 3D enabled mobile devices have higher chances of adoption. Expensive headsets can be a burden to the learners, less expensive headsets [46] like Google Cardboard, Daydream, Samsung Gear as well as options for getting free apps which learners can access through their mobile devices must be employed. To tackle the affordability issue, platforms that allow shared immersion such as edify, that allows the learners to dial into their teacher’s virtual classrooms via video conferencing platforms can be considered as it eliminates the need for the student to use a headset [58]. Another issue found among remote learners using XR technologies was discomfort in wearing the device, eye strain, dizziness. With advancement in technology efforts are being put into designing headsets that are comfortable for the users. Finally, though learners can enhance their skill-based learning using XR, it requires some amount of skill to use the learning tool itself, which can be a deterrent for the learner if not technology savvy [31, 52, 60]. Lamb et al. [60] in their study found that participants who were not technology savvy found the online virtual lab difficult and felt they needed more help and practice in using the technology. Similar observation was mentioned by McFaul and FitzGerald [61] that limited time for familiarization resulted in less engaged participation from learners. To achieve the learning outcomes, learners need to be given clear instructions and proper training or a bridge course on how to use the platform and the technological requirements of the course for remote learning [62]. This will give students time to prepare themselves with necessary technological capacity in advance (Fig. 2.4). While there were some of the issues faced by the learners there were concerns at instructors end also that needs to be addressed for XR based remote learning to happen. Resistance from faculty to adapt new technology and change the mode of instructional delivery is a major setback in the successful adoption of XR in remote learning. It has been observed that where teachers were not familiar with the

2 Extended Reality Based Remote Learning Experience During Pandemic …

Barriers to XR adoption in remote learning Lack of required skills to use XR technologies.

33

1.Future directions Staff training

Management support Access to hardware

Incompatible with the existing pedagogical approaches

Research for improving Technology

Better internet connectivity Lack of quality Content available

Uncomfortable hardware

Clear directives and training for learners

Technology literacy to be integrated in the curriculum Affordability of Technology

Inherent challenges of using 3D virtual environments

Ready to use applications to be considered

Discomfort

Technical problems

Lack of training given to facilitators

Reluctance to adopt

Fig. 2.4 XR in education-Barriers and future directions

technology or when they were not prepared enough to use the AR/VR technologies for their course there was a decline in learners’ performance [52]. As a solution to the problem Nesenbergs et al. [63], suggested to plan workshops for teachers that can help them to prepare/adapt courses using AR/VR and to create a framework for them that would help them to prepare and plan their course/instruction delivery using AR/VR; Ochavillo [48], in the study suggested capacity building sessions are important for faculties to ensure smooth transition to new mode of instruction delivery. Zweifach and Triola in their study [41], have emphasized on the need to involve and motivate faculty early on for integrating XR in medical education. Time is a critical factor while incorporating XR in education be it in assessment plan or content designing. Designing an assessment plan for an experimental subject using

34

P. S. Mathew and A. S. Pillai

XR can be tricky as rubrics for measuring students’ learning may not be available and much thought process is required to identify it for assessment purposes [31, 50]. Ziker et al. [49] have stated that development of 3D content for creating an immersive experience for learners can be a tedious and time-consuming [9, 62] task especially for a faculty without much technical knowledge for 3D content creation. Tzima et al. [9], in their study to understand teachers’ point of view in including AR in education observed that majority of the participant teachers felt that training for 3D content creation would be useful. Working in collaboration with or getting help from somebody who has already been working/worked on XR projects could prove to be beneficial. Authors also stated that ready-made applications as learning tools could prove to be a helpful solution in cases where it is difficult for teachers/students to implement.

2.9 Conclusion Remote learning using XR is making learning more engaging, accessible, iterative and a safe option during the pandemic when the likelihood of face-to-face classroom learning looks uncertain. XR as a tool for remote learning has several advantages to offer the remote learners, however XR certainly does not offer a hurdle free solution. There needs to be some technical breakthroughs which can offer compelling XR experiences through lightweight, convenient to use, less expensive and powerful devices. The standalone Mobile XR and shared immersive experience has tremendous potential to contribute to widespread adoption of XR in education. To exploit the full potential of these XR tools and platforms digital literacy in faculty, learners and trainees must be improved through training sessions. From the study one can get a fair understanding that learners prefer to use XR technologies in education because of the highly engaging and immersive learning experience it provides. Universities need to adopt, invest, encourage, and train faculty and students to use these technologies and develop new methodologies of training and teaching in the science, technology, engineering, arts, and mathematics (STEAM) fields. Most of the XR applications do not measure learning outcome, instead focus on the learner’s experience in using the tool. Instead, educational XR applications must be evaluated not just to assess the students’ learning experience but also to understand if the learning outcomes are met and that there is measurable improvement in the knowledge and skill of the learner. The XR framework discussed in the chapter provides guidelines to facilitate XR in teaching and learning process and for implementing a more effective learning environment.

2 Extended Reality Based Remote Learning Experience During Pandemic …

35

References 1. Qualcomm.: The mobile future of augmented reality (2018). https://www.qualcomm.com/ media/documents/files/the-mobile-future-of-augmented-reality.pdf 2. Rogers, S.: The immersive technology companies offering support during the coronavirus crisis (2020). https://www.forbes.com/sites/solrogers/2020/04/03/the-immersive-technologycompanies-offering-support-during-the-coronavirus-crisis/?sh=5fa8f09a4b2f 3. Papagiannis, H.: How AR is redefining retail in the pandemic. Harv. Bus. Rev. (2020). https:// hbr.org/2020/10/how-ar-is-redefining-retail-in-the-pandemic 4. Nagaraj, M.: Adopting cloud tech, adopting innovation (2020). (fortuneindia.com) 5. Mathew, P.S., Pillai, A.S.: Role of immersive (XR) technologies in improving healthcare competencies: a review. Virtual Augment. Real. Educ. Art Museums (2020). https://doi.org/ 10.4018/978-1-7998-1796-3.ch002 6. Singh, R.P., Javaid, M., Kataria, R., Tyagi, M., Haleem, A., Suman, R.: Significant applications of virtual reality for COVID-19 pandemic. Diabet. Metabol. Syndr. 14(4), 661–664 (2020). https://doi.org/10.1016/j.dsx.2020.05.011 7. Research and Markets.: (2021). https://www.researchandmarkets.com/reports/5239648/ext ended-reality-display-market-growth-trends#rela4-4833338 8. Spiegel, R.: Digital technology—Including augmented reality—Is helping manufacturers stay fully operational during COVID-19 (2020). https://www.designnews.com/automation/digitaltechnology-bolsters-manufacturing-during-pandemic 9. Tzima, S., Styliaras, G., Bassounas, A.: Augmented reality applications in education: teachers point of view. Educ. Sci. 9(2), 99 (2019). https://doi.org/10.3390/educsci9020099 10. Joshi, N.: Reimagine real estate with immersive technologies (2019). https://www.allerin.com/ blog/reimagine-real-estate-with-immersive-technologies 11. ZTE.: 5G cloud XR application white paper (2020). https://www.mobile360series.com/wpcontent/uploads/2019/09/zte-white-paper.pdf 12. Mbaabu, O.: Introduction to extended reality. Section (2020). https://www.section.io/engine ering-education/introduction-to-extended-reality/ 13. Tulgar, A.T.: In between reality and virtuality: augmented reality in teaching english to young learners. Selçuk Üniv. Sos. Bilim. Enst. Derg. 41, 356–364 (2019) 14. Research and Markets.: (2020). https://www.researchandmarkets.com/reports/5006477/glo bal-extended-reality-xr-market-by#rela1-5240209 15. Pellas, N., Mystakidis, S., Kazanidis, I.: Immersive virtual reality in K-12 and higher education: a systematic review of last decade scientific literature. Special Issue: “Virtual Reality and Cognitive Sciences”. Virtual Reality (2020). https://doi.org/10.1007/s10055-020-00489-9 16. Zhang, H.: Head-mounted display-based intuitive virtual reality training system for the mining industry. Int. J. Min. Sci. Technol. 27(4), 717–722 (2017). ISSN 2095-2686. https://doi.org/ 10.1016/j.ijmst.2017.05.005 17. Li, Y., Huang, J., Tian, F., Wang, H.A., Dai, G.-Z.: Gesture interaction in virtual reality. Virtual Real. Intell. Hardw. 1(1), 84–112 (2019). ISSN 2096-5796. https://doi.org/10.3724/SP.J.20965796.2018.0006 18. Kore.: Understanding the different types of AR devices (2018). https://uxdesign.cc/augmen ted-reality-device-types-a7668b15bf7a 19. Mobidev.: (nda). https://mobidev.biz/blog/augmented-reality-development-guide 20. Raghavan, R., Rao, P.: Accenture extended reality (XR): immersive learning for the future workforce (2018). https://www.accenture.com/_acnmedia/pdf-86/accenture-extended-realityimmersive-training.pdf 21. Yang, K., Zhou, X., Radu, I.: XR-Ed framework: designing instruction-driven and learner centered extended reality systems for education 1, 21 (2018). https://doi.org/10.1145/1122445 22. Halabi, O.: Immersive virtual reality to enforce teaching in engineering education. Multimed. Tools Appl. 79, 2987–3004 (2020). https://doi.org/10.1007/s11042-019-08214-8

36

P. S. Mathew and A. S. Pillai

23. Andrews, C., Southworth, M.K., Silva, J., Silva, J.R.: Extended reality in medical practice. Curr. Treat. Options Cardiovasc. Med. 21(4), 18 (2019). https://doi.org/10.1007/s11936-0190722-7 24. Liou, W., Chang, C.: Virtual reality classroom applied to science education. In: 2018 23rd International Scientifc-Professional Conference on Information Technology (IT), pp. 1–4 (2018) 25. Maresky, H.S., Oikonomou, A., Ali, I., Ditkofsky, N., Pakkal, M., Ballyk, B.: Virtual reality and cardiac anatomy: exploring immersive three-dimensional cardiac imaging, a pilot study in undergraduate medical anatomy education. Clin. Anat. 32, 238–243 (2019) 26. Matthews, B., See, Z. S., Day, J.: Crisis and extended realities: remote presence in the time of COVID-19. Media Int. Aust. (2020). 1329878X20967165. https://doi.org/10.1177/132987 8X20967165 27. MeetinVR.: Business meetings & collaboration in virtual reality (nda). https://meetinvr. com/. Accessed 15 June 2020 28. Matney, L.: Microsoft debuts its AR/VR meetings platform Mesh (2021). https://techcrunch. com/2021/03/02/microsoft-debuts-its-ar-vr-meetings-platform-mesh 29. XRC.: Directory of collaborative XR platforms (nda). https://xrcollaboration.com/directory/ 30. Doolani, S., Wessels, C., Kanal, V., Sevastopoulos, C., Jaiswal, A., Nambiappan, H., Makedon, F.: A review of extended reality (XR) technologies for manufacturing training. Technologies 8(4), 77 (2020). https://doi.org/10.3390/technologies8040077 31. Pomerantz, J.: XR for teaching and learning: year 2 of the EDUCAUSE/HP campus of the future project (2020). https://library.educause.edu/-/media/files/library/2019/10/2019hpxr. pdf?la=en&hash=306474918AA2F101DDDCABD59E4366AD7244D572 32. Lerner, D., Mohr, S., Schild, J., Göring, M., Luiz, T.: An immersive multi-user virtual reality for emergency simulation training: usability study. JMIR Ser. Games 8, e18822 (2020) 33. Pedram, S., Palmisano, S., Skarbez, R., Perez, P., Farrelly, M.: Investigating the process of mine rescuers’ safety training with immersive virtual reality: a structural equation modelling approach. Comput. Educ. 153, 103891 (2020) 34. De Ponti, R., Marazzato, J., Maresca, A.M., Rovera, F., Carcano, G., Ferrario, M.M.: Pregraduation medical training including virtual reality during COVID-19 pandemic: a report on students’ perception. BMC Med. Educ. 20, 1–7 (2020) 35. Gopalan, V., Zulkifli, A.N., Abubakar, J.A.A.: A study of students’ motivation using the AR science textbook. AIP Conf. Proc. 1761(1), 27–35 (2016) 36. Skult, N., Smed, J.: Interactive storytelling in extended reality: concepts for the design. In: Game User Experience and Player-Centered Design, pp. 449–467 (2020). https://doi.org/10. 1007/978-3-030-37643-7_21 37. Kang, S., Shokeen, E., Byrne, V.L., Norooz, L., Bonsignore, E., Williams-Pierce, C., Froehlich, J.E.: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp 1–15 (2020). https://doi.org/10.1145/3313831.3376252 38. Ratcliffe J, Soave F, Bryan-Kinns N, Tokarchuk L, Farkhatdinov I.: Extended reality (XR) remote research: a survey of drawbacks and opportunities. ACM (2021). ISBN 978-1-45038096-6/21/05. https://doi.org/10.1145/3411764.3445170 39. Laureanti, J., Brandi, J., Offor, E., Engel, D., Rallo, R., Ginovska, B., Martinez, X., Baaden, M., Baker, N.A.: Visualizing biomolecular electrostatics in virtual reality with UnityMol-APBS. Protein Sci.: Publ. Protein Soc. 29(1), 237–246 (2020). https://doi.org/10.1002/pro.3773 40. Garcia Estrada, J., Prasolova-Førland, E.: Running an XR lab in the context of COVID-19 pandemic: lessons learned from a Norwegian university. Educ. Inf. Technol. (2021). https:// doi.org/10.1007/s10639-021-10446-x 41. Zweifach, S.M., Triola, M.M.: Extended reality in medical education: driving adoption through provider-centered design. Digit Biomark 3, 14–21 (2019). https://doi.org/10.1159/000498923 42. Bucea-Manea-Tonis, R., Bucea-Manea-Tonis, R., Simion, V.E., Ilic, D., Braicu, C., Manea, N.: Sustainability in higher education: the relationship between work-life balance and XR E-learning. Sustainability 12, 5872 (2020). https://doi.org/10.3390/su12145872

2 Extended Reality Based Remote Learning Experience During Pandemic …

37

43. Su, C.-H., Cheng, T.-W.: A sustainability innovation experiential learning model for virtual reality chemistry laboratory: an empirical study with PLS-SEM and IPMA. Sustainability 11(4), 1027 (2019). https://doi.org/10.3390/su11041027 44. Subhash, S., Cudney, E.A.:Gamified learning in higher education: a systematic review of the literature. Comput. Hum. Behav. 87, 192–206 (Elsevier) (2018). ISSN 0747-5632. https://doi. org/10.1016/j.chb.2018.05.028 45. Martín-Gutiérrez, J., Fabiani, P., Benesova, W., Meneses, M.D., Mora, C.E.: Augmented reality to promote collaborative and autonomous learning in higher education. Comput. Hum. Behav. 51, 752–761 (2015). ISSN 0747-5632. https://doi.org/10.1016/j.chb.2014.11.093 46. Berkman Klein Center.: The potential of augmented, virtual, and mixed reality experiences for remote teaching and learning (2020). https://medium.com/berkman-klein-center/extended-rea lity-e5038b38d628 47. Christine Lion-Bailey, Jesse Lubinsky, Micah Shippee. (ND).: The XR ABC framework-a learning framework for augmented and virtual reality. A learning framework for augmented and virtual reality | (teachthought.com) 48. Ochavillo, G.S.: A paradigm shift of learning in maritime education amidst COVID-19 pandemic. Int. J. High. Educ. 9(6). https://doi.org/10.5430/ijhe.v9n6p164 49. Ziker, C., Truman, B., Dodds, H.: Cross reality (XR): challenges and opportunities across the spectrum. In: Innovative Learning Environments in STEM Higher Education: opportunities, Challenges, and Looking Forward, pp. 55–77 (2021). https://doi.org/10.1007/978-3-030-589 48-6_4 50. da Silva, M., Teixeira, J., Cavalcante, P., et al.: Perspectives on how to evaluate augmented reality technology tools for education: a systematic review. J Braz Comput Soc 25, 3 (2019). https://doi.org/10.1186/s13173-019-0084-8 51. Radianti, J., Majchrzak, T.A., Fromm, J., Wohlgenannt, I.: A systematic review of immersive virtual reality applications for higher education: design elements, lessons learned, and research agenda. Comput. Educ. 147, 103778 (2020). ISSN 0360-1315. https://doi.org/10.1016/j.com pedu.2019.103778 52. Hamilton, D., McKechnie, J., Edgerton, E., Wilson, C.: Assessment instrument-immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design. J. Comput. Educ. 8(1), 1–32 (2021). https://doi. org/10.1007/s40692-020-00169-2 53. Johnston, A.P.R., Rae, J., Ariotti, N., Bailey, B., Lilja, A., Webb, R.I., et al.: Journey to the centre of the cell: virtual reality immersion into scientifc data. Traffic 19, 105–110 (2018) 54. Geogebra.: (nda). https://www.geogebra.org/?lang=en 55. Snelling, J.: 25 resources for bringing AR and VR to the classroom (2019). https://www.iste. org/explore/In-the-classroom/25-resources-for-bringing-AR-and-VR-to-the-classroom 56. Peter Graham.: 40 + Resources for bringing AR/VR to the classroom (2020). https://www.vrf ocus.com/2020/06/40-resources-for-bringing-ar-vr-to-the-classroom/ 57. Role of Immersive (XR) technologies in improving healthcare competencies. In: Virtual and Augmented Reality in Education, Art, and Museums, IGI global. Copyright © 2020, IGI Global. https://doi.org/10.4018/978-1-7998-1796-3. https://www.igi-global.com/book/virtual-augmen ted-reality-education-art/234360 58. Chard, S.: Four ways extended reality (XR) will influence higher education in 2020 and beyond (2020). https://theedtechpodcast.com/four-ways-extended-reality-xr-will-influe nce-higher-education-in-2020-and-beyond/ 59. Touchsurgery.: (nda). https://www.touchsurgery.com/ 60. Lamb, R., Lin, J., Firestone, J.B.: Virtual reality laboratories: a way forward for schools? EURASIA J. Math. Sci. Technol. Educ. 16(6), em1856 (2020). ISSN:1305-8223 (online). https://doi.org/10.29333/ejmste/8206 61. McFaul, H., FitzGerald, E.: A realist evaluation of student use of a virtual reality smartphone application in undergraduate legal education. Br. J. Edu. Technol. (2019). https://doi.org/10. 1111/bjet.12850

38

P. S. Mathew and A. S. Pillai

62. Domingo, J.R., Bradley, E.G.: Education student perceptions of virtual reality as a learning tool. Sage J. J. Educ. Technol. Syst. (2017). https://doi.org/10.1177/0047239517736873 63. Nesenbergs, K., Abolins, V., Ormanis, J., Mednis, A.: Use of Augmented and virtual reality in remote higher education: a systematic umbrella review. Educ. Sci. 2, 11, 8 (2021). https://doi. org/10.3390/educsci11010008 64. Sprigg, S.: EON reality re-brands its AVR platform to “EON-XR” and adds new features (2020). https://www.auganix.org/eon-reality-re-brands-its-avr-platform-to-eon-xrand-adds-new-features/

Chapter 3

How Virtual and Augmented Reality Are Reshaping the Fashion Industry During the Covid-19 Pandemic Barbara Silvestri

Abstract The use of interactive media has been one of the most relevant trends in the last years. In this era where the stimuli from media never leave us alone, fashion customers are looking for even more than interactivity. They want to be involved, surprised. The keyword for this new trend is experience, not only as a memorable experience, but as a proper aesthetic experience that is capable to enhance even our sensory perceptions. Virtual reality and augmented reality seem to be the new solution to create experiential marketing, to differentiate one’s brand from competitors and to create immersive storytelling. Covid-19 brought the importance of the digital in fashion even more to evidence. With most of the stores closed, fashion brands had to translate its sector on social media and e-commerce platforms, and that’s when XR stepped in as a notable resource. Social distancing and lockdown highlighted how the digitization of fashion brands must be a priority, not only to recover from the Covid-19 crisis and gain back strength, but also to anticipate the needs of the modern customer. Augmented reality and virtual reality are definitely part of the answer. Keywords Fashion · Augmented reality · Virtual reality · Covid-19 · Experiential marketing · Digitization

3.1 Introduction The first important concept that needs to be understood when talking about virtual and augmented reality is that they come from the same idea and technology, but their use is different. Virtual reality means that we find ourselves in a sensory experience which recreates an artificial reality different from our own. Virtual reality completely surrounds us, leaving almost no hints of our actual reality. Augmented reality is, as the term describes, something that is added to our reality. What we see is a mixed world, where immaterial and material realities collide, forming an augmented world. The first commercial application for augmented and virtual reality arrived B. Silvestri (B) Bologna, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_3

39

40

B. Silvestri

in 2008 when BMW created a printed magazine that, if held in front a computer’s camera, appeared on the screen and could be moved in real time by manipulating the magazine itself: this was the first case of a real time manipulation of a digital model on screen, uniting de facto digital and physical realities [11]. In 2014 and in 2016 both Google and Microsoft released headsets for AR and VR. Google created Google Glass in 2014, while Microsoft created HoloLens in 2016. Google Glass came with a cheaper price, but HoloLens can be defined almost as a wearable computer. Indeed, it allows users to scan their surroundings to create a customized AR experience. Today, there are a lot of different headsets, with different ranges of prices, made by HTC, Nintendo, Oculus and so forth. Furthermore, with the possibility to use Smart Mirrors and Apps, the applications of augmented and virtual reality today are more numerous than ever. One of the fields which found itself to be a perfect fit for these new technologies is indeed fashion. The fashion field is fleeting, changing and striving for innovation and modernity, always looking for novelty just like the tech field. Furthermore, fashion brands today build their identities on storytelling, on lifestyles, on impalpable values that are continuously communicated by events, websites, social media and fascinating advertising ads and experiences. And during the Covid-19 pandemic, all these initiatives had to be moved online, leading to the use of virtual and augmented reality technologies. Why do shoppers choose to come to your store on any given day? How do they feel when they leave? Did you offer unique, empowering experiences and services that leave them excited to come back—again and again? Do individuals feel like your brand is relevant to their lifestyle? Or, as they wander from store to store, do your potential customers forget your brand as it blurs in their minds with those of competitors?. [9]

This quote from an IBM report underlines two main concepts to better understand why XR technologies are a perfect fit for fashion. First of all, today shoppers have a large pool of fashion brands to choose from. The key to attract them, and most importantly, to turn them into loyal customers, is to offer unique experiences, that excite customers and entice them to come back. During the pandemic, with physical shops closed or with limited entrances, the flux of sales moved online. This means that these experiences had to be recreated on the web, where typical store experiences are indeed impossible. Secondly, fashion brands have to be unique. Today, innovation in fashion is faster than ever. Those who arrive later are inevitably forgotten. To be remembered, it is important to be avantgarde from the beginning and adopt innovative solutions as soon as they are available. Especially in fashion, most of the time this means to be bold. VR and AR solve both these two issues. Thanks to them, it is possible to move online not only amazing shopping experiences, but also events such as fashion weeks. Furthermore, since AR and VR technologies are still not widespread, their use results in innovation, freshness, modernity. What we experience today, in fashion but also in other field, is indeed the use of experiential marketing to attract customers and to involve them thanks to interactive experiences.

3 How Virtual and Augmented Reality Are Reshaping …

41

3.2 Main Theories But what does experiential marketing exactly means? Essentially, is it the use of various instruments, such as technologies, visual merchandising, advertising and so forth, to achieve an experience. Modern customers, according to Bernd H. Schmitt, crave this type of interaction from brands: They want products, communications, and marketing campaigns to deliver an experience. The degree to which a company is able to deliver a desirable customer experience—and to use information technology, brands, and integrated communications and entertainment to do so—will largely determine its success in the global marketplace of the new millennium [26].

In particular, Schmitt describes an interesting instrument of experiential marketing called marketing aesthetics: it is the use of aesthetic stimuli to deliver sensory experiences [26]. According to Baumgarten, an aesthetic experience can be defined as an embodied and pleasing sensory experience. Indeed, Baumgarten’s aesthetic theory can be summarized as an approach that sees aesthetic as a way of discovering and perceiving the world through the five senses [17]. Technologies such as augmented reality and virtual reality can recreate experiences in which our senses are stimulated and engaged. According to Marfia and Matteucci [17], an aesthetic experience through VR and AR is reached thanks to two key factors: Immersiveness and Presence. According to the aforementioned study by Marfia and Matteucci Immersiveness requires: • A continuous environment where a person may freely move and look around. • Consistent elements that a user may understand in terms of size, color and interaction patterns. • An interactive scene where objects respond to the delivered stimuli, providing physical feedback (e.g., haptic interfaces) when, for example, touched. • A coherent plot, which may support the development of user engagement. Immersiveness, to better understand, can be seen is as a replication of our reality in a virtual environment. In other words: A stirring narrative in any medium can be experienced as a virtual reality because our brains are programmed to tune into stories with an intensity that can obliterate the world around us […]. The experience of being transported to an elaborately simulated place is pleasurable in itself, regardless of the fantasy content. We refer to this experience as immersion. Immersion is a metaphorical term derived from the physical experience of being submerged in water. We seek the same feeling from a psychologically immersive experience that we do from a plunge in the ocean or swimming pool: the sensation of being surrounded by a completely other reality, as different as water is from air, that takes over all of our attention, our whole perceptual apparatus […] in a participatory medium […] immersion implies learning to swim, to do the things that the new environment makes possible […] the enjoyment of immersion as a participatory activity [17].

The International Society for Presence defines Presence as a psychological and sensory stage in which human beings live a technological generated experience, but part of the human perception fails to recognize the role of technology related to

42

B. Silvestri

the experience itself (Lombard, M. in [17]). Indeed, when we are involved in an experience in a virtual reality environment we know that that environment is not reality. Nonetheless, our senses can be so engaged that, after a span of time, part of us forgets that the experience is only fictional, and this leads to a high level of engagement in the sensory activity. Furthermore, our senses can’t completely distinguish a real stimulus from a virtual or augmented one. According to Jerald, there are four main factors necessary for the achieving of Presence: • The illusion of being in a stable spatial place, in which people can interact with the surrounding objects. • The illusion of self-embodiment to give users the feel of having a body, not necessarily theirs, within the virtual world. This could be achieved, for examples, through the use of avatars. • The illusion of physical interaction, with background sound, a sense of threedimensionality of space and things, vibration of controllers and specific haptic devices to increase touchability. • The illusion of social communication thanks to the communication through avatars, since people are naturally prone to communication (Jerald in [17]). Leanza and Balconi conducted a research which was able to prove how augmented and virtual reality experiences don’t engage only in a psychological way with us, but also bodily: after watching some commercials with Oculus, spectators shown an increase in the prefrontal cortex theta activity and greater SCL (Skin Conductance Level) in response to virtual reality commercials [14]. This further demonstrates how strong the sensory experience achieved with virtual and augmented reality is. If we wanted to go even further, we could theorize not only that virtual and augmented reality are a mean to an aesthetic experience, but also that they can be seen as a prosthetic to our perception. Marshall McLuhan in 1964 theorized how media are proper extensions of man in his work “Understanding media” [19]. According to McLuhan’s book, every new media tries to incorporate in itself all previous existing media. Internet is an example. But with VR and AR we go even further. We are able to perceive video, music, text, interactions, and so a whole new reality. Furthermore, if according to McLuhan the wheel as a medium extends our feet, we could say that AR and VR extend all of our senses, and so our mind as a whole. If we think about AR and VR, the first concept that comes to mind is the extension or creation of a reality. But to extend our reality means also to extend our perception of it at the same time. According to Longo: The effects produced on the body by information technology are particularly interesting. Telematics and virtual reality produce a communicative, perceptive, and functional diffusion of the body. Virtual reality, for instance, extends the body spatially in unprecedented ways and allows it to occupy the whole planet. Distance is annihilated and sensitivity is dis-located, but, paradoxically, by denying the body’s primary attribute—its proximity or presence—through artifice and simulation. Virtual reality enhances the body by denying it. [16]

In his “The Work of Art in the Age of Mechanical Reproduction” Walter Benjamin described cinema as the new medium which was able to describe better the modern

3 How Virtual and Augmented Reality Are Reshaping …

43

age [3]. According to him, the evolution of society was followed by the evolution of new media, and to the adaptation of human senses to those. When he published his essay, it was 1935. He assisted to the development of the new type of town called metropolis, and to the change in pace of the Modern life. Everything had to be faster and more immediate. According to Benjamin, cinema indeed represented this new way of living, and slowly people and their senses adapted to it. Our continuous need for faster interaction with sounds and images was theorized also by Paul Valéry in 1928, as he wrote: Just as water, gas, and electricity are brought into our houses from far off to satisfy our needs in response to a minimal effort, so we shall be supplied with visual- or auditory images, which will appear and disappear at a simple movement of the hand, hardly more than a sign. Just as we are accustomed, if not enslaved, to the various forms of energy that pour into our homes, we shall find it perfectly natural to receive the ultrarapid variations or oscillations that our sense organs gather in and integrate to form all we know (Valéry in [3]).

If we tried to apply the same theories to our times, we could say that the rise of new media such as VR and AR is the reflection of the society of today. Today’s society is hyper-connected, fluid, digital and more and more online. XR could be an expression of the urgent need of our society to fully submerge in the digital world, and the need to see our reality and our senses expanded in this new world. Could our senses and our perceptions fully adapt to VR and AR in the future? The experiences of the past suggest that they will. And since Valéry was able to foresee our need for visual and auditory continuous stimuli, united with the interactivity with those media described as “Appear and disappear with the movement of our hand”, what could represent more those needs today than VR and AR?

3.3 Augmented and Virtual Reality in Fashion During the Covid-19 Pandemic 3.3.1 Virtual Reality and Augmented Reality for Fashion Events Since Covid-19 broke out in march 2020, it has been almost impossible to organize events of any kind. The first thing that had to change in the fashion industry was, of course, fashion weeks. Indeed, it is not difficult to understand why the usual catwalks and events couldn’t take place the same way they did before the pandemic. Every season, fashion weeks events attract an incredible amount of fashion professionals, celebrities and influencers, designers and of course fashion lovers. The first fashion week that was challenged by the virus was Paris Fashion Week. Most buyers, editors, and designers decided to keep up with the usual schedule despite the fear for covid. But according to Binkley, the panic created by the pandemic was of course able to overshadow the fashion week itself. She wrote:

44

B. Silvestri The fixation on coronavirus, both the health concerns and the sweeping impact it’s had on the luxury market, detracted attention from some excellent collections and several topics that have otherwise been top of mind in fashion lately. Talk of sustainability, a rampant discussion throughout New York Fashion Week, died to nearly a whisper. [5]

The fashion world was able to emerge in all its resilience. But still, the efforts to maintain a physical fashion week in Paris debilitated the meaning of the week itself. Shortly after, Tokyo Fashion Week was canceled. Travel, both national and international, and gatherings were discouraged due to the spread of the virus worldwide. It seemed that the fashion world had to reinvent the most important event of the year. And decided to do it through technology. From 12 to 14 June 2020 took place the first all digital European fashion week: London Fashion Week. LFW decided to drop the men’s catwalks and decided to create a mixed gender show, and to showcase on a digital platform new designs, virtual showrooms, podcasts and short films [15]. But are videos and podcasts enough? The fashion show form has lasted for decades precisely because it works so well — in real life. There have been occasional attempts at change, mostly via “movies” that come off like music videos, but they have never been that successful. Generally that’s because they prioritize mood and concept over being able to see the telling detail, or material essence, of a garment. And it’s in that detail and essence that individual desire lies. There’s an opportunity now to provide a different solution. What it demands though is not just recreating events, but rethinking them entirely. [8]

According to Friedman, what is lacking to digital fashion weeks is the sensory approach. This means that what makes fashion weeks events so unique is indeed the chance to see details in fabrics and decoration, the possibility to touch and interact with garments. So, a new approach is needed and it has to be not only digital, but also sensory. The answer could be VR and AR. As we already said, XR can recreate a deep involvement and engagement, while at the same time stimulate the senses. The chance to be submerged in a digital environment, and to interact with elements could be the answer to the problem that Friedman posed. An example is Shanghai Fashion Week. While many cities and designers decided to cancel or do online streaming fashion weeks, Shanghai Fashion Week designers decided to do something more. Xu Zhi, Roderic Wong and Andrea Jiapei Li were able to represent a proper virtual runway, using augmented reality and CG techology. Supported by XCOMMONS, a Shanghai-based platform for art and fashion, they were able to recreate an extremely accurate and immersive experience [27]. But the most recent and interest example of the use of virtual and augmented reality for fashion catwalks was Balenciaga’s Autumn/Winter 2021 show. The show was a reflection of the existential crisis that involves everyone during lockdown, and the designer Demna Gvasalia decided to represent it through a virtual show and a videogame called Afterworld: the age of tomorrow [1]. Gvasalia sent 30 members of the press an Oculus headset showcasing the autumn winter collection, which was filled with themes such as dystopia, existentialism and escapism [1]. According to Anders, fashion critic at british Vogue, the experience was amazing and very captivating: Two days before the show, a big black Balenciaga box arrived on my doorstep. Inside it was a VR headset with two hand controllers. For those of us who haven’t played a video game since

3 How Virtual and Augmented Reality Are Reshaping …

45

Nintendo NES circa 1989, watching a fashion show in virtual reality is completely mindblowing. Set in a blacked-out venue with a raised runway and a soundtrack reminiscent of a video game score, it was literally like being there IRL. So vivid were the details that you could turn to the people around you and study their body language and wardrobe. […] At one point, I caught myself thinking I should be wearing shoes, then remembered I was alone, on my sofa, in London. [1]

Another interesting use of extended reality technologies in fashion events is the possibility to create an entirely digital and virtual fashion exhibition. The Fashion Innovation Agency partnered with the Museum of Other Realities and with the immersive storytelling production house RYOT to create an exhibition which was truly revolutionary. The Museum of Other Realities or MOR is a virtual reality space in which users and artists can create, interact, connect and share experiences in a completely virtual environment [20]. The Fashion Innovation Agency or FIA firmly believes in innovation and technology, and decided to show to the fashion world how limitless are the potentialities of virtual reality, providing an example that is able to recreate the excitement of a traditional fashion exhibition [7]. The three designers Damara, Sabinna and Charli Cohen, partnered with the digital, animation and VR artists Sutu, VRHUMAN, Ana Duncan and JOY to create an interactive and outstanding exhibition [7]. The head of Fashion Innovation Agency Matthew Drinkwater declared: It is vital that the fashion industry recognises the potential of immersive technologies to revolutionise every element of how it creates and communicates, including that most iconic moment – the catwalk itself. This project will reimagine the live show experience and offer an inspirational glimpse into the virtual future of fashion. [7]

Audiences could attend the launch event both in virtual reality and online formats. The garments and the worlds they’re immersed in will still exist within the Museum of Other Realities until summer 2021 [7]. Technologies such as virtual and augmented reality are entering the fabric of fashion, making it more and more digital. The digitization of fashion, started with e-commerce platforms and apps, will definitely continue with VR and AR technologies. The pandemic of Covid-19 was a true accelerator, forcing the agents from the fashion field to think outside the box and to channel innovation and new possibilities through the digital world. With augmented and virtual reality not only it is possible to have the perks of the digital, such as social distancing, but also to reproduce the attention to the detail and the sensory experiences typical of live events.

3.3.2 Virtual Reality and Augmented Reality for Online Retail According to data, online retail platform saw an unprecedented rise in traffic during the coronavirus pandemic: retail websites generated almost 22 billion visits in June 2020, up from 16.07 billion global visits in January 2020 [28]. To shop in physical stores today is harder than ever: restrictions, hygiene and social distancing are

46

B. Silvestri

preventing many customers to fully enjoy their shopping experiences. While the growth of e-commerce platform was predicted long ago, the pandemic was able to accelerate it in an incredible way. According to Adobe’s Digital Insights Manager Vivek Pandya, e-commerce growth was accelerated from 4 to 6 years and total online spending in May hit $82.5 billion, up 77% year-over-year: According to our data, it would’ve taken between 4 and 6 years to get to the levels that we saw in May if the growth continued at the same levels it was at for the past few years, […] We typically don’t expect to see surges at this level, at any time outside of the holiday season. For context, last year’s holiday season drove $142.5 billion dollars from November 1st to December 31st, and that was a 13% year-over-year increase. [13]

At the same time according to Nielsen customers still love shopping in stores, mainly for the chance of touching and feeling products and for the security of seeing what you are buying [21]. But how is it possible to combine the necessity of shopping mainly online due to the pandemic and the need to feel, see and touch products? The answer is of course the use of augmented and virtual reality for online platforms. Not only is it possible to see a product in VR or AR from the comfort of your home, but also to create outfits, try on make up and so forth. This is particularly relevant for the fashion industry. Ji Hyuk Park, Nielsen Commercial leader in South Korea, declared: We are seeing more and more examples of how virtual and augmented reality can be used in retail. The beauty sector is very big in Korea, and brands and retailers are leveraging this technology to enhance the user experience, enabling them to test and try out products virtually. In the challenging times of COVID-19, where hygiene is a big concern, consumers will be less prepared to physically try on products, and this is where augmented reality can replicate experiences without the health implications. [21]

But while VR and AR are helping to maintain social distancing, they have been proved useful to solve other relevant issues. For example, the main problem of fashion e-commerce platforms is the choice of the right size and returns due to fitting issues. Seeing a garment in XR can help customers having a more realistic idea on how they will look, how will they fit, how they can be matched with other accessories, but also in some ways to bond with the product. According to a study made by Brengman, Willems and Van Kerrebroeck virtual and augmented reality can induce a higher level of perceived ownership in customers who buy online [6]. Perceived ownership can be described as: The sensation that something is “mine”. […] Perceived ownership translates itself into a reluctance to give up an object, as psychologically owning it is comparable to actual ownership, thus resulting in a higher valuation of the object. [6]

The study demostrated how VR and AR can lead to a higher perceived ownership and so to higher chances of purchases by customers. Furthermore fashion seems to be the perfect fit for this application of VR and AR: This study also demonstrates that allowing users to examine products marketed online in more detail by digitally touching them generates perceived ownership which appears to be generally more pronounced […] for products with material properties (i.e. requiring

3 How Virtual and Augmented Reality Are Reshaping …

47

sensory information). Therefore, it is suggested that retailers should especially implement AR technology for products that require more than only written or visual information, but need to be inspected more thoroughly […]. This could particularly be interesting for retailers selling apparel. [6]

Furthermore, a study conducted by Rauschnabel, Felix and Hinsch demonstrates how AR based marketing initiatives can increase brand awareness and create a higher customer value: while costs to implement AR in e-commerce are still modest, the advantages are many and evident [25]. According to them AR in the next future could, as technological innovation improves everyday more, reach benefits very similar to those of traditional physical trial and manipulation in stores, while at the same time increase inspirational value: trying, mixing and manipulating elements can inspire the customer and lead to higher brand engagement [25]. Another study made by Mark Yi-Cheon Yim and Sun-Young Park led to an extremely interesting discovery: AR based applications in fashion lead to a more favorable evaluation about AR presentation of fashion products, especially by customers who perceive their own body image as unfavorable [29]. According to the study, this is linked to the fact that: […]when AR is perceived as a highly interactive technology, consumers with a favorable body image perceive more opportunities to customize their appearance and decorate themselves using virtually displayed products and view an enhanced self-image, which results in a more favorable evaluation of their body. […]Furthermore, it is noteworthy that those with a poor body image care less about these features and appear to value other psychological benefits of AR, such as individualized, isolated, and customizable media environments […]. [29]

So, also from a psychological point of view, AR based applications could improve not only sales and interactivity, but also the bodily perception of customers in a positive way. The most helpful uses of AR and VR during the pandemic for online initiatives were AR and VR tools and apps and virtual stores and showrooms. The brand Kendra Scott decided, during the Covid-19 pandemic and with physical stores closed, to introduce an AR tool enabling customers to try on from their own homes different styles of earrings [24]. Beauty brand Ulta Beauty decided to fight the decline of sales and the restriction brought by Covid-19 by enabling a virtual make up try-on function, present on the e-commerce website of Ulta Beauty and via app: the functionality is called GLAMlab and allows customers to virtually try-on more than 4000 items [12]. According to Prama Bhatt, Ulta Beauty’s chief digital officer, GLAMLab usage has increased by five times in the pandemic and they have seen 19 million shades tried on in the post Covid-19 environment [12]. What is interesting about virtual try-on apps for the beauty sector of fashion is the chance of keeping this technologies as default possibilities even after lockdown, to cope with the new hygienic rules and testing issues in physical stores. Another example of XR for online usage is the collaboration between Gucci and Snapchat. In the summer of 2020 the social media platform decided to add a special feature which enabled customers to try out four models of Gucci shoes in AR to see how they looked wearing them, without having to leave the comfort of their house [4]. Not only this kind of initiative helps customers buy and try without crowding shops during the pandemic, but promotes important initiatives such as gamification, interactivity and adoption of XR technologies by customers.

48

B. Silvestri

Now let’s talk about virtual stores and showrooms. A study made by Beck and Criè analyzed how VR fitting rooms for e-commerce could enhance customer experience. Indeed, according to the aforementioned study, the use of Virtual Reality fitting rooms on e-commerce websites not only increases the conversion rate of sales and the amount of gathered data on products, but also the curiosity of customers, who are more willing to explore the website [2]. The most important barrier in the creation of virtual reality stores and showrooms today are headsets: not everyone has a pair in their home, and not everyone can afford them. VR and AR companies are now working to reproduce virtual reality stores and showrooms using 3D advanced rendering. An example is Obsess. The company, led by CEO Neha Singh, is dedicated to the creation of virtual stores and showrooms and is now working to create a more approachable virtual reality [18]. In october 2020, the company received a fund of $3.4 million from Venture Reality Fund, WXR Fund and Jump Capital. According to Jump Capital partner Yelena Shkolnik, the main reason is this new approach to virtual reality: We really like that it doesn’t force you to have a headset, and it allows you to create something akin to a physical store. [18]

The company defines itself as an Augmented & Virtual Reality software platform for experiential shopping, and between the fashion brands which cohoperate with Obsess it is possible to find Ralph Lauren, Chirstian Dior, Tommy Hilfiger, Coach, Charlotte Tilbury, Marni, Diesel, Desigual, Farfetch and Levi’s [23]. A brand which used this type of approach was Tommy Hilfiger. Preparing for massive online sales during the first christmas festivities with Covid-19, Tommy Hilfiger experimented with a virtual store where customers could find festive music, shoppable clothes which cast a realistic shadow on the carpet, and a snow room where virtual snow fell and piled up on the floor [18]. This idea of a more usable virtual reality spaces developed by Obsess and fashion brand who are investing in this type of projects is a mix between accessibility and experiential marketing. Companies are trying not only to create a VR and realistic representation of their clothes thanks to 3D technologies, but also to entertain customers, to create an experience which is memorable, interesting. This is also true for showrooms, which are fundamental for designers to showcase their collections to buyers from all over the world. But can virtual shops successfully recreate this type of experience? During the summer of 2020 the italian brand Brunello Cucinelli, known for its incredible showrooming experience turned to digital showrooming: You can smell the Parmigiano-Reggiano from the sidewalk outside the Brunello Cucinelli showroom. Once Inside, there’s usually pasta being doled out from gigantic pots, piles of focaccia and prosciutto, and olive oil from Cucinelli’s own grove. Cucinelli himself is probably wearing a white sport jacket—incongruous with the sauces, oils, and wines on hand—but he’s unfazed by spillage. Tables and clothing racks are set up at a safe distance where models show off cotton twill drawstring trousers (retail: $1,195) and shearling-trimmed cashmere bomber jackets ($7,995). It’s this sumptuous experience that puts Cucinelli’s ne plus ultra luxury into context. Successful lifestyle brands know how to create an experience for their customers. Transposing the physical pleasures of fashion into satisfying online experiences is the problem. How can a digital synecdoche ever simulate the richness of real life? [30]

3 How Virtual and Augmented Reality Are Reshaping …

49

The company chosen by the Cucinelli brand to create a virtual showroom is NuOrder, a company which is also working with brands such as Steve Madden, Helmut Lang and Acne Studios to create what the company defines an elevated approach to virtual showrooms [22]. The company is able to create high quality 360 imagery which enables buyers to zoom in to see the various garments in high quality and detail and to see the products from all angles, while engaging web customers also with dynamic videos, shoppable hotspots and a high level of usability [22]. Another company which dedicates its mission to create with VR a seamless experience between the digital and physical world is Invrsion [10]. The company released an exclusive interview during the summer of 2020 to better explain what is the mission of Invrsion and how the Covid-19 pandemic is accelerating the use of XR technologies. The interview was originally released in italian and this is a translation by the author.

3.4 An Exclusive Interview with the Experts: The Invrsion Company, Milan

1.

How was Invrsion born and why did you decide to invest in virtual reality?

Invrsion was founded in the spring of 2015 in Milan. The four founders had begun studying virtual reality applications to entertainment with Oculus technology in 2013. The following year, when Facebook acquired Palmer Luckey’s company, they understood the potential of virtual reality as a new environment in which to study consumer behavior, replicate industrial processes and rethink sales models. In October 2015 Invrsion, which already produced stereoscopic 360° videos for large international brands with the aim of self-financing research and development, organized a first Investor Day, where it presented an industrial plan to develop three prototypes of an immersive “room scale” virtual reality simulator: a retail space simulator called ShelfZone, a car simulator and configurator, CarZone, and finally ResiZone, a simulator for Real Estate spaces and furnishing elements. Subsequently Invrsion focused on the development of ShelfZone, which was presented at various industry events, recognizing its realism and high innovation dimension. 2.

What services do you offer?

We simulate retail spaces and showrooms in immersive virtual reality and digitize products and contents in 3D. We cater to the CPG (consumer packaged goods) and retail industry, but we have recently extended our solutions to the industries of Design and Fashion as well. 3.

How does your process of researching and creating virtual environments work?

We create virtual environments after a technical inspection or through reference photos. If the environment does not already exist, we use project files from architects

50

B. Silvestri

and store designers, but very often it is enough to use one of the store templates we have in our library (convenience-superstore-hypermarket) that are made in a representative way, and we make the necessary customization, creating different levels of look-and-feel. Qualitative and quantitative shopper research can be conducted, inviting shoppers to test the virtual solutions implemented to get their feedback (instore navigation path, products picked and put back on the shelf, purchasing behavior and so on). 4.

Have you ever collaborated with companies in the fashion world? If so, which ones?

We have recently extended our solutions to the fashion world, and we are collecting the first expressions of interest from important brands. 5.

How do you expect the relationship between virtual reality and the world of fashion to evolve in the future?

During the COVID period in which companies in the fashion sector were forced to cancel or postpone events and fashion shows, we have identified in virtual reality an answer to their needs. In fact, brands can simulate their showrooms in virtual reality and share them with their buyers anywhere in the world without the need to meet in person, by broadcasting an immersive and engaging multi-sensory brand experience, and sharing their collections without inventory and variant limits. They can digitize their products in hyperrealistic 3D and use them not only in VR, but also in AR and on their e-commerce sites, ensuring a more realistic, interactive product experience than “simple photos”. 6.

How long do you think it will be before virtual reality really takes hold in the world of online and offline retail, and what are the obstacles that slow down the adoption of these new technologies?

We cannot tell; certainly it must be admitted that this is an emerging technology, which is often associated with the gaming and entertainment sector as there is not yet sufficient awareness of its potential for B2B and various sectors such as Retail. However, more and more companies are proposing or adopting these technologies, and also at the media level they are talked about more and more, so this will surely give a boost to their diffusion. Another limitation lies in the fact that to enjoy a completely immersive virtual experience it is necessary to wear a headset. However, headsets are becoming more and more comfortable (maybe just a pair of glasses will suffice in the future?) and cheaper, and this will surely facilitate its adoption at both business and consumer level. The virtual reality market is growing a lot; According to Mordor Intelligence research, it was valued at $11.52 billion in 2019 and is expected to reach $87.97 billion by 2025, with a compound annual growth rate of 48.7% over the 2020–2025 forecast period. 7.

How do you manage, through virtual reality, to create engagement?

Our virtual solutions are currently adopted at the B2B level in the retail sector by brands and retailers to carry out their in-store activities (category management, trade

3 How Virtual and Augmented Reality Are Reshaping …

51

marketing, shopper research, store design and remodeling) in a more efficient and effective way. So it responds to a very “practical” need, in which the engagement component is secondary. However, it is a much appreciated “plus”, also given the “wow effect” it generates, for example in negotiations between brands and retailers. From a B2B2C perspective, when in our view of things virtual reality commerce or v-commerce will allow end consumers to shop in immersive virtual reality from the comfort of thir own home, surely the engagement component will play a predominant role. In the Fashion sector, the element of engagement is certainly of great importance, given the importance for brands to bring experiences to life, both in their meetings with buyers and with their consumers. 8.

What are you currently researching?

Research in our area is constant. We work to enhance our ShelfZone software with new features, in the direction of making the user increasingly independent in carrying out their retail projects. We work on our patented system to scale the digitization of 3D products, etc. 9.

Do you think virtual reality can contribute to a more sustainable world in the future, even in the fashion sector?

If we are already thinking about the possibility for brands and buyers to meet “remotely” in a virtual showroom without having to travel around the world, then this is already an incentive for sustainability. On the consumer side, for example, it comes to mind that, if evaluating products in 3D allows you to have a better perception of the products, and therefore increase conversions and reduce returns, a lot of waste related to returns (which currently represents a sore point for companies) can be avoided. 10.

Do your solutions enter mainly within the b2c or b2b processes, and will we see fashion e-commerce entirely based on AR/VR in the near future?

We currently operate at a B2B level. For what concerns a Fashion e-commerce entirely based on VR, well, let’s hope so! This would mean that we are on the right track. 11.

What do you think of today’s quality of virtual reality headsets? Is the quality they offer sufficient to support fashion?

There are several headsets on the market, and in the face of very high performance, the price variable is constantly decreasing. Providers such as Oculus, HTC, HP provide excellent headsets that guarantee high performance (this also depends on the quality of digitization; Invrsion is globally recognized for the extreme realism of its simulations).

52

12.

B. Silvestri

What will be, in your vision, the path of diffusion of immersive technologies in mass consumption, and what is missing in online purchases using immersive technology compared to traditional purchases, and vice versa?

According to our vision of v-commerce, in the not too distant future consumers will be able to shop from their home in virtual reality. This immersive, unprecedented, fun experience will be able to combine the typical advantages of e-commerce (efficiency, speed) with those typical of shopping in a physical store (emotionality, experiential aspects). 13.

Is covid-19 slowing or accelerating the creation of business processes supported by immersive technologies?

COVID has accelerated the creation of business processes supported by immersive technologies, to the point that many companies have resorted to virtual solutions— just think of the fashion sector affected by the cancellation of fashion shows and physical events.

3.5 Conclusions Covid-19 brought a significant and before unimaginable shift in the fashion world. Fashion weeks were canceled or moved online, stores were closed and e-commerce surged. The pandemic served as a push to make the fashion field evolve and digitize. While before Covid-19 being digital was still an option to some fashion brands and company, after it has become a must. The fashion world needs to be prepared to face challenges as the one of the pandemic, and not take for granted that the old standard way of doing fashion will always be enough. Some brands were developing platforms and initiatives which involved the use of XR technologies before the pandemic, and now have today a great advantage. Others were forced to innovate. The adoption of XR technologies could be soon enough not just an edgy initiative to gain media and consumer attention, but a proper asset to ensure the competitiveness of a brand in a technologically-evolved market. During the pandemic virtual and augmented reality were able to transmit to customers the personal touch of each brand. Fashion weeks were held following safety rules and transmitted through XR performances, shops were dematerialized and rebuilt digitally and visited virtually, products were tried on from the quiet and isolation of consumers’ houses. The perks that accompanied the usage of XR technologies during the first part of the pandemic were clear to fashion companies. The fashion world is ready to welcome new realities, virtual and augmented. But also to mix them with physical reality to create phygital experiences that contain all the best parts of each one and that can face the challenges, present and future, that the pandemic brings.

3 How Virtual and Augmented Reality Are Reshaping …

53

References 1. Anders, C.M.: 5 things to know about Balenciaga’s virtual realityAW21 show, Vogue UK (2020). https://www.vogue.co.uk/news/gallery/balenciaga-autumn-winter-2020 2. Beck, M., Crié, D.: I virtually try it … I want it ! virtual fitting room: a tool to increase on-line and offline exploratory behavior, patronage and purchase intentions. J. Retail. Consum. Serv. (40) (2018). https://doi.org/10.1016/j.jretconser.2016.08.006 3. Benjamin, W.: The work of art in the age of mechanical reproduction (1935) 4. B&T MAGAZINE.: Snapchat has launched its first-ever augmented reality (AR) “try-on” campaign in Australia with luxury Italian brand Gucci (2020). https://www.bandt.com.au/sna pchat-and-gucci-collaborate-for-try-on-campaign/ 5. Binkley, C.: In Paris unceirtanty overshadows sustainability and diversity. Vogue Bus. (2020). https://www.voguebusiness.com/fashion/paris-fashion-week-autumn-winter-2020-cor onavirus-uncertainty-overshadows-sustainability-diversity 6. Brengman, M., Willems, K., Van Kerrebroeck, H.: Can’t touch this: the impact of augmented reality versus touch and non touch interfaces on perceived ownership. Virtual Real. (23), 269– 280 (2019). https://doi.org/10.1007/s10055-018-0335-6 7. FIA.: The fabric of reality—A fully immersive VR fashion show (2020). https://www.fialon don.com/projects/ryot-studios-the-fabric-of-reality/ 8. Friedman, V.: Is this the future of the fashion show? (2020). https://www.nytimes.com/2020/ 05/02/fashion/coronavirus-digital-fashion-show.html 9. IBM.: Report how immersive technology can revitalize the shopping experience (2008). http:// www.artexperience.it/uploads/9/2/9/2/9292963/ibm__immersive_technologies_&_the_sho pping_experience_.pdf 10. Invrsion.: Manifesto section on website (2021). https://invrsion.com/full-reality 11. Javornik, A.: The mainstreaming of augmented reality: a brief history. Harv. Bus. Rev. (2016). https://hbr.org/2016/10/the-mainstreaming-of-augmented-reality-a-brief-history 12. Kavilanz, P.: No testers, no problem: Ulta and Sephora have a new take on try before you buy. CNN Bus. (2020). https://edition.cnn.com/2020/06/18/business/ulta-sephora-virtual-mak eovers/index.html 13. Koetsier, J.: COVID-19 accelerated e-commerce growth ‘4–6 Years’. Forbes (2020). https:// www.forbes.com/sites/johnkoetsier/2020/06/12/covid-19-accelerated-e-commerce-growth-4to-6-years/?sh=a5ed8e4600fa 14. Leanza, F., Balconi, M.: Consumer neuroscience: TV commercials versus virtual reality commercials (2017). http://hdl.handle.net/10807/113705 15. London Fashion Week Website.: (2020). https://londonfashionweek.co.uk/ 16. Longo, O.: Body and technology: continuity or discontinuity? In: Fortunati, L.E., Katz, J., Riccini, R. (eds.) Mediating the Human Body. Routledge (2003) 17. Marfia, G., Matteucci, G.: Some Remarks on Aesthetic and Computer Science (2018). https:// doi.org/10.7413/18258646059 18. McDowell, M.: What to know about virtual stores. Vogue Bus. (2020). https://www.voguebusi ness.com/technology/what-to-know-about-virtual-stores 19. McLuhan, M.: Understanding media: the extensions of man (1964) 20. MOR website.: About us section (2020). https://www.museumor.com/about 21. Nielsen.: Covid-19: the unexpected catalyst for tech adoption (2020). https://www.nielsen. com/us/en/insights/article/2020/covid-19-the-unexpected-catalyst-for-tech-adoption/ 22. NuOrder website.: Virtual showrooms section (2020). https://www.nuorder.com/virtualshowr ooms/ 23. Obsess website.: Customers section (2020). https://obsessar.com/customers/ 24. Papagiannis, H.: How AR is redefining retail in the pandemic. Harv. Bus. Rev. (2020). https:// hbr.org/2020/10/how-ar-is-redefining-retail-in-the-pandemic 25. Rauschnabel, P., et al.: Augmented reality marketing: how mobile AR-apps can improve brands through inspiration. J. Retail. Consum. Serv. 49, 43–53 (2019). https://doi.org/10.1016/J.JRE TCONSER.2019.03.004

54

B. Silvestri

26. Schmitt, B.H.: Experiential Marketing how to Get Customers to Sense, Feel, Think, act, and Relate to your Company and Brands. The Free Press (1999) 27. Sim, A.: Amidst canceled fashion weeks, Shanghai’s virtual runways are a glimpse into the future, Elle (2020). https://www.elle.com.sg/2020/04/22/fashion-week-cancelled-coronavirusshanghai-virtual-runways/ 28. Statista.: Coronavirus impact on retail e-commerce website traffic worldwide as of June 2020, by average monthly visits (2020). https://www.statista.com/statistics/1112595/covid-19-imp act-retail-e-commerce-site-traffic-global/ 29. Yim, M.Y.-C., Park, S.-Y.: I am not satisfied with my body, so I like augmented reality (AR)— Consumer responses to AR-based product presentations (2019). https://doi.org/10.1016/j.jbu sres.2018.10.041 30. Yotka, S.: Fashion is building a virtual future, starting with its showrooms. Vogue (2020). https:// www.vogue.com/article/fashion-is-building-a-virtual-future-starting-with-its-showrooms

Chapter 4

Remix Viralia. For a Reality of the Imaginary Re-imagined. Art and Education in the Era of Covid-19 Giorgio Cipolletta

Teaching does not produce learning but rather “creates a context in which learning occurs”. Etienne Wegner

Abstract There is a before and there is an after. Let’s start with the after. The outbreak of Covid-19 forced cultural institutions to explore alternative digital spaces with online exhibits and the use of virtual, augmented, and mixed reality. The educational world, such as colleges and universities, has also had to rethink a new way of communicating and collaborating. New technologies have re-mediated creating a different degree of participation and connection. In this essay I will argue, on one side, about the art system some concrete examples of extended reality. On the other hand, through the educational system, I will analyze how the interface between teachers and students has radically changed the relationship within an artificial and immersive environment. During the pandemia we have witnessed both the hybrid nature of the spect-actor-in the fruition of the work of art in the museum system, and a convergence towards distance learning where the relationship between teacher and student has been rethought with the help of some technological tools (Oculus, Gear VR) and distance education services (#iorestoacasa). But it is not the tool that is the main factor of this essay, but its instrumental proxy, that is, the quality (affordance) of rethinking it in an ecosystemic direction, giving us a different methodology, a choral art (#restacorale) to capture this variation of the Real, re-imagine the imaginary and remix every degree of reality. Keywords Extended reality · Spect-actor · Re-imagination · Remix · Ecosystemic-imaginaries

G. Cipolletta (B) Macerata, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_4

55

56

G. Cipolletta

4.1 Introduction Almost a year has passed since the press announced the first two patients infected with SARS-CoV-2 in Italy. They were a couple of Chinese tourists, do you remember? On January 30, 2020, China reports to the world the existence in the city of Wuhan of a cluster of cases of pneumonia of unknown etiology (later identified as a new coronavirus Sars-CoV-2), the same day the WHO (World Health Organization) declares a state of international emergency. The following day, January 31nd, the Italian government proclaimed a state of emergency and implemented the first measures to contain the infection throughout the country. One of the first measures adopted is the suspension of all flights to and from China with the implementation of airport controls, using thermoscopes to measure body temperature, in order to monitor the health conditions of passengers coming from China through intermediate stops. On February 21nd, the Italian Ministry of Health introduces mandatory quarantine isolation measures for close contacts with a case that has tested positive for Covid-19, and orders active surveillance with a fiduciary home stay for those who have been in at-risk areas in the previous fourteen days, with the interested party required to report to local health authorities. Starting from this date an increasing number of cases of infection has been detected daily, so much so that in ten days the number of a thousand subjects found infected is exceeded. In this new scenario we found ourselves living a rather extraordinary situation. The first decrees (DCPM) of the President of the Council of Ministers (Prime Ministerial Decree) were signed on March 4th, then on March 8th, up to the most restrictive one of March 9th (#iorestoacasa), stating new measures for the containment and contrast of the spread of the Covid-19 virus over the entire national territory transformed into a “red zone”, sanctioning the closure (suspension of teaching activities) of every order and grade of all educational institutions. It seems that almost a century has passed since that day when the first patients appeared at the door of a hospital in Italy and that soon afterwards it would be the country (the first European one) hit by the violent wave of the pandemic. While we were learning the vocabulary of quarantine and washing our hands for twenty seconds, the temperature of the planet continued to rise. The pandemic (or syndemic,1 Singer, Merril 26) accelerated the changes that were already taking shape. At some point we all (half the world’s population) found ourselves staying home, meaning that 3.9 billion people (more than there were in the world in 1970) were experiencing emotional and financial repercussions of the new coronavirus. According to

Etymologically, “syndemia” is derived from the Greek συν (together) and δημoς ´ (people), with “ν´oσημα” (pathology) implied. The hallmark of a syndemic is indeed the presence of two or more concomitant pathologies, which interact negatively, unfavorably affecting the specific course of each and increasing vulnerability. A “syndemic approach” examines the health consequences of interactions between diseases and the social, environmental, or economic factors that promote that interaction and worsen the disease. See online: http://www.nbst.it/822-pandemia-covid-19-%C3% A8-anche-sindemia-disuguaglianze.html (last access 13rd January 2021). See online: https://www. thelancet.com/series/syndemics (last access 13rd January 2021). 1

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

57

the health psychologist Elke Van Hoof (312 ) of the Free University of Brussels, the lockdown is the most significant psychological experiment ever conducted. Obviously an unintentional experiment, but one from which we are already calculating the first results. The pandemic certainly laid bare the disparities in the health care system and economic security measures, as well as a rethinking of relationships and proxemics3 [17, 19, 10, 20]. The disposition of bodies in space expresses a different type of social relationship that from time to time individuals organize through relationships of equality or inequality, symmetrical or asymmetrical disposition, synchronic or asynchronous, resulting in tendencies to approach or recede (Danziger 1976). The new distance overturns the entire grammar of proxemics during the Covid-19 pandemic. The new “distance of insecurity4 “at least one meter from each other,—experts say—will serve to discourage the spread of the virus and replace interpersonal relationships mediated by appropriate communication technologies, from cell phones to delivery services booked online to the most sophisticated smart applications for e-learning. Through virtual platforms we have found ourselves opening a window into our private physical world, suspending the “rituals of deference and demeanor” [16], (Goffman 14). This hyper-spatial dimension is also reconfigured in the ways we experience relationships and interpersonal distance. This reinvention of ritual and of inhabiting digital spaces generates a “New Dwelling” (Iaconesi 18), where data and computation join the body, psychology, relationships, and experience of the world. The availability of data about our bodies connects to that of the ecosystem in an intimate, responsible and globally interconnected game, but above all with an awareness of self-care. That’s why it’s not just space that re-spatializes itself by squeezing on one side and expanding into the virtual. If our body is thought of as a device through which we have and manage the world, now “infected”, consequently we reposition ourselves, recreating a new dimension of the body itself. 2

See online: https://www.scientificamerican.com/article/the-biggest-psychological-experiment-inhistory-is-running-now1/ (last access 13rd January 2021). 3 Proxemics wants to be a semiology of space as it identifies in it a real channel of communication and, in the different ways of its being organized, it discovers a series of messages that can be interpreted with an anthropological code that, by establishing rules of equivalence between meanings and signifiers, attributes to the various distances a different semantic value, on which affect certain ethnological and psychosociological conditions. In this sense, it can be argued that proxemics recognizes, alongside the three known dimensions of space, the existence of a fourth dimension of a cultural nature. 4 In many countries, the first effects of these new proxemics rules are beginning to be felt. This is how Interior Minister Horste Seehofer decided to shake hands with Chancellor Angela Merkel during a meeting on immigration in Berlin, or how new alternative greetings are proliferating: no handshake, better to touch with the foot. The greeting seems to have been devised by Tanzanian President John Magufuli during the palace welcome to opposition leader Maalim Seif Sharif Hamad. The “foot” salute was promptly renamed the “Wuhan Shake.” Obviously also in China the greetings have been downsized so much so that they have returned to Bao Quan Li, the classic martial arts greeting (closed fist in open hand with bow), which does not involve physical contact. In the American culture, instead, it seems to be more in vogue, as a substitute of the classic handshake, a contact between elbows (elbow bump) already adopted in 2006, at the time of the avian flu.

58

G. Cipolletta

This metric of the body, which I call the “metrobody” [Cipolletta 5, 1, 6], is managed and re-positioned between domestic security and internal psychological insecurity, amplified with the use of personal protective equipment as the outside presents itself. Here the body, as well as the medium, becomes a mediator of viruses and contagion. This new dimension of the sociality of the human causes a transformation of spaces forcing us into social distancing, a horrible word, which I would rectify with physical distancing and the hope of social involvement. In the process of r(i)existence the initiative could be to refound an ecosystemic project, which concerns not only the human, but the living (and also the non-living) in a hybrid connection, where human responsibility could generate a sense of plural community, involving any system of the living. This different way of re-positioning also produced a linguistic vocabulary capable of separating the before and after, B.C. Covid (Before of Covid) and A.C. (After Covid). Many lemmas have been linked to the pandemic, from techno-scientific ones like asymptomatic, anosmia, dysgeusia, dyspnea, morbidity to untranslatable ones like lockdown (word of the year according to Collins Dictionary 85), spillover, contact tracing, social distancing. Another revision of some lemmas that appeared in dictionaries was the inclusion of new neologisms such as apericall, aperizoom, zoombing,5 covidiot,6 workcation,7 staycation that contributed to as many new ways of saying and describing this scenario.

4.2 For an Ecosystem Approach to Schooling This non-resumption of scholastic ritual has reordered the educational system towards a radical change, taking into account the many difficulties in a country like Italy with all its social and territorial problems. There are many implications of the digital divide, an unequal ability to exploit digital technologies, who contours are very complex. Access to information technologies and the Web is seen as a necessary condition for full social inclusion. It is not only the social gap of access, but also the skills of use and the different uses that are made of them. Therefore, there is also a debate on the skills needed for inclusion in the digital world that requires a “human” competence, in order not to be overwhelmed in the communicative overabundance. In fact, it has been noted that there is a gap in access to the Web in skills and uses, but even there is a reversal. Those who surf more nowadays are the most disadvantaged people. 5

Zoom-bombing is when an uninvited guest gate-crashes a virtual meeting and takes control of it, perhaps even sharing inappropriate content. 6 Covidiot is a slang insult for a person who disregards safety measures or goes against public health advice amid the pandemic. Covidiot is a blend of COVID-19 and idiot. 7 A staycation or holistay is a period in which an individual or family stays home and participates in leisure activities within day trip distance of their home and does not require overnight accommodation.

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

59

Through smartphones the relationship has reversed whereby those who had more resources spent more time online and the Web becomes a replacement for other possibilities or missing stimuli. In Italy, among high school students, those who have a social disadvantage are those who surf more, in their lives, the smartphone appears to be more pervasive with a scarcely informative use, but playful-relationship as chat, YouTube, etc. … In this framework of multifaceted disparity the distance learning is inserted. Digitization has in some ways been the right response. At the same time, the emergency use of digital media has given families a great responsibility in organizing teaching and managing digital tools. The urgent push for didactics has left uncovered what was the real urgency, that is, an “ecological” media education [25, 5, 18, 27]. In the digital era, e-learning has seen great development, certainly with very different environments, approaches and ways of delivering courses. First of all, we need to say that today e-learning is also called with other terms, with other syntagms, such as Web Based Learning, Online Learning, Distance Education. In fact, all the tools such as blogs, Wikis, social networks, all social platforms, allow the student to take advantage of a teaching–learning process that is extremely interactive; there is no longer a “one to many” communication, but a “many to many” communication. Distance learning is transformed from a compensatory solution to a precise design choice. Today, beyond the teaching platforms, many social media can be used precisely as environments for the teaching/learning process. Information and communication technologies (ICT) should be learned and used structurally within technological models of education on the basis of a conscious and preventive pedagogical and didactic choice (models of use, dosage of time). It is necessary to pay attention to the risk of using new technologies exclusively as a decorative element or restyling of a traditional didactics and an enhancer of an exclusively transmissive didactics, for example using the IMW (Interactive Multimedia Whiteboard) as a tool to transmit the same contents in the same way, without exploiting the real innovative and cooperative potential of the tool. ICTs, therefore, favor the co-construction of learning objects such as conceptual and mental maps, learning objects, or documentation and access to content such as the platforms already mentioned (such as Moodle), so that their subsequent re-elaboration moves towards a perspective of personalization and individualization for the production and construction of new meanings and materials. It will therefore be fundamental that the whole class, starting with the teachers, use the same applications or educational programs, even if on different supports (Tablet, Notebook, IMW). The American psychologist Ausubel (2) firmly believes in meaningful learning, that is, in that process in which the student relates new information to other information he already has in his cognitive structure. The concepts that are already present form a bridge between what is already known and what is to be learned. A constructivist learning environment is a “space of action” intentionally prepared by the teacher, consisting of a set of structured activities aimed at directing, without directing, or directly guiding, the learning process that is intended to be promoted; based on meaningful experiences for the student and in his own self-efficacy, fundamental dimensions for the learning processes themselves. It is necessary, therefore, that teachers focus not only on content, but also on processes, providing activities that

60

G. Cipolletta

encourage and facilitate observation and reflection on their own mental processes. This is why turning to digital could represent an opportunity that can arise from this condition of crisis. Not digital itself, but all the forms that can come through the possibility of continuing to lecture or reinventing the way of lecturing through systems and platforms, which many schools and universities already have, or more or less proprietary tools that are already being used, but in an occasional and marginal way, but even here let’s not forget the great difficulty for many in being able to take advantage of this digital revolution. Roberto [22, 23], professor of Instructional and Learning Technologies at the University of Roma Tre and author of numerous texts on the subject of distance learning, invites us to reflect on how we are dealing with profoundly backward positions, especially now that with digital technology we are dealing with a technology that is stronger than its predecessors, because it is capable of encompassing and engulfing them (so-called convergence). It would be one more reason to introduce it at school, but this also means a change in method and content, something that obviously arouses distrust and fear, because it is always difficult to question oneself and take new paths. We have become aware of the need for a radical change in the way we do teaching, at the moment in which an event makes our daily lives ordinary, extraordinary.

4.2.1 #LaScuolaContinua (#SchoolContinues) In Italy there have been many tools available to teachers for distance learning such as Classmill,8 a free application that allows you to produce multimedia lessons online. The Edmodo9 platform, on the other hand, presents itself as a true educational social network, while Eliademy as a free virtual classroom that empowers teachers and students to create, share and manage online courses. Also Fidenia10 the “social learning”, all-Italian, dedicated to teaching, is designed to create virtual classes, share resources, create multimedia content, assign tests and dialogue in a “collaborative” way between teachers, students and families. A more well-known platform is WeSchool,11 a free platform for collaborative teaching that uses the methodology of the flipped classroom. Among the content available within WeSchool of teachers there is a collection of online lessons included in the platform Oilproject 12 that collects an Italian community with free lessons. Finally, as we all know, there is the Google suite package with the Classroom13 application that has made teaching more

8

See online: https://classmill.com/ (last access 13rd January 2021). See online: https://new.edmodo.com/ (last access 13rd January 2021). 10 See online: https://www.fidenia.com/ (last access 13rd January 2021). 11 See online: https://www.weschool.com/. (last access 13rd January 2021). 12 See online: https://library.weschool.com/ (last access 13rd January 2021). 13 See online: https://classroom.google.com/u/0/ (last access 13rd January 2021). 9

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

61

productive, collaborative and meaningful in connection with Google Meet,14 the platform dedicated to video lessons. Another platform in contrast to the latter is Zoom Cloud Meeting15 which offers the possibility of creating events in both webinar and webmeeting mode. Another platform that has been used during this period of pandemic is Otus16 where we find a network environment for the creation and management of virtual classrooms, designed to be immediately operational on iPad and Chromebook. Microsoft has also made its online platform available with Office 365 Education17 dedicated to schools and universities (Microsoft Teams). This multimedia interface offers inclusive, student-centered learning environments that allow students to learn through free technologies and tools. An interesting note should be dedicated to Schoology,18 a free online learning environment, with advanced LMS (Learning Management System) features, to promote collaborative teaching, run on any mobile device. Among the most used ones, we have Socloo19 a social educational environment designed and developed exclusively for the Italian School, allowing teachers to involve their students, creating virtual classes and working groups to communicate in real time, to share digital resources and to collaborate in the construction of their e-portfolio. During the Coronavirus emergency, the school, as we have seen, changes its clothes, adapts, modifies itself thanks to new technologies and different modes of use. That’s why in Italy #LaScuolaContinua (distance learning) in a deeper, more authentic and inclusive way. Immersion in distance education remains an open problem but also an opportunity. In recent years, virtual reality (VR) and augmented reality (AR) have attracted interest after Mark Zuckerberg, in 2016 at the CES (International Consumer Electronics Show) in Las Vegas and at the MWC (Mobile World Congress) in Barcelona, announced to bring his own Oculus Rift (AR) visor to the world market by investing two billion dollars. Many other companies such as Sony, Samsung, HTC and Google have begun to capitalize in the VR and AR industry. As we note all of the cameras and screens found in smartphones and other portable devices are tools built to “merge” real-world data with virtual data. Let’s take a step back. The study of Virtual Reality (VR) was born in the field of computer graphics (Sutherland 28) and extends to different disciplines. Let’s think about video games supported by VR tools that today have also become innovative methodologies for research in neuroscience, psychology, biology, etc.… We also find a large availability of free tools for experimental and computational use of VR that has made it easy to access any field. Augmented Reality (AR) is used to increase content comprehension and memory retention, as well as learning motivation. Paul Milgram (24), in this regard, defines AR as a set of circumstances in which a real 14

See online: https://meet.google.com/ (last access 13rd January 2021). See online: https://zoom.us/it-it/meetings.html (last access 13rd January 2021). 16 https://otus.com (last access 13rd January 2021). 17 https://www.microsoft.com/it-it/education/products/office (last access 13rd January 2021). 18 https://www.schoology.com/ (last access 13rd January 2021). 19 https://www.socloo.org/ (last access 13rd January 2021). 15

62

G. Cipolletta

environment is “extended” through virtual objects thus realizing a continuum called “Mixed Reality” (MR) between a real context and a virtual environment. Virtual and augmented experiences are a change, a transformation, in schools, universities, and academies. Many publishers, for example, have become interested in and developed a new type of content by mixing analog paper with multimedia enjoyment through virtual and augmented reality. For example, the union and the combination of an AR system with print are able to provide an added value to the communication. In the scholastic field, we have seen concrete examples such as encyclopedias for children, books for the didactics of the various disciplines and in the various school orders, technical books for professional schools, manuals, guides for visits to cities. Google is experimenting with new ways of publishing, an example of which are the so-called “digital native books20 ”, an evolution of the eBook designed to offer a new literary form, interactive and easy to use. Published books can be, for example, expandable, augmented, collaborative, shareable, geolocalized, but also “cookie-ish”, thus making use of data and information, but also of algorithms, able to make the reading experience increasingly personalized and responsive to the needs of a new generation of readers. Augmented Books21 are another example where the paper book is proposed as an interface for AR, associating the printed text with interactive multimedia elements usable through AR display devices. These types of books can also be enjoyed by multiple users at the same time (multi-user). In this virtual and augmented immersion, teaching becomes experiential, especially at a time when we need to reinvent ourselves. The approach of the virtual and augmented reality, through the versatility of its use in relation to the objectives to be achieved allows the creation of an immersive, innovative and stimulating study environment, where the “digital” content enriches the perception of interaction and fusion between real and virtual objects creating a “mixed reality”, thus promoting a transdisciplinary approach between the different subjects. In this “augmented learning” different thinking styles are produced, as well as creative and divergent solutions to the problems of contemporary life. One of the most widely used augmented reality apps at the moment is certainly the Aurasma22 app developed by the company Autonomy. In Aurasma, each image or object can have its own “aura”, such as a video, a link to a web page or more complex as a 3D animation [11]. This application allows you to associate logos, images, news in newspapers, books or whatever else comes to mind, with videos, animations or sounds, which enrich the content. The use of AR in formal education may prove to be a key component in futuristic learning environments that are richly equipped with hardware and software applications. Another example are GoMeta’s Metaverse23 or Assemblr24 for creating Augmented Reality experiences. Through this application you can arrange components and images on a storyboard and link them together 20

https://editionsatplay.withgoogle.com/#!/ (last access 13rd January 2021). https://www.arbibook.com/?lang=en (last access 13rd January 2021). 22 http://aurasmaproject.weebly.com/getting-started.html (last access 13rd January 2021). 23 https://studio.gometa.io (last access 13rd January 2021). 24 https://assemblrworld.com/ (last access 13rd January 2021). 21

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

63

without having to write any code. These two tools let you create storytelling or create roleplay in different authentic and interactive scenarios. Through these projects of augmented learning we are supposed to strengthen the memorization of content, the motivation of students to learn and participate in the educational dialogue in an autonomous and conscious way. It also enhances the inclusion of all students, leveraging the strengths of each and the need to work on the weaknesses of others. In this way, everyone actively participates in learning and “multiple intelligences” [12, 15] are enhanced. It also amplifies the value of collaboration and cooperation among students who learn to respect the needs of their peers while performing activities. Finally, students learn to have a working method, a strategy and a procedure to follow, thus learning to organize their activities to address and solve the problems they encounter. The creativity developed in immersive education allows to create settings, learning scenarios in virtual worlds that refer to questions and activities using methodologies based on problem-solving, learning by doing and game-based learning. Another example of virtual and augmented reality is offered by Firefox Reality,25 a new browser that allows you to take advantage of different viewers for mixed reality experiences. Mozilla Hubs26 is developed in WebVR a standard supported by every VR viewer on the market, such as Oculus Rift,27 HTV Vive28 and Google Daydream.29 The users can communicate together with each other in virtual rooms created by choosing scenarios and avatars to participate in immersive educational experiences. The use of a visor, such as cardboard30 or VR helmet makes these experiences even more immersive and interactive by catapulting students inside the narrative, discovering a different way of participating in learning and relating to their peers and teachers. The Google for Education31 team in recent years have made virtual and augmented reality learning experiences with Google Explorations and Tour Creator (untile June 30th 2021) will be implemented with Google Arts&Culture.32 The immersive dimension that the video game or virtual reality involves mainly the emotional sphere of those who use it, leading to forms of active involvement highly motivating. The challenge of bringing VR and gamification produces in users a motivation and a critical reflection of the experience itself. Game-based learning situations with or without VR have the ability to overturn the transmissive model for students who become active prosumers of learning and builders of their knowledge in a context of cooperation and collaboration. A VR environment is therefore able to stimulate the human capacity to perceive and imagine creatively translate and convey complex concepts thanks to innovative visualization modes. In other words, 25

https://mixedreality.mozilla.org/firefox-reality/.x\ (last access 13rd January 2021). https://hubs.mozilla.com/ (last access 13rd January 2021). 27 https://www.oculus.com/rift/ (last access 13rd January 2021). 28 https://www.vive.com/us/ (last access 13rd January 2021). 29 https://arvr.google.com/daydream/ (last access 13rd January 2021). 30 https://arvr.google.com/intl/it_it/cardboard/get-cardboard/ (last access 13rd January 2021). 31 https://edu.google.com/intl/it_it/ (last access 13rd January 2021). 32 https://artsandculture.google.com/ (last access 13rd January 2021). 26

64

G. Cipolletta

experiments in VR allow the activation of visual-spatial and kinaesthetic intelligences as in real interaction, so as to provide a perceptual basis for logical intelligence. The degree of immersion is given by the different modalities of fruition starting from the computer, interactive whiteboard or tablet, then moving on to viewers and smartphones to reach the maximum immersive experience in a VR station with helmet, gloves and motion sensors. The construction of learning situations enhanced by virtual reality presupposes an active teaching, leaving space for the student in co-collaboration with the teacher. Another example is CrossLesson33 (Manzoni 21), which is an attempt to connect the immersion of VR environments with Problem based learning. The acronym CROSS indicates the phases on which the lesson is structured (Challenge, Research, Operate, Say, Share), mixing reality and virtuality. The skills that are developed through this teaching methodology are as follows: The first competence is Challenge understood here as the ability to let oneself be questioned by the problems of the world and to compete for the achievement of a goal. The second competence is Research, or information literacy, that is, the competence of selecting and evaluating information that is indispensable in the knowledge society. The third skill is Operate, or the ability to work towards the realization of one’s ideas. The fourth competence is Say, that is the ability to argue in a well-founded way one’s own point of view or simply to tell through multimedia, virtual or not, one’s vision of reality. Finally, the last competence is Share, as the willingness to share with others one’s own solutions, to question them, to revise them, to evaluate and to be evaluated. Virtual, augmented and mixed reality, in the educational field as we have seen are recognized as effective tools to support teaching/learning, in accordance with the constructivist approach. In particular, virtual worlds allow the activation of specific tasks within settings modeled as scenarios aimed at learning objectives.

4.3 A Hybrid Non-conclusion So that the school-world relationship can be mended through virtual practices the closeness that requires a physical distance, but not a mental one. Here is that the school should recover the terms of “proximity” and “imagination”. DxI (Didactics for Imaginaries), where it is the imaginaries themselves that produce proximity and the multiple didactic methodologies are overturned. The interface, the flat and cold screen of the pc has become the new desk, there would be the need to re-imagine teaching from another point of view. These devices of imagery open wide the veil of imagination to infect any form of exploration. I’m not saying it’s easy, but an attempt can be made. If the school is far away, what is close at this moment is the digital platform that must reform and grant a different information, training and form to the shiny flatness of the interface.

33

https://crosslesson.com/ (last access 13rd January 2021).

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

65

To do this surely there is a need for cross-disciplinary methodologies that collaborate and use technological devices. Here if we changed our predisposition to devices, we could dispose these tools to a different function, that is to perform. Teaching performance includes distance to proximity. It is in the play of the parts to rethink a performative didactics and an open school that “saves” the bureaucratic closure of the Italian education system. DaD (distance learning) is not a word of order, but of disorder, and it is precisely in this disorder that the game of didactics understood as “practiced space” [9, 8] should be played out. The school must be “open”, as a flipped school, not only as a flipped classroom34 [20, 29], and from this flipping we must pick up and cultivate an important issue: the ecological transmission of relationships, connecting them. In this interconnection, the school and its actors, the students, should together weave a network, so that there is a school-network. The ability to eradicate the dusty bureaucracy of the school lies precisely in this connective dimension. Why not try right now to activate simulation tests, through virtual performances, augmented to completely uncover the timetables, the frontal interrogations, the examinations. These bureaucratic forms of control exclude the school from something instead that should unite, make community of interconnected cultures. The biggest problem we can find is that of a substantial “failure” of the European investments made in the last fifteen years for the development of ‘technology enhanced learning’ (Giovanella 13). We ended up immersed and not only the school, but also art and many other cultural realities in the arms of Google, the school in Classroom, despite the fact that it is not an environment designed for teaching but for collaborative work and despite the fact that it does not provide interoperability and despite the fact that the data reside in the hands of Google, despite the fact that it does not offer analytics, etc.… (Ibidem) The idea could be to create a European (not only) cultural infrastructure (platform) where to develop learning ecosystems. DxI (Didactics for Imaginaries) is a complex matter that requires complex interventions and simplex (Berthoz 3) answers. In the field of distance (and proximity) learning, despite the technical difficulties that require a European, before national, political intervention, it must adhere to a (platonic) reversal of the relational transmission between teacher and student. There is a need for collaborative co-construction, sharing, abandoning the “perverse” and “hysterical logic of linearity” of the manuals in favor of an approach instead hyper-connective, multimedia and transdisciplinary. The didactics still linked to the traditional division of subjects, hours, classes should adhere to a

34

Flipped teaching is an educational methodology that in recent years has been positively spreading especially in the world of education. The literal translation of the term flipped classroom means a mode of teaching (and learning) supported by digital content where the timing and pattern of work are reversed compared to traditional methods. The latter, in fact, provides a first moment of explanation, where the teacher gives a lecture in the classroom to the class, followed by a second moment where students do their homework individually. The flipped classroom produces a reversal of roles between teachers and students, where pedagogical control of the process shifts decisively from the teacher to the students. In other words, in assuming centrality in the process of learning, students are called upon to assume greater autonomy and responsibility for their own educational success, while the teacher assumes the task of guiding them along their educational path.

66

G. Cipolletta

broader discourse where to include personalized and collective paths both individual and group plural, participating, learning to learn, designing and reimagining. Distance learning is a particular term, but even that terminology can be spilled over into DxI (Didactics for Imaginaries). As soon as students remove their desks, what happens? This is exactly what distance learning has done and Danila [37] explains it well in a TEDx talk, pointing out that this physical absence of the table replaced by the digital platform not only emphasizes even more the social gaps that exist between them, but also activates in students a deep look, gathers a reality that is less visible at school. It is precisely by starting from the differences that we must accelerate the changes of mind (mentality/mindset) and the student is the real engine of change that activates a process of innovation and reaction. In this version of distance learning proposed by Leonori, the “immersive” environment is the underground where a dense network of relationships and ties are woven, just as plants do, adapting and evolving to sudden climatic changes. In this “evolutionary” choice if plants have chosen to remain as we have adhered to remain stationary during lockdown, then in this fixity lies the exploration in this case of the learning environment. In this “climate change” of didactics, we should rethink, redesign and re-imagine a qualitative and ecological approach to didactics According to Leonori, being teachers and students is about making available, welcoming the other, it should be always but even more so with DaD or what we have come to know as DDI (integrated digital didactics).35 Technology, which is never neutral, just like art, is not a solution, but a way of feeling and knowing. Precisely from this argument, the questions posed here are always complex and to address them we must broaden our points of view. Research and education must leave the classroom and the laboratory and join other people, in society, in the city, in the territories, in the environment, whose subjects, actors, people, companies, animals, plants, fish, robots and artificial intelligences must become partners of research and education, it is the condition of ubiquity. Research and education must leave the classroom and the laboratory and join other people, in society, in the city, in the territories, in the environment, whose subjects, actors, people, companies, animals, plants, fish, robots and artificial intelligence must become partners of research and education. It is the condition of ubiquity. And for this condition new cosmologies and new rituals are needed, to reposition ourselves: no longer the human being, the individual, at the center of it all, but the awareness (and consequent assumption of responsibility) of being perpetually in transit between our individuality and part of a network that includes humans, non-humans, computational agents, actors with legal personality (Iaconesi 18). In this ubiquity [4] different methodologies are established to learn a different approach for a hybrid project made of skills, feelings and dreams. In order to 35

Students are those curious creatures who spend much of the morning hiding behind a desk, which they quickly manage to repurpose as if it were a trench of sorts. Their living space behind which they feel safe. If I had to describe a day in the classroom with them, I’d say it’s a mix between riding a roller coaster and the Ramones’ Rock N’ Roll High School video (you know the part where they say “I don’t care about history, because I wouldn’t want to be here”)? There, that might give you the idea. D. Leonori, TEDX, Ascoli 2020. https://www.youtube.com/watch?v=_P7KIwLw06M& fbclid=IwAR2H1jHdPkQPSttkFo-JpO8GYquFvFcTMYdYqhkxgdjmVrxYun8McGioT30.

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

67

have this plural dimension there is a need for trust and responsibility in collaboration. What I am talking about is an always-on [30] didactics, that is a didactics present within hybrid teaching–learning environments, where the real and the virtual merge. This approach contributes to blur the boundary between physical spaces (e.g. the classroom) and digital spaces (online learning environments), thus moving towards a new vision of the “hybrid” interaction space. These hybrid spaces are dynamic spaces created by the movement of people always connected on-life36 [11] to mobile devices. The effect of always-on connection changes our perception of space, including remote contexts in the one experienced at the moment. For there to be an always-on didactic, it is necessary to foster a real didactic innovation functional to the improvement, enrichment and enhancement of teaching–learning processes. The Covid shock has concerned not only schools, but also cinemas, theaters and museums, revealing a cartography of heterogeneous situations. Most cultural institutions around the world are closing indefinitely, and personal exhibitions, events and shows are being cancelled or postponed. In response, there have been intense efforts to provide alternative or additional services through digital platforms, to maintain essential activities with minimal resources, and to document the events themselves through new acquisitions, including new creative works inspired by the pandemic. In this cartography of renewed arts, there have been many examples that reveal a willingness to combine the moment of education with a taste for fun and gamification. We find ourselves immersed in the horizon of infotainment, in which information and entertainment contaminate and overlap. Among the initiatives we can cite the digitization of works (Louvre,37 MoMa,38 Met,39 Rijksmuseum40 ), the creation of apps and the construction of virtual apps with thematic paths on artists and historical periods (MoMa, Met, Smithsonian,41 Rijksmuseum, Vienna State Opera,42 Tallin Art Hall,43 Zacheta National Gallery of Art,44 Banghabandu Musueum of

36

The term Onlife was coined by Luciano Floridi to represent the experience that man lives in hyperhistoric societies where “he no longer distinguishes between online or offline”, and even where “it is no longer reasonable to ask whether one is online or offline”. The word explains the fusion of the digital in the analogue caused by information and communication technologies (ICTs), demarcating the historical period from the hyperhistorical one; in fact “[…] we are probably the last generation to experience a clear difference between offline and online” in fact “the obvious dichotomies such as those between real and digital or human and machine are no longer sustainable in a clear way”. 37 https://www.louvre.fr/en/visites-en-ligne (last access 13rd January 2021). 38 https://www.moma.org/calendar/groups/58 (last access 13rd January 2021). 39 https://www.metmuseum.org/art/online-features/met-360-project (last access 13rd January 2021). 40 https://www.rijksmuseum.nl/en/from-home (last access 13rd January 2021). 41 https://naturalhistory.si.edu/visit/virtual-tour (last access 13rd January 2021). 42 https://ivrpa.org/panorama/vienna-state-opera/ (last access 13rd January 2021). 43 https://www.kunstihoone.ee/en/ (last access 13rd January 2021). 44 https://zacheta.art.pl/en (last access 13rd January 2021).

68

G. Cipolletta

Bangladesh,45 National Museum of Singapore,46 Bahrain National Museum,47 San Paolo Museum of Art48 and the centralized platform managed by the Chinese government49 ). And again, specific curatorial focuses and free image-sharing programs of works in the collection (Rijksmuseum); collections dedicated to Covid’s impact on the arts (Tate); thematic channels on YouTube (Heine Onstadt Kunstsenter50 in Oslo, Lousiana Museum of Modern Art51 in Copenhagen); downloadable radio shows on YouTube and SoundCloud (Gamec in Bergamo52 ); museum interfaces confronts the public (Rijsks From Home at Rijksmuseum and National Museum of Australia53 ). And then immersive journeys (Place Museum of Australia54 ); blogs (Zeit Museum of Contemporary Art55 in Cape Town); smart games (Monterrey Bay Aquarium56 and Nelson Atkins Museum); collaborative projects (Cleveland Museum Art57 ); appointments reserved for the visually impaired (Guggenheim); forays into social networks frequented by young people such as TikTok or its Chinese version Douyun (Uffizi,58 Museum Challenge). Finally, MOOCs59 (Massive Open Online Courses) free academic courses accessible to all (MoMa,60 Pompidu61 ). These examples as well as many others are the expression of a change of the museum in a different perspective where online and offline are mixed. Museums should address this dramatic crisis by developing a set of policies in the use of digital so that they become devices of “creation” to reflect and invent the future [32]. In every field of our life communication technologies with all its derivations of virtual, augmented, mixed reality have become forces and activators of structuring and “restoration” of the environment in which we live and are forced to live 45

http://www.bangabandhusbangladesh.ca/virtual-museum (last access 13rd January 2021). https://www.visitsingapore.com/editorials/covid19-guide/explore-museums-at-home/ (last access 13rd January 2021). 47 http://data.moc.gov.bh/vt/VR/nationail_museum/bnm.html (last access 13rd January 2021). 48 https://joyofmuseums.com/museums/americas-museums/brazil-museums/sao-paulo-museums/ sao-paulo-museum-of-art/ (last access 13rd January 2021). 49 http://virtual.vizen.cn/ (last access 13rd January 2021). 50 https://www.youtube.com/user/Kunstsenteret (last access 13rd January 2021). 51 https://www.youtube.com/channel/UCkB2xFSR7r8iu6nbUdP0EZQ (last access 13rd January 2021). 52 https://soundcloud.com/gamec_bergamo (last access 13rd January 2021). 53 https://www.nma.gov.au/visit-us (last access 13rd January 2021). 54 https://www.nma.gov.au/whats-on/tours-and-experiences/virtual-gallery-tours (last access 13rd January 2021). 55 https://zeitzmocaa.museum/ (last access 13rd January 2021). 56 https://www.montereybayaquarium.org/for-educators/learning-at-home/crafts-and-printables/ be-a-sea-searcher (last access 13rd January 2021). 57 https://www.clevelandart.org/categories/building-project (last access 13rd January 2021). 58 https://www.tiktok.com/@uffizigalleries?lang=it (last access 13rd January 2021). 59 https://www.mooc.org/ (last access 13rd January 2021). 60 https://www.moma.org/research-and-learning/classes (last access 13rd January 2021). 61 https://www.centrepompidou.fr/en/the-centre-pompidou-school/different-ways-of-learningand-training (last access 13rd January 2021). 46

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

69

in the pandemic emergency. In this hybrid approach, natural and artificial realities create new imagery. The “immersiveness” is an interactive experience between body and environment, where the fruition actually configures a meaningful experiential immersion that generates emergent phenomena. In other words, the act and condition of immersion produce both an articulated emergence of sensations, images, emotional phenomena aroused by the immersive environment, as well as the emergence in the body of spontaneous productions that concern, according to the thesis argued by Andrieu and Bernard (2018) both the creator (the artist) and the spectator (“spect-actor”). In immersing oneself in these dimensions, the depth of the body is activated, from the simplest interactions of the skin to the im-sertion (“immersion” and “insertion”) in the deep, empty spaces of our darkest part (Ibidem). In this philosophy of immersion, “osmosis, symbiosis, vertigo, orgasm, and ecstasy participate in a sensory anthropology of the intimate.” At the moment when a body is immersed in a virtual environment, the spectator-actor is in a continuous state of attention in which the real body seems to be abandoned: to be here and there simultaneously. This sensation of bodily detachment offers the possibility for the viewer to be both protagonist and receptor of the immersive experience. In other words, it is the spect-actor himself who performs, who experiences his experience within a tactile, sound and visual environment. This activates a new “condition” of multisensory existence, incorporating on the one hand our “real” body, on the other is the virtualization of the body, in both cases, reality and imagination lose their boundaries. The new technologies as we have seen expand the scope of bodily activity, markedly expand the domain of the pre-personal, while the environment-organism participates in the incorporation by creating a “medium” for our co-bodying. In the immersive paradigm a symbiotic relationship is created with imaginary and real characteristics, where the myth of Narcissus makes McLuhan’s theory [24] of media narcosis relevant and belonging to our being in the world. A body experiences these mixed dimensions (virtual and augmented), it “s-measures” itself in the world and with the world, becoming metric, matrix of the world itself [6].

4.4 #RestaCorale (#stayChoral) In order to adhere to this thought of change, I propose to conclude this article with a “choral” example that on one side abandons the “technological” dimension of virtual and augmented reality, on the other side amplifies the human experience, giving us back an “other”, hybrid and “extended” dimension. Luigi Pagliarini psychologist and artist decides to break time, to tear it to pieces, with the extreme internal need to make its useful non-existence unrecognizable. The Abruzzese-Roman psychologist, artist and professor at the Academy of Macerata, plans a choral path collecting testimonies, faces, signs and feelings of one hundred and fifty young artists-students during the lockdown phase. Choral Art (Pagliarini 2020) is a complex manifesto transcendent and above all of listening. The Choral Art manifesto was born from the strong impulse given to us by the confinement of

70

G. Cipolletta

2020 which, by enclosing and separating us, has deprived us of the most classic need of a human being, sociality. The entire #restaCorale62 project with the subtitle Greetings from the Captive World still in work in progress is broken down into 7 acts (#restaTranscendente, #restaCasa, #restaSicuro, #restaCreativo, #restaUmano, #restaTeStesso, #restaInAscolto. Conversely, I feel it’s critical to mention the smart use during online sessions of operational feedback that, perhaps more than anything else, turn a collective work into a choral work. How did this choral work take place? During the multiple sessions of meeting with the students, Pagliarini hands in all the works sent by his students. The immediate and direct comparison makes this visual and collective work a real Choral Work. Once the immense and tiring work of coordinating the artists and collecting the materials, in a situation like the lockdown, was completed, the editing and realization phase began. Luigi Pagliarini with his #restaCorale welcomes and collects time, and then builds within it a biographical path composed of meetings and dialogues with friends (those of yesterday and today and those physically lost), thoughts and projects, all to weave the transcendental essence of discovery, where time, in fact, shakes its cards, mixing them with the precious ingredients of past lives, suspended and always in tension. This living choral mosaic within a polyphonic breath belongs to and holds the exceptionality of time, taking care of itself and especially of the Other. Just in the dimension of care we should pursue an allocentric vision, which escapes from anthropocentrism to re-establish a contact lost in the pandemic time. This choral manifesto wants to be a metaphor taken from the art world to collect in the hybrid and ecosystemic version of the “extended” life the flower of the imaginary.

62

https://www.abamc.it/mostre-e-eventi/item/restacorale-restatrascendente. Cipolletta [6]. In https://www.arshake.com/intervista-luigi-pagliarini-arte-corale-pt-i/. Cipolletta [5]. https://www. arshake.com/intervista-luigi-pagliarini-arte-corale-pt-2/ (last access 13rd January 2021).

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

71

References 1. Anaïs, B., Bernard, A.: Manifesto Emersivo. Nascita delle Arti Immersive, P.L. Capucci, F. Torriani (a cura di). Ravenna, Noema (2018) 2. Ausubel, D.: In defense of advance organizers—a reply to the critics. Rev. Educ. Res. 48, 251–257 (1978) 3. Berthoz, A.: La Simplexité, Odile Jacob, Paris (2009) 4. Canevacci, M.: Sincretika. Esplorazioni etnografiche sulle arti contemporanee. Roma, Bonanno (2014) 5. Cipolletta, G.: Passages metrocorporei. Per un’estetica della transizione. Macerata, eum (2014) 6. Cipolletta, G.: Intervista | Luigi Pagliarini. Pt I. Arte Corale (2020a). https://www.arshake.com/ intervista-luigi-pagliarini-arte-corale-pt-i/ 7. Cipolletta, G.: Intervista | Luigi Pagliarini. Pt II. Arte corale (2020b). https://www.arshake. com/intervista-luigi-pagliarini-arte-corale-pt-2/ 8. De Certeau, M.: The Practice of Everyday Life. University of California Press, California (1984) 9. Eco, U.: La struttura assente. Milano, Bompiani (1996) 10. Floridi, L.: La quarta rivoluzione. Come l’infosfera sta trasformando il mondo. Milano, Raffello Cortina (2017) 11. Gabbari, M., Gagliardi R., Gaetano A., Sacchi D.: Comunicazione e apprendimento “aumentati” in classe—Fare lezione a scuola con la realtà aumentata. In: Bricks. Rivista online per promuovere l’innovazione nella scuola, n. 1 (2017). http://www.rivistabricks.it/ 12. Gardner, H.: Frames of Mind: the Theory of Multiple Intelligences. Basic Books, NYC (1983) 13. Giovanella, C., Passarelli, M., Persico, N.: The Effects of the Covid-19 Pandemic on Italian Learning Ecosystems—the School Teachers’ Perspective at the steady state (2020). https://doi. org/10.13140/RG.2.2.31552.76809 14. Goffman, E.: Behavior in Public Places. In: Simon, Schuster. (eds.), New York (1985)

72

G. Cipolletta

15. Goffman, E.: Interaction Ritual: essays on Face-to-Face Behavior. Anchor Books, New York (1967) 16. Goffman, E.: The nature of deference and demeanor. Am. Anthropol. (New Seri) 58(3), 473– 502. http://www.jstor.org/stable/665279 17. Granata, P.: Ecologia dei media. Protagonisti, scuole, concetti chiave. Milano, Franco Angeli (2015) 18. Iaconesi, S.: Il primo giorno di una nuova scuola, il benvenuto di Salvatore Iaconesi (2020). https://www.che-fare.com/il-primo-giorno-di-una-nuova-scuola-il-benvenuto-di-sal vatore-iaconesi/ 19. Hall, E.T.: The Silent Language. Fawcett, New York (1966) 20. Maglioni, B.: La classe capovolta. Innovare la didattica con la flipped classroom. Trento, Erickson (2014) 21. Manzoni, M.R.: La CROSSLESSON: un modello didattico tra realtà virtuale e gamification. https://sd2.itd.cnr.it/?r=site/approfondimenti (2020) 22. Maragliano, R.: Zona franca. Per una scuola inclusiva del digitale. Roma, Armando Editore (2019) 23. McLuhan, M.: Understanding media: the extensions of man. McGraw-Hill Education, NYC (1964) 24. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum.In: Proceedings of Telemanipulator and Telepresence Technologies, SPIE, vol. 2351 (1994) 25. Postman, N. (2019). Ecologia dei media. La scuola come contropotere. Roma: Armando Editore 26. Singer, M.: Critical Medical Anthropology. In Encyclopedia of Medical Anthropology—health and Illness in the World’s Cultures. In: Ember, C., Ember, M. (eds). Vol. 1, pp. 23–30. Kluwer, New York (2004) 27. Strayer, JF.: How learning in an inverted classroom influences cooperation, innovation and task orientation. Learn. Environ. Res. 15(2), 171–193 28. Sutherland, I.: A head-mounted three dimensional display in AFIPS ’68 (Fall, part I) 9 December (1968) 29. Trentin, G.: Spazi ibridi di insegnamento-apprendimento per una didattica “always-on” (2015). https://www.researchgate.net/publication/281648096 30. Trione, V.: La rivoluzione dei musei digitali. In: la Lettura. Corriere della Sera #474 (2020) 31. Van Hoof, E.: Lockdown is the world’s biggest psychological experiment—and we will pay the price in. (2020). https://www.elkevanhoof.com/en/author/elke/(last access 13rd January 2021)

Webography Pandemic Literature 32. http://www.nbst.it/822-pandemia-covid-19-%C3%A8-anche-sindemia-disuguaglianze.html 33. https://www.thelancet.com/series/syndemics 34. https://www.scientificamerican.com/article/the-biggest-psychological-experiment-in-historyis-running-now1/ 35. Leonori, D.: TEDX Ascoli 2020 36. https://www.youtube.com/watch?v=_P7KIwLw06M&fbclid=IwAR2H1jHdPkQPSttkFo-JpO 8GYquFvFcTMYdYqhkxgdjmVrxYun8McGioT30

4 Remix Viralia. For a Reality of the Imaginary Re-imagined …

73

School Tools 37. 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. 60.

https://classmill.com/ https://new.edmodo.com/ https://www.fidenia.com/ https://www.weschool.com/ https://library.weschool.com/ https://classroom.google.com/u/0/ https://meet.google.com/ https://zoom.us/it-it/meetings.html https://otus.com https://www.microsoft.com/it-it/education/products/office. https://www.schoology.com https://www.socloo.org/ https://editionsatplay.withgoogle.com/#!/ https://www.arbibook.com/?lang=en http://aurasmaproject.weebly.com/getting-started.htm https://studio.gometa.i https://assemblrworld.com/ https://mixedreality.mozilla.org/firefox-reality/ https://hubs.mozilla.com https://www.oculus.com/rift https://www.vive.com/us/ https://arvr.google.com/daydream/ https://arvr.google.com/intl/it_it/cardboard/get-cardboard/ https://edu.google.com/intl/it_it/ https://artsandculture.google.com/ https://crosslesson.com/

Museum Tools 61. 62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77. 78. 79.

https://www.louvre.fr/en/visites-en-ligne https://www.moma.org/calendar/groups/58 https://www.metmuseum.org/art/online-features/met-360-project https://www.rijksmuseum.nl/en/from-home https://naturalhistory.si.edu/visit/virtual-tour https://ivrpa.org/panorama/vienna-state-opera/ https://www.kunstihoone.ee/en/ https://zacheta.art.pl/en http://www.bangabandhusbangladesh.ca/virtual-museum https://www.visitsingapore.com/editorials/covid19-guide/explore-museums-at-home/ http://data.moc.gov.bh/vt/VR/nationail_museum/bnm.html. https://joyofmuseums.com/mus eums/americas-museums/brazil-museums/sao-paulo-museums/sao-paulo-museum-of-art/ http://virtual.vizen.cn/ https://www.youtube.com/user/Kunstsenteret https://www.youtube.com/channel/UCkB2xFSR7r8iu6nbUdP0EZQ https://soundcloud.com/gamec_bergamo https://www.nma.gov.au/visit-us https://www.nma.gov.au/whats-on/tours-and-experiences/virtual-gallery-tours https://zeitzmocaa.museum/ https://www.montereybayaquarium.org/for-educators/learning-at-home/crafts-and-printables/ be-a-sea-searcher

74 80. 81. 82. 83.

G. Cipolletta

https://www.clevelandart.org/categories/building-project https://www.tiktok.com/@uffizigalleries?lang=it https://www.mooc.org/ https://www.moma.org/research-and-learning/classe. https://www.centrepompidou.fr/en/thecentre-pompidou-school/different-ways-of-learning-and-training 84. https://www.abamc.it/mostre-e-eventi/item/restacorale-restatrascendente 85. https://www.collinsdictionary.com/it/woty (last access 13rd January 2021)

Chapter 5

Leveraging Immersive Technologies During the COVID-19 Pandemic—Opportunities and Challenges Anitha S. Pillai, Radhika Sunil, and Giuliana Guazzaroni Abstract Life has changed dramatically since the COVID-19 virus has been discovered all over the world. There were lockdowns and travel restrictions, and as a result, many activities could not be carried out in the usual way. Students and employees were unable to attend schools, colleges, offices and citizens were not able to visit stores, museums, art galleries, or other tourist attractions for fear of contracting an infection. Due to the advancements in immersive technology, many activities could easily be carried out while sitting in the comfort of one’s own home. Customers could choose their preferred brands while sitting at home, tourism enthusiasts could go on virtual tours, and students could receive training/studies using virtual and augmented reality. Organizations could allow employees to Work From Home (WFH) and periodically conduct discussions and meetings using these technologies. Many conferences, seminars, and events could be conducted successfully using Virtual and augmented reality. This chapter provides an overview of how various sectors harnessed the power of this immersive technology in moving their work/ business forward. Keywords Virtual reality · Augmented reality · Mixed reality · Extended reality · Healthcare · Immersive technology · COVID-19 · Pandemic

A. S. Pillai (B) Professor, Hindustan Institute of Technology and Science, Chennai, India e-mail: [email protected] R. Sunil Politecnico Di Milano, Milan, Italy e-mail: [email protected] G. Guazzaroni Ministero dell’Istruzione, Kingdom of Italy, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_5

75

76

A. S. Pillai et al.

5.1 Introduction 5.1.1 Covid-19 A novel coronavirus (CoV) which was named as ‘2019-nCoV ’ or ‘2019 novel coronavirus’ or ‘COVID-19’ by the World Health Organization (WHO) that began at the beginning of December 2019 in Wuhan City, Hubei Province, China [1–4]. COVID-19 is an infectious disease caused by a newly discovered coronavirus and people infected with this virus experience mild to severe respiratory illness and some recover without requiring special treatment [5] whereas others required hospitalization and some people even succumbed to the disease too. To prevent the virus from spreading, various government agencies around the world have imposed strict lockdowns to prevent people from moving around and becoming infected. Some of the guidelines issued by WHO to prevent the spread of the virus include physical separation, wearing a mask, keeping rooms well ventilated, avoiding crowds, cleaning your hands, and coughing into a bent elbow or tissue [6].

5.1.2 Virtual, Augmented and Mixed Reality The terms "Virtual Reality" and "reality emulation" are derived from the words "Virtual" and "Reality." When the VR headset is worn, it gives the user the impression that they are somewhere else, and the panels inside are refracted by the lenses, filling the field of vision with whatever is being displayed, and visually, users are transported to wherever the headset wants them to go, and the outside world is replaced with a virtual one. Though Virtual reality technology existed for quite some time, it’s value and significance to various magnitudes of everyday life emerged under lockdown following the COVID-19 pandemic [7]. “In a survey conducted in 2020 in the US, 71 percent of the people responded that they spent more time using virtual reality during the COVID-19 pandemic” [6]. Virtual Reality replaces user’s vision whereas Augmented Reality devices are transparent, permitting to see everything in front as if the user is wearing a weak pair of sunglasses and improved version of the real physical world is accomplished through the use of digital visual elements, sound, or sensory stimuli. In Mixed Reality, using sensing and imaging technologies, interaction and manipulation of both physical and virtual environments are obtained. Extended Reality is an emerging term which consist of Augmented reality, Virtual reality, and Mixed reality plus those that are still to be created. Immersive technologies extend the reality by either blending the virtual and “real” worlds or by creating a fully immersive experience to the user [8].

5 Leveraging Immersive Technologies During the COVID-19 …

77

Fig. 5.1 Virtual Reality usage During the Pandemic

5.2 Uses of Immersive Technology During the Pandemic The COVID-19 pandemic has significantly altered our lives in a very short amount of time. Some of the repercussions of the virus include social isolation, travel limitations, school/college/organization closures, an increased need for medical supplies, and hospitalisation [9]. COVID-19 has led to remote working and the use of Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), and Extended Reality (XR) at home. One of the benefits of the current Covid-19 epidemic is that it lets everyone work from various locations, increasing social distance and reducing travel [10].

78

A. S. Pillai et al.

During the epidemic, VR and AR enabled users to feel closer to one another without violating COVID standards, and they also aided in remote working. Employees were able to communicate with their co-workers’, and carry on their work. Friends, and family get-togethers were also feasible with the use of these technologies. During the lockdown, these technologies were used to carry out everyday tasks in healthcare, education, tourism, and media. In reality, these technologies aided us in broadening our horizons. Though AR and VR technologies have been around for a long time, they have grown in popularity in the recent years as they have become the need of the hour. During pandemic, when lockdown is enforced with movement restrictions, various industries including healthcare have begun to use these technologies to carry out their work. The epidemic has also taught us that many sectors are still being able to carry out their work and progress with the assistance of these new technologies. AR and VR have the ability of immersing the user in a virtual world, offering them a real-time experience. When it comes to purchasing groceries or clothes or any other items, everyone wants to go to the store, try on items/clothes, and buy what looks well on them. But what if we could do all of this from the comfort of our own homes, using XR technology to get a personal experience of it all? Nowadays, an increasing number of firms are attempting to investigate various methods to employ this technology to improve their business while also providing users with a real-time experience. For example, the education industry is experimenting with numerous methods to make the teaching–learning process more engaging and student friendly. Traditional marketing methods such as site visit and drone photography are not practical in the current situation due to different regulatory constraints. AR/VR allows real estate companies to promote continuing projects even before they are completed by presenting the interior and exterior of unfinished homes and allowing prospective buyers to experience the entire space without having to travel to the project location. Without in-person showings, AR/VR enabled real estate brokers and landlords to provide property tours to prospective purchasers by requiring only a little investment in these technologies, allowing them to continue operating without interruption. The sections that follow will describe how these technologies were used by various organizations during the pandemic. Many organisations, including education and healthcare, have adopted these technologies in order to continue their job.

5 Leveraging Immersive Technologies During the COVID-19 …

79

Art & Workout

Museum

MarkeƟng

EducaƟon

Cconferences

ApplicaƟons

Manufacturing

Retail

Medicine

Sports

Tourism

& Gaming

ApplicaƟons of immersive Technologies

5.2.1 Marketing Using VR and AR Organizations and brands have replaced the traditional marketing methods and sales touchpoints with individualised, immersive experiences for consumer isolation [11]. With the introduction of COVID-19, the actual world became a much less accessible place, and VR/AR/XR became a smarter and more successful marketing device for organisations across a wide range of industries [12]. Photography and virtual tours have become the most important marketing strategies for selling real estate [11]. Matterplot, the most popular web-based online software for virtual tours, was used by real estate brokers for 3D virtual tours of residential complexes [13]. This pandemic has taught us all that change and innovation are necessary for survival. “There has been an increase in various AR applications with virtual “try-before-youbuy” experiences ranging from presenting furniture and products in your house with daily businesses like IKEA and Home Depot, to digitally trying on premium fashion labels like Louis Vuitton and Gucci” [14].

80

A. S. Pillai et al.

5.2.2 Conferences, Meetings and Planning of Events Many of the conferences and, workshops turned virtual in the year 2020 as they did not want them to be cancelled or postponed. Most of the events made use of AR/VR/MR to make them livelier and more interesting [11]. The advantage was that the participants could attend conferences, seminars, and meetings without leaving their home and also get a complete 360° view of the event. Also, some of the sessions witnessed more participants with the usage of these technologies. Many specialists and audiences could gather using AR/VR methods without incurring the costs of travel, meals, and lodging [11]. “According to Christoph Fleischmann, the founder of Arthur Technologies, most of the organizations brought in VR technologies not only to provide virtual office spaces for teams to meet but also manage work, and these technologies also provide employees with a way to share ideas, concepts, and data during a global pandemic” [15]. Some situations in hospitals required suggestions and opinions from doctors, so for this distant meetings were conducted using immersive VR. This was performed by presenting a clinical case of a patient with heart illness and the authors discovered that the user experience was good [16]. Accenture created an interactive, immersive, and collaborative AR, MR, and VR solution known as Accenture XR Event Planner in collaboration with Qualcomm, Intercontinental Hotel Group (IHG), and InterContinental Los Angeles Downtown that allowed event planners, buyers, and hotel sales staff to remotely visualise, customise, and move through event spaces, significantly improving the way hotels sell meeting space while supporting sales of associated hotel rooms, food and beverages [17]. Because the pandemic has made in-person meetings and conferences impossible, they are now held in 3D utilising VR headsets. According to Ferhan Ozkan, co-founder with Rahel Demant of VR First and XR Bootcamp, a Berlinbased network whose purpose is to assist institutions create XR capabilities [18], the coronavirus epidemic will speed the advent of XR.

5.2.3 Retail Different stores are utilising augmented reality to aid clients in digitally evaluating many cosmetic goods so that they can make better judgments. Customers use Kohl’s AR Virtual Closet to go to the dressing room, mix and match clothes, and buy without leaving their home [14]. Furniture retailers such as Wayfair and Ikea developed the ability, through their websites and applications to virtually place a piece of furniture in a customer’s house to see how it would fit with the current décor without having to visit the store [12]. This is especially useful during the Covid-19 period, when travel is restricted to avoid rush/crowd in stores. “According to IBM’s 2020 U.S. Retail Index report, the pandemic has accelerated the transition to digital commerce by approximately five years” [14]. Kendra Scott’s

5 Leveraging Immersive Technologies During the COVID-19 …

81

boutiques harnessed the capabilities of AR technology by allowing clients to virtually test numerous earring styles, and purchase them immediately using an iPhone with the Safari web browser [14]. The COVID-19 pandemic has produced various disruptions forcing businesses to adapt to it and merchants to respond to the problem quickly demonstrating a variety of intervention types [19]. Gucci, an Italian luxury fashion house, allows users to virtually try on shoes and see how they look before purchasing. The AR lenses on Snapchat is used by beauty brands such as Louis Vuitton and L’Oreal to create immersive digital experiences [20]. Shopify uses VR and AR to allow its consumers to interact with 3D models of things products they want to buy by rotating them, focusing in on their details, and even to envision them in their house using their smartphone camera [7]. COVID-19 has pushed the use of AR in eateries. Every table in a local Mexican restaurant has a QR code that could be scanned to see the menu. Seek has integrated QR codes into AR systems for companies like Walmart and Overstock which saw a 600 percent increase in AR usage on its clients’ websites [21]. Sephora and Ulta, for example, do not allow their customers to physically try makeup products on their skin and instead employ AR to enable users to digitally test beauty goods before making a purchase choice. Levi’s, a company that is utilising an AR retail approach with digital tools such as Squad, an online co-watching video programme where friends can shop together, was established as a way to reconstruct some of the social experiences people missed and yearned for during the pandemic. Burberry collaborated with Snapchat on an in-store AR game that allows users to play, explore, and shop with their friends [14]. Virtual try-on experiences offer significant prospects for AR in retail by allowing customers to preview products digitally in their own homes or on their own bodies, and then purchase them immediately. This pandemic can be seen as a catalyst for this digital change, as immersive technologies have shown to bring great value for consumers while shopping. These virtual shopping experiences could be used by business organisations and prominent brands in the future.

5.2.4 Tourism A lot of tourism organisations have begun to use VR in marketing as well as in the creation of touristic experiences. The German National Tourist Board (GNTB) has recently taken viewers on tours around the country as well as areas of Baltic and North Sea beaches [22]. "The world’s first VR travel and search booking experience was implemented by Amadeus” [22]. This helps travellers to book tours and do the payment using VR. The pandemic had a significant impact on the global leisure and tourism industry, with international travel bans affecting over 90% of the world’s population, resulting in a projected 30% drop in international tourism [23]. Because VR can provide a real-life travel experience, it can be used for virtual travel/tours when actual travel is

82

A. S. Pillai et al.

Fig. 5.2 Immersion room

banned. According to the authors [23] who conducted a survey with 193 participants, travellers use VR-enabled tours as a substitute for an actual trip during and even after the pandemic. Using VR tours during the pandemic posed no danger. Even after COVID-19, there will be a high demand for AR to not only assure tourist safety but also to offer exclusive, personalised, context-specific, profound, and unforgettable experiences [24]. By connecting individuals to nature and to other parts of the world, AVARA Media is able to create immersive, 3D visual experiences. People travel to various regions for live experiences, but now that these excursions are no longer possible, immersive technology systems can deliver the sensory experience without having to travel to that site. Students are also not authorised to go on educational excursions or study trips during this time because it is extremely dangerous, but however, these trips are critical for students to gain hands-on experience or a comprehensive understanding of the operation of various devices or equipments related to their subject of study.

5.2.5 Sports and Gaming During the pandemic, virtual sports platforms enabled athletes to compete in numerous sports competitions organised across the world from their homes, avoiding financial losses caused by the cancellation of many sports activities. Novel applications such as ZWIFT enabled athletes to participate in sports without making physical contact, reducing the propagation of the virus and providing athletes with digital exercise alternatives [25]. These platforms can also be used to train players, which improves both the player’s and the trainer’s safety. Players must train on a regular basis and the only way to do it now is to do it at home using modern technology, so that they do not lose out on their regular training and practices. People had to stay

5 Leveraging Immersive Technologies During the COVID-19 …

83

indoors due to the epidemic, which boosted VR technology to develop immersive gaming worlds for player’s entertainment [6]. With Everest VR, VR can take you from the mundane terrain of your living room to visually interesting settings such as Mount Everest [7].

5.2.6 Medicine Pune-based DesignTech Systems created an AR-based smartphone App that takes users through the process of setting up an oxygen cylinder at home, which is available on both iOS and Android platforms [26]. Due to the pandemic, real meetings were not possible and there was a need for VR/AR enabled technology to hold virtual meetings for the benefit of patients and physicians [16]. In the healthcare industry, XR technology helped to reduce costs while also improving treatment outcomes, particularly when patients were unable to travel. VR aids in the prevention and treatment of stress and anxiety, mainly during COVID [27]. To raise awareness of the problems of stress and anxiety among healthcare workers, the project website (https://mind-vr. com/) offers a free VR application called MIND-VR, which is available in both Italian and English [28]. The COVID-19 pandemic has shown us that further investments and technological interventions are needed in the healthcare industry, especially in XR technology, which can assist to reduce operating costs while providing better diagnosis and patient-doctor communication [29]. Imperial College London collaborated with Microsoft to use the HoloLens 2 MR headset to visit the wards during COVID to reduce exposure to patients where one physician wears the HoloLens 2 into a patient room while the rest of the students join and learn remotely via a Microsoft Teams meeting [30]. VR also provides a peaceful death for COVID-19 patients by simulating a physical world that allows them to experience a place they wanted to visit and can be used to capture all of this for their family to remember their last memorable moments [31]. AR is also a useful tool in the COVID-19 epidemic since it may aid in the depiction of invisible concepts, explanation through virtual world navigation, and storytelling for training reasons. XRHealth created a telemedicine system that may be used throughout the lockdown and the COVID-19 quarantine for post-stroke rehabilitation and brain injury. The benefit of this approach is that patients can control their tension and anxiety while still engaging in physical activity and cognitive activities and having the opportunity to communicate with individuals who have different goals [32, 33]. To the impact of SARS-CoV-2, medical personnel at George Washington University Hospital employed VR to examine the patient’s lungs [34]. Researchers were immersed in the VR environment’s "Protein World" to visualise the details of the SARS-CoV-2 in order to uncover prospective molecular targets for the inhibition of SARS-CoV-2 proteins which aided in drug discovery [35]. During isolation ward visits, doctors donned head-mounted displays and used AR to converse and connect with many other doctors at the same time. Using high-resolution audio and video, patients’ daily recordings could be relayed to the medical system completely

84

A. S. Pillai et al.

contactless [36]. Snap’s AR lens gave information on how monetary donations can aid during the COVID-19 pandemic, making individuals realise how they can play a beneficial role in giving resources while also emphasising on WHO’s statements about the value of donations in pandemic [37]. VR and AR are assisting the healthcare industry in areas such as education, diagnosis, therapy, and even medication development. Many experiments have been conducted during the COVID-19 pandemic to demonstrate the capacity of VR and AR in telemedicine and telehealth technology. The use of VR and AR in telehealth can help patients and healthcare providers deal with the harmful effects of epidemic by providing healthcare assistance remotely. The benefits of these technologies are that they reduce the need for face-to-face contact during the pandemic. This would aid in minimising the transmission of the virus and improving safety for both the general public and frontline personnel. As a result, we are able to fully appreciate the potential of immersed technology to create optimal utilization, particularly during the current situation.

5.2.7 Manufacturing When technology in the manufacturing business changes, it is critical to train the staff involved in order for them to gain the necessary skills or knowledge. Considering the government’s safety standards, conducting in-house training is not possible during this time period. As a result, several companies are utilising immersive technologies to train their personnel in order for them to have a better understanding of the new production process. The production of goods cannot be halted due to lack of available labour. Studies have also indicated that training with immersive technologies improves memory and retention of information when compared to standard training techniques such as video or written text instructions in the form of a book or pamphlet [38]. Strivr, a company, offers modules for training employees to do everything from treating customers with empathy to handling difficult situations. Walmart is one of the largest corporations that uses Strivr for all of their trainings, with over 17,000 headsets supplied to its staff [39]. VR is also widely used to train pilots and employees, particularly in the automotive and industrial industries who work with 3D models, car assembly, and prototype creation. To train a personnel, all details can be displayed via VR/AR technology where the trainer can observe what the learner is doing and provide remote assistance if there are any issues.

5.2.8 Education Online classes in schools and colleges were introduced by educational institutions. Though the students missed the excitement of meeting classmates and professors in their classes, the utilisation of VR and AR ensured that they did not miss their

5 Leveraging Immersive Technologies During the COVID-19 …

85

instructional hours or learning. Would-be freshmen preferred to visit colleges before deciding which one to attend and virtual tours of campuses aided this process. YOUVisit is one such website that offers virtual reality tours of numerous college campuses in the United States, allowing students to immerse themselves in the place where they will be spending the next three to four years [7]. Although this technology existed prior to the epidemic, it became more valuable during the pandemic period due to travel restrictions. AR-Innovative educational strategies are helping students to interact more deeply with their assignments. Terminal Eleven’s SkyView provides students with a map of constellations that they can view on their phones. Medical education was no exception, with AR models being used to instruct and support surgeons and nurses in numerous medical operations. “The healthcare sector is an ideal use case for both augmented reality and virtual reality in training, teaching, and treatment” [40]. The benefit of adopting VR for medical instruction is that it lessens the tension that a student feels when confronted with a challenging case for the first time, and a bad decision can even shatter the student’s confidence. VR assists in simulating a comparable situation and medical students can be prepared for such perilous situations ahead of time without any fear of repercussions.

5.2.9 Workout Within, a virtual reality studio, has created Supernatural, a VR app for fitness enthusiasts that offers tailored full-body exercises and guidance from actual trainers as well as visits to gorgeous photorealistic landscapes during the workout. The software allows users to track their progress and schedule workouts. When the gyms were closed, this app was helpful.

5.3 Challenges Though it is clear that VR/AR/MR/XR are widely employed by various sectors, including education, healthcare, sports, and tourism, to carry out their work because of pandemic limits, it is also clear that it cannot replace human touch. Students undoubtedly prefer to meet and communicate with their classmates and teachers in person rather than online media. Similarly, a patient will feel more at ease sitting and discussing his or her difficulties with the doctor. All of these internet sessions cannot replace face-to-face meetings. But using these technologies and wearing headsets for an extended period of time might be hazardous to one’s health, particularly the eyes. It may also create stress, anxiety, and nausea. When learners use XR equipments by themselves, they may get injured by stumbling over real-world objects [41]. But, rather than waiting for the pandemic to stop and everything to return to normal, we had to rely on these technologies, although we often miss true human touch and personal

86

A. S. Pillai et al.

interaction. Implementing this immersive technology necessitates expenditures on the part of the organisation and is a costly endeavour. Though XR training is realistic, it does not give the same atmosphere as training and working in the actual world, where aspects such as psychological state may be considered [41]. One of the primary concerns in AR/VR systems is privacy [42]. As XR technologies generate a huge amount of personal information, a proper privacy framework and ethical rules are essential with guidelines describing what must be protected [24, 30]. Because XR-related solutions have access to a large amount of private information, they are vulnerable to cyber-attacks and data hacks which can cause significant damage [43].

References 1. Hui, D.S., et al.: The continuing 2019-nCoV epidemic threat of novel coronaviruses to global health—The latest 2019 novel coronavirus outbreak in Wuhan. China. Int. J. Infect. Dis. 91, 264–266 (2020). https://doi.org/10.1016/j.ijid.2020.01.009.[PMCfreearticle][PubMed ][CrossRef][GoogleScholar] 2. Read, J.M., Bridgen, J.R., Cummings, D.A., Ho, A., Jewell, C.P.: Novel coronavirus 2019nCoV: early estimation of epidemiological parameters and epidemic predictions. medRxiv (2020) 3. Corman, V.M., et al.: Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Eurosurveillance 25(3) (2020), [PMC free article] [PubMed] 4. Fong, S.J., Li, G., Dey, N., Crespo, R.G., Herrera-Viedma, E.: Composite monte carlo decision making under high uncertainty of novel coronavirus epidemic using hybridized deep learning and fuzzy rule induction (2020). arXiv:2003.09868 [PMC free article] [PubMed] 5. https://www.who.int/health-topics/coronavirus#tab=tab_1 6. https://www.statista.com/statistics/1178715/coronavirus-impact-vr-usage/ 7. https://www.voices.com/blog/virtual-reality-pandemic/ 8. https://www.forbes.com/sites/bernardmarr/2019/08/12/what-is-extended-reality-technologya-simple-explanation-for-anyone/?sh=6b811a5d7249 9. Nicola, M., Alsafi, Z., Sohrabi, C., Kerwan, A., Al-Jabir, A., Iosifidis, C., Agha, M., Agha, R.: The socio-economic implications of the coronavirus pandemic (COVID-19): a review. Int. J. Surg. (London, England) 78, 185–193 (2020). https://doi.org/10.1016/j.ijsu.2020.04.018 10. https://blog.ifs.com/2020/08/taking-our-meetings-into-virtual-reality/ 11. https://www.cxotoday.com/corner-office/why-ar-vr-will-change-post-covid-marketing/ 12. https://zenmedia.com/blog/virtual-reality-marketing-in-the-age-of-covid/ 13. https://doi.org/10.2991/assehr.k.201202.079 14. https://hbr.org/2020/10/how-ar-is-redefining-retail-in-the-pandemic 15. https://venturebeat.com/2021/01/28/virtual-reality-and-simulation-see-an-uptick-during-thepandemic/ 16. Sadeghi, A.H., Wahadat, A.R., Dereci, A., et al.: Remote multidisciplinary heart team meetings in immersive virtual reality: a first experience during the COVID-19 pandemic. BMJ Innov. 7, 311–315 (2021) 17. https://www.accenture.com/us-en/insights/technology/xr-event-planner 18. https://www.dw.com/en/your-next-business-meeting-could-be-inside-a-vr-headset/a-531 43501 19. Pantano, E., Pizzi, G., Scarpi, D., Dennis, C.: Competing during a pandemic? Retailers’ ups and downs during the COVID-19 outbreak. J. Bus. Res. 116, 209–213 (2020). ISSN 0148-2963. https://doi.org/10.1016/j.jbusres.2020.05.036

5 Leveraging Immersive Technologies During the COVID-19 …

87

20. https://indianexpress.com/article/technology/tech-news-technology/how-brands-are-using-arto-reach-consumers-during-the-pandemic-7159578/ 21. https://risnews.com/could-covid-19-accelerate-augmented-reality-adoption-retailers 22. https://www.hospitalitynet.org/news/4103252.html 23. Sarkady, D., Neuburger, L., Egger, R.: Virtual reality as a travel substitution tool during COVID-19. In: Wörndl, W., Koo, C., Stienmetz, J.L. (eds.) Information and Communication Technologies in Tourism. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-657857_44 24. Mohanty, P., Hassan, A., Ekis, E.: Augmented reality for relaunching tourism post-COVID19: socially distant, virtually connected. Worldw. Hosp. Tour. Them. 12(6), 753–760 (2020). https://doi.org/10.1108/WHATT-07-2020-0073 25. Westmattelmann, D., Grotenhermen, J.G., Sprenger, M., Schewe, G.: The show must go on— Virtualisation of sport events during the COVID-19 pandemic 2020. Eur. J. Inf. Syst. https:// doi.org/10.1080/0960085x.2020.1850186 26. https://indianexpress.com/article/technology/tech-news-technology/pune-company-developsar-based-app-that-teaches-people-how-to-set-up-oxygen-cylinder-at-home-7326634/ 27. Imperatori, C., Dakanalis, A., Farina, B., Pallavicini, F., Colmegna, F., Mantovani, F., et al.: Global storm of stress-related psychopathological symptoms: a brief overview on the usefulness of virtual reality in facing the mental health impact of COVID-19. Cyberpsychol. Behav. Soc. Netw. 23, 782–788 (2020). https://doi.org/10.1089/cyber.2020.0339 28. https://www.frontiersin.org/articles/10.3389/frvir.2021.620225/full 29. https://augray.com/blog/applications-of-extended-reality/ 30. https://www.insidehighered.com/blogs/learning-innovation/5-reasons-we-are-hopeful-aboutfuture-extended-reality-learning 31. https://www.sciencedirect.com/science/article/pii/S2352914821000691 32. https://hitconsultant.net/2020/03/16/xrhealth-launches-virtual-reality-telehealth-supports-gro ups/#.YLnTwagzY-d 33. XRHEALTH.: Common conditions, extraordinary care. https://www.xr.health/ 34. https://www.gwhospital.com/resources/podcasts/covid19-vr-technology 35. https://northernontario.ctvnews.ca/university-researcher-uses-virtual-reality-to-fight-covid19-1.4874535 36. https://www.metrocomms.co.uk/augmented-reality-for-health-professionals/ 37. https://unfoundation.org/blog/post/new-snapchat-lens-brings-covid-19-donations-to-life/ 38. https://www.fierceelectronics.com/electronics/extended-reality-seeing-pickup-from-pan demic-including-growing-interest-industrial 39. https://www.fastcompany.com/90457929/virtual-augmented-reality-most-innovative-compan ies-2020 40. https://www.signiant.com/resources/tech-articles/vr-ar-covid-19-experts/ 41. https://trainingindustry.com/articles/learning-technologies/pros-and-cons-of-using-extendedreality-in-employee-training/ 42. https://www.cnbc.com/2020/07/04/virtual-reality-usage-booms-in-the-workplace-amid-thepandemic.html 43. https://www.sam-solutions.com/blog/what-is-extended-reality-and-what-can-we-do-with-it/

Chapter 6

Virtual and Augmented Reality in Art During the Pandemic Giuliana Guazzaroni

Abstract COVID-19 pandemic brought the whole world to a stand still. One of the biggest issues was a restriction in travel to reduce the spread of the virus. Everybody was forced to stay home and as a result visits to art galleries, museums were all prohibited. Virtual Reality (VR) and Augmented Reality (AR) were used to conduct virtual tours to art galleries and museums even before the pandemic but not to the extent it is being used today. If these immersive technologies were not put into use then the virtual art exhibition nor the virtual tour of museums would have never been a reality. This chapter highlights the use of these technologies to promote art during this pandemic. This situation provided us lots of learning experiences to exploit the maximum capabilities of VR and AR to enhance the virtual tour of Art lovers. Though some people felt that the joy and happiness of visiting such galleries cannot be replaced by virtual tours, there was no other choice to reduce the spread of the virus as well as make them available artifact visible to the public. Keywords Virtual reality · Augmented reality · Mixed reality · Extended reality · Art · Museum

6.1 Introduction The number of smartphone users is rising steadily and now during these COVID times, mobile technology has become a vital part of our day-to-day life. With the advancement in Mobile technology many users are able to benefit from the advantages of Virtual Reality (VR) and Augmented Reality (AR) during this difficult time. It is quite interesting to note that VR and AR have opened up doors to the arts also. Combining AR with art opens up exciting and interesting opportunities for Art lovers and AR enthusiasts. Especially during this pandemic time when movements are restricted art would have just confined to a gallery if not for the advancement in immersive technology. These technologies not just improve visitors’ experience but G. Guazzaroni (B) Rome, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_6

89

90

G. Guazzaroni

also ensures that art reaches a wider audience enabling people to enjoy them with their family sitting at home. Art fairs, like so many other events have been grimly affected by the COVID-19 pandemic. As a result many of them moved Online and paved way for the digital transformation of the art trade. Though Extended Reality (XR) technologies have vast prospective to create deeply immersive and engaging experiences for audiences, their impact on the art industry and elsewhere outside the gaming world has been toughened to date by the poor penetration of VR headsets into the consumer market [1].

6.2 Art and Museums The COVID-19 emergency has created an unprecedented use of digital technologies. Teaching, learning, exams, meetings, seminars, museum visits, consulting doctors take place online using various technologies from simply textual ones, to more complex and immersive technologies capable of generating experiences that simulate face-to-face interactions and more. Natural Language Processing (NLP), Virtual Reality (VR), Augmented Reality (AR), Artificial Intelligence (AI), Intelligent Virtual Environment (IVE) are just some of the technologies that have seen an increase in users in 2020, a year certainly characterized by social distancing and the impossibility of travelling or visiting museums due to the global health emergency. Natural Language Processing (NLP) is one of the best known artificial intelligence techniques and consists in the understanding and processing of natural language. NLP research is producing quite satisfactory results within the reach of millions of citizens. These are intelligent conversation systems that can simulate human dialogue. Amazon’s Alexa and Apple’s Siri are capable not only of imitating language, but also of answering questions on different topics and performing complex tasks, such as planning a visit to a museum. Chatbots or chatterbots are software designed to simulate a conversation with a human being. They are artificial intelligence engines, intelligent agents, able to give answers, joke or at least show interest and empathy towards the user. People can try conversations with existing chatbots, Siri or Alexa or try, for example, Replika, Mitsuku, Rose, Zeve, etc. Users can also write a dialogue or create scripts for non-linear and interactive narratives, with Twine. Born in the field of customer services, intelligent chatbots are increasingly used to talk to "a friend" in moments of loneliness, crisis or difficulty. A chatbot works using machine learning and is equipped with a sort of “artificial brain”. Artificial intelligence (AI) understands the language of the person it interfaces with and becomes more and more intelligent as it gains experience, because it learns from the conversations it has with people. Different psychological models of behavior and different types of personalities are used to create a chatbot engine. The bot becomes recognizable for the user, distinguishes itself from other bots and is perceived as a real person or almost. Replika, for example, is a chatbot capable of feeling empathy thanks to the deep learning sequence-to-sequence that uses neural networks and transcripts of speeches between real people to learn to think and speak. By applying this model, Kuyda, the

6 Virtual and Augmented Reality in Art During the Pandemic

91

architect of Replika, has managed to create a voice assistant with remarkable listening skills. This application has been downloaded, in a Vancouver Mural Festival (VMF), whose new event, VMF Winter Arts, featured 24 AR-driven artworks in 17 locations throughout the Canadian city and residents could download a custom mobile app to find a map of the spots and a 3D digital sculpture to popped up when they scanned the QR code [2]. The artists used Spark AR to execute their visions. Attendees were able to use a custom mobile app to view the artwork, which includes artist Saida Saetgar’s “Journey into a Dreamland”. The Metropolitan Museum of Art created virtual tour experience for art lovers during the pandemic by making use of Met Unframed which is a new virtual art adventure which made use of AR to present paintings from museums into the smart phone of the users. Viewers were able to see 50 works of art from artists like Vincent Van Gogh, Jackson Pollock, and Margareta Haverman [3]. Museums such as the Louvre, the Vatican Museum, and Rijksmuseum in Amsterdam all offer virtual tours of their collections and the use of VR and AR seems to be increasing daily. It’s a human instinct to go out and visit art galleries and museums but the pandemic has made this impossible but in order to satisfy the love for art and museums the alternate solution is only creating virtual tours using immersed technology. Now these enthusiast are logging into the World Wide Web to visit galleries and museums of their interest and the virtual gallery tours are increasing due to lockdown and travel restrictions. The Akron Art Museum in Ohia, USA, used AR to help local residents access public art during the recent Covid-19 shutdown created the perfect lockdown activity for the 200,000 inhabitants of Akron [4]. The art gallery displayed a number of posters and the public could interact with it using a QR code and once scanned into the tablet or the phone the users could interact with the art on display. AR allowed for socially distanced exhibition. As museums and galleries were closed design and art fairs were cancelled or postponed due to the pandemic. Errazuriz and artist Zander Eckblad developed a platform called All Show wherein the artists could self-publish their work, and buyers were able to view them at home and once they found the ones they are interested, they could preview them in “See in AR” toggle, which launches a new page with a moving hand and phone icon. Viewers can peruse artworks on the website on their smartphones. Once they find one they are interested, they can chose to preview them in the accompanying “See in AR” toggle, which launches a new page with a moving hand and phone icon [5] short time, by millions of users. Dialogue, art, and beauty are useful for mental health, to convey the discomfort due to social distancing, illness, physical fragility, towards ideals of beauty and pleasure. Thanks to immersive visits in virtual and / or augmented reality, millions of users have simulated journeys and artistic explorations that are difficult to do in times of restrictions on traveling and moving freely. Art can help and some technologies can make it more usable and literally at hand. With a smartphone, at the Pinacoteca de São Paulo, visitors have an AI at their disposal to communicate with some paintings and sculptures. At the same time, visitors can receive notifications when they

92

G. Guazzaroni

approach interactive works and they are encouraged to ask questions about the pieces of art closest to them. Bruno Pimentel Teixeira, in Chatbots Magazine, states that the symbiotic relationship with the bot and the visual art, has allowed adults, children, laymen and experts to live a very satisfying immersive experience that has surpassed the analog atmosphere provided by the Pinacoteca de São Paulo in Brazil. An increasing number of museums are currently taking this path and are using bots to engage their audiences [6]. The Heinz Nixdorf Museums Forum in Paderborn in Germany presents one of the first experiences in using a bot, through an avatar called MAX. Developed in 2004, MAX is an intelligent conversation agent that interacts directly with visitors through a screen as a virtual museum guide. The MAXXI museum in Rome offers a tool to discover Zaha Hadid’s collections and museum building through Facebook Messenger. This happens thanks to a robotic guide equipped with AI that develops by interacting and challenging the user [7]. In some cases, museums have used chatbots to offer greater support to visitors and to provide prompt assistance during the museum’s closure period due to COVID-19. Moreover, some museums have made use of chatbots to provide an extended experience, beyond the physical opening of the artistic premises. For example, a Canadian museum decided to introduce a chatbot during the global pandemic crisis. This is the Ontario Regiment RCAC Museum, a military institution dedicated to the Second World War. Since the restrictions from COVID-19 went into effect, real-life personnel have been unable to interact with visitors. Therefore, the various tanks and artillery equipment that the museum holds are currently presented by a virtual agent. The AI communication system has many of the attributes of a real employee. Master Corporal Lana is a virtual avatar trained to answer questions about the museum. Lana is displayed on a screen and is specifically designed to converse with the public during the pandemic [8]. Another example is the “Stage of virtual humanity” (multimedia technique). A work by the Italian artist Tomas (2020). A performance as first intervention after the “lockdown” due to Covid-19. Photographed by Roberto Bonfigli. The work has been included in a selection in the library/video room of the Galleria d’Arte Moderna in Rome (via Crispi), in a dedicated location devoted to the 2020 “lockdown” seen by selected artists. These works represent the artists’ personal point of view on the serious global situation, through an operation that wants to break the isolation of museums to create an artistic atelier made up of many voices and languages of art, without linguistic barriers or barriers of judgment and without limits in the use of techniques. In this context, Tomas exhibits the horizontal multimedia work named “The stage of humanity”, a digital theory of stages and theaters among which the Sydney Opera House or the Vicenza Olympic Theater (one of Palladio’s masterpieces). The theaters are enclosed into optical viewers for virtual reality. The choice of combining iconic architecture with ultra-contemporary objects suggests that humanity needs the past to plan the future [9]. Performers: Tomas, Patricia Lee Nicholls, and Giuliana Guazzaroni at the exhibition “Marche Europe in the ideal city” organized and curated by Alessandro Seri for Adam Accademia Delle Arti at the Magazzini Uto ex Tipografia Operaia, Macerata, Italy.

6 Virtual and Augmented Reality in Art During the Pandemic

93

The National Science and Technology Museum in Milan was one of the first to experiment with a monitoring application for Covid-19 spacing. Here, through AI and intelligent cameras, the position and movements of people in the rooms are detected in real time, the users are counted and related to precise volumetric information, respecting the privacy of users and without collecting data on them. This application is particularly useful for reopening the doors of museums and galleries following lockdown periods. AI is a technology that offers a lot of promise, generates turmoil and is currently starting to make its way into the world of art. Machine Learning, Computer Vision, etc. are powerful tools more and more accessible to art lovers and museum customers. These technologies will inevitably lead to interesting discoveries, specific research and new paths within the collections [10, 11].

6.3 Conclusion In the near future, searches for information on the net will occur more and more often through verbal interactions, both by the user and by the bot that responds. The chatbots have also begun to respond to verbal requests on the availability of works of art, representations of the artists, the provenance of the work, etc. providing a useful service to enthusiasts and scholars and making the tangible and intangible artistic heritage more and more usable by the customers.

References 1. https://blog.dataart.com/virtual-art-fairs-and-the-impact-of-covid-19-on-theart-trade 2. https://www.bizbash.com/productionstrategy/strategy/article/21283073/vancouvers-vmf-win ter-arts-festival-isusing-ar-technology-to-create-a-covidsafe-experience 3. https://www.gartner.com/en/marketing/insights/daily-insights/the-appeal-ofaugmented-art 4. https://culturegeek.com/article/how-are-modern-museums-putting-the-ar-intoarts-and-cul ture/ 5. https://www.dezeen.com/2020/04/17/all-show-augmented-reality-exhibitionsebastian-errazu riz/ 6. Pimentel, T.B.: When the art talks (2017). https://chatbotsmagazine.com/when-the-art-talks333e9a753751 7. Boiano, S., et al.: Chatbots in museums: hype or opportunity? https://www.museweb.net/cha tbots-in-museums-hype-or-opportunity-%E2%80%A8/ 8. Charr, M.: Military museum uses chatbot during COVID-19. Museum Next (2020). https:// www.museumnext.com/article/military-museum-uses-chatbot-duringcovid-19/ 9. Galleria D’Arte Moderna (Rome) Domani in Arte. https://www.youtube.com/watch?v=5vE U1LMqWbw 10. Gorla, S.: Museo Scienza e Tecnologia: intelligenza artificiale per distanziamento. Milano FanPage (2020). https://milano.fanpage.it/milano-da-domaniriapre-il-museo-scienza-e-tecnol ogia-intelligenza-artificiale-per-distanziamento/ 11. Styx, L.: How are museums using artificial intelligence, and is AI the future of museums? Museum Next (2020). https://www.museumnext.com/article/artificialintelligence-and-the-fut ure-of-museums/

Chapter 7

Virtual Galleries and Museums in Search of Their Own Specific Identity Mario Gerosa

Abstract In the spring of 2021, with the spread of the covid pandemic, there was a strong acceleration of interest in the virtual worlds. In particular, many galleries and museums without a virtual version had to consider the hypothesis of creating a presence in the web. All this has raised a number of issues related to the identity of virtual spaces. It is necessary to understand, in fact, if they should be twins of real spaces or if they can have their own defined and independent physiognomy. Furthermore, we need to rethink the very identity of virtual space, which does not necessarily must have the features of a walkable and navigable space, but can also be an environment defined by multimedia content. Here, in addition to addressing various crucial issues, such as storytelling in virtual exhibition spaces, some suggestions for good practices within galleries and museums are considered. Keywords Virtual galleries · Virtual museums · Viewing rooms · Storytelling · Responsive architecture Virtual galleries and museums, which have become a very important element in the period of the pandemic, are part of the broader discourse of virtual architecture, a sector still under study and definition that has at the center of its history the debate on the opportunity or less than replicating existing models in real life.1 The first thing to do when considering the possibility of creating a virtual model of a museum or gallery is to ask if there is the need to replicate in whole and in all or partially what there is already in reality. It is a discourse that has arisen for decades now, which had presented itself in a relevant way at the time of Second Life and which always returns current, as a single solution has never been proposed, or at least a shared guideline. The debate on this question is always open: some theorists argue that something very similar to what is found in reality must be created, and 1 Given that it is a subject in continuous evolution, which must be constantly monitored, in December 2020 I created the Virtual Vernissage blog (virtualvernissage.com), with interviews with the protagonists of this world.

M. Gerosa (B) Milano, Italy © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_7

95

96

M. Gerosa

other think that we should deviate, to create something absolutely original. In the period of the pandemic, starting from March 2020, there were interesting examples that tried to overcome the problem of galleries and fairs that were unexpectedly unable to host visitors. In those months, many meanings of virtual have been seen, passing from three-dimensional models that can be navigated to repropositions on websites of various multimedia elements, to compose an ideal virtual environment in a very broad meaning, not characterized by strictly architectural boundaries but by contents to activate and explore. It is therefore necessary to define, first of all, what the concept of a virtual gallery is, since it is not necessarily a traditional space, a walkable path in which the works are exhibited, a space where one walks from one point to another following a trajectory. In truth, it can also be a two-dimensional proposal where various contents are offered, from videos to images, in a kind of collage which revolutionizes and reinvents the discourse of the gallery, the museum and the exhibition. In a general sense, a virtual exhibition space is a place on the internet where one can experience a visit among works of art or other materials on display, a space that is substantiated and consolidated when offers a whole series of added values related to this medium and not always usable in reality.2 Therefore, as of now, it would be appropriate to depart from the idea of a virtual gallery in the classical sense, to do a work of abstraction and go beyond the traditionally three-dimensional image of the gallery, given the fluid nature of the virtual exhibition space. Secondly, there are discussions on use, which concern the interaction and involvement of users and visitors. At this point it is useful to say a few words on the contents of virtual museums and galleries. Here too there are two broad categories: the exhibition spaces that offer the same works presented in reality (e.g. the Louvre or the virtual Uffizi, which sometimes also boast a virtual re-presentation in reality, thanks to engaging and immersive exhibitions3 ) and the virtual places where you can admire works or artifacts that are not seen in any museum of real life. In general, when it comes to emanations from preexisting galleries or museums, there is a tendency to standardize the project to the real exhibition center, while when unpublished works or documents are presented there is greater freedom. Without going into a discussion that would deserve a separate discussion, we point out that in their own way many social networks are themselves museums and vice versa: the Facebook pages of individual users are in some way galleries of the life of those people, who day by day they show off on the web. And the same can be said of many Pinterest or Instagram pages: those collections or sets of images are potentially galleries: they can become that when someone states this intention. This concept was addressed by Suzanne Beer in her book Musées virtuels et realités muséales. We talk about a virtual museum and think of a 3D space, but you 2

Although the idea of a virtual gallery has strongly returned to the fore in the era of Covid, it is important to note that before the platforms widespread today there were many illustrious predecessors, from stereoscopic photographs to View Masters, and notoriously to museums presented in cd-rom. For more information, see: Parry [1]. 3 For example, the exhibitions conceived by Centrica. See online: https://www.virtuitaly.com/milan/ (last access 12th March 2021).

7 Virtual Galleries and Museums in Search of Their Own Specific Identity

97

can create museums with the social media, especially the ones focused on images. So we need to define well what is meant by virtual museum. As Beer says, “I wanted to explore everything that has been called and is still called a ‘virtual museum’. I noticed that the virtual museum is not a concept, as it does not have a precise definition. It could be a notion because it could be seen as a domain, but even in this case its boundaries remain vague. Many productions claim the status of “virtual museum” for many different reasons. In the 1990s there was the boom of the term virtual. The image of the museum is characterized by a serious allure. At the same time, the idea of a museum is also extremely vast, embracing the whole field of memory and visualization of oneself, of what one wants to present as an identity”.4 Furthermore, referring to the collections presented on social networks, the new figure of the collector curator is highlighted. Most of the collections on social media are curated by the collectors themselves, which is quite rare in reality, where the exhibitions are generally signed by curators or gallery owners. What does this mean? It only means that there is a different approach, but not inferior in quality. The ways in which the collection is organized certainly change, but concepts such as rarity and exclusivity are respected. The classic curator in each of his projects tends to maintain an overview, trying to distribute the works equally according to the periods of activity of an artist or a movement. Instead, the curator/collector tends to favor what he likes, often focusing on the wow effect and often thinking about the unusual, the variations on the theme.5 Regarding the subject of virtual art galleries, it is important to consider some broad categories: in this area there are, among other, online viewing rooms, 360° virtual tours, virtual galleries and virtual galleries based on a cross-media system. To these are added the virtual worlds to be attended through your avatar. At this point, a premise must be made: depending on the type of gallery, the visitor is more or less directed in their path. Moving from one example of an exhibition center to another, the degree of virtualization6 increases or decreases and the movement obligations to which one must comply differ.7 In fact, you go from journeys in galleries and museums where you are partially conditioned in your visit by a path that includes suggested points of view, with anchor points on the floor that encourage you to stop at certain points, to very free and fluid visits where you have the impression of having no constraints. It should also be said that these routes in general are very simple, and here one could also think of making the visit experience more challenging, referring to the uniqueness of the virtual experience, creating routes with some difficulties or even labyrinthine routes. For the moment we have always opted for absolutely linear paths in which we are led to proceed directly, always going forward and hardly retracing our steps. In general, therefore, it is noted that the visit to the museum or 4 In: Suzanne Beer, un libro per capire i musei virtuali, interview by Mario Gerosa, in https://www. virtualvernissage.com/suzanne-beer-musees-virtuels-realites-museales/ (last access 12th March 2021). 5 Regarding the new concept of the collector as curator, see also: Kaplan [2]. 6 See: Gerosa [3]. 7 On the accentuation of the feeling of being in a virtual environment, see: Champion [4].

98

M. Gerosa

virtual gallery tends to flatten this experience, at least on a spatial level. Added to this is the fact that generally there is the pure and simple re-proposition of the works that are showed with the same dimensions and proportions of reality. Also in this sense it could be envisaged to create some variations. We also point out right away that in general virtual galleries and museums do not present the possibility of direct involvement that allows interaction between visitors: these are solo experiences, where one cannot exchange opinions live with other people virtually present in the gallery. Unless they are virtual worlds. This could be remedied by integrating video conferencing devices that could enrich the visit experience. An interesting example of a virtual world halfway between these two modes of communication is Metaspace, created by DECOL, a new media studio based in Istanbul. In fact, it includes the idea of presence that video conferencing platforms have accustomed us to. As says Nebi Cihan Çankaya, co-founder e manager of DECOL, “the virtual worlds of Metaspace are designed at the beginning of the pandemic time period. Metaspace is a first of its kind platform. Not just a virtual meeting space with their avatars. Visitors are communicated via face to face, and voice to voice with a specially designed virtual webcam face overlap on avatars. Live events as concerts, presentations, stage shows, and anything in a physical event could be adopted to Metaspace”.8 Before passing to the examination of other possible solutions to make the experience of visiting virtual museums more engaging, let’s quickly review the models implemented so far. As mentioned, in the spring of 2020, on the occasion of art fairs that were abruptly canceled due to the Covid-19 emergency, “online viewing rooms” were organized. In this case, international fairs such as Art Basel and Frieze in London have created real environments with videos in which various gallery owners and curators spoke. The interesting thing is that these are virtual spaces, not web pages but multimedia places: it is an attempt to propose alternative environments that are defined through videos and where there is a continuous multiplication of perspectives and spaces thanks to in the presence of many add-ons. It is in fact a question of spaces within spaces. Apparently it is a matter of two-dimensional galleries, but in reality it is an absolutely interesting and original theme because through the presence of multimedia contents it structures and defines a real virtual space that takes substance also through the story. Among other things, it is important to underline that in the experience of the virtual gallery one can think of both a discourse of space traveled and of space narrated, where the word is fundamental and essential to define an architectural experience as well. Then there are the museums that offer virtual tours made with cameras that make 360° shots. In this case the use of the space is influenced by the medium. The fact 8

In: DECOL: gli eventi mediatici immersivi e Metaspace, un mondo virtuale con webcam incorporata, interview to Nebi Cihan Çankaya by Mario Gerosa in Virtual Vernissage, January 26, 2021, https://www.virtualvernissage.com/decol-gli-eventi-mediatici-immersivi-e-metaspaceun-mondo-virtuale-con-webcam-incorporata/ (last access 12th March 2021).

7 Virtual Galleries and Museums in Search of Their Own Specific Identity

99

of positioning a camera in a certain point of the room and creating panoramic shots where the operator is fixed also deeply affects the subsequent experience. In some way, the decision (which is also in fact an artistic stance) to create spaces starting from a camera, fixed at a certain point, also subsequently influences the way of perceiving spaces. In particular, in the moment in which the experience of the visit is realized, one is in any case forced to follow a path that is never totally fluid and that must in any case be based on a series of reference points. This conditions the experience of the promenade in the gallery or museum: In fact, in the real world it is often the emotionality in front of the work that guides the path, creates accelerations and slowdowns, and here, having to follow a path in any case marked by a certain rhythm it precludes some of the experience. It is true that environments created using 360° cameras can also be visited often with the aid of viewers, and in this way the experience is decidedly more fluid. In certain cases the works have to be seen from a particular perspective, magnifying them, often forcing us to look at them from the bottom up, and sometimes giving the feeling of an unnatural perspective, that perhaps we could define a “technological perspective” or “virtual perspective”.9 Visiting exhibitions and museums with this system sometimes one gets the idea that space can somehow be overwhelming, one feels somehow overwhelmed by the surrounding environment and the perspectives appear slightly forced. Another system for creating virtual galleries are multi-platform graphics engines such as Unity or Unreal, used in different contexts, such as videogames or architectural creations. In this case the perspectives are closer to those of reality, one gets the impression of taking a less forced and more fluid path. With the help of Unity, a virtual gallery called Vspace was created by the Massimo de Carlo Gallery in the first period of the pandemic. The visitor who moves in this gallery has the impression of having greater freedom and moves by choosing their own paths. Another virtual reality platform is HWVR by Hauser & Wirth, an other international contemporary art gallery that has its own virtual exhibition center. At this point another discourse arises, that of the aesthetics of virtual museum galleries, since it is by no means obvious that an anonymous container, a so-called white cube, must be created to propose the works in the liquid space of the internet. In reality, it is also possible to think of devising a gallery that is more relevant to the spirit of web cultures and in particular that refers to the aesthetics of video games.10 Here, too, the discourse deserves an in-depth study that analyzes and investigates the architectural styles that have alternated on the web over the years, both thanks to the mainstream examples developed by developers and by residents of virtual worlds. An aesthetic that balances between two cultures, the classical one and the one generated from below, a recurring concept on the web. And not surprisingly, one of the advantages of virtual galleries is the fact that they are very democratic: there are ready made and “do-it-yourself galleries” where everyone can create their own space in a fairly simple and intuitive way and exhibit their works. This type of galleries are 9

See also: Friedberg [5]. A first examination of styles was developed in 2006, in the golden age of Second Life. See also: Gerosa [6].

10

100

M. Gerosa

proposed by platforms11 that often provide also customized galleries. In their basic version they are rather simple, or at least minimal, but they also have the undoubted merit of introducing this new idea of the virtual gallery to the general public. On the other hand, speaking of aesthetics, the comparison with the latest generation videogames and with the creations made with advanced computer graphics programs is inevitable.12 But now it is a question of understanding how to make the virtual gallery experience unique, which in general, compared to the real one, suffers from some shortcomings. The question of the use of these spaces and of the exhibited works remains unavoidable and complex, linked above all to the ability to move within these places; in fact, visits to the gallery or museum always imply a technical ability, even minimal, which absorbs part of the visitor’s attention, distracting him from the experience of enjoying the work. At all times the visitor must pay some attention to the movements he makes and the technical procedures he has to face and in this he loses part of the involvement during the use of the works. This is a problem to be solved, since the more immersive the experience, the more these technical difficulties increase. It is a question related to the idea of the “sustainability” of virtual architecture. Then there is the need to make up for the shortcomings of the virtual compared to the real. As is known, within the virtual realms some senses are penalized, such as touch, taste and smell.13 For the moment we can only focus on sight and hearing. The lack of tactile sensation has tried to make up for some platforms such as Vortic VR, which offer spaces to visit with Oculus-type viewers and where you have the sensation of touching objects, in fact an almost tactile experience is offered, there is a large attention to the three-dimensional rendering of everything that is seen during the visit. Another relevant aspect is the creation of a “wow effect”, which manages to endow the virtual with a spectacle that makes you forget its criticalities. It occurs, for example, in the virtual version of the Kremer Collection. Created in 2017 by the architect Johan van Lierop, owner of the Bouwery studio, it is a paradigmatic project regarding the good practices of virtual architecture: it is a monumental complex that defies the law of gravity, amazes and excites, perfectly aligned with the most innovative architectures of our time. Van Lierop then had the great intuition of suspending the paintings away from the walls, so that the back of the canvases could also be seen. As he himself said, “Looking at the back of these old paintings is like looking in their

11

Artsteps, Exhibbit, Ikonospace, Kunstmatrix are some of them, and each one of these platforms offer a different kind of solution. 12 A qualitative leap is likely to take place in the near future. In this regard see the interview in Virtual Vernissage to Ilya Gulyaev, author of valuable digital creations suitable for virtual museums and galleries. Ilya Gulyaev, “I musei virtuali saranno belli come i videogame più evoluti”, December 11, 2020, https://www.virtualvernissage.com/ilya-gulyaev-i-musei-virtuali-saranno-belli-come-ivideogame-piu-evoluti/ (last access 12th March 2021). 13 About the topic of touch in virtual reality, see: Price et al. [7].

7 Virtual Galleries and Museums in Search of Their Own Specific Identity

101

passport: you discover their journey through time marks, personal notes, stamps or even scars. It is a very intimate experience that would not be possible in the Louvre”.14 The presence of multimedia content is fundamental to enrich the experience. With platforms such as Matterport you can insert various video and audio content into your gallery and this is certainly a strong point. The presence of multimedia content is one of the simplest and most intuitive things to do within a virtual gallery project. It is a matter inherent in the very nature of the medium, and can have a marked didactic function.15 It is a sort of device that is an integral part of this type of media. When it comes to multimedia content, we refer to videos, but also for example to commentary boards where everyone can leave their own impression on a certain work on display, in favor of those who will come later. In this way the work is more shared, it is an integral part of a sort of social network. The museum and the gallery become social, something that not necessarily happens in reality.16 In fact, for museums in the real world it is conceivable to create support websites where visitors can leave comments or interact, but it is a relatively uncommon practice. Instead, in a virtual museum this could become a habit and a further strength. Another important issue linked to virtual museums and galleries is related to the use in real time of a space that never remains the same, but which adapts to the visitor’s emotions. Walking through the space of a gallery, it would be interesting for the room or the work to become resonance boxes for the user’s emotions. If a visitor stops for a longer time in front of a certain work, it would be useful if the work itself were to be enlarged independently, to offer a greater definition of detail to the observer, without the latter having to worry about enlarging it with the computer tools: there should be an immediate correspondence between the visitor and the work observed, which adapts in size to the interest of the beholder. In the same way, spaces could change in real time, so that the museum or gallery can also become communication tools. And if a visitor lingers for a certain period of time in front of a work of art, it would be interesting if the surrounding space also adapted or in any case reflected this prolonged presence.17 Artificial intelligence is linked to this theme. This resource could be extremely interesting or even decisive in the case of the design of the galleries of virtual museums, as thanks to the data relating to the behavior of the users traveled and the time spent in front of each individual work it allows to modify the proportions and dimensions of the works themselves. In fact, one realizes that visitors linger more in front of a certain work, one could think in real time of also changing the spaces that host that particular work. In fact, the possibility of continuously changing the layout of the rooms is one of the most interesting prerogatives of virtual galleries. 14

See also: https://www.thekremercollection.com/the-kremer-museum/ (last access 12th March 2021). 15 On the subject of virtual museums as a teaching tool, see also: Atamuratov [8] Part II, e a Daniela [9]. 16 About this topic, Jas Brooks in Promises of the Virtual Museum, in: Crossroads, The ACM Magazine for Students, January 2019. See also: Asproni [10]. 17 The theme of an ever-changing architecture, also depending on the visitor’s moods, is one of the points I have foreseen in a handbook for virtual galleries. See: Gerosa [11].

102

M. Gerosa

Still referring to this idea of modifying the gallery and the virtual museum in real time based on the impressions of visitors, one could think of creating customized galleries whose spaces change according to the emotions of the visitor.18 But above all, thanks to the artificial intelligence, the dimension of the spaces and of the works could change constantly, according to the way of perceiving the spirit of the objects shown in a certain room. If the visitor in front of a certain type of work to an impression of melancholy, the room could take on tones that turn to warm colors, if instead it communicates an idea of cheerfulness then one could think of making the room more lively with colors of the walls that change at that moment taking this possibility into account. It could also be thought of introducing music that helps to live the experience of visiting the gallery in a more engaging way. It might be interesting to create an idea of atmosphere. In fact, the problem of the virtual is the presumed coldness, which can be remedied with some expedients: for example, visitors to an exhibition could be sent a small bottle with perfumes to be opened near certain works. In the past, the scents of some ancient works had already been recreated to give the idea in reality what the smell of those of those paintings was. The smells and the atmosphere are fundamental: in the real world there are rooms that have an aftertaste of mold or the patina of time that is found in certain works, in other cases the scent of plastic that smells of industrial production prevails. It would be important to be able to offer these olfactory sensations to those who visit remotely with the galleries.19 The discourse of the exhibition is also central, which does not necessarily have to be like that of the galleries of the real world, since in virtual galleries the interaction with the works is quite different. In the virtual world, the works exhibited in the galleries cannot be smelled or touched, but you can do something else, for example you can walk through a work, feel part of a painting, experience the context of a specific painting firsthand. An experience that in the real world is lived in some immersive exhibitions, with multi-projections of photos and videos. We have already mentioned the issue of the aesthetics of the virtual gallery or museum, which, being within the web, should theoretically refer to the aesthetics of digital architectures such as those of videogames or virtual worlds. However, from these aesthetic considerations, it should not be deduced that the right way to make a virtual museum or gallery more attractive is necessarily to resort to gamification, the transformation of space and experience in a videogame key. It is a fairly widespread opinion that to make the experience of visiting a virtual gallery attractive, you have to create some kind of game. In truth this is not the only option, and above all it is not necessarily the winning one in order to give a “digital oriented” nature to the project, also because it is not certain that the visitor lends himself to an experience that involves him in a playful way. On the other hand, it would always be useful to consider an emotional approach and a strong and original storytelling idea tailored to 18 See also Giuliana Guazzaroni, Role of Emotions in Interactive Museums: How Art and Virtual Reality Affect Emotions, in: Guazzaroni and Pillai [12]. 19 In this regard, we could learn a lot from research on olfactory devices in real museums. See also: Castel [13].

7 Virtual Galleries and Museums in Search of Their Own Specific Identity

103

that project. However, it is not enough to create a kind of game in which the curator now impersonates himself, now the artist, transforming the museum into a playful experience. Placing the museum in a cartoon-like scenario is not always the most interesting path to tackle, especially today when a lot of work is being done on the identity and physiognomy of virtual characters. Presenting a stylized avatar within a playful setting risks distorting the artistic experience and not offering sufficient immersion. The theme of the presence of the virtual counterpart inside the gallery is also linked to this discourse. Is it useful or not to have a character to represent us during this visit? Or does it risk taking away the immersion of the whole experience? Still on the subject of storytelling, in the virtual world, in a broad sense, the word is often and even stronger than the image, so it is necessary to do a lot of work to enhance this aspect.20 In a museum or a virtual gallery, storytelling is closely linked to the idea of community: they are two aspects that go hand in hand and that cannot be considered independently. In fact, the museum must have a double function: that of a place where works of art are exhibited, and that of a social network that favors exchanges, sharing of opinions and insights on the works on display. Therefore the museum must never be seen as an experience to be lived in isolation, rather we must always try to encourage this idea of exchange. At the same time, it is essential to always strengthen the idea of communication of the museum or gallery, in the sense that this structure must become a kind of medium, it must be something “transparent”, non-invasive, a device that allows it to be used to facilitate exchanges and emotions. An interesting solution that goes far beyond the idea of gamification and aims to strengthen both the idea of the curator and that of the community is offered by the virtual world Occupy White Walls. Within this world, you either play the artist or the curator. Inside OWW there are 75.000 art galleries, created by as many users, and everyone can upload (obviously paying attention to copyrights) works of art of all kinds, but also works of creatives and artists who frequent that world. There is everything, from Renaissance painting to cultures developed on the internet, and there is total artistic democracy. As stated by Yarden Yaroshevski, founder of Occupy White Walls and CEO of StikiPixels, the company that manages OWW, a strong point of this virtual world is the presence of artificial intelligence. “In the game there is a proprietary AI system called Daisy and which stores the tastes of individual users. As players visit other galleries, Daisy stores the data in order to make suggestions. Here there is no need to have an artistic culture: every time you find a work you like, AI guides you to discover other works. And we don’t care if an artist is famous or not: artificial intelligence is totally agnostic”.21 The visit to a virtual museum must always be a journey that always involve discoveries, experiences, encounters, testimonies, exchanges, which testify to having 20

About the storytelling in virtual museums, see: Sylaiou and Dafiotis [14]. In: Yarden Yaroshevski: Occupy White Walls, un’idea democratica di arte e la forza dell’intelligenza artificiale, interview by Mario Gerosa in Virtual Vernissage, December 24, 2020, https://www.virtualvernissage.com/occupy-white-walls-yarden-yaroshevski/ (last access 12th March 2021).

21

104

M. Gerosa

been in that particular place but which also allow you to always have an enrichment. During a visit to a museum or a virtual gallery there are some fixed points to keep in mind, which concern the discoveries of the works on display, experiences and meetings with other visitors, or with curators and experts, selfies, comments. In particular, the advent of virtual galleries also implies an updating of the figure and role of the curator, key figure in developing original and innovative content. As explained by Lívia Nolasco-Rózsás, curator of international exhibitions of contemporary art and media art, project manager of Beyond Matter, “I personally hope for projects that produce medium-specific content for physical and virtual spaces alike, regardless of their chosen topic. Creating copies of real spaces is not a sustainable answer for digitally created spaces. Now, in the midst of a pandemic there is clearly a high demand of online exhibitions to substitute the closed physical ones, but even after the period of social distancing and lockdowns is over the currently elaborated formats will have an impact. It might be that online virtual exhibition spaces will function as complementary elements to physical shows and vice versa. In both cases creators and curators of these spaces shall offer different approaches and content in the virtual and in the physical parts. In addition, locative Augmented Reality experiences embedded in the museum context (my favorite is still MoMA’s AR project started in 2018: https://momar.gallery) or in public space (various projects of Acute art use public spaces to stage commissioned AR artworks, the ZKM | Karlsruhe has been featuring AR artworks in public space in the framework of the Seasons of Media Arts) are on the rise”.22 When it comes to virtual worlds it is easy to hypothesize a comparison in real time with other visitors with experts; if this is not possible, it can be remedied with calls that allow you to interact with other people. To facilitate this type of experience, specific thematic paths can also be thought of. The most popular could be the artistic path that provides explanations of critical curators interviews artists, as happened in the viewing rooms that spread in the spring of 2020. It would also be conceivable to foresee the presence of a virtual assistant with whom one can chat in real time to get news about a work, a trend, or an artist. Equally interesting is the path more closely linked to the community, once again from the perspective of a virtual museum or gallery as a social network. People want to feel the gallery or museum very close to them in some way they want to appropriate themselves, share the experience of the visit, because as they said, seeing the museum alone, getting excited by yourself is quite limiting. For this one can think of both the aforementioned system via chat, but then one can also think of multimedia elements such as a panel of visitor comments, such as feedback with ratings for the work in question. And next to each work there could also be a link with the notation “you might also like …” Thereby emotions and comments are shared. In addition, one could also think of some videos in which visitors tell their emotions felt in front of a 22

In: Lívia Nolasco-Rózsás, il ruolo del curatore nelle mostre virtuali e il progetto Beyond Matter, interview by Mario Gerosa in Virtual Vernissage, January, 27, 2021, https://www.virtualvernissage. com/livia-nolasco-rozsas-il-ruolo-del-curatore-nelle-mostre-virtuali-e-il-progetto-beyond-matter/ (last access 12th March 2021).

7 Virtual Galleries and Museums in Search of Their Own Specific Identity

105

certain work or even the possibility of taking a virtual selfie in front of their favorite painting. One uploads her or his own photo and create an image of himself in front of the work. In this regard, one can also create a selfie room that can work by uploading one’s own photo. From the point of view of storytelling, one can think about a path more linked to an idea of short stories.23 A writer is commissioned a short story with some points that refer to the works located in the gallery. When one reads the story, she or he has to go and look for the work that is cited, and next to the work, in a game of crossreferences, will find for example a video or a photo linked to that novel. Then, there is also the theme of guided tours in a museum or gallery. This aspect is an integral part of the visit experience. In a virtual world this is easily achievable, while in a stand alone virtual gallery one should think of connections with a guide at certain times of the day via chat. A resource that could further contribute to humanizing galleries and virtual worlds. But the central point remains the discourse of the content. What can you see in a virtual gallery, and above all how do you make the most of its potential? One of the reference examples is undoubtedly the “Thresholds” exhibition,24 conceived in 2017 by the British artist Mat Collishaw, who used virtual reality to allow the public to visit and relive a photographic exhibition of 1839. It is a clear indication that one must look at the things from another point of view: not to look for the critical points of the virtual in relation to the real, but to discover its strengths, which make it possible to fill in the gaps.25 And then, another crucial point, apart from the lack of touch or smell, is the lack of physical presence and therefore the deprivation of the possibility of creating a collection of one’s presences in museums. “Virtual museums and galleries are not collectible. But many visit museums to compile their own mental catalog of works, for example to say that they have seen the Mona Lisa, or all of Bosch’s paintings, or all the Vermeers, or whatever you want. Generally, you take the catalog home, which is the testimony of having been there, and take photos in front of the works. now, with Instagram, selfies in museums have multiplied, collectible proofs of one’s presence in a place dedicated to preserving collections. On the other hand, if I go to a 360° virtual museum, what can I do to tell that I’ve been there? Can I take a screenshot? A little bit. Can I take a selfie in front of my computer when the virtual museum is seen? Maybe. However, the element of collectability in presence is always missing,

23

In this regard, the presence of avatars as guides or curators should also be considered. These are virtual humans who mainly give information to visitors. See: Sylaiou et al. [15]. 24 https://matcollishaw.com/works/thresholds/ (last access 12th March 2021). 25 Another discourse connected to the virtual revival of past exhibitions is that of the conservation of virtual artefacts. Now that there is a lot of talk about virtual museums, how many museums are there dedicated to virtual museums? Fortunately, there is someone. I discovered, for example, that the Mythologic Museum was recently opened on Fortnite, where among other things some elements of the locations of the most famous events of that virtual world are kept. See also: Baum and Lyons [16].

106

M. Gerosa

which is fundamental. There is no experience of collecting oneself in the world of collections through souvenir photographs”.26 The lack of the practice of artselfies27 in virtual galleries is perhaps more serious than that of the lack of smell. In conclusion, it can be said that the discourse of virtual museums and galleries is still open. It is a subject in the process of being defined that takes shape in real time, as precise needs are revealed. It is not possible to start from a theoretical assumption, but it is necessary to act practically, evaluating all the components and improving along the way.

References 1. Parry, R.: Museums in a Digital Age. Routledge, New York (2010) 2. Kaplan, I.: When Collectors—Not Curators—Dictate Art History, Artsy (2017). https://www. artsy.net/article/artsy-editorial-collectors-curators-dictate-art-history. Accessed 12 Mar 2021 3. Gerosa, M.: Rinascimento Virtuale. Meltemi, Roma (2008) 4. Champion, E. (ed.): The Phenomenology of Real and Virtual Places. Routledge, New York (2018) 5. Friedberg, A.: the virtual window. From Alberti to Microsoft. The MIT Press, Cambridge (2017) 6. Gerosa, M.: Ludico, ionico, corinzio. Arte e architettura nei mondi virtuali. In: Fulco I (ed.) Virtual Geographic. Costa & Nolan, Milano (2006) 7. Price, S., Jewitt, C., Yiannoutsou, N.: Conceptualising touch in VR. Virtual Reality (2021). https://doi.org/10.1007/s10055-020-00494-y. Accessed 12 Mar 2021 8. Atamuratov, R.: The importance of the virtual museums in the educational process. Eur. J. Res. Reflect. Educ. Sci. 8(2) (2020) 9. Daniela, L.: Virtual Museums as Learning Agents. Sustainability (2020) 10. Asproni, P.: Musei social o musei digital? In: ilgiornaledellefondazioni.com (2017). http:// www.ilgiornaledellefondazioni.com/content/musei-social-o-musei-digital. Access 12 Mar 2021 11. Gerosa, M.: Collectors dialogues/Il souvenir, la presenza e il museo da collezione. In: Virtual Vernissage (2020). https://www.virtualvernissage.com/collectors-dialogues-il-souvenir-la-pre senza-e-il-museo-da-collezione/. Accessed 12 Mar 2021 12. Guazzaroni, G., Pillai, A., (ed.): Virtual and Augmented Reality in Education, Art and Museums, Hershey: IGI Global (2020) 13. Castel, M.: Les Dispositifs Olfactifs au Musée. Editions Le Contrepoint, Paris 14. Sylaiou, S., Dafiotis, P.: Storytelling in virtual museums: engaging a multitude of voices. In: Visual Computing for Cultural Heritage. Springer, New York (2020) 15. Sylaiou, S., Kasapakis, V., Gavalas, D., Dzardanova, E.: Avatars as storytellers: affective narratives in virtual museums. In: Personal and Ubiquitous Computing. Springer, New York (2020) 16. Baum, J., Lyons, K.: Librarianship presence in virtual worlds. In: Yang, H.H., Yuen, S.C.-Y. (eds.) Handbook of Research on Practices and Outcomes in Virtual Worlds and Enironments. IGI Global, Hershey (2012) 17. DIS (2014). #artselfie. Notes on Selfies par Douglas Coupland. Conversations entre DIS & Simon Castets. Jean Boîte Editions, Paris 26 27

Gerosa [11]. See DIS [17].

7 Virtual Galleries and Museums in Search of Their Own Specific Identity

107

18. Acidini, L.C., Cappellini, V.: Reale e virtuale nei musei. Due visioni a confronto. Pitagora, Bologna (2008) 19. Antinucci, F.: Musei Virtuali. Roma-Bari, Laterza (2007) 20. Beer, S.: Musées virtuels et realités muséales. L’Harmattan, Paris (2020) 21. Deloche, B.: Le musée Virtuel. Puf, Paris (2001) 22. Guazzaroni, G., Pillai, A.: Virtual and Augmented Regerality in Education, Art and Museums. IGI Global, Hershey (2020) 23. Mandarano, N.: Musei e Media Digitali. Carocci, Roma (2019) 24. Mathey, A.: Le musée virtuel. Quel avenir pour la culture numérique? (2006). Le Manuscrit, Paris. 25. Mathey, A.: Le musée virtuel. Les nouveaux enjeux. Le Manuscrit, Paris (2012) 26. Gerosa, M.: Gallerie virtuali. Un decalogo per crearle come si deve, Artribune.com, 23 settembre 2020. https://www.artribune.com/progettazione/new-media/2020/09/gallerie-vir tuali-decalogo/. Accessed 12 Mar 2021

Chapter 8

Framing Digital Reality Technology Applications Among Museums During COVID-19 Pandemic: A Comparative Text Mining Research Yowei Kang and Kenneth C. C. Yang Abstract Digital reality technologies and their applications have become a truly global phenomenon, attracting enormous interest among museum researchers and professionals. The COVID-19 pandemic in 2020 has drastically changed contemporary human behaviors by forcing people to “shelter in place.” To deter the spread of the COVID-19 virus from adjacent human contacts and indoor congregations has prompted over 90% museums around the world to close their doors to the public. As a result, the COVID-19 also expedites the adoption of a variety of technologies by offering their services virtually. The objectives of this study are to investigate the impacts of digital reality technologies on museums around the world at this critical historical juncture. We employed a global text mining research to examine the current integration of digital reality technologies among museums and to examine how the affordances of these technologies have impacted on the museum sector as described by these media. This text mining study provides empirical data and best practice examples how museums around the world have integrated digital reality technologies in response to the challenges and opportunities due to COVID-19. Results find that virtual reality is still the most commonly used digital reality technology by museums for online viewing and exhibitions. In spite of the hype about extended reality technology, augmented and mixed reality technologies fall behind the popularity of VR. Additionally, the use of these reality-creating technologies among museums in different countries often echoes their economy and technology status. Discussion and implications are provided. Manuscript Submitted for Publication Consideration at Extended Reality Usage during COVID-19 Pandemic Edited by Pillai, Anitha S and Guazzaroni, Giuliana. Y. Kang Inservice Bachelor Program of Culture and Creative Industry, National Chung Hsing University, 145 Xingda Road, South District, Taichung City 402, Taiwan e-mail: [email protected] K. C. C. Yang (B) Department of Communication, The University of Texas at El Paso, 500 West University Avenue, El Paso, TX 79968, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_8

109

110

Y. Kang and K. C. C. Yang

Keywords Augmented reality · Digital reality technology · Extended realities · Framing analysis · Mixed reality · Text mining · Virtual reality

8.1 Introduction 8.1.1 Digital Reality Technologies in the Museum Sector As a truly global phenomenon, digital reality technology has fascinated both museum researchers and professionals who are interested in its integration into the museum sector [34, 36, 63, 64]. The term, digital reality technology, is generally made up of an assortment of emerging and evolving reality-rendering technologies and their applications in augmented reality (AR), mixed reality (MR), virtual reality (VR), extended reality (ER), 360-degree video, and other immersive platforms [15, 34, 36, 63, 64]. Past museum studies have observed that these technologies can generate engaging cultural experiences when museum visitors are able to interact with cultural artefacts by superimposing them with computer-generated images, animation, or videos to create a fictitious reality [12, 13, 41]. For example, Smithsonian American Art Museum (SAAM) [48] uses a mobile-enabled VR technology to provide a wideranging 360° 3D panoramic and immersive views of the Renwick Gallery’s Wonder Exhibition [34, 36]. In the National Museum of Singapore, visitors can use their mobile or tablet devices to interact with animation of animals and plants adapted and rendered from 69 still paintings from its William Farquhar Collection of Natural History Drawings [13]. In Taiwan, the Gold Museum in New Taipei City has integrated both AR and gaming platforms to let young museum visitors to relive the history and experiences of gold miners in the past [53]. With the ubiquitous impacts of the COVID-19 Pandemic, the technical capabilities of new media technologies in offering a remote access to these cultural contents are essential to the survival and continuation of this much affected sector. For example, social media, such as Twitter and Instagram, have enabled museums in the U.S. to share their collections and virtual gallery tours through videos among shelter-in-place museum visitors by clicking #MuseumFromHome and #MuseumMomentofZen [17]. Nevertheless, long before the coming of the COVID-19, Google Arts and Culture website (https://artsandculture.google.com/project/streetviews) has already offered up to 2,000 virtual museum tours to museum visitors [17]. The digitalization of cultural heritage has evidently followed a trend observed by [46] who claim the production and dissemination of cultural contents will be revolutionized by the possibilities of creating multi-media contents to better engage the audience. The COVID-19 Pandemic has functioned as a catalyst of a rapid integration of these new technologies for the survival and continuation of the museum sector. For example, museums around the world have rushed to provide virtual tours of their collections from any remote location, these museums include British Museum, Johannesburg Art Gallery, Musée d’Orsay, National Gallery of Art, National Museum of African

8 Framing Digital Reality Technology Applications Among Museums …

111

American History and Culture, and the Smithsonian Museum of Natural History, among others [17]. Similar to other technologies that have emerged during the past decades, digital reality technology has the same potential to produce innovative contents to encourage museum visitors’ interactions with these cultural contents [34, 36, 41]. This cluster of reality-creating technologies has been predicted to be one of the four trends that could affect the museum sectors, “transforming the cultural landscape and the way audiences experience museums” [17]. Emerging platforms such as Sketchfab (https://ske tchfab.com/museums) have enabled Cleveland Museum of Art, Minneapolis Institute of Art, and The British Museum to share their collections through immersive 3D objects created through AR, MR, VR, and XR applications [17]. The advent of these technologies have posed brand new challenges and opportunities to art exhibitions, galleries, and museums to deliver creative and cultural contents [34, 36]. The museum sector constitutes an important part of the cultural industry ecosystem [34, 36] and it employs over 760,000 Americans and contributes USD$50 billion to the US economy [57]. This sector also most successfully transform its cultural heritage, covering audiovisual, cinematographic productions, crafts and design, phonographic contents, printing and publication, multimedia, among others, [34, 36, 39] to deliver a brand-new museum experiences for visitors [17, 34]. Additionally, these technologies also benefit the museum sector by making it more relevant to their patorns [16], even with potential privacy concerns [35]. Because of their potential to enhance museum visitors’ experiences, digital reality technology is commonly defined as a set of computer-based technologies that are able to immerse users in a reality-creating virtual environment. These technologies are also called immersive technologies in the academic and trade literature [19–24, 26] because of their technical capabilities and affordances to transport users from their physical environment to a fictitious make-believe world. As new technologies emerge and converge, the digital reality ecosystem is also extended to include video games, virtual worlds, and social networking technologies [62]. The futuristic vision of an Intelligent Virtual Environment (IVE) shows the potential integration of digital reality technologies with other artificial intelligence applications [40].

8.1.2 The COVID-19 Pandemic and the Transformation of Museums Coronavirus Disease 2019 (or commonly known as COVID-19) has been tied to China as the epicenter where this deadly global pandemic began [3, 28]. Despite the development of vaccines, the global COVID-19 cases have reached 81.67 million while the deaths top 1.78 million, as of December 29, 2020 [61]. Ironically, the rapid spread of COVID-19 is expected to increase digital reality applications around the world [5, 17] because this pandemic also has transformed how human beings would employ and deploy these technologies in various contexts (such as classroom,

112

Y. Kang and K. C. C. Yang

automation, business, museum, and medical fields) [60]. This global pandemic has forced many people to stay at home during the work day. A recently published study of 14 metro areas in the U.S. (2021) has observed that, after the break-out of the COVID-19, 89% of more people (42.6 million vs. 22.6 million) become active online during the workday [2]. Because of their potential, immersive XR collaboration— which combines AR, MR, or VR technology—is expected to exceed US$400 million by 2025 [44]. One of the major causalities of COVID-19 is the museum sector where 90.9% of the world’s 95,000 museums have closed their door to the public since the pandemic [57]. Among them, 10% are expected to be closed permanently [57]. The global closure of museums, due to preventive lockdown measures to deter the spread of COVID-19 from human contacts and congregations, has prompted museum operators to embrace existing and new digital reality technologies and resources [57]. Before COVID-19, museums around the world have already eagerly adopted digital reality technologies to extend their collections and exhibitions to the public [25]. As early as 2016, the Franklin Institute in Philadelphia has launched a series of VR-based exhibitions [12]. Similarly, the Tower of David Museum in Jerusalem, Israel has launched its “The Holy City” initiative to immerse museum visitors to remotely access cultural heritage related to this holy city [33]. During the lock-down, 360° tours, digital exhibitions, virtual museums, social media, live streaming videos, webinars, among others have been adopted to provide new museum experiences [17, 57]. However, COVID-related closure and subsequent revenue reduction may have posed threats to these initiatives. According to recent data published by the American Alliance of Museums, museums in the U.S. alone have lost about 33 million per day [30]. Additionally, wide-spread digital divide among countries has limited their technical capabilities to ensure their online presence; only 5% of museums in Africa and small less developed states can afford these technologies [57].

8.1.3 Research Questions Given the growing importance of AR, MR, VR, and XR applications by the museum sector around the world, this study provides a comprehensive examination of how media discourses of media outlets and professionals have framed the integration of digital reality technologies to enable the general public to interact with the cultural heritage in the face of COVID-19. This study will answer the following questions: • RQ #1: What is the current state of augmented (AR), mixed (MR), virtual reality (VR), or extended reality (XR) applications by museums around the world in the face of COVID-19? • RQ2: How will text mining methods help museum professionals to understand media discourses in terms of framing augmented (AR), mixed (MR), virtual reality (VR), or extended reality (XR) applications by museums around the world?

8 Framing Digital Reality Technology Applications Among Museums …

113

• RQ3: What are the recurrent keywords and phrases as found in the collected media corpus on the applications of augmented (AR), mixed (MR), virtual reality (VR), or extended reality (XR) applications by museums around the world?

8.2 Literature Review This study is based on a popular theory in communication, media, sociology, and psychology research [47, 56, 58]. The concept of frame is defined as “a central organizing idea or story line that provides meaning to an unfolding strip of events, weaving a connection among them” [27]. The practice of framing by journalist and media professionals can be considered as the purposeful editing, organizing, selecting, and presenting information by media organizations and journalists about an issue, a topic, or a phenomenon [43]. Thus, the process of framing involves the assignment of salience to a topic that media organizations want to shape the perceptions of the aforementioned topic among their audiences [47, 56]. In spite of the importance of the framing research and its widespread applications, rarely have scholars employed new text mining methods to fill the methodological gap in traditional framing research. This study attempts to generate new insights to see what computational framing techniques can contribute to methodological developments in framing research.

8.2.1 Framing AR, MR, VR, and XR Applications in the Museum Sector Reality-creating technologies (such as AR, MR, and VR) can create an immersive extended reality (XR) experience [51]. The global XR industry is expected to grow from USD$18.5 billion in 2019 to over USD$1 trillion by 2020 (P&S Intelligence, as cited in [51]). These technologies help museums around the world to promote cultural heritage by allowing remote access of museum visitors to their collections and exhibitions, thanks to the technology’s interactivity, realism, and visualization [4, 34–36]. Other scholars have also pointed out these technologies have the following features: connectivity, hypertextuality, interactivity, location-specificity, and modality [32]. Overall, reality-creating technologies are able to create a connection between cultural heritage inside the museum and current or potential museum visitors [1, 35]. For example, the VR-based Virtual Exhibition System is designed for archeological museums to transform their finds into more immersive cultural contents to museum visitors remotely [4, 35]. Through a variety of web-based XR applications (e.g., ARCHEOGUIDE, ARCO, and 3D-MURALE), museum-visitors are connected to many archeological artefacts and sites remotely [4, 35]. Similarly, the VR-enhanced exhibition at the National History Museum in Los Angeles enables museum visitors to experience three different underwater VR experience on their own

114

Y. Kang and K. C. C. Yang

premises [1, 35]. With the popularity of mobile technologies, many of the digital reality applications also integrate mobile-enabled AR apps. For example, the Smithsonian American Art Museum (SAAM) [48] employs a VR technology to allow museum goers to experience the Renwick Gallery’s Wonder Exhibition through its downloaded app to enjoy 360 degree 3D panoramic views of the gallery [34–36, 65]. In France, the renowned Lourve launched its multi-lingual “Mona Lisa: Beyond the Glass” exhibition accessible through its VR app store, Android, iOS, and VIVEPORT [13]. This VR-based exhibition enables museum visitors to have a thorough knowledge of this masterpiece by interacting with animated images and sound [13]. Extant literature on the roles of AR, MR, VR, and XR applications in the museum sector has mainly focused on the following areas. First, ample research has provided detailed description of best practice applications by museum professionals. Benefits, challenges, and opportunities of these implementations are often documented in the literature [16, 17]. For example, an ambitious digitalization project by China’s Baidu Baike hosts more than 220 Chinese museums and cultural heritage applications to allow visitors to have a virtual tour of these museums through its AI, VR, videos, and 360° video technologies [8]. This Chinese project also collaborates with over 2,000 museums around the world, including Spanish Gaudi’s Sacrada Familia [8]. In the U.S. the virtual reality experience project, “Beyond the Walls”, launched by The Smithsonian American Art Museum (SAAM), combines the museum’s gallery of images with augmented contents will enable museum visitors to interact with these immersive contents by wearing a HTC Vive or Oculus Rift virtual reality headset [17, 48]. Other innovative project in Asia includes Taiwan’s National Museum for Prehistory where pre-historical Peinan tribal life in south-eastern Taiwan has been visually transformed into an AR exhibition [34, 36, 41]. Digital reality technologies have also played a critical role in promoting the understanding of indigenous peoples in early Taiwan through the digitally-created reconstruction of archaeological finds [34, 36]. The Digital Project for the Education and Promotion of Taiwan’s Prehistory Cultures, The Peinan Site and Peinan Culture is a collaboration with Next Animation Studio [42] that combines both AR and 3D animation project to digitally transform archeological remains at the 3,000-year-old Peinan pre-historical settlement site into authentic and interactive experiences for visitors of Beinan Cultural Park [34, 36]. Visitors can virtually walk among the early settlers of Taiwan to experience how they live 3,000 years ago in the pre-historical Taiwan. The reconstruction of the Peinan archeological site also allows visitors to use their bile device to superimpose computer-generated images over many archeological discoveries from the site. Additionally, through a digital game, museum visitors can virtually participate in the excavation process like an Indian Jones [34, 36, 42] .

8 Framing Digital Reality Technology Applications Among Museums …

115

8.2.2 Framing Best Practices by Their Technology Affordances and Features Second, museum researchers and professionals have examined the characteristics of AR, MR, VR, and XR technologies and their affordance effects on audience’s experiences with cultural artefacts and heritage in a computer-generated environment [36]. Among many significant attributes of these digital reality technologies, the most noticeable is their technical capabilities to allow the museum sector to distribute and share their cultural heritage contents and collections to the public [18, 34, 36]. Technologically, digital reality technology systems are often composed of (1) a graphic rendering system; (2) gloves, trackers, and user interfaces to sense and input users’ movements; (3) output devices to enable aural, haptic, and visual interactions in a computer-generated virtual environment; (4) a software to model virtual objects and to construct databases; (5) a system to deliver visual display technology to offer users with different interactive and immersive museum experiences [11, 34, 36, 65]. Additionally [32], identifies and explained seven key characteristics of these interactive technologies, including interactivity, hypertextuality, modality, connectivity, location-specificity, and mobility (p. 254) that can account for the message elements, communication processes and patterns, and networking effects. In the context of the museum sector, the connectivity attribute refers to the communication processes that visitors can remotely access museum contents and exhibitions and even connect with other patrons. On the hand, hypertextuality, modality, and virtuality characteristics refer to museum visitors’ abilities to multi-modal immersive cultural heritage contents that employ inter-connected and—linked sound, audio, visual, and digital reality message elements to engage museum visitors. Finally, mobility and location-specificity could refer to digital reality technologies enable museum visitors to access cultural heritage contents through remote access points (i.e., mobility) or location-based (i.e., location-specificity) an AR museum guidance system. Many other museum researchers such as [59] have similarly argued that these technological characteristics have offered a “fundamental shift” for many art exhibition and museum professionals to create innovative cultural experiences for museum visitors. As an increasingly popular reality-creating technologies, augmented reality has the capabilities to “dynamically blend real-world environments and contextbased digital information” ([49], p. 59) when museum visitors are able to interact with computer-generated digital graphics, imagery, and objects with an illusion of co-presence in the same space [34, 36, 45, 63–65]. One of the examples is the Greenway Conservancy’s Augmented Reality (AR) exhibition (https://youtu.be/fnBmyumEJK0) in Boston [34, 36]. This AR project demonstrates the characteristics of digital reality technology to overlap physical surroundings in modern time with historical photos of downtown Boston in the 1950s and 1960s [34, 36]. This AR exhibition superimposes interactive present day digital elements with decade old cultural artefacts to engage “people in meaningful experiences, interactions, and dialogue with art and each other” ([54]). To allow visitors to move around at different locations of interest and on their own, this digital reality

116

Y. Kang and K. C. C. Yang

project also takes advantage of the mobility associated with mobile technology to enable users to download a free Hoverlay app to their mobile phone to access sixteen AR application access points where present and past Boston landmarks can be viewed by museum visitors [7, 34, 36]. For example, visitors’ experiences can be further enhanced by descriptive narratives that immerse viewers with multi-sensory and – platform interactions with these cultural artefacts [36]. Additionally, this self-guide function is also used by the Lativian National Museum of Art that museum visitors can use their mobile devices or tablets to guide themselves through different museum exhibits (https://youtu.be/fJiaZBfADrs). This functionality of augment reality applications allow museums to offer “additional virtual space to tell stories while allowing room for individual interpretation” [37].

8.2.3 Framing Digital Reality Technologies by Empirical Effects on Visitors’ Museum Experiences Third, scholars have employed a variety of empirical methods to examine the positive and favorable effects of these emerging technologies and their applications on museum visitors’ experiences that are considered most critical among art exhibitors and museum curators [9] employed a between-subject quasi-experiment design to study the effects of an AR-guided, audio-guided, and non-guided device in an art appreciation context among 135 college participants. Their findings confirm the advantage of an AR technology, in comparison with other traditional self-guide application, to positively improve exhibition visitors’ overall learning effectiveness, flow experience, higher engagement in their gallery experience, and the amount of time visitors spent on the paintings [9, 36]. Scholars have also framed these technologies by examining the benefits of their technical attributes. For example [29], investigated whether the novelty effects of AR applications could have on museum visitors’ attitudes and behaviors to learn the technology [35]. [10] similarly studied whether the implementation of digital reality technologies accessible through their head-mounted display devices and beacons can enhance museum visitors’ overall cultural experiences [34, 35]. Other academic researchers have often similarly framed digital reality technologies by focusing on whether modality of accessing museum collections could also have effects on visitors’ museum experiences [31]. Their study examined whether audio guides, AR presentation, interactive terminals, smartphone, or touch-screen tablets will produce different visitors’ behavioral intention to revisit [31]. Additionally, the modality of telling engaging stories about museum collections is also examined. For example [6], investigated the effects of using engaging narratives to overlay with museum artefacts to enrich and enhance visitors’ overall museum experiences [35]. In recent years, more and more scholars have begun to explore and frame the psychological impacts of digital reality technologies [34]. For example, [14] conducted 21 in-depth qualitative interviews to explore whether AR, MR, VR,

8 Framing Digital Reality Technology Applications Among Museums …

117

and other emerging 360° videos can be instrumental to improving museum visitors’ cognitive, play, and functional values after analyzing the participants’ experiential narratives collected from their study [34–36].

8.3 Research Method and Computational Data Analysis The objectives of this computational text mining study aim to identify important recurrent keywords, key phrases, and topics that were used to frame how digital reality technologies have impacted the museum section. And, in response to these challenges and opportunities, how museum professionals have developed their visions and best practices. This text mining study has employed QDA Miner 5.0 and its affiliated WordStat 8.0 programs to analyze words, phrases, themes, topics, and structures from mass media corpus that will offer both qualitative and quantitative data to the proposed research questions. This study particularly focuses on the framing of digital reality technologies as discussed in major media organizations around the world. As an emerging research and analytical method, Touileb and Salaway [55] observed the powerfulness of the text mining technique to allow researchers to conduct discoursebased analysis of a large corpus. Additionally, this technique offers a feasible solution for scholars to identify and study news frames without human interference as coders of the discourses or media contents [36, 38]. Instead, the text mining technique can automatically generate frames without the problems encountered by scholars of traditional framing analysis, such as a small data set or researchers’ subjective, error-prone, frequently biased interpretation [36]. To compile the media corpus collected from the mainstream English-language media, the authors used the following keywords to search Lexis/Nexis Academic (now Nexis Uni) database. These media discourses will be analyzed to understand how the media organizations have framed digital reality technologies and their applications by museums around the world, we conducted a comparative text mining analysis of mass media articles collected from Nexis Uni database. The authors used the following keywords and phrase pairs to compile the media corpus for later analysis. A total of 11 articles were generated from the search [“extended reality” and “COVID-19” and “museums”], while 55 articles were from [“mixed reality” and “COVID-19” and “museums”]. On the other hand, 302 articles were from [“augmented reality” and “COVID-19” and “museums”], 532 articles were from [“virtual reality” and “COVID-19” and “museums”]. Overall, total of 584 articles were from international media outlets, while 316 articles were from U.S. media outlets (Refer to Table 8.1).

8.4 Findings The previous discussion has provided answer to the first research question to explain the current state of augmented (AR), mixed (MR), virtual reality (VR), or extended

118

Y. Kang and K. C. C. Yang

Table 8.1 Characteristics of the media corpus Lexis-nexis academic (Nexis-Uni)

Total (after removing duplication)

International

US

Search terms “Extended Reality” and COVID-19 and museums

11

5

6

“Digital Reality” and COVID-19 and museums

00

0

0

“Augmented Reality” and COVID-19 302 and museums

178

124

“Virtual Reality” and COVID-19 and museums

532

352

180

“Mixed Reality” and COVID-19 and museums

55

49

6

900

584

316

Total Source The authors

reality (XR) applications by museums around the world in the face of COVID-19. We then conducted text mining analysis of the collected articles to understand how media outlets and professionals have framed augmented (AR), mixed (MR), virtual reality (VR), or extended reality (XR) applications by museums around the world (RQ2). Particularly, we focused on the identification of the recurrent keywords and key phrases related to the applications of these digital reality applications by museums around the world (RQ3). In order to answer our second research question, we employed the extraction of keywords, key phrases, or key terms to estimate their relative importance in relation to the overall media discourses by examining the Term-Frequency (TF) or TF-IDF (Term-Frequency-Inverse document Frequency) statistics, commonly used in the text mining literature [34–36, 52, 65]. Both TF and TF-IDF statistics enables us to identify recurrent keywords or keyphrases that stand out to demonstrate their prominence and importance [35]. To visualize the data, text mining researchers also rely on the technique, called word cloud analysis, to demonstrate the relative frequency of keywords, key phrases, and key terms in a graphical manner [35, 50]. Our second research question aims to identify recurrent keywords and key terms that are used by media outlets and professionals around the world to frame the applications of AR, MR, ER, and VR by the museum sector. The word cloud figure (refer to Fig. 8.1) demonstrates that, when related to the applications of digital reality technologies by museums around the world during the COVID-19 pandemic, the following keywords prominently appear to demonstrate their importance: art (TFIDF = 46.9, 68.6% of the cases), artists (TF-IDF = 25.1, 40.0% of the cases), gallery (TF-IDF = 16.3, 45.7% of the cases), galleries (TF-IDF = 20.8, 51.4% of the cases), museums (TF-IDF = 15.3, 60.0% of the cases), museum (TF-IDF = 12.3, 71.3% of the cases), exhibition (TF-IDF = 23.4, 28.6% of the cases), VR (TF-IDF = 31.7, 31.4% of the cases), virtual (reality) (TF-IDF = 7.5, 88.6% of the cases), technology (TF-IDF = 14.3, 54.3% of the cases). These recurrent keywords show

8 Framing Digital Reality Technology Applications Among Museums …

119

Fig. 8.1 Word cloud analysis. Source The authors

that the majority of applications of digital reality technology focus on virtual reality technology as demonstrated by the keywords: VR and virtual (reality). Mainly, these applications are used by gallery/galleries, museums/ museum, and exhibition.

8.4.1 Key Phrase Extraction The text mining software package, QDA Miner and WordStat, has offered a userfriendly and powerful tool to extra key phrases from the unstructured texts from the corpus [35]. Using the key phrase extraction procedure will enable researchers to extract the most prominent phrases in the media corpus to “document categorization, clustering, indexing, search, and summarization” (as cited in [35]). As shown in the data, in spite of the hype about extended reality (ER) (TF-IDF = 2.5, 5.7% of the cases), most of the digital reality applications by museums during the COVID-19 Pandemic, are virtual reality (TF-IDF = 6.1, 68.6% of the cases), augmented reality (TF-IDF = 7.0, 42.6% of the cases), mixed reality (TF-IDF = 6.3, 20.0% of the cases). The most common applications include online viewing (TF-IDF = 10.0, 17.1% of the cases), online viewing rooms (TF-IDF = 7.6, 14.3% of the cases), and virtual exhibitions (TF-IDF = 6.1, 17.2% of the cases) (Refer to Fig. 8.2).

120

Y. Kang and K. C. C. Yang

Fig. 8.2 Key phrase extraction. Source The authors

8.5 Discussions and Conclusions 8.5.1 Summary of Key Findings As a framing study, we began our research by summarizing and examining how the applications of digital reality technologies by museum professionals have been framed. Previously, we have identified three key framing practices (i.e., FP) of these applications: FP#1) detailed description of current practices and applications and associated benefits, challenges, and opportunities; FP#2) the examination of the characteristics of AR, MR, VR, and XR technologies and their potential affordance effects on audience’s experiences with cultural artefacts and heritage; FP#3) lastly, the employment of empirical methods to examine desirable effects of digital reality technologies and their applications on museum visitors’ experiences. On the basis of our text mining findings, a predominant tilt toward FP#1 by focusing on the current state and best practices of different digital reality technologies by museums during the COVID-19 Pandemic and FP#3 (i.e., the experience of people (FP#2) (71.4%, TF-IDF = 7.9). Technological features and affordances (i.e., FP#2) are rarely found in our textual data among global media outlets and professionals. It is likely that the benefits of these digital reality technologies or their hypes have been taken for granted by media outlets and professionals around the world.

8.5.2 Implications for Museums in the Age of COVID-19 Pandemic This text mining study attempts to provide a realistic description of how various types of digital technologies have been applied by museum professionals as described by media outlets and professionals around the world. We further conducted several crosscountry comparisons and observes an interesting pattern of diffusion of these technologies from more economically advanced to less developed countries. For example,

8 Framing Digital Reality Technology Applications Among Museums …

121

due to its maturity as a technology, virtual reality has been found to be the most commonly used digital reality platform during the COVID-19 Pandemic among the museum professionals. However, economically and technologically advanced countries such as U.S., U.K. and China still dominate its applications, when compared with less advanced countries such as India, Thailand, and Zimbawei. The same trend can be found in the applications of augmented reality by museums; U.S. and U.K. still dominate its applications with many newly launched AR-based museum projects described above. While new emerged reality-creating platforms (such as mixed reality and extended reality) have increasingly grabbed the attention of museum professionals and scholars, its applications are limited and only among China, U.K., and U.S.A., while museum professionals in less developed countries such as Zimbawei and India have rarely seen their discussions.

8.5.3 Study Limitations This study comes with several limitations that need to be considered when interpreting and understanding these text mining results. We caution that, despite the widespread use of text mining techniques, museum scholars and professionals need to be careful when integrating these insights into their professional practices. Several limitations need to be considered. First, the comprehensiveness of articles in our media corpus should be carefully considered for any type of text mining research [34– 36, 65]. In our study, we have included articles from China Daily (5.8%), Financial Times (U.K.) (2.9%), The Times (U.K.) (2.9%), The Telegraph (U.K.) (6.9%), New York Times (U.S.) (2.9%), among other major and domestic news outlets. Second, in spite of the comprehensiveness of the Nexis Uni database that collects both Englishand non-English language media, we only included English language contents to analyze the applications of digital reality technologies by museums. Third, the methodological and analytical limitation only allow us to focus on the processing of words, keywords, key phrases, and associated dictionaries in identifying recurrent linguistic patterns to generate findings [35, 52]. Additionally, the over-reliance on keywords and key phrases have the problems of reducing their importance in different contexts and should be addressed by statistical procedures [34–36, 52, 65].

References 1. Ashley, L.: Facing the future: how museums are embracing new trends and technologies. Lifestyle (2017). https://www.thenational.ae/lifestyle/family/facing-the-future-how-mus eums-are-embracing-new-trends-and-technologies-1.82108. Accessed 8 Sep 2019 2. Broadcasting & Cable.: People active online* at home during the work day. Broadcast. Cable 151(2), 29 (2021) 3. Brown, M.: Fact check: coronavirus originated in China, not elsewhere, researchers and studies say. USA Today (2020). https://www.usatoday.com/story/news/factcheck/2020/2003/

122

4.

5.

6.

7.

8.

9.

10.

11. 12.

13.

14.

15.

16.

17.

18. 19.

Y. Kang and K. C. C. Yang 2016/coronavirus-fact-check-where-did-covid-2019-start-experts-say-china/5053783002/. Accessed 28 Dec 2020 Bruno, F., Bruno, S., De Sensi, G., Luchi, M.L., Mancuso, S., Muzzupappa, M.: From 3D reconstruction to virtual reality: a complete methodology for digital archaeological exhibition. J. Cult. Herit. 11(1), 42–49 (2010) BusinessWire.: Insights on the extended reality market 2020–2024: COVID-19 industry analysis, market trends, market growth, opportunities, and forecast 2024—Technavio. BusinessWire (2020). https://www.businesswire.com/news/home/20201125005656/en/Insightson-the-Extended-Reality-Market-20201125002020-20201125002024-COVID-202011250 05619-Industry-Analysis-Market-Trends-Market-Growth-Opportunities-and-Forecast-202 01125002024---Technavio. Accessed 28 Dec 2020 Capuano, N., Gaeta, A., Guarino, G., Miranda, S., Tomasiello, S.: Enhancing augmented reality with cognitive and knowledge perspectives: a case study in museum exhibitions. Behav. Inf. Technol. 35(11), 968–979 (2016). https://doi.org/10.1080/0144929X.0142016.1208774 Carlton, B.: Boston’s Rose Kennedy greenway is now one of the largest AR exhibits in North America (2019). https://vrscout.com/news/rose-kennedy-greenway-ar-experience/. Accessed 21 July 2019 Cerqueira, D.: Baidu’s Baike takes museums to every corner of the globe. Jing Cult. Commer. (2019). https://jingculturecommerce.com/baidus-baike-takes-museums-to-every-cor ner-of-the-globe/. Accessed 31 Dec 2020 Chang, K.-E., Chang, C.-T., Hou, H.-T., Sung, Y.-T., Chao, H.-L., Lee, C.-M.: Development and behavioral pattern analysis of a mobile guide system with augmented reality for painting appreciation instruction in an art museum. Comput. Educ. 71, 185–197 (2014) Choi, H.-S., Kim, S.-H.: A content service deployment plan for metaverse museum exhibitionscentering on the combination of beacons and HMDs. Int. J. Inf. Manag. 37(1b), 1519–1527 (2017) Claudio, P., Maddalena, P.: Overview: virtual reality in medicine. J. Virtual Worlds Res. 7(1), 1–34 (2014) Coates, C.: Virtual reality is a big trend in museums, but what are the best examples of museums using VR? Museum XR (2020a). https://www.museumnext.com/article/how-mus eums-are-using-virtual-reality/. Accessed 17 May 2021 Coates, C.: How museums are using augmented reality. MuseumNext.com (2020b). https:// www.museumnext.com/article/how-museums-are-using-augmented-reality/. Accessed 26 Dec 2020 Collin-Lachaud, I., Passebois, J.: Do immersive technologies add value to the museumgoing experience? An exploratory study conducted at France’s Paléosite. Int. J. Arts Manag. 11(1), 60–71 (2008) Cook, A.V., Jones, R., Raghavan, A., Saif, I.: Digital reality: the focus shifts from technology to opportunity: tech trends 2018. Deloitte Insights (2018). https://www2012.deloitte.com/us/en/ insights/focus/tech-trends/2018/immersive-technologies-digital-reality.html. Accessed 9 Sep 2019 cuseum.: How museums and attractions are preparing and responding to coronavirus. Cuseum (2020a). https://cuseum.com/blog/2020/2023/2022/how-museums-and-attractions-are-prepar ing-and-responding-to-coronavirus. Accessed 16 May 2021 cuseum.: 4 ways museums can successfully leverage digital content and channels during Coronavirus (COVID-19). Cuseum (2020b). https://cuseum.com/blog/2020/3/24/4-ways-mus eums-can-successfully-leverage-digital-content-and-channels-during-coronavirus-covid-19. Accessed 16 May 2021 Deuze M.: Convergence culture in the creative industries. Int. J. Cult. Stud. 10(2), 243–263 (2007) eMarketer.com.: Chart: virtual and augmented reality device shipment and sales share worldwide, by device type, 2022 (% of total). eMarketer.com (2018a). http://totalaccess.emarketer. com/chart.aspx?r=219212. Accessed 1 Dec 2018

8 Framing Digital Reality Technology Applications Among Museums …

123

20. eMarketer.com.: Chart: UK smartphone users who prefer using smart glasses versus smartphone for select augmented reality activities, April 2018 (% of respondents). eMarketer.com (2018b). http://totalaccess.emarketer.com/chart.aspx?r=220210. Accessed 1 Dec 2018 21. eMarketer.com.: Industries in which augmented reality users in select countries in Western Europe have used AR, Nov. 2017 (% of respondents). eMarketer.com (2018c). http://totala ccess.emarketer.com/chart.aspx?r=219349. Accessed 1 Dec 2018 22. eMarketer.com.: Ownership of VR headsets among US internet users, July 2014–Aug 2018 (% of respondents). eMarketer.com (2018d). http://totalaccess.emarketer.com/chart.aspx?r= 224365. Accessed 1 Dec 2018 23. eMarketer.com.: Executives in select countries whose companies are experimenting with versus. Implementing AR & VR for industrial use, by country (% of respondents, June 2018). eMarketer.com (2018e). http://totalaccess.emarketer.com/chart.aspx?r=222816. Accessed 19 Dec 2018 24. eMarketer.com.: Usage and awareness of augmented reality among internet users in select countries in Western Europe, Nov 2017 (% of respondents). eMarketer.com (2018f). http://tot alaccess.emarketer.com/chart.aspx?r=219347. Accessed 1 Dec 2018 25. Farago, J.: Virtual reality has arrived in the art world. Now what? The New York Times (2017). https://www.nytimes.com/2017/2002/2003/arts/design/virtual-reality-has-arrived-inthe-art-world-now-what.html. Accessed 20 July 2019 26. Forbes, T., Kinnell, P., Goh, M.: A study into the influence of visual prototyping methods and immersive technologies on the perception of abstract product properties. In: Paper presented at the NordDesign: design in the Era of Digitalization, NordDesign 2018 (2018) 27. Gamson, W., Modigliani, A.: Media discourses and public opinions on nuclear power: a constructionist approach. Am. J. Sociol. 95, 1–37 (1987) 28. Hernández, J.C.: China peddles falsehoods to obscure origin of COVID pandemic. The New York Times (2020). https://www.nytimes.com/2020/2012/2006/world/asia/china-covid-originfalsehoods.html. Accessed 28 Dec 2020 29. Hunsucker, A., Baumgartner, E., McClinton, K.: Evaluating an AR-based museum experience. Interaction 15(4), 6 (2018) 30. Itzkowitz, L.: How the coronavirus pandemic is affecting museums. Architect. Dig. (2020). https://www.architecturaldigest.com/story/covid-2019-impact-on-museums. Accessed 29 Dec 2020 31. Jarrier, E., Bourgeon-Renault, D.: Impact of mediation devices on the museum visit experience and on visitors’ behavioural intentions. Int. J. Arts Manag. 15(1), 18–29 (2012) 32. Javornik, A.: Augmented reality: research agenda for studying the impact of its media characteristics on consumer behaviour. J. Retail. Consum. Serv. 30, 252–261 (2016) 33. JPost Editorial.: Virtual tourism, business and education take off amid coronavirus. The Jerusalem Post (2020). https://www.jpost.com/jpost-tech/virtual-tourism-business-and-educat ion-take-off-amid-coronavirus-621652. Accessed 28 May 2021 34. Kang, Y.W.: The applications of digital reality in creative and oceanic cultural industries: The case of Taiwan. In: Yang, K.C.C. (ed.) Cases on Immersive Virtual Reality Techniques, pp. 269–296. IGI-Global Publisher, Hershey, P.A. (2019) 35. Kang, Y.W., Yang, K.C.C.: Privacy concerns about the VR and AR applications in creative cultural industries: a text mining study. In: Loureiro, S.M.C. (ed.) Managerial Challenges and Social Impacts of Virtual and Augmented Reality (Chapter 9), pp. 142–164. IGI-Global Publisher, Hershey, P.A. (2020). ISBN 13:9781799828747 36. Kang, Y.W., Yang, K.C.C.: Employing digital reality technologies in art exhibitions and museums: a global survey of best practices and implications. In: Guazzaroni, G., Pillai, A.S. (eds.) Virtual and Augmented Reality in Education, Art, and Museum (Chapter 8), pp. 139– 161. IGI-Global Publisher, Hershey, P.A (2019). ISBN 13:9781799817963. https://doi.org/10. 4018/978-1-7998-1796-3 37. Klavins, A.: 7 augmented reality ideas for interactive museum experiences. Overly (2020). https://overlyapp.com/blog/2027-augmented-reality-ideas-for-interactive-museum-experi ences/. Accessed 17 May 2021

124

Y. Kang and K. C. C. Yang

38. Lin, F.-R., Hao, D., Liao, D.: Automatic content analysis of media framing by text mining techniques. In: Paper presented at the 2016 49th Hawaii International Conference on System Sciences, Hawaii, USA (2016) 39. Mariani, G.: The cultural and creative industries (2018). https://www.guillaume-mariani.com/ creative-industries/ Accessed 13 Dec 2018 40. Mateus, S., Branch, J.: Intelligent virtual environment using a methodology oriented to agents. In: Lackey, S., Shumaker, R. (eds) Virtual, Augmented and Mixed Reality. VAMR 2016. Lecture Notes in Computer Science, vol. 9740. Springer International Publishing/Springer, Cham, Switzerland (2016) 41. Ministry of Culture [Taiwan].: 2017 Taiwan cultural & creative industries annual report. Ministry of Culture, Taipei, Taiwan (2017). http://cci.culture.tw/upload/cht/attachment/b131e5 55ec34a192be359838c9a4eb07.pdf. Accessed 13 Dec 2018 42. Next Animation Studio.: Next animation studio partners with national museum of prehistory (2017). https://eprnews.com/next-animation-studio-partners-with-national-museum-of-prehis tory-87380/. Accessed 15 Dec 2018 43. Novak, A.N., Hakenan, E.A.: Framing theory. In: Harvey, K., (ed.) Encyclopedia of Social Media and Politics. Sage Publications, Inc., Thousand Oaks. C.A (2013) 44. PR Newswire.: COVID-19 social distancing and remote workforces will accelerate growth in immersive XR collaboration revenue to over US$400 Million by 2025. PR Newswire (2020). https://www.prnewswire.co.uk/news-releases/covid-2019-social-distancing-and-remote-wor kforces-will-accelerate-growth-in-immersive-xr-collaboration-revenue-to-over-us-2400-mil lion-by-2025-879003738.html. Accessed 28 Dec 2020 45. Rese, A., Baier, D., Geyer-Schulz, A., Schreiber, S.: How augmented reality apps are accepted by consumers: a comparative analysis using scales and opinions. Technol. Forecast. Soc. Chang. 124, 306–319 (2017) 46. Ryan, M.D., Hearn, G.: Next-generation “filmmaking”: New markets, new methods and new business models. Media Int. Australia 136, 133–145 (2010) 47. Scheufele, D.A.: Framing as a theory of media effects. J. Commun. 49(1), 103–122 (1999) 48. Smithsonian American Art Museum (SAAM).: Smithsonian American art museum “Beyond The Walls”. https://store.steampowered.com/app/1087320/Smithsonian_American_Art_Mus eum_Beyond_The_Walls/. Accessed 16 May 2021 49. Sommerauer, P., Müller, O.: Augmented reality in informal learning environments: a field experiment in a mathematics exhibition. Comput Educ 79, 59–68 (2014) 50. Srivastava, T.: Build a word cloud using text mining tools of R. Anal. Vidhya (2014). https:// www.analyticsvidhya.com/blog/2014/2005/build-word-cloud-text-mining-tools/. Accessed 25 Apr 2019 51. T HQ.: Extended reality—The technology of the future. T_HQ: Technol. Bus. (2020). https:// techhq.com/2020/2012/extended-reality-the-technology-of-the-future/. Accessed 29 Dec 2020 52. Teso, E., Olmedillab, M., Martínez-Torresc, M.R., Toral, S.L.: Application of text mining techniques to the analysis of discourse in eWOM communications from a gender perspective. Technol. Forecast. Soc. Change 129, 131–142 (2018) 53. The Gold Museum.: AR exploration of Shu-Jin-Jeu gold mine (2021). https://popworld.cc/gol dmuseum/event. Accessed 27 May 2021 54. The Green Way.: Past, present, and future meet on the greenway! (2019). https://www.roseke nnedygreenway.org/. Accessed 21 July 2019 55. Touileb, S., Salway, A.: Constructions: a new unit of analysis for corpus-based discourse analysis. In: Aroonmanakun, W., Supnithi, T., Boonkwan, P., (eds) Proceedings of the 28th Pacific Asia Conference on Language, Information and Computation (PACLIC 28), pp. 634– 644. Chulalongkorn University (2014) 56. Touri, M., Koteyko, N.: Using corpus linguistic software in the extraction of news frames: towards a dynamic process of frame analysis in journalistic texts. Int. J. Soc. Res. Methodol. 18(6), 601–616 (2015). https://doi.org/10.1080/13645579.2014.929878 57. UNESCO.: Museums around the world in the face of COVID-19. UNESCO (2020)

8 Framing Digital Reality Technology Applications Among Museums …

125

58. Vliegenthart, R.: Framing in mass communication research: an overview and assessment. Sociol. Compass 6(12), 937–948 (2012) 59. Walker, J.: Virtual reality to create an ’Internet of experience’ by 2030. Dig. J. (2017). http://www.digitaljournal.com/tech-and-science/technology/virtual-reality-to-createan-internet-of-experience-by-2030/article/497608. Accessed 3 Aug 2018 60. Watson, D.M.: Playing with art: augmented reality site allows people to transplant and play with sculptures around Norfolk and the world. Virginian Pilot (2020). https://www.pilotonline. com/entertainment/arts/vp-db-globalsculpturepark-norfolk-20201221-d20201273x20201257 c20201222rcdrgxruidtigl20201724-story.html. Accessed 30 Dec 2020 61. Worldometer.: COVID-19 coronavirus pandemic. Worldometer (2020). https://www.worldo meters.info/coronavirus/. Accessed 28 Dec 2020 62. Wortley, D.: Immersive technologies and personalised learning: The influence of games-related technologies on 21st century learning. In: Paper presented at the Proceedings of the 4th Annual International Conference on Computer Games, Multimedia and Allied Technology, CGAT 2011 and 2nd Annual International Conference on Cloud Computing and Virtualization, CCV, pp. 74–78 (2011) 63. Yang, K.C.C.: Preface. In: Yang, K.C.C. (ed.) Cases on Immersive Virtual Reality Techniques, pp. 1–16. IGI-Global Publisher, Hershey, P.A. (2019) 64. Yang, K.C.C.: Reality-creating technologies as a global phenomenon. In: Yang, K.C.C. (ed.) Cases on Immersive Virtual Reality Techniques, pp. 1–18. IGI Global Publisher, Hershey, P.A. (2019) 65. Yang, K.C.C., Kang, Y.W.: Integrating virtual reality and augmented reality into advertising campaigns: history, technology, and future trends. In: Lee, N., Wu, X.-M., El Rhalibi, A., (eds) Encyclopedia of Computer Graphics and Games. Springer, New York, N.Y (2018)

Chapter 9

Guided Digital Tours and Covid-19. Comparison of Cases in Europe and Definition of Models Grounded on Data Patrizia Schettino Abstract What is a virtual tour? Is a virtual tour the same as an online guided tour? When museums throughout Europe were closed during the lockdown they tried to offer their public digital experiences. This paper will analyse four cases (in Belgium, UK and Italy) and will attempt to identify and describe the different communication strategies and user experience models they adopted. The paper is based on notes collected using qualitative methods (case study, digital ethnography, narrative inquiry, grounded theory). Focus will be placed on the guided tours organized by Gallerie Estensi (Italy). The paper is also an attempt to define what an online guided tour is and the key elements of its design.

9.1 Introduction This chapter aims to define what an “online interactive guided tour” is and identify a model grounded on data. It also aims to highlight the differences between a “virtual tour” and “an interactive online guided tour”. The two words are commonly used as synonyms but often describe a different design and user experience. In the first section the theory used as theoretical background for this study will be described briefly and the results of recent studies conducted by Nemo during the first lockdown in Europe will be indicated. During the lockdown, many museums advertised “virtual/guided tours” on their social media. Our aim is to understand the difference between virtual and guided tours and identify the different types of guided tours. The method used to collect the data is a combination of digital ethnographic observation, based on digital ethnography, active participation in online guided tours, case study analysis, narrative inquiry and grounded theory. The method is described in the second section. P. Schettino (B) Accademia di Belle Arti di Brera, Milan, ICOM, Italia e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 A. S. Pillai and G. Guazzaroni (eds.), Extended Reality Usage During COVID 19 Pandemic, Intelligent Systems Reference Library 216, https://doi.org/10.1007/978-3-030-91394-6_9

127

128

P. Schettino

In the following sections, four cases will be analysed to identify similarities and differences, and the key design elements are described. The conclusions summarize the three main types of guided tours emerging from this study.

9.2 Theoretical Background Definition of virtual tour/digital tour/digital guided tour. The Cambridge Dictionary defines a virtual tour as: • created by computer technology and appearing to exist but not existing in the physical world; • done using computer technology over the internet, and not involving people physically going somewhere. Let us analyse the two words separately.

9.2.1 Virtual Carina Girvan [1] analysed the definition of “virtual”, with the aim of re-defining and classifying virtual words. “Deleuze (1990) conceptualises the virtual as that which is opposed to actual, but not opposed to real, meaning that the virtual is real, or at least our perception of it is. This highlights a trap which many scholars fall into, describing virtual worlds as “not real. (…) virtual is used to describe a simulated experience. This gives a sense of something that is almost real, something which is perceived to exist, but lacks physical properties beyond the screen”. Can this definition apply to museums? Some museums have no physical substance beyond the screen: for example, the Valentino Garavani museum [2] has no actual museum to showcase this worldrenowned fashion designer and his work. The first “virtual museum” was created by Apple: a CD-ROM of an exhibition created using computer graphics [3]. Can this definition also be also applied to “virtual” archaeology and simulated 3D reconstructions of ancient cities, where there are no, or only very limited, traces of the “physical” evidence”? In this case, the simulation is usually represented by a sort of “travel in time”, in that we see a simulation of life at that time and the reconstruction of buildings as interpreted by archaeologists from the ruins and other findings. Bearing in mind this distinction, can a tour of these museums and cities simulated using computer graphics, such as Valentino’s museum, or the reconstruction of completely destroyed places, be considered “virtual”? In other cases, there museums and cities that can be visited where the museum has digitalized its exhibits and the museum space, using Matterport for example: can

9 Guided Digital Tours and Covid-19 …

129

those institutions create a “virtual tour” or a “guided tour” or both? In this case, there is physical substance beyond the screen. Moreover, can we call a tour using digitalized photographic representation of the physical museum space “virtual”? From the visitor’s point of view, there are three very different experiences: • when they know that the space is solely part of a “fiction”, as they know that it is completely simulated and that they cannot visit it physically (because it never existed or it no longer exists in the forms depicted, either in the present or in the future); • when they know that they can visit it in the future; • when they have visited it and they have “memories of their previous experiences” [4]. According to the Cambridge Dictionary [5], the word “virtual” can be used to define the museum experience “without going there”. However, from the visitor’s point of view, this possibility of visiting the “physical” museum or the memories of the physical museum can profoundly change their interpretation, motivation and expectations. Moreover, “the core concept of the virtual museum has not yet been clearly identified, and the delineation of the perhaps subtle difference between digital collections, online archives, and virtual museums have to be clarified” Schweibenz [6]. Antinucci [7] stated what virtual museum is not, from his point of view: “the virtual museum is not the real museum transposed to the web, it is not an archive, it is not what is missing from the real museum”. Three versions of the definition have been proposed by Sofia Pescarin, Victor Menchero Lopez and Augusto Palombini (2013), all based on the ICOM definition of “museum”. The common keywords are: “digital creation”, “communication product”, “tangible or intangible heritage”, “interactivity and immersion”. In the final report, the V-Must consortium (2014) presented four definitions, with this remark: “the definition is still in progress”. For all the reasons given above, a distinction should be made in this paper and “virtual” should not be defined as the experience of digital photographic representations or video recordings of physical museums, temporarily closed for Covid, that can be visited in the future or have already been visited before the lockdown. More in general, what is a tour?

9.2.2 Definition of Tour Tour is defined by the Cambridge Dictionary as: • a visit to a place or area, especially one during which you look around the place or area and learn about. • a journey made for pleasure, especially as a holiday, visiting several different places in an area;

130

P. Schettino

• a planned visit to several places in a country or area made for a special purpose, such as one made by a politician, sports team, or group of performers. The definition has 3 key elements: a place, a journey, a visit. In this paper, focus will be on • tours for “pleasure” and to “lean about”, because pleasure and learning are important elements also in defining what a museum is (ICOM, Kyoto, General Assembly, discussion on the definition of a museum); • online tours from home, using digital media [8]. As the analysis concentrated on online experiences, it can be said that all these experiences are digital, they are mediated by a device (pc, tablet, mobile phone, etc.) and they imply that some/all museum content is digitalized. At present, these mediated digital experiences cannot encompass all the senses, generally combining only visual and audio elements, but research on interfaces is being carried out to try to include other senses (see in the ICOM glossary) [9]. In this paper, the focus will only be on digital tours, based on digital photographic copies or video recordings, and not on 3D spaces, designed or augmented using 3D computer graphics. Therefore, the word “virtual tour”, but digital tour, will be used to make a clear distinction between an experience in 3D designed spaces and digital spaces that are copies of “physical” spaces. In the glossary published by ICOM Italia, “virtual tour” includes both tours in photographic and 3D reconstructed spaces, because this is the most common use of this word in the museum community: many museums use the word virtual tour to describe what are actually digital tours, but a theoretical difference will be made in this paper [9]. The visitor experience and the design are different, for example, if the user is inside a 3D model such as a 3D reconstruction of a destroyed church, or if he/she is exploring the panorama of the physical ruins of the church. It is important to highlight this difference, making the term virtual tour less ambiguous.

9.2.3 Guided Tour A guided tour can be virtual or not, digital or not, augmented or not. The tour can be experienced by means of visitor self-exploration, without any direct interaction with another human, but mediated by a media or multimedia or a guided tour conducted by a human (usually an educator). The main difference is that the user is free to define path, order, time, duration and she/she can directly choose what element to see/listen to, usually either by clicking on the elements or following a “guide”. In the second case, the key elements are a path in the space and a narrative, as an interpretative tool to support the visitor’s interpretation process. The visitor is not alone; he/she interacts with a guide: the interaction model and the visitor experience is not based on selfexploration but on “following” a set path in the space and an interpretation. Usually,

9 Guided Digital Tours and Covid-19 …

131

the guide’s goal is to share his/her interpretation of an object with the visitors, telling some stories linked to the object. The guide could also merely be a recorded voice, but there is on-going dialogue between visitor and guide, because the visitor decides and accepts to follow the path and to listen to the interpretation proposed by the guide. During the first lockdown, many museums advertised “virtual tours” on their social media and websites. Some of them could be more accurately defined as guided digital online tours of physical spaces (for example a guided tour by a director of an exhibition). To better understand the experience and to see if there are different categories of “digital guided tours”, different cases in Europe will be compared, triangulating data from four cases (UK, Belgium, Italy). The main common elements of those four cases are: 1. 2. 3. 4. 5. 6. 7.

Narrative of a human (curator, director, educator, etc.); Use of a video, recorded live during an online event, via a social media or video call (Zoom, Meet, etc.); Possibility of seeing other participants; Possibility or not of asking questions in a chat; Possibility of asking questions using other social media (Twitter), selected by the moderator; Possibility of seeing different visual media, with different levels of detail; Interaction model based on “following” a set path without self-exploration. I Will not Include:

• tours in 3D environments without a physical museum behind the screen, where the visitor can self-explore a space, without following a set path but choosing which and how many elements to explore and in what order; • digital tours with photographic copies of real spaces, but based on self-exploration (for example, exploration of museums digitalized using Matterport where the visitor can explore a digital photographic copy of the museum’s collections and rooms). Usually, the visitor can click in the space, read content, watch videos, see details of an image; • visits in situ, augmented with augmented reality, for example using 3D glasses, because the focus of this paper is online digital events, organized by the museums when they were closed due to the COVID-19 health emergency; • guided tours by avatars and AI, both in virtual museums or digital copies of physical museums. These will be discussed in a future paper, when there is the chance of performing digital ethnography in a specific case. 9.2.3.1

Virtual Tours, Guided Tours and Other Online Activities from the Nemo Survey 2020

During the lockdown, Nemo and ICOM conducted surveys on museums and their activities [10]. The data of both surveys show an increase in digital activities, including virtual tours and live events.

132

P. Schettino

The Nemo survey showed that museums saw an increase in visits to their websites (in some cases up 500%) and they made an effort to produce or have planned to produce new content for their websites and social media in the near future [11]. Many museums opened an Instagram profile for the first time and they organized “virtual” tours. The International Council of Museums (ICOM) collected data during a survey of digital activities in museums during and after the lockdown. “The forced temporary closure of museums during lockdowns suddenly brought digital communication with the public to the fore. We witnessed a surge in virtual tours, social media posts, remote interactions with the public and much more.” Table 9.1 is the following table is highly relevant for this paper, because it shows that museums increased the number of live events, quizzes and social media more than other digital services. The International Council of Museums (ICOM) collected data with a survey between 7 April and 7 May 2020: 1,600 responses from museums and museum professionals, in 107 countries and across continents. Five themes were covered: the current situation for museums and staff, expected economic impact, digital resources and communication, museum security and conservation of collections, freelance museum professionals. These surveys show that it is important to better understand what happened: how museums reacted to the lockdown and how and what digital services they produced, comparing the quantitative data on virtual tours and live events with more in depth qualitative data about specific cases. Table 9.1 Changes in digital services, Museum, museum professionals and COVID-19, survey results, copyrights: ICOM Collection online (%)

Online exhibitions (%)

Live events (%)

Newsletters (%)

Podcasts (%)

Quizzes contests (%)

Social media (%)

No

33,31

49,46

56,47

31,66

68,26

55,15

7,67

Yes, same as before

43,86

22,18

11,54

52,18

14,67

15,33

42,21

Yes, increased after lockdown

17,97

16,16

18,80

13,36

10,39

19,21

47,49

Started after the lockdown

4,04

10,88

12,28

1,90

5,11

8.57

1,98

9 Guided Digital Tours and Covid-19 …

133

9.3 Methodology I have used a combination of 4 qualitative methods: digital ethnography [12, 13], case study [14], narrative inquiry [15] and grounded theory [16, 17]. I took part as a visitor in several events organized by museums online. Of these events, I selected four guided tours, held in Facebook or via video call (Zoom and Meet), with the purpose to compare them [18]. I was an active user, asking questions and making comments. I took notes about my feelings and my behaviour and about the behaviours of the other participants. I followed all the events organized by four institutions in UK, Italy and Belgium and these are the case studies for this paper. I triangulated the observations from these cases. In the case of the Gallerie Estensi, I also compared (triangulation, [18]) what emerged from the notes with the interview with the museum director and a museum educator.

9.4 Case Studies 9.4.1 Flemish Masters, Different Locations Flemish Masters 2018–2020 is a Facebook page and website where it was possible to attend several guided tours during the lockdown [19]. As the claim says, it is a “personal preview of the real thing”. The link between the “tour” and the goal of creating a desire to visit the physical objects is very well summarized. It is not virtual but a personalized digital preview of “physical” museums, people and objects. https://www.flemishmasters.com/it/eventi/stay-at-home-museum#viewport The idea of a “stay at home museum” is effective: it is a museum that can be experienced from home, as a preview. It offered 5 guided visits of 5 museums with 5 directors/curators: • • • • •

Jan van Eyck- MSK Gent Bruegel- Museum of Fine Arts Brussels Rubers- Rubens House Ensor-MU ZEE Hof van Busleyden.

They were live online events on Facebook, recorded and shared later on the website. The live visit with the director or curator was a walk through the museum in a liverecorded video. After the video, the director or curator was available in the Facebook chat to answer questions. The question and answer session was not mediated, the director or curator directly chose questions from the chat and gave an answer. It was not possible to ask questions during the guided tour but only after the presentation. The content was the video of a museum with some focus on objects and the narrative of the curator, without the possibility of interacting with the director or curator during

134

P. Schettino

the visit. He/she followed his/her path, without changing it based on the questions. The video content is free and is still available on the website. In the case of the Jan van Eyck exhibition, it was the only possibility for visitors to see this temporary exhibition, where for the first time all Jan van Eyck’s masterpieces were located in the same museum, as the opening was just a few days before the lockdown and it closed before the end of the lockdown. I asked questions in the chat after two tours and received answers. The dialogue in the chat with the director or curator was the only interactive part of the visit, but it was very well conducted and there was open discussion. It was not possible to use audio, all the questions were written in the chat.

9.4.2 The Courtauld Gallery, UK, London The museum is located in London. “The collection stretches from the early Renaissance into the twentieth century” and is displayed in the elegant setting of Somerset House [20]. The Gallery is particularly renowned for its collection of Impressionist and Post-Impressionist paintings. In Spring, the museum organized a live event in Zoom for a limited number of visitors. During the event it was not possible to use the chat but the moderator, working from home, invited us to ask questions on Twitter and at the end of the event she selected a few of them and asked the curators or other experts to answer. All the experts were present from their homes. The presentation was made of topics and not about a specific master or object, by 3–4 people who worked for the gallery or were related to it, for example PhD students doing research at the gallery. The main content were slides, photographs and the narrative of the experts. It was not possible to see the gallery spaces live. It could be considered more of a “talk” than a guided tour because the key element of a tour was missing, i.e. following a path in a space. The gallery called them “episodes”. The videos are available on the gallery’s website and on YouTube. Some of the experts also included photographs of objects in the museum in their narrative. In a very broad sense, this can be considered a kind of guided path among objects inside and outside the gallery, but without showing the museum space or moving from room to room. The museum also has full digitalization of the rooms, available on the website, but this was not included by the experts in their narratives and presentations during the events. The list of the live events: • • • •

Art in Isolation. Thursday 30 April; Art and Wellbeing. Thursday 7 May; The Future of Art History. Thursday 14 May; Women Artists. Thursday 21 May.

9 Guided Digital Tours and Covid-19 …

135

9.4.3 Villa Giulia, “Dirigi il Direttore”, Italy, Rome The Museo Nazionale Etrusco di Villa Giulia has in his collections objects from Etruscan civilization [21]. During the lockdown, the museum director, Valentino Nizzo, organized a guided tour in the physical space of the museum and “on the road”, driving through the city and talking about Rome. He used a Facebook live event and recordings, combined with the chat. The interaction between visitors and the director was not only questions, but also indications about where to go and what objects to choose to talk about. It was a kind of “improvisation theatre” where the director chose an area of the museum, but the visitors decided the topics and objects to include in his narrative. The technology was very simple, but used in a very original way. The key element was the use of the chat to collect the public’s directions, “Dirigi il direttore” means “to direct the director”. The quality of the video and audio was quite low, without possibility to zoom in details, but the quality of the narrative by the director was the most relevant part as educational content.

9.4.4 Gallerie Estensi, Italy, Modena Gallerie Estensi is a network of museums [22]. The art collections include works by Bernini, Velázquez, Correggio, Guercino, El Greco, Guido Reni, Cosmè Tura, Dosso Dossi, Mantegna and Carpaccio, a historic library with masterpieces such as Borso d’Este’s Bible, the Museo Lapidario with remains from the ancient Roman town of Mutina, and the ‘baroque delight’ that is the Palazzo Ducale at Sassuolo. Gallerie Estensi offer visitors a comprehensive overview of intellectual and cultural life in the Este states between Ferrara and Modena. During the first lookdown in Italy, Gallerie Estensi organized a program of online guided tours. The model was invented by the director who decided to involve educators from an external company, Prospectiva. The model involves the use of Google Meet, close interaction between educators and visitors, using the live chat, a compelling scenario, two educators to manage the live session, one presenting and the other moderating the chat, the use of digital media during the presentation including documents from the digital library and digital photographic reproduction of the museum spaces, using 3D scanning. From the visitors’ point of view, it was possible to see the presentation of one educator, using all the different media, simulating a museum visit, with Google maps or the digital photographic reproduction of the museum spaces. Zoom Live was also used to explore details of images. The experience was well coordinated, with close interaction between educator and visitor. It was possible to ask questions also using the microphone.

136

P. Schettino

9.4.5 Focus on Gallerie Estensi For this case, I took part in almost all the online guided tours organized from April to July. I was an active participant, asking the educators questions and making comments in the chat. Taking part to these sessions, some patterns started to emerge: 1. 2. 3. 4.

5.

The events created a community; Several visitors commented that the guided tours were greatly appreciated during the lockdown, as a kind of relief from the loneliness they were feeling; Many visitors participated from different countries and asked for the chance to follow the event in English; Older visitors learned for the first time to attend an event in Meet, also making some mistakes, for example clicking on the “share screen” button while the educator was presenting, thus interrupting the session; Visitors suggested topics for future guided tours, for example iconography, the educators reported this to the director and they organized other guided tours around those topics.

One of the keys of the success was the educators’ ability to link the images to the space, using the 3D model and Google maps. The events were so successful that the museum decided to add new 3D scanning of the Palazzo Ducale at Sassuolo and the Pinacoteca di Ferrara, and to hold one of the last events about this building. The successful model designed and tested by Gallerie Estensi was adopted later by other museums, for example the Uffizi. I triangulated my ethnographic observations with the interview with the museum director, Martina Bagnoli [23]. when the lockdown was lifted and also an interview with one of the educators, Lucia Peruzzi. The director confirmed that behind these guided tours there is a very small team of museum employees and external educators from Prospectiva and that they worked very hard to produce the tours. The original idea was to create multimedia sessions in Meet for schools. Meet was used for the first time during the lockdown when the educator worked alongside an expert in communication and one in computer sciences to define a Meet profile for the museum and how to share content with the public, etc. Students and their teachers from around 150 schools throughout Italy, and not only local ones, took part in the online guided tours. MB: “ I invented the guided visits because when we were closed we lost a good many bookings […] I thought we could play a role, assisting teachers who seemed to be in difficulty and who in a short time had been forced to transfer in-presence lessons online without having adequate structures and, since we had 3D scans of the Gallerie Estensi I thought that with media like Zoom and Meet, used by the schools, we could offer satisfactory interactive guided visits…the idea was conceived for schools and then we decided to offer it to the public; so, we launched an educational activity and then offered visits at weekends, because during the lockdown and the melancholic atmosphere, talking about beauty could be useful for the public.”

9 Guided Digital Tours and Covid-19 …

137

Later, she decided to offer the guided tours to a wider public, as free cultural experiences in a difficult moment of isolation at home. Many older visitors and those with scant knowledge of video conference tools decided to attend the events. In some sessions, the educators managed 200 visitors connected in Meet, with live chat sessions and questions and answers. The elements of the guided tour, the presentation and moderation methods were also improved event after event. The director, Martina Bagnoli, emphasized that it was possible to use small teams (3 people) to organize the guided tours because the museum already had a very rich archive of digitalized objects and also a full 3D scan of some of the rooms and the library. Before the pandemic struck, the director had decided to digitalize 250,000 books in the library and 15,000 objects in the museum so the educators had ample choice in the existing databases. With this strategy of digitalization, the museum was ready for the challenge of producing new multimedia in a short time and using a small team. Her strategic digitalization plan was a key element in preparing the museum to go online on social media and for the guided tours when the museum was closed. The plan is to produce guided tours for the disabled and in English in the future. MB: “ We started from a wealth of digitalized content, this has been one of the cornerstones of my job. So, we had 15,000 works published online that we could pick from, we have scans of our museums, we have the library platform, launched at the end of June, where visitors can navigate 250,000 pages of manuscripts in high definition […] So we managed to do this because we had a good base to build on […] we improvised the vehicle on which to channel the content but brick by brick we got there […] this allows us to be more flexible in any emergency […] this is part of the museum’s planning, the result of years of work, for the library it is a huge job, involving 50 people; so counting on the people involved, during the emergency the content already available was coordinated and we found a suitable vehicle…” . The director also gave us her personal definition of a guided tour: MB: “ It is a way for a person to experience a work of art in a way that carries some significance. We know that visits often have little effect on users besides clicking on something in a view while at other times they are bowled over by their encounter with a work of art or object […] our intention is to create, promote and mediate this type of encounter […] this is a guided visit…we want guided visits to be visitor friendly, listening to visitors […] not in the that we say what they want to hear but that we say it in a comprehensible and interesting way.” The educator from Prospectiva also explained what her idea of a guided tour is. The key element is the path in the space (for Gallerie Estensi, using Matterport or Google). Comparing the online experience with guided tours in physical museums, she had the chance to combine several media during her presentation which she usually cannot do during a visit in the physical museum. On some guided online tours, for example about colours, she designed a path through locations, in different cities, and this is possible only online. She said that she sees pupils, especially secondary school ones, more engaged with the online guided visit. In her opinion, an online guided visit should not be just a video of a person speaking in front a work

138

P. Schettino

of art. She believes that interaction with the visitors, using the chat, is important. She worked with a second educator, who moderated the chat: their aim was to give answers immediately without waiting until the end for questions and answers. This was challenging but it allowed her to personalise the guided visit, giving more space and time to specific content. An interesting element emerged from her narrative about the experience of guiding visitors online: her body language became more important and she realised that she had to use her face and voice in a certain way to make the visit more effective. Her voice was of utmost importance, to keep the visitors’ attention and make the content clear. Based on their definitions, a guided tour is also multimedia created with the intention of facilitating the encounter between an object in a museum and the visitor. A guided tour is one of the tools used by the museum as an educational activity. The main hope is that the visitor can learn about the object, listening to the educator’s narrative, having a brief experience of the digital objects available online on the museum’s website and in the library. The visitor can develop their own interpretation, starting from these stories, and come back later to explore the digital resources or to the museum on their own (after the lockdown). These online guided tours can also be preparation for future online and in situ visits.

9.4.6 Comparison of the Different Online Guided Visit Models and Their Design Elements Some common elements and some different ones emerged from analysis and comparison of the case studies. All the guided tours analysed included: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

A narrative by a guide (human guide with human aspect, no avatars, no robots); A live event, with the co-presence of guides and participants connected via a tool; Chance to ask questions and write comments in real time A path proposed by the guide with touch points (key objects from the collections); Some form of interaction with the guide; Use of one medium or several media; Possibility of seeing the recording after the live event or not; A defined plot for the narrative or decided on the spot, following the visitors’ preferences; The use of space; One or more educational objectives.

The main differences are the possibility of interacting with the guide: the possibility of asking questions during the presentation and of having an immediate answer (Gallerie Estensi), with two educators—one moderating the chat and one presenting (Flemish master)—or of asking questions on a separate medium (Twitter, without a live chat, Courthaud Gallery) or of directing the director’s narrative with the visitors’

9 Guided Digital Tours and Covid-19 …

139

preferences and questions from the live chat (Villa Giulia) and direct interaction with the museum director. Also the way space was presented differs: the educator can use photographic and interactive images from a 3D scan of a museum and also zoom in on details, or just walk through the museum, recording live with a camera or using Google maps and other online tools to navigate a space live; in other cases the space is not presented (Courthauld Gallery). If we consider the “tour”, movement in the space, digitalized as a 3D scan or panorama or presented live with video recording, the experience offered by Gallerie Estensi, Villa Giulia and Flemish Masters are guided tours while the events of the Courthauld Gallery are more episodes of a festival, talks and not guided tours. Comparing these 4 experiences, we have a design model for the online guided tour. A possible other form is a guided tour by non-humans (AI). This could be incorporated in future research because all the cases are from the same period—the first lockdown in Europe—and ones which I was able to join, to conduct digital ethnography as an active participant.

9.5 Conclusions Starting from the definition of virtual, guided and tour, the paper focussed on online guided tours in digitalized spaces with a physical spaces and objects (3D scanned or video recorded). The following classification could be used to summarize the three main types of digital online and live guided tours: 1. 2. 3.

Directed by a human, with a defined path and plot, interacting with other visitors in a live chat; Directed by a human, walking in the real space with live video recording, interacting with visitors after the walk; Semi-directed by a human, with visitors participating in the definition of the path.

If a museum wants to design and communicate an event with these elements, it is more accurate to use the term online guided tour and not virtual tour. Guided implies that the user will follow a narrative and path and that he/she will not self-explore the space and the object during the live session. The definition of “personalized preview” of real things is also an attractive way of advertising these experiences, with the goal also of inviting visitors to use the tools to see a preview and later visit the museum as a physical space with material objects. And, as a visitor commented when collecting data for my PhD in Melbourne, “Now I want to touch the things”, thus expressing their desire to travel to and see not only the digital representations of PLACE-Hampi in Melbourne but also the physical Hampi archaeological site in India. Museums can use guided tours not only to support visitors during the lockdown, reducing isolation and offering the possibility of enjoying culture and beauty, but also

140

P. Schettino

to prepare visitors for their visit after the lockdown and to increase the number of tickets sold after the lockdown, using these tours as both educational and promotional tools. A program of online guided tours can create communities, people who will join more than one event and potentially all the events. In a long term strategy, it is important to have a rich archive of digital objects, including a 3D scan of the museum space, and educators with good communication skills who are able to use the technology and moderate the chat.

9.6 Future Research How effective are the different models of online guided tours? After identifying the possible models, future studies could also include an evaluation by visitors, with surveys but also assessing different aspects such as what they have learnt based on the Hooper Green Hill learning model [24], combining qualitative and quantitative methods.

References 1. Girvan, C.: What is a virtual world? Definition and classification. Educ. Tech Res. Dev. 66, 1087–1100 (2018). https://doi.org/10.1007/s11423-018-9577-y 2. Valentino Garavani virtual museum. https://www.valentino.com/it-it/world-of-valentino/news/ valentino-garavani-virtual-museum-30922 3. Apple CD-Rom.: First virtual museum 4. Schettino, P., Kenderdine, S.: Place-Hampi. Narratives of inclusive cultural experience. J. Inclu. Mus. (Common Ground Publisher) 3(3) (2011), 141–156; Denzin, N.: Strategies of multiple triangulation. In: Schweibenz, N. (ed.) 2019, The Virtual Museum: an Overview of its Origins, Concepts, and terminology. Mus. Rev. 4(1) (1970) 5. Cambridge Dictionary.: https://link.springer.com/article/10.1007/s11423-018-9577-y 6. Schweibenz, W. (1998) in Zimmermann, Harald H.; Schramm, Volker (Hg.): Knowledge Management und Kommunikationssysteme, Workflow Management, Multimedia, Knowledge Transfer. Proceedings des 6. Internationalen Symposiums für Informationswissenschaft (ISI 1998), Prag, 3. – 7. November 1998. Konstanz: UVK Verlagsgesellschaft mbH, 1998. S. 185 – 200 7. Antinucci, F.: “The virtual museum”. In Virtual Museums and archaeology. The contribution of the Italian National Research Council, Ed. P. Moscati, Archeologia e Calcolatori, Suppl. 1, 79–86 (2007) 8. ICOM; museum definition.: https://icom.museum/en/resources/standards-guidelines/museumdefinition/#:~:text=%E2%80%9CA%20museum%20is%20a%20non,of%20education%2C% 20study%20and%20enjoyment 9. Glossary digital museology, ICOM Italia.: http://www.icom-italia.org/wp-content/uploads/ 2020/10/ICOMItalia.CommissioneTecnologie.2020.GlossarioTecnologie.pdf 10. ICOM, Museums, museum professionalsand COVID-19, survey results: https://icom.museum/ en/covid-19/surveys-and-data/survey-museums-and-museum-professionals/ (2020) 11. Nemo Survey.: https://www.ne-mo.org/fileadmin/Dateien/public/NEMO_documents/ NEMO_Corona_Survey_Results_6_4_20.pdf

9 Guided Digital Tours and Covid-19 …

141

12. Boellstorff, T.: Rethinking digital anthropology. In: Miller, D., Horst, H.A. (eds.) Digital Anthropology, pp. 39–60. Berg, London (2012) 13. Boellstorff, T., Nardi.B., Pearce, C., Taylor. T.L.: Ethnography and Virtual Worlds. A Handbook of Method. Princeton University Press, Princeton (2012) 14. Yin, R.K.: Case study Research, Design and Methods, 3d edn. Sage, London, New Delhi (2003) 15. Czarniawska, B.: Narratives in Social Science. Sage, London (2004) 16. Charmaz, K.: Constructing Grounded Theory, a Pratical Guide Trougth Qualitative Analysis. Sage, London (2006) 17. Strauss, A., Corbin, J.: Basics of Qualitative Research. Techniques and Procedures for Developing Grounded Theory, 2nd edn. Sage, London, New Delhi (1998) 18. Denzin (ed.): The Research Act in Sociology: a Theoretical Introduction to Sociological Method, pp. 297–313. McGraw-Hill, New York (1970) 19. Flemish masters.: https://www.flemishmasters.com/en/# 20. Courtauld Gallery.: https://courtauld.ac.uk/gallery 21. Museo Etrusco di Villa Giulia.: https://www.museoetru.it/ 22. Gallerie Estensi.: https://www.gallerie-estensi.beniculturali.it/ 23. Interview by phone with Martina Bagnoli 24. Hooper-Greenhill, E.: Museum and Education, Purpose, Pedagogy, Performance. Routledge, Abigdon (2007) 25. V-Must, Virtual Museum Transnational Network (2013), State of the art on Virtual Museums in Europe and outside Europe, available online www.v-must.net 26. V-Must, Virtual Museum Transnational Network (2014), D2.1 Terminology, definitions and types of Virtual Museums, available online www.v-must.net