Stepping into Virtual Reality [2 ed.] 3031364864, 9783031364860

Citation preview

Stepping into Virtual Reality

Mario A. Gutiérrez A. • Frédéric Vexo Daniel Thalmann

Stepping into Virtual Reality Second Edition

Mario A. Gutiérrez A. Logitech Europe S.A. EPFL - Quartier de l’Innovation - Daniel Borel Innovation Center Lausanne, Switzerland

Frédéric Vexo SPYRAL Genève, Switzerland

Daniel Thalmann EPFL, IC Lausanne, Switzerland

ISBN 978-3-031-36486-0 ISBN 978-3-031-36487-7 https://doi.org/10.1007/978-3-031-36487-7

(eBook)

1st edition: © Springer-Verlag London 2008 2nd edition: © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

Since the first edition of this book in 2007, there has been a fast evolution in Virtual Reality. Palmer Luckey designed a prototype for what would become the Oculus Rift Virtual Reality headset in 2010. Facebook bought Oculus for around $3 billion in 2014. This caused VR to gain massive momentum and large companies like HTC, Google, Apple, Amazon, Microsoft, Sony, and Samsung began developing their own VR headsets. The fast development of social networks and their potential integration with VR led to the Metaverse philosophy. At the same time, Artificial Intelligence made its revolution especially with the development of applications based on Deep Learning. Virtual Reality is no longer the same, and the impact of VR will continue to grow in the future. This second edition has been extensively revised and includes all the latest developments. The most important new contributions are the following: • • • • • • • •

A Chapter on Senses and Sensors A Chapter on the Brain-Computer interfaces A Section on Game Engines Two Sections on the Metaverse A Section on Deep Learning and 3D Hand Pose A Section on Immersive Sound A Section on a VR and Cultural Heritage project A Section on Virtual Reality and Sports

Several sections have been updated and extended, including the History of Virtual Reality Section, the Crowd Simulation Section, and the Virtual Surgery Section. The book is based on years of experience creating virtual worlds and virtual realities. Our goal is to transmit this experience to students and ease the learning curve required to master the creation of Virtual Reality (VR) applications. We start by providing some brief answers to key questions such as: Where did VR come from? What are the main concepts that help us understand this research field? What are the current VR trends? And last but not least, how can we create virtual worlds? v

vi

Preface

Throughout the book, we consider the terms “virtual environment,” “virtual world,” and “VR application” as equivalent concepts: computer-generated environments are mainly composed of interactive computer graphics. They are designed to physically and/or psychologically immerse one or more users in an alternative reality. The first part of the book presents a review of the basic theoretical and practical concepts involved in the visual aspect of virtual environments. We start by presenting the basic mathematical foundations for the synthesis of 3D graphics, including basic modeling and rendering techniques. A comprehensive review of the state of the art of computer animation closes this part. The second part provides more details on the components, structure, and types of virtual worlds that can be created. We provide a detailed explanation of the main modeling and animation techniques for virtual characters, which are one of the most important virtual people living in a virtual world. A review and discussion of the main types of VR system architectures serves to define the different alternatives for organizing and designing a VR application. Virtual Reality is only one of the different types of mixed reality environments that belong to the “reality continuum,” defined in Chap. 1. We provide more details on how to create mixed reality applications, such as augmented reality and introduce the Metaverse philosophy. The third part covers the essence of Virtual Reality: The Perception of Virtual Worlds. We first introduce the five main human senses vision, audition, touch, smell, and taste and extra senses. Instead of providing an exhaustive review of the state of the art in VR hardware, which gets outdated soon due to the fast pace of this research area, we present a generic classification of interaction devices using a humancentered approach, based on the five main human senses. In each chapter, we discuss the importance of a specific human sense and the different types of input and output devices that can be used to establish a dialog between human users and virtual worlds. The last part of the book deals with the present and future of Virtual Reality. Successful VR systems and applications are reviewed and discussed in Health Sciences, Cultural Heritage, Manufacturing, Entertainment, and Sport. We highlight how the basic principles explained throughout the book can be applied and we analyze the practical and theoretical problems that are addressed. This work is conceived as a guided tour that will give a practical explanation of each step in the process of creating a Virtual Reality application. It can be used both as a textbook for a Virtual Reality course or as a reference for researchers in Virtual Reality, Computer Graphics, Computer Animation, or Human-Computer Interaction. Lausanne, Switzerland Genève, Switzerland Lausanne, Switzerland May 2023

Mario A. Gutiérrez A. Frédéric Vexo Daniel Thalmann

Acknowledgments

Some of the images included in this book are part of the research work of the following people; they were all members of the research team of the Virtual Reality Laboratory at EPFL: Mireille Clavien, Achille Peternier, Pascal Glardon, Marcelo Kallmann, Ramon Mas, Ronan Boulic, Tom Molet, Zhiyong Huang, Hansrudi Noser, Amaury Aubel, Tolga Abaci, Michal Ponder, Sebastien Schertenleib, Pablo De Heras Ciechomski, Branislav Ulicny, Alejandra Garcia-Rojas, Etienne De Sevin, Barbara Yersin, Jonathan Maim, Julien Pettré, Damien Maupu, Tolga Capin, Guillaume Gauby, Luca Vacchetti, Patrick Salamin, Patrick Lemoine, Bruno Herbelin, Helena Grillon, Anderson Maciel, Sofiane Sarni, Jan Ciger, Rachel de Bondeli, Fatih Erol, Stephanie Noverraz, and Olivier Renault.

vii

Contents

1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Virtual Reality: The Science of Illusion . . . . . . . . . . . . . . . . . . 1.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Immersion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Presence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 A Brief History of Virtual Reality . . . . . . . . . . . . . . . . . . . . . . 1.4 Reality-Virtuality Continuum . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Metaverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.1 Metaverse and Its Future . . . . . . . . . . . . . . . . . . . . . . . 1.5.2 Major Technologies to Empower the Metaverse . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Part I 2

1 1 2 2 3 4 8 9 9 10 12

Fundamentals

Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Mathematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Coordinate Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Vectors, Transformations, and Matrices . . . . . . . . . . . . 2.1.3 Angular Representations . . . . . . . . . . . . . . . . . . . . . . . 2.1.4 Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 3D Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Geometric Representations . . . . . . . . . . . . . . . . . . . . . 2.2.2 Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 3D Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Local Illumination Model . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Global Illumination Model . . . . . . . . . . . . . . . . . . . . . 2.3.3 Textures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.4 Rendering Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . .

15 15 15 18 22 29 30 30 31 38 39 40 42 46 48

ix

x

3

Contents

2.4 Game Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

51 52

Computer Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Animation Types: Real Time and Image by Image . . . . 3.1.2 Articulated Bodies and Virtual Characters . . . . . . . . . . 3.2 Motion Control Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Motion Capture and Performance Animation . . . . . . . . . . . . . . 3.3.1 Optical Motion Capture Systems . . . . . . . . . . . . . . . . . 3.3.2 Nonoptical Motion Capture Systems . . . . . . . . . . . . . . 3.3.3 Motion Capture Advantages and Disadvantages . . . . . . 3.4 Key-Frame Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Shape Interpolation and Parametric Keyframe Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Kochanek-Bartels Spline Interpolation . . . . . . . . . . . . . 3.5 Inverse Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Motion Retargeting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Procedural Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Physics-Based Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Behavioral Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

55 55 56 56 58 59 59 61 63 64

Part II 4

64 65 67 69 71 72 73 75

Virtual Worlds

Virtual Characters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Virtual Humans in Virtual Environments . . . . . . . . . . . . . . . . . 4.2 Character Skinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Skeleton-Based Deformations . . . . . . . . . . . . . . . . . . . 4.2.2 Data-Driven Methods . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Physics-Based Approaches . . . . . . . . . . . . . . . . . . . . . 4.3 Locomotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Locomotion Generation . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 PCA-Based Locomotion . . . . . . . . . . . . . . . . . . . . . . . 4.4 Virtual Human-Object Interaction . . . . . . . . . . . . . . . . . . . . . . 4.4.1 Feature Modeling and Smart Objects . . . . . . . . . . . . . . 4.4.2 Grasping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Motion Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Facial Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Autonomous Characters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.1 Why Autonomous Virtual Characters? . . . . . . . . . . . . . 4.6.2 Properties of Autonomous Virtual Characters . . . . . . . . 4.6.3 Behaviors for Autonomous Virtual Characters . . . . . . . 4.7 Crowd Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

81 81 82 82 85 86 86 86 88 91 91 93 97 99 102 102 103 105 109 117

Contents

xi

5

Architecture of Virtual Reality Systems and Metaverse . . . . . . . . . 5.1 Scene Graph–Based Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Semantic Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Generic System Architecture for VR Systems . . . . . . . . . . . . . . 5.4 Distributed Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Communication Architecture . . . . . . . . . . . . . . . . . . . . 5.4.2 Interest Management . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.3 Concurrency Control . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.4 Data Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.5 Load Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Metaverse and Online Virtual Worlds . . . . . . . . . . . . . . . . . . . . 5.5.1 The Concept of Avatar . . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Online Worlds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Metaverse with the Internet of Things (IoT) . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

125 125 127 129 130 130 131 132 133 133 134 134 136 138 139

6

Mixed Realities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Augmented Reality and Augmented Virtuality . . . . . . . . . . . . . 6.2 Tracking Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Markers-Based Tracking . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Marker-Less Tracking . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Mixed Reality Tool Kits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Deep Learning and 3D Hand Pose . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

141 141 142 142 144 146 146 149

Part III

Perceiving Virtual Worlds

7

Senses and Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 The 5 Senses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 The Extra Senses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Proprioception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 Equilibrioception . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.3 Pallesthesia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.4 Thermoception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.5 Nociception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.6 Chronoception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

153 153 154 154 155 155 155 157 157 158

8

Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Graphical Display Technologies . . . . . . . . . . . . . . . . . . . . . . . . 8.1.1 Cathode-Ray Tubes . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.2 Liquid Crystal Displays . . . . . . . . . . . . . . . . . . . . . . . 8.1.3 Plasma Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Virtual Reality Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 Head-Mounted Displays . . . . . . . . . . . . . . . . . . . . . . .

159 160 160 160 162 164 164

xii

Contents

8.2.2 8.2.3 8.2.4 8.2.5 References . .

Fish Tank VR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Handheld Displays . . . . . . . . . . . . . . . . . . . . . . . . . . Large Projection Screens . . . . . . . . . . . . . . . . . . . . . CAVE Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . .........................................

. . . . .

167 167 169 169 171

9

Audition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 The Need for Sound in VR . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Recording and Reproduction of Spatial Sound . . . . . . . . . . . . . 9.3 Synthesis of Spatial Sound . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 Sound Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Head-Related Transfer Function . . . . . . . . . . . . . . . . . 9.3.3 3D Sound Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.4 Utilization of Loudspeaker Location . . . . . . . . . . . . . . 9.4 Sound Systems for VR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Sound Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.2 Sound Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Immersive Sound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

173 173 174 174 174 175 176 176 177 177 180 181 181

10

Touch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1 The Need for Touch in VR . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Data Gloves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Haptic Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.1 History of Haptic Rendering . . . . . . . . . . . . . . . . . . . . 10.4 Haptic Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.1 Vibrotactile Displays . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.2 Tactile Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.3 Kinesthetic Displays . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

183 183 184 184 185 186 187 188 189 192

11

Smell and Taste . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 The Need for Smells and Tastes in VR . . . . . . . . . . . . . . . . . . 11.2 Smell Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Taste Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

195 195 196 198 199

12

Brain–Computer Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 BCI Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 BCI in Virtual Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

201 201 204 205

Contents

Part IV

xiii

Applications

13

Health Sciences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1 Virtual Surgery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.1 Principles of Virtual Surgery . . . . . . . . . . . . . . . . . . . . 13.1.2 Case Studies in Virtual Surgery . . . . . . . . . . . . . . . . . . 13.1.3 Commercial Virtual Surgery Simulators . . . . . . . . . . . . 13.1.4 The Intelligent Digital Surgeon . . . . . . . . . . . . . . . . . . 13.2 Virtual Rehabilitation and Therapy . . . . . . . . . . . . . . . . . . . . . . 13.2.1 Physiotherapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.2 Psychological Therapy . . . . . . . . . . . . . . . . . . . . . . . . 13.2.3 Autism and Virtual Dolphins . . . . . . . . . . . . . . . . . . . . 13.3 Virtual Anatomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

209 209 209 210 211 212 213 213 215 216 218 220

14

Cultural Heritage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.1 Virtual Campeche and Calakmul . . . . . . . . . . . . . . . . . . . . . . . 14.2 Virtual Dunhuang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3 Terracotta Soldiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4 EU-INCO CAHRISMA and ERATO . . . . . . . . . . . . . . . . . . . . 14.5 EU-IST ARCHEOGUIDE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6 EU-IST LIFEPLUS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7 EU-Horizon 2020 MINGEI . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

223 223 225 225 227 228 230 231 232

15

Other VR Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.1 Vehicle Simulators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.2 Manufacturing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Entertainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4 Virtual Reality and Sport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

235 235 237 239 241 244

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

List of Figures

Fig. 1.1

Fig. 1.2 Fig. 2.1

Fig. 2.2 Fig. 2.3 Fig. 2.4 Fig. 2.5 Fig. 2.6 Fig. 2.7 Fig. 2.8 Fig. 2.9 Fig. 2.10 Fig. 2.11 Fig. 2.12 Fig. 2.13 Fig. 2.14 Fig. 2.15 Fig. 2.16 Fig. 3.1 Fig. 3.2 Fig. 3.3 Fig. 3.4

The first virtual reality systems (a) Norton Heiling Sensorama (1962) (b) Sutherland Sketchpad (1963) (c) Sutherland’s head-mounted display (1968) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reality-virtuality continuum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Point and shapes in polar coordinates. (a) A point P in spherical coordinates. (b) A cardioid function in polar coordinates (ρ, θ, φ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Triangle-based 3D shape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Noncommutativity of transformations composition. (a) R
T
M. (b) T
R
M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gimbal lock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rotation about an arbitrary axis in space . . . . . . . . . . . . . . . . . . . . . . . . . . . Projection. (a) Orthographic. (b) Perspective . . . . .. . . . .. . . . .. . . . . .. . Conic sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A cubic Bézier curve and its defining polygon . . . . . . . . . . . . . . . . . . . . . Spline surfaces: Bézier patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shading algorithms for local illumination . . . . . . . . . . . . . . . . . . . . . . . . . . Ray tracing example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Radiosity example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mix of ray tracing and radiosity rendering . . . . . . . . . . . . . . . . . . . . . . . . . Texture mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Texture mapping techniques. (a) Environment mapping. (b) Bump mapping. (c) Displacement mapping . . . . . . . . . . . . . . . . . . . . Rendering pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H-ANIM skeleton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dongseo’s motion capture green room. (Licensed under the Creative Commons Attribution 2.0 Generic license) . . . .. . . . .. . . .. . Principles of optical motion capture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motion capture using Ascension magnetic sensors . . . . . . . . . . . . . . .

5 9

16 17 22 23 24 29 32 34 38 42 44 45 46 47 48 49 57 59 60 62

xv

xvi

Fig. 3.5 Fig. 3.6

Fig. 3.7 Fig. 3.8 Fig. 3.9 Fig. 3.10 Fig. 3.11 Fig. 4.1

Fig. 4.2 Fig. 4.3 Fig. 4.4 Fig. 4.5 Fig. 4.6 Fig. 4.7 Fig. 4.8 Fig. 4.9 Fig. 4.10 Fig. 4.11 Fig. 4.12 Fig. 4.13 Fig. 4.14 Fig. 4.15

Fig. 4.16 Fig. 4.17 Fig. 4.18 Fig. 4.19 Fig. 4.20 Fig. 5.1 Fig. 5.2 Fig. 5.3

List of Figures

Interpolation between two drawings: (top) an interpolation with 50%; (bottom) interpolations with 25%, 50%, and 100% . . . . . . . . . Influence of tension, continuity, and bias on Kochanek-Bartels spline. (a) Increase of the tension. (b) Increase of the discontinuity. (c) A biased interpolation at K2 . . . . . . . . . . . . . . . . . . . . . Direct and inverse kinematics . . . . . . .. . . . . . . . .. . . . . . . . . .. . . . . . . . .. . . . . Use of an intermediate skeleton for motion retargeting . . .. . . . . .. . . Dynamics-based motion [1] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Direct and inverse dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L-system-based environment . .. . . .. . . .. . . . .. . . .. . . .. . . .. . . .. . . .. . . .. . JLD deformations: early version of skeleton-based deformation from the movie Rendez-vous à Montréal, by Nadia MagnenatThalmann and Daniel Thalmann [1] .. . .. . .. . .. .. . .. . .. . .. .. . .. . .. . .. Metaball-based deformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Walking generated using PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Defining and using smart objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grasping taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grasping (a) Virtual sensors (b) Grasping sensors (c) One-hand grasping (d) Multiple grasping . . . . . . . . . . . . . . . . . . . . . . Multi-agent carrying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (a) Human carrying the balloon (b) Balloon lifting the human into the air . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reaching examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A singing facial expression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FDP feature points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Virtual assistant for basic life support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reflex reactions to a ball . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simplified motivational model of action selection for virtual humans . . . .. . . . . . .. . . . . . .. . . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . Virtual life simulation: By default, the virtual human is working, waiting for motivation (for instance, drinking) to be activated. The food is kept in the kitchen, as seen in the background . . . . . . . Intercommunication .. . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . Crowd simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Crowd simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pedestrians using a hybrid motion planning architecture to reach their goal and avoid each other . . . . . . . . . . . . . . . . . . . . . . . . . . . . Torso crowds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

64

66 67 70 72 73 75

83 84 90 92 93 94 95 95 98 99 101 103 106 107

108 109 111 112 114 116

Semantic representation of an interactive virtual environment . . . . 128 Cybertennis in Telecom Interactive 97, by MIRALab, University of Geneva and EPFL VRlab [21] . . . . . . . . . . . . . . . . . . . . . . 131 The aura nimbus approach for interest management in collaborative worlds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

List of Figures

Fig. 5.4 Fig. 5.5 Fig. 5.6 Fig. 5.7

Fig. 5.8 Fig. 5.9

Fig. 6.1 Fig. 6.2 Fig. 6.3 Fig. 6.4 Fig. 6.5 Fig. 6.6

Fig. 7.1 Fig. 7.2 Fig. 7.3 Fig. 7.4 Fig. 7.5 Fig. 7.6 Fig. 7.7

A colored Emoji from Twitter Emoji project. (Creative Commons Attribution 4.0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Happy Bitmoji. (Creative Commons Attribution-Share Alike 4.0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Facebook Avatars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wouldn’t It Be Nice. (Created by Alicia Chenaux, licensed under the Creative Commons Attribution-Share Alike 2.0 Generic) . . . . . .. . . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . . .. . . . The Palace chat early marketing screenshot. (Wikipedia, fair use) . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . A performance at “LA Club 2020 & Friends” in Second Life. (Licensed under the Creative Commons Attribution 2.0 Generic) . . . . . .. . . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . . .. . . . . .. . . . . . .. . . . . . .. . . . Markers-based tracking using ARToolkit . . . . . . . . . . . . . . . . . . . . . . . . . . . Playing checkers with a virtual human . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3D hand pose estimation from depth images . . . . . . . . . . . . . . . . . . . . . . . Hand surface completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multi-view CNN-based approach for 3D hand pose estimation . . 3D CNN, which takes volumetric representations as input and regresses a lower dimensional representation of 3D joint locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The five senses. (Hans Makart, nineteenth century, public domain) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Walking in the park. (Licensed under the Creative Commons Attribution 2.0) . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . . .. . . . . . . .. . . . . . . . .. . . . The performance of “Jultagi” in Jeonju, South Korea. (Licensed under the Creative Commons Attribution 2.0 Generic license) . . . Maria Leal da Costa, working. Lithuania. (Creative Commons Attribution-Share Alike 4.0) .. . .. . .. . .. .. . .. . .. . .. .. . .. . .. .. . .. . .. . .. Tastant l’aigua d’una piscina, Laia from Reus Catalonia. (Creative Commons License) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Symptoms: Stomach pain. (Public domain, Creative Commons) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Passing Time sculpture by Auckland-based sculptor Anton Parsons. (Creative Commons Zero, Public Domain Dedication) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xvii

135 135 136

136 137

137 143 145 147 147 148

149 154 155 156 156 157 157

158

Fig. 8.4

Human field of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Different types of head-mounted displays . . . . . . . . . . . . . . . . . . . . . . . . . . Use of handheld devices in VR applications (a) 3D rendering on PDA (b) Mobile animator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A CAVE system with four screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Fig. 9.1

Pioneer sound field control system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

Fig. 8.1 Fig. 8.2 Fig. 8.3

165 166 168 170

xviii

Fig. 10.1

List of Figures

Fig. 10.2 Fig. 10.3

Data gloves for touch and force-feedback. (a) CyberTouch™. (b) CyberGrasp™ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 PHANTOM haptic interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 The Haptic Workstation™ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

Fig. 11.1

Food simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

Fig. 12.1

A cap holds electrodes in place while recording an EEG. (Creative Commons Attribution 2.0) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Wearable, wireless EEG solutions in daily life applications: What are we missing? (Licensed under the Creative Commons Attribution-Share Alike 4.0 International license) . . . . . . . . . . . . . . . . . 203

Fig. 12.2

Fig. 13.1 Fig. 13.2 Fig. 13.3 Fig. 13.4

Fig. 13.5 Fig. 13.6

Fig. 14.1

Fig. 14.2 Fig. 14.3 Fig. 14.4 Fig. 14.5 Fig. 14.6 Fig. 15.1 Fig. 15.2 Fig. 15.3 Fig. 15.4 Fig. 15.5 Fig. 15.6 Fig. 15.7

Prototype of the intelligent digital surgeon . . . . . . . . . . . . . . . . . . . . . . . . . VR-based telerehabilitation systems for physiotherapy. (a) Post-stoke therapy. (b) Shoulder and elbow therapy . . . . . . . . . . . VR-based treatment of social phobia. (a) Exposure-based theraphy. (b) Eye-tracking setup . .. . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . (a) The immersive room, (b) Virtual Underwater World visualized on the 3-D screen, and (c) Virtual Dolphinarium visualized on the screen . . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. .. . .. Engaging a child with autism through 3-D immersive visualization and hand gesturing . .. . . .. . . .. . . .. . . .. . . . .. . . .. . . .. . . .. . Virtual model of the musculoskeletal system. (a) Interactive model of human muscles. (b) Analysis of stress on the hip joint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Latin American virtual reality applications: Virtual Calakmul. (Courtesy of Rocío Ruiz Rodarte, Jesús Savage Carmona and José Larios) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Flashlight interaction in a CAVE. (Courtesy of FraunhoferIGD) . . .. . . .. . . .. . . .. . . .. . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . . .. . EU-INCO CAHRISMA (2004) and ERATO (2006) . . . . . . . . . . . . . . Augmented reality and cultural heritage . . . . . . . . . . . . . . . . . . . . . . . . . . . . System architecture for the MINGEI project . . . . . . . . . . . . . . . . . . . . . . . Digital humans in the MINGEI project . . . . . . . . . . . . . . . . . . . . . . . . . . . . A car driving simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Virtual vehicle simulators (a) Vehicle simulator, (b) Virtual cockpit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Augmented reality for operator training .. . .. . .. . .. . .. . .. . .. . .. . .. . .. Simulator or advanced video game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . “Enigma of the Sphinx” using a multimodal interface: the Magic Wand . . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . .. . . . VR volleyball game . . . .. . . .. . .. . . .. . . .. . . .. . .. . . .. . . .. . . .. . . .. . .. . . .. . VR volleyball game input and output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

213 214 215

217 218

219

224 226 227 229 231 231 236 236 239 240 241 243 243

Chapter 1

Introduction

This chapter provides definitions of basic concepts that are essential to understand the field of virtual reality. By the end of the chapter, you should have a good understanding of the fundamental concepts of virtual reality, be familiar with the history of the field, and be aware of the multiple disciplines involved in the development of virtual reality applications. You should also understand the evolution of the metaverse and its future.

1.1

Virtual Reality: The Science of Illusion

When the term virtual reality (VR) started to be used, it produced great expectations. The idea was that this technology would be able to create imaginary worlds that would be indistinguishable from the real world. During the 1990s, other related terms emerged: virtual environments and synthetic environments. Now, much of the original hype of VR has disappeared due to unfulfilled promises. Technology was not ready to make computer-generated worlds as believable as reality. Today we consider that virtual reality is about creating “acceptable” reproductions of real objects or environments for training, entertainment, or design purposes. With the development of new technologies in computing, networking, internet, artificial intelligence (AI), and social media, VR has started a new evolution and new expectations mainly under the framework of metaverse. Such a new development will be explained furthermore in Sect. 1.4. Virtual reality uses computers to create 3D environments in which one can navigate and interact. Navigation implies the ability to move around and explore the features of a 3D scene, such as walking through a forest. Interaction means the ability to select and manipulate objects in the scene, for instance, grabbing and examining a flower found in the forest. Virtual Reality in its most “classical” form requires real-time graphics, a stereoscopic display, used to produce the illusion of 3D, and a tracking system to acquire © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_1

1

2

1 Introduction

head and hands motion. Commonly used technologies include head-mounted displays (HMD) and stereoscopic glasses. Interaction can be achieved using a tracking device, which may be integrated in the HMD itself, for tracking head and body movement. A “data glove” can be used to track hand movements. The glove lets the user point to and manipulate objects in the virtual scene. VR is about simulating reality. The aspect of reality that has been most frequently addressed is the visual one. Sight is, for most people, the dominant perceptual sense, and the principal means for acquiring information. However, reality is not limited to what we can see. Other important components of our perceptual experience are sounds and tactile feedback. Developing VR systems involves different disciplines that address each of the human senses: computer graphics (sight), 3D sound synthesis (hearing), and haptics (touch). Smell and taste play important roles in our daily life but have been less exploited in VR due to the complexity of the required technology. In this book, we dedicate separate chapters to explain how to generate 3D graphics, 3D sounds, and haptic feedback. Smell and taste are grouped in a single chapter because related research is still in a pioneering stage. But before explaining how computers can synthesize stimuli for each of the human senses to produce the illusion of an alternative reality, we discuss some important concepts that help us to understand and measure the effects of virtual reality on users.

1.2

Basic Concepts

The main goal of VR is to create in the user the illusion of being in an environment that can be perceived as a believable place with enough interactivity to perform specific tasks in an efficient and comfortable way. There are two main factors that describe the VR experience from the physical and psychological points of view: immersion and presence.

1.2.1

Immersion

Immersionis related to the physical configuration of the user interface of the VR application. VR systems can be classified as fully immersive (those using an HMD or an Immersive Room), semi-immersive (large projection screens), or non-immersive (desktop-based VR). The classification depends on how much the user can perceive (see, hear, touch) the real world during the simulation. The first VR systems were fully immersive and used variants of head-mounted displays. The idea was to completely isolate the user from the real world, with the hope that this would contribute to the believability and efficiency of the simulation. Early fully immersive systems had different levels of success in creating the illusion

1.2

Basic Concepts

3

of an alternative reality, and some of them were difficult to use. When using an HMD, some people may suffer from motion sickness–related problems. This is known as cybersickness [1]. Several factors that cause it to have been identified, but there is still no definite method to prevent it. Cybersickness seems to appear more often with fully immersive systems. Semi-immersive systems, such as the CAVE, designed by Cruz-Neira et al. in 1992 [2], provide 3D sound and high-resolution graphics. A CAVE is a multi-user workplace surrounded by screens where the virtual world is projected. Images are displayed according to the position and gaze direction of the main user. For additional information about CAVE systems, see Sect. 8.2.5. In general, semiimmersive systems let several users share the simulation; this opens interesting possibilities for collaborative work. Nonimmersive systems have gained popularity due to their lower cost, ease of use, and ease of installation. They are sometimes called desktop-based VR systems; the most representative examples are video games. The good combination of interactivity, ease of use, and appealing graphics and sound can produce in the users a great level of interest and involvement in the simulation. Few VR systems can compete with a good video game in terms of – psychologically – isolating the user from the world and producing strong emotional responses. Psychological aspects of the VR experience are an active research area. It is not completely clear which are the factors in a simulation that can produce specific user reactions in terms of emotional response, involvement, and degree of interest. One of the most important concepts that helps us understand the psychology of the VR experience is the “sense of presence.”

1.2.2

Presence

Presence is a subjective concept, associated with the psychology of the user. According to Slater and Wilbur, “Presence is a state of consciousness, the (psychological) sense of being in the virtual environment” [3]. Professor M. Slater is one of the best-known researchers on this subject. There are several journals1 and international conferences where the psychology of the VR experience is explored and discussed. Presence is when the multimodal simulations (images, sound, haptic feedback, etc.) are processed by the brain and understood as a coherent environment in which we can perform some activities and interact. Presence is achieved when the user is conscious, deliberately, or not, of being in a virtual environment (VE). For example, when playing a video game, a person knows the world in the game is not real but decides to behave as if it were a real situation. A sign of presence is when people

1 One of the main publications on the topic is the MIT Press journal Presence: Teleoperators and Virtual Environments (http://www.mitpressjournals.org/loi/pres).

4

1 Introduction

behave in a VE in a way that is close to the way they would behave in a similar reallife situation. However, presence can be achieved in a VE that does not resemble any real-life environment, for example, fantasy worlds in video games are taken by users as if they existed and users behave accordingly. Presence can lead to involvement and emotional reactions from the user. Once the brain integrates the 3D images, sounds, and other kinds of feedback in the form of a coherent environment, different reactions can arise. It can occur that we feel deeply involved in the simulation and experience a variety of emotions. A well-crafted virtual world could change our emotional state and make us feel anxiety, happiness, or sadness. This phenomenon is what has motivated many researchers to use VR as a therapeutic tool for treating phobias, such as arachnophobia or social phobia. In this sense, research tries to discover what sensory input is needed for people to feel as if they were present in an environment. Depending on the situation, some senses can be more important than others. For instance, when simulating surgery to train doctors, having accurate tactile haptic feedback is more important than the image quality of the simulation. In other scenarios, an accurate 3D sound synthesis can be the key factor to induce a sense of presence. For example, when simulating an orchestra in a concert hall, presence will be achieved if the sound is accurately reproduced, considering the acoustics of the hall. In this case, the realism of the 3D graphics has a lower priority for the sense of presence. In fact, when listening to a high-quality sound recording, it is easier to feel like you are in the concert hall when you close our eyes, because the visual input is less important in this situation. It has been observed that user involvement can induce a sense of presence. It is important to distinguish presence and involvement. As in real life, one can be present but not involved, for example, being in a stadium watching a game that is boring, we know we are in the stadium, but we don’t behave; accordingly, even though we are there, we don’t feel present. Involvement depends on the content, for example, watching an important game of my favorite team on TV can make me react and behave as if I were in the stadium. The content of a simulation can induce presence. This explains the high levels of presence that can be achieved by players of some video games. The content of the VE produces so much involvement in the user that even though the system is non immersive and there is no sophisticated user interface, usually keyboard, mouse and/or joystick, the user feels like he or she is in the virtual world and gets isolated from reality.

1.3

A Brief History of Virtual Reality

The history of virtual reality as a multisensory simulation of the real world can be traced back to the 1960s. The “Sensorama” (see Fig. 1.1a), created by the cinematographer Morton Heilig in 1962, was a multisensory vehicle simulator. The system allowed users to sit in front of a screen where they could choose from different rides prerecorded using motorcycles, bicycles, and even a helicopter. Sensorama used wide field of view optics to view 3D photographic slides and had stereo sound, as

1.3

A Brief History of Virtual Reality

5

Fig. 1.1 The first virtual reality systems (a) Norton Heiling Sensorama (1962) (b) Sutherland Sketchpad (1963) (c) Sutherland’s head-mounted display (1968)

well as smell and wind generators. At that time, computers were not ready to be used for such applications. The system had almost no interactivity; the user was a passive observer. In 1965, Ivan Sutherland published a paper entitled “The Ultimate Display,” in which he described how one day the computer would provide a window into virtual worlds. In 1968, he built a head-mounted display (see Fig. 1.1c) that presented to the user left and right views of a computer-generated 3D scene. The system tracked the user’s head movement to update the images, accordingly, giving the illusion of being in a virtual world. Images were far from being realistic; they were simple line drawings. However, the stereoscopic view produced the impression of looking at solid 3D objects. Sutherland also developed the “Sketchpad” (see Fig. 1.1b), considered to be the ancestor of modern computer-aided drafting (CAD) programs. Sutherland is commonly recognized as the “father of virtual reality systems.” In 1978, A. Lippman, Scott Fisher, and other researchers from MIT developed the “Aspen Movie Map.” The application allowed users to view a simulated ride through the city of Aspen, Colorado. The system used photographs of all the streets of the city, taken with four cameras pointing in different directions mounted on a truck. The user could move in four directions within the simulation; these were the beginnings of interactive virtual environments. In the early 1970s, Myron Krueger experimented with computer-generated environments and developed several computer art projects. Computers responded to gestures of the audience by interpreting and anticipating actions. Audience members could “touch” their video-generated silhouettes and interact with them. The system was called “Videoplace” and was used to generate what Krueger called “artificial reality” [4]. In the mid-1980s, different technologies converged to create the first true VR systems. Researchers at NASA’s Ames Research Center worked on the creation of an affordable pilot training system for manned space missions. The people involved included Scott Fisher, Stephen Ellis, Michael McGreevy, and Warren Robinett. Their efforts led to the development of the Virtual Interface Environment

6

1

Introduction

Workstation (VIEW). The VIEW system consisted of a wide-angle stereoscopic display unit, glove-like devices for multiple degree-of-freedom tactile input, speechrecognition technology, gesture-tracking devices, 3D audio, speech synthesis, computer graphics, and video image generation equipment. In the words of Stephen Ellis, “The technology of the 1980s was not mature enough.”2 At that time, VR helmets (HMD) were too heavy, computers didn’t have enough processing power, and touch feedback systems weren’t reliable enough. Another important pioneer VR project is the work started in 1967 by Frederick Brooks at the University of North Carolina at Chapel Hill. Brooks selected as a “driving problem” the manipulation of molecules. The objective of project GROPE was to develop a haptic interface for molecular forces. The idea was to allow molecule designers to “feel” force constraints when trying to create new molecule combinations. Force feedback was provided by the Argonne Remote Manipulator (ARM) a large ceiling-mounted mechanical device that was used to “grasp” and manipulate molecules. Project GROPE stayed active until 1990 [5]. The first initiatives to commercialize VR products started in the early 1980s. VPL Research was one of the first companies focused on developing VR hardware and software. VPL developed the “DataGlove,” a glove-based input device. Other VPL products included head-mounted displays and software for graphics and sound rendering. VPL Research was founded by Jaron Lanier, who is recognized as having coined the term “virtual reality.” Sun Microsystems acquired the patent portfolio of VPL Research in 1998. Other companies that have played a role in the history of commercial VR products are Polhemus, Ascension, Virtual Technologies, and Immersion. Polhemus, a private company since 2002, was previously owned by major aerospace companies (Northrop, McDonnell Douglas, Kaiser Aerospace). Founded in the late 1960s, Polhemus pioneered electromagnetic position tracking. Ascension Technology was founded in 1986 by Ernie Blood and Jack Scully, former Polhemus employees. Ascension specializes in 3D tracking devices; the “Flock of Birds” set of magnetic trackers, one of their most popular products, is used in animation and medical applications. Virtual Technologies Inc., developers of the CyberGlove™, a glove-based interface, was acquired by Immersion Corporation in 2000. Immersion Corp. sells force-feedback devices and software. The 1980s saw a boom in VR research and commercialization, however, as mentioned before, the technology was not ready to fulfill all the expectations. Unreliable hardware, slow computers, and cybersickness were common problems. In the early 1990s, a change of paradigm in VR interfaces occurred with the design of the CAVE [2]. A CAVE is a room with graphics projected from behind the walls. It was invented at the Electronic Visualization Laboratory of the University of Illinois at Chicago Circle in 1992. The images projected on the walls were in stereo to provide a depth cue. In a CAVE, the user is surrounded by the projected images,

“Whatever happened to... Virtual Reality” http://science.nasa.gov/headlines/y2004/21jun_vr.htm (accessed August 2007).

2

1.3

A Brief History of Virtual Reality

7

offering a field of vision that is much wider than that of an HMD. Instead of wearing an uncomfortable VR helmet, CAVE users put on lightweight stereo glasses and can freely walk inside the CAVE. CAVEs are multi-user systems where people can see their own hands and bodies as part of the virtual world, enhancing the sense of presence and allowing for collaboration and exchange of ideas. A virtual version of space was made available to astronomers in 1990 through NASA’s Virtual Interface Environment Workstation (VIEW). A VR training lab was then built to train astronomers on the ground. In 1991, an engineer at NASA, named Antonio Medina, invented a VR system that lets you pilot a Mars rover, taking time delay into account. A virtuality platform was launched by the Virtuality Group. Gamers could play on these VR arcade machines in a 3D world. Until this system, VR entertainment had never been mass-produced. With the release of The Lawnmower Man in 1992, virtual reality was introduced to a wider audience. Jaron Lanier and his early laboratory days were part of the inspiration for the film, and actual virtual reality equipment from VPL research labs was used in the film. During the 1993 Consumer Electronics Show, Sega announced its VR headset for its Sega Genesis console. Unfortunately, technical development difficulties forced the device to remain in the prototype phase. In the end, this was a huge failure for Sega. However, Sega also released a motion simulator arcade machine, the SEGA VR-1. A VR headset was also released by VictorMaxx called CyberMaxx. Nintendo launched the Virtual Boy console in 1995, which played 3D monochrome video games. A year later, it was discontinued. Georgia Tech and Emory University researchers used VR in 1997 to create war zone scenarios to help veterans with PTSD. This was known as Virtual Vietnam. The MATRIX film from 1999 depicts characters living in a completely simulated world, unaware of the fact they are not living in the real world. The Matrix made a significant cultural impact and brought simulated reality into the mainstream. Google Maps was enhanced with 360-degree street-level images in 2007, captured by special cars equipped with camera equipment. A dodecahedral camera designed by Immersive Media was used for the work. Street View receives a 3D mode just a few years later. Palmer Luckey, an 18-year-old entrepreneur, developed the first prototype of the Oculus Rift headset. It boasted a 90-degree field of view, which had never been seen before, and relied on a computer’s processing power to deliver images. A Kickstarter campaign Luckey launched in 2012 raised $2.4 million. Facebook saw potential in Oculus‘technology and bought it for $2 billion. It marked a turning point in VR’s history because it gained momentum rapidly after this. Sony has announced Project Morpheus, a VR headset for the PlayStation 4 (PS4). With the release of Cardboard, Google has given the world a low-cost and do-ityourself way to view stereoscopic content on smartphones. Samsung Gear VR uses the Samsung Galaxy smartphone as a viewer.

8

1

Introduction

People started exploring the possibilities of VR, including the addition of innovative accessories, such as Cratesmith’s recreation of the skateboard from Back to the Future by pairing the Oculus Rift with a Wii’s balance board. In 2015, VR possibilities have become widely available to the general public, for example, an interactive VR roller coaster was launched by the Wall Street Journal following the ups and downs of the Nasdaq Stock Market. Gloveone had a successful Kickstarter campaign. Users can feel and interact with virtual objects with these gloves. The HTC VIVE SteamVR headset was released in 2016 by HTC. This was the first commercial headset with sensor-based tracking that allowed users to freely move around a space. In 2018, Oculus unveils a new prototype HMD called the “half dome.” Featuring varifocal lenses, this headset offers an extremely wide field of view at 140 degrees. As of 2019, mixed reality systems are now part of standalone VR headsets. With the promise of tethering capability, Oculus Quest is being launched, and smartphone-based VR projects are shutting down. According to Forbes, 2019 is ‘The Year Virtual Reality Gets Real’. In many locations, Oculus Quest sold out and generated $5 million in content sales. As part of its entry into the VR market, Nintendo introduced the Labo. Over one million copies of Beat Saber were sold in under a year. The Oculus Quest 2 was released by Facebook in October 2020. Now, it is the industry’s most powerful and affordable wireless VR headset. A rival to the Oculus Quest 2 headset, Pico Neo 3 launches in 2021. TikTok’s owner, ByteDance, acquired Pico. Facebook (now Meta) planned to spend at least $10 billion on Facebook Reality Labs, its metaverse division that develops AR and VR hardware, software, and content. As explained in Sect. 1.5, decisions changed in 2022. Online science experiments are available at VRLab Academy, a platform that offers Science, Physics, Biology, and Popular Science experiments online. VRLab Academy works with high-definition VR sets, so you can experience science hands-on. Despite its coolness and potential, VR isn’t yet comfortable enough to wear at work every day. While the headsets are relatively light, they can still be a bit irritating to wear for long periods of time. In addition, the display quality is not yet good enough to switch from a monitor. Even though the graphics are impressive, they aren’t as clear or as detailed as those on a traditional monitor or television screen.

1.4

Reality-Virtuality Continuum

In [6], Milgram et al. proposed the idea of a reality-virtuality continuum, a way to understand and classify different types of display technologies. This helps to classify VR simulations with different degrees of reproduction fidelity, presence, and interactivity; see Fig. 1.2.

1.5

Metaverse

9

Fig. 1.2 Reality-virtuality continuum

At one end of the continuum, we can situate “pure” virtual reality: everything one can perceive is artificial; the user is isolated from the real world. On the opposite end, we have real reality, no computer-generated stimuli. In the middle, we find mixed reality, simulations that combine real and virtual images: augmented reality and augmented virtuality. In augmented reality, most of the images are real. For example, using transparent headsets, you could see how new furniture would look in your house or view the 3D schematic of a car engine while working on the engine itself. In augmented virtuality, most of the imagery is computer-generated, such as virtual sets, commonly used in TV shows, where presenters are projected into an imaginary environment. In the last part of this book, we describe VR applications, which are specific examples of simulations situated in different positions along the reality-virtuality continuum.

1.5 1.5.1

Metaverse Metaverse and Its Future

The term “metaverse“originated in the 1992 science fiction novel Snow Crash [7] as a portmanteau of “meta“and “universe.” Metaverse development is often linked to advancing virtual reality technology due to the increasing demands for immersion. Metaverse is essentially the combination of virtual worlds and social media. As of 2019, Facebook has launched a social VR world called Facebook Horizon. It is unclear what the future holds for metaverse (and virtual reality). Mark Zuckerberg announced in 2021 that Facebook would become “Meta Platforms” and that the company would develop a metaverse. The senior vice president of Intel, Raja Koduri, announced in December 2021[8] that “true persistent and immersive computing will require even more, at scale, and accessible to billions of people in real time: a 1000-times increase in computational efficiency from today’s state of the art.”

10

1

Introduction

The metaverse development department of Meta lost over $10 billion in 2021, with Mark Zuckerberg predicting “significant” losses in 2022. Zuckerberg announced in a Facebook post in February 2023 that the company would pivot away from the metaverse to focus on artificial intelligence. According to a new report from Grand View Research, Inc., the global metaverse market is predicted to reach USD 936.6 billion by 2030 [9].

1.5.2

Major Technologies to Empower the Metaverse

Metaverses are digital spaces where users can connect with each other. Listed below are some of the major technologies that will enable the Metaverse in the future. Augmented Reality and Virtual Reality In the virtual world, augmented reality and virtual reality are our entry points. They form the basis of this book. Augmented reality (AR) alters the natural environment by adding digital visuals and characters. The technology is less scary than virtual reality and can be used with any smartphone or digital device with a camera. Virtual reality generates a computer-generated virtual world that is integral to the concept of the metaverse. You can explore it using a virtual headset, sensors, and gloves. Through the inclusion of fake visual content, virtual reality is already transforming the digital environment. It will allow users to hear, sense, and interact with people from all over the world. Edge Computing, 5G and 6G With edge computing, data can be transported faster with little to no delays, resulting in smoother experiences. It has made the metaverse possible since computers need to be able to handle powerful simulations efficiently to eliminate lag and ensure that users have a seamless gaming experience. Another essential element is the availability of 5G networks, which are often paired with edge computing. Participants in the metaverse previously experienced network problems due to slow rendering and slow data transmission due to a lack of high internet connectivity. Since 5G is being scaled out and made more affordable, more people can take advantage of the metaverse from their computers and other devices without experiencing network slowness. A 6G mobile system standard is currently being developed for wireless communications technologies supporting cellular data networks. A planned successor to 5G, it is expected to be significantly faster and support applications beyond current mobile use scenarios, including virtual and augmented reality (VR/AR), ubiquitous instant communications, and the Internet of Things (IoT). 3D Reconstruction and Digital Twins 3D Reconstruction isn’t a new technology, but its application has grown throughout the COVID-19 pandemic, especially in the real estate sector, where lockdowns prevented potential buyers from visiting residences in person. This has led to a

1.5

Metaverse

11

few organizations using 3D reconstruction technology to conduct virtual property tours. Potential buyers can browse potential new houses from afar and buy without even entering, like the metaverse we imagined. Creating an authentic digital world is a challenge in the metaverse. The use of 3D reconstruction enables the creation of realistic and natural-looking scenes. With powerful 3D cameras, humans can create accurate 3D photorealistic representations of buildings, actual locations, and things online. It is also known as a digital twin. Digital twins are replicas of real-world objects. Cryptocurrencies and Blockchain Metaverse implementations sometimes use digital currencies, especially cryptocurrency. nonfungible tokens (NFTs) are traded within the metaverse, and ownership is tracked by blockchain technology. The blockchain technology facilitates digital ownership, value transfer, digital collectability, governance, interoperability, and accessibility. Individuals can exchange value using cryptocurrency while working and socializing in a 3D digital environment. Artificial Intelligence and Nonplayer Characters In everyday life, AI (artificial intelligence) is widely used in corporate strategy planning, decision-making processes, facial recognition, faster computing, and other applications. In recent years, AI experts have begun exploring the possibility of using AI to create immersive metaverses. Data can be processed by AI at an incredibly fast speed. By combining AI with machine learning, AI algorithms can learn from previous iterations, using historical data to produce unique outputs and insights. Artificial intelligence may be applied to NPCs (nonplayer characters) throughout the metaverse. There are NPCs in almost every game; they interact with players and respond to their actions. Using NPCs in a 3D environment, users can engage in lifelike discussions or perform other tasks that require AI skills. In contrast to a human user, an AI NPC can run autonomously and be used by millions of people at the same time. Several languages can also be supported. In Sect. 14.4, we see an example of NPCs in a Virtual Handball game. Artificial intelligence can also be used to model metaverse avatars. To build more accurate and lifelike avatars, AI systems may analyze real people, 3D scans, or 2D photos. Artificial intelligence can also be used to enhance avatars with new facial expressions, hairstyles, and clothing. Machine learning is a branch of artificial intelligence that develops algorithms to make accurate predictions about new or sample data, create highly accurate classification systems for data, and as a result, provide scientists with insights and patterns they wouldn’t otherwise be able to find. Deep learning goes beyond machine learning by automating more aspects of the learning and training process. By decoding unstructured data sets, such as text or images, deep learning algorithms require much less human intervention. For this reason, deep learning has been described as “scalable machine learning.”

12

1

Introduction

Internet of Things Internet of Things (IoT) was first proposed in 1999. IoT is a system that connects everything in our physical world to the Internet through sensors. Once connected to the Internet, these devices will have a unique identification and be able to receive and send data automatically. IoT today connects medical devices, thermostats, voiceactivated speakers, and other devices to data sources. IoT is used in the metaverse for collecting and distributing data from the physical world. This would improve the precision of digital representations. The behavior of a few metaverse objects can be modified by IoT data streams depending on various factors. This allows the creation of live metaverse simulations.

References 1. LaViola JJ (2000) A discussion of cybersickness in virtual environments. SIGCHI Bull 32(1): 47–56 2. Cruz-Neira C, Sandin DJ, DeFanti TA, Kenyon RV, Hart JC (1992) The CAVE: audio visual experience automatic virtual environment. Commun ACM 35(6):64–72 3. Slater M, Wilbur S (1997) A framework for immersive virtual environments (FIVE): speculations on the role of presence in virtual environments. Presence Teleop Virt Environ 6:603–616 4. Krueger M (1991) Artificial reality II. Addison-Wesley, Boston 5. Brooks FP, Ouh-Young M, Batter JJ, Kilpatrick PJ (1990) Project GROPE: Haptic displays for scientific visualization. In: SIGGRAPH ’90: Proceedings of the 17th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 177–185 6. Milgram P, Takemura H, Utsumi A, Kishino F (1995) Augmented reality: a class of displays on the reality-virtuality continuum. Proc SPIE Telemanipul Telepresence Technol 2351:282–292 7. Stephenson N (1992) Snow crash (1993 paperback ed.). Bantam Books, New York, p 440. ISBN 9780553898194 8. Koduri RX (2021, December 14) Powering the metaverse. Intel 9. https://www.bloomberg.com/press-releases/2023-03-06/metaverse-market-to-hit-936-6-billionby-2030-at-cagr-of-41-6-grand-view-research-inc

Part I

Fundamentals

Chapter 2

Computer Graphics

The objectives of this chapter are quite ambitious; you should refer to the references cited in each section to get a deeper explanation of the topics presented. By the end of this chapter, you will be familiar with the main building blocks of a 3D graphics interactive application and will be able to use this knowledge to build your own VR projects. We will give a brief overview of the following topics: • Mathematics: Coordinate systems, vectors, and transformation matrices. These are used to represent, visualize, and animate 3D objects, characters, and the like. • 3D modeling: 3D primitives and geometric representations. This is how we actually create objects with a particular 3D shape. • Image rendering: Illumination models, lighting, colors and textures. These are used to visualize 3D shapes in a realistic and believable way. This knowledge will give you the foundation required to understand any 3D graphics system, be it a software API like OpenGL or DirectX; a modeling/animation tool such as Maya, Blender 3D, or 3DS MAX; or a game engine like Unity or Unreal. Finally, you will be introduced to the game engines.

2.1 2.1.1

Mathematics Coordinate Systems

A coordinate system is a way to determine the position of a point by a set of numbers (coordinates) that can be distances from a set of reference planes, angles subtended with an origin, or a combination of both [1]. Different coordinate systems have been defined; each of them allows representation of 3D shapes; however, depending on the shape, it may be easier to use one © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_2

15

16

2

Computer Graphics

coordinate system than another. The most used coordinate systems are the cartesian and the polar or spherical coordinate systems. Both allow for specifying the position of a point in a 3D Euclidean space, a space in three dimensions just like the world we know. It was named Euclidean after Euclid of Alexandria, a Greek mathematician whose treatise The Elements defined the principles of geometry. There are also non-Euclidean geometries, but the kind of shapes we will work with will be represented in a 3D Euclidean space.

Cartesian Coordinate Systems This is the most used coordinate system in computers. The position of a point is determined by three distances, from each of the planes. The axes are at right angles (orthogonal) and are typically used to specify spatial dimensions: width, height, and depth (x, y, z). Cartesian coordinates are the most common data to represent real, theoretical, and artistic worlds.

Spherical Coordinate Systems A spherical coordinate system is called a polar coordinate system when used in two dimensions. The position of a point is specified using one distance (ρ from the origin) and two angles (θ and φ) measured from the reference planes at right angles (see Fig. 2.1a). Spherical coordinates are commonly used in applications that need to define positions relative to the surface of the Earth, such as meteorology, geography, and oceanography. In such cases, the angles are better known as latitude and longitude. The origin is at the center of the planet, and the reference planes are the

Fig. 2.1 Point and shapes in polar coordinates. (a) A point P in spherical coordinates. (b) A cardioid function in polar coordinates (ρ, θ, φ)

2.1

Mathematics

17

Equator and Greenwich meridian. Spherical coordinates are also used in astronomy to specify the position of celestial objects. In this case, the angles are called right ascension and declination. As shown in Fig. 2.1b, some shapes are easier to specify and plot using polar coordinates than using Cartesian coordinates. However, most computer visualization systems, which are programmed using graphics APIs like OpenGL, use Cartesian coordinates. Thus, it is necessary to convert positions defined in polar/spherical coordinates to Cartesian coordinates through the following formulas: x = ρ sin φ cos θ

y = ρ sin φ sin θ

z = ρ cos φ

In the rest of the book we will use Cartesian coordinates; as mentioned earlier, they are the most commonly used and are better “understood” by 3D APIs and other computer graphics-related tools. In general, 3D virtual worlds are created out of 3D polygons, usually triangles or squares arranged in 3D space to form 3D surfaces that comprise more complex objects; this is illustrated in Fig. 2.2. The simplest polygon and the most efficient to represent, process, and visualize with current graphics hardware is a triangle. A triangle is defined by specifying the three points or vertices that compose it (see Fig. 2.2). Commonly available hardware (graphics cards) can take these three vertices and use them to draw a colored triangle on the screen at very fast speeds. Currently, graphics cards can not only draw hundreds of millions of colored triangles per second; but, for example, the NVIDIA RTX 4080 card can provide real time raytracing. The complete “drawing” process is called rendering; it involves computing the effects of color and textures assigned to each triangle in addition to lights and possibly shadows and reflections. We explain this process in Sect. 2.3.4. Triangles are the basic primitive for creation of 3D shapes, the building block of virtual worlds. To understand how to visualize and animate virtual entities in 3D space, we must be able to visualize and animate each triangle comprising the environment in a coherent way. We will provide more details on animation in Chap. 3.

Fig. 2.2 Triangle-based 3D shape

18

2.1.2

2

Computer Graphics

Vectors, Transformations, and Matrices

To illustrate the basic mathematical principles behind visualization of 3D shapes, we start with a single triangle. In order to represent a triangle shape in a computer, we use a Cartesian coordinate system and three points in the space to delimit the shape; we call them vertices. A vertex v is composed of three Cartesian coordinates (vx, vy, vz). This set of three coordinates is a 3D vector, an array of three values, which are the vector components. A triangle can thus be defined by three 3D vectors that specify the coordinates of each vertex in the space: v1, v2, v3. In a 3D environment, triangles and any other 3D shape can suffer three basic coordinate transformations: they can be translated, rotated, or scaled. When combined, these three basic transformations allow us to produce a great variety of animation, from simple linear motion to complex trajectories and surface deformations. Some important questions to answer are: How can we translate/rotate/scale a 3D shape? What happens to the vertices forming the shape? Translating a 3D shape means adding a vector T to each of its composing vectors. The addition of two vectors consists of summing each of its components: a + b = c, where the components of vector c are: (ax + bx, ay + by, az + bz). Scaling affects the components of each vertex by multiplying them by a scale factor s. The components of vector v when scaled by a given factor are (vx × sx, vy × sy, vz × sz). The rotation transformation is a combination of addition and multiplication operations that result from basic trigonometry. It is easier and computationally efficient to use matrix representation and matrix multiplication to perform a rotation transformation over a vector. In fact, the most convenient way of processing coordinate transformations is to set up matrices containing the parameters of the transformations and use matrix multiplication to reduce a series of transformations to a combined single transformation matrix.

Homogeneous Coordinates To process transformations in a uniform way, it is necessary to add a component to the vectors and increase the dimensions of the matrices. For 3D space, 4D vectors are used, the four components are called homogeneous coordinates. The components of a 4D homogeneous vector are reduced to three dimensions by the following normalization operation: ½x, y, z, h →

x y z , , h h h

Any vertex in 3D space can be represented as [x, y, z, 1]; 4 × 4 matrices should be set up in such a way that multiplying a transformation matrix M by a vector v will

2.1

Mathematics

19

give as the result a vector v′ whose components [x′, y′, z′, 1] correspond to the coordinates of v after applying the transformations parameterized in M. The fourth dimension (fourth component) will normally remain 1. Most of the graphical transformations can be broken down into products of the three basic ones. Transformation matrices are defined as follows.

Translation Vector v′, which is the position of v, after a translation of tx units over the X axis, ty units on Y, and tz units on the Z axis is calculated as follows:

v0 = T  v =

1

0 0

tx

x

0

1 0

ty

y

0 0

0 1 0 0

tz 1



z 1

x þ tx =

y þ ty z þ tz 1

When applied to all the vectors (vertices) that comprise a 3D shape, this transformation will move the shape by the magnitude and direction of vector [tx, ty, tz]. Note that translation cannot be achieved using 3 × 3 matrix multiplication with a 3D vector. This is one of the most important reasons for using homogeneous coordinates: They allow composite transformations to be handled by a single matrix.

Scale The scaled version of v is calculated as follows:

v0 = S  v =

sx 0

0 sy

0 0

0 0

0 0

0 0

sz 0

0 1



x y z 1

=

sx  x sy  y sz  z 1

A scaled 3D shape normally changes both its size and its location. Only shapes centered on the origin will remain in the same position. Scaling may also be done with reference to a point P0 = [x0, y0, z0], which may be the center of the 3D shape. Translating a 3D shape centered at Po can be done through a combined transformation: translating all the vertices of the shape to the origin, scaling, and then translating the vertices again to restore the shape to its original position. The vertices comprising the 3D shape can be arranged in a matrix M with n columns (one per vertex) and four rows (homogeneous coordinates). The transformations will be applied by multiplying the following matrices by M: T-1 (the inverse of T ), for

20

2

Computer Graphics

the initial translation to the origin, S for scaling, and T for restoring the original position: M0 = T - 1  S  T  M 1 0 0 xo M0 =

0 1

0

yo

0 0

1

zo

0 0 0 1 0 0

1 - xo

0

1 0

- yo

0

0 1

- zo

0 0 0 sx 0 0 M0 =



sx

0

0

0

0

sy

0

0

0

0

sz

0

0



1 xo ð1 - sx Þ

0

sy

0

yo 1 - sy

0

0

sz

zo ð1 - sz Þ



0 0 1 x1 x2 . . .

xn

y1

y2

...

yn

z1

z2

...

zn

1

1 x1



... 1 x2 . . .

xn

y1

y2

...

yn

z1

z2

...

zn

0 0 0 1 1 1 ... 1 sx x1 þ xo ð1 - sx Þ sx x2 þ xo ð1 - sx Þ . . . sx xn þ xo ð1 - sx Þ M0 =

sy y1 þ yo 1 - sy

s y y2 þ yo 1 - s y

. . . s y yn þ yo 1 - s y

sz z1 þ zo ð1 - sz Þ

sz z2 þ zo ð1 - sz Þ

...

sz zn þ zo ð1 - sz Þ

1

1

...

1

where M′ contains the vertices of the 3D shape scaled with reference to Po. At this point, it is important to note that matrix multiplication is associative but not commutative: The order in which the transformations are applied must be carefully observed. If the order of the transformations were irrelevant, it would be possible to rearrange the equation M′ = T-1 S T M as follows: M0 = S  T  T - 1  M = S  I  M = S  M which is clearly false.1 This reductio ad absurdum argument shows that, in general, exchanging the multiplication order of transformation matrices produces different results.

1 Multiplying a squarematrix by its inverse (e.g., TT-1) yields the identity matrix I (a matrix that has a 1 for each element on the main diagonal and 0 elsewhere), the identity for matrix multiplication.

2.1

Mathematics

21

Rotation Let us consider rotating a vertex v over the plane formed by the X- and Y- axes: v = [x, y, 0]. Rotation transformation will be applied about the origin by an arbitrary angle θ. To calculate the transformed coordinates v′, we should consider the position vector from the origin to point v. The length of v is r at an angle φ to the X axis. The position vector v is rotated about the origin by the angle θ to v′. Since the z- coordinate of v is 0, we can write the position vectors as follows: v = ½x, y = ½rcos φ, r sin φ v0 = ½x0 , y0  = ½rcosðφ þ θÞ, r sinðφ þ θÞ Using trigonometric identities2 allows writing v′ as: v0 = ½x0 , y0  = ½r ðcos φ cos θ - sin φ sin θÞ, r ðcos φ sin θ þ sin φ cos θÞ Using the definitions of x and y allows rewriting v′ as: v0 = ½x0 , y0  = ½xcos θ - y sin θ, x sin θ þ y cos θ Thus, for a vertex on the plane X - Y, the transformation matrix for a general rotation about the origin by an arbitrary angle θ is: R=

cos θ - sin θ sin θ

cos θ

This corresponds to a rotation around the Z- axis. Any 3D vertex rotated around the Z- axis will keep unaltered its z- coordinate. Thus, we can convert the previous 2 × 2 matrix into a 4 × 4 matrix with homogeneous coordinates: cos θ Rz =

-

sin θ

0

0

sin θ 0

cos θ 0

0 1

0 0

0

0

0

1

Notice how this transformation matrix only modifies the x- and y- coordinates of a 3D vector. Following an analogous procedure, we can obtain the transformation matrices for rotations around the X- and Y- axes:

2

cos(φ ± θ) = cosφcosθ ∓ sinφsinθ sin(φ ± θ) = cosφsinθ ± sinφcosθ

22

2

Rx =

1 0

0 cos θ

0 - sin θ

0 0

0

sin θ

cos θ

0

0

0

0

1

Ry =

Computer Graphics

cos θ 0

0 1

sin θ 0

0 0

- sin θ

0

cos θ

0

0

0

0

1

As in the case of scaling transformation, the rotation matrices we have defined allow us to rotate a 3D shape centered at the origin of the coordinate system. If the shape we want to rotate is not centered, it must first be translated to the origin, rotated and then translated back to its original position: M′ = T-1  Rz  Ry  Rx  T  M, where M and M′ contain the n vectors corresponding to the original and transformed vertices that comprise the 3D shape; see the example for the scaling transformation. As we have shown before, the order in which transformation matrices are multiplied is critical; Fig. 2.3 shows the different results obtained by exchanging the order of translation and rotation transformations.

2.1.3

Angular Representations

The rotation matrices defined earlier correspond to a particular angular representation called Euler angles. Angular representations are mathematical approaches to specify the orientation of a 3D shape. Each angular representation has pros and cons; some are more “intuitive” than others, as is the case of Euler angles, but present some problems when specifying certain types of orientation. Other representations are more robust, in the sense that they allow for representing complex orientations without ambiguity, such as quaternions, but can be more difficult to understand when you are not familiar with them.

y

y 2: Rotation

1: Rotation 2: Translation x

1: Translation x

z

z (a)

(b)

Fig. 2.3 Noncommutativity of transformations composition. (a) RT  M. (b) T RM

2.1

Mathematics

23

Euler Angles The Euler angles are one of the most intuitive angular representations. It was developed by Leonhard Euler to describe the orientation of a rigid body in a 3D Euclidean space. The orientation of an object (3D shape) can be expressed as a sequence of three rotations described by the Euler angles. The order and the axes around which the rotations are applied can be specified in different ways. Other descriptions of rotations in 3D space equivalent to Euler angles are the Tait-Bryan angles, also called Cardano angles or nautical angles. Such representations are used to describe the orientation of an aircraft and name the angles as roll, pitch, and yaw. A common definition of Euler angles is as follows: • θ, the angle between the X- and Y- axes, roll angle • φ, the angle between the Z- and X- axes, the yaw angle • ψ, the angle between the Z- and Y- axes, the pitch angle Precautions must be observed when applying rotation transformations using Euler angles. A well-known problem called “gimbal lock”3 occurs when applying rotations of 90°. As illustrated in Fig. 2.4, when a 90° rotation around the X- axis is applied, subsequent rotations on the Y- or Z- axes give unexpected results. The local coordinate system of the shape is in such a position that, after rotation around X-, the Z- axis has the same orientation as the original Y, but rotations using Euler angles are applied according to the global coordinate system, hence the unexpected

45° on y

90° on x

y

x x

y

z

z Local coordinate system

Unexpected result due to Gimbal lock

Expected result without Gimbal Lock

Fig. 2.4 Gimbal lock

3

A gimbal is a mechanical device that allows the rotation of an object in multiple dimensions. A gimbal lock occurs when two of the three gimbal rings align together so that one of the rotation references is canceled.

24

2

Computer Graphics

result. Other angular representations try to avoid the gimbal lock and provide more flexibility to describe 3D shape orientations.

Axis Angle This angular representation is an alternative way to describe the orientation of an object in 3D space. Rotations are represented by a unit vector and an angle of revolution around the vector; see Fig. 2.5. This angular representation is commonly used in robotics, animation, and simulation. Rotation about an arbitrary axis in space can be calculated using translations and rotations about the coordinate axes if we have an arbitrary axis in space passing through the points O (x0, y0, z0) and P (x1, y1, z1). The rotation about this axis by an angle δ is calculated as follows: • Translate point (x0, y0, z0) to the origin of the coordinate system. • Perform necessary rotations to make the axis of rotation coincident with the Zaxis (the choice of axis is arbitrary). This implies a rotation about the X- axis and a rotation about the Y- axis. • Rotate about the Z- axis by the angle δ. • Apply the inverse of the rotation transformations to put the axis of rotation back to its original orientation. • Perform the inverse of the translation to leave the shape and its rotation axis at the original position. The complete axis-angle transformation can be calculated as follows: M 0 = T - 1  Rx- 1  Ry- 1  Rδ  Ry  Rx  T  M where M and M′ contain the n vectors corresponding to the original and transformed vertices that compose the 3D shape we want to transform. Fig. 2.5 Rotation about an arbitrary axis in space

2.1

Mathematics

25

The required matrices are:

T=

1 0

0

- x0

0 1

0

- y0

0 0

1

- z0

0 0 0 1 0 Rx =

Ry =

Rδ =

1 0

0

1

0

0

0

0

cz =d

- cy =d

0

cz =d

0

0

1

d

0

- cx

0

0

1

0

0

cx

0

d

0

0

0

0

1

0

cos α

- sin α

0

0

sin α

cos α

0

0

0

0

1

0

cos ð - βÞ

0

sin ð - βÞ

0

0

1

0

0

- sin ð - βÞ

0

=

0 cy =d

cos ð - βÞ 0

0 0 cos δ - sin δ 0

0 0

sin δ

cos δ

0

0

0

0

1

0

0

0

0

1

1

0

=

We will now review other more compact angular representations that have interesting properties for robotics, computer graphics, and virtual reality applications.

Quaternions Quaternions were discovered and described by Sir William R. Hamilton in 1844 [2]. In 1985, Ken Shoemake proposed using them as an efficient way to represent 3D rotations of objects in computer animation applications [3]. Quaternions were conceived by Hamilton as extended complex numbers: w + ix + jy + kz, composed of imaginary numbers: i2 = j2 = k2 = -1, ij = k = ji and real numbers: w, x, y, z. Quaternions can be treated as quadruples of real numbers: [w, x, y, z], with operations of addition and multiplication suitably defined. The components can be grouped into a real part, w, called scalar and an imaginary part, (x, y, z), which Hamilton named vector. We usually write a quaternion as [w, v], with v = (x, y, z). A real number s can be represented as a quaternion: [s, 0], a vector v 2 R3 can also be represented as a quaternion: [0, v]. Unit quaternions have unit norms: |q| = 1, x2 + y2 + z2 + w2 = 1.

26

2

Computer Graphics

Quaternion multiplication is defined as follows: q1 q 2 =

w2 w1 w2 - v1  v2 = v2 w1 v2 þ w2 v1 þ v1 × v2

w1 v1

Quaternion multiplication is associative but not commutative: q1 q 2 ≠ q 2 q 1 q 1 ð q 2 q 3 Þ = ð q1 q 2 Þ q 3 The multiplication identity quaternion is: 1

qi =

,

ð0, 0, 0Þ

qqi = q

The conjugate of a quaternion is written q and is defined as: q =

w v



=

w -v

ðq Þ = q ðq1 q2 Þ = q2 q1 Based on these definitions, the inverse of a quaternion is: 1 q-1 =



q jqj

qq - 1 =

0 0 0

the unit length quaternion. Given a point p in 3D space (a 3D vector), represented by a quaternionqp = and a unit quaternion, q =

cos ðθ=2Þ

sin ðθ=2Þr about an arbitrary axis r by an angle θ: qqp q - 1 = =

0 p

,

the product qqpq-1 results in p rotating

w

0

w

v

p

-v

=

w

pv

v

wp - p × v

wp  v - v  wp þ v  p × v = 0 wðwp - p × vÞ þ ðp  vÞv þ v × ðwp - p × vÞ

2.1

Mathematics

27

Thus, we can calculate p′, which is the new position of point p after a rotation about an arbitrary axis with normalized coordinates (xr, yr, zr) by an angle θ as follows: p0 = RðqÞ  p

q=

w = cos ðθ=2Þ x = xr sin ðθ=2Þ y = yr sin ðθ=2Þ z = zr sin ðθ=2Þ

where R(q) is the matrix form of quaternion

RðqÞ =

1 - 2y2 - 2z2

2xy - 2wz

2xy þ 2wz 2xz - 2wy

1 - 2x - 2z 2yz þ 2wx

0

0

2

2

2xz þ 2wy

0

2yz - 2wx 1 - 2x2 - 2y2

0 0

0

1

If q1 and q2 are unit quaternions, the combined rotation of rotating first by q1 and then by q2 is expressed as q3 = q2  q1. Any unit quaternion multiplied by the identity quaternion will not be changed. Representing a rotation as a quaternion requires four numbers. It is easy to construct the quaternion corresponding to any given axis and angle, and vice versa. This is more difficult using Euler angles. Another important advantage of quaternions is that they allow us to produce “smooth rotations,” slow rotations commonly used to control the point of view (camera motion) in a video game or virtual environment. Techniques such as spherical linear interpolation (SLERP) are commonly used with quaternions to interpolate camera rotations between an initial and a final orientation. When multiplying several rotation matrices, rounding errors accumulate due to the finite precision of the microprocessor. When rounding errors have surpassed a threshold, quaternions or matrices do not represent the required rotations anymore and lead to unpredictable results. To solve problems due to rounding errors, quaternions can be normalized; an analogous but harder procedure is required to fix a rotation matrix (make it orthogonal again).

Exponential Maps Exponential maps require only three parameters to describe a rotation. This representation does not suffer from the gimbal lock problem, and its singularities occur in a region of the parameter space that can easily be avoided [4]. The exponential map represents rotations as vectors in R3 (vector with three components), where the axis

28

2

Computer Graphics

of rotation is defined by the vector direction (components) and the rotation angle is the magnitude of the vector. A detailed discussion of the benefits of exponential maps, compared to quaternions and rotation matrices is presented by F. Sebastian Grassia in [5].

Choosing the Best Angular Representation We have presented different approaches to represent the orientation of a shape in 3D space. A very important question to answer is which angular representation to use in a Virtual Reality application. In fact, the answer depends on the part of the application we are referring to: user interface, animation constraints, interpolation, composition, or rendering. User interface implies input data; this is where we “manually” set up the orientation of shapes in a virtual environment or control the camera to watch the virtual scene from different points of view. Common ways to input this information is through keyboard, joystick, mouse, or more sophisticated devices such as magnetic trackers. For the user interface, we need to have an intuitive representation that allows us to easily visualize the orientation; this is commonly achieved with Euler angles. Animation constraints are different restrictions that we impose on the virtual world to achieve useful results. Constraints can be, for example, the joint limits of a virtual character, that prevent the legs or arms from adopting unnatural postures. We can also constrain the camera orientation to ease navigation within a virtual world. This is easily done using Euler angles; sometimes quaternions are used, but they are more difficult to handle. Interpolation refers to the calculation of intermediate orientations to create a smooth transition between two postures/orientations. Interpolation is used when changing the camera point of view or when animating 3D shapes. We have already mentioned that quaternions give very good visual results when interpolated using spherical linear interpolation, commonly known as SLERP. Exponential maps are also suitable for interpolation. Composition is the process of accumulating different transformations to achieve a desired goal; it can be the series of transformations required to move from point A to point B within a virtual environment: translation, interpolation of camera orientation, and the like. Quaternions and matrices are particularly useful for compositing transformations because they can be multiplied to accumulate (compose) complex series of consecutive translations, rotations, and/or scaling operations. Rendering is the process used to display on the screen the virtual environment when viewed from a specific point of view. It is the final phase of the visualization pipeline of any computer graphics application. Orientation matrices are the most suitable representation since they can be processed in a very efficient way (hardware acceleration of the graphics card).

2.1

Mathematics

2.1.4

29

Projections

At this point we have presented the fundamentals of representing points in 3D space that can be used to create 3D shapes. We have also described how to transform those points to translate them, scale them, and rotate them. The next step in this overview is to describe some details about how to “draw” 3D shapes on the screen. So far, we have considered a 3D space; however, the screen is only 2D, we need a way to transform 3D coordinates to 2D. This is known as the projection transformation. Since most computer displays are planar viewing surfaces, the most used projection surface is a plane. Other projection surfaces, such as spheres or cylinders, could also be useful for virtual reality applications displayed on hemispherical or cylindrical screens. We will concentrate on planar projections. The viewing surface is also called the plane of projection, transformation projections are also performed on a vertex basis. Straight lines, called projectors, are traced through the points in 3D space, and the projected point is the intersection between the projector and the plane of projection. There are two main types of planar projections: orthographic and perspective, with their associated transformation matrices.

Orthographic Projection For this projection, we specify a square or rectangular viewing volume formed by the far, near, left, right, top, and bottom clipping planes. Anything outside this volume is not drawn. All objects that have the same dimensions appear the same size, regardless of whether they are far away or near; see Fig. 2.6a. Orthographic projection is frequently used in architectural design and computer-aided design (CAD). Orthographic projections are also used to add text or 2D overlays on top of 3D graphic scenes. The most used plane of projection is one that is parallel to the X - Y plane; in such a case, the transformation matrix Rz with projectors in the z direction is defined as:

Fig. 2.6 Projection. (a) Orthographic. (b) Perspective

30

2

Pz =

1 0

0 0 1 0

0 0

0

0 0

0

0

0 0

1

Computer Graphics

In this projection, all the projectors go through a fixed point, the center of projection, which is sometimes referred to as the eye or viewpoint. The center of projection is usually placed on the opposite side of the plane of projection from the object being projected. Perspective projection works like a pinhole camera, forcing the light rays to go through one single point. This projection adds the effect that distant objects appear smaller than nearby objects. The viewing volume is like a pyramid with the top shaved off. The remaining shape is called the frustum. Objects nearer to the front of the viewing volume appear close to their original size, but objects near the back of the volume shrink as they are projected to the front of the volume; see Fig. 2.6b. This type of projection gives the most realism for simulation and 3D animation. The matrix representation of a perspective transformation is:

Pp =

1 0

0 1

0 0

0 0

0

0

1

0

0

0

1=f

0

where f is the distance between the viewpoint and the plane of projection. Transforming a point p = [x, y, z, 1] using transformation matrix Pp will yield p′ = [x, y, z, z/f]. Applying the rules of homogeneous coordinates, 3D coordinates are obtained by normalizing the 4D vector, dividing its components by z/f. Thus, the coordinates of p′ projected on a 2D plane at a distance f from the viewpoint are: [fx/ z, fy/z].

2.2

3D Modeling

This section will focus on how to “draw” 3D graphics on the computer screen, considering different ways to describe and visualize 3D curves, surfaces, and volumes. This is known as 3D modeling.

2.2.1

Geometric Representations

Geometric representations are ways to describe the arrangement of 3D vertices to create 3D shapes. There are three basic primitives or fundamental ways to create a

2.2

3D Modeling

31

3D shape: points, curves, and polygons. Points are the simplest primitive; a 3D shape can be composed of a large number of points to cover the whole 3D surface, where each point can be represented by a 3D vertex. A more practical and compact way of representing 3D shapes is using curves, which can be approximated by lines or points. We will dedicate most of this section to explaining different ways to describe curves, since these are some of the most powerful geometric representations. Curves can be used to describe complex 3D surfaces. In the end, computer graphics cards use polygons to approximate any 3D surface. The simplest polygon and the most efficient one to render is a triangle. Thus, at the lowest level, any 3D shape should be represented by a set of triangles. The more triangles used, the more accurate is the surface.

2.2.2

Curves

A curve can be represented as a collection of points lying on a 2D or 3D plane. If the points are properly spaced, they can be interconnected using short straight-line segments, yielding an adequate visual representation of the curve. Curves could be represented as a list of points; the higher the number of points used to represent a curve, the more accurate and smoother will be its visualization. However, using an analytical representation has several advantages in terms of precision, compact storage, and ease of calculation of intermediate points. Any point on an analytically represented curve can be precisely determined. When represented by a collection of points, intermediate points must be obtained using interpolation, which leads to losses in drawing precision. In most cases, interpolated points do not lie on the curve. Analytically represented curves are easily reproduced, and such representation is useful when continuous alteration of the curve shape is required during the design phase. Thus, we will concentrate on analytical definition of curves in 3D space, and we will see that this allows us to describe 3D surfaces. From the mathematical point of view, a curve can be represented in a parametric or nonparametric form.

Nonparametric Curves Nonparametric representations are either explicit or implicit. For a 2D curve, an explicit, nonparametric form is given by y = f(x). A typical example is the equation for a straight line, y = mx + b. In this form, for each value of x, only one value of y is obtained. Closed or multiple-value curves, for example, circles, cannot be represented explicitly. Implicit representations of the form f(x,y) = 0 overcome this limitation. For instance, the general second-degree implicit equation, ax2 + 2bxy + cy2 + 2dx + 2ey + f = 0, describes a variety of 2D curves, known as conic sections; see Fig. 2.7. A straight line is obtained by setting a = b = c = 0. In this case, the equation is:

32

2

Computer Graphics

Fig. 2.7 Conic sections

dx þ ey þ f = 0 d f y = - x - = mx þ b0 e e Nonparametric curve representations present some limitations. First, they are axis-dependent; the choice of coordinate system affects the ease of use. Moreover, when points on an axis-dependent nonparametric curve are calculated at equal increments in x or y, they are not evenly distributed along the curve length. An unequal point distribution affects the quality and accuracy of the visualization. Despite these issues, nonparametric representations are useful for several applications. Parametric representations overcome these limitations.

Parametric Curves In parametric form, the coordinates of each point in a curve are represented as functions of a single parameter that varies across a predefined range of values. For a 3D curve with t as the parameter, the Cartesian coordinates of a point on the curve are: x = xð t Þ y = yð t Þ z = z ð t Þ The vector representing the position of a point on the curve is: Pðt Þ = ½xðt Þ, yðt Þ, zðt Þ Since a point on a parametric curve is specified by a single value of the parameter, the parametric form is axis independent. The curve endpoints and length are fixed by the parameter range. In many cases, it is convenient to normalize the parameter range for the curve segment of interest to 0 ≤ t ≤ 1. Being axis-independent, a parametric curve can be manipulated using the transformations we have previously discussed.

2.2

3D Modeling

33

A typical example of a curve in parametric form is the equation of the circle: Pðt Þ = ½r cos t, r sin t , 0 ≤ t ≤ 2π where r is the radius of the circle and the parameter t is associated with the geometric angle (in radians) measured counterclockwise from the positive X- axis. Equal increments in t produce equal arc lengths along the circumference of the circle, and the visual appearance is good. For a detailed discussion on the parametric forms of the circle and other conic sections, you can refer to Chap. 4, “Mathematical Elements for Computer Graphics,” by Rogers and Adams [6]; many other printed and electronic sources are also available on this subject.

Splines Our interest in curve representation will focus on a particularly useful type of parametric curve known as splines. The name spline comes from the shipbuilding industry, where flexible strips of wood, plastic, or metal were used to create full or almost full-size drawings of the boat shape and its parts. The splines were held in place with lead weights, called ducks. Before the development of computers, physical splines were the most common way to reliably draw smooth curves. Computers made possible the development of mathematical models and tools to reproduce and accelerate the process of drawing and manufacturing complex shapes. Pierre Bézier and Paul de Faget de Casteljau developed a way to represent curves and surfaces by specifying a small set of control points. This greatly simplifies the design process of 3D shapes visualized in a computer system, allowing interactive adjustments to the shape of the curve or surface. Such curves have kept the name of Bézier because he was the first to publish his results.4

Bézier Curves The simplest Bézier curve requires only two control points B0 and B1 and produces a straight line between such points. It is a curve of order 2 and degree 1. Control points can be 2D or 3D vectors for curves on a plane or in 3D space, respectively. The parametric form of a linear Bézier curve is: Linear Bezier : Pðt Þ = B0 þ ðB1 - B0 Þt = ð1 - t ÞB0 þ tB1

0≤t≤1

It is interesting to point out that this formula is equivalent to a linear interpolation.

An interesting review of the history of mathematical splines can be found in “A History of Curves and Surfaces in CAGD,” by G. Farin, in Handbook of Computer Aided Geometric Design [7].

4

34

2

Computer Graphics

A quadratic Bézier is a curve of order 3 and degree 2, requiring three control points; it corresponds to the square of the basis equation (1 - t) + t = 1: ðð1 - t Þ þ t Þ2 = 1 ð1 - t Þ2 þ 2ð1 - t Þt þ t 2 = 1 where each term is weighted by a control point: Quadratic Bezier : Pðt Þ = ð1–t Þ2 B0 þ 2ð1–t ÞtB1 þ t 2 B2 Figure 2.8 shows an example of a cubic Bézier curve. These are one of the most frequently used splines. They are used in imaging, animation, and typesetting and are defined as Cubic Bezier : Pðt Þ = ð1–t Þ3 B0 þ 3ð1–t Þ2 tB1 þ 3ð1–t Þt 2 B2 þ t 3 B3 Thus, a Bézier curve of order n + 1 and degree n can be defined as: n

Bi J n,i ðt Þ 0 ≤ t ≤ 1

Pðt Þ = i=0

where the Bézier basis or blending function is defined by the Bernstein polynomials: J n,i ðt Þ =

n i

t i ð1 - t Þn - i

n

= i!ðnn!- iÞ!. i Some important properties of Bézier curves are the following:

with

• The curve generally follows the shape of the polygon formed by the control points. • The first and last points on the curve coincide with the first and last control points. Fig. 2.8 A cubic Bézier curve and its defining polygon

B1 B2

B0

B3

2.2

3D Modeling

35

• The curve is contained within the convex hull of the polygon defined by the control points (see Fig. 2.8). • The tangent vectors at the ends of the curve have the same direction as the first and last points of the defining polygon. This is important for achieving continuity. • The curve can be transformed (translation, scale, rotation) by transforming only its control points.

Applications of Bézier Curves Bézier curves introduced the concept of vector graphics as opposed to raster images. A raster image is described in terms of pixels. Each pixel is coded with a color, depending on various algorithms (jpg, gif, png, bmp). The two major problems of raster images is that they are generally big in terms of file size and they are not scalable, meaning that if you zoom into the image, you will see the square pixels getting bigger and bigger. Bézier curves provided a solution to describe an image in terms of its mathematical representation. A vector graphics file will contain the coordinates of control points to describe a series of curves. The viewer application (renderer) will then draw the graphic. If the user wants to zoom in, the renderer will increase the space between the control points and redraw a smooth curve. Vector graphics are therefore scalable. Up to a certain degree of image complexity, it is generally more size-effective to store control point coordinates rather than pixel information. Nowadays, Bézier curves are the standard for the typesetting industry. The lines and curves required to define the shape (glyphs) of TrueType fonts, used to display and print high-quality text, are described by quadratic Bézier curves. Cubic Bézier curves are frequently used to define curved shapes in imaging, modeling, and animation applications. The advantages described earlier make mathematical splines one of the most useful and versatile tools for 3D shape modeling and animation, which are key elements of a Virtual Reality application. They allow for interactive design; remember they were created to precisely solve this problem: helping automobile designers create new shapes and describe them in a precise and formal (mathematical) way so that computers can process them.

B-Spline Curves Curves generated by using control points are dependent on the interpolation or approximation scheme being used to establish the relationship between the curve and the control points. This scheme is provided by a basis function. As we have explained, the basis function of Bézier curves is defined by the Bernstein basis. The use of Bernstein polynomials as basis functions limits the flexibility of the resulting

36

2

Computer Graphics

curves. One of the most important limitations is that any point of a Bézier curve is a result of blending the values of all control points; a change in one control point is felt throughout the entire curve. This makes it impossible to produce a local change within a curve. This kind of local change is very useful for 3D modeling and animation because they give the designer much more flexibility. There is another basis: the B-spline basis (Basis spline), which contains the Bernstein basis as a special case. In a B-spline curve, each control point Bi is associated with a unique basis function. Thus, each vertex Bi affects the shape of a curve only over a range of parameter values where its associated basis function is nonzero. The B-spline basis also allows us to change the order and degree of the resulting curve without changing the number of control points; this is impossible using the Bernstein basis. The parametric form of a B-spline curve is defined as: nþ1

Bi N i,k ðt Þ t min ≤ t ≤ t max ,

Pðt Þ =

2≤k≤n þ 1

ð2:1Þ

i=1

where the Bi are the position vectors of the n + 1 control points of the defining polygon and the Ni,k are the normalized B-spline basis functions. For the ith normalized B-spline basis function of order k and degree k - 1, the basis functions Ni,k(t) are defined by the Cox–de Boor recursion formulas [8, 9]: N i,1 =

1,

if xi ≤ t ≤ xiþ1

0,

otherwise ðt - xi ÞN i,k - 1 ðt Þ ðxiþk - t ÞN iþ1,k - 1 ðt Þ þ N i,k ðt Þ = xiþk - 1 - xi xiþk - xiþ1 The values of xi are elements of a knot vector satisfying the relation xi ≤ xi+1. The parameter t varies from tmin to tmax along the curve P(t). The convention 0/0 = 0 is adopted. The knot vector has a significant influence on the B-spline basis functions Ni,k(t) and hence on the resulting B-spline curve. The only requirement for a knot vector is that it should be an increasing series of real numbers. There are three types of knot vectors: uniform, open uniform (or open), and nonuniform.

Uniform Knot Vectors In a uniform vector, knot values are evenly spaced, for example, [0 1 2 3 4], [-0.2 – 0.1 0 0.1 0.2]. In practice, knot vectors usually begin at zero and are incremented by one to some maximum value or are normalized in the range [0..1].

2.2 3D Modeling

37

Open Uniform Knot Vectors Open uniform knot vectors have multiple repeated knot values at the ends. The number of repeated values is equal to the order k (in Eq. 2.1) of the B-spline basis function. Examples: k = 2: [0 0 0.25 0.5 0.75 1 1], k = 3: [0 0 0 0.33 0.66 1 1 1]. B-splines using open uniform basis functions are very similar to Bézier curves. When the number of control points is equal to the order of the B-spline basis and an open uniform vector is used, the B-spline basis can be reduced to the Bernstein basis. In such a case, the resulting B-spline is in fact a Bézier curve. For example, using four control points and the fourth-order (k = 4) open uniform knot vector [0 0 0 0 1 1 1 1] yields a cubic Bézier/B-spline curve.

Nonuniform Knot Vectors Nonuniform knot vectors have no specific arrangement; they can be unequally spaced and/or have multiple repeated values. They may be periodic or open, for example, [0 0 1 3 3 4], [0 0.10 0.6 0.7 1]. Nonuniform knot vectors allow us to vary the influence region of each control point so that some control points can affect the shape of the curve more strongly than others.

Nonuniform Rational B-Spline Curves Nonuniform rational B-splines, commonly called NURB curves, are the most general form of mathematical splines. To give the definition of NURB, we must first define what a rational B-spline is. A rational B-spline curve is the projection on 3D space of a 4D nonrational B-spline. Nonrational B-splines are polynomial B-splines, just like the ones we have already described. They are represented using homogeneous coordinates; each control point has three Cartesian coordinates plus the homogeneous coordinate h: [x, y, z, h]. Rational B-spline curves are defined as: nþ1

Pðt Þ =

Bi hi N i,k ðt Þ

i=1 nþ1

i=1

hi ≥ 0

ð2:2Þ

hi N i,k ðt Þ

If hi = 1, 8i the NURB equation reduces to the definition of nonrational B-splines (Eq. 2.1). Increasing hi pulls the curve closer to point Pi, while decreasing it pushes the curve farther from Pi. Setting hi = 0 eliminates the effect of Pi on the curve. Nonuniform rational B-splines are a generalization of nonrational B-splines and share with them most of their analytic and geometric characteristics. NURBs provide a precise way to represent common analytical shapes, such as lines, conic curves,

38

2

Computer Graphics

planes, free-form curves, quadratics, and sculptured surfaces, which are frequently used in Virtual Reality applications and computer-aided design.

2.2.3

Surfaces

Virtual reality applications usually require complex and visually appealing 3D shapes to create the illusion of being in an alternative but believable environment. We usually require modeling everyday objects like furniture, vehicles, tools, structures like houses and all kinds of buildings, and of course living beings, like animals and humans. These shapes can be modeled as 3D surfaces created with splines. Other modeling strategies exist (e.g., surfaces of revolution), but we will focus on surface creation using splines since this is a powerful and versatile method. Modeling with splines provides high interactivity at design time and compact storage of the models. As we have explained, to reconstruct a spline all we need are the control points, and in the case of NURBs, the knot vector. Very complex surfaces can be created with piecewise surfaces composed of Bézier, or more generally, NURB patches. How do we create a surface out of spline curves? We usually define a mesh of polygons out of a grid of splines. Splines are calculated from a net of control points; see Fig. 2.9. For each row of the grid, we calculate a spline at fixed intervals of parameter t; remember this is in fact a linear interpolation between control points. Each point of this spline will serve as a control point for the vertical splines. This

Fig. 2.9 Spline surfaces: Bézier patches

2.3

3D Rendering

39

way we create a grid of splines whose points can be used to create a mesh of triangles. Triangles are the basic 3D primitive for current graphic cards. This means that the most common way to visualize a 3D shape is by representing it as a polygonal mesh; this process is called tessellation. The more polygons we use, the more accurate and smooth the surface will be. Formally, a Bézier surface can be defined as the tensor product of Bézier curves, also known as Bézier patches: n

m

Pðs, t Þ =

Bi,j i=0 j=0

n i

si ð1 - sÞn - i

m j

t j ð1 - t Þm - j

Similar definitions based on the tensor product of the corresponding curve equations are used to define NURB patches. Other techniques, such as recursive subdivision, can provide even more accurate and smooth surfaces. Instead of generating a mesh of polygons from a grid of splines, it is also possible to render directly from the patch definitions. In this algorithm, described by Catmull and Clark [10], a set of control points is divided into two smaller patches, and this procedure is recursively applied until each sub-patch projects to less than one pixel on the screen. Then each patch is rendered by filling in the single pixel it projects to. While this method produces higher quality surfaces, it is also computationally intensive and hence slower. However, subdivision techniques can be hardware-accelerated using modern graphics cards with programmable GPU (graphics processing unit), see the work of Shiue et al. [11] and Loop and Blinn [12].

2.3

3D Rendering

3D rendering is the process of generating 2D projections of 3D shapes, considering the effects of color, light, surface, and other characteristics of the shapes. If we want them to be believable and visually appealing, 3D shapes should be colored, reflect light when illuminated, cast shadows, and so on. Light reflected from objects is what makes them visible for us. Different illumination models have been developed to simulate light reflections. An illumination model is the set of equations used to compute the color and brightness of a point on the surface of an object. It should determine how much light is reflected to the viewer from a visible point on a surface. Reflected light is a function of light source direction, color and intensity, viewer position, surface orientation, and surface properties [13]. Surface properties usually include colors, an opacity factor, a reflectance, a specular power parameter, and the like. Light reflections can be performed at three different scales: microscopic, local, and global. The exact nature of reflection from surfaces is best explained in terms of microscopic interactions between light rays and the surface. However,

40

2

Computer Graphics

simplifications at the local level usually provide acceptable results in real time, an essential requirement for virtual reality applications. In addition to the illumination model, a shading model can be used to avoid the faceted aspect of polygonal surfaces. The most popular shading models are Gouraud and Phong shading, although Phong shading is less frequently used because of its computational cost.

2.3.1

Local Illumination Model

As mentioned earlier, local illumination is an incomplete method, from the physics point of view, but simplification produces acceptable results for real-time interaction. The model considers that the light intensity (I ) at one point on a surface has three components: ambient, diffuse, and specular reflections: I = Iambient + Idiffuse + Ispecular. Rasterization-based algorithms apply this model and are implemented in current graphics boards.

Ambient Reflection The ambient term considers background illumination of unlit surfaces. It allows for some global control of brightness in a scene: Iambient = Iaka, where Ia is an ambient illumination constant defined for the entire scene and ka is an ambient reflection coefficient, usually restricted to values in [0..1].

Diffuse Reflection Diffuse reflection is characteristic of light reflected from a dull, matte surface. Objects illuminated only by diffusely reflected light exhibit an equal light intensity from all viewing directions. The reflected light intensity is independent of the position of the viewer and is proportional to the incident light energy Ii. The incident light energy per unit area depends on the cosine of the angle θ between the incident light direction and the outside normal5 to the surface n. This is known as Lambert’s cosine law: Idiffuse = Iikd cos θ, where kd is the diffuse reflection coefficient. If we calculate the normalized direction vector from the object surface to the light source s, the same equation can be expressed by the dot product of s and n: Idiffuse = Iikd(s  n).

5

The normal vector of a surface, for example, a triangle, is a normalized vector perpendicular to the surface plane.

2.3

3D Rendering

41

Specular Reflection Specular reflection is characteristic of light reflected from a shiny surface, where a bright highlight appears from certain viewing directions. Mirrors reflect incident light in one direction only and are considered perfect specular reflectors. Specular reflection depends on the perfect reflection direction r, the viewer direction v, and the surface normal n: Ispecular = Iiks(rv)n, where Ii is the incident light energy, ks is the specular reflection coefficient, and n can be considered a surface roughness parameter. The parameter n determines how close the surface is to a perfect specular reflector. Values of n larger than 100 specify mirror-like surfaces, values around 1 are used for matte surfaces. The reflection vector r can be expressed in terms of the surface normal and the light source vector s: r = 2n(n  s) - s.

Color The local illumination model we have reviewed calculates the light intensity per surface unit (e.g., per triangle); it is a scalar value that does not represent a specific color. Current display devices generate colors by mixing the three primary colors: red, green, and blue. This is known as the RGB model, in which the maximum intensities for the three primary colors are normalized to the range [0..1]. Different colors are obtained by specifying different intensity levels for each color component. Other color models have been proposed, such as the HSV or the CIE models. The HSV model uses the concepts of hue, saturation, and brightness to define colors. The CIE model was proposed by the Commission Internationale de L’Eclairage (CIE) in 1931. The CIE model specifies colors as the mixture of the three primary colors with different intensities whose sum is 1. There are formulas to specify a given color using any of the three-color models. However, in computer graphics and virtual reality applications, colors are usually described using the RGB model. Producing a colored image requires calculating the light intensity (ambient + diffuse + specular reflections) three times, once for each color component. Surfaces reflect the three-color components with different reflection coefficients. White light contains equal intensities for each primary color. However, diffuse reflection of white light will not necessarily have equal intensities on the three components. The color of diffuse reflection depends on the diffuse reflection coefficients of the surface; this gives the illuminated object its color. Different object colors are obtained when scenes are illuminated using colored light.

Gouraud Shading Light intensity is usually calculated on a per-polygon basis; this is known as flatshading, see Fig. 2.10. To reduce the visual effect of discontinuous changes in light intensity, known as the Mach band effect, intensity interpolation techniques are

42

2

Computer Graphics

Fig. 2.10 Shading algorithms for local illumination

applied. Gouraud shading6 is an intensity interpolation algorithm that partially eliminates intensity discontinuities. Light intensities are calculated for each vertex of the polygon; the normal of each vertex is the average of the surface normals of the adjacent polygons. Linear interpolation is used to determine the color at each point inside the polygon.

Phong Shading Not to be confused with the Phong illumination model, which considers the addition of ambient, diffuse, and specular reflections. Phong shading7 interpolates the surface normals, rather than the intensities of polygon vertices, as is the case with Gouraud shading. At each point on the polygon, the normal is calculated as an interpolation of the normal vectors at the vertices of the polygon. Each vector is then normalized, and for each point, an illumination model is used to determine the light intensity; see Fig. 2.10. Phong shading usually produces better results than Gouraud shading, but it is also more expensive in terms of computation.

2.3.2

Global Illumination Model

The global illumination model produces the most realistic images and has two main components: • Direct illumination: Light arriving directly from sources such as lamps or the sun. • Indirect illumination: Effects of light reflecting off surfaces in the environment. A general global illumination equation, also called the rendering equation describes all energy (light) exchanges between surfaces. This equation describes the flow of light throughout a scene. In contrast to local illumination models, which

6

This algorithm was proposed by the French computer scientist Henri Gouraud in 1971 [14]. Gouraud shading is hardware-accelerated in current graphics cards. 7 Both the illumination model and the shading algorithm were proposed by the Vietnamese computer scientist Bui Tuong Phong in the 1970s [15].

2.3

3D Rendering

43

are approximations, the rendering equation is based on the physics of light and provides theoretically perfect results. The equation states that the transport intensity of light from one surface point to another is the sum of the emitted light and the total light intensity that is scattered toward x from all other surface points [16]: Lo ðx, wÞ = Le ðx, wÞ þ total light

emitted light

Ω

f r ðx, w0 , wÞ Li ðx, w0 Þ ðw0  nÞ dw0 reflected light

where Lo (x,w) is light leaving point x in direction w; Le (x,w) is the light emitted, a property of the surface at point x; Ω    dw′ is an infinitesimal sum over a hemisphere of inward directions. fr(x,w′,w) is the bidirectional reflectance distribution function (BRDF), the proportion of light reflected at point x; Li(x,w′) is the incident light on point x from direction w′; and (w′n) is the attenuation of light due to the incident angle (see diffuse reflection). Some of the most frequently used algorithms for solving the rendering equation are ray tracing, radiosity, and photon mapping.

Ray Tracing The basic principle in ray tracing is to simulate the trajectories of light rays emitted by light sources and reflected by objects in the scene until some of them reach the eye or the camera. To optimize the algorithm, only rays arriving at the eye are simulated. A set of rays are generated from the viewpoint, and the algorithm looks for intersections with objects in the scene. A recursive approach can be applied to simulate reflections: rays intersecting an object generate reflected rays, which are then checked for intersections with other objects. At each intersection the rays contribute a color intensity, the sum of the contributions defines a color for each pixel on the screen. Figure 2.11 shows an example of the kind of images that can be generated. Ray tracing produces very realistic results, including the effects of reflection, refraction, and shadows, which are difficult to generate using local illumination. Although this technique has been traditionally considered too slow for real-time applications, several optimizations have been implemented. The most common optimizations of ray tracing include the use of binary space partitioning data structures such as k-d trees to speed up the search for ray intersections [17, 18]; the parallelization of the algorithm and its execution on PC clusters [19]; and the use of programmable hardware (GPU) [19, 20]. It is now possible to have interactive ray tracing; a few images per second, enough for examining a complex scene [21]. In the near future, this technique is expected to be used in real-time applications requiring at least 30 images per second, such as video games [22].

44

2

Computer Graphics

Fig. 2.11 Ray tracing example

Radiosity The radiosity method was originally introduced in the 1950s as a method for computing radiant heat exchanges between surfaces. In 1984, Goral et al. [23] proposed the application of this method to the simulation of light transfer between surfaces. This method considers that most surfaces in a real environment are diffuse reflectors: an incident ray of light is reflected or scattered in all directions within the entire hemisphere above the reflecting surface. The basic principle of radiosity is the energy balance in a closed scene: the radiosity (energy per unit surface) of a surface depends on its self-emitted radiosity and on the energy (light) emitted by all the other surfaces, received and reflected (re-emitted) by the surface. A very representative characteristic of images generated using radiosity is the effect of diffuse color bleeding; see Fig. 2.12. Color bleeding cannot be obtained using classical ray tracing methods. If the scene is subdivided into discrete surface elements or patches with constant radiosity, the energy balance can be expressed by the following equation: N

Bi = Ei þ ρi

F ij Bj j=1

2.3

3D Rendering

45

Fig. 2.12 Radiosity example

where Bi is the radiosity of patch Pi; Ei is the self-emitted radiosity of patch Pi; ρi is the reflectivity of Pi, (0 < ρi < 1); and Fij is the form factor of patches Pi and Pj; the proportion of the light emitted by Pi that is received by Pj. Writing this equation for each patch leads to a linear system of N equations. The most computationally expensive part of the radiosity method is the computation of the form factors. Several techniques for the computation of form factors have been proposed: projection methods with z-buffer schemes or exploration methods with ray tracing schemes. Efforts have also focused on parallelization of the algorithm [24]. A comprehensive review of the radiosity method and optimization algorithms is presented by Sillion and Puech in [25]. Radiosity methods calculate light interactions in an environment in a view-independent way. Different views can be rendered in real time; this characteristic makes radiosity a very interesting method for realistic visualization in virtual reality applications [26]. Implementations using programmable graphics hardware are making it feasible to generate images in real-time using radiosity [27].

Photon Mapping The classical ray tracing algorithm does not handle diffuse interreflections that produce color bleeding nor caustics (e.g., shimmering waves at the bottom of a

46

2

Computer Graphics

Fig. 2.13 Mix of ray tracing and radiosity rendering

swimming pool). Photon mapping is able to handle such effects. Diffuse interreflections and caustics are due to indirect illumination of diffuse surfaces. The photon mapping method estimates this kind of illumination using precomputed photon maps. This method was proposed by Jensen [28–30] and is implemented as an extension to ray tracing. Photon mapping is a two-pass method. In the first pass, photons are shot into the scene and stored in a photon map when they hit non specular objects. The second pass, the rendering pass, uses statistical techniques on the photon map to extract information about incoming light and reflected radiance at any point in the scene. The photon map is decoupled from the geometric representation of the scene. This means that the algorithm can simulate global illumination in complex scenes containing millions of triangles. Modified versions of this algorithm have been implemented using programmable graphics hardware [31]. In order to render the most realistic images, artists use a mix of all the techniques we have explained. Figure 2.13 shows an example of mixing radiosity and ray tracing.

2.3.3

Textures

The creation of believable images for virtual reality applications requires more than complex 3D shapes and realistic illumination. Most of the objects in the real world

2.3

3D Rendering

47

are composed of a variety of materials that affect their appearance; for example, objects are made of wood, marble, metal, fur, and so on. Materials can have very complex colored patterns that are difficult to reproduce. One of the most frequently used techniques to improve the visual appearance of 3D shapes is texture mapping [32]. The basic principle is to map a 2D image (texture) to a 3D shape in such a way that the 3D surface gets colored by the image; see Fig. 2.14. Texture mapping can also be the basis for other sophisticated rendering algorithms for improving visual realism and quality. For example, environment mapping is a view-dependent texture mapping technique that supplies a specular reflection to the surface of objects [33]. This makes it appear that the environment is reflected in the object; see Fig. 2.15a. Texture mapping can be considered as a method for modifying parameters of the shading equation such as the surface normal. This technique is known as bump mapping, it allows us to produce an effect of 3D relief [34, 35] or wrinkles [36] on a surface (see Fig. 2.15b). The coordinates of the point being shaded can be modified based on a parameterization of the surface defined by the texture coordinates. This technique is known as displacement mapping [37–39]; see Fig. 2.15c. In displacement mapping, the surface is actually modified, in contrast to bump mapping, where only the surface normal is modified. This means that displacement mapped surfaces produce correct self-occlusions, shadows, and silhouettes. The drawback of this technique is the large number of micro polygons required to deform the surface, which is undesirable for virtual reality applications due to the performance penalty that this induces. Texture mapping, including environment mapping and bump mapping, are implemented in hardware in current graphics cards and supported by the most common 3D graphics APIs: OpenGL and DirectX. Texture mapping is a very

Fig. 2.14 Texture mapping

48

2

Computer Graphics

Fig. 2.15 Texture mapping techniques. (a) Environment mapping. (b) Bump mapping. (c) Displacement mapping

cheap way to produce realistic images: scenes with or without texture mapping usually take the same time to be rendered.

2.3.4

Rendering Pipeline

We will explain how standard computer graphics application programming interfaces (APIs) such as OpenGL and DirectX process data in several steps to render 3D images; this process is known as the rendering pipeline and is the main purpose of graphics hardware. If no graphics hardware is present (e.g., no OpenGL; DirectXcompatible graphics card), then data processing is emulated in software, which usually results in slow performance. We will focus the discussion on the OpenGL API because it is now the standard supported by most operating systems and computer architectures, including Windows, Mac OSX, Linux, and all UNIX systems, in contrast to Microsoft’s DirectX, which is only supported by Windows PCs.

2.3

3D Rendering

49

OpenGL is an industry-standard, cross-platform API whose specification was finalized in 1992. The first implementations appeared in 1993. It was largely compatible with a proprietary API called Iris GL (Graphics Library), designed and supported by Silicon Graphics, Inc. To establish an industry standard, Silicon Graphics collaborated with various other graphics hardware companies to create an open standard: OpenGL. All versions of OpenGL through 1.5 were based on a fixed-function pipeline: The user could control various parameters, but the underlying functionality and order of processing were fixed. OpenGL 2.0 (September 2004) provided users with the ability to program data processing at the level of vertices and pixels (fragments). With this version of OpenGL, application developers are able to implement their own rendering algorithms, using a high-level shading language: GLSL.

Fixed Rendering Pipeline Figure 2.16 shows a simplified version of the rendering pipeline according to the OpenGL specification. The process starts with primitives (points, lines, polygons) formed by vertices and their associated attributes, such as color and surface normals. Different transformation matrices, as defined in Sect. 2.1.2, are applied to convert vertices into “eye coordinates.” There are two kinds of transformations: modeling transformations, used to place/ move 3D shapes in the scene, and viewing transformations, which specify the location of the viewer or the camera (transforming the whole scene to view it from a specific position and angle). Both modeling and viewing transformations are combined (multiplied in the required order: viewing x modeling) into a single transformation, called the modelview matrix. If lighting is enabled, the lighting calculations are performed using the vertices in eye coordinates, surface normals, light source positions, material properties, and other lighting information to produce a primary and a secondary color for each vertex. OpenGL uses a local illumination model; see Sect. 2.3.1. The final step of vertex operations involves texture coordinates. Storage

Transform

Rasterization

Fixed Transformation Geometry

Texture

Fixed Fragment Shader Rasterization

Programmable Transformation

Fig. 2.16 Rendering pipeline

Fragment Operations Programmable Fragment Shader

Color buffer

50

2

Computer Graphics

Lit and textured vertices in eye space are transformed by a projection matrix (see Sect. 2.1.4) into “clip coordinates.” Vertices out of the clipping volume are discarded. Depending on the type of primitive, new vertices may be generated at the intersection of the primitive and the view volume. Colors, texture coordinates, and other vertex attributes are assigned to the newly generated vertices by interpolating their values along the clipped edge. Vertices in clip space are still homogeneous coordinates (see Sect. 2.1.2); the next step is to divide them by the h-coordinate to yield normalized device coordinates. The resulting coordinate triplet is mapped to a 2D plane by the viewport transformation: mapping coordinates in clip space to physical window coordinates. In the next step, primitives (points, lines, or polygons) are decomposed into smaller units corresponding to pixels in the destination frame buffer. This process is called rasterization. Each of these smaller units generated by rasterization is called a fragment or a pixel. For example, the process of rasterization converts a triangle defined by three vertices into a set of fragments (pixels) covering the triangle area. A fragment has several attributes: window coordinate, depth (Z-coordinate), color, texture coordinates, and so on. The value for each of these attributes is determined by interpolation between the values specified or computed at the vertices of the primitive. When they are rasterized, vertices have a primary color and a secondary color. Color values are interpolated between the vertices if smooth shading (Gouraud shading, see Sect. 2.3.1) is used. Otherwise, the color values for the last vertex of the primitive are used for the entire primitive (flat shading). After fragments have been generated by rasterization, several fragment operations are performed. One of the most important operations at this stage is texture mapping. The texture coordinates associated with the fragment are used to access the texture memory in the graphics card. Textures can be accessed and applied in several ways specified within the application. Other fragment operations are fog: modifying the color of the fragment depending on its distance from the viewpoint, and color sum: combining the values of the fragment’s primary and secondary colors. The goal of the rendering pipeline is to take the graphics primitives defined by the application and convert them into pixels to fill in a frame buffer that will be used for display. For a more detailed explanation of the rendering pipeline, you can refer to the book by R. Rost, OpenGL Shading Language [40]. What we have described is the fixed functionality that is implemented in hardware in current graphics cards. However, modern hardware also offers the possibility to let the user define many steps of the rendering pipeline.

Programmable Rendering Pipeline Figure 2.16 shows how most of the operations at the level of vertices and fragments can be replaced by shaders. Operations are specified by programs called shaders, which are compiled and linked at run time and executed by the graphics processing unit (GPU) in modern graphics cards. There are several types of shaders, including:

2.4

Game Engines

51

– Vertex shaders: These shaders manipulate the position, color, and orientation of vertices in a 3D object. – Pixel shaders: These shaders control the color, lighting, and texture of individual pixels on the screen. – Geometry shaders: These shaders manipulate the geometry of an object, allowing for things like procedural generation of meshes. – Compute shaders: These shaders are used for general-purpose computing on the GPU and can be used for tasks like physics simulation and AI. When using OpenGL, shaders are written in a high level language called GLSL (OpenGL Shading Language) [40]. Direct3D, the 3D graphics component of the DirectX API, uses an analogous language called HLSL (High Level Shading Language). The latest versions of OpenGL (v4.6) and Direct3D (v12) offer several features that are useful for virtual reality developers. These include support for advanced shading techniques such as tessellation and geometry shaders; they provide support for multi-threaded rendering, which can help improve performance in complex VR environments. Finally, these modern Graphics APIs now offer support for GPU-accelerated ray tracing, which can be used for more realistic lighting and reflections in VR applications. Common uses of shaders include approximations to global illumination (see references in Sect. 2.3.2); skinning (displacement of vertices in articulated structures to give the illusion of bones covered by muscles and skin, essential for virtual humans and other animated characters); particle simulation (commonly used to simulate water, dust, etc.); and various effects derived from texture mapping (bump mapping, displacement mapping, environment mapping). Nowadays, the use of shaders is more and more frequent. Even for pedagogical purposes, shaders are useful for teaching computer graphics because they allow for implementing algorithms and equations in a more direct way [41], for example, implementing Phong illumination as a vertex shader. In terms of interactivity and ease of learning and deployment of interactive graphics applications, WebGL implements a standardized Javascript API that closely follows the OpenGL ES specification. OpenGL ES is optimized for mobile devices, such as the latest standalone HMDs from Meta and HTC, and has been used as the foundation for the WebGL API. WebGL is now supported by most desktop and mobile web browsers, and there are several online learning resources, such as webgl3d (http://webgl3d.info/), showing how to use shaders (using GLSL) to create 2D and 3D interactive content.

2.4

Game Engines

Developing a virtual reality application requires implementing a rendering pipeline following the principles described above. This can be a time-consuming task, requiring expertise in multiple areas of computer graphics and software engineering.

52

2

Computer Graphics

Instead of developing their own VR rendering software, industries and researchers are turning to available game engines and other interactive authoring software issued from the video game industry. There are free (mostly open source) and commercial products, and most of the time they allow one to reduce the costs of development, in terms both of time and financial budget. Among the most popular and advanced game engines we can consider: – Unity (https://unity.com/), a cross-platform game engine developed by Unity Technologies. It is based on a C# scripting API that provides easy access to commonly required game features. The engine features a built-in physics engine, and its Asset Store allows developers to upload and download creations from the Unity community. Unity is known for being easy-to-use and popular for indie game development, and it has been adopted by several industries outside gaming, such as film, architecture, automotive, engineering, construction, etc. (https:// gamedevacademy.org/what-is-unity/). It is one of the most popular tools currently used to develop VR applications for the most common platforms (SteamVR, Meta, Pico). – Unreal Engine (https://www.unrealengine.com) is a real-time 3D creation tool developed by Epic Games that enables the development of video games, architectural and automotive visualizations, linear film and television content, VR applications, etc. It includes a wide array of tools, including a level editor, animation and visual effects tools, and a visual scripting system called Blueprint. Similar to Unity, Unreal Engine offers a marketplace where developers can share and sell their creations. – OpenSceneGraph (http://www.openscenegraph.org) ia an open source high- performance 3D graphics toolkit. It is used by application developers in fields such as visual simulation, games, virtual reality, scientific visualization, and modeling. OpenSceneGraphs was written in Standard C++ and OpenGL; it runs on all Windows platforms, OSX, GNU/Linux, IRIX, Solaris, HP-Ux, AIX, and FreeBSD operating systems. – Godot (https://godotengine.org/) ia a free open-source game development tool that supports both 2D and 3D graphics. It includes a powerful game editor, builtin scripting language, and a rich library of customizable assets. In comparison to other game engines, such as Unity and Unreal, Godot may lack some of the more advanced features and industry support, but it provides a cost-effective and easyto-use alternative for developers looking to create smaller-scale games or prototypes.

References 1. Martin R (ed) (1993) Directions in geometric computing. Information Geometers Ltd, Winchester 2. Hamilton WR (1844) On quaternions; or on a new system of imaginaries in algebra. Philosophical Magazine XXV:10–13

References

53

3. Shoemake K (1985) Animating rotation with quaternion curves. In: SIGGRAPH ’85: Proceedings of the 12th annual conference on computer graphics and interactive techniques. ACM Press, New York, p 2454 4. Lepetit V, Fua P (2005) Monocular model-based 3D tracking of rigid objects: a survey. Found Trends Comput Graph Vision 1(1):1–89 5. Grassia FS (1998) Practical parameterization of rotations using the exponential map. J Graphics Tools 3(3):29–48 6. Rogers D, Adams JA (1990) Mathematical elements for computer graphics, 2nd edn. McGrawHill, New York 7. Farin G (2002) Handbook of computer aided geometric design. Elsevier Science, Amsterdam 8. Cox MG (1972) The numerical evaluation of B-splines. J Inst Math Appl 10:134–149 9. de Boor C (1972) On calculation with B-splines. J Approx Theory 6:50–62 10. Catmull E, Clark J (1978) Recursively generated B-spline surfaces on arbitrary topological meshes. Comput Aided Des 10(6):350–355 11. Shiue LJ, Jones I, Peters J (2005) A realtime GPU subdivision kernel. In: SIGGRAPH ’05: ACM SIGGRAPH 2005 papers. ACM Press, New York, pp 1010–1015 12. Loop C, Blinn J (2006) Real-time GPU rendering of piecewise algebraic surfaces. In: SIGGRAPH ’06: ACM SIGGRAPH 2006 papers. ACM Press, New York, pp 664–670 13. Whitted T (1980) An improved illumination model for shaded display. Commun ACM 23(6): 343–349 14. Gouraud H (1971) Continuous shading of curved surface. IEEE Trans Comput C-20(6): 623–629 15. Phong BT (1975) Illumination for computer generated pictures. Commun ACM 18(6):311–317 16. Kajiya JT (1986) The rendering equation. In: SIGGRAPH ’86: Proceedings of the 13th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 143–150 17. Bentley JL, Friedman JH (1979) Data structures for range searching. ACM Comput Surv 11(4): 397–409 18. Glassner AS (1984) Space subdivision for fast ray tracing. IEEE Comput Graph Appl 4(10): 15–22 19. Purcell TJ, Buck I, Mark WR, Hanrahan P (2005) Ray tracing on programmable graphics hardware. In: SIGGRAPH ’05: ACM SIGGRAPH 2005 courses. ACM Press, New York, p 268 20. Horn DR, Sugerman J, Houston M, Hanrahan P (2007) Interactive k-d tree GPU raytracing. In: I3D ’07: Proceedings of the 2007 symposium on Interactive 3D graphics and games. ACM Press, New York, pp 167–174 21. Wald I, Slusallek P (2001) State-of-the-art in interactive ray-tracing. Proc EUROGRAPHICS 2001:21–42 22. Friedrich H, Günther J, Dietrich A, Scherbaum M, Seidel HP, Slusallek P (2006) Exploring the use of ray tracing for future games. In: Sandbox ’06: Proceedings of the 2006 ACM SIGGRAPH symposium on videogames. ACM Press, New York, pp 41–50 23. Goral CM, Torrance KE, Greenberg DP, Battaile B (1984) Modeling the interaction of light between diffuse surfaces. In: SIGGRAPH ’84: Proceedings of the 11th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 213–222 24. Guitton P, Roman J, Subrenat G (1995) Implementation results and analysis of a parallel progressive radiosity. In: PRS ’95: Proceedings of the IEEE symposium on parallel rendering. ACM Press, New York, pp 31–38 25. Sillion F, Puech C (1994) Radiosity and global illumination. Morgan Kaufmann Publishers, San Francisco 26. Schöffel F (1997) Online radiosity in interactive virtual reality applications. In: VRST ’97: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 201–208

54

2

Computer Graphics

27. Carr NA, Hall JD, Hart JC (2003) GPU algorithms for radiosity and subsurface scattering. In: HWWS ’03: Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardware. Eurographics Association, Aire-la-Ville, pp 51–59 28. Jensen HW (1996) Global illumination using photon maps. In: Rendering techniques ’96: 7th Eurographics workshop on rendering, pp 21–30 29. Jensen HW (2001) Realistic image synthesis using photon mapping. AK Peters Ltd, Natick 30. Jensen HW (2004) A practical guide to global illumination using ray tracing and photon mapping. In: SIGGRAPH ’04: ACM SIGGRAPH 2004 Course notes. ACM Press, New York, p 20 31. Purcell TJ, Donner C, Cammarano M, Jensen HW, Hanrahan P (2003) Photon mapping on programmable graphics hardware. In: HWWS ’03: Proceedings of the ACM SIGGRAPH/ EUROGRAPHICS conference on Graphics hardware. Eurographics Association, Aire-laVille, pp 41–50 32. Haeberli P, Segal M (1993) Texture Mapping as a fundamental drawing primitive. In: Proceedings of the 4th Eurographics workshop on rendering, pp 259–266 33. Blinn JF, Newell ME (1976) Texture and reflection in computer generated images. Commun ACM 19(10):542–547 34. Policarpo F, Oliveira MM, Comba JLD (2005) Real-time relief mapping on arbitrary polygonal surfaces. In: I3D ’05: Proceedings of the 2005 symposium on Interactive 3D graphics and games. ACM Press, New York, pp 155–162 35. Policarpo F, Oliveira MM (2006) Relief mapping of non-height-field surface details. In: I3D ’06: Proceedings of the 2006 symposium on Interactive 3D graphics and games. ACM Press, New York, pp 55–62 36. Blinn JF (1978) Simulation of wrinkled surfaces. In: SIGGRAPH ’78: Proceedings of the 5th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 286–292 37. Cook RL (1984) Shade trees. In: SIGGRAPH ’84: Proceedings of the 11th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 223–231 38. Gumhold S, Huttner T (1999) Multiresolution rendering with displacement mapping. In: HWWS ’99: Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on graphics hardware. ACM Press, New York, pp 55–66 39. Wang L, Wang X, Tong X, Lin S, Hu S, Guo B, Shum HY (2003) View- dependent displacement mapping. In: SIGGRAPH ’03: ACM SIGGRAPH 2003 papers. ACM Press, New York, pp 334–339 40. Rost R (2006) OpenGL shading language. Addison-Wesley, Boston 41. Talton JO, Fitzpatrick D (2007) Teaching graphics with the OpenGL shading language. In: SIGCSE ’07: Proceedings of the 38th SIGCSE technical symposium on computer science education. ACM Press, New York, pp 259–263

Chapter 3

Computer Animation

This chapter introduces you to the world of computer animation, which is essential to show virtual worlds that change over time. The main motion control methods are explained, including motion capture, keyframe animation, inverse kinematics, physics-based animation, and behavioral animation.

3.1

Introduction

Most virtual worlds are dynamic; they change over time: objects move, rotate, and transform themselves; in inhabited virtual worlds, virtual characters live. Object and character motion is an essential part of these virtual worlds. The main goal of computer animation is to synthesize the desired motion effect, which is a mixing of natural phenomena, perception, and imagination. The animator designs the object’s dynamic behavior with a mental representation of causality. He or she imagines how it moves, gets out of shape, or reacts when it is pushed, pressed, pulled, or twisted. So the animation system has to provide the user with motion control tools that can translate his or her wishes from his or her own language. Computer animation methods may also help to understand physical laws by adding motion control to data to show their evolution over time. Visualization has become an important way of validating new models created by scientists. When the model evolves over time, computer simulation is generally used to obtain the evolution of time, and computer animation is a natural way of visualizing the results obtained from the simulation. Computer animation may be defined as a technique in which the illusion of movement is created by displaying on a screen, or recording on a recording device, a series of individual states of a dynamic scene. Formally, any computer animation sequence may be defined as a set of objects characterized by state variables evolving over time. For example, a human character is normally characterized using its joint angles as state variables.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_3

55

56

3.1.1

3

Computer Animation

Animation Types: Real Time and Image by Image

Two types of animation are generally distinguished: • Animation in real time • Animation image by image For virtual reality, we are only interested in real time, where the computer calculates the movements and the transformations sufficiently quickly that the user in front of his or her graphic display can see these movements and transformations. It is also what we can find in video games and flight simulators. Image-by-image computer animation corresponds to traditional animation. It is initially necessary to calculate the images, record them on videotape (or film), and then visualize them or project at a fast rate (e.g., 25 images/second for video PAL). The calculated images can take a split second to several hours. This is not what we want for virtual reality. To produce a computer animation sequence, the animator has two principal techniques available. The first is to use a model that creates the desired effect; a good example is the growth of a green plant. The second is used when no model is available. In this case, the animator produces “by hand” the real-world motion to be simulated. Until recently, most computer-generated films have been produced using the second approach: traditional computer animation techniques like key-frame animation, or spline interpolation. Then animation languages, scripted systems, and director-oriented systems were developed. In the next generation of animation systems, motion control tends to be performed automatically using artificial intelligence (AI) and robotics techniques. Motion is planned at a task level and computed using physical laws. More recently, researchers have developed models of behavioral animation and simulation of autonomous creatures.

3.1.2

Articulated Bodies and Virtual Characters

An important class of entities that are very often animated are virtual characters, and especially virtual humans. They are articulated figures modeled with multiple layers: a virtual skin is usually attached to an underlying skeleton that animates the whole body. We will discuss in more details virtual characters and virtual humans in Chapter 4. However, we will present in this chapter the concept of skeleton, as most animation methods are applied to skeletons. The skeleton is a hierarchically organized set of joints that depends on the animation requirements, for example, it might be relevant for a medical surgery tool to model each vertebra of the spine, while a simpler decomposition into four joints could be sufficient for a video game character. Of course, it is better to create morphologically correct skeletons, but this can turn out to be quite costly. Real humans have so many degrees of freedom that virtual characters frequently omit some of them.

3.1

Introduction

57 head_top r_scapula (2)

1_scapula (2) head vt6 (3)

r_shoulder (3) r_elbow (2) r_wrist (2) r_clavicle (2) r_hand_center

pelvis (3) ; v11 (1) r_hip (3)

r_knee (2)

r_ankle (1) r_subtalar (1) r_mid_foot (1) r_toe (1)

vc8 (3) vc7 (3) 1_shoulder (3) 1_elbow (2) 1_wrist (2) vt5 (2) l_clavicle (2) vt4 (3) vl3 (2) vl2 (2) l_hip (3)

1_hand_center

Joint (nb dof) Functional location (not a joint)

l_knee (2)

l_ankle (1) l_subtalar (1) l_mid_foot (1) l_toe (1)

Fig. 3.1 H-ANIM skeleton

A skeleton can be described as a hierarchically organized set of joints, with each joint having one or more rotational degrees of freedom (DOF). It also often stores a 4D transformation matrix for representing the length and orientations of the bones, together with joint limits to avoid unnatural rotations. Since matrices are always expressed locally, the resulting global posture is the composition of all previous transformation matrices in the hierarchy, starting from the root. For example, the shoulder is a joint with three DOFs, and the elbow joint is its successor in the hierarchy (see Fig. 3.1). If the matrix of the shoulder is modified, then the elbow moves, but its local matrix stays the same. In an effort to standardize on a common representation for virtual humans, the Web3D Consortium defined the Humanoid Animation Specification (H-Anim) for VRML and MPEG4. They specify a fixed topology and a naming convention for the articulated structure as well as tools for adding an outer mesh, the skin or cloth. As shown in Fig. 3.1, the skeleton consists of several segments (such as the forearm, hand, and foot), which are connected to each other by the joints (such as the elbow, wrist, and ankle). For an application to animate a virtual character, it needs to obtain access to the joints and alter the joint angles. The application may also need to retrieve information about such things as joint limits and segment masses. Animation of the skeleton is performed by modifying 4D matrices associated with the

58

3 Computer Animation

bones, either by synthesizing the motion or with the help of inverse kinematics and key-framing, as we present in the next sections.

3.2

Motion Control Methods

We start from the classification introduced by Thalmann and Thalmann [1] based on the method of controlling motion. A motion control method specifies how an object or an articulated body is animated and may be characterized according to the type of information to which it is privileged in animating the object or the character. For example, in a key-frame system for an articulated body, the privileged information to be manipulated is joint angles. In a forward dynamics-based system, the privileged information is a set of forces and torques; of course, in solving the dynamic equations, joint angles are also obtained in this system, but we consider this as derived information. In fact, any motion control method will eventually have to deal with geometric information (typically joint angles), but only geometric motion control methods are explicitly privileged to this information at the level of animation control. The nature of privileged information for the motion control of characters falls into three categories: geometric, physical, and behavioral, giving rise to three corresponding categories of motion control methods. • The first approach corresponds to methods heavily relied on by the animator: motion capture, shape transformation, parametric key-frame animation. Animated objects are locally controlled. Methods are normally driven by geometric data. Typically, the animator provides a lot of geometric data corresponding to a local definition of the motion. • The second way guarantees a realistic motion by using physical laws, especially dynamic simulation. The problem with this type of animation is controlling the motion produced by simulating the physical laws that govern motion in the real world. The animator should provide physical data corresponding to the complete definition of a motion. Motion is obtained by the dynamic equations of motion relating the forces, torques, constraints, and the mass distribution of objects. As trajectories and velocities are obtained by solving the equations, we may consider actor motions as globally controlled. Functional methods based on biomechanics are also part of this class. • The third type of animation, called behavioral animation, considers the relationship between each object and the other objects. Moreover, the control of animation may be performed at a task level, but we may also consider animated objects as autonomous creatures. In fact, we will consider as a behavioral motion control method any method that drives the behavior of objects by providing high-level directives indicating a specific behavior without any other stimulus.

3.3

Motion Capture and Performance Animation

3.3

59

Motion Capture and Performance Animation

Performance animation or motion capture consists of measurement and recording of direct actions of a real person or animal for immediate or delayed analysis and playback. The technique is used today especially in production environments for 3D character animation. It involves mapping measurements onto the motion of the digital character. This mapping can be direct: for example, human arm motion controlling a character’s arm motion, or indirect, mouse movement controlling a character’s eye and head direction. Real-time motion capture is very important in virtual reality, as it can provide the computer with information on the motion of the user: position and orientation of the limbs, postures, and gestures. Figure 3.2 shows an example of motion capture session. We may distinguish two kinds of systems, optical and nonoptical.

3.3.1

Optical Motion Capture Systems

Passive optical systems use markers coated with a retroreflective material to reflect light back that is generated near the camera’s lens. The camera’s sensitivity can be adjusted, taking advantage of most cameras’ narrow range of sensitivity to light so that only the bright markers will be sampled, ignoring skin and fabric. The centroid

Fig. 3.2 Dongseo’s motion capture green room. (Licensed under the Creative Commons Attribution 2.0 Generic license)

60

3 Calibration motion

Computer Animation

Centers of rotation and scaled skeleton

Marker model

Fig. 3.3 Principles of optical motion capture

of the marker is estimated as a position within the 2D image that is captured. The gray-scale value of each pixel can be used to provide subpixel accuracy. Markers are attached to an actor’s body and to several cameras focused on performance space. By tracking the positions of markers, one can get locations corresponding to key points in the animated model; for example, we attach markers at joints of a person and record the position of markers from several different directions. Figure 3.3 shows the principles. We then reconstruct the 3D position of each key point at each time. The main advantage of this method is freedom of movement; it does not require any cabling. There is, however, one main problem: occlusion; that is, the lack of data resulting from hidden markers, for example, when the performer lies on his back. Another problem comes with the lack of an automatic way of distinguishing reflectors when they get very close to each other during motion. These problems may be minimized by adding more cameras, but at a higher cost, of course. Most optical systems operate with four or six cameras, but for high-end applications like movies, 16 cameras is usual. A good example of an optical system is the VICONTM system. “Active optical systems” triangulate positions by illuminating one LED at a time very quickly or multiple LEDs at once, but sophisticated software identifies them by their relative positions, somewhat akin to celestial navigation. Rather than reflecting light back that is generated externally, the markers themselves are powered to emit their own light. The use of conventional video cameras is still a dream, as it is very difficult to automatically detect the joints in three dimensions and to make a correct

3.3

Motion Capture and Performance Animation

61

correspondence between 3D points on images captured by several cameras. This is an active area of research in computer vision.

3.3.2

Nonoptical Motion Capture Systems

Inertial Systems Motion capture using inertial sensors is a technology that incorporates miniature sensors, biomechanical models, and sensor fusion algorithms. Inertial sensors transmit motion data wirelessly to a computer, where it is recorded or viewed. A combination of a gyroscope, magnetometer, and accelerometer is used in many inertial systems to measure rotational rates. Rotations are translated into skeletons in the software. These systems can capture the full six degrees of freedom of a human body in real time and can also provide limited direction information, though they have a much lower resolution and are susceptible to electromagnetic noise. Inertial systems have several advantages, including: the ability to capture in a variety of environments, no need for solution, portability, and large capture areas. The disadvantages include lower positioning accuracy and positional drift over time.

Exoskeleton Motion Capture Systems Exoskeleton motion capture systems directly measure joint angles. It consists of an articulated mechanical structure attached to a performer's body, and as the performer moves, so do the mechanical parts, thereby measuring the performer's relative motion. Exoskeleton motion capture systems are real time, relatively low cost, free from occlusion, wireless, and offer unlimited capture volume. In general, they are rigid structures consisting of jointed, straight metal or plastic rods connected via potentiometers that articulate at joints throughout the body. There are some suits that provide limited force feedback or haptic input.

Magnetic Position/Orientation Trackers and Systems Position and orientation are determined by the magnetic flux relationship between three orthogonal coils at both the transmitter and receiver. By mapping the tracking volume meticulously, these systems can calculate both range and orientation based on the relative intensity of the voltage or current of three coils. Six degrees of freedom are available from the sensor, which enables useful results to be obtained using only two thirds of the markers needed in optical systems: one on the upper arm and one on the lower arm for elbow angle and position. The markers are not occluded by nonmetallic objects but are susceptible to magnetic and electrical interference from metal objects in the environment, like

62

3

Computer Animation

Fig. 3.4 Motion capture using Ascension magnetic sensors

rebar (steel reinforcing bars in concrete) or wiring, which affects the magnetic field, and electrical sources such as monitors, lights, cables, and computers. The sensor response is nonlinear, especially toward the edges of the capture area. With the magnetic systems, there is a distinction between AC and DC systems: one uses square pulses, the other uses sine wave pulses. Magnetic motion capture systems require the real actor to wear a set of magnetic sensors (see Fig. 3.4), as defined earlier, which can measure their spatial relationship to a centrally located magnetic transmitter. The position and orientation of each sensor are then used to drive an animated character. One problem is the need to synchronize receivers. For complete human body motion, 11 sensors are generally needed: one on the head, one on each upper arm, one on each hand, one in the center of the chest, one on the lower back, one on each ankle, and one on each foot. The most common way to calculate the rest of the necessary information is to use inverse kinematics (see Sect. 3.5). The capture volumes for magnetic systems are dramatically smaller than for optical systems. These magnetic sensors can be used in virtual reality to track the position and orientation of the head in head-mounted displays (see Sect. 8.2.1), the position and orientation of the listener in audio space systems (see Sect. 9.3), or the position and orientation of the hand in data gloves (see Sect. 10.2). They were also the basis for two well-known historic kinds of magnetic motion capture systems manufactured by Polhemus and Ascension. Polhemus has been delivering award-winning motion trackers for over 50 years, and they offer today wireless tracking products that provide full freedom of movement, delivered with their proprietary 6DOF electromagnetic technology. FASTRAK set the gold standard in motion tracking years ago and remains a top seller. This award-winning 6DOF product is a highly accurate, low-latency 3D

3.3

Motion Capture and Performance Animation

63

motion tracking and digitizing system. But, today, G4 is a wireless motion tracking system that delivers full 6DOF tracking, providing both position and orientation without hybrid technologies. Ascension Technologies manufactured several different types of trackers including the MotionStar Turnkey, a motion capture tracker for character animation. It captured the motions of up to 120 receivers simultaneously over a long range without metallic distortion. Ascension Technology Corporation was acquired by Roper Technologies Inc. in 2012 and became an affiliate company of NDI also acquired by Roper Technologies in 2011. Today, NDI’s Aurora® and 3D Guidance® electromagnetic (EM) tracking solutions generate a defined EM field in which EM micro-sensors are tracked without requiring a line of sight. Aurora® electromagnetic 5DOF and 6DOF sensors can be embedded into OEM medical instruments such as catheters, endoscopes, and ultrasound probes. The Aurora and 3D Guidance solutions share a common foundation of advanced product engineering, expert integration, and superior tracking performance.

3.3.3

Motion Capture Advantages and Disadvantages

The motion capture approach offers the following advantages: • It works in real time. • The amount of work does not vary with the complexity or length of the performance. • Complex movement and realistic physical interactions such as secondary animation, weight, and exchange of forces can be more easily recreated in a physically accurate manner. • Motion capture allows one actor to play multiple roles within a single film. Unfortunately, the motion capture approach has the following drawbacks: • Specific hardware and special programs are required to obtain and process the data. • The cost of software and equipment and personnel required can be prohibitive. • Motion capture systems may have specific requirements for the space in which they are operated. • Applying motion capture to quadruped characters can be difficult. • Technology can become obsolete every few years as better software and techniques are invented. • Results are limited to what can be performed within the capture volume without extra editing of the data. • Movement that does not follow the laws of physics generally cannot be represented. • If the computer model has different proportions from the capture subject, artifacts may occur. For example, if a cartoon character has large, oversized hands, these may intersect strangely with any other body part when the human actor brings

64

3

Computer Animation

them too close to his body. In the same way, feet may enter the floor if characters’ legs are longer than the human actors’ legs. • Motion capture does not bring any new concept to animation methodology. For any new motion, it is necessary to record the reality again. Moreover, motion capture is not appropriate, especially in real-time simulation activities, where the situation and actions of people cannot be predicted ahead of time, and in dangerous situations, where one cannot involve a human actor.

3.4 3.4.1

Key-Frame Animation Shape Interpolation and Parametric Keyframe Animation

Key-frame animation consists of the automatic generation of intermediate frames, called in-betweens, based on a set of key-frames supplied by the animator. There are two fundamental approaches to key-frame: 1. The in-betweens are obtained by shape interpolation. This technique was introduced by Burtnyk and Wein [2]. It plays a major role in film production and has been improved continuously by using more and more sophisticated mathematical tools [3, 4]. This method makes it possible to transform one geometrical form into another during an animation. Fig. 3.5 shows the principle. There is a serious problem for the image-based key-frame: the motion can be distorted due to interpolation. The problem becomes complicated when the two drawings do not have the same number of vertices. In this case, it is initially necessary to carry out a preprocessing step to equalize the number of vertices in the two drawings.

Fig. 3.5 Interpolation between two drawings: (top) an interpolation with 50%; (bottom) interpolations with 25%, 50%, and 100%

3.4

Key-Frame Animation

65

2. A way of producing better images is to interpolate parameters of the model instead of the object itself. This technique is called parametric key-frame animation by Parke [5] and Steketee and Badler [6] and key transformation animation by Zeltzer [7]. The parameters are normally spatial parameters, physical parameters and visualization parameters that decide the models’ behavior. Parametric key-frame animation is considered a direct kinematics method in motion control when the interpolated parameters are defined in the joint space of the articulated figure. Efficient and numerically well-behaving methods exist for the transformation of position and velocity from joint space to Cartesian space. In both key-frame methods, linear interpolation produces undesirable effects, such as lack of smoothness in motion, discontinuities in the speed of motion, and distortions in rotations. For these reasons, spline interpolation methods are used. Splines can be described mathematically as piecewise approximations of cubic polynomial functions. Two kinds of splines are very popular: interpolating splines with C1 continuity at knots, and approximating splines with C2 continuity at knots. For animation, the most interesting splines are the interpolating splines: cardinal splines, Catmull-Rom splines, and Kochanek-Bartels [8] splines (see Sect. 3.4.2).

3.4.2

Kochanek-Bartels Spline Interpolation

This method consists of interpolating splines with three parameters for local control: tension, continuity, and bias. Consider a list of points Pi and the parameter t along the spline to be determined. A point V is obtained from each value of t from only the two nearest given points along the curve (one behind Pi, one in front of Pi+1). But the tangent vectors Di and Di+1 at these two points are also necessary. This means that we have: V = THCT where T is the matrix [t3t2t 1], H is the Hermite matrix, and C is the matrix [Pi Pi+1 Di Di+1]. The Hermite matrix is given by:

H=

2 -3

-2 3

1 -2

1 -1

0 1

0 0

1 0

0 0

This equation shows that the tangent vector is the average of the source chord Pi - Pi-1 and the destination chord Pi+1 - Pi. Similarly, the source derivative (tangent vector) DSi and the destination derivative (tangent vector) DDi may be considered at any point Pi.

66

3

Computer Animation

Fig. 3.6 Influence of tension, continuity, and bias on Kochanek-Bartels spline. (a) Increase of the tension. (b) Increase of the discontinuity. (c) A biased interpolation at K2

Using these derivatives, Kochanek and Bartels propose the use of three parameters to control the splines: tension, continuity, and bias. The tension parameter t controls how sharply the curve bends at a point Pi. As shown in Fig. 3.6a, in certain cases a wider, more exaggerated curve may be desired, while in other cases, the desired path may be much tighter. The continuity c of the spline at a point Pi is controlled by the parameter c. Continuity in the direction and speed of motion is not always desirable. Animating a bouncing ball, for example, requires the introduction of a discontinuity in the motion of the point of impact, as shown in Fig. 3.6b. The direction of the path as it passes through a point Pi is controlled by the bias parameter b. This feature allows the animator to have a trajectory anticipate or overshoot a key position by a certain amount, as shown in Fig. 3.5c. Equations combining the three parameters may be obtained: DSi = 0:5½ð1 - t Þð1 þ cÞð1 - bÞðPiþ1 - Pi Þ þ ð1 - t Þð1 - cÞð1 þ bÞðPi - Pi - 1 Þ DDi = 0:5½ð1 - t Þð1 - cÞð1 - bÞðPiþ1 - Pi Þ þ ð1 - t Þð1 þ cÞð1 þ bÞðPi - Pi - 1 Þ A spline is then generated using V =THCT with DDi and DSi instead of D-i and Di+1.

3.5

3.5

Inverse Kinematics

67

Inverse Kinematics

Formally, the direct kinematics problem consists in finding the position of endpoint positions (e.g., hand, foot) with respect to a fixed-reference coordinate system as a function of time without regard to the forces or moments that cause the motion. Efficient and numerically well-behaved methods exist for the transformation of position and velocity from joint space (joint angles) to Cartesian coordinates (end of the limb). The inverse kinematics problem is the opposite of the direct kinematics problem (see Fig. 3.7). This is the determination of the joint variables given the position and the orientation of the end of the manipulator, or end effector, with respect to the reference coordinate system. It can be solved by various methods, such as inverse transform, screw algebra, dual matrices, dual quaternian, iterative, and geometric approaches. More details of each method can be found in [9]. Inversion is possible if the dimensions of joint space and Cartesian space are the same. However, a general articulated structure may contain more DOFs in joint space, which are highly redundant in accomplishing tasks. Inversion is not always possible. The solution is the first-order approximation of the system: to linearize the direct geometric model. Because of the linearization, the solution’s validity of inverse kinematics is limited to the neighborhood of the current state and, as such, any desired motion has to comply with the hypothesis of small movements.

A α

Direct kinematics

Inverse kinematics β

γ

Joint angles

Fig. 3.7 Direct and inverse kinematics

B

Limb position/orientation

68

3

Computer Animation

The position and orientation vector of the end effector in Cartesian space are called the main task (or behavior). If its dimension m (usually six: three rotations and three translations) is less than the dimension n of the joint space, the (n - m) vectors in joint space are projected to the null vector in Cartesian space by the linear transformation J. They do not modify the achievement of the main task. The (n m) vector in Cartesian space is called the secondary task (or behavior). The discrete form of the general solution provided by inverse kinematics is: Δq = J þ Δx þ ðI - JJ þ ÞΔz Where Δq is the unknown vector in the joint variation space of dimension n; Δx describes the main task as a variation of the end effector position and orientation in Cartesian space; the dimension m of the main task is usually less than or equal to the dimension n of the joint space; J is the Jacobian matrix of the linear transformation, representing the differential behavior of the controlled system over the dimensions specified by the main task; J+ is the unique pseudo-inverse of J providing the minimum norm solution, which realizes the main task; I is the identity matrix of the joint variation space (nn)(I - J+J ), is a projection operator on the null space of the linear transformation J; any element belonging to this joint variation subspace is mapped by J into the null vector in the Cartesian variation space; Δz describes a secondary task in the joint variation space. This task is partially realized via the projection on the null space. In other words, the second part of the equation does not modify the achievement of the main task for any value of Δz. Usually Δz is calculated to minimize a cost function. If the main task belongs to the image space of J, then the null space is (n - m) dimensional in the joint variation space. This information is fundamental to evaluate the potentiality of the secondary task and clearly leads to a tradeoff between realization of main and secondary tasks. We can summarize the steps using inverse kinematics as shown in Algorithm 3.1: Algorithm 3.1: Inverse Kinematics Steps 1: Define the main task. 2: Evaluate the potential of the secondary ask 3: Construct the Jacobian matrix. 4: Invert the Jacobian matrix. 5: Handle the joint limits. 6: Manage singularities. The so-called Jacobian of the system is the matrix gathering the first order variations. It is inverted to obtain the joint variation realizing a desired variation of the end effector. A more detailed explanation of these steps is given in [10]. Practically, the problem is to determine a joint configuration for which a desired task, usually expressed in Cartesian space, is achieved. For example, the shoulder, elbow, and wrist configurations must be determined so that the hand reaches a position precisely in space. The equations that arise from this problem are generally nonlinear and are difficult to solve in general. In addition, a resolution technique must deal with the difficulties described later. For the positioning and animation of articulated

3.6

Motion Retargeting

69

figures, the weighting strategy is the most frequent: some typical examples are given for posture manipulation [11, 12] and to achieve smooth solution blending [13, 14].

3.6

Motion Retargeting

As we saw in Sect. 3.2, there is a lot of interest in recording motion using motion capture systems (magnetic or optical), then trying to edit such a motion to create this individuality. This process is tedious, and there is no reliable method at this stage. Even if it is easy to correct one posture by modifying its angular parameters (with an inverse kinematics engine, for instance), it becomes a difficult task to perform this over the whole motion sequence while ensuring that some spatial constraints are respected over a certain time range and that no discontinuities arise. When one tries to adapt a captured motion to a different character, the constraints are usually violated, leading to problems such as the feet going into the ground or a hand unable to reach an object that the character should grab. The problem of adaptation and adjustment is usually referred to as the motion retargeting problem. Witkin and Popovic [15] proposed a technique for editing motions by modifying the motion curves through warping functions and produced some of the first interesting results. In another paper [16], they extended their method to handle physical elements, such as mass and gravity, and described how to use characters with different numbers of degrees of freedom. Their algorithm is based on reducing the character to an abstract character that is much simpler and only contains the degrees of freedom that are useful for a particular animation. The edition and modification are then computed on this simplified character and mapped again onto the end-user skeleton. Bruderlin and Williams [17] have described some basic facilities to change the animation by modifying the motion parameter curves. The user can define a particular posture at time t, and the system is then responsible for smoothly blending the motion around t. They also introduced the notion of a motion displacement map, which is an offset added to each motion curve. The motion retargeting problem term was brought up by Michael Gleicher [18]. He designed a space-time constraint solver, to which every constraint is added, leading to a big optimization problem. He mainly focused on optimizing his solver to avoid enormous computation time, and he achieved very good results. Given a captured motion associated to its performer skeleton, Monzani et al. [19] decompose the problem of retargeting the motion to the end-user skeleton into two steps: 1. Compute the intermediate skeleton matrices by orienting the intermediate skeleton bones to reflect the performer skeleton posture (motion converter). 2. Set the end-user skeleton matrices to the local values of the corresponding intermediate skeleton matrices. The first task is to convert the motion from one hierarchy to a different one. An intermediate skeleton model is introduced to solve this, implying three more subtasks: manually set at the beginning the correspondences between the two

70

3

Motion Capture

Computer Animation

Virtual creature design

Anatomic Binding (skeleton root)

End User Skeleton

Performer Skeleton

Motion Composer (GUI)

Motion Converter

Intermediate Skeleton Fig. 3.8 Use of an intermediate skeleton for motion retargeting

hierarchies, create the intermediate skeleton, and convert the movement. The resulting motion is then corrected, and Cartesian constraints are enforced using inverse kinematics. When considering motion conversion between different skeletons, one quickly notices that it is very difficult to directly map the performer skeleton values onto the end-user skeleton, due to their different proportions, hierarchies, and axis systems. This raised the idea of having an intermediate skeleton: depending on the performer’s skeleton posture, its bones are reoriented to match the same directions. It is then easy to map the intermediate skeleton values onto the end-user skeleton (see Fig. 3.8). Bindiganavale and Badler [20] also addressed the motion retargeting problem, introducing new elements: using the zero-crossing of the second derivative to detect significant changes in the motion, visual attention tracking (and the way to handle the gaze direction), and applying inverse kinematics to enforce constraints, by defining six subchains (the two arms and legs, the spine, and the neck). Finally, Lee and Shin [21] used in their system a coarse-to-fine hierarchy of B-splines to interpolate the solutions computed by their inverse kinematics solver. They also

3.7

Procedural Animation

71

reduced the complexity of the IK problem by analytically handling the degrees of freedom for the four human limbs. Lim and Thalmann [22] have addressed the issue of solving customers’ problems when applying evolutionary computation. Rather than the seemingly more impressive approach of wow-it-all-evolved-from-nothing, tinkering with existing models can be a more pragmatic approach. Using interactive evolution, they experimentally validate this point on setting parameters of a human walk model for computer animation while previous applications are mostly about evolving motion controllers of far simpler creatures from scratch. Glardon et al. [23] have proposed a novel approach to generate new generic human walking patterns using motion-captured data, leading to a real-time engine intended for virtual human animation. The method based on PCA (principal component analysis) is described in Sect. 4.3.2.

3.7

Procedural Animation

Procedural animation corresponds to the creation of a motion by a procedure describing specifically the motion. Procedural animation should be used when the motion can be described by an algorithm or a formula. For example, consider the case of a clock based on the pendulum law: α = A sinðωt þ φÞ A typical animation sequence may be produced using a program, as shown in Algorithm 3.2: Algorithm 3.2: Procedural Animation create CLOCK(...) for FRAME = 1 to NBFRAMES do TIME = TIME + 1/25 ANGLE = A  SIN(OMEGA  TIME + PHI) modify(CLOCK, ANGLE) draw CLOCK record CLOCK erase CLOCK end for

The generation of motion using a procedure is not really a method; it is more a framework. There are so many possibilities to express motion using procedures or laws that the possible algorithms are unlimited. Two important cases are when the laws are physical laws and when they are behavioral; these two cases will be presented in the next two sections.

72

3.8

3

Computer Animation

Physics-Based Animation

A great deal of work exists on the dynamics of articulated bodies [24], and efficient direct dynamics algorithms have been developed in robotics for structures with many degrees of freedom [25]. In computer animation, these algorithms have been applied to the dynamic simulation of the human body [26]. Given a set of external forces (like gravity or wind) and internal forces (due to muscles) or joint torques, these algorithms compute the motion of the articulated body according to the laws of rigid body dynamics. Impressive animations of passive structures like falling bodies on stairs can be generated in this way with little input from the animator. Figure 3.9 shows an example of animation generated using dynamics. In fact, an important issue that arises in this direct or forward dynamics is how to control the model. Mathematically, forward dynamics translates into differential equations, which are typically posed as an initial-value problem; the user has little control other than setting up the initial configuration. This is the opposite to key-frames, where the animator has full control. Control of physics-based models remains an open research issue.

Fig. 3.9 Dynamics-based motion [1]

3.9

Behavioral Animation

73

Fig. 3.10 Direct and inverse dynamics

More generally, it is unpractical for an animator to specify a temporally coordinate sequence of force/torque activation to generate a desired active behavior. This is the inverse dynamics problem, which has been addressed for walking in [27]. Figure 3.10 shows the relation between forward and inverse dynamics. Impressive results have also been achieved by Hodgins et al. [28] with the simulation of dynamic human activities such as running and jumping. Some controller architectures have also been proposed by Lazlo et al. [29] to generate a wide range of gaits and to combine elementary behaviors [30]. The use of constraints to avoid the direct specification of torques has also been researched; for example, in the computer graphics community, the satisfaction of space-time constraints has been proposed by Witkin and Kass [31] with the minimization of an objective function, such as the total energy expenditure.

3.9

Behavioral Animation

A common way of describing behavior is by scripting, where the interaction must be described step by step. This approach provides full control and flexibility; however, if used alone, it does require programming of every possible outcome of an interaction, which may not be practical for larger systems. An interesting solution to this problem is to use scripting in a complementary role to other methods of specifying behavior. For example, the lower-level details (usually related to animation since behavior and animation cannot be fully separated) can be scripted, whereas a higher-level module can carry out reasoning. Specially designed or general-purpose interpreted programming languages can be used. If a general-purpose programming language is used, some extensions may need to be implemented to simplify the process of writing scripts for nonprogrammers (e.g., designers). Behaviors can also be specified based on rules. Rule-based systems are rather simplistic, and rules essentially correspond to if-then statements. For example, the Improv system [32] combines scripting and rule-based behavior, trying to tackle the problems of animation and behavior simultaneously. It consists of an animation

74

3 Computer Animation

engine, used for the motion generation aspects, and a behavior engine, used for describing the decision-making process through rules. State machines are also widely used for specifying behaviors as they have useful graphical representations. In fact, in most cases, even if scripting is used, there will be a state machine implemented. The disadvantage is that implementing complex systems with state machines requires a lot of states and transitions, so a state machine can easily become difficult to maintain. Vosinakis and Panayiotopoulos have introduced the task definition language [33]. This work aims at filling the gap between higher-level decision processes and agent interaction with the environment. This language supports complex highlevel task descriptions through a combination of parallel, sequential, or conditionally executed built-in functions. The main advantage of this language is that it enables tasks to be reused by different agents in different environments. Reynolds [34] introduced a distributed behavioral model simulating flocks of birds, herds of land animals and fish schools. For birds, the simulated flock is an elaboration of a particle system with the simulated birds being the particles. A flock is assumed to be the result of the interaction between the behaviors of individual birds. Working independently, the birds try to both stick together and avoid collisions with one another and with other objects in their environment. In a module of behavioral animation, positions, velocities, and orientations of the actors are known from the system at any time. The animator may control several global parameters, such as weight of the obstacle-avoidance component, weight of the convergence to the goal, weight of the centering of the group, maximum velocity, maximum acceleration, and minimum distance between actors. The animator provides data about the leader trajectory and the behavior of other birds relative to the leader. Wilhelms and Van Gelder [35] propose a system based on a network of sensors and effectors. Ridsdale [36] proposes a method that guides lower-level motor skills from a connectionist model of skill memory, implemented as collections of trained neural networks. Another approach for behavioral animation is based on timed and parameterized L-systems [37] with conditional and pseudo-stochastic productions. With this production-based approach a user may create any realistic or abstract shape, play with fascinating tree structures, and generate any concept of growth and life development in the resulting animation. Figure 3.11 shows an example. More advanced research in behavioral animation can be found in autonomous characters, which is detailed further in Sect. 4.6, as well as crowd simulation, detailed in Sect. 4.7.

References

75

Fig. 3.11 L-system-based environment

References 1. Thalmann NM, Thalmann D (1991) Complex models for animating synthetic actors. IEEE Comput Graph Appl 11(5):32–44 2. Burtnyk N, Wein M (1980) Computer-generated key-frame animation. J Soc Motion Telev Eng:149–153 3. Catmull E (1978) The problems of computer-assisted animation. In: SIGGRAPH ’78: Proceedings of the SIGGRAPH ’78 conference, pp 348–353 4. Reeves W (1981) Inbetweening for computer animation utilizing moving point constraints. In: SIGGRAPH ’81: Proceedings of the SIGGRAPH ’81 conference, pp 263–269 5. Parke F (1982) Parameterized models for facial animation. IEEE Comput Graph Appl 2(9): 61–68 6. Steketee S, Badler N (1985) Parametric keyframe interpolation in incorporating kinetic adjustment and phrasing control. In: SIGGRAPH ’85: Proceedings of the SIGGRAPH ’85 conference, pp 255–262 7. Zeltzer D (1982) Representation of complex animated figures. In: Proceedings of the graphics Interface82 conference, pp 205–221 8. Kochanek D, Bartels R (1984) Interpolating splines with local tension, continuity, and bias control. In: SIGGRAPH ’84: Proceedings of the SIGGRAPH ’84 conference, pp 33–41 9. Fu K, Gonzalez CS, Lee G (1987) Robotics, control, sensing, vision, and intelligence. McGrawHill, New York, pp 52–76 10. Boulic R, Mas R (1996) Interactive computer animation. Prentice Hall, London. Chap. Hierarchical Kinematic Behaviors for Complex Articulated Figure 11. Badler N, Manoochehri K, Walters G (1987) Articulated figure positioning by multiple constraints. IEEE Comput Graph Appl 7(6):28–38 12. Zhao J, Badler N (1994) Inverse kinematics positioning using nonlinear programming for highly articulated figures. ACM Trans Graph 13(4):313–336

76

3

Computer Animation

13. Philips C, Zhao J, Badler N (1991) Interactive real-time articulated figure manipulation using multiple kinematic constraints. In: SIGGRAPH ’91: Proceedings of the SIGGRAPH ’91 conference, pp 245–250 14. Phillips C, Badler N (1991) Interactive behaviors for bipedal articulated figures. In: SIGGRAPH ’91: Proceedings of the SIGGRAPH ’91 conference, pp 359–362 15. Witkin A, Popovic Z (1995) Motion warping. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 105–108 16. Popovic Z, Witkin A (1999) Physically based motion transformation. In: SIGGRAPH ’99: Proceedings of the SIGGRAPH ’99 conference, pp 11–20 17. Bruderlin A, Williams L (1995) Motion signal processing. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 97–104 18. Gleicher M (1998) Retargeting motion to new characters. In: SIGGRAPH ’98: Proceedings of the SIGGRAPH ’98 conference, pp 33–42 19. Monzani JS, Baerlocher P, Boulic R, Thalmann D (2000) Using an intermediate skeleton and inverse kinematics for motion retargeting. In: EUROGRAPHICS’00: Proceedings of the EUROGRAPHICS’00 conference, pp 11–19 20. Bindiganavale R, Badler NI (1998) Modeling and motion capture techniques for virtual environments, Springer: Lecture notes in artificial intelligence, pp 70–82. Chap. Motion abstraction and mapping with spatial constraints 21. Lee J, Shin SY (1999) A hierarchical approach to interactive motion editing for human-like figures. In: SIGGRAPH ’99: Proceedings of the 26th annual conference on computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Company, New York, pp 39–48 22. Lim IS, Thalmann D (2000) Solve customers’ problems: interactive evolution for tinkering with computer animation. In: SAC2000: Proceedings of the ACM symposium on applied computing, pp 404–407 23. Glardon P, Boulic R, Thalmann D (2004) PCA-based walking engine using motion capture data. In: CGI04: proceedings of computer graphics international 2004, pp 292–298 24. Huston R (1990) Multibody dynamics. Butterworth-Heinemann, Stoneham 25. Featherstone R (1986) Robot dynamics algorithms. Kluwer Academic Publishers, Norwell 26. Kenna MM, Zeltzer D (1996) Dynamic simulation of a complex human figure model with low level behavior control. Presence 5(4):431–456 27. Ko H, Badler N (1996) Animating human locomotion with inverse dynamics. IEEE Comput Graph Appl 16:50–59 28. Hodgins J, Wooten W, Brogan D, Brien JO (1995) Animating human athletics. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 71–78 29. Lazlo J, Panne MVD, Fiume E (1996) Limit cycle control and its application to the animation of balancing and walking. In: SIGGRAPH ’96: Proceedings of the SIGGRAPH ’96 conference, pp 155–162 30. Faloutsos P, van de Panne M, Terzopoulos D (2001) Composable controllers for physics-based character animation. In: SIGGRAPH ’01: Proceedings of the SIGGRAPH ’01 conference, pp 251–260 31. Witkin A, Kass M (1988) Spacetime constraints. In: SIGGRAPH ’88: Proceedings of the SIGGRAPH ’88 conference, pp 159–168 32. Perlin K, Goldberg A (1996) Improv: a system for scripting interactive actors in virtual worlds. In: SIGGRAPH ’96: Proceedings of the SIGGRAPH ’96 conference, pp 205–216 33. Vosinakis S, Panayiotopoulos T (2003) A task definition language for virtual agents. WSCG 2003 J WSCG 11:512–519 34. Reynolds C (1987) Flocks and herds and schools: a distributed behavioral model. In: SIGGRAPH ’87: Proceedings of the SIGGRAPH ’87 conference, pp 25–34 35. Wilhelms J, Van Gelder A (1997) Anatomically based modeling. In: SIGGRAPH ’97: Proceedings of the SIGGRAPH ’97 conference, pp 173–180

References

77

36. Ridsdale G (1990) Connectionist modelling of skill dynamics. J Visualiz Comput Anim 1(2): 66–72 37. Noser H, Turner R, Thalmann D (1992) Interaction between L-systems and vector forcefield. In: CGI92: Proceedings of the computer graphics international ’92 conference, pp 747–761

Part II

Virtual Worlds

Chapter 4

Virtual Characters

Most virtual worlds are inhabited by virtual characters; this chapter explains how to populate virtual worlds with virtual characters, especially virtual humans. Character skinning is an important topic, because a representation of a character needs a skin, and the skin should be deformable during the motion. Two essential tasks will be presented in this chapter: the locomotion and the interaction between a virtual human and an object. Facial animation is also an essential topic, especially for the interaction between the virtual humans and the user, a key aspect in virtual reality. Most virtual humans in virtual reality need to be autonomous; we emphasize this aspect in the chapter. Finally, we present an important section on crowd simulation.

4.1

Virtual Humans in Virtual Environments

As we already explained, virtual reality (VR) refers to a technology that can shift a subject into a different environment without physically moving him or her. Immersion is a key issue in VR systems, as it is central to the paradigm in which the user becomes part of the simulated world, rather than the simulated world being a feature of the user’s own world. Presence is the psychological sense of “being there” in the environment based on the technologically founded immersive base. However, any given immersive system does not necessarily always lead to presence for all people. The problem is that immersion is too often considered a hardware problem. The use of an HMD or a CAVE is supposed to provide immersion and increase the sense of presence. But how can we believe we are in a world where there is no inhabitant even when immersed? We may think about virtual worlds inhabited by a virtual character society, where virtual humans will cooperate, negotiate, make friends, communicate, and group and ungroup depending on their likes, moods, emotions, goals, fears, and so on. We may also think about a virtual world inhabited by animals or fictitious creatures. This chapter is dedicated to virtual characters and their role in virtual worlds. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_4

81

82

4

Virtual Characters

In the case of virtual humans, one important point in the simulation is the believability of the individual virtual humans; they should behave like real humans, including having capabilities such as perception, language understanding, and generation, emotions, goal-driven behavior, reactivity to the environment including with other virtual humans, memory, inference, appearance of thought and personalities, interpersonal interactions, and social skills. To be believable an actor must be affected by what takes place around it and needs to engage in social behaviors with other actors and the user. Therefore, the behavioral model of an actor needs to be versatile enough to allow a change of behavior, the emotional state of the actor must be reflected and must affect its behavior, and the interpersonal relationships with the other actors must be considered and possibly bring the actor to engage in social interactions. These actors should be able to act freely and emotionally. Ideally, they should be conscious and unpredictable. But we are far from such an ideal situation. In this chapter, we will discuss the representation of the skin of the virtual character in Sect. 4.2. Concerning the motion control of the skeleton, we will not explain again the general motion control methods described in Chap. 3, but we will present two actions which are specific to virtual humans: locomotion (Sect. 4.3) and object manipulation (Sect. 4.4). In Sect. 4.5, we survey methods to animate the characters’ face. The concept of autonomous virtual character is the key concept of Sect. 4.6, while crowd simulation is presented in Sect. 4.7.

4.2

Character Skinning

Many different approaches have been proposed to connect a deformable skin to its underlying skeleton. They can be roughly subdivided into three main categories: skeleton-based deformations, data-driven methods, and physics-based approaches. However, we will emphasize the two first categories, because they are more appropriate to real-time deformations needed for VR.

4.2.1

Skeleton-Based Deformations

The skeleton-driven deformation, a classical method for basic skin deformation is probably the most widely used technique in 3D character animation. In research literature, an early version was introduced in 1987 by Magnenat-Thalmann and Thalmann [1], who introduced the concept of joint-dependent local deformation (JDL) with operators to smoothly deform the skin surface (see Fig. 4.1); the method was also extended to handle hand deformations [2]. This technique has been given various names, such as subspace deformation (SSD), linear blend skinning, or smooth skinning. This method works first by assigning a set of joints 1. . ., n with weights to each vertex v of the character. The deformation of a vertex v0 is then

4.2

Character Skinning

83

Fig. 4.1 JLD deformations: early version of skeletonbased deformation from the movie Rendez-vous à Montréal, by Nadia Magnenat-Thalmann and Daniel Thalmann [1]

computed by a weighted combination of the transformation Tj of the influencing joint j with the following function: n

Sð v 0 Þ =

wj T j

v0

ð4:1Þ

j=0

The skeletal deformation makes use of an initial neutral pose. While this method provides fast results and is compact in memory, its drawbacks are the undesirable deformation artifacts in case of important variation of joint angles among the influencing joints. The method has been extended [3] to address a crucial issue related to linear blending skinning: setting the appropriate vertex weight values. An interesting extension of SSD-based approaches is the multi weighting approach [4], which assigns each vertex one weight for each coefficient of each influencing joints transformation matrix instead of a single weight per influencing joint. It therefore defines a new deformation function with a high-dimensional input space: 12 weights/vertex/bone, to provide the resulting skinning approach with an increased flexibility to control and transform each skin vertex. As it would be almost impossible to define the multiweight values interactively, an automated process automatically estimates the set of values. The multi weights can be derived from a set of example poses. This initial set of poses can be produced from any variety of sources. Each bone must also be assigned an influence map, which defines its influence over the surface vertices. The influence maps are user-defined. The user must also provide a scaling parameter to control the smoothness of the deformation solution and avoid overfitting, when large weights deform skin badly on a new pose. The training poses are then used as examples to estimate the multiweight vectors using a statistical analysis (principal component analysis) and a modified least square optimization method. The weights computed from the set of training poses generalize to other postures.

84

4

Virtual Characters

Fig. 4.2 Metaball-based deformations

More complex models include intermediate layers to improve and refine the control of the skin. For example, an intermediate muscle layer is introduced between the flexible skin and the rigid skeleton [5]. This intermediate structure is made of free-form deformation (FFD) control boxes attached to the skeleton. The control box shapes are parameterized with the joint angles of the skeleton. Two main deformation operators are applied to the control boxes, the first controls the bending of the mesh around the joint, and the second mimics the muscle behavior by inflating the skin along the bones to replicate the muscle contraction effect. Another approach [6] combines implicit surfaces and B-spline patches. In this method, the implicit surfaces are ellipsoidal metaballs that represent the gross shape of bone, muscle, and fat tissue. The motion/deformation of each primitive with respect to underlying joints is specified via a graphical interface. A skin of fixed topology is extracted by casting rays from the skeleton segments in a star-shaped manner and using the intersection points as control points of B-spline patches. Figure 4.2 shows an example. Character skinning has also been investigated to simulate anatomy and reproduce as closely as possible the anatomical behavior of characters [7, 8]. Anatomy-based modeling considers that using flexible surfaces at joints and along the limbs is an oversimplification and thus adopts a similar approach as artistic anatomy, assuming that the skin surface changes may be more accurately represented as the result of the behavior of the underlying anatomical structures, such as muscles, fat, and connective tissues. Complex volumes are used to approximate the real shape of muscles with tendon attachments to the bones.

4.2

Character Skinning

85

Various approaches have been investigated: • Passive muscles, where geometric or physics-based deformations are used to adjust the shape of the muscle to the current posture of the skeleton • Active muscles, where biomechanical physics-based deformation models are used to simulate the dynamic behavior of muscles Muscle elongation and contraction (see Eq. 4.2) are used to activate the skeleton joints and produce motions [4]: r = ð1 - t Þr n þ ktr n = ð1 - t þ kt Þr n

ð4:2Þ

where k is a tension-control parameter, t a tension parameter, r the ratio of the width and height of the muscle, and rn this ratio in a fully relaxed state. An interesting approach [9] introduces the idea of abstracting muscles by an action line (a polyline in practice), representing the force produced by the muscle on the bones, and a surface mesh deformed by an equivalent mass- spring mesh. To smooth out mesh discontinuities, they employ special springs termed angular springs that tend to restore the initial curvature of the surface at each vertex. However, angular springs cannot deal with local inversions of the curvature. Another method [10] also uses an action line and a muscle mesh. The action line, represented by a polyline with any number of vertices, is moved for each posture using a predefined behavior and a simple physically based simulation. It is then used as a skeleton for the surface mesh, and the deformations are produced in a usual way. Each vertex of the muscle mesh is parameterized by one or two underlying action lines so that the deformations of the muscle shape are driven by the associated action lines. It, therefore, reduces the 3D nature of the elasticity problem to one dimension for fusiform muscles (one action line) and two dimensions for flat muscles (several action lines). Each action line is approximated by a polyline associated with a 1D mass-spring-damper system. The muscle insertion and origin are anchored to the skeleton joints in order to estimate the action line deformations from skeleton motions. Collision avoidance between components is handled with repulsive force fields.

4.2.2

Data-Driven Methods

Several attempts have been made to overcome the limitation of geometric skin deformation by using examples of varying postures and blending them during animation. Aimed mostly at real-time applications, these example-based methods essentially seek solutions to efficiently leverage realistic shapes that come from the captured skin shape of real people, physically based simulation results, or sculpted by skilled designers. Data-driven methods have recently arisen with the growth of 3D acquisition devices, particularly with full-body 3D scanners. These acquisition devices allow us to get reliable and accurate data from complex shapes like the human body. The basic idea of these approaches is to use real data that cover the

86

4 Virtual Characters

space of shape variation of the character to train a deformable model. The underlying hypothesis is that it is possible to generalize from a set of individuals by, for example, applying statistical analysis to the training set. This family of methods initially started by modeling static shape variations (morphology), such as the human face [11], and has been further applied to address dynamic shape variations (poses), such as deformable human bodies. We have now reached the stage where both types of variations (morphologies and poses) are simultaneously handled. The approach named Skinning Mesh Animations (SMAs) [12], presents an original skin example-based scheme. The main difference is that it does not require a predefined skeleton. The method takes as input a set of deformed meshes representing the pseudo-articulated deformable shape in different poses. It then automatically estimates statistically relevant bones based on the hypothesis that clustering triangles with analogous rotation sequences indicates the near rigid structure of the mesh animation. It further determines bone transformations, bonevertex influence sets, and vertex weight values for producing skinned animations that approximate the original deformable animation. The skinning approximation is particularly suited for shapes with a sufficient near-rigid structure and does not apply well for highly deformable shapes. SMAs support hardware rendering and pose editing.

4.2.3

Physics-Based Approaches

Various approaches have been proposed to introduce more realism by introducing dynamic-based deformations [13]. For example, it could be based on the introduction of a framework for skeleton driven animation of elastically deformable characters [14]. The proposed framework is somehow similar to FFD-based animation approaches as it embeds the object in a control lattice, but it uses continuum elasticity and FEM to compute the dynamics of the object being deformed. Bones of the control skeleton are restricted to lying along the edges of the control lattice, so that the skeleton can be considered a constraint in the dynamic system.

4.3 4.3.1

Locomotion Locomotion Generation

Walking has global and specific characteristics. From a global point of view, every human walking has comparable joint angle variations. However, close up, we notice that individual walk characteristics are overlaid to the global walking gait. We will take as an example the walking engine described in [15]. Walking is defined as a motion in which the center of gravity alternatively balances from the right to the left side. It has the following characteristics:

4.3

Locomotion

87

• At any time, at least one foot is in contact with the floor, the single support duration (ds). • there exists a short instant during the walk cycle, where both feet are in contact with the floor, the double support duration (dds). • it is a periodic motion that has to be normalized in order to adapt to different anatomies. The joint angle variations are synthesized by a set of periodic motions which we briefly describe here: • Sinus functions with varying amplitudes and frequencies for the humanoid’s global translations (vertical, lateral, and frontal) and the humanoid’s pelvic motions (forward/backward, left/right, and torsion). • Periodic functions based on control points and interpolating hermite splines. They are applied to hip flexion, knee flexion, ankle flexion, chest torsion, shoulder flexion, and elbow flexion. More generally, many works have been dedicated to the locomotion of virtual humans [16, 17]. The key-framing technique allows an animator to specify key postures at specific key times. Using appropriate software (e.g., Autodesk’s 3ds Max or Maya), skilled designers can control the motion in detail. However, this technique is quite labor-intensive, as any motion parameter change entails the animators to modify every key-frame. Kinematics approaches generate motions from parameters such as position feet or speed value [18, 19]. Motions are generated by giving a predefined set of foot positions (footprints) and timing information. These data are generally computed by a motion planning technique, which has to be as interactive as possible to be comfortable for animators. Dynamics approaches aim to describe a motion by applying physics laws. For example, it is possible to use control algorithms based on finite state-machine to describe a particular motion and proportional derivative servos to compute the forces [20]. However, even if these methods produce physically correct animations, the configuration of their algorithms remains difficult. It is not easy to determine the influence of each parameter on the resulting motions. Many methods based on empirical data and bio-mechanical observations can generate walking [21] or running patterns [22], reactive to given user parameters. Other similar approaches consider the environment, to walk on uneven or sloped terrains [23] or to climb stairs [24]. Despite their real-time capability, all these methods lack realism, as the legs’ motion is considered symmetrical, for example. Another class of animation techniques reuses original motion capture data. Treated as a time-varying signal, a new motion can be generated by modifying its frequency bands [25] or its Fourier coefficients [26]. Other methods [27, 28] define each motion by B-spline coefficients. New motions are then computed by setting weights on the various original motions and performing interpolations using polynomial and RBF (radial basis function) functions. The Kovar and Gleicher [29] method wraps input motions into a data structure that ensures consistent time warping, root alignment, and constraint matching.

88

4

Virtual Characters

We will explain in more details in the next section the method introduced by Glardon et al. [30], which applies PCA (principal component analysis) on a motioncapture database composed of various walking and running cycles. The generation of new parameterized motions is performed by interpolation and extrapolation into a hierarchical structure of PCA spaces.

4.3.2

PCA-Based Locomotion

A method from statistics, PCA [31] has become attractive in animation synthesis. It is used either to compress the data [32] or to emphasize similarities between input data [11, 33] to generate motion according to control parameters such as age or gender. In this section, we introduce a model allowing efficient generation of a whole locomotion sequence only at each high-level parameter update. To illustrate our methodology, we use two examples of locomotion: walking and running. The first step consists in creating a motion database; for example, we use an optical motion-capture system and a treadmill to record five subjects differing in age and gender (two women and three men). The physical parameter speed of the various sequences varies from 3.0 to 7.0 km/h, by increments of 0.5 km/h in the case of walking, and from 6.0 to 12.0 km/h, by increments of 1.0 km/h for running. The sequences were then segmented into cycles (one cycle includes two steps, starting at right heel strike), and four of them have been selected. These cycles are aligned to an identical locomotion direction, converted to joint angle space (represented by axis angles, according to the standard H-ANIM skeleton [34]), and finally normalized so that each sequence is represented by the same number of samples. In addition, a standing (neutral) position sequence of each subject has been inserted to represent the speed value 0 km/h. Consequently, the database is composed of 180 walking cycles and 140 running cycles. In practice, a person’s posture, or a body pose, in a given key-frame can be defined by the position and orientation of a root node and a vector of joint angles. A motion can then be represented by an angular motion vector μ, which is a set of such joint angle vectors measured at regularly sampled intervals. Computing the entire locomotion sequence is time-consuming; the PCA technique [31] is applied, drastically reducing the dimension of the input motion-capture data space. The resulting space, referred to as the main PCA, is computed with the input motion matrix M composed of all motion vectors μ from our database with k subjects. To center this space with respect to the whole data set, we define μ0 as an average vector of all n motion vectors. The basis vectors describing this space are the m first orthogonal PCs (principal components) necessary to compute an approximation of the original data. Let α = (α1,α2,. . .,αm) be a coefficient vector and E = (e1,e2,. . .,em) a vector matrix of the first PCs (or eigenvectors) of M, a motion θ can be expressed as:

4.3

Locomotion

89 m

θ ffi θ0 þ

αi ei = θ0 E

ð4:3Þ

i=1

As mentioned, the purpose of this PCA is to reduce the dimensionality of the input data, a very important aspect for real-time purposes. To generate a new and entire motion, a blending technique could be applied on various α, according to three high-level parameters: personification vector p, where pi is the weight for the i subject; type of locomotion T (walk or run); and speed S. In theory, a linear interpolation between coefficient vectors is performed, while original data are nonlinear (axis angle). This ambiguity can be solved by applying the method presented in [35]. However, in practice, the method here is intended for locomotion, where the different motions show small variations between postures, mostly in the sagittal plane, therefore allowing linear interpolation. Unfortunately, blending is not appropriate for motion extrapolation. As the goal of a locomotion engine is to propose extrapolation of physical parameters, speed in our example, an alternative technique must be applied. A complete method proposes a hierarchical structure of PCA spaces that first helps to classify the motions and second allows a linear least square in a very low dimension, instead of in the main PCA space. Data are then retargeted to different human sizes from those captured, and there is a process, based on motion analysis, to unwrap the normalized data. To produce animation adaptable to any kind of virtual human, the generalization of the heterogeneous input data we used is an important aspect. Indeed, we captured various subjects, not only with differences in the style of motion but also, and more importantly, with differences in size. First, all 3D positions (i.e., the humanoid root joint) of the motion vectors are divided by the leg length H of the captured subject. Murray [36] has shown that all the leg relative angles in the sagittal plane (hip, knee, ankle) show very similar trajectories for all adult men for the same value of normalized speed V, obtained by dividing the walking velocity v (in m/s) by the hip joint height H (i.e., the leg length in meters). We can generalize this statement to the running motion, too. The normalization of the input data is performed at the speed control level, as the end user must additionally give the leg length parameter H of the human he wants to animate. Every input locomotion sequence and each generated sequence contain a fixed number of frames, due to the normalization step performed during preprocessing. The time warp induced is handled using a walking cycle frequency function f that links a given normalized speed to a cycle frequency [37] called the Inman law. We adapted this law to our observations, performed on a treadmill, and extended it to the case of running motion. We fit the data to an approximation function of the form axb, similar to the Inman law. The resulting frequency functions are described by the equation:

90

4

Virtual Characters

Fig. 4.3 Walking generated using PCA

f ðV Þ = 0:85V 0:4

ð4:4Þ

f ðV Þ = 1:07V 0:24

ð4:5Þ

for walking motions and by:

for running motions. Therefore, the animation engine can continuously vary the speed and compute the “phase update” as in the walking engine in [21], where the phase varies between [0. . .1] and the phase update is computed with the following equation: Δφ = Δtf ðV Þ

ð4:6Þ

where Δt is the elapsed time between two updates and V the normalized speed. The phase Δφ multiplied by the number of frames in the normalized sequences returns the frame to display. Finally, the goal of the method is to allow the end user to set some high- level parameters: a personification vector p composed of weights assigned to each subject, the type of locomotion T , the speed S, and the human size H (i.e., the leg size). Figure 4.3 shows an example.

4.4

Virtual Human-Object Interaction

4.4 4.4.1

91

Virtual Human-Object Interaction Feature Modeling and Smart Objects

Making virtual characters, especially virtual humans, capable of manipulating objects usually requires solutions to closely related issues on two fronts: the specification of object behavior and its reflection through animation. The former usually takes place on a more abstract level, whereas the latter is mainly concerned with generation of valid and believable motion for humans and objects. There are works in literature that have addressed both, together or separately. The human hand is a complicated articulated structure with 27 bones. Not only must the movements of these joints be calculated, but also the reaching motion of the arm and the body needs to be considered. For real-time performance in a VR system with many agents, fast collision-detection, and inverse kinematics algorithms [38] will be necessary in most cases. The calculation of the hand and body postures is not the only difficulty in grasping: realistic grasping also requires significant input about the semantics of the object. Even if the geometric and physical constraints permit, sometimes an object is simply not grasped that way. For example, a door handle must not be grasped from the neck section if the goal is to turn it. A fully automatic grasping algorithm that only takes the geometry of the object into account cannot always come up with solutions that are satisfactory in this sense. Fortunately, the grasping problem for virtual characters is easier than its robotics counterpart. Simply put, we do not have to be as accurate and physical constraints are much less of a problem. The main criterion is that the grasp must look realistic. In fact, the apparent physical realities of a virtual environment can be very different from those of the real world, with very different constraints being imposed. For example, we can imagine a virtual human holding an object that is several times his size and weight in air, while grasping it at a small site on the edge. This does not conflict with the previous examples addressing the reality issue, as for an autonomous virtual human in a virtual setting this is more a question of what he intends to do with the object (semantics) than the actual physics of grasping. On the behavior front, virtual human-object interaction techniques were first specifically addressed in the object specific reasoner (OSR) [39]. The primary aim of this work is to bridge the gap between high-level AI planners and the low-level actions for objects, based on the observation that objects can be categorized with respect to how they are to be manipulated. There is little consideration given to interaction with more complex objects, although there are examples that employ object grasping. The closely related work on parameterized action representation [40] addresses the issue of natural language processing for virtual human-object interactions. Natural language usually describes actions at a high level, leaving out many crucial details. A PAR describes an action by specifying conditions and execution steps. Components of a natural language instruction are mapped to the parameters of a PAR, which is linked to the motion generators.

92

4

Virtual Characters

Fig. 4.4 Defining and using smart objects

Feature modeling is an interesting approach to modeling objects for manipulation. In feature modeling, object models contain more than geometric information, for example, the function of some part of the object or its semantics. For example, in a CAD/CAM application, the definition of an object may contain instructions of how a certain part of the object is to be assembled. The smart objects paradigm has been introduced as a feature modeling approach for interactions of virtual humans with virtual objects [41, 42]. In essence, smart objects provide not only the geometric information necessary for drawing them on the screen but also semantic information useful for manipulation purposes. The smart objects paradigm is based on extending objects (shapes) with additional information on their semantics. Its focus is on autonomous virtual humans within virtual environments. The semantic information that a smart object carries is mainly about the behavior of the object when an interaction occurs between the object and a virtual human. By “behavior,” we mean the changes in the appearance and state of an object because of the interaction (i.e., a virtual human opening a door). Attributes are the primary means of specifying information on how a virtual human manipulates its environment. They convey various kinds of information (e.g., where and how to approach to manipulate the object or to position the hands to grasp it), animation sequences (e.g., a door opening), and general nongeometric information associated with the object (e.g., weight or material properties). The semantic information in the smart object is used by the virtual characters to perform actions on/or with the object, such as grasping it, moving it, and operating it (e.g., a machine or an elevator). Figure 4.4 shows how to define a smart object (here a CD-ROM) and how a virtual character can then use it. Furthermore, the smart objects paradigm is focused on behavioral animation and virtual human actor-object interactions, where predefined interaction plans, and finite state machines are strongly bound to the objects. This imposes a limitation on situations where direct or indirect interactions between agents and multiple

4.4

Virtual Human-Object Interaction

93

objects, or simply multiple objects, are required. Recent work in [43] applied the smart objects approach to implementing “attentive” agents. Attention points and face descriptors are specified on objects in the modeling phase, which are then used by a gaze manager to automatically generate gaze behaviors for an agent when he needs to look at an object. Three low level gaze functions are used: glance, look, and stare.

4.4.2

Grasping

Grasping is perhaps the most important and complicated motion that manipulation of objects involves. The difficulty comes not only from properly “wrapping” the fingers around the object but also from the fact that the grasp must be suitable for the intended manipulation. In [44], a classification of the hand postures commonly used for grasping in manufacturing tasks is given. One of the earlier works on grasping that uses this classification to automatically generate grasping motions is the approach described in [45]. This approach is based on three steps: • Heuristic grasping decision. Based on a grasp taxonomy, Mas and Thalmann [46] proposed a completely automatic grasping system for synthetic actors. In particular, the system can decide to use a pinch when the object is too small to be grasped by more than two fingers or to use a two-handed grasp when the object is too large. Figure 4.5 shows the taxonomy. • Inverse kinematics to find the final arm posture. Inverse kinematics is also commonly used for creation of reaching motions for articulated structures [38, 47, 48], but it is still difficult to obtain realistic full-body postures without substantial tweaking. On the other hand, database-driven methods [49] cope better with full body postures. These methods are based on capturing motions for reaching inside a discrete and fixed volumetric grid around the actor. The reaching motion for a specific position is obtained through interpolation of the motions assigned to the neighboring cells.

Fig. 4.5 Grasping taxonomy

94

4

Virtual Characters

Fig. 4.6 Grasping (a) Virtual sensors (b) Grasping sensors (c) One-hand grasping (d) Multiple grasping

• Multisensor hand. Huang et al. [50] approach is adapted from the use of proximity sensors in robotics [51], the sensor-actuator networks [52], and their own work on human grasping. In their work, the sphere multisensors have both touch and length sensor properties, and have been found very efficient for synthetic actor grasping problem. Multisensors are considered as a group of objects attached to the articulated fig. A sensor is activated for any collision with other objects or sensors. Here we select sphere sensors for their efficiency in collision detection. Figure 4.6a, b show the multisensors and Fig. 4.6c a cube grasped. In case of large objects, such as furniture, grasping simultaneously involves two or more persons. Therefore, we focused on a multiagent grasp action for encumbering objects (see Fig. 4.6d). As the object’s weight and geometry is distributed over several hand support points of different agents, the heuristic motion planning schemes have to be different from the ones for an object grasp performed by a single individual. For example, a large object might be grasped frontally by the first agent and from behind by the second agent (see Fig. 4.7). The humanoid is the active agent, the balloon the passive agent. We can reverse the role of active and passive agents, for example, the balloon can be active and the human passive (see Fig. 4.8). The choice of the active and passive agents depends on which agent is supposed to control the other one: Is the human carrying the balloon or is the balloon lifting the human into the air? By extension, any agent can be active and passive at the same time, for example, a box attaches to a balloon and is attached to a humanoid. Another approach used in the Jack system [53, 54], which is based on Cutkosy’s grasp classification [44], is described in [55]. This approach uses specialized controllers for each different type of grasp to close the fingers. It uses parallel transition networks (PaT-Nets) [56] to control each finger. Transitions in the PaT-Nets are triggered by collisions between the fingers and the object. Different PaT-Nets are

4.4

Virtual Human-Object Interaction

95

Fig. 4.7 Multi-agent carrying

Fig. 4.8 (a) Human carrying the balloon (b) Balloon lifting the human into the air

used to simulate different grasp types; the differing responses of the different PaT-Nets actually define how the grasp is to take place. A different approach in [57] takes the interactions between the fingers and the object into account by means of dynamics simulation. This work proposes passive and active controllers for the hand. The passive controller is constructed to attract the

96

4 Virtual Characters

hand posture toward a naturally comfortable pose and to force it to remain within joint limits. The active controller is used for the actual grasping and drives the hand toward a pose that is a blend of two out of a total of six setpoints that are automatically extracted from motion- capture examples. The authors demonstrate that the method can be used for manipulation of simple objects by interactively switching between set points for the active controller. This work looks at the problem of grasping from an interesting angle since properly modeling physical interactions between the fingers and the object is important. However, the method is limited to executing grasps that are in the library of examples. Furthermore, the amount of control one can exercise on the grasping may not be enough if a specific and meaningful object manipulation sequence is to be realized. Knowledge about the presence of elongated features is relevant in the context of animation for the definition of posture and grasping motion for virtual humans. While tubular or elongated features can be quite easily defined during the design processes, their automatic extraction from unstructured 3D meshes is not a trivial task. Moreover, geometric parameters such as tube axis or section size should be made readily available to the animation tool. Among the many methods for shape analysis, skeleton extraction techniques are the most suitable for identifying tubular features. Topology-based skeletons, for example, code a given shape by storing the evolution of the level sets of a mapping function defined on its boundary. A geometric skeleton is usually associated with this coding, defined by the barycenters of the contours. Ciger et al. [58] introduces a new grasping framework, which brings together a tubular feature-classification algorithm, a hand grasp posture–generation algorithm, and an animation framework for human-object interactions. This unique combination is capable of handling grasping tasks within the proper context of virtual human object manipulation. This is very important because how an object is to be grasped depends strongly on how it is be used. The method has the advantage that it can work with relatively complex objects, where manual approximation with simple geometrical primitives may not be possible or practical. Furthermore, the method supports many intuitive parameters for controlling the grasping posture, such as the finger spread or the thumb configuration. Since the grasp parameters are specified as ranges, it is possible to generate a different posture each time a virtual human attempts to grasp an object, depending on the current configuration of the virtual human. The algorithm to detect tubular features is called Plumber, and it is a specialized shape-classification method for triangle meshes. The Plumber method analyzes the shape of an object by studying how the intersection of spheres centered at the mesh vertices evolve while the sphere radius changes. For example, for a thin limb, the curve of intersection between the mesh and a sphere will be simply connected for a small radius and then will rapidly split into two components when the radius increases and becomes greater than the tube size. A detailed description of the shape-analysis technique, which uses the intersecting sphere, and of the Plumber method can be found in [59, 60]. The algorithm segments a surface into connected components that are either body parts or elongated features, that is, handle-like and

4.4

Virtual Human-Object Interaction

97

protrusion-like features, together with their concave counterparts, narrow tunnels, and wells. The segmentation can be done at single- or multiscale and produces a shape graph that codes how the tubular components are attached to the main body parts. Moreover, each tubular feature is represented by its skeletal line and an average cross-sectional radius. Grasping is perhaps the most important part of a manipulation sequence, but it is not alone. A full sequence can consist of walking to and reaching for the object, looking at it, grasping it multiple times, and keeping the hands constrained to the object while it is moving. Therefore, the smart objects are required to provide a full manipulation sequence, putting the grasping action into the proper context.

4.4.3

Motion Planning

An issue commonly encountered in virtual character animation is the problem of collisions with the environment. This problem is common to locomotion and object manipulation. Most animation algorithms (e.g., inverse kinematics) operate only on virtual characters and do not take the environment into account. When these motions are played, collisions between the virtual human and the scene may occur, detracting from the believability of the virtual environment. If care is taken during the design stage, the probability of collisions happening can be reduced; however, it is not possible to eliminate them, especially if we are not able to directly control what is happening in the virtual environment (e.g., if virtual characters are present). In the field of robotics, researchers have been working on motion-planning methods for robots to avoid collisions [61]. These methods can be applied to virtual characters, but a virtual character is an articulated structure with many degrees of freedom; therefore, the dimensionality of the search space is very high. Methods based on probabilistic roadmaps [62, 63] are particularly suitable for structures of this complexity. A probabilistic roadmap is a data structure (graph) that is used to capture the connectivity of the search space. Nodes in the roadmap correspond to randomly sampled configurations of the robot (e.g., joint angles), and an edge between two nodes in the roadmap means that the robot can move between corresponding configurations by means of a local planner. Among these, visibilitybased roadmap [64] construction techniques aim at reducing the number of nodes while the rapidly exploring random trees (RRT) [63, 65] focus on sufficiently exploring the configuration space at the expense of increasing the number of nodes. The latest trend in motion planning for virtual characters is the use of motion capture data together with roadmap techniques. In [19], the authors attempt to solve the problem of biped locomotion by using randomly sampled feet positions to construct the roadmap, which is augmented afterward with a posture transition graph. Nodes in the roadmap are connected using data from input motion clips. In [66], motion-planning algorithms based on probabilistic roadmaps are used to control 22 degrees of freedom (DOFs) of human-like characters in interactive applications. The main purpose is the automatic synthesis of collision-free reaching

98

4

Virtual Characters

Fig. 4.9 Reaching examples

motions for both arms, with automatic column control and leg flexion. Generated motions are collision- free, in equilibrium, and respect articulation range limits. To deal with the high (22) dimension of the configuration space, the random distribution of configurations is biased to favor postures most useful for reaching and grasping. Figure 4.9 shows examples. In addition, there are extensions to interactively generate object manipulation sequences: a probabilistic inverse kinematics solver for proposing goal postures matching predesigned grasps; dynamic update of roadmaps when obstacles change position; online planning of object location transfer; and an automatic stepping control to enlarge the character’s reachable space. The work in [67] also focuses on the problem of object manipulation. The path of the object to be moved is computed using the RRT algorithm. An inverse kinematics algorithm generates poses that match the object’s position and orientation. Using soft constraints, it also biases the poses towards those in a posture database. As commonly seen with many virtual humans’ motion planning methods, some postprocessing steps are used to increase the realism of the generated motions. The authors aim to reduce the dimensionality of the configuration space by planning only for the position and orientation of the object being manipulated. Most existing work targeting motion planning for virtual humans assumes that the virtual environment is static. However, if motion planning is to be used for object

4.5

Facial Animation

99

manipulation, then it is important to consider dynamics of the environment. Possible changes in the workspace can be included as additional dimensions in the configuration space, but a large number of dimensions is undesirable since it will reduce planning performance. To avoid this, dynamic roadmaps (DRM) can be used. For example, the work of Kallmann in [68] proposes a method to construct a dynamic roadmap [69] on top of an RRT planner for application to a humanoid robot.

4.5

Facial Animation

The goal of facial animation systems has always been to obtain a high degree of realism using optimum-resolution facial mesh models and effective deformation techniques. Various muscle-based facial models with appropriate parameterized animation systems have been effectively developed for facial animation [70– 72]. The Facial Action Coding System [73] defines high-level parameters for facial animation, on which several other systems are based. FACS defines fundamental basic actions known as Action Units (AUs). Each AU describes the contraction of one facial muscle or a group of related muscles. Thus, FACS was derived by analysis of the anatomical basis for facial movements. There are 46 action units. This repertoire of action units can be regarded as a “kit” for creating and compositing facial expressions. FACS is, however, limited to those muscles that can be controlled voluntarily. Most facial animation systems typically follow these following steps: 1. Define an animation structure on a facial model by parameterization. 2. Define “building blocks” or basic units of the animation in terms of these parameters, for example, static expressions and visemes (visual counter parts of phonemes). 3. Use these building blocks as key-frames and define various interpolation and blending functions on the parameters to generate words and sentences from visemes and emotions (see an example singing in Fig. 4.10) from expressions. Fig. 4.10 A singing facial expression

100

4

Virtual Characters

The interpolation and blending functions contribute to the realism for a desired animation effect. Generate the mesh animation from the interpolated or blended key-frames. Given the tools of parameterized face modeling and deformation, the most challenging task in facial animation is the design of realistic facial expressions and visemes. The complexity of the key-frame based facial animation system increases when we incorporate natural effects such as coarticulation for speech animation and blending between a variety of facial expressions during speech. The use of speech synthesis systems and the subsequent application of coarticulation to the available temporized phoneme information is a widely accepted approach [74]. Coarticulation is a phenomenon observed during fluent speech, in which facial movements corresponding to one phonetic or visemic segment are influenced by those corresponding to the neighboring segments. Two main approaches taken for coarticulation are by Pelachaud [75] and Cohen and Massaro [76]. Both these approaches are based on the classification of phoneme groups and their observed interaction during speech pronunciation. Pelachaud arranged the phoneme groups according to the deformability and context dependence in order to decide the influence of the visemes on each other. Muscle contraction and relaxation times were also considered, and the Facial Action Units were controlled accordingly. For facial animation, the MPEG-4 standard is particularly important. The Facial Definition Parameter (FDP) set, and the Facial Animation Parameter (FAP) set are designed to encode facial shape; animation of faces, thus, reproduces expressions, emotions, and speech pronunciation. The FDPs are defined by the locations of the feature points and are used to customize a given face model to a particular face. Figure 4.11 shows the FDPs. They contain 3D feature points such as mouth corners and contours, eye corners, and eyebrow centers. FAPs are based on the study of minimal facial actions and are closely related to muscle actions. Each FAP value is simply the displacement of a particular feature point from its neutral position expressed in terms of the Facial Animation Parameter Units (FAPU). The FAPUs correspond to fractions of distances between key facial features (e.g., the distance between the eyes). For example, the MPEG-4-based facial animation engine [77] for animating 3D facial models works in real time and can display a variety of facial expressions, including speech pronunciation with the help of 66 low-level Facial Animation Parameters. Recently, the efforts in the field of phoneme extraction have resulted in software systems capable of extracting phonemes from synthetic as well as natural speech and generating lip-synchronized speech animation from these phonemes, thus creating a complete talking-head system. It is possible to mix emotions with speech in a natural way, thus imparting to the virtual character emotional behavior. The ongoing efforts are concentrated on imparting “emotional” autonomy to the virtual face, enabling a dialogue between the real and the virtual humans with natural emotional responses. Kshirsagar and Magnenat-Thalmann [78] use a statistical analysis of the facial feature point movements. As the data are captured for fluent speech, the analysis reflects the dynamics of the facial movements related to speech production. The

4.5

Facial Animation

Fig. 4.11 FDP feature points

101

102

4 Virtual Characters

results of the analysis were successfully applied for more realistic speech animation. Also, this approach makes possible an easy blending between various facial expressions and speech. The use of MPEG-4 feature points for data capture and facial animation allows a restriction of the quantity of data being processed, at the same time offering more flexibility with respect to the facial model. A system incorporating a personality model [79] for an emotional autonomous virtual human has also been developed.

4.6 4.6.1

Autonomous Characters Why Autonomous Virtual Characters?

Virtual characters are not just for movies and games. In the future, they can be at the heart of the simulation of activities for a variety of purposes, including education and training, treatment of psychological problems, and emergency preparedness. Imagine the following scenarios: • A user is being trained to perform some complex tasks, such as repairing a copy machine. He uses an interactive user manual, where an autonomous character plays an expert, showing him how to proceed. At every stage, the user is able to see what to do next, even when mistakes are made. • A therapist is helping a patient overcome a fear of public speaking. To overcome this fear, the patient has to perform while immersed in a virtual environment consisting of a seminar room and a virtual audience, which can react to the user in an autonomous way. The therapist can choose the type of virtual audience (for instance, one that is aggressive or sexist) that will result in a more effective treatment for the patient. • A user is learning basic life support (BLS) procedures. She is immersed in a virtual setting and discovers a victim lying on the ground. She has to give him BLS through her proxy, a virtual assistant. The user navigates the scene, assesses the situation, and makes decisions by issuing natural voice commands. The virtual assistant waits for commands and executes the actions. If the user’s commands are correct, the victim recovers. In cases where the user provides incorrect commands, the virtual assistant may refuse to do harm to the victim; in such situations, the virtual assistant may prompt the user for retrial, or may suggest an alternative possibility. Figure 4.12 shows the scene. To be able to accomplish such tasks, virtual characters should be able to act on their own. This means that they should be autonomous virtual characters (AVCs).

4.6

Autonomous Characters

103

Fig. 4.12 Virtual assistant for basic life support

4.6.2

Properties of Autonomous Virtual Characters

Autonomy is generally the quality or state of being self-governing. Rather than acting from a script, an AVC is aware of its changing virtual environment, making its own decisions in real time in a coherent and effective way. An AVC should appear to be spontaneous and unpredictable, making the audience believe that the character is alive and has its own will. To be autonomous, an AVC must be able to perceive its environment and decide what to do to reach an intended goal. The decisions are then transformed into motor control actions, which are animated so that the behavior is believable. Therefore, an AVC’s behavior consists of always repeating the following sequence: perception of the environment, action selection, and reaction. The problem with designing AVCs is determining how to decide on the appropriate actions at each point in time, to work toward satisfaction of the current goal, which represents the AVC’s most urgent need. At the same time, there is a need to pay attention to the demands and opportunities coming from the environment, without neglecting, in the long term, the satisfaction of other active needs. There are four properties that determine how AVCs make their decisions: perception, adaptation and intelligence, memory, and emotions. Perception: Perception of the elements in the environment is essential for AVCs, as it gives them an awareness of what is changing. An AVC continuously modifies its environment, which, in turn, influences its perceptions. Therefore, sensorial information drastically influences AVC behavior. This means that we cannot build believable AVCs without considering the way they perceive the world and each other. It is tempting to simulate perception by directly retrieving the location of each perceived object straight from the environment. To realize believable perception, AVCs should have sensors that simulate the functionality of their organic counterparts, mainly for vision, audition, and tactile sensation. These sensors should be used as a basis for implementing everyday behaviors, such as

104

4

Virtual Characters

visually directed locomotion, responses to sounds and utterances, and the handling of objects. What is important is the functionality of a sensor and how it filters the information flow from the environment. It is not necessary or efficient to model sensors with biological accuracy. Therefore, virtual eyes may be represented by a Z-buffered color image representing a character’s vision [80]. Each pixel of the vision input has semantic information giving the object projected on this pixel and numerical information giving the distance to this object. So, it is easy to know, for example, that there is a table just in front at 3 m. A virtual nose, or other tactile point-like sensors, may be represented by a simple function evaluating the global force field at the sensor’s location. The virtual ear of a character may be represented by a function returning the ongoing sound events. With these virtual sensors, AVCs should be able to perceive the virtual world in a way that is very similar to the way they would perceive the real one. In a typical behavioral animation scene, the actor perceives the objects and the other actors in the environment, which provides information on their nature and position. This information is used by the behavioral model to decide the action to take, which results in a motion procedure. The synthetic actor perceives his environment from a small window in which the environment is rendered from his point of view. Adaptation and intelligence: Adaptation and intelligence define how the character is capable of reasoning about what it perceives, especially when unpredictable events happen. An AVC should constantly choose the best action so that it can survive in its environment and accomplish its goals. As the environment changes, the AVC should be able to react dynamically to new elements, so its beliefs and goals may evolve over time. An AVC determines its next action by reasoning about what it knows to be true at a specific time. Its knowledge is decomposed into its beliefs and internal states, goals, and plans, which specify a sequence of actions required to achieve a specific goal. When simulating large groups or communities of AVCs, it is possible to use bottom-up solutions that use artificial life techniques, rather than top-down, plan-based approaches, such as those that are common in artificial intelligence. This allows new, unplanned behaviors to emerge. Memory: It is necessary for an AVC to have a memory so that similar behaviors can be selected when predictable elements reappear. Memory plays an important role in the modeling of autonomy, as actions are often decided based on memories. But imagine an AVC in a room containing 100 different objects. Which objects can be considered memorized by the virtual character? It is tempting to decide that whenever an object is seen by the AVC, it should be stored in its memory. But if you consider humans, nobody can remember every single object in a room. Therefore, the memory of a realistic AVC should not be perfect either. Noser et al. [81] proposed the use of an octree as the internal representation of the environment seen by an actor because it can represent the visual memory of an actor in a 3D environment with static and dynamic objects. Emotions: The believability of an AVC is made possible by the emergence of emotions clearly expressed at the right moment. An emotion is an emotive

4.6

Autonomous Characters

105

reaction to a perception that induces a character to assume a physical response, facial expression, or gesture or to select a specific behavior. The apparent emotions of an AVC and the way it reacts are what give it the appearance of a living being with needs and desires. Without them, an actor would just look like an automaton. Apart from making them appear more realistic, AVC’s visible emotions can provide designers with a direct way of influencing the user’s emotional state. To allow AVCs to respond emotionally to a situation, they could be equipped with a computational model of emotional behavior. Emotionally related behavior, such as facial expressions and posture, can be coupled with this computational model, which can be used to influence their actions. The development of a good computational model is a challenge.

4.6.3

Behaviors for Autonomous Virtual Characters

The four properties just discussed are very important in creating believable autonomous characters and should greatly contribute to their behavior. Behaviors can range from very simple, like reflexes, to very complex, although it is not really clear what a complex behavior is. We could find in daily life many examples of simple but complex situations. Let’s take two examples. Suppose a pedestrian walking along a boardwalk suddenly perceives another pedestrian coming exactly on the same line. What to do? This is a common problem that we solve without explaining the strategy; this is dependent on psychological factors. How to make people eating realistic? Some specific behaviors have to be taken into account, like eating a T-bone steak. In this case, people generally (when they are right-handed) use the right hand for the knife and the left for the fork when they cut their steak, but then they switch the fork into their right hand to eat. Reflexes The simplest spontaneous behavior in human beings produced by external stimuli is reflex. Reflexes are unconscious movements regulated by the nervous system. They can be considered the basis of movements or even as the root of human movement. We can also consider reactions produced by reflexes as primitive responses that protect our body from danger and help us adjust to our surroundings. The movement performed during a reaction depends on many factors. The most evident factors are the physical characteristics: gender, age, body constitution, etc. For example, aged people have uncertain or slow reactions. Other less evident factors are the internal or psychological parameters of the person such as personality and emotional state. Consider a person who has been frightened; if she is highly neurotic, she might shout out very loudly as a reactive response. A typical example where people react to a stimulus is when a ball is thrown to them [82]. The reactive behavior controller algorithm tracks the position of the objects all the time. In this case the object is a ball that has an initial position and

106

4

Virtual Characters

Fig. 4.13 Reflex reactions to a ball

initial velocity. When the Euclidean distance between the object and the character is less than the attention distance, we can say that the character can see the object. At this point, the perception algorithm will compute a projection of the trajectory of the ball to know if the ball can be reached by the arms of the character. If the object is reachable, the character starts reacting. The emotional state produced in people by a specific event could influence the kind of movement they may perform. If the characters are afraid of a stimulus, they will contract their body; if they are surprised, they will do an expansive movement, and so on. In this kind of movement, we can modify some characteristics such as amplitude or velocity, which are also factors that influence the variation of movements. Therefore, the inputs of the reactive behavior controller are the individual descriptors and the properties of the stimuli. The algorithm chooses which kind of behavior the character will perform. The first part of the algorithm chooses a movement according to the gender and age. Having the movement type defined, we can know from the ontology that “intercept reaction” implies putting the hands in the stimuli, “protect” implies raising the lower part of the arms, and “avoid” implies an inclination of the column. The second part of the algorithm provides other configurations of simple movements that will integrate the final one. It defines the recruiting level for the spine inclination according to the level of danger of the stimuli. And finally, according to the emotional state, we add other movements, where we suppose that a person who is afraid may contract the body (e.g., flex the knees); and a person that is surprised will perform a broad movement (e.g. extend the arms). In Fig. 4.13, we present some results of combinations of movements by changing individual parameters and parameters of the stimuli.

Motivations Motivation is also a key cause of action, and we will consider basic motivations as essential to model life in the virtual world, providing a true virtual life. We adapt basic actor motivations and needs to urban situations, where most actions involve interactions with the environment, especially manipulation of objects in natural situations like eating or drinking. We focus on common life situations, where the actor senses and explores his environment and following an action-selection mechanism determines the suitable actions to take. For this, we can consider a mechanism

4.6

Autonomous Characters

–

107

+ Internal Variable

Internal Variables

Environment Information Others Motivations

Motivation

Goal-oriented Behaviors

Goal-oriented Behavior(s)

HCS

Motivated Action(s)

Intermediate Action(s)

Actions

Fig. 4.14 Simplified motivational model of action selection for virtual humans

of action selection based on a free-flow hierarchy associated with a hierarchical classifier [83]. The hierarchy of our model will contain four levels (depicted in Fig. 4.14). Social Behavior To model human society, it is important to consider motivations and social behavior. Interesting implications of introducing communication abilities for agents could be in modeling emergent collective behavior. Emergent crowds (see next section) can be formed based on transfer of both emotional states and behaviors, for example, by verbal and nonverbal communication between physically close agents. In order to realistically simulate how humans interact in a specific social context, it is necessary to precisely model the type of relationship they have and specify how it affects their interpersonal behavior. Sociologists have identified several dimensions that are important to provide for the simulation of any group behavior: (1) power (dominance and submissiveness of the agents), (2) attraction (friendliness and unfriendliness), (3) instrumental control (hierarchical rank), and (4) emotional expressiveness. For the development of the agents’ behavior in an organizational context, considering the hierarchical relationship is, of course, crucial, but the importance of the more informal dimensions should not be underestimated. The free-flow hierarchy [84] permits us to consider different types of motivations and information coming from the environment perception. The key idea is that, during the propagation of the activity in the hierarchy, no choices are made before the lowest level in the hierarchy represented by the actions is reached. The hierarchical classifier [85] provides a good solution to model complex hierarchies by reducing the search domain of the problem and using rules with weights. Also, we can easily make behavioral sequences (composed of a sequence of actions). As a result, the virtual character can move to a specific place to perform a physiological action and satisfy the motivations no

108

4

Virtual Characters

Hunger

Known food is remote

R4

R1

Reach food location

Known food is remote

R2

Go to food

R3

Food near Mouth

Eat

Near food

Take food

Fig. 4.15 Virtual life simulation: By default, the virtual human is working, waiting for motivation (for instance, drinking) to be activated. The food is kept in the kitchen, as seen in the background

matter where it is. In other words, the main role of the action-selection mechanism is to maintain the internal variables under the threshold by choosing the correct actions. Actions involving interactions are preferably chosen because they are defined to be directly beneficial for the virtual human. Otherwise, the virtual human is instructed to walk and reach the place where the motivation can be satisfied. During the simulation, the model is fed with parameters describing the current state of the actor concerning each of the motivations, and by flowing inside the hierarchical structure will correctly trigger the concerned actions. After an action is selected as a response to satisfy one of the actor’s motivations, the state parameter of the internal variable is adapted accordingly. An example is shown in Fig. 4.15. The agent’s behavior within a group is highly dependent on their location in the sociometric structures but also on their own social identity. This social characterization should be done using social-statistical variables: (1) culture, (2) gender, and (3) age, considering the agents’ roles within the group. A distinction between (4) task roles (e.g., function) and (5) socio-emotional roles (e.g., confident) is often used. An additional (6) status rating (prestige) can also be introduced. Social interactions require us to reproduce verbal communication. In such a graphical environment, interagent communication would be more than just an exchange of messages. We need to create models for sound propagation, which should be suitable for verbal communication. To increase the believability of behaviors resulting from emotions we include nonverbal communication elements. Nonverbal communication is concerned with

4.7

Crowd Simulation

109

Fig. 4.16 Intercommunication

postures and their indications of what people are feeling. Postures are the means to communicate and are defined by a specific position of the arms, legs, and angles of the body. It is also essential to notice that the effects of nonverbal communication, though unconscious, are nevertheless important. For example, two persons communicating with each other (see Fig. 4.16) and rejecting a third one. The first two people are standing in an open triangle formation commonly used for friendly conversation. It leaves a potential opening for a third person to join them. Such a person attempts to join the group, but the first two respond by rejecting him. One raises his left arm, forming a barrier. The other deliberately avoids making eye contact with the newcomer. They have formed a closed formation, suppressing any possibility for a new person to join them. The newcomer feels rejected and leaves unhappily.

4.7

Crowd Simulation

Real-time crowds bring different challenges compared to the systems, either involving a small number of interacting characters (e.g., most contemporary computer games) or nonreal-time applications (as crowds in movies or visualizations of crowd evacuations after offline model computations). In comparison with single-agent

110

4

Virtual Characters

simulations, the main conceptual difference is the need for efficient variety of management at every level, whether it is visualization, motion control, animation, or sound rendering. As everyday experiences hint, virtual humans comprising a crowd should look different, move different, react different, sound different, and so forth. Even if we assume that perfect simulation of a single virtual human is possible, creating a simulation involving multiple such humans would be a difficult and tedious task. Methods easing control of many characters are needed; however, such methods should still preserve the ability to control individual agents. For an extended study of crowd simulation, see the book on crowd animation by Thalmann and Musse [16] and the recent history of crowd simulation by Musse et al. [86]. In comparison with nonreal-time simulations, the main technical challenge is increased demand for computational resources, whether it is general processing power, graphics performance, or memory space. One of the foremost constraining factors for real-time crowd simulations is crowd rendering. Fast and scalable methods to compute behavior, that can consider inputs not known in advance and render large and varied crowds are needed. While nonreal-time simulations can take advantage of knowing a full run of the simulated scenario (and therefore, e.g., can run iteratively over several possible options selecting the globally best solution), real-time simulations must react to the situation as it unfolds in the moment. Animating crowds [87] is challenging in both character animation and virtual city modeling. Though different textures and colors may be used, the similarity of virtual people would be soon detected even by nonexperts, for example, if everybody walks the same in this virtual city! It is, hence, useful to have a fast and intuitive way of generating motions with different personalities depending on gender, age, emotions, etc. from an example motion, say, a genuine walking motion. The problem is basically to generate variety among a finite set of motion requests and then to apply it to either an individual or a member of a crowd. It also needs very good tools to tune the motion [88]. Bouvier et al. [89] used a combination of particle systems and transition networks to model human crowds in visualization of urban spaces; lower level enables people to avoid obstacles using attraction and repulsion forces analogous to physical electric forces. Higher-level behavior is modeled by transition networks, with transitions depending on timing, visiting of certain points, changes of local densities, and global events. Brogan and Hodgins [90] simulated group behavior for systems with significant dynamics. They presented an algorithm for controlling the movements of creatures traveling as a group. The algorithm has two steps: First, a perception model determines the creatures and obstacles visible to each individual, and then a placement algorithm determines the desired position for everyone given the locations and velocities of perceived creatures and obstacles. Simulated systems included groups of legged robots, bicycle riders, and point-mass systems. Musse and Thalmann’s [86] proposed solution addresses two main issues: crowd structure and crowd behavior. Considering crowd structure, the approach deals with a hierarchy composed of crowd, groups and agents, where the groups are the most complex structure containing the information to be distributed among the individuals. Concerning crowd behavior, virtual agents are endowed with different levels of

4.7

Crowd Simulation

111

Fig. 4.17 Crowd simulation

autonomy. They can act according to an innate and scripted crowd behavior (programmed behavior), react as a function of triggered events (reactive or autonomous behavior), or be guided by an interactive process during simulation (guided behavior) [91]. Figure 4.17 shows a crowd guided by a leader. For emergent crowds, Ulicny and Thalmann [92] proposed a behavior model based on a combination of rules and finite state machines [93] for controlling agent’s behavior using a layered approach. The first layer deals with the selection of higherlevel complex behavior appropriate to the agent’s situation; the second layer implements these behaviors using low-level actions provided by the virtual human [94]. At the higher level, rules select complex behaviors (such as flee) according to the agent’s state (constituted by attributes) and the state of the virtual environment (conveyed by events). In rules, it is specified for whom (e.g., agent or agents in a particular group) and when the rule is applicable (e.g., at a defined time, after receiving the event, or when some attribute reaches a specified value), and what is the consequence of rule firing (e.g., change of the agent’s high-level behavior or attribute). At the lower level, complex behaviors are implemented by hierarchical finite state machines. In terms of rendering, the goal of a real-time crowd visualizer [95] is to render a large number of entities according to the current simulation state, which provides the position, orientation, and velocity for each individual. System constraints are believability, real-time updates (25 frames per second), and several digital actors ranging in the tens of thousands. Also, actors are made believable by varying their appearance (textures and colors) and animation; we may also add accessories like hats, glasses, or mobile phones (see Fig. 4.18). Their graphical representation is derived from a template, which holds all the possible variations. Thus, with only a limited set of such templates, we can achieve a varied crowd, leading to considerable time savings for designers. A template is defined as:

112

4

Virtual Characters

Fig. 4.18 Crowd simulation

• set of three meshes with decreasing complexity (LODs) • a set of textures in gray scale (except for the skin) identifying color modulation areas (pants, shirt, hair, etc.) • a skeleton (kinematic structure) • a corresponding animation database as skeletal orientations (here 1000 different walk cycles generated using a motion-blending-based locomotion engine) Each human in the visualization system is called an instance and is derived from a template. Individualization comes from assigning a specific gray-scale texture and a color combination for each identifiable region. Instances have individualized walk velocities and are animated by blending the available walk animations. The rendering pipeline advances consecutively in four steps. The first consists in culling, that is, determining visibility, and choosing the rendering fidelity for each simulated human. Three fidelities are defined: dynamic meshes, static meshes, and impostors. The next step of the pipeline is the rendering of dynamic meshes, which are the most detailed fidelity capable to interpolate animations based on skeletal postures. According to the current instance state (linear and angular walk velocities and time), animations are retrieved from the database and interpolated, yielding a smooth animation, with continuous variations of velocities, and no foot-sliding. The

4.7

Crowd Simulation

113

resulting skeletal posture is sent to a hardware vertex shader and fragment shader deforming and rendering the human on the graphics card. Then static meshes (also called baked or predeformed) constitute the second rendering fidelity, which keeps a pretransformed set of animations using the lowest-resolution mesh of the deformed mesh in the previous step. Precomputing deformations allows substantial gains in speed but constrains the animation variety and smoothness. The final rendering fidelity is the billboard model that, compared to previous approaches, uses a simplified scheme of sampling and lighting. World aligned billboards are used, with the assumption that the camera will never hover directly above the crowd. Thus, only sampled images around the waist level of the character are needed. For example, the templates are sampled at 20 different angles, for each of the 25 key-frames composing a walk animation. When constructing the resulting texture, the bounding box of each sampled frame is detected to pack them tightly together. When rendering billboarded pedestrians, cylindrical lighting is applied: Each vertex normal is set to point in the positive Z- direction, plus a small offset on the X- axis so that it points slightly outside the frame. We then interpolate the light intensity for each pixel in the fragment shader. Figure 4.18 shows a crowd with three fidelities. If we consider real crowds, they are formed by thousands of individuals that move in a bound environment. Each pedestrian has individual goals in space that he wants to reach, avoiding obstacles. People perceive their environment and use this information to choose the shortest path in time and space that leads to their goal. Emergent behaviors can also be observed in crowds. For example, in places where the space is small and very crowded, people form lanes to maximize their speed. Also, when dangerous events such as fires occur, pedestrians tend to react in very chaotic ways to escape. Planning crowd motion in real time is a very expensive task, which is often decoupled into two distinct parts: path planning and obstacle avoidance. Path planning consists in finding the best way to reach a goal. Obstacles can either be other pedestrians or objects that comprise the environment. The path selection criteria are the avoidance of congested zones and minimization of distance and travel time. Path planning must also offer a variety of paths to spread pedestrians in the whole scene. Avoidance, on the other hand, must inhibit collisions of pedestrians with obstacles. For real-time simulations, such methods need to be efficient as well as believable. Multiple motion-planning approaches for crowds have been introduced. As of today, several fast path planning solutions exist. Avoidance, however, remains a very expensive task. Agent-based methods offer realistic pedestrian motion planning, especially when coupled with global navigation. This approach gives the possibility to add individual and cognitive behaviors for each agent but becomes too expensive for a large number of pedestrians. Potential field approaches [96] handle long- and short-term avoidance. Long-term avoidance predicts possible collisions and inhibits them. Short-term avoidance intervenes when long-term avoidance alone cannot prevent collisions. These methods offer less-believable results than agent-based approaches, because they do

114

4

Virtual Characters

Fig. 4.19 Pedestrians using a hybrid motion planning architecture to reach their goal and avoid each other

not provide the possibility to individualize each pedestrian. However, this characteristic also entails much lower computational costs. A hybrid architecture [97] has been proposed to handle realistic crowd motion planning in real time. In order to obtain high performance, the approach is scalable. The authors divide the scene into multiple regions of varying interest, defined at initialization and modifiable at run time. According to its level of interest, each region is ruled by a different motion planning algorithm. Zones that attract the attention of the user exploit accurate methods, while computation time is saved by applying less expensive algorithms in other regions. The architecture also ensures that no visible disturbance is generated when switching from an algorithm to another. The results show that it is possible to simulate up to 10,000 pedestrians in real time with a large variety of goals. Moreover, the possibility of introducing and interactively modifying the regions of interest in a scene offers a way for the user to select the desired performance and to distribute the computation time accordingly. A simulation of pedestrians taking advantage of this architecture is illustrated in Fig. 4.19. Renault et al. [80] were the first to introduce synthetic vision for virtual humans in 1990. In 2010, Ondrej et al. [98] and in 2014, Dutra et al. [99] explored a visionbased approach to preventing collisions between agents within a crowd. There are still some researchers who work with synthetic vision in crowds today. There were also data-driven techniques for panic situations proposed by Oguz et al. [100], which were first described in 2007 and data-driven techniques [101–103]. Ju et al. [104]

4.7

Crowd Simulation

115

proposed a method that blends existing crowd data to create morphable crowd animations. In 2011, the Optimal Reciprocal Collision Avoidance (ORCA) method has been widely used in crowd domain benchmarks since then. Bicho et al. [105] proposed BioCrowds as the first crowd algorithm with no collisions. In this method, agents compete for space with each other and move only if space is available. Personalities [106], human relationships [107], and populations with purpose [108] are examples of features used to endow agents into the crowds. Durupinar et al. [106] extend the HiDAC (High-Density Autonomous Crowds) proposed by Pelechano et al. [109] by providing each agent with an ocean-based personality model. Low-level behavior parameters are automated based on each personality trait. Based on BDI and Helbing’s model, Okaya [107] proposes a model of human behavior in evacuation scenarios. Human relationships influence BDI states, and altruism forces affect Helbing’s computed movements. A 4D histogram with velocities, orientations, and positions of agents in the space is proposed by Musse et al. [110]. Based on real-world observations of crowd movements, Wolinski et al. [111] presented a framework for evaluating multiagent crowd simulation algorithms. Shoulson et al. [112] propose ADAPT, a tool to create prototyping testbeds for crowd scenarios. Furthermore, Berseth et al. [113] propose SteerPlex, a model for estimating crowd simulation complexity, along with SteerFit, an automatic parameter fitting for steering algorithms [114]. A learned crowd-simulation model is combined with real crowd data to describe efficient trajectory [115]. According to Ahmet et al. [116], crowd simulations can be tuned, validated, and optimized by learning parameters from real-life videos. Liu et al. [117] describe a technique for predicting crowd dynamics using regression neural networks (NN). Using reinforcement learning (RL), Ravichandran et al. [118] model pedestrians as autonomous, learning, and reactive agents. In their study, Amirian et al. [119] present a data-driven crowd simulation method that mimics pedestrians’ observed traffic in a given environment. With artificial neural networks, Testa et al. [120] propose a novel method for estimating the evacuation time of a given complex environment with an average error of 5%. There are many applications and real case scenarios presented in a book published by Cassol et al. [121]. An algorithm is presented by Wong et al. [122] to compute the optimal route for each local region of an environment. Using a specified time, Bianco et al. [123] describe a statistical method for predicting crowd motion in the future. Matthew [124] proposes an inverse approach that is useful for both crowd simulation in virtual environments and urban crowd planning. Using an optimization-based formulation, Ren et al. [125] combine physics-based simulation methods with data-driven techniques to simulate tens or hundreds of agents at interactive rates. An authoring system is proposed by Ho et al. [126], which models the environment as an influence map based on attractors that guide crowd simulation activities. Stüvel et al. [127] present a novel method for dense crowd simulation in 2017 for navigation and collision avoidance. In real crowds of high density, people maneuvering the crowd need to twist their torsos to pass others. To plan such torso

116

4

Virtual Characters

Fig. 4.20 Torso crowds

orientations, they use capsule-shaped agents instead of traditional disc-shaped agents. Figure 4.20 shows an example. There are also papers in virtual, augmented, and mixed realities [128– 130]. Moussaïd et al. [131] propose to conduct crowd experiments using a shared three-dimensional virtual environment instead of real people. Researchers have also proposed papers that evaluate and measure the impact of user’s presence, such as [128, 132, 133]. For instance, in [134], authors propose to investigate performance and user experience in Social Virtual Reality (SVR) embodied, immersive, and providing face-to-face encounters. There are also papers in cultural and personality aspects of crowds. Favaretto et al. [135, 136] introduce a new method to crowd analysis extracting Hofstede dimensions from video sequences, to later be used in crowd simulation [137]. Durupinar et al. [135] propose to parameterize mobs’ common properties to create collective behaviors. In addition, group formation is a recurring theme. Zhang et al. [138] propose a method of controlling crowds by constrained and controlled group formation. A crowds-specific version of Bosse’s model [139] was proposed by Borg et al. [140]. Among the agents in the crowd, authors propose propagating emotion. According to Basak et al. [141], crowd simulation can be tuned with parameters from real-life videos. Besides, authors propose methods to individualize agents into the crowd, as for instance providing door and doorway etiquette [142], distracted agents [143], providing agents which motion is impacted by fluid forces [144] and obeying social distance in pandemic situations [145]. In [146], authors study the effects on users during interaction with a virtual human crowd in an immersive virtual reality environment or still exploring tactile feedback during immersive walking in VR applications [147]. Artificial neural networks have also been explored in papers to predict crowd emotions [148] and to integrate data-driven walking behaviors [149]. According to Li et al. [108], a multistage clustering strategy is used to quantify and detect collective motion, as well as the use of histograms of magnitude and momentum

References

117

for crowd anomaly detection and localization [150]. Sun et al. [151] propose using graph convolutional networks to predict crowd flows. Study of the impact of panic on crowd movement using information entropy [152] and investigation of evacuation path optimization (EPO) using ant colony inspiration [153]. Finally, high abstractions have been proposed, such as BioClouds and Legion. These models simulate huge big crowds, for example, one million people. Some methods have already been proposed to assist people in moving from buildings, dangerous situations, such as terrorist attacks in 2021. There were considerations about the impact of groups of agents on crowd efficiency, strategies to find optimal routes, the impact of groups of agents on the efficiency of the crowd, and even to simulate human mobility during the COVID-19 situation. According to Mirahadi and McCabe [154], a real-time evacuation model can be developed based on Dijkstra’s algorithm [155]. To study pediatric hospital evacuation, Li et al. [108] developed a rule-based model based on an extended cellular automata model. Xie et al. [156] analyze evacuation dynamics in pedestrian crowds impacted by groups of agents using a social force framework. Other authors have also proposed approaches with group behaviors in recent years, for example [157, 158]. Zhang et al. [159] propose a guidance strategy that maximizes the computation efficiency. Using Helbing’s model [160], Charrier et al. [161] propose an approach to include nervousness propagation in a classical simulation setup. Further, Shi et al. [162] propose a study aiming to verify that Viswalk software can reproduce the effect of egress flow under normal and emergency conditions. As can be seen, many papers have been proposed in the context of evacuation and crowd mobility, as in Orallo and Martinez [163].

References 1. Magnenat-Thalmann N, Thalmann D (1987) The direction of synthetic actors in the film “Rendez-vous a Montreal”. IEEE Comput Graph Appl 7(12):9–19 2. Magnenat-Thalmann N, Laperriere R, Thalmann D (1988) Joint-dependent local deformations for hand animation and object grasping. In: Proceeding of the graphics interface, vol 88, pp 26–33 3. Mohr A, Gleicher M (2003) Building efficient, accurate character skins from examples. ACM Trans Graph 22(3):562–568 4. Wang X, Phillips C (2002) Multi-weight enveloping: least-squares approximation techniques for skin animation. In: Proceedings of the symposium on computer animation, SCA02, pp 129–138 5. Chadwick J, Haumann D, Parent R (1989) Layered construction for deformable animated characters. In: SIGGRAPH ’89: Proceedings of the SIGGRAPH ’89 conference, pp 243–252 6. Thalmann D, Jianhua S, Chauvineau E (1996) Fast realistic human body deformations for animation and VR applications. In: Proceedings of computer graphics international. IEEE Computer Society Press, pp 166–174 7. Wilhelms J, Van Gelder A (1997) Anatomically based modeling. In: SIGGRAPH ’97: Proceedings of the SIGGRAPH ’97 conference, pp 173–180

118

4

Virtual Characters

8. Scheepers F, Parent R, Carlson W, May S (1997) Anatomy-based modeling of the human musculature. In: SIGGRAPH ’97: Proceedings of the SIGGRAPH ’97 conference, pp 163–172 9. Porcher-Nedel L, Thalmann D (1998) Real time muscle deformations using mass- spring systems. In: CGI98: Proceedings of the computer graphics international ’98 conference. IEEE Computer Society Press, pp 156–165 10. Aubel A, Thalmann D (2001) Interactive modeling of the human musculature. In: Proceedings of the computer animation 2001 conference, Seoul, pp 167–173 11. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: SIGGRAPH ’99: Proceedings of the SIGGRAPH ’99 conference, pp 187–194 12. James DL, Twigg CD (2005) Skinning mesh animations. ACM Trans Graph 24(3):399–407 13. Faloutsos P, Van de Panne M, Terzopoulos D (1997) Dynamic freeform deformations for animation synthesis. IEEE Trans Vis Comput Graph 3(3):201–214 14. Capell S, Green S, Curless B, Duchamp T, Popovic Z (2002) Interactive Skeleton- driven dynamic deformations. In: SIGGRAPH ’02: Proceedings of the SIGGRAPH ’02 conference, pp 41–47 15. Boulic R, Magnenat-Thalmann N, Thalmann D (1990) A global human walking model with real-time kinematics personification. Vis Comput 6(6):344–358 16. Thalmann D, Musse SR (2013) Crowd simulation, 2nd edn. Springer, New York 17. Multon F, France L, Cani-Gascuel M, Debunne G (1999) Computer animation of human walking: a survey. J Visualiz Comput Anim 10(1):39–54 18. van de Panne M (1997) From footprints to animation. Computer Graphics Forum, pp 211–223 19. Choi MG, Lee J, Shin SY (2003) Planning biped locomotion using motion capture data and probabilistic roadmaps. ACM Trans Graph 22(2):182–203 20. Wooten W, Hodgins J (2000) Simulating leaping, tumbling, landing and balancing humans. In: Proceedings of the IEEE international conference on robotics and automation, pp 656–662 21. Boulic R, Ulciny B, Thalmann D (2004) Versatile walk engine. J Game Dev:29–50 22. Bruderlin A, Calvert T (1996) Knowledge-driven, interactive animation of human running. In: Proceedings of the graphics interface 96 conference, pp 213–221 23. Sun H, Metaxas D (2001) Automating gait generation. In: SIGGRAPH ’01: Proceedings of the SIGGRAPH ’01 conference, pp 261–270 24. Chung S, Hahn J (1999) Animation of human walking in virtual environments. In: Proceedings of computer animation 1999 conference, pp 4–15 25. Bruderlin A, Williams L (1995) Motion signal processing. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 97–104 26. Unuma RTM, Anjyo K (1995) Fourier principles for emotion-based human figure. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 91–96 27. Rose C, Cohen M, Bodenheimer B (1998) Verbs and adverbs: multidimensional motion interpolation. IEEE Comput Graph Appl 18(5):32–41 28. Park S, Shin H, Shin S (2002) On-line locomotion generation based on motion blending. In: Proceedings of the SIGGRAPH/eurographics symposium on computer animation 2002, pp 105–111 29. Kovar L, Gleicher M (2003) Flexible automatic motion blending with registration curves. In: Proceedings of the SIGGRAPH/eurographics symposium on computer animation 2003, pp 214–224 30. Glardon P, Boulic R, Thalmann D (2004) A coherent locomotion engine extrapolating beyond experimental data. In: Proceedings of Computer Animation Social Agent 2004, pp 73–84 31. Jolliffe IT (1986) Principal component analysis, Springer series in statistics. Springer, New York 32. Alexa M, Müller W (2000) Representing animations by principal components. In: Proceedings of the eurographics 2000 conference, vol 19, pp 291–301

References

119

33. Lim I, Thalmann D (2002) Construction of animation models out of captured data. In: Proceedings of IEEE conference multimedia and expo 2002, pp 829–832 34. H-anim: The humanoid animation working group. http://www.h-anim.org. Accessed July 2007 35. Alexa M (2002) Linear combination of transformations. In: SIGGRAPH ’02: Proceedings of the SIGGRAPH ’02 conference, pp 380–387 36. Murray MP (1967) Gait as a total pattern of movement. Am J Phys Med 46(1):290–333 37. Inman V, Ralston H, Todd F (1981) Human walking. Wiliams & Witkins, Baltimore 38. Tolani D, Goswami A, Badler N (2000) Real-time inverse kinematics techniques for anthropomorphic limbs. Graph Models Image Process 62(5):353–388 39. Levison L (1996) Connecting planning and acting via object-specific reasoning. PhD thesis, Philadelphia 40. Badler N, Bindiganavale R, Allbeck WSJ, Zhao L, Palmer M (2000) Parameterized action representation for virtual human agents. In: Embodied conversational agents. MIT Press, Cambridge, pp 256–284 41. Goncalves L, Kallmann M, Thalmann D (2002) Defining behaviors for autonomous agents based on local perception and smart objects. Comput Graph 26(6):887–897 42. Kallmann M, Thalmann D (2002) Modeling behaviors of interactive objects for real-time virtual environments. J Visual Lang Comput 13(2):177–195 43. Peters C, Dobbyn S, McNamee B, O’Sullivan C (2003) Smart objects for attentive agents. J WSCG 11 44. Cutkosky MR (1989) On grasp choice, grasp models, and the design of hands for manufacturing tasks. IEEE Trans Robot Autom 5(3):269–279 45. Mas R, Boulic R, Thalmann D (1997) Extended grasping behavior for autonomous human agents. In: AGENTS ’97: Proceedings of the first international conference on autonomous agents. ACM Press, New York, pp 494–495 46. Mas R, Thalmann D (1994) A hand control and automatic grasping system for synthetic actors. Comput Graph Forum 13(3):167–177 47. Baerlocher P (2001) Inverse kinematics techniques for the interactive posture control of articulated figures. PhD thesis, Ecole Polytechnique Fédérale de Lausanne (EPFL) 48. Wang X, Verriest JP (1998) A geometric algorithm to predict the arm reach posture for computer-aided ergonomic evaluation. J Visualiz Comput Anim 9(1):33–47 49. Wiley DJ, Hahn JK (1997) Interpolation synthesis of articulated figure motion. IEEE Comput Graph Appl 17(6):39–45 50. Huang Z, Boulic R, Magnenat Thalmann N, Thalmann D (1995) A multi-sensor approach for grasping and 3D interaction. In: Computer graphics: developments in virtual environments, pp 235–253 51. Espiau B, Boulic R (1985) Collision avoidance for redondants robots with proximity sensors. In: Third international symposium of robotics research 52. van de Panne M, Fiume E (1993) Sensor-Actuator Networks. In: Proceedings of the of SIGGRAPH-93: computer graphics, Anaheim, pp 335–342 53. Phillips CB, Badler NI (1988) JACK: a toolkit for manipulating articulated figures. In: UIST ’88: Proceedings of the 1st annual ACM SIGGRAPH symposium on user interface software. ACM Press, New York, pp 221–229 54. Badler NI, Phillips CB, Webber BL (1993) Simulating humans: computer graphics animation and control. Oxford University Press, New York 55. Douville B, Levison L, Badler NI (1996) Task-level object grasping for simulated agents. Presence 5(4):416–430 56. Trias TS, Chopra S, Reich BD, Moore MB, Badler NI, Webber BL, Geib CW (1996) Decision networks for integrating the behaviors of virtual agents and avatars. In: VRAIS ’96: Proceedings of the 1996 virtual reality annual international symposium (VRAIS 96). IEEE Computer Society, Washington, DC, p 156

120

4

Virtual Characters

57. Pollard NS, Zordan VB (2005) Physically based grasping control from example. In: SCA ’05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation. ACM Press, New York, pp 311–318 58. Ciger J, Abaci T, Thalmann D (2005) Planning with smart objects. In: WSCG ’2005 59. Mortara M, Patane G, Spagnuolo M, Falcidieno B, Rossignac J (2004) Blowing bubbles for the multi-scale analysis and decomposition of triangle meshes. Algorithmica Spec Issues Shape Algorithms 38(2):227–248 60. Mortara M, Patane G, Spagnuolo M, Falcidieno B, Rossignac J (2004) Plumber: a method for a multi-scale decomposition of 3D shapes into tubular primitives and bodies. In: SMI ’04: Proceedings of the ninth ACM symposium on solid modeling and applications. Eurographics Association, Aire-la-Ville, pp 339–344 61. Latombe J (1991) Robot motion planning. Kluwer Academic Publishers, Norwell 62. Kavraki L, Svestka P, Latombe J, Overmars M (1994) Probabilistic roadmaps for path planning in high-dimensional configuration spaces, Technical report. Stanford University, Stanford 63. LaValle SM (1998) Rapidly-exploring random trees: a new tool for path planning, Technical report 98-11. Department of Computer Science, Iowa State University 64. Simon T, Laumond J, Nissoux C (2000) Visibility based probabilistic roadmaps for motion planning. Adv Robot J 14(2) 65. Kuffner JJ, LaValle SM (2000) RT-connect: an efficient approach to single-query path planning. In: Proceedings of the ICRA 2000 conference, pp 995–1001 66. Kallmann M, Aubel A, Abaci T, Thalmann D (2003) Planning collision-free reaching motions for interactive object manipulation and grasping. In: Proceedings of the eurographics 2003 conference, vol 22, pp 313–322 67. Yamane K, Kuffner JJ, Hodgins JK (2004) Synthesizing animations of human manipulation tasks. ACM Trans Graph 23(3):532–539 68. Kallmann M, Mataric M (2004) Motion planning using dynamic roadmaps. In: Proceedings of the ICRA 2004 conference, pp 4399–4404 69. Leven P, Hutchinson S (2000) Motion planning using dynamic roadmaps. In: Proceedings of the fourth international workshop on the algorithmic foundations of robotics (WAFR), pp 363–376 70. Parke F (1982) Parameterized models for facial animation. IEEE Comput Graph Appl 2(9): 61–68 71. Waters K (1987) A muscle model for animating three-dimensional facial expression. In: SIGGRAPH ’87: Proceedings of the SIGGRAPH ’87 conference, pp 17–24 72. Terzopoulos D, Waters K (1990) Physically based facial modelling, analysis and animation. J Visualiz Comput Anim 1(2):73–90 73. Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto 74. Hill DR, Pearce A, Wyvill B (1988) Animating speech: an automated approach using speech synthesized by rule. Vis Comput 3:277–289 75. Pelachaud C (1991) Communication and coarticulation in facial animation. PhD thesis, University of Pennsylvania 76. Cohen MM, Massaro DW, Thalmann NM, Thalmann D (1993) Models and techniques in Computer Animation. Springer, New York, pp 139–156. Chap. Modelling coarticulation in synthetic visual speech 77. Kshirsagar S, Garchery S, Magnenat-Thalmann N (2001) Feature point based mesh deformation applied to MPEG-4 facial animation. In: DEFORM ’00/AVATARS ’00: Proceedings of the IFIP TC5/WG5.10 DEFORM’2000 workshop and AVATARS’2000 workshop on deformable avatars. Kluwer, B.V., Deventer, pp 24–34 78. Kshirsagar S, Magnenat Thalmann N (2001) Principal components of expressive speech animation. In: Proceedings of the computer graphics international 2001 conference. IEEE Computer Society, pp 38–44

References

121

79. Kshirsagar S, Magnenat Thalmann N (2002) Virtual humans personified. In: Proceedings of the autonomous agents conference (AAMAS) 2002, pp 356–359 80. Renault O, Magnenat Thalmann N, Thalmann D (1990) A vision-based approach to behavioural animation. J Visualiz Comput Anim 1(1):18–21 81. Noser H, Renault O, Thalmann D, Magnenat Thalmann N (1990) Navigation for digital actors based on synthetic vision, memory and learning. Comput Graph Pergamon Press 19:7–19 82. Garcia-Rojas A, Vexo F, Thalmann D (2007) Semantic representation of individualized reaction movements for virtual human. Int J Virt Reality 6(1):25–32 83. de Sevin E, Thalmann D (2005) A motivational model of action selection for virtual humans. In: Proceedings of the computer graphics international (CGI2005) conference. IEEE Computer Society Press, New York, pp 213–220 84. Tyrrell T (1993) The use of hierarchies for action selection. Adapt Behav 1(4):387–420 85. Donnart JY, Meyer JA (1996) Learning reactive and planning rules in a motivationally autonomous animal. IEEE Trans Syst Man Cybern 26(3):381–395 86. Musse S, Cassol V, Thalmann D (2021) A history of crowd simulation: the past, evolution, and new perspectives. Vis Comput 37(12) 87. Musse S, Thalmann D (2001) A behavioral model for real-time simulation of virtual human crowds. IEEE Trans Vis Comput Graph 7(2):152–164 88. Emering L, Boulic R, Molet T, Thalmann D (2000) Versatile tuning of humanoid agent activity. Comput Graph Forum 19(4):231–242 89. Bouvier E, Guilloteau P (1996) Crowd simulation in immersive space management. In: Eurographics workshop on virtual environments and scientific visualization ’96. Springer, New York, pp 104–110 90. Brogan D, Hodgins J (1994) Robot herds: group behaviors for systems with significant dynamics. In: in Proceedings of artificial life IV, pp 319–324 91. Musse S, Babski C, Capin T, Thalmann D (1998) Crowd modeling in collaborative virtual environments. In: Proceedings of the ACM VRST 98, Taiwan, pp 115–123 92. Ulicny B, Thalmann D (2001) Crowd simulation for interactive virtual environments and VR training systems. In: Proceedings of the eurographics workshop on animation and simulation. Springer, New York, pp 163–170 93. Cremer J, Kearney J, Papelis Y (1995) HCSM: framework for behavior and scenario control in virtual environments. ACM Trans Model Comput Simul 5(3):242–267 94. Boulic R, Becheiraz P, Emering L, Thalmann D (1997) Integration of motion control techniques for virtual human and avatar real-time animation. In: Proceedings of the ACM VRST ’97, pp 111–118 95. Pettre J, de Heras P, Maim J, Yersin B, Laumond J, Thalmann D (2006) Real time navigating crowds: scalable simulation and rendering. Comput Anim Virt Worlds 16:445–456 96. Treuille A, Cooper S, Popovic Z (2006) Continuum crowds. In: Proceedings of the SIGGRAPH 06 conference, pp 1160–1168 97. Morini F, Mam J, Yersin B, Thalmann D (2007) Real-time scalable motion planning for crowds. In: Proceedings of the CyberWorlds 2007 conference, pp 144–151 98. Ondřej J, Pettré AH, Olivier SD (2010) A synthetic-vision based steering approach for crowd simulation. ACM Trans Graph 99. Dutra T, Priem G, Cavalcante-Neto J, Vidal C, Pettre J (2014) Synthetic vision-based crowd simulation: reactive vs. reactive planning approaches. In: Proceedings of the 27th conference on computer animation and social agents, pp 1–4 100. Oğuz O, Akaydın A, Yılmaz T, Güdükbay U (2010) Emergency crowd simulation for outdoor environments. Comput Graph 34(2):136–144 101. Charalambous P, Chrysanthou Y (2010) Learning crowd steering behaviors from examples. In: Boulic R, Chrysanthou Y, Komura T (eds) Motion in games. Springer, Berlin/ Heidelberg, p 35 102. Flagg M, Rehg JM (2013) Video-based crowd synthesis. IEEE Trans Visual Comput Graph 19(11):1935–1947

122

4

Virtual Characters

103. Zhao M, Turner SJ, Cai W (2013) A data-driven crowd simulation model based on clustering and classification. In: 2013 IEEE/ACM 17th international symposium on distributed simulation and real time applications, DS-RT ’13. IEEE Computer Society, New York, pp 125–134 104. Ju E, Choi MG, Park M, Lee J, Lee KH, Takahashi S (2010) Morphable crowds. ACM Trans Graph 29:6 105. Bicho AL, Rodrigues RA, Musse SR, Jung CR, Paravisi M, Magalh LP (2012) Simulating crowds based on a space colonization algorithm. Comput Graph 36(2):70–79 106. Durupinar F, Pelechano N, Allbeck J, Güdükbay U, Badler NI (2011) How the ocean personality model affects the perception of crowds. IEEE Comput Graph Appl 31(3):22–31 107. Okaya M, Takahashi T (2011) Human relationship modeling in agent-based crowd evacuation simulation. In: Kinny D, Hsu JYJ, Governatori G, Ghose AK (eds) Agents in principle, agents in practice. Springer, Berlin/Heidelberg 108. Li W, Allbeck JM (2011) Populations with purpose. In: Allbeck JM, Faloutsos P (eds) Motion in games. Springer, Berlin/Heidelberg, pp 132–143 109. Pelechano N, Allbeck JM, Badler NI (2007) Controlling individual agents in high-density crowd simulation. In: Proceedings of the 2007 ACM SIGGRAPH/eurographics symposium on computer animation, SCA ’07. Eurographics Association, Goslar, DEU, pp 99–108 110. Musse SR, Cassol VJ, Jung CR (2012) Towards a quantitative approach for comparing crowds. Comput Anim Virt Worlds 23(1):49–57 111. Wolinski D, Guy JS, Olivier AH, Lin M, Manocha D, Pettre J (2014) Parameter estimation and comparative evaluation of crowd simulations. Comput Graph Forum 33(2):303–312 112. Shoulson A, Marshak N, Kapadia M, Badler NI (2013) Adapt: the agent development and prototyping testbed. In: Proceedings of the ACM SIGGRAPH symposium on interactive 3D graphics and games, I3D ’13. Association for Computing Machinery, New York, pp 9–18 113. Berseth G, Kapadia M, Faloutsos P (2013) Steerplex: estimating scenario complexity for simulated crowds. In: International conference of motion on games. ACM, New York, pp 45: 67–45:76 114. Berseth G, Kapadia M, Haworth, Faloutsos P (2015) Steerfit: automated parameter fitting for steering algorithms. In: Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation, SCA ’14. Eurographics Association, Goslar, DEU, pp 113–122 115. Bera A, Kim S, Manocha D (2015) Efficient trajectory extraction and parameter learning for data-driven crowd simulation. In: Proceedings of graphics interface 2015, GI 2015. Canadian Human-Computer Communications Society, Toronto, pp 65–72 116. Başak AE, Güdükbay U, Durupınar F (2018) Using real life incidents for creating realistic virtual crowds with data-driven emotion contagion. Comput Graph 72:70–81 117. Liu W, Pavlovic V, Faloutsos KHP, Yoon S, Kapadia M (2017) Characterizing the relationship between environment layout and crowd movement using machine learning. In: Proceedings of the tenth international conference on motion in games, MIG ’17. Association for Computing Machinery, New York 118. Ravichandran NB, Yang F, Peters C, Lansner A, Herman P (2018) Pedestrian simulation as multi-objective reinforcement learning. In: Proceedings of the 18th international conference on intelligent virtual agents, IVA ’18. Association for Computing Machinery, New York, pp 307–312 119. Amirian J, van Toll W, Hayet JB, Pettré J (2019) Data-driven crowd simulation with generative adversarial networks. In: Proceedings of the 32nd International Conference on Computer Animation and Social Agents, CASA ’19. Association for Computing Machinery, New York, pp 7–10 120. Testa E, Barros RC, Musse S (2019) Crowdest: a method for estimating (and not simulating) crowd evacuation parameters in generic environments. Vis Comput 35:1119–1130 121. Cassol VJ, Musse SR, Jung CR, Badler NI (2017) Simulating crowds in egress scenarios. Springer, London 122. Wong S, Wang YS, Tang PK, Tsai T (2017) Optimized evacuation route based on crowd simulation. Comput Visual Media 3:243–261

References

123

123. Dal Bianco CM, Braun A, Brasil J, Musse SR (2017) Preserving the motion features in non avoiding collision crowds. Comput Entertain 15(3):1–15 124. Mathew CDT, Knob PR, Musse SR, Aliaga DG (2019) Urban walkability design using virtual population simulation. Comput Graph Forum 38(1):455–469 125. Ren J, Xiang W, Xiao Y, Yang R, Manocha D, Jin X (2019) Heter-Sim: heterogeneous multiagent systems simulation by interactive data-driven optimization. IEEE Trans Vis Comput Graph:1 126. Ho RC, Wong SK, Chou YH, Lin GW, Tsai TY, Huang PH, Wang YS, Yeh IC (2016) An authoring framework for time dependent crowd simulation. In: SIGGRAPH ASIA 2016 posters, SA’16. Association for Computing Machinery, New York 127. Stüvel SA, Magnenat Thalmann N, Thalmann D, Stappen AF, Egges A (2017) Torso crowds. IEEE Trans Visual Comput Graph 23(7):1823–1837 128. Dickinson P, Gerling K, Hicks K, Murray J, Shearer J, Greenwood J (2019) Virtual reality crowd simulation: effects of agent density on user experience and behaviour. Virtual Reality 129. Hürst W, Geraerts R (2019) Augmented and virtual reality interfaces for crowd simulation software-a position statement for research on use-case-dependent interaction. In: 2019 IEEE virtual humans and crowds for immersive environments (VHCIE), pp 1–3 130. Moussaïd M, Kapadia M, Thrash T, Sumner RW, Gross M, Helbing D, Hölscher C (2016) Crowd behaviour during high-stress evacuations in an immersive virtual environment. J R Soc Interf 131. Kyriakou M, Chrysanthou Y (2018) How responsiveness, group membership and gender affect the feeling of presence in immersive virtual environments populated with virtual crowds. In: Proceedings of the 11th annual international conference on motion, interaction, and games, MIG ’18. Association for Computing Machinery, New York 132. Pelechano N, Allbeck JM (2016) Feeling crowded yet? Crowd simulations for VR. In: 2016 IEEE virtual humans and crowds for immersive environments (VHCIE), pp 17–21 133. Yang F, Shabo J, Qureshi A, Peters C (2018) Do you see groups? The impact of crowd density and viewpoint on the perception of groups. In: Proceedings of the 18th international conference on intelligent virtual agents, IVA ’18. Association for Computing Machinery, New York, pp 313–318 134. Latoschik ME, Kern F, Stauffert JP, Bartl A, Botsch M, Lugrin JL (2019) Not alone here? Scalability and user experience of embodied ambient crowds in distributed social virtual reality. IEEE Trans Visual Comput Graph 25(5):2134–2144 135. Durupınar F, Güdükbay U, Aman A, Badler NI (2016) Psychological parameters for crowd simulation: from audiences to mobs. IEEE Trans Visual Comput Graph 22(9):2145–2159 136. Favaretto R, Dihl L, Barreto R, Musse SR (2016) Using group behaviors to detect hofstede cultural dimensions. In: IEEE international conference on image processing (ICIP) 137. Dihl L, Testa ES, Knob P, da Silva GLB, Favaretto RM, de Alcântara MF, Musse SR (2017) Generating cultural characters based on hofstede dimensions. In: 2017 IEEE virtual humans and crowds for immersive environments (VHCIE), pp 1–5 138. Zhang P, Liu H, Ding Y (2015) Crowd simulation based on constrained and controlled group formation. Vis Comput 31(1):1432–2315 139. Bosse T, Hoogendoorn M, Klein MCA, Treur J, van der Wal CN (2011) Agent-based analysis of patterns in crowd behaviour involving contagion of mental states. In: Mehrotra KG, Mohan CK, Oh JC, Varshney PK, Ali M (eds) Modern approaches in applied intelligence: 24th international conference on industrial engineering and other applications of applied intelligent systems, IEA/AIE 2011, Syracuse, June 28–July 1, 2011, Proceedings, Part II, p. 677. Springer, Berlin/Heidelberg 140. Borges Fortes Neto A, Pelachaud C, Musse SR (2017) Giving emotional contagion ability to virtual agents in crowds. In: Beskow J, Peters C, Castellano G, O’Sullivan C, Leite I, Kopp S (eds) Intelligent virtual agents. Springer, Cham, pp 63–72 141. Başak A, Gudukbay U, Durupinar F (2018) Using real life incidents for realistic virtual crowds with data-driven emotion contagion. Comput Graph

124

4

Virtual Characters

142. Huang W, Terzopoulos D (2020) Door and doorway etiquette for virtual humans. IEEE Trans Visual Comput Graph 26(3):1502–1517 143. Kremer M, Haworth B, Kapadia M, Faloutsos P (2021) Modelling distracted agents in crowd simulations. Vis Comput 37(1):1432–2315 144. Schaffer D, Neto AB, Antonitsch A, Musse SR (2020) Towards animating virtual humans in flooded environments. In: ACM SIGGRAPH conference – motion in games, p 1 145. Usman M, Lee TC, Moghe R, Zhang X, Faloutsos P, Kapadia M (2020) A social distancing index: EVALUATING navigational policies on human proximity using crowd simulations. In: ACM SIGGRAPH conference – motion in games, p 1 146. Volonte M, Hsu Y, Liu K, Mazer JP, Wong S, Babu SV (2020) Effects of interacting with a crowd of emotional virtual humans on users’ affective and non-verbal behaviors. In: 2020 IEEE conference on virtual reality and 3D user interfaces (VR), pp 293–302 147. Koilias A, Mousas C, Anagnostopoulos CN (2020) I feel a moving crowd surrounds me: exploring tactile feedback during immersive walking in a virtual crowd. Comput Anim Virt Worlds 31(4–5):1963. https://doi.org/10.1002/cav.1963 148. Tripathi G, Singh K, Vishwakarma DK (2020) Crowd emotion analysis using 2d convents. In: 2020 third international conference on smart systems and inventive technology (ICSSIT), pp 969–974 149. Kielar P, Borrmann A (2020) An artificial neural network framework for pedestrian walking behavior modeling and simulation. Collect Dynam 5:290–298 150. Bansod S, Nandedkar A (2020) Crowd anomaly detection and localization using histogram of magnitude and momentum. Vis Comput 36:609–620 151. Sun J, Zhang J, Li Q, Yi X, Liang Y, Zheng Y (2020) Predicting citywide crowd flows in irregular regions using multi-view graph convolutional networks. IEEE Trans Knowl Data Eng 152. Zhao R, Hu Q, Liu Q, Li C, Dong D, Ma Y (2020) Panic propagation dynamics of high-density crowd based on information entropy and Aw-Rascle model. IEEE Trans Intell Trans Syst 21(10):4425–4434 153. Huang ZM, Chen WN, Li Q, Luo XN, Yuan HQ, Zhang J (2020) Ant colony evacuation planner: an ant colony system with incremental flow assignment for multipath crowd evacuation. IEEE Trans Cybern:1–14 154. Mirahadi F, McCabe BY (2021) Evacusafe: a real-time model for building evacuation based on Dijkstra’s algorithm. J Build Eng 155. Dijkstra EW (1959) A note on two problems in connexion with graphs. Numer Math 1(1): 269–271 156. Xie W, Lee EWM, Li T, Shi M, Cao R, Zhang Y (2021) A study of group effects in pedestrian crowd evacuation: experiments, modelling and simulation. Saf Sci 133:105029 157. Karamouzas I, Overmars M (2012) Simulating and evaluating the local behavior of small pedestrian groups. IEEE Trans Visual Comput Graph 18(3):394–406 158. Moussaïd M, Perozo N, Garnier S, Helbing D, Theraulaz G (2010) The walking behaviour of pedestrian social groups and its impact on crowd dynamics. PLoS One 5(4):1–7 159. Zhang Z, Jia L (2021) Optimal guidance strategy for crowd evacuation with multiple exits: a hybrid multiscale modeling approach. Appl Math Model 90:488–504 160. Helbing D, Molnar P (1995) Social force model for pedestrian dynamics. Phys Rev E 51: 42–82 161. Charrier R, Philippe J, Bertelle C (2021) Study of a model of nervousness propagation within virtual crowds. In: Prendinger H, Lester J, Ishizuka M (eds) Complex systems, smart territories and mobility. Springer, Cham, pp 187–200 162. Shi X, Xue S, Feliciani C, Shiwakoti N, Lin J, Li D, Ye Z (2021) Verifying the applicability of a pedestrian simulation model to reproduce the effect of exit design on egress flow under normal and emergency conditions. Physica A Stat Mech Application 562:125–347 163. Hernández-Orallo E, Armero-Martánez A (2021) How human mobility models can help to deal with covid-19. Electronics 10(1):33

Chapter 5

Architecture of Virtual Reality Systems and Metaverse

This chapter is dedicated to the foundations of virtual reality systems, emphasizing the distributed aspect. Communication, concurrency, and data sharing are analyzed. The second part of the chapter emphasizes the online worlds and the metaverse, including avatars and the role of Internet of Things. The creation of VR systems to support virtual environments (VE) is a challenging problem requiring diverse areas of expertise, ranging from networks to psychology. Developing VEs is a very expensive task in terms of time and financial and human resources. VEs can be applied in a broad range of areas, such as scientific visualization, socializing, training, psychological therapy, and gaming (for more details, see the “Applications” part of this book). Such a diversity of applications produces a set of requirements that make it very difficult, if not impossible, to build a single system that fits all needs. The result has been the creation of monolithic systems that are highly optimized to a particular application, with very limited reusability of components for other purposes. According to Oliveira et al. [1], the problem of lack of reusability is due to the current trend in the VE community: developing a new VE system for each different application. The “reinventing the wheel” and “not invented here” syndromes limit the innovation and delay the use of VEs in wider areas for the public. Monolithic systems such as DIVE [2], MASSIVE [3], NPSNET [4], SPLINE [5], and dVS/dVISE [6], among others, proliferated in the past due to the lack of system flexibility for a particular application [1]. The introduction of more modular architectures led to the emergence of toolkits such as WorldToolkit [7], Avocado [8], VR Juggler [9], and VHD++ [10]. These software suites have different degrees of flexibility. Frameworks like VHD++ differ from others due to its specialized skills in a particular domain, such as virtual humans’ simulation technologies. All of them are based on a hierarchical representation of the virtual environment: a scene graph.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_5

125

126

5.1

5

Architecture of Virtual Reality Systems and Metaverse

Scene Graph–Based Systems

A scene graph is an abstract logical access structure, a “tree.” It is used to represent objects comprising the environment (scene data) and the relationships between them. The scene graph consists of parent nodes, child nodes, and data objects. The parent nodes, also called group nodes, organize and, in some cases, control how to interpret their descendants. Group nodes serve as the glue that holds a scene graph together. Child nodes can be either group nodes or leaf nodes. Leaf nodes have no children. They encode the core semantic elements of a scene graph, for example, what to draw (geometry), what to play (audio), how to illuminate objects (lights), or what code to execute (behavior). Leaf nodes refer to data objects. These objects are not scene graph nodes, but they contain the data that leaf nodes require, such as the geometry to draw or the sound sample to play. Scene graphs should not be confused with data structures used to do visibility culling or collision queries, such as octrees, BSP, ABT, KdT. Scene graphs are used to connect game rules, physics, animation, and AI systems to the graphics engine. Popular implementations of scene graph programming interfaces (APIs) include: Cosmo3D (SGI), Vega Scene Graph (http://www.presagis.com), Java3D, OpenSceneGraph (http://www.openscenegraph.org), and OpenGL Performer (https://en.wikipedia.org/wiki/OpenGL_Performer). All of them were designed for creating real-time visual simulations and other performance-oriented 3D graphics applications. OpenGL Performer (https://en.wikipedia.org/wiki/OpenGL_Performer) is a commercial toolkit that evolved from Open Inventor, which is currently an open-source project.5 Open Inventor is considered the archetypical example of a scene graph library. It presents an object-oriented programming model based on a 3D scene database (scene graph) that provides a higher layer of programming for OpenGL. Java 3D gained popularity as the main scene graph-based API for developing 3D applications with Java. It is frequently used for developing web-based applications enhanced with real-time 3D graphics [11–13]. It is also a very representative example of a scene graph-based 3D toolkit. Scene graph implementations are based on a hierarchical spatial representation of the objects in the scene, for example, a terrain contains a house, and inside the house there is a person holding a hammer in her right hand. Usually, the semantic information that is encoded in the scene graph corresponds mainly to visualization aspects, geometry to draw, and associated effects, such as a sound sample. Only elemental relationships between objects can be specified, for example, smart objects [14] (simple tools and objects such as a hammer or a drawer) that contain information describing how they can be grasped by a virtual human and a predetermined behavior to perform (see Sect. 4.4.1). Efforts aimed at enhancing the adaptability and reusability of VE applications and entities within them have focused on designing software component frameworks to manage the resources and building blocks of a VE system. Such is the case of the Java Adaptive Dynamic Environment (JADE) [1], which permits dynamic runtime

5.2

Semantic Virtual Environments

127

management of all components and resources of a VE system. While this kind of initiative is successful in terms of providing reusability and interoperability at the level of the code source, they do not address the fundamental problem of reusing the virtual entities that participate in the VE application. The use of scene graphs as hierarchical spatial representations is not questioned. As a result, source code implementing animation, visualization, and interaction algorithms can be reused to some extent. But the knowledge associated with a virtual entity remains difficult to reuse. In [15] Gutiérrez proposed the “semantic virtual environments” approach, which considers virtual entities as complex items with different types of data and knowledge associated with them.

5.2

Semantic Virtual Environments

The main idea behind semantic virtual environments (SVE) [15] is that a higherlevel semantic representation of virtual environments can enhance the reusability and adaptation capabilities of the virtual entities participating in a VE. The semantics-based representation that is proposed builds on the concept of a scene graph, with the difference that it does not focus on visual or spatial relationships between entities but on higher-level semantics. The SVE approach is based on the semantics of the virtual entities and not on their geometry, as is the case in most of the VE applications. A semantic model provides a way to specify alternative geometric representations for each entity and a range of functionalities. The entities are called digital items and can be not only virtual objects to be rendered as part of the scene but also independent user interfaces and controls for animation or interaction. A semantic model of a virtual environment can be defined keeping in mind that the main attribute of a virtual entity (objects or characters) is its 3D shape. Virtual environments are geometry-based applications. However, depending on the context, the shape of the entity can change. Moreover, there are contexts in which multiple shapes representing the same virtual entity must be manipulated and synchronized. For instance, when displaying a virtual character on a large projection screen, we require information for rendering a high-resolution (hi-res) 3D shape. Nevertheless, the hi-res shape could be controlled by the user interactions performed through a different representation, either a simplified shape or an abstract manipulator: a menu, a slider, or any other GUI control. Using independent representations for each case is costly in terms of data synchronization. The semantic model encapsulates the set of geometric and/or abstract representations belonging to each virtual entity and associated descriptors to inform the rendering and interface systems about how to handle each object. Figure 5.1 shows a general class diagram of the semantic model for the VE and the tools that can be used to control and interact with it.

128

5

Architecture of Virtual Reality Systems and Metaverse

Scene

1..1

0..* Semantic Descriptor 1..1

Controller

1..*

1..1

Digital Item

0..*

0..*

0..1

0..* 1..1

0..*

0..1

Geometric Descriptor

1..1 0..* Shape

Fig. 5.1 Semantic representation of an interactive virtual environment

The scene is the main container of the VE model; it has references to the digital items contained in the VE. It is the data repository used by digital items and other software tools such as viewers or interaction devices. A semantic descriptor provides human- and machine-readable information about a particular digital item (we also call them virtual entities). They are the entry points for the scene controller to choose the correct geometry and interface to present. The semantic descriptor is the placeholder for any information describing how the digital item is to be used and how it is related to other items in the scene. The geometric descriptor of a digital item specifies the type of shape associated with the entity: a deformable mesh, an articulated body (joints and segments), and the like. For instance, hierarchical structures for skeleton-based animation can be defined by the geometric descriptors, such as H-Anim (https://www.web3d.org/ working-groups/humanoid-animation-hanim), the standard specification of a hierarchical structure for humanoid character animation. Moreover, alternative skeletons with different levels of detail on the number of joints and/or the complexity of the segments (3D shapes associated with each joint) can be specified as well. Handling alternative geometric representations for each virtual entity in the environment is not enough to provide a complete representation of a virtual world. The model reflects also the relations between entities. The semantic descriptors characterize each object or character in the scene and constitute a scene graph that can be used for both rendering and extracting the underlying information about its

5.3

Generic System Architecture for VR Systems

129

contents. Digital items can contain other items or be related to each other in different ways, for example, entities that move together or trigger events on other entities; the model is general enough to express a variety of relations. A digital item can be controlled in a variety of ways: interactive controls (requiring user input), autonomous animation controllers (implementation of algorithms to synthesize some kind of behavior or animation), etc. A controller specifies the interaction possibilities: predefined object behaviors that react to human intervention and the way to provide access (interfaces) to them. For instance, an interactive control can describe the animation to open and close a door and expose the parameters controlling this behavior. This information can be used to implement a variety of interfaces to access the same parameter: 3D manipulators or 2D GUI controls. Usually, digital items in a VE are animated in either an autonomous or an event-triggered way. The virtual entities contained in a scene can be controlled and displayed in several ways. Different viewers and interfaces can be used depending on the devices available and the information associated with each virtual object or character. For example, a single virtual character can be rendered as a hi-res 3D model in an OpenGL-based viewer, or a simplified version can be displayed in a PDA screen, provided that the character’s descriptor contains alternative representations associated with it. The character can be controlled through a 3D manipulator or a 2D GUI control. All the previous cited information can be stored in the corresponding semantic descriptor. This model has been successfully applied in the development of VR applications with adaptive multimodal interfaces. The work presented in [16] uses a graphic interface to design real-time multimodal interfaces to interact within a virtual world. For example, virtual characters can be controlled either through an optical gesture recognition system, which detects face gestures, or by means of a PDA, in a similar way as with the “mobile animator” [17] (see Sect. 8.2.3).

5.3

Generic System Architecture for VR Systems

In addition to the SVE approach explained earlier, other researchers and developers have designed system architectures for VR systems. The main goal is to maximize the reusability of software and data while maximizing the variety of VR applications that can be developed. Steinicke et al. [18] have proposed a generic architecture for VR systems composed of three main layers: application, graphics, and rendering. The application layer is dedicated to create the VR user interface. Various GUI (graphical user interface) toolkits can be used for this purpose, such as Qt (https:// www.qt.io/), Gtk (http://www.gtk.org), wxWidgets (http://www.wxwidgets.org). The graphics layer deals with 3D models (graphics objects) managed through a scene graph and controlled by behavior graphs, which can describe event-dependent and time-dependent behavior in the virtual scene. Both graphs are contained in a

130

5 Architecture of Virtual Reality Systems and Metaverse

canvas, which can be the equivalent of the scene container of the SVE model (Sect. 5.2). The rendering layer allows us to plug adapters for different rendering technologies, such as OpenGL, global illumination software (see Sect. 2.3.2) like RenderMan (https://renderman.pixar.com), or POVRay (http://www.povray.org). An additional component in the architecture is used to control VR devices such as trackers or data gloves. In many cases, those devices are controlled by their own separate computer. In this case, data exchange between the device controller and the main application is usually done through the network (e.g., using UDP or TCP sockets). Network connections also serve to share data describing the virtual environment among several users, making it possible to create distributed and multiuser virtual environments.

5.4

Distributed Virtual Environments

This section is based on the definitions provided by Lee et al. in [19] and [20]. A distributed virtual environment (DVE) is a software system that allows users in a network to interact with each other by sharing a common view of their states. As users are geographically distributed over large networks like the Internet and the number of users increases, scalability is a key aspect to consider for real-time interaction. In [21], Molet et al. present an application of a distributed virtual environment called Cybertennis (see Fig. 5.2). This was done for the Telecom conference in Geneva in 1997, to illustrate the concept of DVE. Two users, one in Lausanne and one in Geneva (40 miles away), were playing tennis together on a virtual moon, and the virtual clone of Marilyn Monroe was acting as referee. The approaches for improving scalability of DVE can be classified into the following categories: communication architecture, interest management, concurrency control, data replication, and load distribution.

5.4.1

Communication Architecture

Depending on how the communication is coordinated, the communication architecture can be characterized as follows: client-server, peer-to-peer, or peer-to-server. In the client-server model, all messages are sent to a server and then the server distributes them to all or some of the users according to synchronization requirements. As the number of participants in a virtual world increase, the server becomes a bottleneck. Even if additional servers are used, the delay due to additional communication overhead in servers is inevitable.

5.4

Distributed Virtual Environments

131

Fig. 5.2 Cybertennis in Telecom Interactive 97, by MIRALab, University of Geneva and EPFL VRlab [21]

The peer-to-peer model allows users to directly exchange messages. However, each user has to assume all the responsibility of message filtering and synchronization. The peer-to-server model exploits the benefits of the two other models. Consistency management is done by a server and communication among users is performed using multicast.

5.4.2

Interest Management

Computing power and rendering speed are rapidly increasing; however, network resources remain very expensive compared with computational resources. To overcome the limitations of the network resources, various relevance-filtering mechanisms are used. Interest-management strategies exploit the fact that users do not need to receive all update messages related to the whole world. Instead, they receive only messages in which they are interested. Two methods can be used, depending on the fidelity of the message filtering: dividing the virtual world into several regions

132

5

Architecture of Virtual Reality Systems and Metaverse B’s nimbus

A’s focus

A’s awareness of B A

B

Fig. 5.3 The aura nimbus approach for interest management in collaborative worlds

(usually done offline), or localizing the area of interest of the participants in real time. As proposed by Benford et al. [22], the awareness level between two objects A and B can be defined by four strength values: A’s focus on B, A’s nimbus to B, B’s focus on A and B’s nimbus to A. If A’s and B’s aurae don’t have common components (i.e., there is no adequate aura intersection), these values are null. Focus describes the observers’ allocation of attention. The concept of nimbus describes the observed object’s manifestation or observability. The observer’s awareness of the observed is then some combination of the observed’s nimbus and the observer’s focus. Aura is defined as the subspace that effectively bounds the presence of an object within a given medium. Aura, focus, and nimbus may all be multivalued functions, may have arbitrary extents and shapes, and are medium specific. Figure 5.3 illustrates these concepts.

5.4.3

Concurrency Control

Shared information in DVEs is often replicated at each user’s site to provide acceptable interactive performance, especially where users are geographically distributed over large networks like the Internet. Replication enables users to locally access and update data. On the other hand, the cost of replication is to maintain synchronization among replicas in the presence of multiple concurrent updates, which eventually leads to inconsistent views among users. As communication delay increases, the probability of conflicts between operations does as much. Therefore, concurrency control is required to maintain synchronization among replicas. Approaches to concurrency control have been broadly categorized into pessimistic, optimistic, and prediction schemes.

5.4

Distributed Virtual Environments

133

The pessimistic scheme blocks users until a lock request for an object is granted and then allows them to manipulate the object. It is simple and guarantees absolute consistency. However, interactive performance can be deteriorated by high communication delays due to an increased number of users. The optimistic scheme allows users to update objects without conflict checks with others and thus to interact with objects more naturally. However, when conflicts occur, a repair must be done. This increases the system complexity and produces strange effects when undoing and redoing previous actions on user interfaces. When using the prediction-based concurrency control, the owner of an object predicts the next owner before users request ownership of the object. The prediction is calculated on the basis of the position, orientation, and navigation speed of the users.

5.4.4

Data Replication

In order to support real-time interaction, it is common to replicate virtual world data from the server at the client. Each client then updates its replicated data by local changes or notification of remote changes. The initial download delay increases proportionally to the size of the virtual world. To reduce the transmission overhead, several on-demand transmission (partial- replication) techniques are proposed. Instead of downloading all the virtual world objects, only the required objects are copied. Other approaches use compression of geometry and image data. A key aspect in partial replication is how to efficiently replicate the required data, to minimize the artifacts (incomplete scenes) due to unavailable data. Two schemes are used together: prioritized transfer of objects and caching and prefetching techniques. The prioritized transfer of objects filters objects within a user’s viewing range and transmits only the objects that provide high fidelity to the user using level of details (LODs) or multiresolution techniques. This maximizes the graphical fidelity of world objects as well as interactive performance by mediating the graphical detail and the transmission overhead of the objects. Caching and prefetching make the demanded data immediately available in a timely manner by exploiting the locality of data. This approach exploits the spatial relationship based on the distance between a user and objects. It is based on the observation that the nearer the object lies to the user, the more probability it must be accessed again.

5.4.5

Load Distribution

Another approach for enhancing scalability is the multiple-server architecture, adopted in several DVE systems and in many commercial multiplayer network

134

5 Architecture of Virtual Reality Systems and Metaverse

games such as Second Life (see Sect. 5.5.2) or Lineage II (www.lineage2.com). Partitioning a virtual world into multiple regions and distributing the responsibilities for managing the regions across multiple servers can significantly reduce the workloads of individual servers. This enables a system to support more concurrent users and a larger virtual environment. However, nonuniform distribution of users over the virtual environment would incur a workload imbalance among servers. Some servers would handle more crowded regions than others and suffer from heavy workload and produce low interactive performance. To avoid such degradation of interactive performance, dynamic load distribution schemes have been introduced: overloaded servers transfer their excessive workloads to less-loaded ones. There are three main approaches: local, global, and adaptive dynamic load distribution. In the local approach, an overloaded server performs load distribution locally with only its neighboring servers, whose regions are adjacent to that of the overloaded server. The local approach incurs less overhead than the global approach because an overloaded server requires the information only on its neighboring servers and incurs a small amount of user migrations. However, in the case of a highly skewed workload imbalance (most overloaded servers are neighbored with each other rather than dispersed), an overloaded server cannot give its excessive workload to its neighboring servers because they are also heavily loaded. The global approach can balance the workloads evenly even in a highly skewed case because all the servers perform load distribution together with the global information. A central coordinator repartitions the whole virtual environment using a graph-partitioning algorithm. However, the repartitioning overhead and the required amount of user migrations increase sharply as do the size of a virtual environment and the number of servers in the system. In the adaptive approach, an overloaded server balances its workload with a set of servers (beyond its neighboring servers) according to their workload status.

5.5 5.5.1

Metaverse and Online Virtual Worlds The Concept of Avatar

Computer avatars are graphical representations of a user or their persona. In Internet forums and other online communities, avatars are sometimes referred to as profile pictures, userpics, or formerly as picons (personal icons). They are derived from stickers used in messaging platforms. Stickers are illustrations of characters representing emotions or actions that are a mix of cartoons and Japanese smileylike “emojis” sent through instant messaging platforms. Emojis were preceded by emoticons, a concept developed in 1982 by computer scientist Scott Fahlman who

5.5

Metaverse and Online Virtual Worlds

135

Fig. 5.4 A colored Emoji from Twitter Emoji project. (Creative Commons Attribution 4.0)

Fig. 5.5 Happy Bitmoji. (Creative Commons Attribution-Share Alike 4.0)

suggested text-based symbols like :-) and :-( could replace language. Figure 5.4 shows an example of emoji. Bitstrips, a media and technology company based in Toronto, Canada, launched Bitmoji in 2014, an app that allows users to create personalized stickers for messaging apps as shown in Fig. 5.5. After Snap, Inc. acquired Bitstrips, the comic service was shut down, but Bitmoji remains operational and has been given greater prominence within Snapchat’s overall platform. Facebook introduced Facebook Avatars in 2016, saying it wanted to give users a better way to express themselves. These characters are like Bitmoji characters, and they can be created with apps for Android and iOS. They were described as: “Avatars enable you to share a range of emotions and expressions via a digital persona that’s uniquely representative of you, so we’re excited to bring this new form of self-expression to more people around the world. With so many emotions and expressions to choose from, avatars let you to react and engage more authentically with family and friends across the app.” Figure 5.6 shows examples of Facebook Avatars. An avatar can also be a three-dimensional model, as in online worlds (see Sect. 5.5.2) and video games. The appearance of a 3D avatar is very similar to that of a virtual character, as described in Chap. 4. However, its animation is either very limited or completely controlled by the user. Figure 5.7 shows a 3D Avatar used in Second Life.

136

5

Architecture of Virtual Reality Systems and Metaverse

Fig. 5.6 Facebook Avatars

Fig. 5.7 Wouldn’t It Be Nice. (Created by Alicia Chenaux, licensed under the Creative Commons Attribution-Share Alike 2.0 Generic)

5.5.2

Online Worlds

As already mentioned in Sect. 1.5, metaverse is essentially the combination of virtual worlds and social media. At this stage, we can imagine a virtual world where billions of people live, work, shop, learn, and interact with each other, but today, the current implementations of metaverses are online virtual worlds. One of the first online virtual world was Palace [23], a computer program to access graphical chat room servers, called palaces, in which users may interact with one another using 2D graphical avatars overlaid on a graphical backdrop (see Fig. 5.8). The software concept was originally created by Jim Bumgardner and produced by Time Warner in 1994 and was first opened to the public in November 1995. Another precursor of

5.5

Metaverse and Online Virtual Worlds

137

Fig. 5.8 The Palace chat early marketing screenshot. (Wikipedia, fair use)

Fig. 5.9 A performance at “LA Club 2020 & Friends” in Second Life. (Licensed under the Creative Commons Attribution 2.0 Generic)

metaverse is ActiveWorlds [24], which allows users to own worlds and universes and develop custom 3D content. Typically, access points for the metaverse include general-purpose computers and smartphones, augmented reality, mixed reality, and virtual reality. In 2003, the San Francisco-based firm Linden Lab launched Second Life [25], an online multimedia platform that allows people to create an avatar for themselves and then interact with other users and user-created content within a multiplayer online virtual world. Figure 5.9 shows an example of Second Life session. In 2019, the social network company Facebook launched a social VR world called Facebook Horizon [26]. In 2021, Facebook was renamed “Meta Platforms”

138

5 Architecture of Virtual Reality Systems and Metaverse

and its chairman, Mark Zuckerberg, declared a company commitment to developing a metaverse.

5.6

Metaverse with the Internet of Things (IoT)

The Internet of things (IoT) [27] describes physical objects (or groups of such objects) with sensors, processing ability, software and other technologies that connect and exchange data with other devices and systems over the Internet or other communications networks. IoT will allow virtual spaces to interact and access the real world seamlessly whereas Metaverse technology will offer the necessary 3D user interface for the IoT device cluster [28]. This will provide a user centered IoT and metaverse experience to the users. This means that IoT’s main objective will be to provide the necessary link between the outside world and the virtual world. IoT is the internet of 3D physical things. The metaverse adds a 3D user interface providing a more tailored user interface environment for IoT. As such, the metaverse brings the physical and digital worlds together, allowing people and things to collaborate more intuitively with complex situations. The metaverse, as a 3D interface for IoT, will make the physical and digital indistinguishable and therefore augment our human ability to make better-informed decisions. Training on real-world extreme situations can be done through virtual simulations using digital twins in the metaverse, allowing us to develop and test training approaches in situations that we cannot do in the real world. A digital twin is a virtual representation of a real-world physical system or process (a physical twin) that serves as the indistinguishable digital counterpart of it for practical purposes, such as system simulation, integration, testing, monitoring, and maintenance. Digital twins can be vehicles, buildings, factories, or people. The most complex digital twin is the digital human or virtual human. A digital twin may be used in real time and regularly synchronized with the corresponding physical system. As metaverses evolve to reflect reality, training simulations will help prepare people and AI/ software to work together to better detect problems and mitigate impact more closely in real life. Complex virtual simulation will allow us to run various long-term planning scenarios and determine the most optimized designs for our energy, transportation, and healthcare systems and to dynamically operate these approaches as the real-world changes. Enabled by AR and IoT, the metaverse will establish an entirely new precedent for work and collaboration, allowing enterprises to operate at much greater speed and scale. The central part of perception of the elements in the environment is essential for the second generation of virtual humans, as it gives them an awareness of what is changing and what is static. A virtual human must continuously update its position and activities, which, in turn, influences its perceptions. Therefore, sensorial information and IoT drastically influence virtual human behavior. Virtual humans should have sensors that simulate the functionality of their organic counterparts, mainly for vision, audition, and tactile sensation. Virtual senses convey decisions about the

References

139

limb motion allowing not only to avoid obstacles, but also to walk on uneven terrains and climb stairs or even jump in case of narrow obstacles like little rivers or rocks. To achieve a realistic sense of autonomy – a key human attribute, virtual humans should have an internal representation of self-comprising drives, motivation, personality and emotion. Such internal states can then drive the behavior of virtual humans in response to the events and interact in a dynamic real-time environment. The major challenge here is to develop a coherent computational model that simulates these internal characteristics and their impact on the reasoning process and the resultant behavior of virtual humans. Empowering virtual humans to better interact with us and offer us quality-based services is the ultimate goal of metaverse. It is the ultimate goal to empower our capacity to learn, to get help if we have disabilities, to prepare us in advance for unexpected events and enable us to answer the question “what will happen if.” IoT is set to transform the metaverse as it seamlessly connects the 3D world to many real-life devices. The real connection between IoT and the metaverse relies on cloud technology. Central to the metaverse’s success will be the seamless interoperability between AR and IoT data, contributing to a shared vision of user-centric multimedia services for modern nomadic people, who has a high level of mobility and is connected to network for most of times. It will extend personal media spaces for modern nomadic life by removing spatial constraints in our daily activities.

References 1. Oliveira M, Crowcroft J, Slater M (2003) An innovative design approach to build virtual environment systems. In: in EGVE ’03: Proceedings of the workshop on virtual environments 2003. ACM Press, New York, pp 143–151 2. Hagsand O (1996) Interactive multiuser VEs in the DIVE system. IEEE Multimedia 3(1):30–39 3. Greenhalgh C, Purbrick J, Snowdon D (2000) InsideMASSIVE-3:flexiblesupport for data consistency and world structuring. In: CVE ’00: Proceedings of the third international conference on collaborative virtual environments. ACM Press, New York, pp 119–127 4. Macedonia MR, Brutzman DP, Zyda MJ, Pratt DR, Barham PT, Falby J, Locke J (1995) NPSNET: a multi-player 3D virtual environment over the Internet. In: SI3D ’95: Proceedings of the 1995 symposium on interactive 3D graphics. ACM Press, New York, pp 93–94 5. Anderson D, Barrus J, Howard J, Rich C, Waters R (1995) Building multiuser interactive multimedia environments at MERL. IEEE Multimedia 2(4):77–82 6. Grimsdale C (1991) dVS-Distributed virtual environment system. In: Proceedings of Computer Graphics’91. Bleinheim Online, London, pp 163–170 7. Sense8 Corporation (1998) World toolkit reference manual – release 7. Mill Valley 8. Tramberend H (1999) Avocado: a distributed virtual reality framework. In: VR ’99: Proceedings of the IEEE virtual reality. IEEE Computer Society, pp 14–21 9. Bierbaum A, Just C, Hartling P, Meinert K, Baker A, Cruz-Neira C (2001) VR Juggler: a virtual platform for virtual reality application development. In: Proceedings of the virtual reality 2001 conference (VR’01). IEEE Computer Society, pp 89–96 10. Ponder M, Papagiannakis G, Molet T, Magnenat-Thalmann N, Thalmann D (2003) VHD++ development framework: towards extendible, component based VR/AR simulation engine featuring advanced virtual character technologies. In: Proceedings of computer graphics international (CGI). IEEE Computer Society Press, pp 96–104

140

5

Architecture of Virtual Reality Systems and Metaverse

11. Huang S, Baimouratov R, Nowinski WL (2004) Building virtual anatomic models using Java3D. In: VRCAI ’04: Proceedings of the 2004 ACM SIGGRAPH international conference on virtual reality continuum and its applications in industry. ACM Press, New York, pp 402–405 12. Dörner R, Grimm P (2000) Three-dimensional Beans–creating Web content using 3D components in a 3D authoring environment. In: VRML ’00: Proceedings of the fifth symposium on virtual reality modeling language (Web3D-VRML). ACM Press, New York, pp 69–74 13. Yimin C, Tao Z, Di W, Yongyi H (2002) A robot simulation, monitoring and control system based on network and Java3D. In: Proceedings of the 4th world congress on intelligent control and automation, pp 139–143 14. Kallmann M, Thalmann D (1999) Direct 3D interaction with smart objects. In: VRST ’99: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 124–130 15. Gutierrez M, Thalmann D, Vexo F (2005) Semantic virtual environments with adaptive multimodal interfaces. In: Proceedings of the 11th international conference on multimedia modelling (MMM2005), pp 277–283 also in Gutierrez M (2005) Semantic virtual environments. PhD thesis, Ecole Polytechnique Fédérale de Lausanne (EPFL) 16. Gutiérrez M, Lemoine P, Thalmann D, Vexo F (2004) Telerehabilitation: controlling haptic virtual environments through handheld interfaces. In: VRST ’04: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 195–200 17. Gutiérrez M, Vexo F, Thalmann D (2004) The mobile animator: interactive character animation in collaborative virtual environments. Proc IEEE VR:27–31 18. Steinicke F, Ropinski T, Hinrichs K (2005) A generic virtual reality software system’s architecture and application. In: ICAT ’05: Proceedings of the 2005 international conference on augmented tele-existence. ACM Press, New York, pp 220–227 19. Lee D, Lim M, Han S, Lee K (2007) ATLAS: a scalable network framework for distributed virtual environments. Presence Teleop Virt Environ 16(2):125–156 20. Lee D, Lim M, Han S (2002) ATLAS: a scalable network framework for distributed virtual environments. In: CVE ’02: Proceedings of the 4th international conference on collaborative virtual environments. ACM Press, New York, pp 47–54 21. Molet T, Aubel A, Capin T, Carion S, Lee E, Thalmann NM, Noser H, Pandzic I, Sannier G, Thalmann D (1999) Anyone for tennis? Presence MIT 8(2):140–156 22. Benford S, Bowers J, Fahlen L, Greenhalgh G, Mariani J, Rodden T (1995) Networked virtual reality and cooperative work. Presence MIT 4(4):364–386 23. Suler J (1997) Psychology of cyberspace – history of the palace. In: Psychology of cyberspace 24. https://www.activeworlds.com/ 25. https://en.wikipedia.org/wiki/Second_Life 26. Constine J (2019) Facebook announces Horizon, a VR massive-multiplayer world. TechCrunch, September 25 27. https://en.wikipedia.org/wiki/Internet_of_things 28. Sutikno T, Thalmann D (2022) Insights on the internet of things: past, present, and future directions. TELKOMNIKA Telecommun Comput Electron Control 20(6):1399–1420

Chapter 6

Mixed Realities

Not only real and virtual worlds can coexist, but they can be mixed. In this chapter, we emphasize the techniques of augmented reality and mixed reality. Augmented reality generally requires tracking methods to make strict correspondence between virtual and real worlds. Such methods need a collaboration between computer graphics and computer vision. The role of deep learning methods and AI methods is today essential. This chapter will discuss the role of deep learning in 3D hand pose.

6.1

Augmented Reality and Augmented Virtuality

In Chap. 1, we introduced the concept of reality continuum (Sect. 1.4) to classify variants of virtual reality. Such classification depends on the proportion of virtual imagery with respect to a real-world scene that is displayed in an application. We can distinguish three main categories: virtual reality (VR), in which only virtual images are used; augmented reality (AR), where computer graphics are superimposed over images of the real world; and augmented virtuality (AV), where imaged portions of the real world are integrated within a virtual world. The term mixed realities (MR) is used to refer to either augmented reality or augmented virtuality, since both of them combine elements from real and virtual worlds. All the VR techniques, algorithms, interaction devices, and so on that are described in this book can also be applied in both AR and AV applications. In addition, an important component should be incorporated: a tracking mechanism. Tracking is usually done applying computer-vision and image-processing techniques to data acquired using a video camera. The objective of tracking is to detect relevant objects in the real world; acquire their position, orientation, contour, and other visual features; and use this information to “attach” computer-generated images to real objects or portions of the scene.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_6

141

142

6

Mixed Realities

If an MR application is to operate in real time, where computer-generated graphics are superimposed on live images of the surrounding world, all the MR processes must be able to keep pace with the captured image frame rate. Knowing what is present in the real world and where is a vital component for the effective operation of many MR systems. This description can be used to aid virtual object placement, occlusion culling, collision detection, or many other visual or simulated effects. Building up and reconstructing this real-world description in real time is the main challenge faced by MR systems. Different tracking algorithms and techniques have been developed to solve this problem.

6.2

Tracking Techniques

This section is based on the survey of 3D tracking methods presented in [1]. In addition to computer vision, other techniques can be used to achieve 3D tracking. Mechanical trackers, such as those based on exoskeletons (see Sect. 10.4.3), are accurate enough, although they constrain the user to a limited working space. Magnetic trackers, like those used for motion capture (see Sect. 3.3) are vulnerable to distortions by metallic structures in the environment and limit the range of displacements. Ultrasonic trackers, such as the Polhemus Isotrac, suffer from noise and tend to be inaccurate at long ranges because of variations in the ambient temperature. Inertial trackers drift with time. Computer vision has the potential to yield noninvasive, accurate, and low-cost solutions. Different methods with variable degrees of accuracy and robustness can be applied.

6.2.1

Markers-Based Tracking

A popular tracking method for augmented reality applications is adding fiducials, such as LEDs or special markers, to the scene or target object to ease the registration task. This assumes that one or more fiducials are always visible. Otherwise, the registration cannot be done. A drawback of this method is that it is not always possible to place fiducials. In fact, AR end users do not like them because they are visible in the scene, and it is not always possible to modify the environment before running the application. The addition in the scene of fiducials, also called landmarks or markers, greatly helps accomplish two main tasks: extracting information from the image and estimating the pose of the tracked objects. Markers constitute image features that are easy to extract. They also provide reliable, easy to exploit measurements for the pose estimation. There are two types of fiducials: “point fiducials,” which give one-point correspondence between the scene and the image, and “planar fiducials.” To obtain more

6.2

Tracking Techniques

143

Fig. 6.1 Markers-based tracking using ARToolkit

information from each point fiducial, it is possible to turn it into a planar shape with identifiable corners. A single planar fiducial provides all six spatial constraints needed to define a coordinate frame (position and orientation). Point fiducials are commonly circular markers because the appearance of circular patterns is relatively invariant under perspective distortion, facilitating their detection. To facilitate their identification, the markers can be arranged in a distinctive geometric pattern. This is the technique used by most optical motion capture systems (see Sect. 3.3.1). Planar rectangular fiducials have gained popularity because they yield a robust, low-cost solution for real-time 3D tracking. The ARToolkit software library uses this approach. ARToolkit markers have a black border on a white background to facilitate the detection (see Fig. 6.1). An inner pattern allows identification of the different markers. The image is first thresholded, and the system looks for connected regions of black pixels. Regions whose outline contour can be fitted by four-line segments are kept. Then each region is corrected to remove the perspective distortion and compare it by template matching with the known patterns. The whole process, including marker detection and pose estimation, can run at 30 frames per second. The 3D tracking system does not require any manual initialization and is robust to fiducial occlusion. Under good lighting conditions, the recovered pose is accurate enough for AR applications. ARToolkit can be a good solution for 3D tracking if it is possible to add markers to the scene. Due to its low CPU requirements, similar marker-based applications have been implemented on mobile devices such as PDAs and mobile phones [2].

144

6.2.2

6

Mixed Realities

Marker-Less Tracking

In some applications, it is either undesirable or very difficult to add markers to the scene, for example, outdoor environments or unknown environments. In such cases, it is necessary to rely on naturally present object features, such as edges, corners, or texture. However, this increases the complexity of the tracking algorithm. Finding and following feature points or edges can be difficult because they are hard to detect, and in many cases, there are not enough of them on typical objects. Total or even partial occlusion of the tracked objects typically results in tracking failure. The camera can easily move too fast so that the images are motion-blurred; the lighting during a shot can change significantly; reflections and specularity may confuse the tracker. It should also be considered that an object may drastically change its aspect due to displacement, for example, changes in perspective or appearance of hidden parts when changing the point of view. In such cases, the features to be followed always change, and the tracker must deal with features coming in and out of the picture. 3D knowledge is often used to ease the task of marker-less tracking algorithms. The 3D knowledge can come in the form of a CAD model, a set of planar parts, or even a rough 3D model such as an ellipsoid. Two kinds of approaches can be distinguished, depending on the nature of the image features being used: edge-based methods and methods that rely on information provided by pixels inside the object’s projection (optical flow, template matching, or interest-point correspondences). Edge-Based Methods Edge-based methods try to match the projections of the edges of the target object to an area of high image gradient. Most of the early tracking approaches were edge-based. These methods are both computationally efficient and relatively easy to implement. They are also naturally stable in lighting changes, even for specular materials. One strategy consists in looking for strong gradients in the image around a first estimation of the object pose without explicitly extracting the contours. This is fast and general. Another strategy is to first extract image contours, such as straight-line segments, and to fit the model outlines to these image contours. The loss in generality can be compensated for by a gain in robustness. In the following example (see Fig. 6.2), we can see an application of edge-based tracking methods for creating a checkers game with a virtual character. Optical Flow-Based Methods Optical flow is the apparent motion of the image projection of a physical point in an image sequence, where the velocity at each pixel location is computed under the assumption that projection’s intensity remains constant. The optical flow can be computed using the Lucas-Kanade method [3].

6.2

Tracking Techniques

145

Fig. 6.2 Playing checkers with a virtual human

Template Matching The Lucas-Kanade algorithm [3, 4] was originally designed to compute the optical flow at some image locations, but it can be used for a more general purpose; to register a 2D template to an image under a family of deformations. It does not necessarily rely on local features such as edges or interest points, but on global region tracking, which means using the whole pattern of the object to be tracked. Such a method can be useful to treat complex objects that are difficult to model using local features. Although it can be computationally expensive, it has been shown that under some conditions, it can be effectively formulated. Interest-Point Methods Interest-point methods use localized features instead of global ones; this has several advantages. In the same way as edge-based methods, they rely on matching individual features across images and are able to cope with partial occlusions or matching errors. Illumination invariance is also simple to achieve. But unlike edge-based methods, they do not get confused by background clutter and they exploit more of the image information, which tends to make them more robust. They are similar in some ways to optical-flow approaches, but they allow faster motions since no small interframe motion assumption is made. A representative example of this kind of method is the face-tracking system described by Vacchetti et al. in [5]; their tracking algorithm uses interest points and one reference image (initialization phase).

146

6

Mixed Realities

Tracking Without 3D Knowledge Using a 3D model as reference for the tracking algorithm increases its reliability, but it can be too expensive or inconvenient to have a model in some applications. If no 3D knowledge is available, different image features should be exploited. It is possible to simultaneously estimate both camera motion and scene geometry without a 3D reference model. The recovered trajectory and 3D structure are expressed in an arbitrary coordinate system, for example, the one corresponding to the initial camera position. This problem is known as simultaneous localization and mapping (SLAM). SLAM is a common problem for the self-localization of a moving robot. Some approaches rely on projective properties that provide constraints on camera motion and 3D point locations from 2D correspondences. A representative example is the work presented by Nister et al. [6].

6.3

Mixed Reality Tool Kits

The first and most popular solutions for rapid development of augmented reality applications was the ARToolkit (https://en.wikipedia.org/wiki/ARToolKit). ARToolkit was originally developed by Dr. Hirokazu Kato, and its ongoing development was supported by the Human Interface Technology Laboratory (HIT Lab) at the University of Washington; the HIT Lab NZ at the University of Canterbury, New Zealand, and ARToolworks, Inc. in Seattle. ARToolkit was made available as an open source in 2001. In 2015, former ARToolworks chief officers created artoolkitX to keep developing and maintaining the software (http:// www.artoolkitx.org/). Nowadays, the most popular AR applications are deployed on mobile devices and use software frameworks based mainly one of the following AR SDKs: ARKit for iOS (https://developer.apple.com/augmented-reality/arkit/), ARCore for Android (https://developers.google.com/ar), and Vuforia for cross-platform development (https://developer.vuforia.com/). These SDKs provide libraries for 3D rendering, camera calibration, and real-time tracking, as well as support for various sensors, such as GPS, accelerometers, and gyroscopes.

6.4

Deep Learning and 3D Hand Pose

Deep learning has changed the way of considering motion capture. In machine learning, deep learning is a method based on artificial neural networks that incorporate representation learning. There are three types of learning: supervised, semi-supervised, and unsupervised. Multilayer deep learning algorithms extract higher-level features from raw input by progressively layering them. For example, in image processing, lower layers may identify edges, while higher layers may identify the concepts relevant to a human such as digits or letters or faces.

6.4

Deep Learning and 3D Hand Pose

147

Fig. 6.3 3D hand pose estimation from depth images

Fig. 6.4 Hand surface completion

In this section, we will not describe a full deep learning-based approach to motion capture, because it is not really available, but we will emphasize the potential on the specific case of 3D hand pose. 3D hand pose estimation plays a significant role in human-computer interaction, especially in virtual reality (VR) and augmented reality (AR) applications. Figure 6.3 shows the principle of hand pose estimation. As shown in Fig. 6.4, there are several challenges: Severe self-occlusion problem High similarity among fingers Noisy depth image and data missing We can consider three approaches to 3D hand pose: 1. Model-Driven Approach We find the optimal hand pose parameters via fitting a deformable 3D hand model to input image observations. We need to predefine hand size and motion constraints; sensitive to initialization.

148

6

Mixed Realities

view 1 z z

x x

y

Depth Image

Segmented Hand 3D Points

y

view 2

Multi-view Fusion

view 3

3D Joints

view N Projections

CNNs

Heat-maps

Fig. 6.5 Multi-view CNN-based approach for 3D hand pose estimation

2. Data-Driven Approach It needs to train discriminative models to map image features to hand pose parameters: – Random Forest ([7, 8], etc.) – Convolutional Neural Networks (CNNs) [9, 10] It does not require model calibration and robust to poor initialization 3. CNN-Based Approach With the success of deep neural networks and the emergence of large hand pose datasets [7, 8], many methods are based on CNNs. Many CNN-based approaches [11–14], directly take the depth image as input to 2D CNNs, which cannot fully exploit 3D information in the depth image. Image-based features extracted by 2D CNNs are not directly suitable for 3D hand pose estimation due to the lack of 3D spatial information. Ge et al. [11] better utilize the depth cues by projecting the depth image onto three views and applying multi-view CNNs to regress three views’ heat-maps (Fig. 6.5). However, the multi-view CNNs still cannot fully exploit 3D spatial information in the depth image, since the projection from 3D to 2D will lose certain information. Ge et al. propose, for example, a 3D CNN-based hand pose estimation approach that can capture the 3D spatial structure of the input and accurately regress full 3D hand pose in a single pass, as illustrated in Fig. 6.6. There have been successful applications of 3D CNNs for video and dynamic hand gesture analysis [14–17], which rely on time as a third dimension. CAD models and depth images are also analyzed using 3D CNNs to extract 3D features. For modeling 3D shapes, 3D ShapeNets [18] use Convolutional Deep Belief Networks. By

References

149

Fig. 6.6 3D CNN, which takes volumetric representations as input and regresses a lower dimensional representation of 3D joint locations

applying subvolume supervision and anisotropic probing, Qi et al. [19] demonstrate that 3D CNN with low input volume resolution can still perform well in object classification. According to Song and Xiao [20], 3D CNN can be used to detect 3D objects in RGB-D images. A 3D CNN called VoxNet is proposed by Motaena and Scherer [21] to process LiDAR, RGB-D, and CAD data to recognize objects. The 3D CNN is also used for detecting landing zones [22]. For 3D shape deformation, Yumer and Mitra [23] propose using a 3D CNN to learn deformation flows from CAD models. The 3D CNN proposed by Song et al. [24] can be used to complete semantic scenes. The works achieve state-of-the-art results, but none focuses on estimating articulated pose from single depth images in real time from a set of articulated joints. Ge et al. [25] propose a 3D CNN-based hand pose estimation method that uses a volumetric representation as its input. A volumetric representation of the observed hand depth image is intended to provide sufficient and meaningful information about the hand in 3D space in real time. Using the 3D volumes, 3D CNNs will be trained to predict the 3D location of the hand joints in 3D space.

References 1. Martin R (ed) (1993) Directions in geometric computing. Information Geometers Ltd, Winchester 2. https://en.wikipedia.org/wiki/Internet_of_things 3. Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: International joint conference on artificial intelligence, pp 674–679 4. Baker S, Matthews I (2004) Lucas-Kanade 20 years on: a unifying framework. Int J Comput Vision:221–255 5. Vacchetti L, Lepetit V, Fua P (2004) Stable real-time 3D tracking using online and offline information. IEEE Trans Pattern Anal Mach Intell 26(10):1385–1391 6. Nister D, Naroditsky O, Bergen J (2004) Visual odometry. In: Conference on computer vision and pattern recognition, pp 652–659 7. Tang D, Chang HJ, Tejani A, Kim T-K (2014) Latent regression forest: structured estimation of 3D articulated hand posture. In: Proceedings of the IEEE conference on computing visual pattern recognition, Columbus, pp 3786–3793 8. Sun X, Wei Y, Liang S, Tang X, Sun J (2015) Cascaded hand pose regression. In: Proceedings of the IEEE conference on computing visual pattern recognition, Columbus, pp 824–832

150

6

Mixed Realities

9. Tompson J, Stein M, Lecun Y, Perlin K (2014) Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans Graph 33(5):169 10. Oberweger M, Wohlhart P, Lepetit V (2015) Hands deep in deep learning for hand pose estimation. In: Proceedings of the computer vision winter workshop, pp 21–30 11. Ge L, Liang H, Yuan J, Thalmann D (2016) Robust 3D hand pose estimation in single depth images: from single-view CNN to multi-view CNNs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3593–3601 12. Sinha A, Choi C, Ramani K (2016) Deephand: robust hand pose estimation by completing a matrix with deep features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4150–4158 13. Ye Q, Yuan S, Kim T-K (2016) Spatial attention deep net with partial PSO for hierarchical hybrid hand pose estimation. In: Proceedings of the European conference on computer vision, pp 346–361 14. Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4207–4215 15. Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231 16. Tran D, Bourdev LD, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE conference on computer vision, pp 4489–4497 17. Camgoz NC, Hadfield S, Koller O, Bowden R (2016) Using convolutional 3D neural networks for user-independent continuous gesture recognition. In: Proceedings of the IEEE conference on pattern recognition, pp 49–54 18. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3D shapenets: a deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1912–1920 19. Qi CR, Su H, Niessner M, Dai A, Yan M, Guibas LJ (2016) Volumetric and multi-view CNNs for object classification on 3D data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5648–5656 20. Song S, Xiao J (2016) Deep sliding shapes for amodal 3D object detection in RGB-D images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 808–816 21. Maturana D, Scherer S (2015) VoxNet: A 3D convolutional neural network for real-time object recognition. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, pp 922–928 22. Maturana D, Scherer S (2015) 3D convolutional neural networks for landing zone detection from LiDAR. In: Proceedings of the IEEE conference on robotics and automation, pp 3471–3478 23. Yumer ME, Mitra NJ (2016) Learning semantic deformation flows with 3D convolutional networks. In: Proceedings of the European conference on computer vision, pp 294–311 24. Song S, Yu F, Zeng A, Chang AX, Savva M, Funkhouser T (2017) Semantic scene completion from a single depth image. In: Proceedings of the IEEE conference on computer vision pattern recognition, pp 1746–1754 25. Ge L, Liang H, Yuan J, Thalmann D (2019) Real-time 3D hand pose estimation with 3D convolutional neural networks. IEEE Trans Pattern Anal Mach Intell 41(4):956–970

Part III

Perceiving Virtual Worlds

Chapter 7

Senses and Sensors

Virtual reality is known as dependent on specific hardware. This chapter explains why this hardware is needed. Users in VR should be able to perceive the virtual world. This means that the user of a VR system should have a way of feeling the virtual world through his or her senses, which is only possible with the use of sensors, which are hardware dependent. This chapter presents the various senses that exist and should be ideally perceived in a virtual world.

7.1

Introduction

When users are immersed in the virtual worlds, what can they really feel in terms of senses and what is a sensor? Our senses are our primary way of interacting with the world. Long before we have learnt how to use language, or reason, or be in control of our emotions, we are able to make use of our senses. There are many different types of senses, and they all correspond to specific groups of brain regions, where they receive and interpret signals that correspond to various physical phenomena. A sense is a way of perceiving our surroundings. A human’s senses include sight, sound, touch, smell, and taste. A sensor detects something; for example, your nose is a sensor that detects smell. In Chapters 8, 9, 10, 11, and 12, we discuss how senses play a role in virtual reality and how sensors can contribute. There are a multitude of senses available to humans.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_7

153

154

7

Senses and Sensors

Fig. 7.1 The five senses. (Hans Makart, nineteenth century, public domain)

7.2

The 5 Senses

When we think of human senses, we think of eyesight, hearing, taste, touch, and smell. These five senses are shown in Fig. 7.1 to be exampled by five nude female figures who, from left to right, experience and show touch, hearing, looking, smelling, and tasting. Yet we have always known that we are capable of sensing much more than this. Exactly what, however, is still subject to ongoing scientific research.

7.3

The Extra Senses

Neuroscientists are well aware that we are a bundle of senses. As this video by Aeon (https://aeon.co/videos) explains, many would argue that we have anywhere between 22 and 33 different senses.

7.3

The Extra Senses

7.3.1

155

Proprioception

By using this sense, you can determine where your body parts are in relation to other body parts. It allows you to locate the parts of your body without looking. For example, we can walk without looking at our feet (Fig. 7.2). Proprioception is also called body awareness sense. It also helps us figure out how much force to use in different activities (think of picking up a fragile glass versus a gallon of milk).

7.3.2

Equilibrioception

In terms of acceleration and directional changes, it is the sense that helps you maintain your balance. This sense also allows for perceiving gravity. It keeps us upright and allows us to move around safely. When you are driven in a bus or train and you see the landscape, you can have the nausea. This is based on equilibrioception. Located in the inner ear, this sensory system is called vestibular labyrinthine system (Fig. 7.3).

7.3.3

Pallesthesia

Pallesthesia, or vibratory sensation, is the ability to perceive vibration (Fig. 7.4). It is usually generated by mechanoreceptors such as Pacinian corpuscles, Merkel disk receptors, and tactile corpuscles. These receptors stimulate action potentials in afferent nerves (sensory neurons) in the skin and body.

Fig. 7.2 Walking in the park. (Licensed under the Creative Commons Attribution 2.0)

156 Fig. 7.3 The performance of “Jultagi” in Jeonju, South Korea. (Licensed under the Creative Commons Attribution 2.0 Generic license)

Fig. 7.4 Maria Leal da Costa, working. Lithuania. (Creative Commons Attribution-Share Alike 4.0)

7

Senses and Sensors

7.3

The Extra Senses

7.3.4

157

Thermoception

Thermoception refers to the ability to sense temperature changes in the body (Fig. 7.5). The skin’s thermal receptors monitor heat, cold, and sudden changes in temperature before transmitting that information to the brain.

7.3.5

Nociception

It is the ability to feel pain (Fig. 7.6). This was previously considered the result of overloading other senses, such as touch. But now it’s viewed as its own unique Fig. 7.5 Tastant l’aigua d’una piscina, Laia from Reus Catalonia. (Creative Commons License)

Fig. 7.6 Symptoms: Stomach pain. (Public domain, Creative Commons)

158

7

Senses and Sensors

sensory system. There are three distinct types of pain receptors: cutaneous (skin), somatic (bones and joints), and visceral (body organs).

7.3.6

Chronoception

Chronoception is studied by psychologists, cognitive linguists, and neuroscientists as a subjective experience, characterized by the perception of duration of indefinite and unfolding events. Figure 7.7 illustrate the passing time. Fig. 7.7 Passing Time sculpture by Aucklandbased sculptor Anton Parsons. (Creative Commons Zero, Public Domain Dedication)

Chapter 8

Vision

This chapter presents the primary sense of the human: the vision. The main hardware corresponding to this sense is the display, very common for almost everybody using a computer or a smartphone. But for VR, there are more specific equipments that bring the essential aspect of immersion. We emphasize the head-mounted display and the CAVE-like projection systems. Historically, one of the first human senses that was addressed in virtual reality applications was vision. The first computer-generated images were produced in the 1950s. Some of the first images generated by an electronic – analog – machine were created by Ben Laposky. This mathematician and artist from Iowa produced images by manipulating electronic beams displayed across the fluorescent face of an oscilloscope’s cathode-ray tube. Laposky called his oscillographic artworks “oscillons” and “electronic abstractions.” The history of virtual reality presented in Chap. 1 shows that the first VR systems, like Sutherland’s ultimate display or Heilig’s Sensorama, were based on graphical displays. As will be discussed in the following chapters, vision is not necessarily the most relevant of human senses, but it is recognized as the primary source of information about the outside world. Vision is commonly used to double check the accuracy of other senses, for example, we direct our sight toward unexpected sources of sound, such as an ambulance siren, to verify their location and evaluate potential danger. The human vision system allows us to recognize shapes in a three-dimensional space with different colors and light intensities. In Chap. 2, we covered in detail the mathematical and computational foundations for computer graphics generation. Chapter 3 explained how to put such graphics in motion. This chapter describes the technologies that are currently used to display the graphical aspect of a virtual world.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_8

159

160

8.1 8.1.1

8

Vision

Graphical Display Technologies Cathode-Ray Tubes

The cathode-ray tube (CRT) was invented by German physicist Karl Ferdinand Braun in 1879. The “cathode” is a heated filament (electron gun) in a vacuum created inside a glass “tube.” The “ray” is a stream of electrons that naturally pour off a heated cathode into the vacuum. The stream of electrons is focused by a focusing anode into a tight beam and then accelerated by an accelerating anode. This beam of electrons flies through the vacuum in the tube and hits the flat screen at the other end of the tube. This screen is coated with phosphor,1 which glows when struck by the beam. The tube is wrapped in coils of wire used to steer the electron beam. These coils can create magnetic fields inside the tube, and the electron beam responds to the fields. Two sets of coils create magnetic fields that move the electron beam vertically and horizontally. By controlling the voltages in the coils, the electron beam can be positioned at any point on the screen. In a CRT, phosphor coats the inside of the screen. When the electron beam strikes the phosphor, it makes the screen glow. In a black-and-white screen, there is one phosphor that glows white when struck. In a color screen, there are three phosphors arranged as dots or stripes that emit red, green, and blue light. There are also three electron beams to illuminate the three different colors together. CRTs still find adherents in computer gaming because of higher resolution per initial cost and small response time. They are also still popular in the printing and broadcasting industries as well as in the professional video, photography, and graphics fields due to their greater color fidelity and contrast, better resolution when displaying moving images, and better view from angles, although improvements in LCD technology increasingly alleviate these concerns.

8.1.2

Liquid Crystal Displays

Liquid crystals were first discovered in 1888 by Austrian botanist Friedrich Reinitzer. Reinitzer observed that when he melted a cholesterol-like substance (cholesteryl benzoate), it first became a cloudy liquid and then cleared up as its temperature rose. Upon cooling, the liquid turned blue before finally crystallizing. In 1968, RCA made the first experimental LCD. Liquid crystals are affected by electric current. A particular sort of nematic liquid crystal, called twisted nematics (TN), is naturally twisted. Applying an electric current to these liquid crystals will untwist them to varying degrees, depending on 1

A phosphor is any material that, when exposed to radiation, emits visible light. The radiation might be ultraviolet light or a beam of electrons. Fluorescent colors absorb invisible ultraviolet light and emit visible light at a characteristic color.

8.1

Graphical Display Technologies

161

the current’s voltage. LCDs use these liquid crystals because they react predictably to electric current in such a way as to control light passage. If an electric charge is applied to liquid crystal molecules, they untwist. When they straighten out, they change the angle of the light passing through them so that it no longer matches the angle of a top polarizing filter. Consequently, no light can pass through that area of the LCD, making it darker than the surrounding areas. Small and inexpensive LCDs reflect light from external light sources, for example, in LCD watches, numbers appear where small electrodes charge the liquid crystals and make the layers untwist so that light does not pass through the polarized film. Most computer displays are lit with built-in fluorescent tubes above, beside, and sometimes behind the LCD. A white diffusion panel behind the LCD redirects and scatters the light evenly to ensure a uniform display. There are two main types of LCDs, passive- and active-matrix displays. Passive-matrix LCDs use a simple grid to supply the charge to a particular pixel on the display. The grid is created with two glass layers called substrates. One substrate is given columns and the other is given rows made from a transparent conductive material, usually indium-tin oxide. The rows or columns are connected to integrated circuits that control when a charge is sent to a particular column or row. The liquid crystal material is inserted between the two glass substrates, and a polarizing film is added to the outer side of each substrate. To turn on a pixel, the integrated circuit sends a charge down the correct column of one substrate and a ground activated on the correct row of the other. The row and column intersect at the designated pixel, and that delivers the voltage to untwist the liquid crystals at that pixel. Passive-matrix systems have significant drawbacks, notably slow response time and imprecise voltage control. Response time refers to the LCD’s ability to refresh the image displayed. Imprecise voltage control makes it difficult to influence only one pixel at a time. When voltage is applied to untwist one pixel, the pixels around it also partially untwist, which makes images appear fuzzy and lacking in contrast. Active-matrix LCDs depend on thin film transistors (TFT). TFTs are tiny switching transistors and capacitors. They are arranged in a matrix on a glass substrate. To address a particular pixel, the proper row is switched on, and then a charge is sent down the correct column. Since all of the other rows that the column intersects are turned off, only the capacitor at the designated pixel receives a charge. The capacitor is able to hold the charge until the next refresh cycle. The amount of voltage supplied to a crystal determines the amount of light that goes through. To show colors, an LCD requires three subpixels with red, green, and blue color filters to create each color pixel. Through the careful control and variation of the voltage applied, the intensity of each subpixel can range over 256 shades. Combining the subpixels produces a possible palette of 16.8 million colors (2563). Color displays require many transistors, one per subpixel; a typical LCD with a resolution of 1024 × 768 pixels uses 2, 359, 296 transistors. A problem with one transistor creates a “bad pixel” on the display. It is very likely to have a few bad pixels scattered across the screen, particularly in very large LCDs. LCDs are found in many devices, including watches, handheld devices, and large computer monitors. Desktop-based applications can use this kind of display due to

162

8

Vision

the limited space required by the screen, the brightness, and the flicker-free2 image they provide. Important factors to consider when evaluating an LCD monitor include the following: • Resolution: VR applications usually require high resolutions to produce realistic images. Unlike CRT monitors, LCDs have a native-supported resolution for best display effect. • Dot pitch: The distance between the centers of two adjacent pixels. The smaller the dot pitch size, the less granularity is present, producing a sharper image. • Active display area: The size of an LCD panel measured diagonally. • Response time: The minimum time necessary to change a pixel’s color or brightness. The faster the response time, the smoother will be the animation. • Matrix type (active or passive): Nowadays, most of the LCDs on the market are active. • Viewing angle: More specifically known as viewing direction. The direction from which the display will look the best. For multiple-user applications, a wider viewing angle is desirable. • Color support: How many colors are available, also known as color gamut. • Brightness: Refers to the emitted luminous intensity on screen and is measured in candelas per square meter (cd/m2 or nits). A higher cd/m2 or nit value means higher onscreen brightness. Office applications usually require lower brightness to prevent eye strain due to the long periods the user spends in front of the screen. On the contrary, VR applications benefit from high luminous intensity. • Contrast ratio: The ratio of the intensity of the brightest bright to the darkest dark. • Aspect ratio: The ratio of the width to the height, as is 4:3 (used by most CRTs), 16:9 (common format for films), or 16:10. • Input ports: Most LCD panels still use the traditional analog VGA connector known as the DSUB-15 or HD15 connector. This is the same connector that is used on all CRT monitors and on most PC graphics cards. Modern LCDs and graphics cards use the DVI connector. This is a digital interface that is supposed to allow for a cleaner and brighter picture compared to standard VGA connectors. Some monitors may also come with composite video connectors to allow them to function as a TV screen.

8.1.3

Plasma Displays

The plasma display was invented at the University of Illinois at Urbana-Champaign by Donald L. Bitzer, H. Gene Slottow, and graduate student Robert Willson in 1964 The constant refreshing of a CRT produces an effect called flickering, which can cause headaches in migraine sufferers and seizures in epileptics, if they are photosensitive. Screen filters are available to reduce these effects. A high refresh rate (above 75 Hz) also helps to prevent these effects. 2

8.1

Graphical Display Technologies

163

for the PLATO computer system. The original monochrome (orange, green, or yellow) panels enjoyed a surge of popularity in the early 1970s because the displays were rugged and needed neither memory nor circuitry to refresh the images. A long period of sales decline followed in the late 1980s as semiconductor memory made CRT displays cheaper than plasma displays. Plasma’s relatively large screen size and thin profile made the displays attractive for high-profile placement, such as in lobbies and stock exchanges. In 1997, Pioneer started selling their first plasma televisions to the public. Plasma televisions compete with projector screens. Screen sizes have increased since the 21-inch (533 mm) display in 1992. The largest plasma video display in the world was shown at the 2006 Consumer Electronics Show in Las Vegas, a 103-inch (2.616-m) unit manufactured by Matsushita Electrical Industries (Panasonic). A plasma display panel (PDP) is a type of flat panel display now commonly used for large TV displays (typically above 37 inches or 940 mm). Many tiny cells located between two panels of glass hold an inert mixture of noble gases (neon and xenon). The gas in the cells is electrically turned into plasma which then excites phosphors to emit light. Each pixel is made up of three fluorescent lights (red, green, and blue). The central element in fluorescent light is plasma, a gas made up of free-flowing ions (electrically charged atoms) and electrons. Under normal conditions, a gas is mainly made up of uncharged particles. If many free electrons are introduced into the gas by establishing an electrical voltage across it, the free electrons collide with the atoms, knocking loose other electrons. With a missing electron, an atom loses its balance. It has a net positive charge, making it an ion. To ionize gas in a particular cell, the plasma display’s computer charges electrodes that intersect at that cell. This process is done thousands of times in a fraction of a second, charging each cell in turn. When the intersecting electrodes are charged, an electric current flow through the gas in the cell. The current creates a rapid flow of charged particles, which stimulates the gas atoms to release ultraviolet photons. The released ultraviolet photons interact with the phosphor material coated on the inside wall of the cell. When an ultraviolet photon hits a phosphor atom in the cell, one of the phosphor’s electrons jumps to a higher energy level and the atom heats up. When the electron falls back to its normal level, it releases energy in the form of a visible light photon. The phosphors in a plasma display give off colored light when they are excited. Every pixel is made up of three separate subpixel cells, each with different colored phosphors. By varying the pulses of current flowing through the different cells, the control system can increase or decrease the intensity of each subpixel color to create hundreds of different combinations of red, green, and blue, producing colors across the entire spectrum. Plasma display technology makes it possible to produce very wide screens using extremely thin materials. And because each pixel is lit individually, the image is very bright and looks good from almost every angle. The image quality isn’t quite up to the standards of the best cathode-ray tube sets, but it certainly meets most people’s expectations.

164

8

Vision

Until quite recently, the superior brightness, faster response time, greater color spectrum, and wider viewing angle of color plasma video displays, when compared with LCD televisions, made them one of the most popular forms of display for HDTV flat panel displays. However, improvements in LCD technology have narrowed the technological gap. The lower weight, falling prices, higher available resolution, and often lower electrical power consumption of LCDs make them competitive against plasma displays in the television set market.

8.2

Virtual Reality Displays

Virtual reality displays use the technologies described in the previous section to provide the user with a system to visually explore a virtual world. Based on the number of users that can participate in the VR experience, displays can be roughly classified in two main categories: individual and collective displays. A second classification criterion is the degree of visual immersion. Following this criterion, VR displays range from fully immersive systems that completely isolate the user’s field of view from the real world (HMDs) up to handheld displays that provide a small visualization area and are used as auxiliary tools in real-world tasks.

8.2.1

Head-Mounted Displays

Most head-mounted displays (HMD) have either one or two (for stereoscopic vision) small displays with lenses and semitransparent mirrors embedded in a helmet, eyeglasses, or visor. The display units are miniaturized and may use CRT, LCDs, liquid crystal on silicon (LCos), or OLED technology. Many head-mounted displays include speakers or headphones so that it can provide both video and audio output. HMDs almost always include a tracking device so that the point of view displayed in the monitor changes as the user moves her head. HMDs have the potential to display a different image to each eye. This can be used to show stereoscopic images. An object located at a distance of 9 m, or more is seen from essentially the same perspective by each of the average human eyes, which have an inter-pupillary distance of between 2.5 and 3 inches. At shorter distances, the perspective from each eye is significantly different and the expense of generating two different images becomes worthwhile. Humans have a field of view (FOV) of approximately 180°; see Fig. 8.1. However, most HMDs offer a more limited FOV, ranging from 25° to 45°. To achieve their full potential, HMDs are used in combination with tracking sensors that detect changes of angle and orientation and even position. When such data is available in the computer controlling the VR application, it can be used to render the images corresponding to the angle of look at a particular time. This allows

8.2

Virtual Reality Displays

165

Fig. 8.1 Human field of view

the user to “look around” a virtual environment simply by moving the head without needing a separate controller to change the viewpoint. Tracking devices can be wired or wireless (radio transmission) and use either magnetic fields or accelerometers to acquire orientation (3 DOF) and even position (6 DOF) in space. Early HMD manufacturers include: Fifth Dimension Technologies (5DT), Cybermind, Kaiser Electro-Optics (KEO), Virtual Research Systems, nVision, I-O Display Systems, and eMagin. Since the introduction of the Oculus Rift DK1 in 2013, several companies have developed a new generation of HMDs. Figure 8.2 shows representative VR headsets manufactured by Meta, HTC, and Acer. Other companies such as Sony have also produced HMDs as a new way to experience console video games. Some HMDs allow viewing of a computer-generated image superimposed on a real-world view. This is done by projecting the synthetic image on a semitransparent glass. This kind of application is referred to as augmented reality (AR). The HoloLens (https://www.microsoft.com/en-us/hololens) and Magic Leap (https:// www.magicleap.com/) are some of the most representative AR devices. These devices, also known as see-through HMDs, have applications in the maintenance of complex systems as they can provide the technician with “X-ray vision” by combining computer graphics (e.g., system diagrams) with natural vision.

166

8

Vision

Fig. 8.2 Different types of head-mounted displays

See-through HMDs also have applications in surgery, as they allow the combination of radiographic data (CAT scans and MRI imaging) with the surgeon’s natural view of the operation. Police, soldiers, and firefighters can use see-through HMDs to display tactical information such as maps or thermal imaging data while viewing the real scene. Commercially available HMDs can provide compelling immersive experiences. They are frequently used in combination with vision-based hand tracking, and/or 6-DOF tracked controllers. Most HMDs include hand-held controllers whose position and orientation can be accurately estimated and used to interact with the virtual content. Despite the progress in screen resolution and field of view, HMDs can still produce discomfort in some users. An additional drawback is the “social isolation” that can be felt by some users when using an HMD. Being “disconnected” from the real world can be uncomfortable for some people. VR display technologies, such as fish tank VR (desktop-based VR), large projection screens, or CAVE systems, still offer advantages in terms of keeping the user “in touch” with the real world, making it easier to share and collaborate with interactive 3D content. HMD systems can be classified as “inside-out” displays that let the user look outward from inside the virtual world. Other technologies, like fish tank VR, are “outside-in” displays that can be used to look inward from outside the virtual environment. HMDs can be an appropriate choice for VR applications requiring frequent turns and spatial orientation, such as navigation tasks, for example, to explore virtual cities or buildings. Virtual environments (VE) presented through HMDs are often explored on foot. The area of the VE that can be explored is limited to the dimensions of the tracking space of the HMD. Some strategies have been proposed to overcome this limitation. In [1], Williams et al. describe a strategy that takes advantage of people’s natural ability to spatially update. The authors propose the method of “resetting,” which involves manipulating the user’s location in physical space to move out of the path of the physical obstruction while maintaining her spatial awareness of the virtual space. Other strategies include using different navigation paradigms such as teleportation or joystick-based navigation using a hand-held controller [2].

8.2

Virtual Reality Displays

8.2.2

167

Fish Tank VR

Ware et al. [3] defined “fish tank virtual reality” as a system using a high- resolution monitor supplemented with hardware for stereoscopic viewing, such as shutter glasses, and a head-tracking device for adjusting the perspective transformation to a user’s eye position can simulate the appearance of stable 3D objects positioned behind or just in front of the screen. Stereoscopic view is commonly achieved using LCD shutter glasses. These devices use polarizing filters and take advantage of the property of liquid crystals, which become dark when a voltage is applied but are otherwise transparent. A pair of eyeglasses can be made using this material and connected to a computer video card. The video card alternately darkens over one eye and then the other, in synchronization with the refresh rate of the monitor, while the monitor alternately displays different perspectives for each eye. This is called alternate-frame sequencing. At sufficiently high refresh rates, the viewer’s visual system does not notice the flickering; each eye receives a different image, and the stereoscopic effect is achieved. Recent technology developments allow the manufacturing of LCD panels that include a lenticular lens layer, designed to show different images when viewed from slightly different angles. When used in combination with eye tracking by means of a stereo camera mounted on top of the display, the optical lens structure presents a unique, parallel image for each eye. This removes the need for shutter glasses to see content with dynamic perspective and parallax effect (https://www.dimenco.eu/). Fish tank VR systems afford a few practical advantages over HMDs that make them an attractive alternative. The term “fish tank virtual reality” is used to describe such systems because the experience is like looking at a fish tank inside which there is a virtual world [4]. As reported by Qi et al. [5], studies reveal that VR display technologies like fish tanks, also referred to as 3D displays [6], allow users to have better performance in terms of response time and accuracy in tasks involving judging the shape, density, connectivity, and other characteristics of objects in a virtual world. This is true for scientific visualization applications that allow users to carefully examine complex structures.

8.2.3

Handheld Displays

One of the first examples of using a mobile device as an interface to a virtual environment is the work of Fitzmaurice et al. [7]. Actually, the authors implemented their own handheld device. A 4-inch color monitor with an attached 6-DOF input device was used to navigate and interact through the virtual world. The small screen was the only “window” into the VE. The experiment showed the feasibility of using a handheld display in a VE and its advantages: user mobility and an intuitive way to interact with the computer.

168

8

Vision

The work of Watsen, Darken, and Capps [8] was one of the first implementations of a true handheld-based interface to a virtual environment. The authors implemented a 2D interface on a 3Com PalmPilotR to be used while standing inside a CAVE system. Basic scene and object manipulation were done through slider controls on the handheld. While this prototype confirmed the benefits of a mobile interface to VE, the CAVE opposed some problems to the use of a handheld. Being a full-immersion system, the CAVE made it difficult for the user to switch between the surrounding 3D world and the handheld. More recent research has continuously focused on the implementation of 2D GUIs for basic interaction: camera control and scene management (object picking, navigation through a simplified representation of the 3D environment) [9, 10]. Emphasis has been made on the portability and mobility of the GUI itself. Some other problems starting to be targeted are peer-to-peer communications between handhelds, together with client-server (handheld – main graphic workstation) interactions to control the VE and interact with other users at the same time [11]. In [12], Gutiérrez et al. describe a PDA-based interface for real-time control of virtual characters in multiuser semi-immersive virtual environments using a large rear-projection screen. The “mobile animator” eliminates the display of floating menus and other widgets over the simulation screen and shows the potential of handheld devices as powerful interfaces to VR applications (see Fig. 8.3b). Other authors still consider the PDA a virtual viewer. In the Touch-Space system [13], the PDA is optically tracked and used to display virtual objects superposed on the real world (augmented reality). Handheld devices represent a real alternative to head-attached devices for mobile applications. PDAs were the first iterations of the current smartphones, and as expected, the latest smartphones have made augmented reality applications available to the mass market. As is the case for PCs, the graphics capabilities of hand-held devices are clearly driven by the game industry. Vendors such as ATI and NVidia offered the first 3D graphics acceleration chips for mobile phones and PDAs (e.g., Dell’s Axim, see Fig. 8.3a) [14]. Current developments by Apple and Qualcomm

Fig. 8.3 Use of handheld devices in VR applications (a) 3D rendering on PDA (b) Mobile animator

8.2

Virtual Reality Displays

169

have allowed the deployment of advanced real-time 3D rendering, together with computer vision and sensor fusion tracking algorithms that ease the creation of mobile AR applications.

8.2.4

Large Projection Screens

This semi-immersive approach consists in positioning one or more users in front of a large rear-projection screen displaying the virtual world. Stereo glasses and 3D surround sound enhance the experience. This approach has gained popularity due to the possibilities it provides for direct interaction and communication between users. It has been observed that a semiimmersive VE improves the overall user experience [15]. Moreover, this kind of system can be used as a collaborative tool by design and development teams, making the implementation of a VE application more interactive and effective. Besides flat rear-projection screens, other systems have been designed to produce a more immersive experience, like a CAVE system. A representative example is the Allosphere, developed by the University of California, Santa Barbara [16]. This is a large-scale instrument for immersive visualization and simulation. It is one of the world’s largest immersive environments. The three-story-high cubical space comprises an anechoic chamber with a spherical display screen, 10 meters in diameter, surrounding from one to thirty users standing on a bridge structure. The Allosphere is equipped with high-resolution active stereo projectors, a complete 3D sound system with hundreds of speakers, and interaction technology. This system is positioned between CAVE environments, which give fully immersive experiences to a small number of users, and full-dome planetarium style theaters, which have a limited sense of immersion.

8.2.5

CAVE Systems

As mentioned in Chap. 1, the CAVE system was designed by Cruz-Neira et al. in 1992 [15, 17]. The name “CAVE” is both a recursive acronym (CAVE automatic virtual environment) and a reference to “the simile of the cave” found in Plato’s Republic, in which the philosopher discusses inferring reality (ideal forms) from projections (shadows) on the cave wall [18]. The main idea behind the CAVE project was to create a VR system without the common limitations of previous VR solutions, like poor image resolution, inability to share the experience directly with other users, and isolation from the real world. A head-tracking system is used to produce the correct stereo perspective. This allows the user to see her entire environment from the correct viewpoint, thus creating a compelling illusion of reality. Real and virtual objects are blended in the same space, and the user can see her body interacting with the environment.

170

8

Vision

Interest and development of CAVEs and spatially immersive displays (SIDs) have become increasingly significant. Commercial products by Barco (www.barco. com), and ARCANE (https://arcanetech.io/produit/vr-cave/ have been created. Despite the amount of development in this direction, both professional and customized solutions are still extremely expensive. Most of the available systems are close to the prototype stage and require very specific care and maintenance. A variety of software libraries have been developed to produce the individual images required. Some work has been done to adapt popular video games to work in CAVE environments [19]. There are also reduced versions of CAVE systems, with fewer walls or that are even transportable. In [20], Peternier et al. present a tutorial for building a low-cost CAVE with four screens (see Fig. 8.4).

Fig. 8.4 A CAVE system with four screens

References

171

References 1. Williams B, Narasimham G, Rump B, McNamara TP, Carr TH, Rieser J, Bodenheimer B (2007) Exploring large virtual environments with an HMD when physical space is limited. In: APGV ’07: Proceedings of the 4th symposium on applied perception in graphics and visualization. ACM Press, New York, pp 41–48 2. Langbehn E, Lubos P, Steinicke F (2018) Evaluation of locomotion techniques for roomscale VR: joystick, teleportation, and redirected walking. In: Proceedings of the virtual reality international conference – Laval Virtual (VRIC ’18). Association for Computing Machinery, New York, pp 1–9. Article 4 3. Ware C, Arthur K, Booth KS (1993) Fish tank virtual reality. In: CHI ’93: Proceedings of the INTERACT ’93 and CHI ’93 conference on human factors in computing systems. ACM Press, New York, pp 37–42 4. Arthur KW, Booth KS, Ware C (1993) Evaluating 3D task performance for fish tank virtual worlds. ACM Trans Inf Syst 11(3):239–265 5. Qi W, Russell I, Taylor M, Healey CG, Martens JB (2006) A comparison of immersive HMD, fish tank VR and fish tank with haptics displays for volume visualization. In: APGV ’06: Proceedings of the 3rd symposium on applied perception in graphics and visualization. ACM Press, New York. pp. 51–8 6. Pan X, Xu X, Dev S, Campbell AG (2021) 3D displays: their evolution, inherent challenges and future perspectives. In: Arai K (ed) Proceedings of the future technologies conference (FTC) 2021, volume 3. FTC 2021, Lecture notes in networks and systems, vol 360. Springer, Cham 7. Fitzmaurice GW, Zhai S, Chignell MH (1993) Virtual reality for palmtop computers. ACM Trans Inf Syst 11(3):197–218 8. Watsen K, Darken R, Capps W (1999) A handheld computer as an interaction device to a virtual environment. In: Proceedings of Third International Immersive Projection Technology Workshop (IPT 1999), Stuttgart 9. Hill L, Cruz-Neira C (2000) Palmtop interaction methods for immersive projection technology systems. In: Fourth international immersive projection technology workshop (IPT 2000) 10. Farella E, Brunelli D, Bonfigli M, Benini L, Ricco B (2002) Using palmtop computers and immersive virtual reality for cooperative archaeological analysis: the Appian way case study. In: International Conference on Virtual Systems and Multimedia (VSMM), Gyeongju 11. Farella E, Brunelli D, Bonfigli M, Benini L, Ricco B (2003) Multi-client cooperation and wireless PDA interaction in immersive virtual environment. In: Euromedia 2003 conference, Plymouth 12. Gutiérrez M, Lemoine P, Thalmann D, Vexo F (2004) Telerehabilitation: controlling haptic virtual environments through handheld interfaces. In: VRST ’04: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 195–200 13. Cheok A, Yang X, Ying Z, Billinghurst M, Kato H (2002) Touch-space: mixed reality game space based on ubiquitous, tangible, and social computing. Pers Ubiquit Comput 6:430–442 14. Bimber O, Raskar R (2007) Modern approaches to augmented reality. In: SIGGRAPH ’07: ACM SIGGRAPH 2007 courses. ACM Press, New York 15. Mine M (2003) Towards virtual reality for the masses: 10 years of research at Disney’s VR studio. In: EGVE ’03: Proceedings of the workshop on virtual environments 2003. ACM Press, New York, pp 11–17 16. Höllerer T, Kuchera-Morin J, Amatriain X (2007) The allosphere: a large-scale immersive surround-view instrument. In: EDT ’07: Proceedings of the 2007 workshop on emerging displays technologies. ACM Press, New York, p 3 17. Cruz-Neira C, Sandin DJ, DeFanti TA, Kenyon RV, Hart JC (1992) The CAVE: audio visual experience automatic virtual environment. Commun ACM 35(6):64–72 18. Cruz-Neira C, Sandin DJ, DeFanti TA (1993) Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In: SIGGRAPH ’93: Proceedings of the 20th

172

8

Vision

annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 135–142 19. Jacobson J, Hwang Z (2002) Unreal tournament for immersive interactive theater. Commun ACM 45(1):39–42 20. Peternier A, Cardin S, Vexo F, Thalmann D (2007) Practical design and implementation of a CAVE environment. In: International conference on computer graphics, theory and applications GRAPP, pp 129–136

Chapter 9

Audition

In this chapter, we discuss the second major sense in VR audition. Sound is important in nature and the real world in general. The lack of sounds in virtual worlds could be natural but could also make the virtual world poor and unnatural. This chapter explains how to produce natural sound in virtual worlds. Typical equipment for producing 3D sounds and recent development in immersive sound are explained.

9.1

The Need for Sound in VR

Sounds are part of our real-life experience and offer rich information about our environment. Sound perceived as coming from different spatial locations as we hear it in everyday life is also known as spatial sound. Spatial sound is a key aspect in producing realistic virtual environments. It has been shown that the combination of sound and graphics enhances the sense of presence [1]. Sound can play different roles in a virtual reality (VR) system. Complementary information: Sound allows us to provide additional and richer information about the simulated environment. Slight echoes and reverberations in the surrounding environment give the brain cues about the direction and distance of objects as well as information about the size of the environment. For example, an office has fewer echoes than a cathedral. The use of sound contributes to enhance the user awareness of the simulated space, for example, sounds coming from parts of the environment beyond the field of view let the user know that “there is more than meets the eye.” Sound can convey simulated properties of the entities within the virtual environment, for example, surface features (different sounds produced when touching soft or rough objects), weight, and impact force.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_9

173

174

9

Audition

Alternative feedback: Sound feedback can enhance user interfaces (e.g., sounds can indicate the reception of user commands or confirm selection of objects). Sound can help people with visual impairments. Alternative interaction modality: In the form of voice, sound can be an efficient communication channel. Although it is still not completely reliable, speech recognition has been recognized as a convenient input for human–computer interaction in VR (see the VIEW system in Sect. 1.3). Speech synthesis can be used as an alternative way to provide information to the user within a virtual environment.

9.2

Recording and Reproduction of Spatial Sound

There are three main methods for sound recording: monaural, stereo, and binaural. A monaural sound recording uses a single microphone. No sense of sound positioning is present in monaural sound. Stereo sound is recorded with two microphones separated by a certain distance. This gives a sense of the sound position as recorded by the microphones. Sound sources in stereo recordings are often perceived as being positioned inside the listener’s head. This is because humans do not normally hear sounds in the manner they are recorded in stereo, separated by empty space. When we hear a sound, our head acts as a filter. Binaural recordings sound more realistic because they are recorded in a way that resembles the human acoustic system. The microphones are embedded in a dummy head. Binaural recordings sound closer to what humans hear in the real world, as the dummy head filters sound in a manner similar to the human head.

9.3

Synthesis of Spatial Sound

Spatial sound synthesis consists in processing sound signals to produce realistic audio, considering individual sound signals and parameters describing the sound scene: position and orientation of each source and acoustic characterization of the room or space. Several techniques can be applied, as discussed in the next subsections.

9.3.1

Sound Rendering

Sound rendering is mostly used to generate synchronized soundtracks for animation. This method creates spatial sound by attaching a characteristic sound to each object in the scene. Sound sources can come from sampling or artificial synthesis.

9.3

Synthesis of Spatial Sound

175

The technique is a pipelined process of four stages: • Generation of each object’s characteristic sound (recorded, synthesized, modal analysis collisions) • Sound instantiation and attachment to moving objects within the scene • Calculation of the necessary convolutions to describe the sound source interaction within the acoustic environment • Convolutions are applied to the attached instantiated sound sources The technique exploits the similarity of light and sound to provide the necessary convolutions. A sound source in space propagates sound waves in all directions. Sound waves can be reflected and refracted due to the acoustic environment. A sound wave interacts with many objects in the environment as it makes its way to the listener. The final sound that the listener hears is the integral of the signals from the multiple simultaneous paths existing between the sound source and the listener. Sound rendering is very similar to raytracing and has a unique approach to handling reverberation. But just as with raytracing, it is difficult to use in real-time applications.

9.3.2

Head-Related Transfer Function

Between a sound source and your eardrum (the thin membrane that transmits sound from the air to the ossicles inside the middle ear), there are a few obstructions that cause reflections and absorption of the sound signal. Obstructions include shadowing and boundary effects of the head, reflections off the pinna (visible part of the ear) and shoulders, and resonance in the ear canal. The signal that reaches your eardrum includes all the effects caused by these components and more. The combination of these effects depends on the position of the sound source in space and the orientation of the head. The sound arriving at your two ears will be different unless the sound source is on the median plane (plane dividing your body into right and left sides). If we consider the effect of the body on the sound signal arriving from a given direction and with a given head rotation as a filter, then we can measure the transfer function of that filter. This measurement is called a head-related transfer function or HRTF and consists in recording sounds with tiny probe microphones in the ears of a person. These recordings are then compared with the original sounds to compute the person’s HRTF. The HRTF is a function of four variables: three space coordinates of the sound source and frequency. For sound sources situated at more than 1 m, the source is said to be in the far field, and the HRTF falls off inversely with range. Most HRTF measurements are made in the far field, reducing the HRTF to a function of azimuth, elevation and frequency. Once the HRTF for the left ear and the right ear are calculated, accurate binaural signals can be synthesized from a monaural source. The computations required by

176

9

Audition

the HRTF are so demanding that they currently cannot be performed in real time without special hardware. The Convolvotron1 is a DSP engine for this purpose. Dimensionality reduction techniques could be applied to accelerate the computation of the HRTF in software, but this is still a work in progress [3].

9.3.3

3D Sound Imaging

The 3D sound imaging technique consists in approximating binaural spatial audio through the interaction of a 3D environment simulation. The following steps are required: 1. Compute line-of-sight information between the virtual user and the sound sources. 2. The sounds emitted by these sources are processed based on their location, using some software DSP algorithms or simple audio effects modules with delay, filter, pan, and reverb capabilities. 3. A headphone set is used to play the final stereo sound sample, according to the user’s position. 3D sound imaging is suitable for simple VR systems where a sense of space is more important than high accuracy in the localization of sound sources.

9.3.4

Utilization of Loudspeaker Location

Utilization of loudspeaker location does not attempt to simulate many of the human localization cues,2 instead it focuses on attaching sampled sounds to objects in 3D space. Visual Synthesis Incorporated’s Audio Image Sound Cube uses this approach with eight speakers to simulate spatial sound. Two speakers are in each corner of the cube, one up high and one down low. Pitch and volume of the sampled sounds are used to simulate sound location; volume is distributed through the speakers appropriately to give the perception of a sound source’s spatial location. This technique is less accurate than convolving sound, but it is fast and efficient, making it a good choice for less-expensive real-time spatial sound systems.

1

Crystal River Engineering and the NASA Ames Research Center designed a special digital signal processing (DSP) engine called Convolvotron, which computes 128 convolution points in parallel [2]. 2 Interaural time difference, head shadow, pinna response, shoulder echo, head movement, reverberation, vision. See [4].

9.4

Sound Systems for VR

9.4

177

Sound Systems for VR

The feeling of being there is considered a prerequisite for the effectiveness of virtual reality (VR). (Non)availability of auditory stimulation has been assessed in several studies [5], but because the study design differs, it is difficult to generalize the results and estimate the effect of the auditory component (e.g., the types of sounds included, the rendering technology employed). Virtual reality applications need an audio rendering pipeline that must meet several requirements, as explained in [6]: 3D localization: Accurate 3D positioning of the virtual sound source. The perceived position of the sound source should match the spatial location of the associated object in the virtual environment. Acoustics simulation: Acoustic room simulation is essential to perceive spatial properties of the virtual environment such as size of the room and reflection properties of the walls. Speed and efficiency: There is a trade-off between accurately simulating the physical properties of spatial sound and the efficient generation of sound in real time. Moreover, realistic virtual environments require a certain number of simultaneous virtual sound sources. The most important acoustic phenomena can be simulated in software using sound engines. Simplified models of physical properties of spatial sound can produce convincing results. One of the challenges is to map sound sources at arbitrary virtual positions to a limited number of speakers, whose real position is often constrained by the physical setup of the VR installation. Using LCD shutter glasses and a VR-CUBE with 3D graphics displays on all four walls and the floor, Larsson et al. [7] presented a virtual representation of a church in Gothenburg (Sweden). Virtual sound was organ music, which was reproduced by eight active loudspeakers in each corner of the VR cube to achieve surround sound. The task for the 30 participants (26 males, 4 females) was to navigate through five numbered cubes in the correct order.

9.4.1

Sound Hardware

Electroacoustic Playback Spatial sound can be generated using electroacoustic devices: headphones or loudspeakers. Headphones allow more precise control of the spatial cues because the signals reaching the two ears can be controlled independently. An additional advantage is that there is no indirect sound, such as echoes or reverberation.

178

9

Audition

Headphones can be more expensive than loudspeakers and may be impractical for applications in which the user does not want to wear a device on the head, as in collaborative virtual environments using a CAVE. It is more difficult to control the spatial information reaching the listener in a loudspeaker-based system. However, these systems are relatively simple and inexpensive to implement and do not interfere with the user. Multiple Loudspeaker Systems The total acoustic signal that reaches each ear is the sum of the signals that can be perceived by that ear coming from each sound source in an environment. Based on this property, spatial auditory cues such as ITD3 and IID4 can be modulated by controlling the audio played from multiple speakers situated around the listener. The signals at the two ears cannot be independently manipulated as is the case with headphones. Consequently, it is more difficult to control the binaural signal reaching the user, to reproduce the sound that would be perceived in a real scene. However, there are methods to specify the signals played from each loudspeaker to simulate spatial auditory cues. The accuracy of spatial sound generation using loudspeakers depends on the number of speakers used. If we put an infinite number of speakers around the listener, spatial sound generation would be reduced to simply playing the desired signal from the loudspeaker at the desired location of the source to achieve a “perfect” reproduction. Panning between multiple pairs of loudspeakers arrayed in front of and behind the listener is often used to improve spatial sound simulations. These “surround-sound” technologies are popular in the entertainment industry and are implemented via a variety of formats. One of the first surround-sound systems was the one created for the movie Fantasia in 1939. “Fantasound” was a three-channel system whose loudspeakers were in front of the listener at left, middle, and right. Approximately 54 speakers surrounded the audience and carried a mixture of sound from the left and right front speakers. It was not true stereo. Few theaters were ever equipped to play “Fantasound,” because it was very expensive. A representative example of surround-sound systems is the Pioneer Sound Field Control System (PSFC); see Fig. 9.1. The PSFC is a DSP-driven hemispherical 14-loudspeaker array. It was installed at the University of Aizu Multimedia Center. The system could control an entire sound field, including sound direction, virtual

3 Depending on the angle between the interaural axis and a sound source, one ear may receive the sound earlier than the other. The resulting interaural time differences (ITDs) are the main cue indicating the laterality (left/right location) of the direct sound [4]. 4 At the high end of the audible frequency range, the head of the listener reflects and diffracts signals so that less acoustic energy reaches the far side of the head (causing an “acoustic head shadow”). Due to the acoustic head shadow, the relative intensity of the sound at the two ears varies with the lateral location of the source. The resulting interaural intensity differences (IIDs) generally increase with source frequency and the angle between the source and the median plane. IIDs are perceptually important for determining source laterality for frequencies above about 2 kHz [8].

9.4

Sound Systems for VR

179

LOUDSPEAKERS

RS-232C

WORK STATION

SOURCE 1

SOURCE 2

A D

DASP UNIT

D A

PERSONAL COMPUTER DATA IF

CONTROL DATA

Fig. 9.1 Pioneer sound field control system

distance, and simulated environment for each source. The PSFC speaker dome is about 14 m in diameter, allowing about 20 users at once to comfortably stand. Nowadays, one of the most common surround-sound formats is the 5.1 speaker system, where speakers are located at the left, middle, and right in front and left and right behind the listener. The 0.1 speaker is a subwoofer, used to reproduce bass audio frequencies, which are difficult to play with conventional loudspeakers. The middle speaker, located in front of the listener, reproduces most of the speech information. Typical 5.1 surround formats are Dolby Digital Surround and Digital Theater Systems (DTS). Newer surround-sound formats include the Dolby Digital Surround EX, a 6.1speaker system that adds a center speaker behind the listener. Another format is the Sony Dynamic Digital Sound (SDDS) system, which is a 7.1- speaker system adding a left-center and right-center speaker in front of the listener. Surround-sound technologies provide a rudimentary sense of location.

180

9

Audition

Sound Cards Most PC systems have sound cards capable of at least stereo playback. In the consumer hardware category, we find multichannel sound cards for 5.1, 6.1, and 7.1 surround sound. Some of the main vendors are: Creative, which overtook the OpenAL project in 2003 and is the developer of EAX,5 Terratec, and Guillemot/ Hercules. Consumer hardware sound cards have limited performance but good native 3D sound engines. Their development is driven by the gaming industry and thus they are continuously updated. Some cards have limited audio quality. Professional hardware includes special sound cards for custom multichannel sounds, they are mostly DSP boards. Their signal processing capabilities allow for creating custom 3D sound engines. Professional sound cards have very good performance and are mainly used in 3D sound research. Representative vendors include: RME and M-Audio.

9.4.2

Sound Engines

Also known as sound libraries or APIs, sound engines are software components that allow us to interface with the sound card and/or implement 3D sound generation in software. 3D sound engines are the audio counterparts of graphics APIs like OpenGL and Direct3D (the graphics component of Microsoft’s DirectX API). Some of the most powerful 3D sound engines are OpenAL, fmod, Miles, and DirectSound. OpenAL, the Open Audio Library is a free software cross-platform audio API. It is designed for efficient rendering of multichannel 3D positional audio. The library models a collection of audio sources moving in a 3D space that are heard by a single listener somewhere in that space. Its style and conventions are very similar to those of OpenGL. The fmod music and sound effects system is a library that supports multiple platforms, including game consoles. It is a commercial sound engine that can work with multiple sound file formats, such as MP3. The Miles Sound System (https://en.wikipedia.org/wiki/Miles_Sound_System) is very popular in the game industry. It is a commercial product that supports multiple platforms and multiple sound file formats. DirectSound is the audio component of Microsoft’s DirectX API. It only supports Windows and the Xbox game console. Most of the video games for the Windows platform use this sound engine.

5

The Environmental Audio eXtensions are a number of digital signal processing presets for audio accessible from the DirectSound API. The latest-version, EAX HD 5.0, supports 128 simultaneous sound sources with up to four effects applied to each source.

References

9.5

181

Immersive Sound

Immersive sound (or immersive surround) refers to any of the new “3D” surround sound formats with height speakers to enhance the listening experience. AURO-3D was the first immersive sound format in theaters (2011). AURO-3D was introduced to the US home market in December 2014. Auro-3D consists of the Auro-3D Engine and the Creative Tool Suite. In the engine, legacy content is converted to Auro-3D using Auro-Codec and Auro-Matic upmixing algorithm. A set of plugins called the Creative Tool Suite can be used to create native immersive 3D audio content. The Auro-3D format is fully compatible with all existing theater systems and production processes, and the format also offers many compatibility features. The Dolby Atmos system was introduced in 2012, but more than 230 titles have been announced or released since then. As of August 2014, Dolby Atmos became available on mass market home theater processors and receivers in the US for the first time in any real numbers. The Dolby Atmos surround sound technology is a product of Dolby Laboratories. With height channels, existing surround sound systems can be expanded to interpret sounds as three-dimensional objects without horizontal or vertical limitation. The DTS:X format is the third major immersive sound format to hit the market. In DTS:X, “objects” (audio tracks) can be located for each speaker by specifying polar coordinates. Depending on the number and position of speakers available, the audio processor dynamically renders sound output. Since DTS:X is an object-based encoding system, sounds can be placed with great precision within the soundstage and can be rendered differently depending on the specific home implementation. In addition, object-based sound formats enable the end user to create custom playback mixes for live and recorded content. Immersive Sound is either channel-based (content is mixed for a specific predefined speaker layout) or object-based (sounds are stored with meta data that defines their location in space and speed and direction if moving). Dolby Atmos and DTS:X are object-based, while AURO-3D is channel-based.

References 1. Larsson P, Vstfjll D, Kleiner M (2002) Better presence and performance in virtual environments by improved binaural sound rendering. In: Proceedings of the AES 22nd international conference on virtual, synthetic and entertainment audio, pp 31–38 2. Jacobson J, Hwang Z (2002) Unreal tournament for immersive interactive theater. Commun ACM 45(1):39–42 3. Burgess DA (1992) Techniques for low cost spatial audio. In: UIST ’92: Proceedings of the 5th annual ACM symposium on user interface software and technology. ACM Press, New York, pp 53–59

182

9

Audition

4. Naef M, Staadt O, Gross M (2002) Spatialized audio rendering for immersive virtual environments. In: VRST ’02: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 65–72 5. Kapralos B, Mekuz N (2007) Application of dimensionality reduction techniques to HRTFS for interactive virtual environments. In: ACE ’07: Proceedings of the international conference on advances in computer entertainment technology. ACM Press, New York, pp 256–257 6. Kern AC, Ellermeier W (2020) Audio in VR: effects of a soundscape and movement-triggered step sounds on presence. Front Robot AI Sect Virt Environ 7 7. Larsson P, Västfjäll D, Olsson P, Kleiner M (2007) When what you hear is what you see: presence and auditory-visual integration in virtual environments. In: Proceedings of the 10th annual international workshop on presence (Barcelona), pp 11–18 8. Shilling RD, Shinn-Cunningham B (2002) Handbook of virtual environments: design, implementation, and applications. Lawrence Erlbaum Associates, Mahwah, pp 65–92. Chap. Virtual Auditory Displays

Chapter 10

Touch

Another very important human sense is touch. This chapter discusses the tactile, kinesthetic, and haptics senses. It presents the various hardware systems to try to reproduce the sensation of touch in Virtual Worlds.

10.1

The Need for Touch in VR

In general, virtual reality (VR) applications focus on providing visual and auditory feedback and much less on touch or force-feedback. It is easier to develop visual virtual worlds with basic sound effects; it requires less specialized equipment. However, the human haptic system, which involves the tactile and kinesthetic senses, is a very important perception channel. It has been shown that the ability to “touch” virtual objects increases the sense of presence [1]. The term haptic comes from the Greek word haptesthai, which refers to the sense of touch as the way in which the human body perceives objects and space. Touch is one of the most important human senses; it can be divided into cutaneous, kinesthetic, and haptic systems. The cutaneous system is composed of mechanoreceptors embedded in the skin. It constitutes the tactile sense, which processes any stimulation on the body surface, the skin. The kinesthetic system is composed of receptors located in the muscles, tendons, and joints. It constitutes the kinesthetic sense, which allows us to be aware of our limb positions and movements and muscle tensions. It is also called proprioception, from the Latin proprius, meaning “one’s own.” An example of proprioception is when you touch your nose with closed eyes. To perform such a movement, you rely on the kinesthetic sense to know where your fingertip is relative to your nose. In this and most of the cases, the tactile sense is also involved; the fingertip touches the nose. However, the information that is determinant to the correct execution of the action comes primarily from the kinesthetic

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_10

183

184

10 Touch

sense. Thus, the distinction between tactile and kinesthetic senses is functional and task-dependent [2]. The haptic sensory system can be considered the combination of both cutaneous and kinesthetic systems. In [3], Robles-De-La-Torre talks about some real-life cases that show how vision cannot fully compensate for the major loss of somesthesis – sense of touch – due to injury or disease. The author discusses the importance of the sense of touch and why haptic feedback enhances virtual environments. Haptic feedback has different modalities: force, tactile, and proprioceptive feedback. Force-feedback integrated in a VR simulation helps the user estimate the hardness, weight, and inertia of virtual objects. Tactile feedback provides a feel of the object’s surface contact geometry, smoothness, temperature, etc. Proprioceptive feedback helps to estimate the relative position of body limbs – body posture within the virtual environment.

10.2

Data Gloves

The hand is one of the main input channels concerning touch. As we have explained, the whole body is able to receive and process haptic feedback, but our hands are the main interface to touch and manipulate our external environment. One of the first interaction devices created to take advantage of hand gestures was the data glove. See Chap. 1 for additional information on the role that glove-based interfaces play in virtual reality applications. Various sensor technologies are used to capture physical data such as bending fingers. Often a motion tracker (magnetic or inertial) is attached to capture the global position/rotation data of the glove. These movements are then interpreted by the software that accompanies the glove. Gestures can be categorized into useful information, such as recognizing sign language or other symbolic functions. Some devices, like Immersion’s CyberTouch™ (Fig. 10.1a) and CyberGrasp™ (Fig. 10.1b), can also provide vibrotactile and force-feedback, respectively (see Sects. 10.4.1 and 10.4.3). This allows a data glove to also be used as an output device.

10.3

Haptic Rendering

As defined by Salisbury et al. [4], haptic rendering is the process of computing and generating forces in response to user interactions with virtual objects. We can distinguish two main types of haptic rendering algorithms:

10.3

Haptic Rendering

185

Fig. 10.1 Data gloves for touch and force-feedback. (a) CyberTouch™. (b) CyberGrasp™

3-DOF haptic rendering: these algorithms consider the paradigm of touching virtual objects with a single contact point, which has only three degrees of freedom (3 DOF); only the position in 3D space can be modified. 6-DOF haptic rendering: these algorithms are designed to render the forces and torques arising from the interaction of two virtual objects. In this case, the grasped object has six degrees of freedom: position and orientation in 3D space. The haptic feedback consists of 3D force and torque. Many everyday-life tasks such as eating with a fork or writing with a pen imply the manipulation of objects in 3D space. In such tasks, we feel the interaction with other objects. These kinds of activities can be simulated with 6-DOF object manipulation with force-and-torque feedback.

10.3.1

History of Haptic Rendering

Haptic rendering can be traced back to 1965 when Ivan Sutherland proposed a multimodal display that would incorporate haptic feedback into interaction with virtual worlds. By that time, haptic feedback had already been used in flight simulators and master-slave robotic teleoperation. In 1954, at the Argonne National Laboratories, Goertz and Thompson developed the ARM (Argonne Remote Manipulator), an electrical servomechanism that received feedback signals from sensors mounted on a slave robot and applied forces to the master manipulator. The ARM was used in the GROPE project to address the synthesis of force-feedback from simulated interactions (see Sect. 1.3). In 1980, Bejczy and Salisbury [5] developed a computer-based Cartesian control for teleoperator systems. Cartesian control was applied to the manipulation of simulated slave robots [6]. The use of tactile feedback in VR was pioneered by

186

10 Touch

Fig. 10.2 PHANTOM haptic interfaces

several researchers at MIT. Patrick [7] used voice coils (those found in the cone of a loudspeaker) to provide vibrations at the fingertips of a user wearing a Dextrous Hand Master Exoskeleton. The “Sand-paper,” developed by Minksy et al. [8], was a tactile joystick that mapped image texels to vibrations, allowing one to feel textures. Burdea et al. [9] at Rutgers University developed a light and portable force-feedback glove called the Rutgers Master. In 1994, Massie and Salisbury [10] designed the PHANTOM (Personal HAptic iNTerface Mechanism), a stylus-based haptic interface that was later commercialized and has since become one of the most commonly used force-feedback devices (see Fig. 10.2). Other commercial force-feedback interfaces are the Impulse Engine, created in 1995 [11] and the CyberGrasp™ glove, which appeared in 1998 and was developed by Virtual Technologies (acquired by Immersion, see Sect. 1.3). Inexpensive haptic joysticks for video games became available in the late 1990s.

10.4 Haptic Interfaces Haptic interfaces generate mechanical signals that stimulate human kinesthetic and touch channels. Tactile and kinesthetic channels let us perceive and act in our environment [12]. The objective of haptic interfaces is to facilitate the manipulation of virtual objects by reproducing their physical characteristics. Objects in nature can be roughly classified as inert or active. Inert objects are unanimated; they only dissipate mechanical energy. In the case of active objects, they are animated, and they can supply some energy. Thus, there can be two types of haptic interfaces: passive or active. Passive haptic interfaces can only exert resistance forces against the user’s movement. These devices dissipate the mechanical energy (force) exerted by the

10.4

Haptic Interfaces

187

user during the interaction with virtual objects. This kind of haptic interface is inherently safe because of the passive working of the device. The user will never experience accidental forces or torques exerted by a passive haptic interface because the user is the only energy supplier in the human-machine system. User energy is redirected or dissipated through clutches and brakes. Other passive haptic interfaces utilize variable transmissions to redirect user-supplied energy or to reduce the degrees of freedom of a device to follow a specified path. Active haptic interfaces utilize actuators that are capable of supplying energy. Such actuators can be electric motors, hydraulic systems, piezoelectric, etc. Active systems pose potential stability and safety issues when simulating stiff environments. Hybrid haptic interfaces use both active and passive actuators [13]. Active, passive, and hybrid haptic interfaces can be considered as displays that provide different kinds of stimulation to the human haptic system. Depending on the type of feedback they provide, they can be classified into vibrotactile, tactile, and kinesthetic displays.

10.4.1

Vibrotactile Displays

Vibrotactile displays are composed of a single vibrating component used to encode information in temporal parameters of the vibration signal, such as frequency, amplitude, waveform, or duration. Vibration can be generated by means of solenoids, voice coils, or electric motors like the ones used in mobile phones. Vibrotactile information has been successfully used in video games, providing touch feedback in handheld controllers. Tactile sensations are used to represent impacts or firing of weapons, or to encode abstract information such as progress bars or tactile “radars.” Current research is focused on encoding more complex information in the vibrotactile signal and on combining this type of feedback with other sensory modalities. For example, in [14] Cardin et al. present a vibrotactile system integrated in the clothes of an aircraft pilot. The device sends vibrotactile feedback to the pilot when the aircraft becomes off-balance. The system also dynamically localizes the position of the actuators to ensure accurate feedback independently from the pilot’s posture and movements. In [15], the authors developed a vibrotactile system that provides navigation cues (obstacle detection and warning) for visually impaired people. Cassinelli et al. [16] present a vibrotactile interface with similar applications: augmentation of spatial awareness in hazardous working environments, as well as enhanced obstacle awareness for car drivers.

188

10.4.2

10 Touch

Tactile Displays

Tactile displays can be classified into three types of stimulation: electrotactile, thermal, and mechanical.

Electrotactile Displays Electrotactile (electrocutaneous) displays create touch sensations by passing a small electric current through the skin. Their main advantage is the absence of moving parts, which facilitates their construction, control, and maintenance. The main problem with this technology is the relatively small difference in the thresholds between signal detection and the onset of pain. In [17], Yamamoto et al. present an application of electrotactile (electrostatic) displays to telepresentation systems.

Thermal Displays Thermal feedback is another concept in haptic research. This kind of display can assist in object identification in virtual environments by simulating the thermal cues associated with contacting materials with different thermal properties. Thermal displays consist of thermal stimulators, thermal sensors, and a temperature-control system that monitors and controls the surface temperature of the thermal display. A crucial element in developing a thermal display is to characterize the thermal responses that occur on the hand as it contacts a material. In most thermal displays the simulation of a virtual material in contact with the skin has been based on empirically derived data of the changes in skin temperature as the hand contacts the corresponding real material [13, 18, 19]. The main problems faced by this type of display are the lack of an actuator of suitable bandwidth and the fact that human thermal sensing is relatively little understood.

Mechanical Displays Mechanical displays are used to convey spatial information to the skin. Tactile displays attempt to reproduce the small-scale skin deformations that occur during finger-tip interactions with real objects. The initial interest on this kind of interface was motivated by sensory substitution devices for visually impaired people. Current research also addresses applications in virtual environments, telemedicine, minimally invasive surgery, and virtual training. The most common design approach uses arrays of moving pins or “tactors” that contact the skin to approximate arbitrary shapes [20, 21].

10.4

Haptic Interfaces

189

There is a second category of mechanical tactile displays: vibration displays. Vibration displays present shape information by activating patterns of spatially configured transducers at high temporal frequencies (200–250 Hz) of operation. These transducers are much smaller than those used by vibrotactile displays; they can be pins or wires driven by solenoids or piezoelectric actuators. They have been used to convey abstract forms such as letter shapes for sensory substitution applications [22, 23]. A novel mechanical display that provides tactile feedback is the “gravity grabber” developed by Minamizawa et al. [24]. This is a wearable haptic display that applies pressure to the fingerpads to provide the sensation of grasping objects of variable weight. The device is based on the insight that the deformation on fingerpads produces a reliable weight sensation even when the proprioceptive sensation is absent. The wearable haptic display uses dual motors and a belt to produce deformations on the fingerpads of the thumb and index, allowing users to perceive grip force, gravity, and inertia of a virtual object.

10.4.3

Kinesthetic Displays

Kinesthetic feedback can be provided by devices exerting forces to simulate the texture and other high-spatial-frequency details of virtual objects. Various types of force-feedback devices have been designed: single-point interaction, exoskeletons, and haptic surfaces.

Single-Point Force-Feedback Single-point force-feedback is the most common method for rendering haptic sensations with force-feedback devices. Using this kind of interface, the user is represented in the virtual environment by a single point corresponding to the tip of a probe through which the user interacts with the device. This follows the metaphor of exploring objects remotely through an intermediary link, such as a stick. The PHANTOM device, a 3-DOF pantograph cited in Sect. 10.3.1 (Fig. 10.2), is one of the best-known examples of single point force-feedback devices. Other 3-DOF and 6-DOF pantographs, such as the Haptic Master, developed by Iwata et al., are described in [25]. Solazzi et al. [26] have developed a haptic interface that combines a fingerpad (tactile) interface and a kinesthetic interface. Kinesthetic feedback is provided by a 3-DOF manipulator. Tactile feedback is obtained by a plate that can contact the fingertip with different orientations, according to the direction of the surface normal of the virtual object being examined. An interesting single-point haptic device is the SPIDAR (SPace Interface Device for Artificial Reality); its development started in 1989 by Sato [27] and has been continuously improved since then. The SPIDARG&G [28], one of the latest versions, consists of a pair of string-based 6-DOF

190

10 Touch

haptic devices, named SPIDAR-G. By grasping a special grip on each device, the user can interact with virtual objects using both hands. The grip is used to track hand movement and to grasp or release virtual objects. Each hand has its own workspace and can feel different haptic sensations such as contact, weight, or inertia. Single-point haptic devices can have interesting applications in 2D environments. This is the case of the 2-DOF planar parallel manipulator developed by Solis et al. [29]. This haptic interface is an example of reactive robot technology, cooperative systems that reproduce and simulate human actions. The haptic interface is used to teach people to write Japanese characters. Following the same principle, the interface has been used to assist on design and sketching tasks [30, 31].

Exoskeletons An exoskeleton is a set of actuators attached to a hand, arm, or body. They have been commonly used in robotics as master-manipulators for teleoperation, for example, the ARM device used in project GROPE (see Sect. 9.3.1). One of the first examples of a compact exoskeleton suitable for desktop use was developed by Iwata in 1990 [32]. Burdea et al. [9] developed a lightweight portable exoskeleton that uses pneumatic cylinders to apply force to the finger-tips. A popular commercial exoskeleton is the CyberGraspTMTM, also mentioned in Sect. 9.3.1. Recent improvements include a new version that allows two-hand interaction, the Haptic WorkstationTM, commercialized by Immersion (see Fig. 10.3). TheHapticWorkstationTM combines several peripherals: CyberGloveTM, CyberGraspTM, CyberTrackTM, and CyberForceTM, A pair of CyberGloveTM is used to acquire the hand’s posture. On each hand, 22 angles are evaluated, on all phalanges and finger abducts, making it possible to have a good representation of the current hand posture. A pair of CyberGraspTM exoskeletons of the hand can pull each finger separately using wires. This device cannot constrain the user to close his fingers. The CyberGraspTM can be used to simulate object grasping. The exoskeleton prevents it from closing the fist by applying a force of 10 N maximum. Two

Fig. 10.3 The Haptic Workstation™

10.4

Haptic Interfaces

191

CyberTracksTM, encapsulated in the CyberForceTM exoskeletons, allow gathering the position and the orientation of the wrist. The precision and the refresh rate (1 kHz) are higher than using a magnetic or optical system. For positioning, the accuracy is around a tenth of a millimeter and around a tenth of a degree for orientation. The CyberForceTM can be used to apply a 3D force of up to 80 N on the wrist. This is a 3-DOF device because it cannot be used to constrain hand orientation. In combination with an HMD, the Haptic WorkstationTM has been used in telerehabilitation [33], 3D modeling of organic shapes [34], and teleoperation applications [35]. Other commercial haptic interfaces are the 3-DOF and 6-DOF devices developed by Force Dimension (http://www.forcedimension.com). One of the most recent advances in exoskeleton developments is the device presented by Frisoli et al. [36]. This is a tendon-driven wearable arm exoskeleton with five DOFs. The interface is lightweight and can be used for simulating touch by a hand of large objects. In one of its possible configurations, it can be used in combination with a hand exoskeleton that can apply forces on the thumb and index fingers of the right hand. This way it can provide force-feedback on three points: palm, forearm, and the fingers. Bergamasco et al. [37] described the use of a previous version of this exoskeleton in cultural heritage applications using CAVE systems, the “museum of pure form.” This type of exoskeleton minimizes visual interference and maximizes the workspace. Other applications include rehabilitation or simulation of industrial assembly and maintenance operations.

Haptic Surfaces Haptic surfaces are also known as shape displays. The main idea is to directly create 3D physical shapes out of a modifiable surface. This kind of device moves or deforms to simulate the shapes of virtual objects. One of the first shape displays was FEELEX [38]. This device was composed of a flexible silicon screen, an array of actuators and a projector. The flexible screen was deformed by the actuators to simulate the shape of virtual objects. An image of the virtual object was projected onto the surface of the flexible screen and synchronized with the actuators. This configuration allowed the user to touch and feel the projected images with his hands. Lumen [39] is a 13 × 13-pixel bitmap display where each pixel can physically move up and down. The display can present 2D images and moving physical shapes that can be touched with the hands. A 2D position sensor lets users input commands and manipulate shapes with their hands. Haptic surfaces are still rather primitive but, in the future, could be used to create on-the-fly tangible user interfaces where the components are not physical objects but dynamically created shapes. A review of other shape displays that do not qualify as haptic devices, since they cannot be touched, is presented in [40].

192

10 Touch

References 1. Insko B (2001) Passive hapics significantly enhance virtual environments. PhD thesis, University of North Carolina, Department of Computer Science 2. Tan HZ, Pentland A (2001) Fundamentals of wearable computers and augmented reality. Lawrence Erlbaum Associates, Mahwah, pp 579–598. Chap. Tactual Displays for Sensory Substitution and Wearable Computers 3. Robles-De-La-Torre G (2006) The importance of the sense of touch in virtual and real environments. IEEE MultiMedia 13(3):24–30 4. Salisbury K, Brock D, Massie T, Swarup N, Zilles C (1995) Haptic rendering: programming touch interaction with virtual objects. In: SI3D ’95: Proceedings of the 1995 symposium on interactive 3D graphics. ACM Press, New York, pp 123–130 5. Bejczy A, Salisbury JK (1980) Kinematic coupling between operator and remote manipulator. Adv Comput Technol 1:197–211 6. Kim W, Bejczy A (1991) Graphical displays for operator aid in telemanipulation. In: Proceedings of the IEEE international conference on systems, man and cybernetics, pp 1059–1067 7. Patrick N (1990) Design, construction, and testing of a fingertip tactile display for interaction with virtual and remote environments. Master’s thesis Department of Mechanical Engineering, MIT Press 8. Minsky M, Young Ming O, Steele O, Brooks FP, Behensky M (1990) Feeling and seeing: issues in force display. In: SI3D ’90: Proceedings of the 1990 symposium on Interactive 3D graphics. ACM Press, New York, pp 235–241 9. Burdea G, Zhuang J, Roskos E, Silver D, Langrana N (1992) A portable dextrous master with force feedback. Presence Teleop Virt Environ 1(1):18–27 10. Massie T, Salisbury JK (1994) The PHANTOM haptic interface: a device for probing virtual objects. In: Proceedings of the ASME winter annual meeting: symposium on haptic interfaces for virtual environment and teleoperator systems, pp 295–300 11. Jackson B, Rosenberg L (1995) Interactive technology and the new paradigm for healthcare. IOS Press, pp 147–151. Chap. Force Feedback and Medical Simulation 12. Hayward V, Astley OR, Cruz-Hernandez M, Grant D, Robles-De-La-Torre G (2004) Haptic interfaces and devices. Sens Rev 4(1):16–29 13. Caldwell D, Gosney C (1993) Enhanced tactile feedback (tele-taction) using a multi-functional sensory system. In: Proceedings of the IEEE international conference on robotics and automation, pp 955–960 14. Cardin S, Vexo F, Thalmann D (2006) Vibro-tactile interface for enhancing piloting abilities during long term flight. J Robot Mechatron 18(4):381–391 15. Cardin S, Vexo F, Thalmann D (2006) Wearable system for mobility improvement of visually impaired people. Visual Comput J 23:109–118 16. Cassinelli A, Reynolds C, Ishikawa M (2006) Haptic radar/extended skin project. In: SIGGRAPH ’06: ACM SIGGRAPH 2006 sketches. ACM Press, New York, p 34 17. Yamamoto A, Nagasawa S, Yamamoto H, Higuchitokyo T (2004) Electrostatic tactile display with thin film slider and its application to tactile telepresentation systems. In: VRST ’04: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 209–216 18. Ho HN, Jones LA (2007) Development and evaluation of a thermal display for material identification and discrimination. ACM Trans Appl Percept 4(2):13 19. Caldwell D, Lawther S, Wardle A (1996) Tactile perception and its application to the design of multi-modal cutaneous feedback systems. In: Proceedings of the IEEE international conference on robotics and automation, pp 3215–3221 20. Wall SA, Brewster S (2006) Sensory substitution using tactile pin arrays: human factors, technology and applications. Signal Process 86(12):3674–3695

References

193

21. Howe RD (2004) The shape of things to come: pin-based tactile shape displays. In: Proceedings of the 4th international conference eurohaptics 2004, Munich, 5–7 June 2004, vol 1996, pp 2–11 22. Summers IR, Chanter CM (2002) A broadband tactile array on the fingertip. J Acoust Soc Am 112(5):2118–2126 23. Bliss JC, Katcher MH, Rogers CH, Shepard RP (1970) Optical-to-tactile image conversion for the blind. IEEE Trans Man Mach Syst 11(1):58–65 24. Minamizawa K, Fukamachi S, Kajimoto H, Kawakami N, Tachi S (2007) Gravity grabber: wearable haptic display to present virtual mass sensation. In: SIGGRAPH ’07: ACM SIGGRAPH 2007 emerging technologies. ACM Press, New York, p 8 25. Iwata H (2005) Art and technology in interface devices. In: VRST ’05: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 1–7 26. Solazzi M, Frisoli A, Salsedo F, Bergamasco M (2007) A fingertip haptic display for improving local perception of shape cues. In: WHC ’07: Proceedings of the second joint eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems. IEEE Computer Society, Washington, DC, pp 409–414 27. Sato M (2002) SPIDAR and virtual reality. In: Proceedings of the 5th biannual world automation congress, pp 17–23 28. Murayama J, Bougrila L, Luo Y, Akahane K, Hasegawa S, Hirsbrunner B, Sato M (2004) SPIDAR G&G: a two-handed haptic interface for bimanual VR interaction. In: Proceedings of the 4th international conference eurohaptics 2004, Munich, 5–7 June 2004 29. Solis J, Avizzano CA, Bergamasco M (2002) Teaching to write Japanese characters using a haptic interface. In: HAPTICS ’02: Proceedings of the 10th symposium on haptic interfaces for virtual environment and teleoperator systems. IEEE Computer Society, Washington, DC, pp 255–262 30. Avizzano C, Portillo-Rodriguez O, Bergamasco M (2007) Assisting to sketch unskilled people with fixed and interactive virtual templates. In: Proceedings of IEEE international conference on robotics and automation, pp 4013–4017 31. Portillo-Rodriguez O, Avizzano C, Chavez-Aguilar A, Raspolli M, Marcheschi S, Bergamasco M (2006) Haptic desktop: the virtual assistant designer. In: Proceedings of the 2nd IEEE/ASME international conference on mechatronic and embedded systems and applications, pp 1–6 32. Iwata H (1990) Artificial reality with force-feedback: development of desktop virtual space with compact master manipulator. In: SIGGRAPH ’90: Proceedings of the 17th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 165–170 33. Gutiérrez M, Lemoine P, Thalmann D, Vexo F (2004) Telerehabilitation: controlling haptic virtual environments through handheld interfaces. In: VRST ’04: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 195–200 34. Ott R, Thalmann D, Vexo F (2006) Organic shape modeling through haptic devices. Comput Aided Des Appl 3(1–4):79–88 35. Ott R, Gutierrez M, Thalmann D, Vexo F (2006) Advanced virtual reality technologies for surveillance and security applications. In: VRCIA ’06: Proceedings of the 2006 ACM international conference on virtual reality continuum and its applications. ACM Press, New York, pp 163–170 36. Frisoli A, Rocchi F, Marcheschi S, Dettori A, Salsedo F, Bergamasco M (2005) A new forcefeedback arm exoskeleton for haptic interaction in virtual environments. In: WHC 2005: first joint Eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems, pp 195–201 37. Bergamasco M, Frisoli A, Barbagli F (2002) Haptics technologies and cultural heritage applications. In: CA ’02: Proceedings of the computer animation. IEEE Computer Society, Washington, DC, p 25 38. Iwata H, Yano H, Nakaizumi F, Kawamura R (2001) Project FEELEX: adding haptic surface to graphics. In: SIGGRAPH ’01: Proceedings of the 28th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 469–476

194

10 Touch

39. Poupyrev I, Nashida T, Maruyama S, Rekimoto J, Yamaji Y (2004) Lumen: interactive visual and shape display for calm computing. In: SIGGRAPH ’04: ACM SIGGRAPH 2004 emerging technologies. ACM Press, New York, p 17 40. Poupyrev I, Nashida T, Okabe M (2007) Actuation and tangible user interfaces: the Vaucanson duck, robots, and shape displays. In: TEI ’07: Proceedings of the 1st international conference on Tangible and embedded interaction. ACM Press, New York, pp 205–212

Chapter 11

Smell and Taste

This chapter introduces a few developments about the two remaining senses among the five fundamental senses: smell and taste. Although these senses are important in real life, they are very difficult to reproduce in the virtual world.

11.1

The Need for Smells and Tastes in VR

The olfactory nerve is the only sense organ that connects the external world directly to the brain, to the limbic system. The limbic system is composed of structures involved in emotion, motivation, and emotional association with memory. From the evolutionary point of view, the limbic system is one of the oldest structures in the brain and has evolved as part of the olfactory (smell) sense. It is commonly accepted that smells influence how we act and feel. The sense of smell can stimulate the memorization of concepts or experiences. Odors are well known for their high influence as contextual retrieval cues not only for autobiographic memories, but also for various other types of memories, including visuospatial memories. For example, a recent study has shown that the use of olfactory stimuli for cueing memories during sleep is useful for memory consolidation [1]. Strong correlations have been found between smell and attention, reaction times, mood, and emotional state [2]. The sense of smell was addressed by one of the first virtual reality systems, the Sensorama (see Sect. 1.3). Studies show that olfactory stimuli can enhance the sense of presence by recalling previous experiences and modifying the emotional state of the user. The sense of taste is the result of a complex interaction between multiple sensory mechanisms: Whenever food is placed in the mouth, taste receptors are stimulated. Simultaneously, different types of sensory fibers that monitor several food attributes, such as texture, temperature, and odor, are activated [3]. Taste perception serves as a primary gatekeeper controlling voluntary ingestion of substances. Humans can distinguish five major taste classes: sweet, sour, bitter, salty, and umami, a savory © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_11

195

196

11

Smell and Taste

flavor exemplified by the amino acid glutamate. Everyone has different sensitivity to many different substances. Some of this variation is known to be genetic in origin [4]. Other factors, such as age, sex, and exposure to different diets and foods, are also known to affect some taste abilities [5]. Recent studies have demonstrated that anxiety or depression are associated with taste disturbances. For instance, altered taste and appetite seen in affective disorders may reflect an actual change in the gustatory system [6]. Whether the sense of taste can be used as an interaction modality in virtual worlds is still an open research question. However, it is clear that this sense plays a very important role in perceiving the real world.

11.2

Smell Interfaces

Smell interfaces are also called olfactory displays. In general, an olfactory display consists of a palette of odorants, a flow-delivery system, and a control algorithm that determines the mixing ratios, concentration, and timing of the stimulus [7]. Smell-O-Vision [8] was a system that released odor during film projection so that viewers could “smell” what was happening. It was created by Hans Laube and made its only appearance in the 1960 film Scent of Mystery [9], produced by Mike Todd Jr. Movie theater seats were injected with 30 odors when the soundtrack triggered the process. Olfactory displays have been mainly used to provide alerts and notifications. One of the most frequently cited developments is the work of Kaye [10]. The systems he has developed are used to provide ambient notification information. In the system inStink, a spice rack with sensors attached to each spice jar was linked to a set of spray guns at a remote location. Whenever a jar was removed from the rack, the corresponding essential oil was released at the remote location, conveying some kind of sense of presence. It was reported that some scent combinations were rather unpleasant. This is an important issue to consider. Mixing smells is much more complex than mixing colors. A second system developed by Kaye was Dollars & Scents. Two different scents were used to represent the current value of the NASDAQ stock market. The device would emit a mint smell if the market was rising or lemon if it was falling. Scent Reminder used the same technology but was integrated in MS Outlook. Scents were released as appointment reminders. In [11] Washburn and Jones present an overview on olfactory displays for data visualization. Some studies concerning the effects of olfactory cues on the sense of presence are described. In general, the addition of smells to virtual environments increases the perception of realism and presence. Several issues concerning the dissipation, duration, and mixing of scents are reported. In general, there are two types of olfactory displays, multiuser and individual. Multiuser olfactory displays are used in special rooms, for example, when using a CAVE system. An example of a multiuser olfactory display is Scents of Space

11.2

Smell Interfaces

197

(http://www.haque.co.uk/scentsofspace.php). An interactive smell system that allows for 3D placement of fragrances without dispersion. This project demonstrates how smell can be used spatially to create fragrance collages that form soft zones and boundaries that are configurable on the fly. An airflow within space is generated by an array of fans. Moving air is then controlled by a series of diffusion screens. Computer-controlled fragrance dispensers and careful air control enable parts of the space to be selectively scented without dispersing through the entire space. Individual olfactory displays include devices such as the one developed by Yanagida et al. [12]. The system uses an air cannon that tracks the user’s nose to present smells. This fires a focused vortex of air so that other users do not smell anything. However, the system had some problems, one of the most important was that the temporal duration in which users could detect the smell was too short. The smell would not be perceived if the users exhaled when a scented vortex reached their noses. The Scent Collar, developed by the Institute for Creative Technologies (http:// www.ict.usc.edu) and AnthroTronix (http://www.anthrotronix.com), is another individual olfactory display that fits around a user’s neck. It holds four scent cartridges and is controlled by a wireless interface. An application of individual olfactory displays for smell-based interaction is Olfoto [13]. The system is an olfactory photo browsing and searching tool where smells are used to tag photos. The use of olfactory displays in virtual reality has not been fully explored. In [10], Kaye overviews some other applications of smell in games and virtual environments. An interesting application of olfactory cues in virtual reality is “Friend Park” [14]. The authors paid special attention to providing olfactory cues and proposed the concept of Aroma Aura. Two types of smells are used: scent of the environment and scent of the object. The environmental scents provide the smell of a place (aroma aura). This helps users feel like they are in a particular location (sense of presence). The scent of the object gives information about a particular item in the environment. VAQSO, a Tokyo-based company, has developed an attachment for VR headsets that emits odors [15]. The VASQO VR adds scent to the VR experience. It is possible to smell in virtual reality by synchronizing VR scenes or action. Five different scent cartridges can be placed and removed easily. Scents last about one month on average. A Velcro strap on the device allows it to be compatible with any headgear. During a computer game, for instance, Vaqso VR exhales a certain aroma based on the events in the virtual world. The Japanese startup offers 15 different aromatic compositions as of November, 2018. Among them are natural (ocean, burning/gunpowder, forest, soil), food-related (coffee, caramel, chocolate, curry roast chicken), and several others. Researchers from Stockholm and Malmö Universities have brought smell to VR gaming in 2022 [16]. In the Nosewise Wine Game, players enter a virtual wine cellar. From a bar containing different types of wine, they choose virtual wine glasses. When the participants lift the glass, the scent machine attached to the VR system releases a scent. In this way, participants attempt to identify the aromas of each option, and score with a correct guess. “The possibility to move on from a

198

11

Smell and Taste

passive to a more active sense of smell in the game world paves the way for the development of completely new smell-based game mechanics based on the players’ movements and judgments,” says Simon Niedenthal, interaction and game researcher at Malmö University. The scent is delivered by a handheld smell machine.

11.3

Taste Interfaces

The sense of taste has never been directly addressed in a virtual reality system. This is still an unexplored research area. Very few examples of what we call “taste interfaces” could be found in the scientific literature when we were writing this book. Food Simulator The “food simulator” developed by Iwata et al. [17] is an interaction device that addresses the sense of taste. This interface could be defined as a “food texture display.” The system simulates the action of chewing different kinds of food. The simulator releases flavoring chemicals onto the tongue and plays back the prerecorded sound of a chewing jawbone corresponding to the food being simulated. The mechanical part of the simulator that is inserted into the mouth has cloth and rubber covers and is intended to resist the user’s bite in a similar way to real food. When someone chews the device, a sensor registers the force of the bite and a motor provides appropriate resistance; see Fig. 11.1. The researchers say their device can be used to design new foods and may even allow young designers to experience the difficulty older people may face in chewing food. Drink Simulator The device developed by Hashimoto et al. [18] focuses on providing stimuli to the mouth and lips, simulating the action of drinking. This is a straw-like user interface that simulates the sensations of drinking by reproducing pressure, vibration, and sound.

motor force sensor

biting force

Fig. 11.1 Food simulator

References

199

The system is composed of a disposable straw; a speaker to reproduce prerecorded sounds of different kinds of food (liquids, solids, gels); and a valve controlled by a solenoid and a servomotor. Different tests have been conducted to evaluate the effectiveness of the interface for discriminating several kinds of food with an average success between 55% and 72%. The straw-like user interface has been presented in several exhibitions (e.g., ARS Electronica 2005). The researchers affirm that due to the high sensitivity of the mouth and lips, it is possible to develop unique interfaces for interactive arts and entertainment. Edible User Interfaces In his master’s thesis, B.I. Resner [19] proposed the concept of “edible user interfaces” (EUI), which can be seen as a kind of tangible interface used by animals (e.g., dogs) to communicate with humans by means of computers. Based on the EUI concept, Maynes-Aminzade [20] has developed two edible interfaces: BeanCounter and TasteScreen. The BeanCounter is a low-resolution gustatory display that dispenses jellybeans according to the use of system resources. Jellybeans drop into a clear chamber when memory is allocated, then fall out into a bowl when the memory is released, enabling programmers to monitor up to six processes. The volume of jellybeans dispensed corresponds to the amount of memory associated with the operation. Besides the fact that jellybeans are edible, in this interface, the sense of taste is not directly exploited. This is more an example of an alternative data visualization system. The tasteScreen is a USB device that attaches to the top of a flat-panel monitor. The device contains 20 small plastic cartridges filled with flavoring agents that can be mixed and dripped down the front of the monitor. A user can sample the dispensed flavor by touching his tongue to the computer screen. Certain combinations of flavoring agents re-create a flavor appropriate to the user’s task. While the hygienic aspect of the tasteScreen is questionable, the idea of using the sense of taste to convey information should be explored further.

References 1. Rasch B, Büchel C, Gais S, Born J (2007) Odor cues during slow-wave sleep prompt declarative memory consolidation. Science 315(5817):1426–1429 2. Michael G, Jacquot L, Millot J, Brand G (2003) Ambient odors modulate visual attentional capture. Neurosci Lett 352(3):221–225 3. Simon SA, Ivan RG, deAraujo E, Nicolelis MAL (2006) The neural mechanisms of gustation: a distributed processing code. Nat Rev Neurosci 7:890–901 4. Drayna D (2005) Human taste genetics. Annu Rev Genomics Hum Genet 6:217–235 5. Bartoshuk L, Duffy V, Miller I (1994) PTC/PROP tasting: anatomy, psychophysics and sex effects. Physiol Behav 56:1165–1171 6. Heath TP, Melichar JK, Nutt DJ, Donaldson LF (2006) Human taste thresholds are modulated by serotonin and noradrenaline. J Neurosci 26(49):12664–12671

200

11

Smell and Taste

7. Gutierrez-Osuna R (2004) Encyclopedia of human-computer interaction. Berkshire Publishers, pp 507–511. Chap. Olfactory Interaction 8. https://en.wikipedia.org/wiki/Smell-O-Vision 9. https://en.wikipedia.org/wiki/Scent_of_Mystery 10. Kaye JJ (2004) Making scents: aromatic output for HCI. Interactions 11(1):48–61 11. Washburn DA, Jones LM (2004) Could olfactory displays improve data visualization? Comput Sci Eng 6(6):80–83 12. Yanagida Y, Kawato S, Noma H, Tomono A, Tetsutani N (2004) Projection- based olfactory display with nose tracking. In: VR ’04: Proceedings of the IEEE Virtual Reality 2004 (VR’04). IEEE Computer Society, Washington, DC, p 43 13. Brewster S, McGookin D, Miller C (2006) Olfoto: designing a smell-based interaction. In: CHI ’06: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press, New York, pp 653–662 14. Tominaga K, Honda S, Ohsawa T, Shigeno H, Ichi Okada K, Matsushita Y (2001) “Friend park”-expression of the wind and the scent on virtual space. In: VSMM ’01: Proceedings of the seventh international conference on virtual systems and multimedia (VSMM’01). IEEE Computer Society, Washington, DC, p 507 15. https://www.makertour.fr/projects/vaqso-bring-scent-to-virtual-reality 16. https://www.designboom.com/technology/sense-smell-vr-gaming-world-nosewise-odormachine-10-25-2022/ 17. Iwata H, Yano H, Uemura T, Moriya T (2004) Food simulator: a haptic interface for biting. In: VR ’04: Proceedings of the IEEE virtual reality 2004 (VR’04). IEEE Computer Society, Washington, DC, pp 51–57 18. Hashimoto Y, Nagaya N, Kojima M, Miyajima S, Ohtaki J, Yamamoto A, Mitani T, Inami M (2006) Straw-like user interface: Virtual experience of the sensation of drinking using a straw. In: ACE ’06: Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology. ACM Press, New York, p 50 19. Resner BI (2001) Rover@Home: computer mediated remote interaction between humans and dogs. Master’s thesis program in media arts and sciences, School of Architecture and Planning, MIT Press 20. Maynes-Aminzade D (2005) Edible bits: seamless interfaces between people, data and food. In: CHI 2005: alt.chi, extended abstracts

Chapter 12

Brain–Computer Interfaces

This short chapter is dedicated to a research approach that is promising but still in the infancy and questionable from an ethical point of view: direct connection to the brain. In this chapter, we present the main techniques of brain–computer interfaces and their role in virtual reality.

12.1

BCI Principles

Virtual reality (VR) technology has grown significantly over the past few years. Whether it is car design or surgery, the virtual world has become more and more prevalent in our everyday lives. Virtual reality users, however, are still required to wear clunky headsets and possibly other gear. The equipment is becoming more user-friendly, but we wonder when we will be able to bypass all that equipment and connect VR to our minds directly. There is probably no better answer today than the brain–computer interface (BCI). Now, BCI is not only used in neuroscience research but also for rehabilitation. The purpose of a BCI is to create a direct communication channel between the human brain and a computer or other external device. Using technologies to interface directly with the brain, neuroreality refers to a reality powered by the brain. In contrast to traditional VR, which requires a physical reaction from the user (e.g., swinging a controller to wield a virtual sword), neuroreality systems are linked directly to the brain through brain–computer interfaces. BCI research was historically aimed at developing communication and control solutions for patients with severe disabilities [1]. Using brain–computer interfaces, tremendous progress has been made in controlling prosthetics, wheelchairs, and robotic devices. Brain–computer interfaces use sensors that transmit brain waves. With these sensors, the computer can receive direction from the fingers, voice, eye gaze, skin, head, and body position of a connected user. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_12

201

202

12 Brain–Computer Interfaces

Fig. 12.1 A cap holds electrodes in place while recording an EEG. (Creative Commons Attribution 2.0)

Brain–computer interfaces (BCIs) use direct brain activity measurements, provide feedback to the user, operate in real time, and require intentional control. A brain–computer interface measures central nervous system (CNS) activity, converting it into artificial output to replace, restore, enhance, supplement, and improve CNS output and its interaction with the external and internal environment. BCIs can be invasive (microelectrode arrays), partially invasive (ECoG and endovascular), or noninvasive (EEG, MEG, EOG, MRI) [2]. An invasive BCI involves implanting electrodes under the scalp to communicate brain signals. Its main advantage is its ability to provide more accurate reading; its downside is the side effects of the surgery. The partially invasive BCI devices are implanted inside the skull, but rest outside the brain rather than within the gray matter. Compared to noninvasive BCIs, they produce better resolution signals. Noninvasive methods are used for monitoring and recording EEG signals from the scalp (Fig. 12.1). A BCI operates by translating volitional modulation of brain signals into computer commands, which can be recorded from the scalp using electroencephalography (EEG), from the dura mater or cortical surface using electrocorticography (ECoG), or from cortical neurons. The majority of BCI systems use real-time electroencephalography (EEG) to convert brain waves into control signals for virtual reality software. A classification algorithm is applied to these signals after they have been preprocessed by the system, which converts the extracted features into commands for use in the VR simulation. One of the major barriers to using EEG as a brain–computer interface in the early days of BCI research was the extensive training required before users could work the technology. As a means of streamlining the collection and analysis of EEG brain data and reducing researchers’ cost-benefit barriers, neuroscientists developed an EEG classification processing pipeline to simplify their steps, refine their strategies and techniques, and expand their applications. There are approximately 100 billion neurons in the human brain. In order to fully comprehend the complex relationships between these neurons and their synaptic connections, it is necessary to examine vast amounts of brain data holistically. A

12.1

BCI Principles

203

Brain activity analysis Preprocessing Signal acquisition

Artifact handling Headset system Feature extraction & interpretation

EEG headset

ETI Information fusion

Conductive tissue Neuronal activity Feedback

Sensory Input

Personal traits

Feedback generation

Output interface

Signal generation

Fig. 12.2 Wearable, wireless EEG solutions in daily life applications: What are we missing? (Licensed under the Creative Commons Attribution-Share Alike 4.0 International license)

primary rate-limiting step in the utility of EEG readings has been the ability to isolate meta-level neural circuitry patterns from EEG readings. Typically, EEG signals make a lot of noise and require high temporal precision, so researchers are trying to filter the noise and make the devices more efficient at processing signals from users. BCI research primarily focuses on noninvasive EEG-based BCIs. While EEG-based interfaces are easy to wear and do not require surgery, they have relatively low spatial resolution and cannot utilize high-frequency signals because the skull dampens and scatters the electromagnetic waves created by neurons. In addition, EEG-based interfaces require some time and effort beforehand, whereas non-EEG-based ones, as well as invasive ones, require no prior training. It has been shown that machine learning, based on statistical temporal features extracted from frontal lobe (EEG brainwave) data, can successfully classify mental states in human–computer interactions (relaxed, neutral, concentrating) [3], mental emotional states (negative, neutral, positive) [4], and thalamocortical dysrhythmia [5]. We must consider these aspects when designing and developing wireless EGG solutions: signal generation, signal acquisition, brain activity analysis, and output interface (Fig. 12.2).

204

12.2

12 Brain–Computer Interfaces

BCI in Virtual Reality

BCI technology has been incorporated by a few companies into VR devices. The company MindMaze SA, Lausanne, Switzerland has developed a unique headset called NeuroGoggles™ that combines VR with bio-sensing capabilities. A 31-channel EEG embedded in the headgear net of the HMD, an electrooculogram (EOG), a 5-lead clinical electrocardiogram (ECG), and eight EMG channels are included for measuring electrophysiological properties of the body. With NeuroGoggles, you can experience true augmented and virtual reality with unique biofeedback based on brain and body signals. In this system, two stereoscopic color cameras are integrated into a wearable head-mounted display for video-through viewing, with a depth sensor for object tracking, an inertial sensor for head tracking and electrophysiological measurements for brain and bodily signals. It is designed to ensure low-latency synchronization across all functional modules, which is essential for multimodal real-time data analysis. The integration of BCI into VR is a promising area for entertainment purposes, since it is possible for virtual reality headsets to include EEG sensors, so games could respond differently depending on the user’s mood or how they respond to certain game elements. Incorporating a P300-based brain–computer interface (BCI) into virtual reality (VR) environments is promising for the video games industry. Nevertheless, it has several limitations, mostly related to hardware constraints and the stimulation required by the BCI. According to Cattan et al. [6], there are several ways to resolve or circumvent these limitations. The BCI should only be used for small and noncritical tasks in the game. When it comes to software limitations, using actions that are naturally integrated into the virtual world is important. Cooperative games are also a good solution because they enable multiple actions and enhance social interaction and entertainment. Furthermore, passive BCIs seem essential to bringing a unique perspective to VR technology. Daniel R. Cook, founder of EyeMynd Quantum Physics Research Institute (https://www.eyemynd8.com/) compares the experience to dreaming. “In a dream, you can run around without moving your physical legs. Dreaming and imagining creates brain signals that we can read”. “With what we want to do, you won’t need eyeballs to see, or ears to hear, or hands and feet. We can bypass all of that.” Unlike other devices that require implantation, EyeMynd’s system does not require implantation. An EEG headset would be worn instead to monitor their brainwaves. “Ten years from now, this will seem obvious, computers are becoming fast enough that we can detect and interpret all the signals of the brain in real time,” says Cook. Seven years later, it does not seem anymore obvious, and we should surely wait for more years. It isn’t just Cook who is exploring brainwave-detecting external tech to make VR more seamless. Neurable is a Boston-based startup that uses easy-to-use devices to measure brain activity to generate simple, real-time insights. A pioneer in bioinformatics, EMOTIV empowers the neuroscience community using EEG. The EMOTIV

References

205

EEG headset is equipped with algorithms that detect Frustration, Interest, Relaxation, Excitement, Engagement, Stress, and Attention. OpenBCI (https://openbci.com/) is a Brooklyn-based company that develops open-source tools for neuroscience and biosensing. They have developed Galea, a hardware and software platform that combines multimodal biometrics with mixed reality, enabling researchers to have access to real-time physiological and brain data in VR experiments. Galea integrates multiple sensors such as EEG, EMG, EDA, PPG, and eye-tracking into a single HMD. Elon Musk founded Neuralink in 2016 with a device that has a chip that processes and transmits neural signals to devices like a computer or phone. In the future, it is hoped that a person could control a mouse, keyboard, or text messaging functions using their thoughts. As Musk himself has said, we will eventually be able to create computer simulations that are almost indistinguishable from reality, and if these brain interfaces become a reality, they could serve as the platform through which we experience these simulations, enabling us to experience a world that is not only realistic, but also feels real. Popular writer Tim Urban described how Neuralink could impact our understanding of reality in a detailed report announcing the launch of the technology. Rather than tricking our senses into believing what we encounter in virtual reality is real by using external hardware like goggles, gloves, and headphones, our brains could be programmed to respond the same way they would if the experiences really happened. “There would be no more need for screens of course—because you could just make a virtual screen appear in your visual cortex. Or jump into a VR movie with all your senses,” asserted Urban.

References 1. Wolpaw JR et al (2002) Brain-computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791 2. Martini ML, Oermann EK, Opie NL, Panov F, Oxley T, Yaeger K (2020) Sensor modalities for brain-computer interface technology: a comprehensive literature review. Neurosurgery 86(2): E108–E117 3. Bird JJ, Manso LJ, Ribeiro EP, Ekárt A, Faria DR (2018) A study on mental state classification using EEG-based brain–machine interface. 9th International Conference on Intelligent Systems, Madeira Island 4. Bird JJ, Ekart A, Buckingham CD, Faria DR (2019) Mental emotional sentiment classification with an EEG-based brain-machine interface. St Hugh’s College, University of Oxford, United Kingdom: The International Conference on Digital Image and Signal Processing (DISP’19) 5. Vanneste S, Song JJ, De Ridder D (2018) Thalamocortical dysrhythmia detected by machine learning. Nat Commun 9(1):1103 6. Cattan G, Andreev A, Visinoni E (2020) Recommendations for integrating a P300-based brain– computer interface in virtual reality environments for gaming: an update. Computers 9(4):92. https://doi.org/10.3390/computers9040092

Part IV

Applications

Chapter 13

Health Sciences

The areas of health science that have been more frequently addressed by virtual reality are virtual surgery, various forms of rehabilitation therapy (psychological, kinesthetic), anatomy, and biomechanics. We present first the main trends in virtual surgery and a few commercials systems. In the areas of virtual rehabilitation and psychological aspects, we present VR techniques to help for social phobia and autism. Virtual anatomy is also discussed.

13.1 13.1.1

Virtual Surgery Principles of Virtual Surgery

Virtual surgery is an active research subject. Such systems allow physicians to practice and improve their skills in a virtual environment, performing surgery on virtual patients, before entering the operating room (OR). Studies have shown that using VR surgical simulation can significantly improve the performance in the OR [1], in particular for endoscopic and laparoscopic procedures. Although virtual reality (VR) and virtual environments are not commonly used in professional surgery, they are being incorporated into the training and education of surgeons. Professional surgeons generally believe that this type of technology can be very beneficial for surgical education. Real-time computer graphics are required for surgery simulations as well as haptic interfaces. The main objective of a virtual surgery system is to accurately simulate the behavior of tissues (skin, muscles, internal organs, bones) under cutting operations. Realistic collisions should be computed to simulate the interaction of surgical instruments with different kinds of tissue. Simulating cutting operations requires surface deformation and other geometric operations. The topology of the 3D shapes representing tissues suffers changes; new geometry should be created during the © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_13

209

210

13

Health Sciences

simulation. The behavior of the tissue should be accurately reproduced. This requires the use of finite element models (FEM) and other free-form surface deformation algorithms [2]. Besides FEM, other methods like the boundary element method (BEM) are used as alternative computational mechanics techniques. BEM differs from FEM in that only the surface is discretized in order to apply boundary conditions to the equations defining the behavior of the body whose surface is being discretized. This reduces the size of the system of equations to be solved and allows for faster calculations. Together with haptic and visual feedback, BEM has been applied to simulate three basic actions performed on virtual deformable objects: prodding, pinching, and cutting. This research is targeted at the development of a neurosurgery simulator [3]. The user interface is a very important element; haptic interfaces, in particular single-point force-feedback devices like the PHANTOM, are commonly used, as in VR training in gynecologic laparoscopy [4]. Special-purpose haptic interfaces have been developed to simulate the complex interaction involved in endoscopy [5]. Most of the virtual surgery systems are designed to be used for training and planification before real surgery. For example, the EyeSi simulation system [6] used for training surgeons in intraocular surgery. EyeSi immerses the surgeon in an environment of real surgical instruments and virtual tissue deformations provided by interactive 3D graphics. Other systems are already being used for real-time assistance in the operating room. This is the case of the system for real-time biomechanical simulation for image-guided neurosurgery developed by Warfield et al. [7]. By using VR and AR (augmented reality) in surgery, surgeons can visualize hearts, eyes, knee joints, and more. Surgeons sometimes use haptic feedback gloves to simulate buzzing drills and saws. These resources enable doctors to prepare for operations, study realistic models of patients, and access real-time anatomical references and information. Furthermore, these extended reality (XR) tools can help hospitals save time and money. 3D scans allow surgeons to prepare thoroughly for each case and implement proactive, timesaving procedures. As an added benefit, storing images and patient data on one AR/VR platform reduces the need for expensive screens and unnecessary staff.

13.1.2

Case Studies in Virtual Surgery

Using augmented and virtual reality techniques, Rahman et al. [8] propose approaches to minimally invasive surgery (MIS). There are several reasons why the best surgeons and scientists utilize MIS surgery techniques: minimizing the patient’s suffering, the duration of the operation, and the difficulty level. A realistic model of cardiovascular system has been developed by Sorensen et al. [9] to be used for virtual surgery training. In addition to illustrating various elements

13.1

Virtual Surgery

211

of heart surgery, this simulator allowed surgeons to practice open-heart surgery before any cardiovascular surgery. George Washington Hospital’s Department of Neurosurgery uses VR technology to treat thoracic conditions. Like the brain, the chest also has major blood vessels and anatomical features where millimeters play a crucial role when performing advanced medical procedures. Precision virtual reality technology is now being used by thoracic surgeons at the hospital to support surgical planning and patient education. Patients with chest masses that may be invading other structures or adjacent to them may benefit from the VR platform. A 360-degree view enhances surgeons’ ability to see where structures, vessels, and airways are in relation to one another. By doing so, you can determine whether certain surgeries can be safely and effectively performed. It is possible to use VR technology to detect if a mass has invaded a certain area, thereby avoiding unnecessary procedures. More precise views can help direct treatment and possibly change the approach if surgery is recommended. If the surgeon prefers to go through the side instead of the front, then he or she may decide to go through the side. A new software system at Stanford Medicine combines images from MRIs, CT scans, and angiograms to create a three-dimensional model that physicians and patients can manipulate, much like a virtual reality game. Using virtual reality, residents are trained, surgeons are assisted in planning upcoming operations, and patients are educated. Additionally, it assists surgeons in the operating room by guiding them in a three-dimensional space. This VR system was developed by Colorado startup Surgical Theater to train residents, help surgeons plan upcoming surgeries, and educate patients. Surgeons can also use it in the operating room, guiding them in a three-dimensional space. Instructors can highlight different components of the brain, such as arteries to show an aneurysm, bones to illustrate skull deformities, or tissues to illustrate tumors, all while rotating the view to illustrate how the structures appear from different angles. Additionally, avatars can follow the steps for removing a tumor or fixing an aneurysm, starting outside the skull. Surgeons practice their upcoming surgery in the Neurosurgical Simulation Lab. Rather than practicing on a generic brain, they practice on images of the actual patient, which allows them to plan out the surgery ahead of time. Surgery can be planned easier and more accurately with threedimensional imagery, resulting in safer procedures. For the brain and spinal cord, Stanford Medicine doctors use VR technology because these organs are stable and lend themselves to images – in contrast to other parts of the body, which move with blood flow and breathing. The new VR technology adds a three-dimensional view to the real-time video that surgeons can overlay on video.

13.1.3

Commercial Virtual Surgery Simulators

Several commercially available virtual surgery simulators are listed below:

212

• • • • • • • •

13

Health Sciences

SIMENDO Laparoscopy (SIMENDO, Netherlands) Medical (Immersion Inc., USA) LapSim (Surgical Science Sweden AB, Goteborg, Sweden) Mentice MIST (Mentice AB, Goteborg, Sweden) SimSurgery Education Platform (SimSurgery AS, Oslo, Norway) Prom IS (Hapticc Ltd., Dublin, Ireland) HystSim (VirtaMed AG, Zurich, Switzerland) VOXEL-MAN TempoSurg (VOXEL-MAN Group, Hamburg, Germany)

OramaVR, a spin-off from Foundation for Research and Technology-Hellas, is a Swiss startup that streamlines medical training for healthcare professionals. The startup uses an immersive VR head-mounted display that supports individual remote training and enables tracking and development of skills. It also combines an analytics engine and cloud computing for user assessments and tracking progress. The solution also supports multi-user training for medical school students, integrating up to fifty students remotely. This facilitates gamified prototyping of the curriculum and improves knowledge and skill development.

13.1.4

The Intelligent Digital Surgeon

A new project was launched in 2022, the Intelligent Digital Surgeon (IDS) [10]. This project aims to design and model a 3D intelligent digital surgeon to guide trainees in virtual surgery. The digital surgeon will identify the trainee’s gestures by recognizing and analyzing the trainee’s hand/arm gestures and provide personalized assessment, real-time feedback, instructions, and recommendations. The proposed system will adopt a distributed software architecture taking advantage of edge-cloud resources and 5G networking to enable ultra-low latency untethered VR. The IDS motion captured data containing digital surgeon’s know-how gestures are provided by Hospital of Geneva (HUG), and then visualized by the company MIRALab Sarl to allow digital representation of these surgeon’s gestures. A deep learning model will be derived by the University of Geneva from recording skilled real surgeons’ gestures to assist the feedback decision engine of the digital surgeon. A distributed VR system architecture will be implemented utilizing edge computing and 5G networks by the University of Geneva. The company OramaVR will implement the deep learning model in their MAGE system. Furthermore, it will embed methods of cutting and tearing physics-based deformable surfaces that advance the state-ofthe-art. Figure 13.1 shows a prototype of the Intelligent Digital Surgeon.

13.2

Virtual Rehabilitation and Therapy

213

Fig. 13.1 Prototype of the intelligent digital surgeon

13.2

Virtual Rehabilitation and Therapy

Virtual reality can be used to provide rehabilitation based on physiotherapy (e.g., stroke-based disorders) and also for psychological therapy (phobia treatment).

13.2.1 Physiotherapy As explained in [9], VR-based rehabilitation involves multiple disciplines, including physical therapy, computer science, psychology, communication. VR systems for rehabilitation integrate haptics and modern sensor technology. The methodology to develop this kind of system includes identification of movement patterns, development of simulated tasks (therapy exercises), and diagnostics (through data acquisition and monitoring). Several research projects address the motor rehabilitation needs of stroke patients (physical therapy) [11, 12], while others focus on the treatment of balance disorders [13]. The VR-based rehabilitation systems are commonly implemented through stereoscopic displays, force-feedback devices (haptic interfaces), and modern sensing techniques that have game-like features and can capture accurate data for further analysis. Diagnostics and evaluation can be made through an artificial intelligencebased model (data mining, dimensionality reduction, etc.) using collected data, and clinical tests have been conducted. Haptic interfaces, such as the “Rutgers ankle” [14], provide 6-DOF resistive forces on the patient’s foot, in response to Virtual Reality–based exercises. This kind of physiotherapy for post-stroke rehabilitation has demonstrated to be effective. Results showed that, over six rehabilitation sessions, the patient improved on clinical measures of strength and endurance, which corresponded well with torque and power output increases measured by the Rutgers ankle. There were also substantial

214

13

Health Sciences

improvements in task accuracy and coordination during the simulation and the patient’s walking and stair-climbing ability. The combination of haptic interfaces, Web technologies for data transmission, 3D graphics, and other multimodal stimuli can be used to develop telerehabilitation systems. The basic concept is to provide the patient with an interface that will allow us to perform the physiotherapy designed and monitored by a doctor or therapist in a remote location. Video and audio streaming can be used to keep close contact between patient and therapist regardless of the physical distance. Data transmission over the Internet or dedicated networks can be used to control the virtual environment and acquire data for therapy evaluation purposes and progress monitoring. An example of such a system is the telerehabilitation application presented in [15]. A Haptic WorkstationTM is used as an interface to a virtual environment controlled and monitored by a therapist in a remote location. The patient performs exercises defined by the therapist. A virtual window displaying live video allows the patient to keep visual contact with people in the remote location. The system illustrated in Fig. 13.2a is intended to be used in post-stroke rehabilitation. This kind of therapy helps stroke survivors relearn skills that are lost when part of the brain is damaged. These skills can include coordinating leg or arm movements in order to walk, manipulating objects, or other tasks involved in any complex activity. In this telerehabilitation system, the therapist can design new exercises by means of drawing lines on a PDA; such lines will be transformed into 3D shapes with haptic feedback. The patient will be able to touch and interact with the 3D shapes, relearning arm coordination skills. A second version of the telerehabilitation system (see Fig. 13.2b) has been used for shoulder and elbow therapy. This system provides a better performance monitoring interface (3D character showing the position in real time on a PDA) and has been validated by a physiotherapist with the help of muscular effort measurements acquired by an ElectroMyoGraph (EMG).

Fig. 13.2 VR-based telerehabilitation systems for physiotherapy. (a) Post-stoke therapy. (b) Shoulder and elbow therapy

13.2

Virtual Rehabilitation and Therapy

13.2.2

215

Psychological Therapy

Phobias can often be treated effectively by using gradual exposure therapy. Patients are subjected to anxiety-provoking stimuli in a gradual order, from the least difficult stimulus to the most difficult one, attenuating their anxiety in a progressive way. Traditionally those stimuli are looked for in actual physical situations (in vivo) or by having the patient imagine the stimulus (in vitro). Virtual reality (VR) allows a third option of exposure therapy in a virtual setting that is safer, less embarrassing, and less costly than reproducing real-world situations. An important advantage of VR is that it can create situations that are difficult to find in real life. Experiments have proven VR to be a useful tool in treating specific phobias such as fear of heights [16], fear of spiders [17], fear of flying [16], claustrophobia [18], and social phobia [19–21]. For instance, the work of Grillon et al. [19] and Herbelin et al. [20] has focused on the treatment of social phobia and social anxiety: fear of speaking in public. The researchers have developed an exposure-based therapy that consists of immersing patients in a virtual world; the patient should face a group of virtual characters that react to the patient’s performance. Different public-speaking situations can be simulated, for example, a job interview, meeting in a bar, speech in an auditory (see Fig. 13.3a). In a preliminary work [22], the authors exposed subjects to a VR situation representing a 3D audience composed of emergent gazes in the dark and surrounding the subject. It was observed that the virtual audience was able to provoke more anxiety in social phobics than in non-phobics. The hypothesis that eye contact is an important factor of social phobia was proposed. An experiment reported in [20] used an eye-tracking setup integrated in the virtual reality exposure system, composed of a large screen with stereoscopic projection (see Fig. 13.3b). The virtual audience was rendered in a more realistic way, using animated virtual characters and photo-realistic scenarios. The study

Fig. 13.3 VR-based treatment of social phobia. (a) Exposure-based theraphy. (b) Eye-tracking setup

216

13

Health Sciences

attempted to evaluate objectively a specific parameter present in social phobia, namely, eye contact avoidance, by using the eye-tracking system. The results show a tendency to improve in the subject’s feedback to specific assessment scales, which is correlated to the decrease of eye contact avoidance. The presented virtual reality exposure therapy protocol could be successfully applied to social therapy. Eye-tracking technology could provide therapists with an objective evaluation of gaze avoidance and can give tangible feedback to the patients to estimate their progress during gaze behavior exercises.

13.2.3

Autism and Virtual Dolphins

There has been a growing interest in dolphin-assisted autism therapy in the public and in research in recent years. The program involves dolphin encounters with children with autism. The goal of this 3-D Virtual Dolphinarium [23] is to provide children with autism with a virtual learning environment through interaction with virtual dolphins. As the name implies, the Virtual Dolphinarium simulates a dolphin lagoon. Virtual Dolphinarium was created in the Immersive Room (Fig. 13.4a) by the Institute for Media Innovation (IMI) at Nanyang Technological University (NTU). In this Immersive Room, the latest software and hardware were used to deliver immersive graphics experiences. The room was equipped with Microsoft Kinect devices for human gesture recognition. Dolphin motions and performances are physically simulated. Real-time collision detection and avoidance algorithms are required. A number of simulations are displayed graphically, including waves, splashing, fountains, clouds, balloons, etc. Geometrical modeling is applied to the digitally represented geometry of dolphins, underwater environments, dolphin lagoons, etc. (Fig. 13.4b, c). In place of swimming with dolphins, the program allows these children to act as dolphin trainers by the poolside. Children with autism have been learning nonverbal communication with virtual dolphins through immersive visualization and gesture-based interaction as an initial implementation. There is no doubt that dolphins are highly intelligent marine mammals. Oceanariums with captive dolphins are always popular with children. The dolphins are typically trained by instructors to follow their gestures while performing in public. With inspiration from dolphin shows, which are mostly controlled by gestures, the Virtual Dolphinarium models all virtual pink dolphins with gesture recognition functions. Children with autism are shown a video of dolphin training and performance before they enter the immersive room. Gestures and dolphin training can support children with autism’s nonverbal communication skills. There is always something appealing about dolphin lagoons, dolphins, and dolphin shows for children. Dolphins and the lagoon are modeled in great detail in the Virtual Dolphinarium. Using a 320-curve 3-D screen, immersive visualization simulates virtual dolphins with emphasis on fidelity and real-time interaction. During the interactive program, children with autism will wear 3-D shutter glasses to enable an active stereographic view. Through hand gestures, they communicate

13.2

Virtual Rehabilitation and Therapy

217

Fig. 13.4 (a) The immersive room, (b) Virtual Underwater World visualized on the 3-D screen, and (c) Virtual Dolphinarium visualized on the screen

nonverbally with virtual dolphins. Contrary to the limited number of hand gestures children with autism can learn by communicating with real dolphins, a major advantage of using virtual dolphins is that children with autism can learn more functional hand gestures by programming them and teaching them. It is possible to incorporate functional sign language into these hand gestures at a higher level for various learning purposes. In Fig. 13.5, in the Virtual Dolphinarium, a child with autism interacts with virtual pink dolphins to learn to communicate using appropriate hand gestures. UWS Dolphin Lagoon trainers use selected hand gestures to control virtual pink dolphins. Its design is consistent with many gesture-controlled games that are supported by Microsoft Kinect. As dolphin trainers, children with autism can stand close to the screen of the immersive room while the graphics visualize on the 3-D screen the Virtual Dolphinarium immersively and allow them to gesture poolside.

218

13

Health Sciences

Fig. 13.5 Engaging a child with autism through 3-D immersive visualization and hand gesturing

13.3

Virtual Anatomy

The visible human project (VHP) is one of the most important anatomy- related initiatives that have involved the use of virtual reality technologies, in particular, visualization techniques. The VHP was conceived to collect data from human subjects to serve as a guidebook and baseline dataset in modern anatomy research and education [24, 25]. The dynamic 3D model of the inguinal region developed by Lopez-Cano et al. [26] is an illustrative example of the different components of a virtual anatomy system. The interactive 3D model was created for didactic purposes. A geometric model adjusted to real data was created by means of semiautomatic contour segmentation of anatomic units obtained from the visible human project (https://www. nlm.nih.gov/research/visible/visible_human.html) and data generated from classical anatomic information. Contour extraction was based on definition of a set of NURB curves. Realistic rendering allows on-demand visualization of each independent unit of the model. Current implementation enables surface deformation associated with pushing and stretching interaction. This simulator would be a relevant addition to existing tools for anatomical education after clinical validation. The example cited earlier deals with the modeling of internal organs of a particular region of the body. Virtual anatomy also includes modeling the whole musculoskeletal human system. In [27], Aubel and Thalmann use a multilayered

13.3

Virtual Anatomy

219

Fig. 13.6 Virtual model of the musculoskeletal system. (a) Interactive model of human muscles. (b) Analysis of stress on the hip joint

approach for human body modeling and deformation. The model is split into three general anatomical structures: the skeleton, musculature, and skin. Each of these layers is modeled and deformed using fast, procedural, ad hoc methods that are easy to reproduce. The modeling approach is generic enough to handle muscles of varying shape, size, and characteristics and does not break in extreme skeleton poses. The authors have developed an integrated MuscleBuilder system whose main features are easy and quick creation of muscle deformation models and automatic deformation of an overlying skin. Visually realistic results can be obtained at interactive frame rates. See Fig. 13.6a for an example of an interactive 3D model of some muscles and their interaction with the skeleton. Other example of applications of a virtual model of the musculoskeletal system is the work of Maciel et al. [28, 29]. The authors developed a visualization tool for the analysis of stress distribution on the heap joint in pre- and postoperative conditions; see Fig. 13.6b. Internationel research initiatives are focusing on the development of integrated simulation platforms and databases to concentrate current and future knowledge about the human anatomy and physiology. One of those initiatives is the virtual physiological human. The virtual physiological human (VPH) is a methodological and technological framework established to enable investigation of the human body as a single complex system. It is expected that this project will continue to be the basis for personalized (patient-specific) health-care solutions as well as early diagnostics and predictive medicine [30]. The VPH is a European initiative based on the international Physiome Project. The NSR Physiome Project (https://www.imagwiki.nibib.nih.gov/physiome) is a worldwide effort of several research groups to define the physiome via database integration and development of integrated quantitative and descriptive models. The physiome describes the physiological dynamics of a normal intact organism. In the

220

13

Health Sciences

context of the Physiome Project, it includes integrated models of components of organisms, such as particular organs or cell systems, biochemical, or endocrine systems. Virtual reality plays an important role in both the VPH and the Physiome Project. By means of interactive simulation through haptic interfaces and real-time visualization, it will be possible to study and better understand human physiology and its disfunctions. Simulation technologies will allow for in silico validation of the effects of drugs, surgical techniques, etc. It will also be possible to design more targeted implants and artificial organs [30].

References 1. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4):458–464 2. Sela G, Subag J, Lindblad A, Albocher D, Schein S, Elber G (2007) Real-time haptic incision simulation using FEM-based discontinuous free-form deformation. Comput Aided Des 39(8): 685–693 3. Wang P, Becker AA, Jones IA, Glover AT, Benford SD, Greenhalgh CM, Vloeberghs M (2007) Virtual reality simulation of surgery with haptic feedback based on the boundary element method. Comput Struct 85(7-8):331–339 4. Székely G, Brechbühler C, Dual J, Enzler R, Hug J, Hutter R, Ironmonger N, Kauer M, Meier V, Niederer P, Rhomberg A, Schmid P, Schweitzer G, Thaler M, Vuskovic V, Tröter G, Haller U, Bajka M (2000) Virtual reality-based simulation of endoscopic surgery. Presence Teleop Virt Environ 9(3):310–333 5. Körner O, Männer R (2003) Implementation of a haptic interface for a virtual reality simulator for flexible endoscopy. In: HAPTICS ’03: Proceedings of the 11th symposium on haptic interfaces for virtual environment and tele-operator systems (HAPTICS’03). IEEE Computer Society, Washington, DC, p 278 6. Wagner C, Schill M, Männer R (2002) Intraocular surgery on a virtual eye. Commun ACM 45(7):45–49 7. Warfield SK, Ferrant M, Gallez X, Nabavi A, Jolesz FA (2000) Real-time biomechanical simulation of volumetric brain deformation for image guided neurosurgery. In: Supercomputing ’00: Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM). IEEE Computer Society, Washington, DC, p 23 8. Rahman M, Mahmud P, Mashuk M et al (2013) Augmented and virtual reality based approaches in minimally invasive surgery training. In: International conference on informatics, electronics and vision (ICIEV), pp 1–4 9. Yeh SC, Rizzo A, Zhu W, Stewart J, McLaughlin M, Cohen I, Jung Y, Peng W (2005) An integrated system: virtual reality, haptics and modern sensing technique (VHS) for post-stroke rehabilitation. In: VRST ’05: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 59–62 10. http://miralab.com/project/intelligent-digital-surgeon-ids/ 11. Jung Y, Yeh SC, Stewart J (2006) Tailoring virtual reality technology for stroke rehabilitation: a human factors design. In: CHI ’06: CHI ’06 extended abstracts on Human factors in computing systems. ACM Press, New York, pp 929–934 12. Jack D, Boian R, Merians A, Adamovich SV, Tremaine M, Recce M, Burdea GC, Poizner H (2000) A virtual reality-based exercise program for stroke rehabilitation. In: Assets ’00: Proceedings of the fourth international ACM conference on Assistive technologies. ACM Press, New York, pp 56–63

References

221

13. Jacobson J, Redfern MS, Furman JM, Whitney SL, Sparto PJ, Wilson JB, Hodges LF (2001) Balance NAVE: a virtual reality facility for research and rehabilitation of balance disorders. In: VRST ’01: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 103–109 14. Deutsch JE, Latonio J, Burdea GC, Boian R (2001) Post-stroke rehabilitation with the rutgers ankle system: a case study. Presence Teleop Virt Environ 10(4):416–430 15. Gutiérrez M, Lemoine P, Thalmann D, Vexo F (2004) Telerehabilitation: controlling haptic virtual environments through handheld interfaces. In: VRST ’04: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 195–200 16. Krijn M, Emmelkamp PMG, Olafsson RP, Bouwman M, van Gerwen LJ, Spinhoven P, Schuemie MJ, van der Mast CAPG (2007) Fear of flying treatment methods: virtual reality exposure vs. cognitive behavioral therapy. Aviat Space Environ Med 78(2):121–128 17. Bouchard S, Côté S, St-Jacques J, Robillard G, Renaud P (2006) Effectiveness of virtual reality exposure in the treatment of arachnophobia using 3D games. Technol Health Care 14(1):19–27 18. Garcia-Palacios A, Hoffman HG, Richards TR, Seibel EJ, Sharar SR (2007) Use of virtual reality distraction to reduce claustrophobia symptoms during a mock magnetic resonance imaging brain scan: a case report. Cyberpsychol Behav 10(3):485–488 19. Grillon H, Riquier F, Herbelin B, Thalmann D (2006) Virtual reality as therapeutic tool in the confines of social anxiety disorder treatment. Int J Disabil Hum Dev 5(3) 20. Herbelin B, Benzaki P, Riquier F, Renault O, Grillon H, Thalmann D (2005) Using physiological measures for emotional assessment: a computer-aided tool for cognitive and behavioural therapy. Int J Disabil Hum Dev 4(4):269–277 21. Wilcox LM, Allison RS, Elfassy S, Grelik C (2006) Personal space in virtual reality. ACM Trans Appl Percept 3(4):412–428 22. Herbelin B, Riquier F, Vexo F, Thalmann D (2002) Virtual reality in cognitive behavioral therapy: a preliminary study on social anxiety disorder. In: 8th international conference on virtual systems and multimedia, VSMM2002 23. Cai Y, Chia NKH, Thalmann D, Kee NKN, Lu D, Zheng J, Thalmann NM (2013) Design and development of a virtual dolphinarium to increase functional developmental learning in children with autism spectrum disorder. IEEE Trans Neural Syst Rehabil Eng 21(2):208–217 24. Ackerman MJ (1998) The visible human project. Proc IEEE 86(3):504–511 25. North C, Shneiderman B, Plaisant C (1996) User controlled overviews of an image library: a case study of the visible human. In: DL ’96: Proceedings of the first ACM international conference on digital libraries. ACM Press, New York, pp 74–82 26. Lopez-Cano M, Rodriguez-Navarro J, Rodriguez-Baeza A, Armengol Carrasco M, Susın A (2007) A real-time dynamic 3D model of the human inguinal region for surgical education. Comput Biol Med 37(9):1321–1326 27. Aubel A, Thalmann D (2004) MuscleBuilder: a modeling tool for human anatomy. J Comput Sci Technol 19(5):585–595 28. Maciel A, Boulic R, Thalmann D (2007) Efficient collision detection within deforming spherical sliding contact. IEEE Trans Vis Comput Graph 13(3):518–529 29. Maciel A, Sarni S, Boulic R, Thalmann D (2005) Stress distribution visualization on pre and post-operative virtual hip joint. Comput Aided Orthop Surg 2005:298–301 30. Virtual physiological human: “Towards virtual physiological human: multilevel modelling and simulation of the human anatomy and physiology. WHITE PAPER, November 2005

Chapter 14

Cultural Heritage

This chapter presents a very important field of application of VR: the cultural heritage. VR is the perfect tool to make a revival of the past. We present in this chapter a few projects of VR-based project in culture heritage. They recreate old scenes of the Maya, China, Greek, and Roman civilizations. Due to lack of information, the significance of many important sites of cultural interest is often misunderstood by visitors. Learning about the function and relevance of archaeological sites typically involves consultation of specific literature or on-site explanations by tourist guides. A very attractive way to discover and better understand these cultural sites would be to see and visit them at their glory time. For this, travel in time and space would be best, and virtual reality is offering it through cultural heritage applications. In cultural heritage applications, virtual reality techniques are used to visually recreate historical elements that have today disappeared in archaeological sites. This chapter reviews some applications of virtual reality and augmented reality in the cultural heritage field. We discuss their benefits to the visitors and/or viewers and the associated technical challenges. We start with archaeological data acquisition and reconstruction. Then we present projects that allow visiting ancient places with different degrees of realism and interaction. We start with virtual tours of buildings followed by more complex applications, where you could meet their inhabitants. A virtual tour is one of the most traditional applications of VR in the field of cultural heritage. These virtual tours allow the user to walk through, see, and interact with 3D models of old cities or buildings that have historical value. In this section, we present several tours that are representative of different cultures and technologies.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_14

223

224

14.1

14

Cultural Heritage

Virtual Campeche and Calakmul

Campeche is an old Mexican city, founded by the Spanish in 1540, located on the Yucatan peninsula. UNESCO declared Campeche as a World Cultural Heritage site in 2001. To protect this city, the king of Spain decided to build during the sixteenth century a fortress with ramparts. The military sites and the center of the city remain almost intact today. The project virtual Campeche presented in [1] allows a virtual visit through a Web-based application that runs on standard PCs using a mouse and keyboard as interface. Virtual Campeche was created following a standard approach, used by many cultural heritage projects: data acquisition (orthographic photos of the buildings) and building reconstruction (buildings are modeled in 3D using the postprocessed photos as textures). As the content is supposed to be accessible through the Web, three main techniques are applied to maximize interactivity and system responsiveness: Level of detail model of the city and different versions of the same model are used depending on the position of the current viewpoint. Objects far away in the scene are displayed using a low-detail model with fewer polygons, which loads and renders faster. Progressive scene display: The 3D scene is structured into small parts for rapid delivery over the Internet. Potential visibility computation is used to adapt the visual quality of the content according to the available bandwidth for data transmission. Nearby, in the same Mexican state of Campeche, is an old Maya metropolis called Calakmul, declared a Protected World Heritage Site by UNESCO in 2002. Calakmul is deeply covered by wild tropical forest. Because of it, visitors endure a 5-hour trip to get to this ancient city. The bountiful vegetation prevents visitors from fully appreciating the complex urban design of the city. To partially overcome these problems, scientists and archaeologists collaborated to create a digital reproduction (see Fig. 14.1) of this Maya archaeological site. This project is presented in [2] and is composed of three main applications. A physical installation takes advantage of augmented reality techniques to virtually reconstruct funeral chambers and other sites. A virtual intelligent agent with speech

Fig. 14.1 Latin American virtual reality applications: Virtual Calakmul. (Courtesy of Rocío Ruiz Rodarte, Jesús Savage Carmona and José Larios)

14.3

Terracotta Soldiers

225

recognition capabilities guides the visitors during their walk through the virtual Calakmul. Finally, it offers interactive recreations of stelae and death masks.

14.2

Virtual Dunhuang

Dunhuang was an important historical site on the silk road in northwest China close to the Taklamakan desert. It was declared a World Heritage site by UNESCO in 1987. This oasis town is very well known for the Mogao Grottos, which were dug and decorated by the Buddhist monks during the glory time of the silk trade. Today, the site is composed of almost 550 caves, 3000 statues, and 50,000 m2 of painted murals, which have badly suffered from the Taklamakan climate and visitors. This is one of the reasons why less than 10% of the site is currently open to the public. A virtual environment, for preservation and dissemination of Dunhuang art, was developed in 1999 by the Zheijiang University in Hangzhou (China) and the Fraunhofer-IGD in Darmstadt/Rodstock (Germany). This project was divided into three main parts: Dunhaung Info Web, Internetbased virtual reality, and high-end VR presentation. We describe only the last part, which is presented in [3]. The use of a CAVE system (see Sect. 8.2.5 for a description of this kind of display device) produces good immersion in a Dunhaung grotto. As the historical content was aiming to be presented in a CAVE, several technical challenges came to light. First, all museum pieces are illustrated by text describing the age and materials used for sculptures. To avoid disturbing the user while he is immersed in the 3D world, it was decided to provide this information by a virtual voice as if there was a real guide providing explanations. Secondly, a real environment such as a grotto is more complex than a square room, and a mechanism had to be developed to enable free navigation. For that, the researchers developed a “flashlight interaction paradigm,” where the user could play the role of the famous archaeologist “Indiana Jones” discovering a lost historical site with his flashlight (see Fig. 14.2). The “virtual Dunhuang” project remains one of the best examples of a fully immersive virtual tour for cultural heritage applications.

14.3

Terracotta Soldiers

A vast field of life-size terracotta statues depicting soldiers, servants, and horses was discovered in China in the 1970s. An estimated total of 6000 pieces based on the real army of the Ch’in emperor. Thanks to the Xian project presented in [4], it is now possible to look at them outside of China. The Xian project aimed to recreate and give “life” to the terracotta army by using virtual reality techniques. As a result, a short movie was presented during the Digital

226

14

Cultural Heritage

Fig. 14.2 Flashlight interaction in a CAVE. (Courtesy of Fraunhofer-IGD)

97 event in Geneva, Switzerland. In it, we first see a scene with the 3D terracotta soldiers inside the earth, exactly where they were found. It is dark with a starry sky. The day is coming, so more and more light is appearing. This suddenly awakens one terracotta soldier. He is extremely astonished to see the scene around himself. He notices the presence of another soldier near him and his head, which is on the ground. He took the head on the ground and put it on the next soldier’s body. This latter starts to live again. They look at each other, and in the same time, all the army is slowly coming to life. When the historical part still existed, classically we used acquisition technologies such as 3D scanners to create virtual representations of the real models. However, free access to the object of interest is not always easy. Moreover, the data acquired using a scanner are a “point cloud” a set of unconnected 3D points, not a coherent surface (3D mesh), which is required to have an acceptable 3D model. Remeshing operations should be performed, involving hole filling and denoising algorithms. The meshes obtained are not adapted for virtual reality applications: There are too many triangles and no textures. To overcome the problems mentioned before, Magnenat-Thalmann et al. [4] propose different approaches based on clay sculpting simulation for modeling the faces and meta-balls [19] for modeling the soldiers’ bodies. Despite the judicious choices made during this project, several challenges have been raised. First, the individuality of the soldiers’ faces had to be taken into

14.4

EU-INCO CAHRISMA and ERATO

227

account: All faces are different, and the generated face should be adapted to support a facial animation approach [5] based on a hierarchical model (emotions, phrases of speech, head movements). On the other hand, the goal with the soldiers was to make them realistic enough and with deformation capabilities for animation. The approach used by the research group was to build on top of an animation skeleton a deformable layer based on meta-balls aiming to simulate soft human tissues such as skin and muscles. Finally, clothes were added on the virtual soldiers. These require an optimized mechanical model to manage collision detection and self-collision. To complete this work, the authors used similar techniques to model and animate the horses that accompanied the terracotta soldiers. This project is old in terms of the virtual reality techniques and technology that were used. However, it clearly points out the main problems that should be faced when creating a film or an interactive virtual world.

14.4

EU-INCO CAHRISMA and ERATO

Can you imagine not only visiting a recreation of an ancient building but meeting its inhabitants and hearing their sounds with accurate acoustics as if you were in this place a long time ago? This is possible today with advanced real-time 3D rendering techniques and dynamics synthesis of sound. The ERATO project and the earlier CAHRISMA (see Fig. 14.3) propose a strong improvement in the believability by integrating visual and acoustical simulations. The ERATO project offers virtual simulations of very well-known ancient theatres and Odea from the Mediterranean region, such as Jerash (Jordan), Aphrodisias [6], Aspendos (Turkey), Aosta (Italy), and Lyon (France). CAHRISMA project created interactive simulation of “Istanbul” mosques (see Fig. 14.3a) (Sokullu, Sleymaniye) by the great Ottoman architect Sinan [7]. They were founded by the

Fig. 14.3 EU-INCO CAHRISMA (2004) and ERATO (2006)

228

14

Cultural Heritage

European Union under the framework of the EU-INCO-MED (International Research Cooperation with Mediterranean Partner countries). The virtual environments include, of course, modeling the ancient environments and their acoustics and simulation of the people and their motion inside the selected spaces. These projects present two main challenges related to real time: virtual crowd rendering and interactive sound simulation. The real-time rendering of virtual crowds that composed the audience requires advanced GPU programming techniques (vertex and pixel shaders) for accelerating the skinning of the virtual characters and the rendering 3D textured meshes, as presented in [8]. Advances in dynamic sound rendering have been made by Rindel et al. [9] during this project. Now it becomes possible to quantify the acoustical properties of the ancient theatres and odea in each condition, leading to valuable information regarding the acoustical consequences and degree of authenticity of permanent restorations and temporary amendments. Musical instruments from the Hellenistic and Roman periods were reproduced, and some music samples were designed to make sound recordings. Based on such recorded samples of authentic-style music/speech the acoustic simulations of the theatres also produced audible sound demonstrations of virtual performances with a high degree of acoustic realism. These sound files were integrated with the visual restorations to form the audio in 3D virtual realizations of performances in the selected venues. The previous work on the cultural heritage was mainly focusing on the visual realism and not on the user interactivity. In the ERATO and CAHRISMA projects, the research groups propose a unification of graphic and sound models for interactive simulation to obtain authentic and realistic virtual environments. The realism of the experience is enhanced by their inhabitants (virtual crowds).

14.5

EU-IST ARCHEOGUIDE

Can you imagine visiting ruined buildings in their former glory and walking through them interactively? This is possible today with augmented reality (AR) technologies. The ARCHEOGUIDE project presented in [10] (see Fig. 14.4b), allows visitors see not only ruined monuments, but they can also admire and visit the temples in their former glory. The ARCHEOGUIDE provides a new approach for accessing information at cultural heritage sites in a compelling, user-friendly way through the development of a system based on advanced IT including augmented reality, 3D visualization, mobile computing, and multimodal interaction techniques [11]. The system, described in [12], was tried out in the ancient city of Olympia in Greece. Cultural site visitors were provided with a video through a head-mounted display (HMD), earphone, and mobile computing equipment. A tracking system determines the location of the visitor within the site. Based on the visitor’s profile

14.5

EU-IST ARCHEOGUIDE

229

Fig. 14.4 Augmented reality and cultural heritage

and position, audio and visual information are presented to guide and allow him or her to gain more insight into relevant aspects of the site. The project introduces new techniques of sensors (global positioning system and image-based recognition of landmarks), aggregation for position, and gaze tracking. To display virtual reality through the HMD, the research had to develop an optimized render for mobile platform [13]. The project also explores multimodal interfaces using gesture recognition and speech processing. Efficient wireless data transmission between the site server, which contains all archaeological data, and the user has been developed. The approach using optical tracking without markers being attached to the surroundings presented in [14] and the wearability of the system made ArcheoGuide an ideal tool for ancient sites that do not allow disturbance of the surroundings.

230

14.6

14

Cultural Heritage

EU-IST LIFEPLUS

Since antiquity, fresco paintings were used as records of both events and lifestyles, and for decorations. Thanks to the technologies developed in the framework of EU-IST LIFEPLUS [15] project, we can bring those frescos to life. We can now meet the inhabitants of ancient Pompeii in the ruined houses and shops (see Fig. 14.4a). The LIFEPLUS project was funded by the European Union under the framework of IST (Information Society Technologies). Its goal was to push the limits of current augmented reality technologies, exploring the processes of narrative design of fictional spaces (e.g., fresco paintings) where users can experience a high degree of realistic interactive immersion. Based on a captured or real-time video of a real scene, the project enhanced these scenes by allowing the possibility to render realistic 3D simulations of virtual flora and fauna (humans, animals and plants) in real time. According to its key mobile AR technology, visitors were provided with a video see-through HMD (see Fig. 14.4a), earphones, and mobile computing equipment. A tracking system determines their location within the site, and audio-visual information is presented to them in context with their exploration, superimposed on their current view of the site. LIFEPLUS extends existing AR systems and provides key new technologies to render lively, real-time animations and simulations of ancient virtual life (3D human groups, animals, and plants). This project was quite challenging in several domains, and it is certainly one of the more advanced projects in augmented reality applications for cultural heritage purposes. As with all augmented reality applications, it was mandatory to track the user’s gaze for superimposing 3D computer graphics footage of the real world. This has been realized by the aggregation of DGPS (Differential Global Position System) technology and marker-less video tracking provided by one of the industrial leaders in the field 2d3. On the other hand, the project requested the development of real-time “realistic” virtual life, and these are complex tasks. In [16], the authors explain that it is crucial first to have reliable data sources from archaeologists for a scientifically correct and accurate restitution but also to advance the research in the field of real-time: hair simulation, cloth simulation [17], skin rendering, fauna and flora simulation, realistic facial emotion expression and finally artificial-life methods for behavioral animation of the virtual characters. The integration of these techniques in one application was realized by development of an integrated framework, presented in [20], distributed over two mobile computers. These insures a possible use in ancient sites without disturbing the surroundings.

14.7

EU-Horizon 2020 MINGEI

14.7

231

EU-Horizon 2020 MINGEI

Using craft as cultural heritage, Mingei [18] explores the possibilities for representing and making accessible both tangible and intangible aspects of the craft. As a form of Intangible Cultural Heritage, Heritage Crafts (HCs) involve craft artefacts, materials, and tools. As part of intangible dimensions of HC, there is dexterity, know-how, and proficiency with tools, as well as tradition and identity of the communities where they are practiced. Figure 14.5 shows the system architecture of MINGEI. Digital humans (see Fig. 14.6) have been developed for the Mingei project to enhance exhibits at the Chios Mastic Museum. A specific methodology was followed when creating digital humans. Visitors to the museum can see digital humans standing next to the machines, ready to share their stories and explain how they work.

System Architecture to implement the Scientific protocol for the digital representation a HC (Objective 1) WPS

WP7

Advanced Digitisation Technologies (O2)

Impact Assessment

Motion capturing and gesture recognition T5.1

Offline human motion capture

Real time human motion capture

Gesture and action recognition

RGB-based tracking of isolated hands and body

Human motion analytics & visualisation

RGB-based tracking of hands and body in interaction with objetcs

T5.3

WP6

T6.5

Virtual Human as story tellers T5.4

T5.5

System Interface & Integration (O5 & O6)

T5.2

T6.1

Pilots

Emotional autonomous interactive virtual characters

Glass

Mastic WP3

Semantic Representation of Crafts (O3, O4) Narratives & storytelling

Crafts Model T3.1

HC repository

T3.2

Storytelling & training T3.4

T6.2

Online platform

Mobile AR

MR training surface

T3.3

Semantic annotation

T3.5

Silk

Narratives generation T6.3

Storytelling authoring tools T6.5

Usability / UX evaluation

Fig. 14.5 System architecture for the MINGEI project

Fig. 14.6 Digital humans in the MINGEI project

232

14

Cultural Heritage

Based on the project scenario, two types of digital humans were developed: realistic talking digital humans for the exhibition sites, with full body and facial animations, and low-resolution digital humans for the mobile application.

References 1. Zara J, Benes B, Rodarte RR (2004) Virtual Campeche: a web based virtual three-dimensional tour. In: Proceedings of the Fifth Mexican International Conference in Computer Science (ENC 2004). IEEE Press, pp 133–140 2. Ruiz R, Weghorst S, Savage J, Oppenheimer P, Furness TF, Dozal Y (2002) Virtual reality for archeological maya cities. Congreso Virtual, INAH-UNESCO) Ciudad de Mexico 3. Lutz B, Weintke M (1999) Virtual dunhuang art cave: a cave within a CAVE. EG ’99: Comput Graphic Forum 18(3):257–264 4. Magnenat-Thalmann N, Pandzic I, Moussaly JC (1997) The making of the Terra-Cotta Xian soldiers. Digitalized’97: Proceedings in Digital Creativity, pp 66–73 5. Kalra P, Magnenat-Thalmann N, Moccozet L, Sannier G, Aubel A, Thalmann D (1998) Realtime animation of realistic virtual humans. IEEE Comput Graph Appl 18(5):42–56 6. Erim KT (1967) De Aphrodisiade. American J of Archaeology 71:233–243 7. Rogers J (2006) Sinan: makers of Islamic civilization. I. B. Tauris, London 8. de Heras Ciechomski P, Schertenleib S, Maim J, Thalmann D (2005) Reviving the Roman Odeon of Aphrodisias: dynamic animation and variety control of crowds in virtual heritage. VSMM’05 9. Rindel JH, Gade AC, Lisa M (2006) The virtual reconstruction of the ancient Roman concert hall in Aphrodisias, Turkey. Proceedings of the Institute of Acoustics: The 6th International Conference on Auditorium Acoustics Vol. 28 (5–7 May 2006), pp 316–323 10. Vlahakis V, Ioannidis N, Karigiannis J, Tsotros M, Gounaris M, Stricker D, Gleue T, Daehne P, Almeida L (2002) Archeoguide: an augmented reality guide for archaeological sites. IEEE Comput Graph Appl 22(5):52–60 11. Vlahakis V, Karigiannis J, Tsotros M, Gounaris M, Almeida L, Stricker D, Gleue T, Christou IT, Carlucci R, Ioannidis N (2001) Archeoguide: first results of an augmented reality, mobile computing system in cultural heritage sites. In: VAST ’01: Proceedings of the 2001 conference on virtual reality, archeology, and cultural heritage. ACM Press, New York, pp 131–140 12. Dähne P, Karigiannis JN (2002) Archeoguide: system architecture of a mobile outdoor augmented reality system. In: ISMAR ’02: Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR’02). IEEE Computer Society, Washington, DC, p 263 13. Gleue T, Dähne P (2001) Design and implementation of a mobile device for outdoor augmented reality in the ARCHEOGUIDE project. In: VAST ’01: Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage. ACM Press 14. Stricker D, Kettenbach T (2001) Real-time markerless vision-based tracking for outdoor augmented reality applications. In: IEEE and ACM International Symposium on Augmented Reality (ISAR 2001). IEEE Computer Society, New York, pp 29–30 15. Papagiannakis G, Schertenleib S, O’Kennedy B, Arevalo-Poizat M, Magnenat-Thalmann N, Stoddart A, Thalmann D (2005) Mixing virtual and real scenes in the site of ancient Pompeii. Computer Animation & Virtual Worlds 16(1):11–24 16. Magnenat-Thalmann N, Papagiannakis G, Foni A, Arevalo-Poizat M, Cadi-Yazli N (2004) Simulating life in ancient sites using mixed reality technology. CEIG’04: XIV Congreso Espan ˜ol de Informatica Grafica 17. Cordier F, Magnenat-Thalmann N (2002) Real-time animation of dressed virtual humans. In: Eurographics Conference Proceedings. Blackwell, pp 327–335

References

233

18. Cadi Yazli N, Baka E, Magnenat-Thalmann N, Kaplanidi D, Partarakis N, Karuzaki E, Zidianakis M, Pattakos A, Zabulis X (2022) Modelling craftspeople for cultural heritage: a case study. Comput Animat Virtual Worlds 33:3–4 19. Mohr A, Gleicher M (2003) Building efficient, accurate character skins from examples. ACM Transactions on Graphics 22(3):562–568 20. Bierbaum A, Just C, Hartling P, Meinert K, Baker A, Cruz-Neira C (2001) VR juggler: a virtual platform for virtual reality application development. In: Proceedings of the Virtual Reality 2001 Conference (VR’01). IEEE Computer Society, pp 89–96

Chapter 15

Other VR Applications

This last chapter presents a few different VR applications in vehicle simulation, manufacturing, and entertainment.

15.1

Vehicle Simulators

It is common practice to develop vehicle simulators based on physical mock-ups. They might be equipped with hydraulic platforms or based on a real car placed in front of large rear-projection screens. The primary application of vehicle simulators is driver training and behavior analysis (see Fig. 15.1); they have also been used for virtual prototyping. Most car manufacturers use vehicle simulators as part of product conception. Car simulators allow engineers to test the car before it is built and evaluate ergonomic aspects, interior design, and even road behavior. For virtual prototyping, simulators based on physical reproduction of cabins require substantial time and cost to be manufactured. Therefore, they cannot be reconstructed each time to reflect every part updated on the CAD design. This difficulty appears when performing ergonomic and visibility evaluations according to changes in the car interior. Immersive virtual reality (VR) provides a natural alternative. Virtual prototypes can replace physical mock-ups for the analysis of design aspects like layout and packaging efficiency; visibility of instruments, controls, and mirrors; reachability and accessibility; clearances and collisions; human performance; and aesthetics and appeal. The goal is to immerse a person in the virtual car interior, to study the design and interact with the virtual car. Virtual car simulators suitable for analysis activities are still unfeasible due to the limitations of the technology, concerning the haptic feedback. Nevertheless, some prototypes have been developed and tested.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7_15

235

236

15

Other VR Applications

Fig. 15.1 A car driving simulator

Fig. 15.2 Virtual vehicle simulators (a) Vehicle simulator, (b) Virtual cockpit

In [1], Kallman et al. present a simulator system built for both training and ergonomics-related tasks. The system can be used in two different configurations: The first is based on a physical mock-up of the vehicle, equipped with a forcefeedback steering wheel, gearshift, and pedals (see Fig. 15.2a); the second

15.2

Manufacturing

237

configuration is based on a fully virtual control metaphor, allowing one to interact with the vehicle only through the use of motion trackers and data gloves. Virtual reality interfaces can also be used to teleoperate real vehicles. An example of this kind of application is presented by Ott et al. in [2]. The authors developed a virtual cockpit with haptic feedback provided by a Haptic WorkstationTM. Four alternative teleoperation interfaces were implemented. Each interface exploited different aspects of virtual reality and haptic technologies: realistic 3D virtual objects, haptic force-feedback, and free arm gestures. Tests with multiple users were conducted to evaluate and identify the best interface in terms of efficiency and subjective user appreciation. The interface that got the best evaluation was a gesture-based interface that used free arm gestures to drive the vehicle (a small robot). The authors concluded that an efficient interface for direct teleoperation must have rich visual feedback in the form of passive controls such as speedometers and direction indicators. Such visual aids were appreciated by users once they were released from the burden of manipulating the virtual steering wheel and throttle. Force feedback shall be exploited not to simulate tangible objects (interface resembling reality) but to drive the user’s movements (gesture-based interface). The free-form (gesture-based) interface was efficient because it did not require precise manipulations. It reduced the amount of concentration required to drive. The user could direct her attention to the rest of the visuals and use them to improve the driving. Virtual interfaces resembling real cockpits are usually more intuitive in the sense that users know immediately how they work, from previous real-world experience. Nevertheless, the available hardware can make them less efficient due to problems with the grasping mechanisms and force-feedback inherent to the Haptic WorkstationTM Other more advanced haptic interfaces may provide better results. The work of Salamin et al. [3] is another example of experimentation with virtual cockpits using the Haptic WorkstationTM In collaboration with the Renault Trucks Cockpit Studies Department, the authors developed a virtual cockpit with a tangible gearbox. The user was able to grasp and manipulate the gearbox lever and feel the mechanical forces due to the gearbox. Such forces were generated according to the specifications provided by Renault Trucks. Other forces were applied to constrain the user’s movements and provide some comfort by partially compensating for the weight of the exoskeleton (a “gravity compensation” mechanism described in [4]). Figure 15.2b shows the virtual cockpit and the virtual representation of the user’s hand.

15.2

Manufacturing

Manufacturing comprises some basic classes of engineering tasks. Such tasks can benefit from the application of VR through high-fidelity and interactive visualizations. The simulation of assembly processes requires considering several assembly

238

15

Other VR Applications

conditions: availability of assembly room, supply of materials, and handling of parts and assembly tools. A virtual environment allows one to interactively produce an assembly sequence plan that considers the real production environment. The basic principles of this planning are the structural and geometric construction of the product and all restrictions that arise from the limited assembly space and the restricted accessibility. In [5], Ou et al. present a Web-based manufacturing simulation system based on VRML (Virtual Reality Modeling Language). Another interesting work [6] uses X3D, the successor to VRML, as the basis for a distributed virtual manufacturing system for small and medium enterprises. The system allows one to perform collaborative tasks, including product design, manufacturing, and resource sharing, through the Web. In [7], Salonen et al. present a system that demonstrates the use of augmented reality and multimodal interfaces for assisting human operators during assembly tasks. The system uses a wooden 3D puzzle as a simplified assembly task in a factory. The prototype system uses 3D models generated using CAD-system and translated to STL (the standard for rapid prototyping systems) or VRML. The visualization can be done using a monitor or HMD. A Webcam is used to acquire the images of the user’s hands and pieces of the 3D puzzle. The wooden pieces have fiducial markers that can be recognized through image processing, using the Augmented Reality Toolkit ARToolkit see Sect. 6.3). Once the markers are recognized, the orientation of each part can be computed, and 3D images can be superimposed on the real image. The assembly work is described using an XML file describing the marker board file, the models, and their identifiers (the markers), and the work phases and parts that belong to each phase. When a work phase is going on, the application visualizes the part that belongs to that work phase and shows the required action. For example, pointing with an arrow to the next part that should be taken and displaying an animation of how to mount this part in the object being assembled. The user can move forward and backward in assembly phases using different modalities: keyboard, speech, or gesture commands. The most convenient modalities are speech and gesture user interfaces. The system can be controlled by choosing the appropriate function from a virtual menu using hand gestures. Augmented and mixed reality seem to be excellent tools for manufacturing planning, simulation, and training. In [8], Liverani et al. describe a system for realtime control of human assembling sequences of mechanical components. The development method they propose involves a CAD environment, a hardware system called Personal Active Assistant (PAA), and a set of mixed reality features. The whole scheme is targeted at reducing the gap between engineers and manual operators by means of CAD and mixed reality technologies. The system is based on a CAD assembly module and mixed reality wearable equipment. It can be used to improve several activities in the industrial field, such as operator professional training, optimal assembly sequence seeking, or on-field teleconferencing (suitable for remote collaboration or for full exploitation of concurrent engineering suggestions during design and setup stages). The main characteristic of the PAA is a realtime wireless linkage to a remote server or designer workstation, where project geometric database is stored. The mixed reality wearable equipment is a see-through

15.3

Entertainment

239

Fig. 15.3 Augmented reality for operator training

HMD and a head-mounted camera. The user can freely operate in the mixed environment, while the camera can record the human-driven assembly sequence and check the efficiency and correctness via object recognition: An incremental subassembly detection algorithm has been developed to achieve complex dataset monitoring. Conversely, designers or assembly planners can exploit the peculiarities of mixed reality–based assembly: A straightforward interaction with the assembly operator can be obtained by sending vocal advice or displaying superimposed visual information on the real scene. Another example of the use of augmented reality for operator training in a manufacturing context is the work of Vacchetti et al. [9], partially developed in the framework of the IST STAR European project. The system tracks the 3D camera position by means of a natural feature tracker, which, given a rough CAD model, can deal with complex 3D objects. The tracking algorithm is described in [10]. The tracking method is robust and can handle large camera displacements and aspect changes (see Fig. 15.3). The target applications are industrial maintenance, repair, and training. The tracking robustness makes the augmented reality (AR) system able to work in real environments, such as industrial facilities, not only in the laboratory. Real-time object recognition has been used to augment real scenes with virtual humans and virtual devices that show how to operate various types of industrial equipment.

15.3

Entertainment

Virtual reality constantly borrows video game technology (advanced rendering algorithms, user interaction paradigms and interfaces) and vice versa; some of the advanced concepts and technologies that were created in research laboratories have found their way into the game industry. VR devices like HMDs, data gloves, and sensors have been used with different degrees of success in video games. In some cases, advanced computer graphics are complemented with highly realistic interfaces to produce a compelling experience; see Fig. 15.4. One of the most successful stories is the case of the Nintendo Wii. The Wii remote game controller,

240

15

Other VR Applications

Fig. 15.4 Simulator or advanced video game

equipped with accelerometers and an infrared sensor that detects its spatial orientation, makes possible the creation of appealing games. VR and human–computer interaction researchers have studied this kind of gesture-based interface before and continue to test novel concepts. Sometimes researchers use commercially available devices. This is the case of Shirai et al. [11], who developed some motion-analysis methods using the Wii remote. A very representative example of VR research applied to the entertainment industry is the case of VR Studio, the Walt Disney Company’s center of excellence in real-time graphics and interactive entertainment. The studio was founded in 1992 to explore the potential of virtual reality technology for theme park attractions. They have also developed Toontown (https://en.wikipedia.org/wiki/Toontown_Online), one of the first massively multiplayer online worlds designed for children aged 7 and older. One of the first VR-based attractions they developed was an interactive system that uses HMDs to immerse multiple users in the movie Rocketeer. Another attraction based on the movie Aladdin also used an HMD but was a single-user system. VR Studio was responsible for the development of three of the main DisneyQuest attractions (a theme park that opened in Orlando in 1998): • Aladdin’s Magic Carpet Ride, a four-person race through the world of Aladdin. The players use custom-designed HMDs and climb aboard motorcycle-like vehicles to begin the game. • Hercules in the Underworld. This attraction uses a five-screen immersive projection theater (IPT). This is similar to a CAVE, but with an hexagonal shape. The players used 3D glasses and a joystick to navigate through the virtual world. Pirates of the Caribbean: Battle for Buccaneer Gold The game takes place in a four-screen immersive theater built around a ship-themed motion platform. The ship has a real steering wheel and six physical cannons. The players wear stereo glasses and all screens are in stereo.

15.4

Virtual Reality and Sport

241

Fig. 15.5 “Enigma of the Sphinx” using a multimodal interface: the Magic Wand

A historical review of VR Studio describing its research experience is presented by Mine in [12]. One of the most interesting lessons learned reported by Mine is the fact that immersive multiuser (CAVE-like) displays are very effective not only as game interfaces but also as design tools. This kind of display allows one to share experiences and ease communication between participants while examining a prototype or playing a game. Another important point reported by VR Studio is that the use of physical interfaces helps to immerse users in a VR experience. An example of this is the work of Abaci et al. [13], Enigma of the Sphinx. This is a VR-based game that uses a large rear-projection screen and a multimodal interface called “Magic Wand.” The Magic Wand is a stick equipped with a magnetic tracker. It uses both posture and speech recognition to let the user interact with objects and characters in a virtual world based on ancient Egypt (see Fig. 15.5). The VR game was played by several users in a special event; the Magic Wand was an intuitive interface that helped players to get more involved in the virtual world. The industry of video games has produced highly advanced interactive 3D (I3D) authoring tools and game engines [14]. These software tools are being used not only for developing new games but also all kinds of “serious” applications in the areas of simulation, training and education.

15.4 Virtual Reality and Sport One of the main benefits of VR in sports training is portability. VR can turn any gym or empty room into a state-of-the-art practice space. Additionally, VR allows users to interact with lifelike scenes and potential hazards without putting their bodies at risk. When athletes are injured, VR can allow positive sports psychology and faster recovery, especially for contact sports such as rugby or basketball. Even when sidelined, athletes can practice plays and mentally prepare for the next game.

242

15

Other VR Applications

Using VR in sports training allows coaches to collect critical metrics to improve a player’s performance. There is also a large impact of augmented reality in sports as in recent years, trackers, computer vision, and artificial intelligence have been widely used to analyze players’ performances. As a result, athletes will train differently, coaches will scout and make decisions differently, and people will watch sports differently. Fans will have a better understanding of how the players move on the court, where they like to shoot, how the defensive lines are positioned, etc., when AR graphics are displayed on the field. High-end graphics on the field will enhance the real game to the fullest. Fans will have the opportunity to interact with players and events on the court to display as much information as they want, just like they do in a video game. VR can provide an extensive immersive experience using specific simulators. For example, MIT developed in 2016 the Amphibian system [15]. Amphibian is a virtual reality system that provides an immersive SCUBA diving experience through a convenient terrestrial simulator. Users lie on their torso on a motion platform with their outstretched arms and legs placed in a suspended harness. Users receive visual and aural feedback through the Oculus Rift head-mounted display and a pair of headphones. Additionally, we have a simulation of buoyancy, drag, and temperature changes through various sensors. Preliminary deployment shows that the system has potential to offer a high degree of presence in VR. VR is the ideal technology to provide the way of playing together to collective games. September 15, 2022, marks the launch of NFL PRO ERA, the first officially licensed football simulation for VR headsets. The game was developed by athleteled sports technology company StatusPRO in collaboration with the National Football Association and features real NFL teams, players, and stadiums. This shows the impact of VR on the most popular sport in USA. But in this section, we will discuss two other sports where VR had an interesting development. Historically, one of the first experiences is “AnyOne for Tennis” [16], discussed in Sect. 5.4 already, developed in 1997 for Interactive Telecom in Geneva. Players were integrated into the virtual environment through head-mounted displays (HMDs), magnetic sensors, and data gloves. At the opening session, the player from University of Geneva was live on stage and approximately 60 kilometers from the other player from EPFL’s Computer Graphics Lab (VRLab) in Lausanne, Switzerland. To manage and control this shared, networked, virtual environment, the researchers used their Virtual Life Network (VLNet), a client/server network system with realistic virtual humans (avatars). Avatars support the deformation of the body while moving. In addition to virtual humans, synthetic referees are also included in the tennis game simulation. Recently, a complex VR game of volleyball was developed in Singapore with avatars and non-players characters (NPCs) communicating using a ball [17, 18]. The game of volleyball involves two teams with six players each. NPCs and avatars interact in the VR Volleyball game by gazing, detecting collisions, and performing miscellaneous actions (like picking up a volleyball and passing the ball to the server). It is possible to combine autonomous players with avatars in this implementation. A Kinect and Oculus are used to capture each user’s movements in the game. Efforts have been focused on the following:

15.4

Virtual Reality and Sport

243

• Interactions between computer agents and avatars: gazing volleyball, transferring ball, etc. • Physically modeling realistic behaviors of objects. For instance, the ball cannot appear or disappear at one place without player’s manipulation. • Realistic animations during interactions: walking, gazing, transferring, etc. Further, a preliminary comparative user study illustrates the importance of realistic interactions in object-driven interactive multiparty VR. Currently, the focus is on gazing and miscellaneous interactions. Audio feedback has also been introduced using 3D spatial sounds specific to each type of interaction. There is a unique sound for each type of ball interaction: contact (player, ground, net) or action (underhand, overhand, spike). A background commentary is also added when the game state changes. Future research should incorporate behavior and mood into the autonomous agent characteristics. Figure 15.6 shows the game and Fig. 15.7 the input and output.

Fig. 15.6 VR volleyball game Input

Head pose estimation

Occulus

User skeleton & position (Body tracking)

Depth camera

User skeleton & position (Body tracking)

Output

Occulus

3D immersive reality

Fig. 15.7 VR volleyball game input and output

Monitor

244

15

Other VR Applications

To assess the significance and effects of including realistic interactions in a multiparty VR setup, the authors conducted a preliminary comparative survey with 10 participants (eight males and two females). Their ages ranged from 20 to 40 years. Two versions of the game are available, one with realistic interactions and the other without them. Participants were asked to play both versions. They had no idea which version of the game they were playing. The participants are then asked to fill out the presence questionnaire. Nine out of 10 participants agree that adding realistic interactions to VR improves their experience. Additionally, all participants agreed that the addition of gazing interaction increased their feeling of being present in the same way as computer players. In addition, eight of 10 participants reported a better feeling of presence with virtual players when collision avoidance was included.

References 1. Kallmann M, Lemoine P, Thalmann D, Cordier F, Magnenat-Thalmann N, Ruspa C, Quattrocolo S (2003) Immersive vehicle simulators for prototyping, training and ergonomics. In: Proceedings of computer graphics international, pp 90–95 2. Ott R, Gutierrez M, Thalmann D, Vexo F (2005) VR haptic interfaces for teleoperation: an evaluation study. In: IV’2005, pp 788–793 3. Salamin P, Thalmann D, Vexo F (2007) Visualization learning for visually impaired people. In: The 2nd international conference of e-learning and games: edutainment 2007, pp 171–181 4. Ott R, Gutierrez M, Thalmann D, Vexo F (2005) Improving user comfort in haptic virtual environments through gravity compensation. In: Proceedings. First Joint Eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems. World Haptics conference, pp 401–409 5. Ou SC, Sung WT, Hsiao SJ (2002) Development of intelligent virtual reality web-based robotics for manufacturing applications. In: IEEE international conference on industrial technology (IEEE ICIT’02), pp 348–353 6. Jin L, Wen Z, Oraifige IA (2007) Distributed VR for collaborative design and manufacturing. In: IV ’07: Proceedings of the 11th international conference information visualization. IEEE Computer Society, Washington, DC, pp 792–797 7. Salonen T, Säaski J, Hakkarainen M, Kannetis T, Perakakis M, Siltanen S, Potamianos A, Korkalo O, Woodward C (2007) Demonstration of assembly work using augmented reality. In: CIVR ’07: Proceedings of the 6th ACM international conference on Image and video retrieval. ACM Press, New York, pp 120–123 8. Liverani A, Amati G, Caligiana G (2006) Interactive control of manufacturing assemblies with mixed reality. Integr Comput-Aided Eng 13(2):163–172 9. Vacchetti L, Lepetit V, Ponder M, Papagiannakis G, Thalmann D, Magnenat-Thalmann N, Fua P (2004) A stable real-time AR framework for training and planning in industrial environments. In: Ong S, Nee A (eds) Virtual and augmented reality applications in manufacturing. Springer, New York, pp 129–146 10. Vacchetti L, Lepetit V, Fua P (2004) Stable real-time 3D tracking using online and offline information. IEEE Trans Pattern Anal Mach Intell 26(10):1385–1391 11. Shirai A, Geslin E, Richir S (2007) WiiMedia: motion analysis methods and applications using a consumer video game controller. In: Sandbox ’07: Proceedings of the 2007 ACM SIGGRAPH symposium on Video games. ACM Press, New York, pp 133–140

References

245

12. Mine M (2003) Towards virtual reality for the masses: 10 years of research at Disney’s VR studio. In: EGVE ’03: Proceedings of the workshop on virtual environments 2003. ACM Press, New York, pp 11–17 13. Abaci T, de Bondeli R, Ciger J, Clavien M, Erol F, Gutíerrez M, Noverraz S, Renault O, Vexo F, Thalmann D (2004) Magic wand and the Enigma of the Sphinx. Comput Graph 28(4): 477–484 14. Blackman S (2005) Serious games...and less! SIGGRAPH Comput Graph 39(1):12–16 15. MIT’s virtual scuba simulator makes you feel like you are really underwater. Insider, May 4, 2016 16. Molet T, Aubel A, Capin T, Carion S, Lee E, Thalmann NM, Noser H, Pandzic I, Sannier G, Thalmann D (1999) Anyone for tennis? Presence MIT 8(2):140–156 17. Jain N, Mishra N, Thalmann NM (2021) A behavior model for improving realism in a VR team sport. In: Proceedings of the 30th IEEE international conference on robot and human interactive communication (RO-MAN) 18. Hai W, Jain N, Wydra A, Thalmann NM, Thalmann D (2018) Increasing the feeling of social presence by incorporating realistic interactions in multi-party VR. In: Proceedings of the 31st international conference on computer animation and social agents. ACM

References

1. LaViola JJ (2000) A discussion of cybersickness in virtual environments. SIGCHI Bull 32(1): 47–56 2. Cruz-Neira C, Sandin DJ, DeFanti TA, Kenyon RV, Hart JC (1992) The CAVE: audio visual experience automatic virtual environment. Commun ACM 35(6):64–72 3. Slater M, Wilbur S (1997) A framework for immersive virtual environments (FIVE): speculations on the role of presence in virtual environments. Presence Teleop Virt Environ 6:603– 616 4. Krueger M (1991) Artificial reality II. Addison-Wesley, Boston 5. Brooks FP, Ouh-Young M, Batter JJ, Kilpatrick PJ (1990) Project GROPE: Haptic displays for scientific visualization. In: SIGGRAPH ’90: Proceedings of the 17th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 177–185 6. Milgram P, Takemura H, Utsumi A, Kishino F (1995) Augmented reality: a class of displays on the reality-virtuality continuum. Proc SPIE Telemanipul Telepresence Technol 2351:282– 292 7. Stephenson N (1992) Snow crash (1993 paperback ed.). Bantam Books, New York, p 440. ISBN 9780553898194 8. Koduri RX (2021, December 14) Powering the metaverse. Intel 9. https://www.bloomberg.com/press-releases/2023-03-06/metaverse-market-to-hit-936-6-bil lion-by-2030-at-cagr-of-41-6-grand-view-research-inc 10. Martin R (ed) (1993) Directions in geometric computing. Information Geometers Ltd, Winchester 11. Hamilton WR (1844) On quaternions; or on a new system of imaginaries in algebra. Philosophical Magazine XXV:10–13 12. Shoemake K (1985) Animating rotation with quaternion curves. In: SIGGRAPH ’85: Proceedings of the 12th annual conference on computer graphics and interactive techniques. ACM Press, New York, p 2454 13. Lepetit V, Fua P (2005) Monocular model-based 3D tracking of rigid objects: a survey. Found Trends Comput Graph Vision 1(1):1–89 14. Grassia FS (1998) Practical parameterization of rotations using the exponential map. J Graphics Tools 3(3):29–48 15. Rogers D, Adams JA (1990) Mathematical elements for computer graphics, 2nd edn. McGraw-Hill, New York 16. Farin G (2002) Handbook of computer aided geometric design. Elsevier Science, Amsterdam 17. Cox MG (1972) The numerical evaluation of B-splines. J Inst Math Appl 10:134–149

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7

247

248

References

18. de Boor C (1972) On calculation with B-splines. J Approx Theory 6:50–62 19. Catmull E, Clark J (1978) Recursively generated B-spline surfaces on arbitrary topological meshes. Comput Aided Des 10(6):350–355 20. Shiue LJ, Jones I, Peters J (2005) A realtime GPU subdivision kernel. In: SIGGRAPH ’05: ACM SIGGRAPH 2005 papers. ACM Press, New York, pp 1010–1015 21. Loop C, Blinn J (2006) Real-time GPU rendering of piecewise algebraic surfaces. In: SIGGRAPH ’06: ACM SIGGRAPH 2006 papers. ACM Press, New York, pp 664–670 22. Whitted T (1980) An improved illumination model for shaded display. Commun ACM 23(6): 343–349 23. Gouraud H (1971) Continuous shading of curved surface. IEEE Trans Comput C-20(6): 623–629 24. Phong BT (1975) Illumination for computer generated pictures. Commun ACM 18(6): 311–317 25. Kajiya JT (1986) The rendering equation. In: SIGGRAPH ’86: Proceedings of the 13th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 143–150 26. Bentley JL, Friedman JH (1979) Data structures for range searching. ACM Comput Surv 11(4):397–409 27. Glassner AS (1984) Space subdivision for fast ray tracing. IEEE Comput Graph Appl 4(10): 15–22 28. Purcell TJ, Buck I, Mark WR, Hanrahan P (2005) Ray tracing on programmable graphics hardware. In: SIGGRAPH ’05: ACM SIGGRAPH 2005 courses. ACM Press, New York, p 268 29. Horn DR, Sugerman J, Houston M, Hanrahan P (2007) Interactive k-d tree GPU raytracing. In: I3D ’07: Proceedings of the 2007 symposium on Interactive 3D graphics and games. ACM Press, New York, pp 167–174 30. Wald I, Slusallek P (2001) State-of-the-art in interactive ray-tracing. Proc EUROGRAPHICS 2001:21–42 31. Friedrich H, Günther J, Dietrich A, Scherbaum M, Seidel HP, Slusallek P (2006) Exploring the use of ray tracing for future games. In: Sandbox ’06: Proceedings of the 2006 ACM SIGGRAPH symposium on videogames. ACM Press, New York, pp 41–50 32. Goral CM, Torrance KE, Greenberg DP, Battaile B (1984) Modeling the interaction of light between diffuse surfaces. In: SIGGRAPH ’84: Proceedings of the 11th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 213–222 33. Guitton P, Roman J, Subrenat G (1995) Implementation results and analysis of a parallel progressive radiosity. In: PRS ’95: Proceedings of the IEEE symposium on parallel rendering. ACM Press, New York, pp 31–38 34. Sillion F, Puech C (1994) Radiosity and global illumination. Morgan Kaufmann Publishers, San Francisco 35. Schöffel F (1997) Online radiosity in interactive virtual reality applications. In: VRST ’97: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 201–208 36. Carr NA, Hall JD, Hart JC (2003) GPU algorithms for radiosity and subsurface scattering. In: HWWS ’03: Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardware. Eurographics Association, Aire-la-Ville, pp 51–59 37. Jensen HW (1996) Global illumination using photon maps. In: Rendering techniques ’96: 7th Eurographics workshop on rendering, pp 21–30 38. Jensen HW (2001) Realistic image synthesis using photon mapping. AK Peters Ltd, Natick 39. Jensen HW (2004) A practical guide to global illumination using ray tracing and photon mapping. In: SIGGRAPH ’04: ACM SIGGRAPH 2004 Course notes. ACM Press, New York, p 20 40. Purcell TJ, Donner C, Cammarano M, Jensen HW, Hanrahan P (2003) Photon mapping on programmable graphics hardware. In: HWWS ’03: Proceedings of the ACM SIGGRAPH/

References

249

EUROGRAPHICS conference on Graphics hardware. Eurographics Association, Aire-laVille, pp 41–50 41. Haeberli P, Segal M (1993) Texture Mapping as a fundamental drawing primitive. In: Proceedings of the 4th Eurographics workshop on rendering, pp 259–266 42. Blinn JF, Newell ME (1976) Texture and reflection in computer generated images. Commun ACM 19(10):542–547 43. Policarpo F, Oliveira MM, Comba JLD (2005) Real-time relief mapping on arbitrary polygonal surfaces. In: I3D ’05: Proceedings of the 2005 symposium on Interactive 3D graphics and games. ACM Press, New York, pp 155–162 44. Policarpo F, Oliveira MM (2006) Relief mapping of non-height-field surface details. In: I3D ’06: Proceedings of the 2006 symposium on Interactive 3D graphics and games. ACM Press, New York, pp 55–62 45. Blinn JF (1978) Simulation of wrinkled surfaces. In: SIGGRAPH ’78: Proceedings of the 5th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 286–292 46. Cook RL (1984) Shade trees. In: SIGGRAPH ’84: Proceedings of the 11th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 223–231 47. Gumhold S, Huttner T (1999) Multiresolution rendering with displacement mapping. In: HWWS ’99: Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on graphics hardware. ACM Press, New York, pp 55–66 48. Wang L, Wang X, Tong X, Lin S, Hu S, Guo B, Shum HY (2003) View- dependent displacement mapping. In: SIGGRAPH ’03: ACM SIGGRAPH 2003 papers. ACM Press, New York, pp 334–339 49. Rost R (2006) OpenGL shading language. Addison-Wesley, Boston 50. Talton JO, Fitzpatrick D (2007) Teaching graphics with the OpenGL shading language. In: SIGCSE ’07: Proceedings of the 38th SIGCSE technical symposium on computer science education. ACM Press, New York, pp 259–263 51. Thalmann NM, Thalmann D (1991) Complex models for animating synthetic actors. IEEE Comput Graph Appl 11(5):32–44 52. Burtnyk N, Wein M (1980) Computer-generated key-frame animation. J Soc Motion Telev Eng:149–153 53. Catmull E (1978) The problems of computer-assisted animation. In: SIGGRAPH ’78: Proceedings of the SIGGRAPH ’78 conference, pp 348–353 54. Reeves W (1981) Inbetweening for computer animation utilizing moving point constraints. In: SIGGRAPH ’81: Proceedings of the SIGGRAPH ’81 conference, pp 263–269 55. Parke F (1982) Parameterized models for facial animation. IEEE Comput Graph Appl 2(9): 61–68 56. Steketee S, Badler N (1985) Parametric keyframe interpolation in incorporating kinetic adjustment and phrasing control. In: SIGGRAPH ’85: Proceedings of the SIGGRAPH ’85 conference, pp 255–262 57. Zeltzer D (1982) Representation of complex animated figures. In: Proceedings of the graphics Interface82 conference, pp 205–221 58. Kochanek D, Bartels R (1984) Interpolating splines with local tension, continuity, and bias control. In: SIGGRAPH ’84: Proceedings of the SIGGRAPH ’84 conference, pp 33–41 59. Fu K, Gonzalez CS, Lee G (1987) Robotics, control, sensing, vision, and intelligence. McGraw-Hill, New York, pp 52–76 60. Boulic R, Mas R (1996) Interactive computer animation. Prentice Hall, London. Chap. Hierarchical Kinematic Behaviors for Complex Articulated Figure 61. Badler N, Manoochehri K, Walters G (1987) Articulated figure positioning by multiple constraints. IEEE Comput Graph Appl 7(6):28–38 62. Zhao J, Badler N (1994) Inverse kinematics positioning using nonlinear programming for highly articulated figures. ACM Trans Graph 13(4):313–336

250

References

63. Philips C, Zhao J, Badler N (1991) Interactive real-time articulated figure manipulation using multiple kinematic constraints. In: SIGGRAPH ’91: Proceedings of the SIGGRAPH ’91 conference, pp 245–250 64. Phillips C, Badler N (1991) Interactive behaviors for bipedal articulated figures. In: SIGGRAPH ’91: Proceedings of the SIGGRAPH ’91 conference, pp 359–362 65. Witkin A, Popovic Z (1995) Motion warping. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 105–108 66. Popovic Z, Witkin A (1999) Physically based motion transformation. In: SIGGRAPH ’99: Proceedings of the SIGGRAPH ’99 conference, pp 11–20 67. Bruderlin A, Williams L (1995) Motion signal processing. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 97–104 68. Gleicher M (1998) Retargeting motion to new characters. In: SIGGRAPH ’98: Proceedings of the SIGGRAPH ’98 conference, pp 33–42 69. Monzani JS, Baerlocher P, Boulic R, Thalmann D (2000) Using an intermediate skeleton and inverse kinematics for motion retargeting. In: EUROGRAPHICS’00: Proceedings of the EUROGRAPHICS’00 conference, pp 11–19 70. Bindiganavale R, Badler NI (1998) Modeling and motion capture techniques for virtual environments, Springer: Lecture notes in artificial intelligence, pp 70–82. Chap. Motion abstraction and mapping with spatial constraints 71. Lee J, Shin SY (1999) A hierarchical approach to interactive motion editing for human-like figures. In: SIGGRAPH ’99: Proceedings of the 26th annual conference on computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Company, New York, pp 39–48 72. Lim IS, Thalmann D (2000) Solve customers’ problems: interactive evolution for tinkering with computer animation. In: SAC2000: Proceedings of the ACM symposium on applied computing, pp 404–407 73. Glardon P, Boulic R, Thalmann D (2004) PCA-based walking engine using motion capture data. In: CGI04: proceedings of computer graphics international 2004, pp 292–298 74. Huston R (1990) Multibody dynamics. Butterworth-Heinemann, Stoneham 75. Featherstone R (1986) Robot dynamics algorithms. Kluwer Academic Publishers, Norwell 76. Kenna MM, Zeltzer D (1996) Dynamic simulation of a complex human figure model with low level behavior control. Presence 5(4):431–456 77. Ko H, Badler N (1996) Animating human locomotion with inverse dynamics. IEEE Comput Graph Appl 16:50–59 78. Hodgins J, Wooten W, Brogan D, Brien JO (1995) Animating human athletics. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 71–78 79. Lazlo J, Panne MVD, Fiume E (1996) Limit cycle control and its application to the animation of balancing and walking. In: SIGGRAPH ’96: Proceedings of the SIGGRAPH ’96 conference, pp 155–162 80. Faloutsos P, van de Panne M, Terzopoulos D (2001) Composable controllers for physicsbased character animation. In: SIGGRAPH ’01: Proceedings of the SIGGRAPH ’01 conference, pp 251–260 81. Witkin A, Kass M (1988) Spacetime constraints. In: SIGGRAPH ’88: Proceedings of the SIGGRAPH ’88 conference, pp 159–168 82. Perlin K, Goldberg A (1996) Improv: a system for scripting interactive actors in virtual worlds. In: SIGGRAPH ’96: Proceedings of the SIGGRAPH ’96 conference, pp 205–216 83. Vosinakis S, Panayiotopoulos T (2003) A task definition language for virtual agents. WSCG 2003 J WSCG 11:512–519 84. Reynolds C (1987) Flocks and herds and schools: a distributed behavioral model. In: SIGGRAPH ’87: Proceedings of the SIGGRAPH ’87 conference, pp 25–34 85. Wilhelms J, Van Gelder A (1997) Anatomically based modeling. In: SIGGRAPH ’97: Proceedings of the SIGGRAPH ’97 conference, pp 173–180

References

251

86. Ridsdale G (1990) Connectionist modelling of skill dynamics. J Visualiz Comput Anim 1(2): 66–72 87. Noser H, Turner R, Thalmann D (1992) Interaction between L-systems and vector forcefield. In: CGI92: Proceedings of the computer graphics international ’92 conference, pp 747–761 88. Magnenat-Thalmann N, Thalmann D (1987) The direction of synthetic actors in the film “Rendez-vous a Montreal”. IEEE Comput Graph Appl 7(12):9–19 89. Magnenat-Thalmann N, Laperriere R, Thalmann D (1988) Joint-dependent local deformations for hand animation and object grasping. In: Proceeding of the graphics interface, vol 88, pp 26–33 90. Mohr A, Gleicher M (2003) Building efficient, accurate character skins from examples. ACM Trans Graph 22(3):562–568 91. Wang X, Phillips C (2002) Multi-weight enveloping: least-squares approximation techniques for skin animation. In: Proceedings of the symposium on computer animation, SCA02, pp 129–138 92. Chadwick J, Haumann D, Parent R (1989) Layered construction for deformable animated characters. In: SIGGRAPH ’89: Proceedings of the SIGGRAPH ’89 conference, pp 243–252 93. Thalmann D, Jianhua S, Chauvineau E (1996) Fast realistic human body deformations for animation and VR applications. In: Proceedings of computer graphics international. IEEE Computer Society Press, pp 166–174 94. Scheepers F, Parent R, Carlson W, May S (1997) Anatomy-based modeling of the human musculature. In: SIGGRAPH ’97: Proceedings of the SIGGRAPH ’97 conference, pp 163–172 95. Porcher-Nedel L, Thalmann D (1998) Real time muscle deformations using mass- spring systems. In: CGI98: Proceedings of the computer graphics international ’98 conference. IEEE Computer Society Press, pp 156–165 96. Aubel A, Thalmann D (2001) Interactive modeling of the human musculature. In: Proceedings of the computer animation 2001 conference, Seoul, pp 167–173 97. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: SIGGRAPH ’99: Proceedings of the SIGGRAPH ’99 conference, pp 187–194 98. James DL, Twigg CD (2005) Skinning mesh animations. ACM Trans Graph 24(3):399–407 99. Faloutsos P, Van de Panne M, Terzopoulos D (1997) Dynamic freeform deformations for animation synthesis. IEEE Trans Vis Comput Graph 3(3):201–214 100. Capell S, Green S, Curless B, Duchamp T, Popovic Z (2002) Interactive Skeleton- driven dynamic deformations. In: SIGGRAPH ’02: Proceedings of the SIGGRAPH ’02 conference, pp 41–47 101. Boulic R, Magnenat-Thalmann N, Thalmann D (1990) A global human walking model with real-time kinematics personification. Vis Comput 6(6):344–358 102. Thalmann D, Musse SR (2013) Crowd simulation, 2nd edn. Springer, New York 103. Multon F, France L, Cani-Gascuel M, Debunne G (1999) Computer animation of human walking: a survey. J Visualiz Comput Anim 10(1):39–54 104. van de Panne M (1997) From footprints to animation. Computer Graphics Forum, pp 211–223 105. Choi MG, Lee J, Shin SY (2003) Planning biped locomotion using motion capture data and probabilistic roadmaps. ACM Trans Graph 22(2):182–203 106. Wooten W, Hodgins J (2000) Simulating leaping, tumbling, landing and balancing humans. In: Proceedings of the IEEE international conference on robotics and automation, pp 656–662 107. Boulic R, Ulciny B, Thalmann D (2004) Versatile walk engine. J Game Dev:29–50 108. Bruderlin A, Calvert T (1996) Knowledge-driven, interactive animation of human running. In: Proceedings of the graphics interface 96 conference, pp 213–221 109. Sun H, Metaxas D (2001) Automating gait generation. In: SIGGRAPH ’01: Proceedings of the SIGGRAPH ’01 conference, pp 261–270

252

References

110. Chung S, Hahn J (1999) Animation of human walking in virtual environments. In: Proceedings of computer animation 1999 conference, pp 4–15 111. Unuma RTM, Anjyo K (1995) Fourier principles for emotion-based human figure. In: SIGGRAPH ’95: Proceedings of the SIGGRAPH ’95 conference, pp 91–96 112. Rose C, Cohen M, Bodenheimer B (1998) Verbs and adverbs: multidimensional motion interpolation. IEEE Comput Graph Appl 18(5):32–41 113. Park S, Shin H, Shin S (2002) On-line locomotion generation based on motion blending. In: Proceedings of the SIGGRAPH/eurographics symposium on computer animation 2002, pp 105–111 114. Kovar L, Gleicher M (2003) Flexible automatic motion blending with registration curves. In: Proceedings of the SIGGRAPH/eurographics symposium on computer animation 2003, pp 214–224 115. Glardon P, Boulic R, Thalmann D (2004) A coherent locomotion engine extrapolating beyond experimental data. In: Proceedings of Computer Animation Social Agent 2004, pp 73–84 116. Jolliffe IT (1986) Principal component analysis, Springer series in statistics. Springer, New York 117. Alexa M, Müller W (2000) Representing animations by principal components. In: Proceedings of the eurographics 2000 conference, vol 19, pp 291–301 118. Lim I, Thalmann D (2002) Construction of animation models out of captured data. In: Proceedings of IEEE conference multimedia and expo 2002, pp 829–832 119. H-anim: The humanoid animation working group. http://www.h-anim.org. Accessed July 2007 120. Alexa M (2002) Linear combination of transformations. In: SIGGRAPH ’02: Proceedings of the SIGGRAPH ’02 conference, pp 380–387 121. Murray MP (1967) Gait as a total pattern of movement. Am J Phys Med 46(1):290–333 122. Inman V, Ralston H, Todd F (1981) Human walking. Wiliams & Witkins, Baltimore 123. Tolani D, Goswami A, Badler N (2000) Real-time inverse kinematics techniques for anthropomorphic limbs. Graph Models Image Process 62(5):353–388 124. Levison L (1996) Connecting planning and acting via object-specific reasoning. PhD thesis, Philadelphia 125. Badler N, Bindiganavale R, Allbeck WSJ, Zhao L, Palmer M (2000) Parameterized action representation for virtual human agents. In: Embodied conversational agents. MIT Press, Cambridge, pp 256–284 126. Goncalves L, Kallmann M, Thalmann D (2002) Defining behaviors for autonomous agents based on local perception and smart objects. Comput Graph 26(6):887–897 127. Kallmann M, Thalmann D (2002) Modeling behaviors of interactive objects for real-time virtual environments. J Visual Lang Comput 13(2):177–195 128. Peters C, Dobbyn S, McNamee B, O’Sullivan C (2003) Smart objects for attentive agents. J WSCG 11 129. Cutkosky MR (1989) On grasp choice, grasp models, and the design of hands for manufacturing tasks. IEEE Trans Robot Autom 5(3):269–279 130. Mas R, Boulic R, Thalmann D (1997) Extended grasping behavior for autonomous human agents. In: AGENTS ’97: Proceedings of the first international conference on autonomous agents. ACM Press, New York, pp 494–495 131. Mas R, Thalmann D (1994) A hand control and automatic grasping system for synthetic actors. Comput Graph Forum 13(3):167–177 132. Baerlocher P (2001) Inverse kinematics techniques for the interactive posture control of articulated figures. PhD thesis, Ecole Polytechnique Fédérale de Lausanne (EPFL) 133. Wang X, Verriest JP (1998) A geometric algorithm to predict the arm reach posture for computer-aided ergonomic evaluation. J Visualiz Comput Anim 9(1):33–47 134. Wiley DJ, Hahn JK (1997) Interpolation synthesis of articulated figure motion. IEEE Comput Graph Appl 17(6):39–45

References

253

135. Huang Z, Boulic R, Magnenat Thalmann N, Thalmann D (1995) A multi-sensor approach for grasping and 3D interaction. In: Computer graphics: developments in virtual environments, pp 235–253 136. Espiau B, Boulic R (1985) Collision avoidance for redondants robots with proximity sensors. In: Third international symposium of robotics research 137. van de Panne M, Fiume E (1993) Sensor-Actuator Networks. In: Proceedings of the of SIGGRAPH-93: computer graphics, Anaheim, pp 335–342 138. Phillips CB, Badler NI (1988) JACK: a toolkit for manipulating articulated figures. In: UIST ’88: Proceedings of the 1st annual ACM SIGGRAPH symposium on user interface software. ACM Press, New York, pp 221–229 139. Badler NI, Phillips CB, Webber BL (1993) Simulating humans: computer graphics animation and control. Oxford University Press, New York 140. Douville B, Levison L, Badler NI (1996) Task-level object grasping for simulated agents. Presence 5(4):416–430 141. Trias TS, Chopra S, Reich BD, Moore MB, Badler NI, Webber BL, Geib CW (1996) Decision networks for integrating the behaviors of virtual agents and avatars. In: VRAIS ’96: Proceedings of the 1996 virtual reality annual international symposium (VRAIS 96). IEEE Computer Society, Washington, DC, p 156 142. Pollard NS, Zordan VB (2005) Physically based grasping control from example. In: SCA ’05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation. ACM Press, New York, pp 311–318 143. Ciger J, Abaci T, Thalmann D (2005) Planning with smart objects. In: WSCG ’2005 144. Mortara M, Patane G, Spagnuolo M, Falcidieno B, Rossignac J (2004) Blowing bubbles for the multi-scale analysis and decomposition of triangle meshes. Algorithmica Spec Issues Shape Algorithms 38(2):227–248 145. Mortara M, Patane G, Spagnuolo M, Falcidieno B, Rossignac J (2004) Plumber: a method for a multi-scale decomposition of 3D shapes into tubular primitives and bodies. In: SMI ’04: Proceedings of the ninth ACM symposium on solid modeling and applications. Eurographics Association, Aire-la-Ville, pp 339–344 146. Latombe J (1991) Robot motion planning. Kluwer Academic Publishers, Norwell 147. Kavraki L, Svestka P, Latombe J, Overmars M (1994) Probabilistic roadmaps for path planning in high-dimensional configuration spaces, Technical report. Stanford University, Stanford 148. LaValle SM (1998) Rapidly-exploring random trees: a new tool for path planning, Technical report 98-11. Department of Computer Science, Iowa State University 149. Simon T, Laumond J, Nissoux C (2000) Visibility based probabilistic roadmaps for motion planning. Adv Robot J 14(2) 150. Kuffner JJ, LaValle SM (2000) RT-connect: an efficient approach to single-query path planning. In: Proceedings of the ICRA 2000 conference, pp 995–1001 151. Kallmann M, Aubel A, Abaci T, Thalmann D (2003) Planning collision-free reaching motions for interactive object manipulation and grasping. In: Proceedings of the eurographics 2003 conference, vol 22, pp 313–322 152. Yamane K, Kuffner JJ, Hodgins JK (2004) Synthesizing animations of human manipulation tasks. ACM Trans Graph 23(3):532–539 153. Kallmann M, Mataric M (2004) Motion planning using dynamic roadmaps. In: Proceedings of the ICRA 2004 conference, pp 4399–4404 154. Leven P, Hutchinson S (2000) Motion planning using dynamic roadmaps. In: Proceedings of the fourth international workshop on the algorithmic foundations of robotics (WAFR), pp 363–376 155. Waters K (1987) A muscle model for animating three-dimensional facial expression. In: SIGGRAPH ’87: Proceedings of the SIGGRAPH ’87 conference, pp 17–24 156. Terzopoulos D, Waters K (1990) Physically based facial modelling, analysis and animation. J Visualiz Comput Anim 1(2):73–90

254

References

157. Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto 158. Hill DR, Pearce A, Wyvill B (1988) Animating speech: an automated approach using speech synthesized by rule. Vis Comput 3:277–289 159. Pelachaud C (1991) Communication and coarticulation in facial animation. PhD thesis, University of Pennsylvania 160. Cohen MM, Massaro DW, Thalmann NM, Thalmann D (1993) Models and techniques in Computer Animation. Springer, New York, pp 139–156. Chap. Modelling coarticulation in synthetic visual speech 161. Kshirsagar S, Garchery S, Magnenat-Thalmann N (2001) Feature point based mesh deformation applied to MPEG-4 facial animation. In: DEFORM ’00/AVATARS ’00: Proceedings of the IFIP TC5/WG5.10 DEFORM’2000 workshop and AVATARS’2000 workshop on deformable avatars. Kluwer, B.V., Deventer, pp 24–34 162. Kshirsagar S, Magnenat Thalmann N (2001) Principal components of expressive speech animation. In: Proceedings of the computer graphics international 2001 conference. IEEE Computer Society, pp 38–44 163. Kshirsagar S, Magnenat Thalmann N (2002) Virtual humans personified. In: Proceedings of the autonomous agents conference (AAMAS) 2002, pp 356–359 164. Renault O, Magnenat Thalmann N, Thalmann D (1990) A vision-based approach to behavioural animation. J Visualiz Comput Anim 1(1):18–21 165. Noser H, Renault O, Thalmann D, Magnenat Thalmann N (1990) Navigation for digital actors based on synthetic vision, memory and learning. Comput Graph Pergamon Press 19:7–19 166. Garcia-Rojas A, Vexo F, Thalmann D (2007) Semantic representation of individualized reaction movements for virtual human. Int J Virt Reality 6(1):25–32 167. de Sevin E, Thalmann D (2005) A motivational model of action selection for virtual humans. In: Proceedings of the computer graphics international (CGI2005) conference. IEEE Computer Society Press, New York, pp 213–220 168. Tyrrell T (1993) The use of hierarchies for action selection. Adapt Behav 1(4):387–420 169. Donnart JY, Meyer JA (1996) Learning reactive and planning rules in a motivationally autonomous animal. IEEE Trans Syst Man Cybern 26(3):381–395 170. Musse S, Cassol V, Thalmann D (2021) A history of crowd simulation: the past, evolution, and new perspectives. Vis Comput 37(12) 171. Musse S, Thalmann D (2001) A behavioral model for real-time simulation of virtual human crowds. IEEE Trans Vis Comput Graph 7(2):152–164 172. Emering L, Boulic R, Molet T, Thalmann D (2000) Versatile tuning of humanoid agent activity. Comput Graph Forum 19(4):231–242 173. Bouvier E, Guilloteau P (1996) Crowd simulation in immersive space management. In: Eurographics workshop on virtual environments and scientific visualization ’96. Springer, New York, pp 104–110 174. Brogan D, Hodgins J (1994) Robot herds: group behaviors for systems with significant dynamics. In: in Proceedings of artificial life IV, pp 319–324 175. Musse S, Babski C, Capin T, Thalmann D (1998) Crowd modeling in collaborative virtual environments. In: Proceedings of the ACM VRST 98, Taiwan, pp 115–123 176. Ulicny B, Thalmann D (2001) Crowd simulation for interactive virtual environments and VR training systems. In: Proceedings of the eurographics workshop on animation and simulation. Springer, New York, pp 163–170 177. Cremer J, Kearney J, Papelis Y (1995) HCSM: framework for behavior and scenario control in virtual environments. ACM Trans Model Comput Simul 5(3):242–267 178. Boulic R, Becheiraz P, Emering L, Thalmann D (1997) Integration of motion control techniques for virtual human and avatar real-time animation. In: Proceedings of the ACM VRST ’97, pp 111–118 179. Pettre J, de Heras P, Maim J, Yersin B, Laumond J, Thalmann D (2006) Real time navigating crowds: scalable simulation and rendering. Comput Anim Virt Worlds 16:445–456

References

255

180. Treuille A, Cooper S, Popovic Z (2006) Continuum crowds. In: Proceedings of the SIGGRAPH 06 conference, pp 1160–1168 181. Morini F, Mam J, Yersin B, Thalmann D (2007) Real-time scalable motion planning for crowds. In: Proceedings of the CyberWorlds 2007 conference, pp 144–151 182. Ondřej J, Pettré AH, Olivier SD (2010) A synthetic-vision based steering approach for crowd simulation. ACM Trans Graph 183. Dutra T, Priem G, Cavalcante-Neto J, Vidal C, Pettre J (2014) Synthetic vision-based crowd simulation: reactive vs. reactive planning approaches. In: Proceedings of the 27th conference on computer animation and social agents, pp 1–4 184. Oğuz O, Akaydın A, Yılmaz T, Güdükbay U (2010) Emergency crowd simulation for outdoor environments. Comput Graph 34(2):136–144 185. Charalambous P, Chrysanthou Y (2010) Learning crowd steering behaviors from examples. In: Boulic R, Chrysanthou Y, Komura T (eds) Motion in games. Springer, Berlin/ Heidelberg, p 35 186. Flagg M, Rehg JM (2013) Video-based crowd synthesis. IEEE Trans Visual Comput Graph 19(11):1935–1947 187. Zhao M, Turner SJ, Cai W (2013) A data-driven crowd simulation model based on clustering and classification. In: 2013 IEEE/ACM 17th international symposium on distributed simulation and real time applications, DS-RT ’13. IEEE Computer Society, New York, pp 125–134 188. Ju E, Choi MG, Park M, Lee J, Lee KH, Takahashi S (2010) Morphable crowds. ACM Trans Graph 29:6 189. Bicho AL, Rodrigues RA, Musse SR, Jung CR, Paravisi M, Magalh LP (2012) Simulating crowds based on a space colonization algorithm. Comput Graph 36(2):70–79 190. Durupinar F, Pelechano N, Allbeck J, Güdükbay U, Badler NI (2011) How the ocean personality model affects the perception of crowds. IEEE Comput Graph Appl 31(3):22–31 191. Okaya M, Takahashi T (2011) Human relationship modeling in agent-based crowd evacuation simulation. In: Kinny D, Hsu JYJ, Governatori G, Ghose AK (eds) Agents in principle, agents in practice. Springer, Berlin/Heidelberg 192. Li W, Allbeck JM (2011) Populations with purpose. In: Allbeck JM, Faloutsos P (eds) Motion in games. Springer, Berlin/Heidelberg, pp 132–143 193. Pelechano N, Allbeck JM, Badler NI (2007) Controlling individual agents in high-density crowd simulation. In: Proceedings of the 2007 ACM SIGGRAPH/eurographics symposium on computer animation, SCA ’07. Eurographics Association, Goslar, DEU, pp 99–108 194. Musse SR, Cassol VJ, Jung CR (2012) Towards a quantitative approach for comparing crowds. Comput Anim Virt Worlds 23(1):49–57 195. Wolinski D, Guy JS, Olivier AH, Lin M, Manocha D, Pettre J (2014) Parameter estimation and comparative evaluation of crowd simulations. Comput Graph Forum 33(2):303–312 196. Shoulson A, Marshak N, Kapadia M, Badler NI (2013) Adapt: the agent development and prototyping testbed. In: Proceedings of the ACM SIGGRAPH symposium on interactive 3D graphics and games, I3D ’13. Association for Computing Machinery, New York, pp 9–18 197. Berseth G, Kapadia M, Faloutsos P (2013) Steerplex: estimating scenario complexity for simulated crowds. In: International conference of motion on games. ACM, New York, pp 45: 67–45:76 198. Berseth G, Kapadia M, Haworth, Faloutsos P (2015) Steerfit: automated parameter fitting for steering algorithms. In: Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation, SCA ’14. Eurographics Association, Goslar, DEU, pp 113–122 199. Bera A, Kim S, Manocha D (2015) Efficient trajectory extraction and parameter learning for data-driven crowd simulation. In: Proceedings of graphics interface 2015, GI 2015. Canadian Human-Computer Communications Society, Toronto, pp 65–72 200. Başak AE, Güdükbay U, Durupınar F (2018) Using real life incidents for creating realistic virtual crowds with data-driven emotion contagion. Comput Graph 72:70–81 201. Liu W, Pavlovic V, Faloutsos KHP, Yoon S, Kapadia M (2017) Characterizing the relationship between environment layout and crowd movement using machine learning. In:

256

References

Proceedings of the tenth international conference on motion in games, MIG ’17. Association for Computing Machinery, New York 202. Ravichandran NB, Yang F, Peters C, Lansner A, Herman P (2018) Pedestrian simulation as multi-objective reinforcement learning. In: Proceedings of the 18th international conference on intelligent virtual agents, IVA ’18. Association for Computing Machinery, New York, pp 307–312 203. Amirian J, van Toll W, Hayet JB, Pettré J (2019) Data-driven crowd simulation with generative adversarial networks. In: Proceedings of the 32nd International Conference on Computer Animation and Social Agents, CASA ’19. Association for Computing Machinery, New York, pp 7–10 204. Testa E, Barros RC, Musse S (2019) Crowdest: a method for estimating (and not simulating) crowd evacuation parameters in generic environments. Vis Comput 35:1119–1130 205. Cassol VJ, Musse SR, Jung CR, Badler NI (2017) Simulating crowds in egress scenarios. Springer, London 206. Wong S, Wang YS, Tang PK, Tsai T (2017) Optimized evacuation route based on crowd simulation. Comput Visual Media 3:243–261 207. Dal Bianco CM, Braun A, Brasil J, Musse SR (2017) Preserving the motion features in non avoiding collision crowds. Comput Entertain 15(3):1–15 208. Mathew CDT, Knob PR, Musse SR, Aliaga DG (2019) Urban walkability design using virtual population simulation. Comput Graph Forum 38(1):455–469 209. Ren J, Xiang W, Xiao Y, Yang R, Manocha D, Jin X (2019) Heter-Sim: heterogeneous multiagent systems simulation by interactive data-driven optimization. IEEE Trans Vis Comput Graph:1 210. Ho RC, Wong SK, Chou YH, Lin GW, Tsai TY, Huang PH, Wang YS, Yeh IC (2016) An authoring framework for time dependent crowd simulation. In: SIGGRAPH ASIA 2016 posters, SA’16. Association for Computing Machinery, New York 211. Stüvel SA, Magnenat Thalmann N, Thalmann D, Stappen AF, Egges A (2017) Torso crowds. IEEE Trans Visual Comput Graph 23(7):1823–1837 212. Dickinson P, Gerling K, Hicks K, Murray J, Shearer J, Greenwood J (2019) Virtual reality crowd simulation: effects of agent density on user experience and behaviour. Virtual Reality 213. Hürst W, Geraerts R (2019) Augmented and virtual reality interfaces for crowd simulation software-a position statement for research on use-case-dependent interaction. In: 2019 IEEE virtual humans and crowds for immersive environments (VHCIE), pp 1–3 214. Moussaïd M, Kapadia M, Thrash T, Sumner RW, Gross M, Helbing D, Hölscher C (2016) Crowd behaviour during high-stress evacuations in an immersive virtual environment. J R Soc Interf 215. Kyriakou M, Chrysanthou Y (2018) How responsiveness, group membership and gender affect the feeling of presence in immersive virtual environments populated with virtual crowds. In: Proceedings of the 11th annual international conference on motion, interaction, and games, MIG ’18. Association for Computing Machinery, New York 216. Pelechano N, Allbeck JM (2016) Feeling crowded yet? Crowd simulations for VR. In: 2016 IEEE virtual humans and crowds for immersive environments (VHCIE), pp 17–21 217. Yang F, Shabo J, Qureshi A, Peters C (2018) Do you see groups? The impact of crowd density and viewpoint on the perception of groups. In: Proceedings of the 18th international conference on intelligent virtual agents, IVA ’18. Association for Computing Machinery, New York, pp 313–318 218. Latoschik ME, Kern F, Stauffert JP, Bartl A, Botsch M, Lugrin JL (2019) Not alone here? Scalability and user experience of embodied ambient crowds in distributed social virtual reality. IEEE Trans Visual Comput Graph 25(5):2134–2144 219. Durupınar F, Güdükbay U, Aman A, Badler NI (2016) Psychological parameters for crowd simulation: from audiences to mobs. IEEE Trans Visual Comput Graph 22(9):2145–2159 220. Favaretto R, Dihl L, Barreto R, Musse SR (2016) Using group behaviors to detect hofstede cultural dimensions. In: IEEE international conference on image processing (ICIP)

References

257

221. Dihl L, Testa ES, Knob P, da Silva GLB, Favaretto RM, de Alcântara MF, Musse SR (2017) Generating cultural characters based on hofstede dimensions. In: 2017 IEEE virtual humans and crowds for immersive environments (VHCIE), pp 1–5 222. Zhang P, Liu H, Ding Y (2015) Crowd simulation based on constrained and controlled group formation. Vis Comput 31(1):1432–2315 223. Bosse T, Hoogendoorn M, Klein MCA, Treur J, van der Wal CN (2011) Agent-based analysis of patterns in crowd behaviour involving contagion of mental states. In: Mehrotra KG, Mohan CK, Oh JC, Varshney PK, Ali M (eds) Modern approaches in applied intelligence: 24th international conference on industrial engineering and other applications of applied intelligent systems, IEA/AIE 2011, Syracuse, June 28–July 1, 2011, Proceedings, Part II, p. 677. Springer, Berlin/Heidelberg 224. Borges Fortes Neto A, Pelachaud C, Musse SR (2017) Giving emotional contagion ability to virtual agents in crowds. In: Beskow J, Peters C, Castellano G, O’Sullivan C, Leite I, Kopp S (eds) Intelligent virtual agents. Springer, Cham, pp 63–72 225. Başak A, Gudukbay U, Durupinar F (2018) Using real life incidents for realistic virtual crowds with data-driven emotion contagion. Comput Graph 226. Huang W, Terzopoulos D (2020) Door and doorway etiquette for virtual humans. IEEE Trans Visual Comput Graph 26(3):1502–1517 227. Kremer M, Haworth B, Kapadia M, Faloutsos P (2021) Modelling distracted agents in crowd simulations. Vis Comput 37(1):1432–2315 228. Schaffer D, Neto AB, Antonitsch A, Musse SR (2020) Towards animating virtual humans in flooded environments. In: ACM SIGGRAPH conference – motion in games, p 1 229. Usman M, Lee TC, Moghe R, Zhang X, Faloutsos P, Kapadia M (2020) A social distancing index: EVALUATING navigational policies on human proximity using crowd simulations. In: ACM SIGGRAPH conference – motion in games, p 1 230. Volonte M, Hsu Y, Liu K, Mazer JP, Wong S, Babu SV (2020) Effects of interacting with a crowd of emotional virtual humans on users’ affective and non-verbal behaviors. In: 2020 IEEE conference on virtual reality and 3D user interfaces (VR), pp 293–302 231. Koilias A, Mousas C, Anagnostopoulos CN (2020) I feel a moving crowd surrounds me: exploring tactile feedback during immersive walking in a virtual crowd. Comput Anim Virt Worlds 31(4–5):1963. https://doi.org/10.1002/cav.1963 232. Tripathi G, Singh K, Vishwakarma DK (2020) Crowd emotion analysis using 2d convents. In: 2020 third international conference on smart systems and inventive technology (ICSSIT), pp 969–974 233. Kielar P, Borrmann A (2020) An artificial neural network framework for pedestrian walking behavior modeling and simulation. Collect Dynam 5:290–298 234. Bansod S, Nandedkar A (2020) Crowd anomaly detection and localization using histogram of magnitude and momentum. Vis Comput 36:609–620 235. Sun J, Zhang J, Li Q, Yi X, Liang Y, Zheng Y (2020) Predicting citywide crowd flows in irregular regions using multi-view graph convolutional networks. IEEE Trans Knowl Data Eng 236. Zhao R, Hu Q, Liu Q, Li C, Dong D, Ma Y (2020) Panic propagation dynamics of high-density crowd based on information entropy and Aw-Rascle model. IEEE Trans Intell Trans Syst 21(10):4425–4434 237. Huang ZM, Chen WN, Li Q, Luo XN, Yuan HQ, Zhang J (2020) Ant colony evacuation planner: an ant colony system with incremental flow assignment for multipath crowd evacuation. IEEE Trans Cybern:1–14 238. Mirahadi F, McCabe BY (2021) Evacusafe: a real-time model for building evacuation based on Dijkstra’s algorithm. J Build Eng 239. Dijkstra EW (1959) A note on two problems in connexion with graphs. Numer Math 1(1): 269–271 240. Xie W, Lee EWM, Li T, Shi M, Cao R, Zhang Y (2021) A study of group effects in pedestrian crowd evacuation: experiments, modelling and simulation. Saf Sci 133:105029

258

References

241. Karamouzas I, Overmars M (2012) Simulating and evaluating the local behavior of small pedestrian groups. IEEE Trans Visual Comput Graph 18(3):394–406 242. Moussaïd M, Perozo N, Garnier S, Helbing D, Theraulaz G (2010) The walking behaviour of pedestrian social groups and its impact on crowd dynamics. PLoS One 5(4):1–7 243. Zhang Z, Jia L (2021) Optimal guidance strategy for crowd evacuation with multiple exits: a hybrid multiscale modeling approach. Appl Math Model 90:488–504 244. Helbing D, Molnar P (1995) Social force model for pedestrian dynamics. Phys Rev E 51:42– 82 245. Charrier R, Philippe J, Bertelle C (2021) Study of a model of nervousness propagation within virtual crowds. In: Prendinger H, Lester J, Ishizuka M (eds) Complex systems, smart territories and mobility. Springer, Cham, pp 187–200 246. Shi X, Xue S, Feliciani C, Shiwakoti N, Lin J, Li D, Ye Z (2021) Verifying the applicability of a pedestrian simulation model to reproduce the effect of exit design on egress flow under normal and emergency conditions. Physica A Stat Mech Application 562:125–347 247. Hernández-Orallo E, Armero-Martánez A (2021) How human mobility models can help to deal with covid-19. Electronics 10(1):33 248. Oliveira M, Crowcroft J, Slater M (2003) An innovative design approach to build virtual environment systems. In: in EGVE ’03: Proceedings of the workshop on virtual environments 2003. ACM Press, New York, pp 143–151 249. Hagsand O (1996) Interactive multiuser VEs in the DIVE system. IEEE Multimedia 3(1): 30–39 250. Greenhalgh C, Purbrick J, Snowdon D (2000) InsideMASSIVE-3:flexiblesupport for data consistency and world structuring. In: CVE ’00: Proceedings of the third international conference on collaborative virtual environments. ACM Press, New York, pp 119–127 251. Macedonia MR, Brutzman DP, Zyda MJ, Pratt DR, Barham PT, Falby J, Locke J (1995) NPSNET: a multi-player 3D virtual environment over the Internet. In: SI3D ’95: Proceedings of the 1995 symposium on interactive 3D graphics. ACM Press, New York, pp 93–94 252. Anderson D, Barrus J, Howard J, Rich C, Waters R (1995) Building multiuser interactive multimedia environments at MERL. IEEE Multimedia 2(4):77–82 253. Grimsdale C (1991) dVS-Distributed virtual environment system. In: Proceedings of Computer Graphics’91. Bleinheim Online, London, pp 163–170 254. Sense8 Corporation (1998) World toolkit reference manual – release 7. Mill Valley 255. Tramberend H (1999) Avocado: a distributed virtual reality framework. In: VR ’99: Proceedings of the IEEE virtual reality. IEEE Computer Society, pp 14–21 256. Bierbaum A, Just C, Hartling P, Meinert K, Baker A, Cruz-Neira C (2001) VR Juggler: a virtual platform for virtual reality application development. In: Proceedings of the virtual reality 2001 conference (VR’01). IEEE Computer Society, pp 89–96 257. Ponder M, Papagiannakis G, Molet T, Magnenat-Thalmann N, Thalmann D (2003) VHD++ development framework: towards extendible, component based VR/AR simulation engine featuring advanced virtual character technologies. In: Proceedings of computer graphics international (CGI). IEEE Computer Society Press, pp 96–104 258. Huang S, Baimouratov R, Nowinski WL (2004) Building virtual anatomic models using Java3D. In: VRCAI ’04: Proceedings of the 2004 ACM SIGGRAPH international conference on virtual reality continuum and its applications in industry. ACM Press, New York, pp 402–405 259. Dörner R, Grimm P (2000) Three-dimensional Beans–creating Web content using 3D components in a 3D authoring environment. In: VRML ’00: Proceedings of the fifth symposium on virtual reality modeling language (Web3D-VRML). ACM Press, New York, pp 69–74 260. Yimin C, Tao Z, Di W, Yongyi H (2002) A robot simulation, monitoring and control system based on network and Java3D. In: Proceedings of the 4th world congress on intelligent control and automation, pp 139–143

References

259

261. Kallmann M, Thalmann D (1999) Direct 3D interaction with smart objects. In: VRST ’99: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 124–130 262. Gutierrez M, Thalmann D, Vexo F (2005) Semantic virtual environments with adaptive multimodal interfaces. In: Proceedings of the 11th international conference on multimedia modelling (MMM2005), pp 277–283 also in Gutierrez M (2005) Semantic virtual environments. PhD thesis, Ecole Polytechnique Fédérale de Lausanne (EPFL) 263. Gutiérrez M, Lemoine P, Thalmann D, Vexo F (2004) Telerehabilitation: controlling haptic virtual environments through handheld interfaces. In: VRST ’04: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 195–200 264. Gutiérrez M, Vexo F, Thalmann D (2004) The mobile animator: interactive character animation in collaborative virtual environments. Proc IEEE VR:27–31 265. Steinicke F, Ropinski T, Hinrichs K (2005) A generic virtual reality software system’s architecture and application. In: ICAT ’05: Proceedings of the 2005 international conference on augmented tele-existence. ACM Press, New York, pp 220–227 266. Lee D, Lim M, Han S, Lee K (2007) ATLAS: a scalable network framework for distributed virtual environments. Presence Teleop Virt Environ 16(2):125–156 267. Lee D, Lim M, Han S (2002) ATLAS: a scalable network framework for distributed virtual environments. In: CVE ’02: Proceedings of the 4th international conference on collaborative virtual environments. ACM Press, New York, pp 47–54 268. Molet T, Aubel A, Capin T, Carion S, Lee E, Thalmann NM, Noser H, Pandzic I, Sannier G, Thalmann D (1999) Anyone for tennis? Presence MIT 8(2):140–156 269. Benford S, Bowers J, Fahlen L, Greenhalgh G, Mariani J, Rodden T (1995) Networked virtual reality and cooperative work. Presence MIT 4(4):364–386 270. Suler J (1997) Psychology of cyberspace – history of the palace. In: Psychology of cyberspace 271. https://www.activeworlds.com/ 272. https://en.wikipedia.org/wiki/Second_Life 273. Constine J (2019) Facebook announces Horizon, a VR massive-multiplayer world. TechCrunch, September 25 274. https://en.wikipedia.org/wiki/Internet_of_things 275. Sutikno T, Thalmann D (2022) Insights on the internet of things: past, present, and future directions. TELKOMNIKA Telecommun Comput Electron Control 20(6):1399–1420 276. Pasman W, Woodward C, Hakkarainen M, Honkamaa P, Hyväkka J (2004) Augmented reality with large 3D models on a PDA: implementation, performance and use experiences. In: VRCAI ’04: Proceedings of the 2004 ACM SIGGRAPH international conference on virtual reality continuum and its applications in industry. ACM Press, New York, pp 344–351 277. Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: International joint conference on artificial intelligence, pp 674–679 278. Baker S, Matthews I (2004) Lucas-Kanade 20 years on: a unifying framework. Int J Comput Vision:221–255 279. Vacchetti L, Lepetit V, Fua P (2004) Stable real-time 3D tracking using online and offline information. IEEE Trans Pattern Anal Mach Intell 26(10):1385–1391 280. Nister D, Naroditsky O, Bergen J (2004) Visual odometry. In: Conference on computer vision and pattern recognition, pp 652–659 281. Tang D, Chang HJ, Tejani A, Kim T-K (2014) Latent regression forest: structured estimation of 3D articulated hand posture. In: Proceedings of the IEEE conference on computing visual pattern recognition, Columbus, pp 3786–3793 282. Sun X, Wei Y, Liang S, Tang X, Sun J (2015) Cascaded hand pose regression. In: Proceedings of the IEEE conference on computing visual pattern recognition, Columbus, pp 824–832 283. Tompson J, Stein M, Lecun Y, Perlin K (2014) Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans Graph 33(5):169 284. Oberweger M, Wohlhart P, Lepetit V (2015) Hands deep in deep learning for hand pose estimation. In: Proceedings of the computer vision winter workshop, pp 21–30

260

References

285. Oberweger M, Wohlhart P, Lepetit V (2015) Training a feedback loop for hand pose estimation. In: Proceedings of the IEEE conference on computer vision, pp 3316–3324 286. Ge L, Liang H, Yuan J, Thalmann D (2016) Robust 3D hand pose estimation in single depth images: from single-view CNN to multi-view CNNs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3593–3601 287. Sinha A, Choi C, Ramani K (2016) Deephand: robust hand pose estimation by completing a matrix with deep features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4150–4158 288. Ye Q, Yuan S, Kim T-K (2016) Spatial attention deep net with partial PSO for hierarchical hybrid hand pose estimation. In: Proceedings of the European conference on computer vision, pp 346–361 289. Molchanov P, Yang X, Gupta S, Kim K, Tyree S, Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4207–4215 290. Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231 291. Tran D, Bourdev LD, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE conference on computer vision, pp 4489–4497 292. Camgoz NC, Hadfield S, Koller O, Bowden R (2016) Using convolutional 3D neural networks for user-independent continuous gesture recognition. In: Proceedings of the IEEE conference on pattern recognition, pp 49–54 293. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3D shapenets: a deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1912–1920 294. Qi CR, Su H, Niessner M, Dai A, Yan M, Guibas LJ (2016) Volumetric and multi-view CNNs for object classification on 3D data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5648–5656 295. Song S, Xiao J (2016) Deep sliding shapes for amodal 3D object detection in RGB-D images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 808–816 296. Maturana D, Scherer S (2015) VoxNet: A 3D convolutional neural network for real-time object recognition. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, pp 922–928 297. Maturana D, Scherer S (2015) 3D convolutional neural networks for landing zone detection from LiDAR. In: Proceedings of the IEEE conference on robotics and automation, pp 3471–3478 298. Yumer ME, Mitra NJ (2016) Learning semantic deformation flows with 3D convolutional networks. In: Proceedings of the European conference on computer vision, pp 294–311 299. Song S, Yu F, Zeng A, Chang AX, Savva M, Funkhouser T (2017) Semantic scene completion from a single depth image. In: Proceedings of the IEEE conference on computer vision pattern recognition, pp 1746–1754 300. Williams B, Narasimham G, Rump B, McNamara TP, Carr TH, Rieser J, Bodenheimer B (2007) Exploring large virtual environments with an HMD when physical space is limited. In: APGV ’07: Proceedings of the 4th symposium on applied perception in graphics and visualization. ACM Press, New York, pp 41–48 301. Langbehn E, Lubos P, Steinicke F (2018) Evaluation of locomotion techniques for roomscale VR: joystick, teleportation, and redirected walking. In: Proceedings of the virtual reality international conference – Laval Virtual (VRIC ’18). Association for Computing Machinery, New York, pp 1–9. Article 4

References

261

302. Ware C, Arthur K, Booth KS (1993) Fish tank virtual reality. In: CHI ’93: Proceedings of the INTERACT ’93 and CHI ’93 conference on human factors in computing systems. ACM Press, New York, pp 37–42 303. Arthur KW, Booth KS, Ware C (1993) Evaluating 3D task performance for fish tank virtual worlds. ACM Trans Inf Syst 11(3):239–265 304. Qi W, Russell I, Taylor M, Healey CG, Martens JB (2006) A comparison of immersive HMD, fish tank VR and fish tank with haptics displays for volume visualization. In: APGV ’06: Proceedings of the 3rd symposium on applied perception in graphics and visualization. ACM Press, New York. pp. 51–8 305. Pan X, Xu X, Dev S, Campbell AG (2021) 3D displays: their evolution, inherent challenges and future perspectives. In: Arai K (ed) Proceedings of the future technologies conference (FTC) 2021, volume 3. FTC 2021, Lecture notes in networks and systems, vol 360. Springer, Cham 306. Fitzmaurice GW, Zhai S, Chignell MH (1993) Virtual reality for palmtop computers. ACM Trans Inf Syst 11(3):197–218 307. Watsen K, Darken R, Capps W (1999) A handheld computer as an interaction device to a virtual environment. In: Proceedings of Third International Immersive Projection Technology Workshop (IPT 1999), Stuttgart 308. Hill L, Cruz-Neira C (2000) Palmtop interaction methods for immersive projection technology systems. In: Fourth international immersive projection technology workshop (IPT 2000) 309. Farella E, Brunelli D, Bonfigli M, Benini L, Ricco B (2002) Using palmtop computers and immersive virtual reality for cooperative archaeological analysis: the Appian way case study. In: International Conference on Virtual Systems and Multimedia (VSMM), Gyeongju 310. Farella E, Brunelli D, Bonfigli M, Benini L, Ricco B (2003) Multi-client cooperation and wireless PDA interaction in immersive virtual environment. In: Euromedia 2003 conference, Plymouth 311. Cheok A, Yang X, Ying Z, Billinghurst M, Kato H (2002) Touch-space: mixed reality game space based on ubiquitous, tangible, and social computing. Pers Ubiquit Comput 6:430–442 312. Bimber O, Raskar R (2007) Modern approaches to augmented reality. In: SIGGRAPH ’07: ACM SIGGRAPH 2007 courses. ACM Press, New York 313. Mine M (2003) Towards virtual reality for the masses: 10 years of research at Disney’s VR studio. In: EGVE ’03: Proceedings of the workshop on virtual environments 2003. ACM Press, New York, pp 11–17 314. Höllerer T, Kuchera-Morin J, Amatriain X (2007) The allosphere: a large-scale immersive surround-view instrument. In: EDT ’07: Proceedings of the 2007 workshop on emerging displays technologies. ACM Press, New York, p 3 315. Cruz-Neira C, Sandin DJ, DeFanti TA (1993) Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In: SIGGRAPH ’93: Proceedings of the 20th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 135–142 316. Jacobson J, Hwang Z (2002) Unreal tournament for immersive interactive theater. Commun ACM 45(1):39–42 317. Peternier A, Cardin S, Vexo F, Thalmann D (2007) Practical design and implementation of a CAVE environment. In: International conference on computer graphics, theory and applications GRAPP, pp 129–136 318. Larsson P, Vstfjll D, Kleiner M (2002) Better presence and performance in virtual environments by improved binaural sound rendering. In: Proceedings of the AES 22nd international conference on virtual, synthetic and entertainment audio, pp 31–38 319. Burgess DA (1992) Techniques for low cost spatial audio. In: UIST ’92: Proceedings of the 5th annual ACM symposium on user interface software and technology. ACM Press, New York, pp 53–59

262

References

320. Kapralos B, Mekuz N (2007) Application of dimensionality reduction techniques to HRTFS for interactive virtual environments. In: ACE ’07: Proceedings of the international conference on advances in computer entertainment technology. ACM Press, New York, pp 256–257 321. Kern AC, Ellermeier W (2020) Audio in VR: effects of a soundscape and movement-triggered step sounds on presence. Front Robot AI Sect Virt Environ 7 322. Naef M, Staadt O, Gross M (2002) Spatialized audio rendering for immersive virtual environments. In: VRST ’02: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 65–72 323. Larsson P, Västfjäll D, Olsson P, Kleiner M (2007) When what you hear is what you see: presence and auditory-visual integration in virtual environments. In: Proceedings of the 10th annual international workshop on presence (Barcelona), pp 11–18 324. Shilling RD, Shinn-Cunningham B (2002) Handbook of virtual environments: design, implementation, and applications. Lawrence Erlbaum Associates, Mahwah, pp 65–92. Chap. Virtual Auditory Displays 325. Insko B (2001) Passive hapics significantly enhance virtual environments. PhD thesis, University of North Carolina, Department of Computer Science 326. Tan HZ, Pentland A (2001) Fundamentals of wearable computers and augmented reality. Lawrence Erlbaum Associates, Mahwah, pp 579–598. Chap. Tactual Displays for Sensory Substitution and Wearable Computers 327. Robles-De-La-Torre G (2006) The importance of the sense of touch in virtual and real environments. IEEE MultiMedia 13(3):24–30 328. Salisbury K, Brock D, Massie T, Swarup N, Zilles C (1995) Haptic rendering: programming touch interaction with virtual objects. In: SI3D ’95: Proceedings of the 1995 symposium on interactive 3D graphics. ACM Press, New York, pp 123–130 329. Bejczy A, Salisbury JK (1980) Kinematic coupling between operator and remote manipulator. Adv Comput Technol 1:197–211 330. Kim W, Bejczy A (1991) Graphical displays for operator aid in telemanipulation. In: Proceedings of the IEEE international conference on systems, man and cybernetics, pp 1059–1067 331. Patrick N (1990) Design, construction, and testing of a fingertip tactile display for interaction with virtual and remote environments. Master’s thesis Department of Mechanical Engineering, MIT Press 332. Minsky M, Young Ming O, Steele O, Brooks FP, Behensky M (1990) Feeling and seeing: issues in force display. In: SI3D ’90: Proceedings of the 1990 symposium on Interactive 3D graphics. ACM Press, New York, pp 235–241 333. Burdea G, Zhuang J, Roskos E, Silver D, Langrana N (1992) A portable dextrous master with force feedback. Presence Teleop Virt Environ 1(1):18–27 334. Massie T, Salisbury JK (1994) The PHANTOM haptic interface: a device for probing virtual objects. In: Proceedings of the ASME winter annual meeting: symposium on haptic interfaces for virtual environment and teleoperator systems, pp 295–300 335. Jackson B, Rosenberg L (1995) Interactive technology and the new paradigm for healthcare. IOS Press, pp 147–151. Chap. Force Feedback and Medical Simulation 336. Hayward V, Astley OR, Cruz-Hernandez M, Grant D, Robles-De-La-Torre G (2004) Haptic interfaces and devices. Sens Rev 4(1):16–29 337. Caldwell D, Gosney C (1993) Enhanced tactile feedback (tele-taction) using a multi-functional sensory system. In: Proceedings of the IEEE international conference on robotics and automation, pp 955–960 338. Cardin S, Vexo F, Thalmann D (2006) Vibro-tactile interface for enhancing piloting abilities during long term flight. J Robot Mechatron 18(4):381–391 339. Cardin S, Vexo F, Thalmann D (2006) Wearable system for mobility improvement of visually impaired people. Visual Comput J 23:109–118 340. Cassinelli A, Reynolds C, Ishikawa M (2006) Haptic radar/extended skin project. In: SIGGRAPH ’06: ACM SIGGRAPH 2006 sketches. ACM Press, New York, p 34

References

263

341. Yamamoto A, Nagasawa S, Yamamoto H, Higuchitokyo T (2004) Electrostatic tactile display with thin film slider and its application to tactile telepresentation systems. In: VRST ’04: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 209–216 342. Ho HN, Jones LA (2007) Development and evaluation of a thermal display for material identification and discrimination. ACM Trans Appl Percept 4(2):13 343. Caldwell D, Lawther S, Wardle A (1996) Tactile perception and its application to the design of multi-modal cutaneous feedback systems. In: Proceedings of the IEEE international conference on robotics and automation, pp 3215–3221 344. Wall SA, Brewster S (2006) Sensory substitution using tactile pin arrays: human factors, technology and applications. Signal Process 86(12):3674–3695 345. Howe RD (2004) The shape of things to come: pin-based tactile shape displays. In: Proceedings of the 4th international conference eurohaptics 2004, Munich, 5–7 June 2004, vol 1996, pp 2–11 346. Summers IR, Chanter CM (2002) A broadband tactile array on the fingertip. J Acoust Soc Am 112(5):2118–2126 347. Bliss JC, Katcher MH, Rogers CH, Shepard RP (1970) Optical-to-tactile image conversion for the blind. IEEE Trans Man Mach Syst 11(1):58–65 348. Minamizawa K, Fukamachi S, Kajimoto H, Kawakami N, Tachi S (2007) Gravity grabber: wearable haptic display to present virtual mass sensation. In: SIGGRAPH ’07: ACM SIGGRAPH 2007 emerging technologies. ACM Press, New York, p 8 349. Iwata H (2005) Art and technology in interface devices. In: VRST ’05: Proceedings of the ACM symposium on virtual reality software and technology. ACM Press, New York, pp 1–7 350. Solazzi M, Frisoli A, Salsedo F, Bergamasco M (2007) A fingertip haptic display for improving local perception of shape cues. In: WHC ’07: Proceedings of the second joint eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems. IEEE Computer Society, Washington, DC, pp 409–414 351. Sato M (2002) SPIDAR and virtual reality. In: Proceedings of the 5th biannual world automation congress, pp 17–23 352. Murayama J, Bougrila L, Luo Y, Akahane K, Hasegawa S, Hirsbrunner B, Sato M (2004) SPIDAR G&G: a two-handed haptic interface for bimanual VR interaction. In: Proceedings of the 4th international conference eurohaptics 2004, Munich, 5–7 June 2004 353. Solis J, Avizzano CA, Bergamasco M (2002) Teaching to write Japanese characters using a haptic interface. In: HAPTICS ’02: Proceedings of the 10th symposium on haptic interfaces for virtual environment and teleoperator systems. IEEE Computer Society, Washington, DC, pp 255–262 354. Avizzano C, Portillo-Rodriguez O, Bergamasco M (2007) Assisting to sketch unskilled people with fixed and interactive virtual templates. In: Proceedings of IEEE international conference on robotics and automation, pp 4013–4017 355. Portillo-Rodriguez O, Avizzano C, Chavez-Aguilar A, Raspolli M, Marcheschi S, Bergamasco M (2006) Haptic desktop: the virtual assistant designer. In: Proceedings of the 2nd IEEE/ASME international conference on mechatronic and embedded systems and applications, pp 1–6 356. Iwata H (1990) Artificial reality with force-feedback: development of desktop virtual space with compact master manipulator. In: SIGGRAPH ’90: Proceedings of the 17th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 165–170 357. Ott R, Thalmann D, Vexo F (2006) Organic shape modeling through haptic devices. Comput Aided Des Appl 3(1–4):79–88 358. Ott R, Gutierrez M, Thalmann D, Vexo F (2006) Advanced virtual reality technologies for surveillance and security applications. In: VRCIA ’06: Proceedings of the 2006 ACM international conference on virtual reality continuum and its applications. ACM Press, New York, pp 163–170

264

References

359. Frisoli A, Rocchi F, Marcheschi S, Dettori A, Salsedo F, Bergamasco M (2005) A new forcefeedback arm exoskeleton for haptic interaction in virtual environments. In: WHC 2005: first joint Eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems, pp 195–201 360. Bergamasco M, Frisoli A, Barbagli F (2002) Haptics technologies and cultural heritage applications. In: CA ’02: Proceedings of the computer animation. IEEE Computer Society, Washington, DC, p 25 361. Iwata H, Yano H, Nakaizumi F, Kawamura R (2001) Project FEELEX: adding haptic surface to graphics. In: SIGGRAPH ’01: Proceedings of the 28th annual conference on computer graphics and interactive techniques. ACM Press, New York, pp 469–476 362. Poupyrev I, Nashida T, Maruyama S, Rekimoto J, Yamaji Y (2004) Lumen: interactive visual and shape display for calm computing. In: SIGGRAPH ’04: ACM SIGGRAPH 2004 emerging technologies. ACM Press, New York, p 17 363. Poupyrev I, Nashida T, Okabe M (2007) Actuation and tangible user interfaces: the Vaucanson duck, robots, and shape displays. In: TEI ’07: Proceedings of the 1st international conference on Tangible and embedded interaction. ACM Press, New York, pp 205–212 364. Rasch B, Büchel C, Gais S, Born J (2007) Odor cues during slow-wave sleep prompt declarative memory consolidation. Science 315(5817):1426–1429 365. Michael G, Jacquot L, Millot J, Brand G (2003) Ambient odors modulate visual attentional capture. Neurosci Lett 352(3):221–225 366. Simon SA, Ivan RG, deAraujo E, Nicolelis MAL (2006) The neural mechanisms of gustation: a distributed processing code. Nat Rev Neurosci 7:890–901 367. Drayna D (2005) Human taste genetics. Annu Rev Genomics Hum Genet 6:217–235 368. Bartoshuk L, Duffy V, Miller I (1994) PTC/PROP tasting: anatomy, psychophysics and sex effects. Physiol Behav 56:1165–1171 369. Heath TP, Melichar JK, Nutt DJ, Donaldson LF (2006) Human taste thresholds are modulated by serotonin and noradrenaline. J Neurosci 26(49):12664–12671 370. Gutierrez-Osuna R (2004) Encyclopedia of human-computer interaction. Berkshire Publishers, pp 507–511. Chap. Olfactory Interaction 371. https://en.wikipedia.org/wiki/Smell-O-Vision 372. https://en.wikipedia.org/wiki/Scent_of_Mystery 373. Kaye JJ (2004) Making scents: aromatic output for HCI. Interactions 11(1):48–61 374. Washburn DA, Jones LM (2004) Could olfactory displays improve data visualization? Comput Sci Eng 6(6):80–83 375. Yanagida Y, Kawato S, Noma H, Tomono A, Tetsutani N (2004) Projection- based olfactory display with nose tracking. In: VR ’04: Proceedings of the IEEE Virtual Reality 2004 (VR’04). IEEE Computer Society, Washington, DC, p 43 376. Brewster S, McGookin D, Miller C (2006) Olfoto: designing a smell-based interaction. In: CHI ’06: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press, New York, pp 653–662 377. Tominaga K, Honda S, Ohsawa T, Shigeno H, Ichi Okada K, Matsushita Y (2001) “Friend park”-expression of the wind and the scent on virtual space. In: VSMM ’01: Proceedings of the seventh international conference on virtual systems and multimedia (VSMM’01). IEEE Computer Society, Washington, DC, p 507 378. https://www.makertour.fr/projects/vaqso-bring-scent-to-virtual-reality 379. https://www.designboom.com/technology/sense-smell-vr-gaming-world-nosewise-odormachine-10-25-2022/ 380. Iwata H, Yano H, Uemura T, Moriya T (2004) Food simulator: a haptic interface for biting. In: VR ’04: Proceedings of the IEEE virtual reality 2004 (VR’04). IEEE Computer Society, Washington, DC, pp 51–57 381. Hashimoto Y, Nagaya N, Kojima M, Miyajima S, Ohtaki J, Yamamoto A, Mitani T, Inami M (2006) Straw-like user interface: Virtual experience of the sensation of drinking using a

References

265

straw. In: ACE ’06: Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology. ACM Press, New York, p 50 382. Resner BI (2001) Rover@Home: computer mediated remote interaction between humans and dogs. Master’s thesis program in media arts and sciences, School of Architecture and Planning, MIT Press 383. Maynes-Aminzade D (2005) Edible bits: seamless interfaces between people, data and food. In: CHI 2005: alt.chi, extended abstracts 384. Wolpaw JR et al (2002) Brain-computer interfaces for communication and control. Clin Neurophysiol 385. Martini ML, Oermann EK, Opie NL, Panov F, Oxley T, Yaeger K (2020) Sensor modalities for brain-computer interface technology: a comprehensive literature review. Neurosurgery 86(2):E108–E117 386. Bird JJ, Manso LJ, Ribeiro EP, Ekárt A, Faria DR (2018) A study on mental state classification using EEG-based brain-machine interface. In: 9th international conference on intelligent systems, Madeira Island 387. Bird JJ, Ekart A, Buckingham CD, Faria DR. Mental Emotional sentiment classification with an EEG-based Brain-machine interface. St Hugh’s College, University of Oxford. The international conference on digital image and signal processing (DISP’19), p 1029 388. Vanneste S, Song JJ, De Ridder D (2018) Thalamocortical dysrhythmia detected by machine learning. Nat Commun 9(1):1103 389. Cattan G, Andreev A, Visinoni E (2020) Recommendations for integrating a P300-based brain–computer interface in virtual reality environments for gaming: an update. Computers 9(4):92. https://doi.org/10.3390/computers9040092 390. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4):458–464 391. Sela G, Subag J, Lindblad A, Albocher D, Schein S, Elber G (2007) Real-time haptic incision simulation using FEM-based discontinuous free-form deformation. Comput Aided Des 39(8): 685–693 392. Wang P, Becker AA, Jones IA, Glover AT, Benford SD, Greenhalgh CM, Vloeberghs M (2007) Virtual reality simulation of surgery with haptic feedback based on the boundary element method. Comput Struct 85(7-8):331–339 393. Székely G, Brechbühler C, Dual J, Enzler R, Hug J, Hutter R, Ironmonger N, Kauer M, Meier V, Niederer P, Rhomberg A, Schmid P, Schweitzer G, Thaler M, Vuskovic V, Tröter G, Haller U, Bajka M (2000) Virtual reality-based simulation of endoscopic surgery. Presence Teleop Virt Environ 9(3):310–333 394. Körner O, Männer R (2003) Implementation of a haptic interface for a virtual reality simulator for flexible endoscopy. In: HAPTICS ’03: Proceedings of the 11th symposium on haptic interfaces for virtual environment and tele-operator systems (HAPTICS’03). IEEE Computer Society, Washington, DC, p 278 395. Wagner C, Schill M, Männer R (2002) Intraocular surgery on a virtual eye. Commun ACM 45(7):45–49 396. Warfield SK, Ferrant M, Gallez X, Nabavi A, Jolesz FA (2000) Real-time biomechanical simulation of volumetric brain deformation for image guided neurosurgery. In: Supercomputing ’00: Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM). IEEE Computer Society, Washington, DC, p 23 397. Rahman M, Mahmud P, Mashuk M et al (2013) Augmented and virtual reality based approaches in minimally invasive surgery training. In: International conference on informatics, electronics and vision (ICIEV), pp 1–4 398. Sørensen TS, Mosegaard J (2006) Virtual open heart surgery: training complex surgical procedures in congenital heart disease. In: ACM SIGGRAPH 2006 emerging technologies. ACM, p 35 399. http://miralab.com/project/intelligent-digital-surgeon-ids/

266

References

400. Yeh SC, Rizzo A, Zhu W, Stewart J, McLaughlin M, Cohen I, Jung Y, Peng W (2005) An integrated system: virtual reality, haptics and modern sensing technique (VHS) for post-stroke rehabilitation. In: VRST ’05: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 59–62 401. Jung Y, Yeh SC, Stewart J (2006) Tailoring virtual reality technology for stroke rehabilitation: a human factors design. In: CHI ’06: CHI ’06 extended abstracts on Human factors in computing systems. ACM Press, New York, pp 929–934 402. Jack D, Boian R, Merians A, Adamovich SV, Tremaine M, Recce M, Burdea GC, Poizner H (2000) A virtual reality-based exercise program for stroke rehabilitation. In: Assets ’00: Proceedings of the fourth international ACM conference on Assistive technologies. ACM Press, New York, pp 56–63 403. Jacobson J, Redfern MS, Furman JM, Whitney SL, Sparto PJ, Wilson JB, Hodges LF (2001) Balance NAVE: a virtual reality facility for research and rehabilitation of balance disorders. In: VRST ’01: Proceedings of the ACM symposium on Virtual reality software and technology. ACM Press, New York, pp 103–109 404. Deutsch JE, Latonio J, Burdea GC, Boian R (2001) Post-stroke rehabilitation with the rutgers ankle system: a case study. Presence Teleop Virt Environ 10(4):416–430 405. Krijn M, Emmelkamp PMG, Olafsson RP, Bouwman M, van Gerwen LJ, Spinhoven P, Schuemie MJ, van der Mast CAPG (2007) Fear of flying treatment methods: virtual reality exposure vs. cognitive behavioral therapy. Aviat Space Environ Med 78(2):121–128 406. Bouchard S, Côté S, St-Jacques J, Robillard G, Renaud P (2006) Effectiveness of virtual reality exposure in the treatment of arachnophobia using 3D games. Technol Health Care 14(1):19–27 407. Garcia-Palacios A, Hoffman HG, Richards TR, Seibel EJ, Sharar SR (2007) Use of virtual reality distraction to reduce claustrophobia symptoms during a mock magnetic resonance imaging brain scan: a case report. Cyberpsychol Behav 10(3):485–488 408. Grillon H, Riquier F, Herbelin B, Thalmann D (2006) Virtual reality as therapeutic tool in the confines of social anxiety disorder treatment. Int J Disabil Hum Dev 5(3) 409. Herbelin B, Benzaki P, Riquier F, Renault O, Grillon H, Thalmann D (2005) Using physiological measures for emotional assessment: a computer-aided tool for cognitive and behavioural therapy. Int J Disabil Hum Dev 4(4):269–277 410. Wilcox LM, Allison RS, Elfassy S, Grelik C (2006) Personal space in virtual reality. ACM Trans Appl Percept 3(4):412–428 411. Herbelin B, Riquier F, Vexo F, Thalmann D (2002) Virtual reality in cognitive behavioral therapy: a preliminary study on social anxiety disorder. In: 8th international conference on virtual systems and multimedia, VSMM2002 412. Cai Y, Chia NKH, Thalmann D, Kee NKN, Lu D, Zheng J, Thalmann NM (2013) Design and development of a virtual dolphinarium to increase functional developmental learning in children with autism spectrum disorder. IEEE Trans Neural Syst Rehabil Eng 21(2):208–217 413. Ackerman MJ (1998) The visible human project. Proc IEEE 86(3):504–511 414. North C, Shneiderman B, Plaisant C (1996) User controlled overviews of an image library: a case study of the visible human. In: DL ’96: Proceedings of the first ACM international conference on digital libraries. ACM Press, New York, pp 74–82 415. Lopez-Cano M, Rodriguez-Navarro J, Rodriguez-Baeza A, Armengol Carrasco M, Susın A (2007) A real-time dynamic 3D model of the human inguinal region for surgical education. Comput Biol Med 37(9):1321–1326 416. Aubel A, Thalmann D (2004) MuscleBuilder: a modeling tool for human anatomy. J Comput Sci Technol 19(5):585–595 417. Maciel A, Boulic R, Thalmann D (2007) Efficient collision detection within deforming spherical sliding contact. IEEE Trans Vis Comput Graph 13(3):518–529 418. Maciel A, Sarni S, Boulic R, Thalmann D (2005) Stress distribution visualization on pre and post-operative virtual hip joint. Comput Aided Orthop Surg 2005:298–301

References

267

419. Virtual physiological human: “Towards virtual physiological human: multilevel modelling and simulation of the human anatomy and physiology. WHITE PAPER, November 2005 420. Zara J, Benes B, Rodarte RR (2004) Virtual campeche: a web based virtual three-dimensional tour. In: Proceedings of the fifth Mexican international conference in computer science (ENC 2004). IEEE Press, New York, pp 133–140 421. Ruiz R, Weghorst S, Savage J, Oppenheimer P, Dozal YTF III (2002) Virtual reality for archeological Maya cities. In: Congreso Virtual, INAH- UNESCO) Ciudad de Mexico 422. Lutz B, Weintke M (1999) Virtual Dunhuang Art Cave: a cave within a CAVE. EG ’99 Comput Graph Forum 18(3):257–264 423. Magnenat-Thalmann N, Pandzic I, Moussaly JC (1997) The making of the Terra- Cotta Xian soldiers. In: Digitalized’97: Proceedings in digital creativity, pp 66–73 424. Kalra P, Magnenat-Thalmann N, Moccozet L, Sannier G, Aubel A, Thalmann D (1998) Realtime animation of realistic virtual humans. IEEE Comput Graph Appl 18(5):42–56 425. Erim KT (1967) De Aphrodisiade. Am J Archaeology 71:233–243 426. Rogers J (2006) Sinan: makers of Islamic civilization. I. B. Tauris, London 427. de Heras Ciechomski P, Schertenleib S, Maim J, Thalmann D (2005) Reviving the Roman Odeon of Aphrodisias: dynamic animation and variety control of crowds in virtual heritage. In: VSMM’05 428. Rindel JH, Gade AC, Lisa M (2006) the virtual reconstruction of the Ancient Roman Concert Hall in Aphrodisias, Turkey. In: Proceedings of the Institute of Acoustics: The 6th international conference on auditorium acoustics, vol 28, 5–7 May 2006, pp 316–323 429. Vlahakis V, Ioannidis N, Karigiannis J, Tsotros M, Gounaris M, Stricker D, Gleue T, Daehne P, Almeida L (2002) Archeoguide: an augmented reality guide for archaeological sites. IEEE Comput Graph Appl 22(5):52–60 430. Vlahakis V, Karigiannis J, Tsotros M, Gounaris M, Almeida L, Stricker D, Gleue T, Christou IT, Carlucci R, Ioannidis N (2001) Archeoguide: first results of an augmented reality, mobile computing system in cultural heritage sites. In: VAST ’01: Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage. ACM Press, New York, pp 131–140 431. Dähne P, Karigiannis JN (2002) Archeoguide: System Architecture of a Mobile Outdoor Augmented Reality System. In: ISMAR ’02: Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR’02). IEEE Computer Society, Washington, DC, p 263 432. Gleue T, Dähne P (2001) Design and implementation of a mobile device for outdoor augmented reality in the ARCHEOGUIDE project. In: VAST ’01: Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage. ACM Press, New York 433. Stricker D, Kettenbach T (2001) Real-time markerless vision-based tracking for outdoor augmented reality applications. In: IEEE and ACM international symposium on augmented reality (ISAR). IEEE Computer Society, New York, pp 29–30 434. Papagiannakis G, Schertenleib S, O’Kennedy B, Arevalo-Poizat M, Magnenat-Thalmann N, Stoddart A, Thalmann D (2005) Mixing virtual and real scenes in the site of ancient Pompeii. Comput Anim Virt Worlds 16(1):11–24 435. Magnenat-Thalmann N, Papagiannakis G, Foni A, Arevalo-Poizat M, Cadi-Yazli N (2004) Simulating life in ancient sites using mixed reality technology. In: CEIG’04: XIV Congreso Espan ˜ol de Informatica Grafica 436. Cordier F, Magnenat-Thalmann N (2002) Real-time animation of dressed virtual humans. In: Eurographics conference proceedings. Blackwell, pp 327–335 437. Cadi Yazli N, Baka E, Magnenat-Thalmann N, Kaplanidi D, Partarakis N, Karuzaki E, Zidianakis M, Pattakos A, Zabulis X (2022) Modelling craftspeople for cultural heritage: a case study. Comput Anim Virt Worlds 33:3–4 438. Kallmann M, Lemoine P, Thalmann D, Cordier F, Magnenat-Thalmann N, Ruspa C, Quattrocolo S (2003) Immersive vehicle simulators for prototyping, training and ergonomics. In: Proceedings of computer graphics international, pp 90–95 439. Ott R, Gutierrez M, Thalmann D, Vexo F (2005) VR haptic interfaces for teleoperation: an evaluation study. In: IV’2005, pp 788–793

268

References

440. Salamin P, Thalmann D, Vexo F (2007) Visualization learning for visually impaired people. In: The 2nd international conference of e-learning and games: edutainment 2007, pp 171–181 441. Ott R, Gutierrez M, Thalmann D, Vexo F (2005) Improving user comfort in haptic virtual environments through gravity compensation. In: Proceedings. First Joint Eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems. World Haptics conference, pp 401–409 442. Ou SC, Sung WT, Hsiao SJ (2002) Development of intelligent virtual reality web-based robotics for manufacturing applications. In: IEEE international conference on industrial technology (IEEE ICIT’02), pp 348–353 443. Jin L, Wen Z, Oraifige IA (2007) Distributed VR for collaborative design and manufacturing. In: IV ’07: Proceedings of the 11th international conference information visualization. IEEE Computer Society, Washington, DC, pp 792–797 444. Salonen T, Säaski J, Hakkarainen M, Kannetis T, Perakakis M, Siltanen S, Potamianos A, Korkalo O, Woodward C (2007) Demonstration of assembly work using augmented reality. In: CIVR ’07: Proceedings of the 6th ACM international conference on Image and video retrieval. ACM Press, New York, pp 120–123 445. Liverani A, Amati G, Caligiana G (2006) Interactive control of manufacturing assemblies with mixed reality. Integr Comput-Aided Eng 13(2):163–172 446. Vacchetti L, Lepetit V, Ponder M, Papagiannakis G, Thalmann D, Magnenat-Thalmann N, Fua P (2004) A stable real-time AR framework for training and planning in industrial environments. In: Ong S, Nee A (eds) Virtual and augmented reality applications in manufacturing. Springer, New York, pp 129–146 447. Shirai A, Geslin E, Richir S (2007) WiiMedia: motion analysis methods and applications using a consumer video game controller. In: Sandbox ’07: Proceedings of the 2007 ACM SIGGRAPH symposium on Video games. ACM Press, New York, pp 133–140 448. Abaci T, de Bondeli R, Ciger J, Clavien M, Erol F, Gutíerrez M, Noverraz S, Renault O, Vexo F, Thalmann D (2004) Magic wand and the Enigma of the Sphinx. Comput Graph 28(4): 477–484 449. Blackman S (2005) Serious games...and less! SIGGRAPH Comput Graph 39(1):12–16 450. MIT’s virtual scuba simulator makes you feel like you are really underwater. Insider, May 4, 2016 451. Jain N, Mishra N, Thalmann NM (2021) A behavior model for improving realism in a VR team sport. In: Proceedings of the 30th IEEE international conference on robot and human interactive communication (RO-MAN) 452. Hai W, Jain N, Wydra A, Thalmann NM, Thalmann D (2018) Increasing the feeling of social presence by incorporating realistic interactions in multi-party VR. In: Proceedings of the 31st international conference on computer animation and social agents. ACM

Index

A ActiveWorlds, 137 Ambient reflection, 40 Angular representations, 22–30 AnyOne for Tennis, 242 Argonne Remote Manipulator (ARM), 6, 185, 190 Aroma Aura, 197 Artificial intelligence (AI), v, 1, 11, 51, 56, 91, 104, 126, 138, 242 Artificial reality, 5, 189 Ascension Technology, 6, 63 Augmented reality (AR), vi, 8, 10, 137–139, 141–143, 146, 147, 165, 168, 210, 223, 224, 228–230, 238, 239, 242 Augmented reality toolkit (ARToolkit), 143, 146, 238 Augmented virtuality (AV), 9, 141–142 AURO-3D, 181 Autism, 209, 216–218 Avatar, 11, 125, 134, 136, 137, 211, 242 Axis angle, 24–25, 88, 89 B Behavioral animation, 56, 58, 73–74, 92, 104, 230 Bernstein polynomials, 34, 35 Bézier curves, 33–37, 39 Binaural sound, 174 Bitmoji, 135

Blockchain, 11 Brain-computer interface (BCI), v, 201–205 Brooks, F., 6 B-splines, 35–38, 70, 84, 87 C Cardano angles, 23 Cartesian coordinate system, 16, 18 Cathode-ray tubes (CRTs), 159, 160, 162–164 CAVE, 3, 6, 7, 81, 166, 168–170, 178, 191, 196, 225, 226, 240 Chronoception, 158 Color CIE model, 41 HSV model, 41 RGB model, 41 Computer animation, vi, 25, 55–75 Computer-vision, 61, 141, 142, 169, 242 Convolutional Neural Networks (CNNs), 148, 149 Convolvotron, 176 Coordinate systems, 15–17, 22–24, 32, 67, 146 Cryptocurrency, 11 Cultural heritage, v, vi, 191, 223–225, 228–231 CyberForceTM, 190 CyberGloveTM, 6, 190 CyberGraspTM, 184, 190 Cybersickness, 3, 6 CyberTouch™, 184, 185 CyberTrackTM, 190

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Gutiérrez A. et al., Stepping into Virtual Reality, https://doi.org/10.1007/978-3-031-36487-7

269

270 D Data glove, 2, 62, 130, 184, 185, 237, 239, 242 Deep learning, v, 11, 141, 146–149, 212 Diffuse reflection, 40, 41, 43 Digital twin, 10, 11, 138 Direct kinematics method, 65 Disney, Walt, 240 Dolby Atmos, 181 DTS:X, 181 Dynamic animation, 55, 72 Dynamic simulation, 58, 72 E Edge computing, 10, 212 Electroencephalography (EEG), 202–205 Emoji, 134, 135 “Enigma of the Sphinx”, 241 Entertainment, vi, 1, 7, 178, 199, 204, 239–241 Equilibrioception, 155 Euclidean space, 16, 23 Euler angles, 22–24, 27, 28 Exponential maps, 27–28 Eye-tracking, 167, 205, 215, 216 F Facebook, v, 7–9, 135, 137 Facebook Avatar, 135, 136 Facial Animation Parameters (FAPs), 100 Facial Definition Parameter (FDP), 100, 101 Fantasound, 178 FEELEX, 191 Fisher, S., 5 Fish tank VR, 166, 167 5G, 10, 212 Flock of Birds, 6 Free-form deformation (FFD), 84 G Game engine, v, 15, 51–52, 241 Gesture, 5, 59, 105, 129, 148, 184, 212, 216, 217, 229, 237, 238 Gimbal lock, 23, 24, 27 Global illumination, 42–46, 51, 130 Godot, 52 Gouraud shading, 41–42, 50 GPU programming techniques, 228 Grasping, 91–98, 189, 190, 237 GROPE, 6, 185, 190

Index H H-ANIM, 57, 88 Haptic interfaces active interfaces, 186, 187 electrotactile displays, 188 exoskeletons, 189, 190 haptic surfaces, 191 kinesthetic displays, 187, 189 mechanical displays, 188 passive interfaces, 186, 187 single point force-feedback, 189 tactile displays, 187, 188 thermal displays, 188 vibrotactile displays, 187, 189 Haptic Master, 189 Haptic rendering, 184–186 Haptics, 2–4, 6, 61, 183–191, 209, 210, 213, 214, 220, 235, 237 Haptic sensory system, 184 Haptic WorkstationTM, 190, 191, 214, 237 Head-mounted display (HMD) field of view, 164, 166 see-through, 230 Head-related transfer function (HRTF), 175–176 Health sciences, vi, 209 Heilig, M., 4 History of virtual reality, v, 4–8 Homogeneous coordinates, 18–19, 21, 30, 37, 50 Human localization cues (sound), 176 I Illumination models, 39–46, 49 Immersion, 2–3, 6, 9, 81, 164, 169, 184, 186, 190, 212, 225, 230 Immersion Corporation, 6 In-between frame, 64 Individuality, 69, 226 Intelligent Digital Surgeon (IDS), 212, 213 Interactive 3D software OpenSceneGraph, 52, 126 Quest3D, 8 Unreal game engine, 52 Internet of Things (IoT), 10, 12, 125, 138–139 Interpolation, 27, 28, 31, 33, 35, 38, 41, 42, 50, 56, 64–66, 87–89, 93, 99 Interpolation between frames, 56, 64–65 Inter-pupillary distance, 164 Inverse kinematics, 58, 62, 67–70, 91, 93, 97, 98

Index J Joint angles, 55, 57, 58, 61, 67, 83, 84, 86–88, 97 K Key-frame, 58, 64, 65, 87, 88, 99, 100, 113 Key-frame animation, 56, 58, 64–66 Kinesthetic system, 183, 184 L Lambert’s cosine law, 40 Lanier, J., 6, 7 Laposky, B., 159 Level of details (LODs), 112, 133 Liquid crystal displays active display area, 162 active-matrix, 162 aspect ratio, 162 brightness, 162 color support, 162 contrast ratio, 162 dot pitch, 162 input ports, 162 passive-matrix, 162 resolution, 162 response time, 162 viewing angle, 162 Local illumination, 40–43, 49 Loudspeaker location, 176 M Magnetic motion capture, 62 Manufacturing, vi, 33, 93, 167, 237–239 Meta-ball, 226, 227 Metaverse, v, vi, 1, 8–12, 125–139 Mixed reality, vi, 9, 116, 137, 141–149, 205, 238, 239 Mobile animator, 129, 168 Monaural sound, 174 Motion capture, 58–64, 69, 87, 88, 97, 142, 146, 147 Motion control, 55, 56, 58, 65, 82, 110 MPEG4, 57 Multimodal interaction, 228 Multimodal interface Magic Wand, 241 N Nociception, 157–158 Nonparametric curves, 31–32 Non-player character (NPC), 11

271 Nonrational B-splines, 37 NURB, 37–39, 218 O Oculus, v, 7, 8, 165, 242 Olfactory displays, 196, 197 Online world, 135–138, 240 OpenSceneGraph, 52, 126 Optical motion capture, 59–61, 88, 143 Orthographic projection, 29–30 P Palace, 136, 137 Pallesthesia, 155 PHANTOM®, 186, 189, 210 Phong shading, 40, 42 Photon mapping, 43, 45–46 Physiome Project, 219, 220 Plasma displays, 162–164 Polar coordinate system, 16 Polhemus, 6, 62, 142 Posture, 28, 57, 59, 69, 70, 83, 85, 87–89, 91, 93, 96–98, 105, 108, 109, 112, 113, 184, 187, 190, 241 Presence, 2–4, 7, 8, 81, 96, 116, 132, 173, 183, 195–197, 226, 242, 244 Principal component analysis (PCA), 71, 83, 88–90 Procedural animation, 71 Projection matrices, 50 Proprioception, 155, 183 Psychological therapy, 125, 213, 215–216 Q Quaternions, 22, 25–28 R Radiosity, 43–46 Raster images, 35 Rational B-splines, 37–38 Ray tracing, 43–46, 51 Reality continuum, vi, 141 Real-time tracking, 146 Rendering, vi, 10, 15, 17, 28, 39–52, 86, 110–113, 127–129, 131, 174, 175, 177, 180, 184, 185, 227, 228, 230, 239 Rendering equation, 42, 43 Rendering pipeline fixed pipeline, 49–50 programmable pipeline, 50–51

272 Rutgers ankle, 213 Rutgers Master, 186 S Second Life, 134, 135, 137 Sense, v, vi, 2–4, 7, 10, 22, 81, 91, 106, 138, 139, 153–155, 157, 159, 169, 173, 174, 176, 179, 183, 184, 195–199, 205, 237 Sensorama, 4, 159, 195 Sensors, v, 10, 12, 61–63, 74, 94, 103, 104, 138, 146, 153, 159–170, 184, 185, 188, 191, 196, 198, 201, 204, 205, 213, 229, 239, 240, 242 Shaders fragment, 50, 113 vertex, 51, 113, 228 Shutter glasses, 167, 177, 216 6G, 10 Skinning, 51, 81–86, 228 Smart object, 91–93, 97, 126 Smell interfaces, 196–198 Smell, sense of, 195, 198 Snow Crash, 9 Sound cards, 180 Sound engines, 177, 180 Sound rendering, 6, 110, 174–175, 228 Spatial sound, 173–178, 243 Specular reflection, 40–42, 47 Spherical coordinate system, 16–17 SPIDAR, 189 Splines bias, 66 continuity, 66 tension, 66 Stereo sound, 4, 174, 176 Surround-sound, 169, 177–181 Sutherland, I., 5, 159, 185 T Tait-Bryan angles, 23 Taste interfaces drink simulator, 198 edible user interfaces, 199 food simulator, 198 Taste, sense of, 195, 196, 198, 199 Teleoperation, 185, 190, 191, 237 Teleoperator, 185 Telerehabilitation, 191, 214 Texture mapping bump mapping, 47, 51 displacement mapping, 47, 51 environment mapping, 47, 51

Index Textures, 15, 17, 46–51, 110–113, 144, 186, 189, 195, 198, 224, 226 Thermoception, 157 3D hand pose, v, 146–149 3D reconstruction, 10, 11 3D sound imaging, 176 Transformation matrices, 15, 18–22, 29, 30, 49, 57, 83 Transformations modeling transformations, 49 modelview matrix, 49 viewing transformations, 49 Trigonometric identities, 21 U Unity, 15, 52 Unreal Engine, 52 V VAQSO, 197 Vector, 15, 18–22, 24–27, 30, 32, 33, 35, 36, 38, 41, 42, 65, 68, 83, 88, 89 Vector graphics, 35 Vehicle simulators, 4, 235–237 VIEW system, 6, 174 Virtual anatomy, 218–220 Virtual assembly, 238 Virtual characters locomotion, 86, 87 Virtual cockpits, 237 Virtual dolphin, 216–217 Virtual guide, 212 Virtual humans clothes simulation, 230 hair simulation, 230 Virtual life, 106, 108, 230, 242 Virtual physiological human (VPH), 219, 220 Virtual reality exposure system (VRE), 215 Virtual Reality Modeling Language (VRML), 57, 238 Virtual rehabilitation, 213–217 Virtual surgery, v, 209–212 Virtual training, 188 Visible human project (VHP), 218 Vision, vi, 7, 61, 103, 104, 114, 138, 139, 141, 142, 176, 184, 242 VPL Research, 6, 7 X X3D, 238