Digital Medicine: Bringing Digital Solutions to Medical Practice 9789814968737, 9781003386070

Citation preview

Digital Medicine

Jenny Stanford Series on Next-Generation Medicine: Vol. 1

Digital Medicine

Bringing Digital Solutions to Medical Practice

edited by

Ralf Huss

Published by Jenny Stanford Publishing Pte. Ltd. 101 Thomson Road #06-01, United Square Singapore 307591

Email: [email protected] Web: www.jennystanford.com British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

Digital Medicine: Bringing Digital Solutions to Medical Practice Copyright © 2023 by Jenny Stanford Publishing Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. Cover image: Designed by Bloom Project, Munich & Nuremberg, Germany ISBN 978-981-4968-73-7 (Hardcover) ISBN 978-1-003-38607-0 (eBook)

Contents

Foreword xxiii Acknowledgments xxvii Introduction to Digital Medicine: Bringing Digital Solutions to Medical Practice xxix

Part I: Digital Science 1. Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge 3 Julia E. Vogt, Ece Ozkan, and Riĉards Marcinkeviĉs 1.1 Machine Learning: An Overview 4 1.1.1 Navigating through Concepts 5 1.1.2 Machine Learning Approaches and Tasks 5 1.2 Machine Learning for Healthcare 8 1.2.1 Supervised Learning 9 1.2.2 Semi-supervised Learning 11 1.2.3 Unsupervised Learning 11 1.2.4 Reinforcement Learning 12 1.3 Limitations, Challenges, and Opportunities 12 1.3.1 Lack of Data, Labels, and Annotations 14 1.3.2 Learning across Domains, Tasks, and Modalities 14 1.3.3 Data Sharing, Privacy, and Security 15 1.3.4 Interpretable and Explainable Machine Learning 15 1.3.5 Toward Causally Informed Models 16 1.4 Conclusion: Quo Vadis? 16

2. Data Access and Use 35 Iñaki Soto-Rey, Sebastian Hofmann, Holger Storf, Dennis Kadioglu, Danny Ammon, and Michael Storck 2.1 Legislation 36

vi

Contents



2.2

2.3

Data Access 2.2.1 Governance 2.2.1.1 Roles and rights 2.2.1.2 Data access and governance process 2.2.2 Pseudonymization and Anonymization 2.2.2.1 Pseudonymization 2.2.2.2 Anonymization Data Use 2.3.1 Storage as a Specific Use Case 2.3.1.1 Patient registries 2.3.1.2 Research databases 2.3.2 Data Sharing

38 39 39 40 42 42 43 45 45 45 46 46

3. Data Integration 51 Holger Storf, Dennis Kadioglu, Michael Storck, Iñaki Soto-Rey, Sebastian Hofmann, and Danny Ammon 3.1 Introduction 52 3.2 Interoperability 53 3.2.1 Syntactic and Structural Interoperability 53 3.2.2 Semantic Interoperability 53 3.2.3 Process Interoperability 54 3.3 Extract, Transform, Load Process (ETL Process) 55 3.3.1 Introduction 55 3.3.2 Extract 55 3.3.3 Transform 55 3.3.4 Load 56 3.4 Data Provisioning and Data Storage 56 3.4.1 Data Lake 57 3.4.2 Data Warehouse 57 3.4.3 Data Provisioning “On the Fly” 57 3.5 Data Quality and Data Reliability 58 3.5.1 Definition and Relevance 58 3.5.2 FAIR Principles 59 3.6 Reintegration of Data 60

Contents

4. Data Analysis in Genomic Medicine: Status, Challenges, and Developments 63 Matthias Schlesner 4.1 Introduction 63 4.2 Genome Sequencing in Clinical Applications 64 4.2.1 Read Mapping to the Human Reference Genome 67 4.2.1.1 Toward a complete human reference genome 70 4.2.2 Variant Detection 72 4.2.2.1 Germline variant calling 72 4.2.2.2 Somatic variant calling in cancer 74 4.2.2.3 Toward best practices for cancer somatic variant calling 76 4.2.3 Variant Annotation 77 4.2.3.1 Functional annotation 77 4.2.3.2 Functional consequence prediction 79 4.2.3.3 Population prevalence 81 4.2.3.4 Variant knowledge bases 81 4.2.3.5 Integrated annotation platforms 83 4.2.4 Genomic Variant Information from Primary Literature 84 4.2.5 Identification of Driver Mutations 85 4.2.6 Conclusions 85

Part II: Digital Health and Innovation

5. Ethical Aspects of mHealth Technologies: Challenges and Opportunities 101 Tereza Hendl, Bianca Jansky, Victoria Seeliger, Ayush Shukla, and Verina Wild 5.1 Introduction 102 5.2 Ethical Implication and Challenges 105 5.3 The Ontologies and Epistemologies Shaping mHealth 105

vii

viii

Contents



5.4 Concerns of Accuracy, Safety, and Security 5.5 Support for User Health Decision-Making 5.6 Protection from Physical and Mental Harm 5.7 Increasing Benefit 5.8 Intersectional Benefit and Health Justice 5.9 Conclusion

6. i-Learning: The Next Generation e-Learning

107 108 110 113 115 118 129

Felix Müller-Sarnowski

6.1 6.2



6.3 6.4 6.5 6.6

What Is e-Learning? 130 History and Domains of e-Learning 133 6.2.1 History of Educational Technology: Teaching Machines 134 6.2.2 The History of Programmed Learning 139 6.2.3 History of Computer-Based Learning 140 6.2.4 History of Cybernetic Learning and Personalized Learning 141 6.2.5 History of Multimedia Learning 142 6.2.6 History of Distance Learning 143 The Concepts and Technical Domains of e-Learning 143 The Driving Forces Behind e-Learning 145 e-Learning in Medicine 146 What Comes Next? 147 6.6.1 Lost in Information 148 6.6.2 Medicine Is Increasingly Ruled by “Big Data” 149 6.6.3 Information: Seed of a New Age? 150 6.6.4 Coding: The Next Generation Literacy 150 6.6.5 Should Every Medical Student Learn to Program? 152 6.6.6 i-Learning: The Next Generation of e-Learning 153 6.6.7 What Drives i-Learning? 156 6.6.8 i-Learning and Medical Information Science (MIS) 158

Contents

7. Data-Driven Nursing Research: An Overview of Underlying Concepts and Enablers 171 Steffen Netzband, Volker Hammen, and Frank Kramer 7.1 Status Quo 171 7.2 Nursing Data Collection for Research Purposes 172 7.2.1 Nursing Minimum Data Sets 172 7.2.2 Nursing Sensitive Indicators 174 7.3 Interoperability of Data Exchange Across Different Systems and Facilities 175 7.3.1 Semantic Interoperability 176 7.3.2 Syntactic Interoperability 178 7.3.3 Inter-institutional Data Transfer 179 7.4 Discussion 180

8. Designing the Hospital of the Future: A Framework to Guide Digital Innovation 185 Christina C. Bartenschlager, Steffen Heider, Julian Schiele, Jennifer Kunz, and Jens O. Brunner 8.1 Sharpening the Vision 186 8.2 The Conceptual HoF Framework 188 8.2.1 Patient (P) 189 8.2.2 Staff (S) 190 8.2.3 Treatment and Intervention (T) 191 8.2.4 Logistics and Supply (L) 192 8.2.5 Management and Organization (M) 193 8.2.6 Data and Control (D) 193 8.2.7 Infrastructure (I) 194 8.3 Evaluation of HoF Dimensions 195 8.4 Enablers 196 8.5 The Four-Step Approach 197 9. Combining Digital Medicine, Innovative Geospatial and Environmental Data in Environmental Health Sciences to Create Sustainable Health Franziska Kolek, Claudia Künzer, Dac-Loc Nguyen, and Claudia Traidl-Hoffmann 9.1 Introduction and Setting the Stage

201

202

ix

x

Contents



9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9

The Exposome Environmental Health Services Data for Environmental Health Services Measuring Exposomes by Personal Samples, Individual Exposome The Role of Earth Observation Data in Public Health Research Imitating Natural Exposure Conditions in Human Exposure Chambers Challenges of Digital Medicine in Environmental Health Future Prospects

10. Digitalization and (Nano)Robotics in Nanomedicine

203 204 206 208 210 211 211 213 217

Stefan Lyer, Pascal Blersch, Christian Huber, Rainer Tietze, and Christoph Alexiou

10.1 Introduction 217 10.2 Imaging in Nanomedicine 218 10.3 Simulations in Nanomedicine 224 10.4 Robotics and Nano- and Microbots in Biomedicine 226

Part III: Digital Diagnostics

11. Neural Networks in Molecular Imaging: Theory and Applications

241

Andreas Schindele, Johannes Tran-Gia, and Constantin Lapa

11.1 Introduction 241 11.2 Theory 242 11.2.1 Machine Learning 242 11.2.2 Neural Networks 244 11.3 Applications 248 11.3.1 Alzheimer’s Disease 248 11.3.2 Lung Cancer 249 11.4 Outlook 251

Contents

12. Precision Oncology: Molecular Diversification of Tumor Patients 255 Sebastian Dintner 12.1 Introduction 256 12.2 Genomics 259 12.2.1 Short Nucleotide Variants and Structural Variants 264 12.2.2 Copy Number Alterations 265 12.2.3 Homologous Recombination Repair and Deficiency 265 12.2.4 Tumor Mutational Burden 266 12.2.5 Mutational Signatures 266 12.2.6 Germline Variants 267 12.3 Single-Cell Sequencing 268 12.4 Liquid Biopsy 269 12.5  Data Integration and Molecular Tumor Boards 269 12.5.1 Data Integration 270 12.5.2 Standard Nomenclature 271 12.6 Conclusion 273 13. Digital Applications in Precision Pathology 279 Ralf Huss, Johannes Raffler, Claudia Herbst, Tina Schaller, and Bruno Märkl 13.1 Introduction 279 13.2 Precision Pathology 280 13.2.1 Machine Learning 281 13.2.2 Machine Intelligence 282 13.3  Computational, Algebraic, and Encoded Pathology 283 13.4 Applications of Precision Pathology 285 13.5 Digital Image and Data Analysis 287 14.

Computational Pathology 293 Peter Schüffler and Wilko Weichert 14.1 Introduction 293 14.2 AI in Pathology is a New Field 294 14.3 Impact on Clinical Routine 295

xi

xii

Contents



14.4 Impact on Research 296 14.5 Structured Reports Are Essential for AI Development 298 14.6 Datasets at Scale via Federated Learning 298 14.7 Outlook 299

15. Digital Neuropathology 303 Friederike Liesche-Starnecker, Georg Prokop, and Jürgen Schlegel 15.1 Introduction: The Roots of Neuropathology 303 15.2 Traditional Histology in Future Lights 304 15.3 Advanced Neuro-oncologic Diagnostics 305 15.4 In Situ Microscopy in Real Time 307 15.5 Conclusion 309

16. Application of Artificial Intelligence in Gastrointestinal Endoscopy 313 Alanna Ebigbo, Friederike Prinz, Michael Meinikheim, Markus Scheppach, and Helmut Messmann 16.1 Introduction 314 16.2 Esophagus 315 16.3 Stomach 318 16.4 Small Intestine 321 16.5 Colon 322 16.6 Computer-Aided Detection of Polyps 324 16.7  Computer-Aided Characterization (CADX) of Polyps 324 16.8 Conclusion 326

Part IV: Digital Therapeutics

17. Digital Transformation Processes in Acute Inpatient Care in Germany Andreas Mahler and Kerstin Lamers 17.1 Introduction 17.2 Approaches to Implementation 17.2.1 Top-Down 17.2.2 Bottom-Up

337 338 339 340 340

Contents



18.

17.2.3 The Holistic Approach 341 Direct Care Perspective 342 17.3.1 Digitalization in the Context of Care Relief 343 17.4 Care Management Perspective 346 17.4.1 Sector-Specific Challenges 346 17.4.1.1 Structural difficulties 346 17.4.1.2 Professional difficulties 347 17.4.2 Communication in the Care Team 347 17.4.3 Qualification of Employees 348 17.5 Perspective of Nursing Science 349 17.5.1 The Evaluability of Data: New Ways and Opportunities for Care 349 17.5.2 Nursing Research 350 17.5.2.1 Nursing indicators as a clinical and cross-sectoral management tool 350 17.5.2.2 Imagine the given scenario 351 17.6 Prospects 353 17.3

Digital Medicine in Neurology 357 Antonios Bayas, Monika Christ, and Markus Naumann 18.1 Introduction 357 18.2 Multiple Sclerosis and Parkinson’s Disease 360 18.2.1 Multiple Sclerosis 360 18.2.1.1 Monitoring MS symptoms and disease course 362 18.2.1.2 Digital treatment support 366 18.2.1.3 Prediction of disease activity and treatment decisions 366 18.2.2 Parkinson’s Disease 367 18.2.2.1 Monitoring of symptoms and activities of daily living 368 18.2.2.2 Digital treatment support 370 18.2.2.3 Automated diagnosis, prediction of disease severity, and treatment decisions 371 18.3 Summary and Outlook 374

xiii

xiv

Contents

19.



Neurorehabilitation Medicine 383 Andreas Bender, Thomas Platz, and Andreas Straube 19.1 Introduction 383 19.2 Therapeutic Potential of Digital Medicine in Neurorehabilitation 384 19.2.1 Virtual Reality Rehabilitation 384 19.2.2 Telerehabilitation 389 19.2.3 Robotic Therapy in Neurorehabilitation 391 19.2.4 Humanoid Robot Assistance in Neurorehabilitation 394 19.3 Diagnostic and Prognostic Potential of Digital Medicine in Neurorehabilitation 396 19.4 Digital Medicine as a Long-Term Medical Aid in and after Neurorehabilitation 398 19.5 Conclusion 399

20. Digital Psychiatry Sophie-Kathrin Kirchner, Anna Hirschbeck, and Irina Papazova 20.1 Entering a New World of Mental Health Care 20.2 Diagnosis and Prevention, Prognosis, and Treatment Selection 20.2.1 Diagnosis and Prevention 20.2.2 Prognosis, Treatment Selection, and Outcome Prediction 20.3 Digital Treatment Options for Mental Health 20.3.1 Psychotherapy at a Distance (Text-Based and Video-Based Treatment) 20.3.2 Internet- and Mobile-Based Psychological Interventions 20.3.2.1 Types 20.3.2.2 Evidence 20.3.2.3 Quality assurance 20.4 Limitations and Challenges for the New World of Mental Health 20.4.1 Efficacy vs. Effectiveness 20.4.2 Data Safety and Legal Concerns

407

407 409 412

413 414 415 416 416 418 419

420 420 420

Contents



21.

20.4.3 Ethical Challenges 20.5 Conclusion

Digital Neurosurgery Björn Sommer and Ehab Shiban 21.1 Introduction 21.2 Intraoperative Imaging 21.2.1 Intraoperative Ultrasound 21.2.2 Intraoperative Computed Tomography 21.2.3 Intraoperative Magnetic Resonance Imaging 21.3 Neuronavigation 21.3.1 Intraoperative Navigation: Brain 21.3.2 Intraoperative Navigation: Spine 21.3.3 Hybrid Operating Rooms 21.4 Robotics 21.4.1 Augmented Reality

22. Digital Surgery: The Convergence of Robotics, Artificial Intelligence, and Big Data Philipp Jawny and Michael Beyer 22.1 Introduction 22.2 State-of-the-Art Surgery 22.3  Digital Infrastructure in the Surgical Environment 22.3.1 The Modern Operating Room 22.3.2 Preoperative Planning and Intraoperative Decision-Making 22.3.3 Documentation and Reporting 22.3.4 Surgical Education and Training 22.4 Digital Revolution of Surgery 22.4.1 Surgical Robotics 22.4.1.1 Visualization and cognition 22.4.1.2 Advanced instruments 22.4.1.3 Autonomous robotic systems 22.4.2 Artificial Intelligence 22.4.2.1 Machine learning 22.4.2.2 Natural language processing

422 423 433

433 435 435 436

439 440 440 442 444 444 446 453

453 454

455 455 457 458 459 460 461 462 463 463 464 464 465

xv

xvi

Contents



23.

22.5 22.6

22.4.2.3 Artificial neural networks 22.4.2.4 Computer vision 22.4.3 Big Data Problems and Challenges 22.5.1 Technical Infrastructure and Interoperability 22.5.2 Technical Expertise Future of Surgery 22.6.1 The Need for Interprofessional Teams 22.6.2 The Future Role of Surgeons

Digital Urology Alexander Buchner 23.1 Introduction 23.2 Telemedicine 23.3 Robotics 23.3.1 Robot-Assisted Surgery 23.3.2 Radiosurgery 23.4 Artificial Intelligence 23.5 Conclusion

24. Digitalization in Anesthesiology and Intensive Care Philipp Simon, Ludwig Christian Hinske, and Axel Heller 24.1 Introduction 24.2 The Perioperative Process: Digitalization in Anesthesiology and Critical Care 24.3 Digital Anesthesiology and Intensive Care in Research and Education 24.4 Fair Anesthesia: Data Sharing and Open Science 24.5 Clinical Decision Support 24.6 Perspective and Vision

465 465 466 466 467 467 468 468 469 475

475 475 479 479 480 482 486 493

493

494 495 496 498 500

25. Digital Palliative Care 505 Irmtraud Hainsch-Müller, Christoph Aulmann, and Eckhard Eichner 25.1 Introduction 505

Contents





26.

25.2

Integration of Digital Palliative Care 507 25.2.1 Early Integration/Advance Care Planning 507 25.2.2 Symptom Control in the Context of Palliative Care Treatment 508 25.2.2.1 PROMs, the basis of patient-centered medicine 509 25.2.3 Patient Care/Management/Limits and Opportunities 511 25.2.4 Cross-Sectoral Care 513 25.2.5 Digital Bereavement Counseling/ Settlement of the Digital Estate 514 25.3  Research in the Field of Digital Palliative Care 515 25.4 Education 516 25.5 Prospects 516

Digital Medicine in Pulmonary Medicine 521 Olaf Schmidt 26.1 Introduction 521 26.2  Overview of the Applications Mentioned on the Homepage of the Deutsche Atemwegsliga e.V. 522 26.2.1 Kaia COPD App 522 26.2.2 Atemwege Gemeinsam Gehen App/ Breath Walk Together App 523 26.2.3 OMROM Asthma Diary App 523 26.2.4 Vivatmo App 523 26.2.5 breaszyTrack – dein Asthma-Helfer App/breaszyTrack – Your Asthma Helper App 524 26.2.6 www.copd-aktuell.de is an Online Portal 524 26.2.7 “Kata – Deine Inhalationshilfe für die Anwendung Eines Dosieraerosols” App/“Kata – Your Inhalations Aid for the Use of a Metered Dose Inhaler” App 524 26.2.8 “myAir” App 525

xvii

xviii

Contents









26.2.9 “Nichtraucher Helden” App/ “Non-Smoking Heroes” App 525 26.2.10 “SaniQ Asthma!” App 525 26.2.11 “Therakey” is a COPD Online Portal 525 26.3 Description of the Functioning of the SaniQ App 526 26.4 What Studies Have Already Been Conducted with This Application? 529 26.4.1 Rhineland-Palatinate Breathes Through: Telemedicine for Healthy Lungs 529 26.4.2 Experiences With Digital Care for Patients With Chronic and Acute Lung Diseases During the SARS-CoV-2 Pandemic530 26.4.3 COVID-19@Home: App-Based Telemonitoring in the GP Practice 534 26.5  What Functions Should a Telemedical Application Fulfill in the Future? A Concept for a Digital Supply 536 26.5.1 Description of Telemedicine 537 26.5.2 Communication 537 26.5.3 Medication 537 26.5.4 Monitoring 538 26.5.5 Interfaces 538 26.5.6 Extensions 538 26.5.7 Conclusion 538

27. Digital Rheumatology Johannes Knitza, Martin Krusche, and Philipp Sewerin 27.1 Introduction 27.2 Acceleration of Diagnosis 27.2.1 Status Quo: Shortage of Rheumatologists and Diagnostic Odyssey 27.2.2 Diagnostic Decision Support Systems 27.3 Personalized Disease Monitoring 27.3.1 Video Consultations

543

543 544 544 544 546 546

Contents





27.3.2 Electronic Patient-Reported Outcomes 546 27.3.3 Wearables and Smartphones 547 27.3.4 Self-Sampling 548 27.4 Digital Therapy 548 27.4.1 Patient Education 548 27.4.2 Digital Therapy for Rheumatic Complaints 549 27.4.3 Digital Therapy for Comorbidities 550 27.5 Artificial Intelligence in Rheumatology 550 27.6  Limitations and Potential of Digital Rheumatology 552 27.6.1 Limitations 552 27.6.2 Potential 552

28. Digital Dermatology 563 Julia Welzel, Julia K. Winkler, Holger A. Hänßle, Michael Jünger, Charlotte Kiani, Stephan A. Braun, Linda M. Wittbecker, Regine Gläser, and Alexander Zink 28.1 Introduction 563 28.2 AI-Supported Image Analysis in Dermatology 564 28.3 Teledermatology 567 28.4 Smart Skin and Wearables 568 28.5 Digital Dermatopathology 572 28.6 Digital Teaching 574 28.7 Summary 576 29. Digital Neonatology Jonathan Schaeff, Wilfried Schenk, and Michael C. Frühwald 29.1 Introduction 29.2 Challenges 29.3 Delivery Room Management 29.4 Defining Diseases 29.5 Medication and Nutrition 29.6  Monitoring, Event Detection, and Automated Therapy 29.7 Bacterial Infections

583

583 585 586 588 589 592 595

xix

xx

Contents



29.8 Neurology and Neurodevelopment 597 29.9 Conclusion 597

30. Digital Medicine and Artificial Intelligence in the Area of Breast and Gynecologic Cancer

611

Nina Ditsch and Christian Dannecker

30.1 Introduction 612 30.2 AI in the Special Field of Gynecologic Oncology 612 30.3 AI in the Field of Breast Cancer 613 30.4 AI and Gynecologic Cancers 620 30.5 Cervical Cancer 621 30.6 Ovarian Cancer 623 30.7 Endometrial Carcinoma 626 30.8 Summary and Visions 627

31. Digital Otorhinolaryngology (Ear-Nose-Throat)

637

Stephan Lang, Anke Daser, Benjamin Kansy, Kerstin Stähr, Timon Hussain, Benedikt Höing, Diana Arweiler-Harbeck, and Stefan Mattheis

31.1 Introduction 637 31.2  Surgical Planning Using Virtual Reality Systems 638 31.3  Intraoperative 4K and 3D Visualization, Navigation Systems, and Navigated Instruments 641 31.3.1 Intraoperative 4K and 3D Visualization 641 31.3.2 Navigation Systems and Navigated Instruments 642 31.4 Robotics in Head and Neck Surgery 643 31.4.1 Robotic Systems and Their Application in Head and Neck Surgery 644 31.5 Future Perspectives 646 31.6  Digital Ear Surgery: Intraoperative Live Imaging of Electrocochleography During Cochlear Implant Surgery 647

Contents

32.

Digital Orthopedics and Traumatology 653 Karl Braun and Dominik Pförringer 32.1 Introduction 653 32.2 Processes and Divisions 654 32.2.1 Telemedicine 654 32.2.2 Pre- and Postoperative Processes 655 32.2.3 Intraoperative Digitization 656 32.2.4 Education and Teaching 656 32.2.5 Ethical and Legal Framework 657 32.3 Outlook and Potential Next Steps 657 32.3.1 Information on Two Columns 657 32.3.2 Teaching and Training 658 32.3.3 Budget and Legal Support for Clinics and Outpatients 658

Index

661

xxi

Foreword

Our life has – over many decades – evolved into a very safe and civilized space. Still, we are confronted, ever so often, with risks, fatal accidents, unintended human errors, and seemingly unpredictable illnesses. In many areas, society has reduced these risks with dramatic success through intelligent development and engineering of iterative prevention strategies, particularly in the areas of occupational safety, air, and road transport. We have been relentlessly and constantly turning every stone at every level in order to achieve the vision and goal of zero avoidable deaths. The prominent example, Vision Zero for road traffic, which began more than 30 years ago in Sweden, is based on the premise that making mistakes from time to time is human and natural and has therefore been trying to change the prevailing conditions and systems in such a way that these mistakes do not pose a vital threat to anyone. This is done through official programs at regional, national, and European levels, in many small and large steps, all of which collectively reach enormous proportions. Airplanes hardly ever crash and being at work is statistically safer than staying at home or even in bed. Today, there is a seatbelt obligation for car drivers, it is compulsory to wear a helmet for motorcyclists, there are antilock braking systems, airbags, electronic stability control systems, and much more. When serious injuries or fatal accidents occur at a traffic junction, the situation is always investigated and subsequently translated into counteractions. For example, a roundabout is often built instead, which does not necessarily reduce the number but the seriousness of accidents. When people die frequently because of swerving on the opposite roadway, the lanes are robustly separated and the speed limit is reduced. Why don’t we transfer this rigorousness of action to other fields and, in particular, to medicine? If people die early or otherwise from avoidable widespread diseases such as cardiovascular diseases or cancer, do we set an investigation commission to prevent the reoccurrence of such events? In fact, and unbelievably so, we do not! Instead, the way we talk about cancer nowadays, ironically

xxiv

Foreword

resembles how we talked about accidents during the last century. “Bitter fate,” “presumably familial predisposition,” “he smoked heavily,” “she was overweight,” “was not careful,” “went too late to the doctor,” and other excuses are accentuating our shrugging and the non-action. We seem to agree with a death sentence for minor human imperfection. Or to put it in the words of the famous German singer Herbert Grönemeyer: “Wie eine träge Herde Kühe schauen wir kurz auf und grasen dann gemütlich weiter” (“Like a sluggish herd of cows, we briefly look up and then continue to graze in comfort.”) Well-made digitization can become a vital element in obtaining a Vision Zero in medicine, which will become evident while reading the chapters of this book. Even with our current knowledge, up to half of all widespread diseases could be avoided through prevention and early detection. But as of now, we are not doing what we could and should do! When two people in one family fall ill with the same problem, it is often not even noticed. This has two reasons: Firstly, we already know a lot about many diseases today, but we do not use knowledge of the known unknowns. If we could use digital tools to analyze genetic information or even only anamneses and case histories, then some diseases could be completely avoided, prevention individualized, and therapies improved. Secondly, we still know very little. For example, our knowledge about familial cancer is rather small because this information has not been collected systematically. Digitized medical care can, however, generate a lot of new knowledge about the many unknown unknowns. Medicine urgently needs the much-quoted, learning healthcare system. As part of the national network on genomic medicine in Germany, many of us are currently trying to give a tangible example of what this can look like, using lung cancer as an example. There are many forms of molecular diagnosis in this disease. But we do not know much about reproducibility, and the interpretation of the results is based on the approach of the respective institution. Many colleagues are struggling to always find and realize the right therapy recommendations for their patients, not least because much is changing quickly. What we have in mind for these patients is a digitally based network at the interface between academic centers and wide-ranging care, which evaluates the quality of examinations and data from all patients: How often are specific diagnostics

Foreword

requested? What recommendations are made? Tools to analyze this kind of data are available. However, in Germany, data processing tools from the last century are used or not used at all. The scenario described above for lung cancer would be a huge advancement not only for the research community but also for all patients. If we would systematically collect, although with little delay, the measures that are taken in various constellations and the results achieved in each case, we could make this information available at the point of care, making it highly relevant for patient care. However, the benefits of digitalization start much earlier than all relevant information and findings are made available in a consolidated form in one place. This is the goal of the electronic healthcare record, the EHR, called ePA in Germany. In addition, navigation applications would provide information on therapy and clinical studies for patients, or their families, as well as individual context-related information that differs with illness, condition, place of residence, and so on. Using digital health tools, patients can also get a voice themselves to deliver results and receive answers to their questions. In the right way, digitalization can democratize access to and use of healthcare. Not to mention that it probably also would be much more efficient. Some may ask “... and data protection?” Well, what about data protection, really? The pandemic has taught us that this topic has been kept in a stranglehold of intransparent pseudo-security in Germany for too long with an “only dead data are good data” mindset. Health data must be particularly safe, no doubt. But data protection in a strict sense means patient protection. Medical data is supposed to help and protect patients, and data protection must have the same goals, without unreasonably compromising data security. So far, we have been looking at it from one side only – we have been trying to protect data optimally, creating barriers to data access and irrational constructions such as data minimization. But in case of doubt, we are also protecting the data from the patients themselves, thus leaving them unprotected from their disease. In the context of health data, patient welfare must be the utmost and mandatory guiding principle for data protection. Data protection must also be liable for the availability of data. As long as good data protection is considered to be equivalent to the least possible processing of data, we will not make any progress. The

xxv

xxvi

Foreword

processing and analysis of medical data can be life-prolonging, even life-saving. This, in turn, makes an understanding of data protection, which focuses on data use rather than patient use, a danger for patients. Data analysis must not only take place in the ivory towers of a few health service researchers with massive regulation and a time lag. It must take place wherever it is needed, where it helps to protect patients. We know from workshops, for instance, that cancer patients demand this. We have had this discussion time and again, and we were surprised at the beginning but it has become quite clear: The patients want their data to be used. There is a great awareness that the solidarity-based health care system in Germany enables expensive therapies and correspondingly a great willingness to give something back exists. I would even go so far as to state that it is ethically questionable not to allow such analyses of your data. About the initial considerations concerning the Vision Zero concept, if we have a traffic light at dangerous crossroads, will we turn it off and accept traffic fatalities? But this is exactly what we are doing in the healthcare sector and it must change. This book allows the readers to know how this works. It gives a comprehensive overview of the necessary technical and organizational requirements, including impressive examples for specific implementations in prevention, diagnosis, and therapy, that shows how digital medicine can help to prevent avoidable deaths. Each chapter of this book remarkably reflects the progress made in the digitization of medicine, collectively forming the basis for Vision Zero in medicine. Prof. Dr med. Christof von Kalle Charité-BIH Clinical Study Center, Berlin, Germany

Acknowledgments

As with any book of such magnitude, complexity, and novelty, it simply cannot be done alone. Therefore, I am very grateful to all authors who contributed chapters and provided their expert insight and deep understanding of their subjects. Some chapters required more joint efforts than others, given the unequal level of digital maturity and adoption of digitization in the past. But everyone, alone or as a team of co-authors, did their utmost to present the statethe-art digital solution in their speciality of medicine or biomedical research as of the date of publication. The writing and editing of a textbook like this has been a learning experience for me as well as for my many esteemed colleagues. I would like to thank everyone who joined me along the way and supported this project. Special thanks goes to Mrs Claudia Grass, who reviewed and edited each chapter, and to Dr Johannes Raffler, who guided me through some of the technical and digital pitfalls. I would also like to thank Jenny Stanford Publishing, Singapore, for encouraging me to start a series on “next-generation medicines,” after our textbook Tissue Phenomics was well received by the audience, which was published a few years back. Ralf Huss

Introduction to Digital Medicine: Bringing Digital Solutions to Medical Practice Ralf Huss Institute for Digital Medicine, University Hospital Augsburg, Augsburg, Germany Institute of Pathology and Molecular Diagnostics, Medical Faculty, University of Augsburg, Augsburg, Germany

Definition and Principles of Digital Medicine There are different definitions and, in turn, principles of the term “digital medicine.” Currently “digital medicine” is considered a field within digital health that focuses on the combination of medicines (digital therapeutics) with sensor technologies or digital diagnostics. For this purpose, “digital medicine” uses specialized and regulated hardware along with softwarebased solutions including medicinal products (if available) that rely on digital solutions like algorithms to measure, intervene, sustain, and/or improve human health. Therefore, some authors prefer the broader term “digital health” which is concerned with the use of digital technologies independently or “in concert” with (bio)pharmaceuticals, biologics, devices, or other (nano-sized) products to optimize patient care and disease outcome [2]. “Digital medicine” is intended to digitally support and assist the practice of medicine, including risk prediction, disease prevention, recovery, and health promotion for individuals and across entire populations using large data sets and machine-based intelligence [5]. Digital medicine “empowers” all stakeholders, i.e., patients, healthcare providers, and professionals to make the best medical decisions through the most effective use of “big” data and data-driven decision-support tools. The availability and accessibility to data will change best practices not only in interventional or pharmaceutical treatments but also in caregiving and nursing and will further improve proper patient management through education and change of behavior in an aging society. “Digital medicine”

xxx

Introduction to Digital Medicine: Bringing Digital Solutions to Medical Practice

enables conventional medicine to move continuously from sick care to more preventive measures and to lower the risk of disease aggravation. The affordable access to appropriate hard- and software-driven technologies has made it possible to make use of health data and information. Data from individual patients or entire patient cohorts from clinical trials will produce and optimize digital diagnostics and therapeutics. “Digital medicine” will change how we will practice medicine through many different digital measures including transparent (telemedicine) communication and educational (webbased) tools. More and more healthcare professionals adopt digital technologies engaging in new cooperative networks. The same stakeholders who now practice “conventional medicine” will practice “digital medicine” side-by-side with data scientists and IT engineers in a multidisciplinary team [1]. “Digital medicine” connects expert knowledge and medical experience with the use of digital tools generating novel evidence and disruptive insights to further improve patient care. “Digital medicine” needs to be incentivized also financially to ensure the responsible and sustainable adoption of digitization in all areas of healthcare. This requires determining the value and costs of “digital medicine.” All stakeholders, i.e., patients, legislators, payers, regulatory and political authorities as well as the healthcare industry and IT providers need to be involved to find an optimum balance between data ethics, altered workflows, rising costs, and the value proposition for society [4]. The development and deployment of digital solutions are certainly costly and require initial investments to improve the access to the healthcare space and eventually reduce cost. A sustainable reimbursement of digital medicine will ensure the continuous quality of patient care, healthcare equity, and mitigation of potential bias, and thereby contribute to realizing and monetizing the potential of digitization in medicine. The different parts and chapters of the book are written by experts in their fields. They provide examples and guidance on the use and application of digital solutions in clinical research and medical practice. While some disciplines are further advanced, others are following fast while some are lagging [3].

Introduction to Digital Medicine: Bringing Digital Solutions to Medical Practice

Radiology has always been an innovator in the field of digitization and digital practice and is not included in this book. We refer to the vast body of special literature on radiology and medical imaging. The book starts with an introduction to artificial intelligence and machine learning for physicians to introduce the value of (big) medical data. The availability of data lakes demands a structured approach to access, use, and integrate relevant data before being utilized in clinical decision support. Even outside the immediate medical practice, digital solutions are essential to improve the training and education of all stakeholders in healthcare. They are also used in hospital management, in understanding the influence of environmental factors like climate change, or in developing disruptive diagnostic and therapeutic tools through nanotechnology and robotics. The latter includes the development of wearables and sophisticated sensor technologies to retrieve data in real-time and “on-the-fly.” The last two parts of the book provide examples on the current use of digital solutions in many clinical disciplines and subjects like pathology, surgery, neurology, and even psychiatry. Digital medicine is a game changer in the practice of medicine in many different aspects and also in establishing patient care as an interdisciplinary effort to manage the financial and resource challenges of the future. Caregivers who will not participate in digital medicine will fall behind and eventually lose the trust of patients and society.

References

1. Berisha, V., Krantsevich, C., Hahn, P. R., Hahn, S., Dasarathy, G., Turaga, P., Liss, J. (2021). Digital medicine and the curse of dimensionality. NPJ Digit Med.; 4(1): 153. doi: 10.1038/s41746-021-00521-5. PMID: 34711924; PMCID: PMC8553745. 2. Dang, A., Arora, D., Rane, P. (2020). Role of digital therapeutics and the changing future of healthcare. J Family Med Prim Care; 9(5): 2207– 2213. doi: 10.4103/jfmpc.jfmpc_105_20. PMID: 32754475; PMCID: PMC7380804. 3. Lim, J. S., Goh, H. L., Au Yong, T. P. T., Boon, C. S., Boon, I. S. (2021). Patient-centred digital medicine. Clin Oncol (R Coll Radiol). 2022 Jan; 34(1): e80. doi: 10.1016/j.clon.2021.10.011. Epub 2021 Oct 31. PMID: 34732294.

xxxi

xxxii

Introduction to Digital Medicine: Bringing Digital Solutions to Medical Practice

4. Odone, A., Buttigieg, S., Ricciardi, W., Azzopardi-Muscat, N., Staines, A. (2021). Public health digitalization in Europe. Eur J Public Health. 2019 Oct 1; 29(Supplement_3): 28–35. doi: 10.1093/eurpub/ ckz161. Erratum in: Eur J Public Health. 2021 Dec 1; 31(6): e1. PMID: 31738441; PMCID: PMC6859512. 5. Steinhubl, S. R., Topol, E. J. (2018). Digital medicine, on its way to being just plain medicine. NPJ Digit Med.; 1: 20175. doi: 10.1038/s41746017-0005-1. PMID: 31304349; PMCID: PMC6550249.

Part I

Digital Science

Chapter 1

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

Julia E. Vogt, Ece Ozkan, and Riĉards Marcinkeviĉs Department of Computer Science, ETH Zürich, Switzerland [email protected]

Machine learning (ML) is a discipline emerging from computer science [21] with close ties to statistics and applied mathematics. Its fundamental goal is the design of computer programs [103], or algorithms, that learn to perform a certain task in an automated manner. Without explicit rules or knowledge, ML algorithms observe and possibly, interact with the surrounding world by the use of available data. Typically, as a result of learning, algorithms distill observations of complex phenomena into a general model which summarizes the patterns, or regularities, discovered from the data. Modern ML algorithms regularly break records achieving impressive performance at a wide range of tasks, e.g., game playing

Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

4

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

[147], protein structure prediction [74], searching for particles in high-energy physics [13], and forecasting precipitation [127]. The utility of machine learning methods for healthcare is apparent: it is often argued that given vast amounts of heterogeneous data, our understanding of diseases, patient management, and outcomes can be enriched with the insights from machine learning [40, 55, 98, 109, 123, 128, 146]. In this chapter, we will provide a nontechnical introduction to the ML discipline aimed at a general audience with an affinity for biomedical applications. We will familiarize the reader with the common types of algorithms and typical tasks these algorithms can solve (Section 1.1) and illustrate these basic concepts through concrete examples of current machine learning applications in healthcare (Section 1.2). We will conclude with a discussion of the open challenges, limitations, and potential impact of machinelearning-powered medicine (Sections 1.3 and 1.4).

1.1 Machine Learning: An Overview

Why Now? Computer-based systems have become an integral part of modern hospitals [144] in numerous routine activities, including medical record keeping [72], imaging [44], and patient monitoring [154]. Computer-aided diagnosis and treatment planning have been contemplated ever since the early days of computing. An iconic example is MYCIN – an artificial intelligence-based expert system developed in 1972 for diagnosing blood infections [37, 130]. In contrast to the purely data-driven machine learning perspective, MYCIN relied on about 450 hardcoded rules [130] and logical reasoning to deduce a patient’s diagnosis. The machine learning approach is strikingly different: harnessing large and complex datasets, inducing rules or other types of patterns automatically, and alleviating the need for tedious and costly knowledge engineering. With recent technological advances, we can now collect, store, and share health data at previously unprecedented scales [145], for instance, as of December 2021, The Cancer Imaging Archive [35] contains almost 250,000 computerized tomography images [152]. Such scales alone warrant the need for automated and data-driven methods in healthcare and clinical decision-making.

Machine Learning: An Overview

1.1.1 Navigating through Concepts In the media sphere and even academic literature, terms such as artificial intelligence, machine learning, deep learning, and statistics are sometimes used interchangeably without drawing explicit borders [61, 78]. Below we provide a remark on the differences between these fields, which might be useful to a non-expert reader. Artificial intelligence (AI) [99, 130] tackles the most general problem of building intelligent machines and is not restricted to a particular set of methods: both a simple expert system with a few hand-engineered rules and a logical inference engine [37] or a deep artificial neural network playing the board game of Go [147] could be seen as instances of AI. On the other hand, ML is a subfield of AI [130] focusing on the machines that learn from experience, namely, from the given data. Deep learning (DL) [56, 139] refers to an even smaller subset of machine learning methods: it studies deep neural networks (DNNs), a family of ML techniques that learn many layers of complex, highly nonlinear concepts directly from the raw data, e.g., images, sound recordings, text, and videos. A well-informed reader might have noticed before that many ML methods rely on statistical reasoning and in some cases, ML and statistical models can be used for similar purposes [27]. A simplified, stereotypical delineation between the two fields, sufficient for the current chapter, is that classical statistical models are probabilistic, focus on inference, and make strict structural assumptions; whereas ML methods offer an algorithmic solution to the prediction problem allowing for very general and complex relationships [27, 61].

1.1.2 Machine Learning Approaches and Tasks

The three most common categories of machine learning methods are (i) supervised learning, (ii) unsupervised learning, and (iii) reinforcement learning [21, 105] (Fig. 1.1). These approaches have been tailored toward principally different problem types, which will be outlined below. Regardless of the method, ultimately, a model that generalizes well is desired, i.e., a model with good performance at the considered task across as many different unobserved settings as possible.

5

6

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge



(a) Supervised learning



(b) Unsupervised learning



(c) Reinforcement learning

Figure 1.1 Types of machine learning approaches: (a) supervised, (b) unsupervised, and (c) reinforcement learning. (a) In supervised learning, an ML model extracts predictive patterns from labeled data. (b) In unsupervised learning, a model discovers structure from unlabeled data. (c) In reinforcement learning, an agent interacts with the surrounding environment by sequentially performing actions to maximize the received reward.

Machine Learning: An Overview

In supervised learning [21, 105], the goal is to learn a predictive relationship between a set of input variables, also called features, attributes, or covariates, and an output variable, also called a response, target, or label. Following the example from Fig. 1.1(a), let us assume that we want to design a computer system to recognize objects, such as apples, bananas, or dogs from hand-drawn images. For this task, our features could be given by digital images stored on a personal computer, and labels could be verbal descriptions of the depicted objects, such as “It’s a dog,” written down in a text file. Typically, we would use some learning algorithm to extract predictive patterns from the observed training data. The algorithm serves as a ‘recipe’ for distilling the raw data into a sufficiently abstract and general model. We would then apply the trained model to unseen test data to predict unobserved labels, and evaluate its performance, e.g., in terms of accuracy. The setting where labels come from a finite number of unordered categories, as in the hand-drawn image recognition example, is known as the classification task; whereas in the regression task, labels are usually real-valued. Note, that more generally, we could even learn relationships between input variables and multiple differently valued outputs. The unsupervised learning [21, 70, 105] approach is not as well-defined as supervised learning and attempts to solve a more challenging open-ended problem: given only the input variables without labels, unsupervised learning algorithms typically seek to discover some ‘interesting’ structure in the data. For instance, we might want to stratify our dataset into groups of similar observations – this problem is known as clustering. Following the hand-drawn image example above, a clustering algorithm would group images of similar objects together, e.g., into groups of fruit and animals (Fig. 1.1(b)). Dimensionality reduction is another typical unsupervised task where usually for visualization purposes, we reduce the data to two or three informative dimensions by combining the input features. Although it might seem that supervised and unsupervised learning solve completely disjoint sets of problems, there is a whole spectrum of practical settings in-between [155, 178], e.g., when the output variable is partially missing because it is too expensive to measure for all of the subjects or when the output does not correspond to the exact target, which is impossible or unethical

7

8

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

to measure. Techniques that deal with such settings fall under the categories of semi-supervised [155] and weakly supervised [178] learning. In their simplest forms, both supervised and unsupervised learning usually assume that the model is trained on a set of examples collected prior to learning and that model’s predictions or decisions do not affect each other. Reinforcement learning (RL) [21, 105, 150] ventures beyond these assumptions: the learning algorithm, in this context referred to as an agent, interacts with the surrounding environment by observing it and performing sequences of actions in order to maximize the occasionally obtained reward. For example, a baby learning to walk across a room and being praised by its parents or a robotic arm learning to place objects into a container [Fig. 1.1(c)] could be thought of as reinforcement learning scenarios. An important feature of this setting is the interactive and sequential nature of the learning process. While the three approaches discussed above have become active research areas in their own right, in practice, real-world applications often require a combination of different methods and solving multiple tasks at a time. In the following, we will focus on biomedical and healthcare applications of ML supplementing our discussion with concrete examples of the current research in this area.

1.2 Machine Learning for Healthcare

In healthcare, ML methods are usually leveraged to extract patterns that correlate with medical conditions. They are applied to healthcare records and other patient data to provide clinicians with decision support by predicting, e.g., diagnosis, management, and outcome in an automated manner. There is a plethora of data types in healthcare to which ML algorithms can be applied, including



∑ clinical data from electronic health records (EHR) which contain demographics, laboratory test results, medication, allergies, vital signs, clinical reports [73, 101], ∑ imaging data from different modalities such as X-ray, retinal, mammography, dermoscopy, MR (magnetic resonance), CT

Machine Learning for Healthcare



(computerized tomography), and ultrasound (US) images and echocardiograms [49, 59, 66], ∑ sensors or mobile data from wearable devices or sensors recorded and stored as time series [91, 136], ∑ omics data which are complex and high-dimensional genomic, epigenomic, transcriptomic, metabolomic, exposomic, and proteomic measurements [5, 177].

An advantage of ML methods over conventional statistical modeling is their flexibility and scalability in exploiting diverse and complex data types. Table 1.1 summarizes a few selected application examples in terms of their domain, type of data, ML approach, and task. Below we compile further recent works using various methods and types of data mentioned above.

1.2.1 Supervised Learning

Arguably, the most common ML approach in healthcare is supervised learning. Linear [159] and logistic [148] regression models and decision trees [119] are simple, reliable, and effective supervised learning algorithms used on a variety of datasets to this day. Some examples include the classification of sagittal gait patterns in neurology [172], prediction of diabetes severity [77], breast cancer survival prediction [9, 79], diagnosis of acute appendicitis [181] using clinical data, activity classification [117] and fall detection in elderly people [166] from wearable sensors, and prediction of interactions between target genes and drugs [165] using omics data. However, simpler models often under-perform on data featuring nonlinear effects and interactions, and therefore, more flexible methods may be required, such as random forests [26] or gradient boosting machines [50]. Some of the applications of random forests are the prediction of the risk of developing hypertension [46], the severity and outcome of COVID-19 [68], breast cancer recurrence [149], identification of various diseases, such as hepatitis, Parkinson’s [4, 91] or type 2 diabetes mellitus [176] using clinical and wearable sensors data. Furthermore, applications based on omics data also exist, e.g., for improving hazard characterization in microbial risk assessment [108] or identifying gene signatures for the diagnosis of tuberculosis [22].

9

10

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

In medical images, the most commonly used models are based on DNNs, applied to a number of different modalities including













∑ X-ray images for predicting pneumonia [120], tuberculosis [125] or other pathologies [66], multi-organ segmentation [86], image registration [100], pathology localization [7, 112, 122]; ∑ retinal images for predicting diabetic retinopathy grading [59, 137], age-related macular degeneration [167] or retinopathy of prematurity [163], simultaneous segmentation of retinal anatomical structures, such a retinal vessel and optic disk [97], or pathologies, such as exudates, hemorrhages, microaneurysms, or retinal neovascularization [11, 82]; ∑ heart echocardiograms for cardiac image segmentation [31, 164, 168], identification of local cardiac structures and estimation of cardiac function [54, 110, 170], view classification [183], and the reduction of user variability in data acquisition [1]; ∑ mammography images to improve breast cancer classification [93, 143], lesion localization [43, 48], and risk assessment [84, 104]; ∑ dermoscopy images for classifying skin cancer [36, 49, 90], skin lesion segmentation [3], or detection and localization of cutaneous vasculature [80]; ∑ MR images for brain tumor grading assessment [116], classification of prostate cancer [138], age estimation [64] or Alzheimer’s disease detection [67] from brain images, cardiac multi-structures segmentation [20], or left ventricle segmentation in cardiac MRI [8]; ∑ CT images for liver tumor assessment [38], classification of brain images for diagnosis of Alzheimer’s disease [51], hemorrhage detection [33, 57], COVID-19 pneumonia classification [6], abdominal [160] or urinary bladder [30] segmentation.

There is accumulating evidence that DNNs have already led to improved accuracy for computer-aided applications in various medical imaging modalities. Moreover, interest and advances in deep learning are still developing rapidly in the ML for the healthcare

Machine Learning for Healthcare

community, bringing the performance of ML methods to a level that is acceptable to clinicians.

1.2.2 Semi-supervised Learning

The cost of acquiring raw medical images is often negligible in comparison to expert annotations, labels, or scores. Thus, instead of using expensive supervised learning, many authors leverage semi-supervised learning (SSL) approaches [155], which rely on unsupervised learning techniques to work with a mixture of labeled and unlabeled data. For medical images, the scarcity of labeled data is often a limitation for both segmentation and classification tasks. To this end, SSL has been leveraged for X-ray [24], MR [12, 85, 107] and CT [94] image, MS lesion [17] and gland [173] segmentation. For the classification task, SSL methods have been applied for cardiac abnormality classification in chest X-rays [95], skin lesion diagnosis [89], or gastric disease identification from endoscopic images [142]. Pulmonary nodule detection in CT scans [156] using SSL is also possible.

1.2.3 Unsupervised Learning

Unsupervised learning methods further loosen the burden of manual annotation by exploiting unlabelled data without any supervision. It might be helpful to find interesting patterns or structures in the data, which can be, e.g., used for anomaly detection. Various problems in the healthcare setting can be solved using unsupervised learning: disease clustering and patient subgrouping using EHR [158], fMRI time series [174] data or genomic makeup [92], genomic segmentation [63], dimensionality reduction prior to anomaly detection [180] or classification of mammography images [151]. Unsupervised learning has been applied to various imaging modalities, e.g., X-ray image retrieval [2], blood vessel segmentation in retinal images [169], breast density segmentation and scoring in mammography [76], probabilistic atlas-based segmentation [41] and denoising of contrast-enhanced sequences in MR images [19], and lung nodule characterization in CT images [65].

11

12

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

1.2.4 Reinforcement Learning In RL, an agent learns from new experiences through a continuous trial-and-error process [96]. This approach has been used in different medical scenarios, e.g., for optimal and individualized treatment strategies for HIV-infected patients [47], for cancer trials [175], for sepsis patients [134], forecasting the risk of diabetes [179], segmenting transrectal US images to estimate location and volume of the prostate [132], surgical decision-making [81], predicting CpG islands in the human genome [34], and increasing the accuracy of biological sequence annotation [126]. The rapid progress in ML triggered considerable interest and created numerous opportunities for data-driven applications in healthcare, which in the future might lead to advancements in clinical practice like (semi) automated and more accurate diagnosis or the development of novel and more personalized treatment strategies. Despite these reasons for cautious optimism, plenty of underexplored opportunities, unaddressed limitations and concerns exist. In the following, we elaborate on some open problems.

1.3 Limitations, Challenges, and Opportunities

While it is undoubted that ML can be instrumental in developing data-driven decision support systems for healthcare, the use of machine learning and promises associated with it have come under scrutiny [32, 109, 162]. Machine learning is not a panacean magic crystal ball: it relies on large amounts of high-quality, representative, and unbiased data [109] and is not a substitute for rigorous study and data system design or causal inference [162]. Prior to applying ML methods to health data, we should be critical and ask ourselves whether there actually exists use cases for our ML model [32], whether we possess a sufficiently large dataset representative of the population of interest, whether the features we will use are commonly available in practice, whether we would be able to validate the model externally, whether our model will be accepted by its target users [52, 87].

Table 1.1

A few selected healthcare application examples summarized in terms of the application domain, type of data, ML approach, and task

Application Domain

Data Type

ML Approach

ML Task

Description

Ref.

Ophthalmology

Imaging

Supervised

Classification

Predicting diabetic retinopathy grading using retinal fundus photographs

[59]

Cardiology

Imaging

Supervised

Segmentation

Segmentation of fetal left ventricle in echocardiographic sequences

[168]

Dermatology

Infectiology Oncology

Endocrinology

Clinical Omics

Imaging Clinical

Supervised

Supervised

Unsupervised

Unsupervised

Unsupervised

Urology

Imaging

RL

Oncology

Clinical

RL

Clinical

RL

Classification

Segmentation Denoising

Dim. reduction, anomaly detection Segmentation Regression

Classification

Classifying skin lesions comprising 2,032 different diseases Predicting the severity and outcome of COVID-19 Semi-automated genomic annotation

Spatio-temporal denoising of contrastenhanced MRI sequences

Anomaly detection for several public benchmark datasets including the thyroid dataset

[49]

[68]

[63]

[19]

[180]

Segmenting transrectal US images to estimate the location and volume of the prostate

[132]

Personalized patient-centered decisionmaking

[81]

Individualized treatment strategies for advanced metastatic stage lung cancer

[175]

13

ICU

Classification

Limitations, Challenges, and Opportunities

Genetics

Imaging

14

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

Some of the recent lines of work in ML attempt to alleviate and address these and similar concerns. Below we discuss a few open challenges and areas of active research which could be of interest to both the ML methodologists and practitioners working in healthcare.

1.3.1 Lack of Data, Labels, and Annotations

Data scarcity is a very important problem since data lie at the heart of any ML project. For most applications, in addition to data collection, annotation is an expensive task. Furthermore, medical data are personal, and accessing and labeling them has its challenges. Moreover, in the case of rare diseases, class imbalance makes the use of ML algorithms more challenging. To overcome these problems, different terms for learning are proposed that do not entirely depend on manually labeled data: self-supervised learning, where the model leverages the underlying structure of the data [83], semisupervised learning using a small amount of labeled combined with a large amount of unlabeled data during training [155], weakly supervised learning using the supervision of noisy labels [178], and few-shot learning for generalizing from a small amount of labeled data without using any unlabeled data [157].

1.3.2 Learning across Domains, Tasks, and Modalities

ML methods generally assume that the training and test sets feature similar patterns and relationships, which usually do not hold for healthcare applications. Domain adaptation is a promising solution and gets increasing attention in recent years [58]. Furthermore, trained models usually suffer from overspecializing in individual tasks which tend to not generalize either [42]. As a response, multitask learning was proposed, which is inspired by human learning and proposes to use of shared concepts to extract the common ideas among a collection of related tasks [28, 39]. Moreover, clinicians typically make use of multiple data modalities in their decision-making, including imaging, time series data such as ECG signals, clinical data such as lab results, and non-structured data such as clinical notes. Combining data from different modalities lies at the center of multimodal learning [14, 15].

Limitations, Challenges, and Opportunities

1.3.3 Data Sharing, Privacy, and Security Despite the spectacular advances in the ML domain, there is a concern regarding privacy and security in the proposed data-driven methods. This is normally quite a challenging issue in the healthcare setting due to the fact that ML models need to work with personal information [118]. The need for patient privacy while implementing ML methods using large datasets triggers the urge for automated models which respect privacy and security. The issue has recently been picked up by official authorities, e.g., in the EU the General Data Protection Regulation (GDPR) was implemented [10, 25]. This decision accelerated the field of secure and privacy-preserving ML research to bridge the gap between data protection and utilization for clinical routine [75, 171].

1.3.4 Interpretable and Explainable Machine Learning

Machine learning models are increasingly incorporated into highstakes decision-making [87, 129], including healthcare [60]. With methodological advances and empirical success of deep learning, ML models have become even more performant, yet larger and more complex. As a response, there is a surge of interest in designing ML systems that are transparent and trustworthy. Interpretable [45, 88, 129] and explainable [133] machine learning typically refers to models that are either directly comprehensible or can explain and present their decisions post hoc in terms understandable to the target user. Many interesting research questions originate from this line of work, pertinent to healthcare as well. For instance, does ML even need to be interpretable or explainable, what is a ‘good’ interpretation/explanation in the considered application, and how can we make the model’s interpretations/explanations insightful and/or actionable? Fair Machine Learning The fact that ML methods are data-driven does not necessarily make their decisions fair, ethical, or moral [16]. Datasets and ML models are products of the larger sociotechnical system we live in and inevitably reflect the state of society with all of its disparities [182]. Fairness [16, 124] has become an important principle for machine-learning-based algorithmic decision-making

15

16

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

and has given rise to many technical challenges, such as mitigating demographic disparities captured by ML models [18], documenting and reporting ML models in a transparent manner [102], and documenting datasets, their contents, characteristics, and intended uses [53]. Many of these considerations are equally applicable to ML for healthcare [124] where personalization based on sensitive information could sometimes lead to undesirable disparate outcomes [16].

1.3.5 Toward Causally Informed Models

Many clinical and scientific research questions inherently require causal reasoning [113], e.g., treatment effect estimation or counterfactual outcome prediction. The ‘big data’ alone are meaningless unless researchers are equipped with adequate tools that actually can address their questions [121]. Causality [114] is a field of study which formalizes our reasoning about cause-andeffect relationships. In recent years, it has been argued that many challenging open ML problems are closely related to causality and the inability of the naïve purely data-driven models to reason causally [115, 140, 141]. This realization is an important step toward data collection and ML model development that is informed by the causal perspective. Naturally, causal considerations matter for healthcare applications of ML as well: causally informed ML models tend to be more stable and avoid the pitfalls of a purely predictive approach. For example, when analyzing medical imaging data [29] with highly heterogeneous anatomical and acquisition conditions, causality could help mitigate irrelevant correlations picked up by a predictive model due to confounding features in the training data.

1.4 Conclusion: Quo Vadis?

In this chapter, we have introduced the basics of machine learning, and the types of tasks it can tackle, explained its relationships with other well-established fields, and illustrated its utility in healthcare with a variety of applied research. We have seen that opportunities for application are numerous: ML offers an algorithmic solution to the automated analysis of complex and unwieldy datasets.

References

It is beyond doubt that ML and AI hold promises of automating routine clinical tasks [71], reducing costs [106], and improving healthcare access and quality [135], especially, in developing economies. Some researchers have even optimistically proclaimed that deep learning will replace human specialists altogether, e.g., in radiology [62, 66, 131]. Not surprisingly, their bold predictions have not come to pass yet, and a widespread, clinically meaningful adoption of ML ‘in the wild’ still stands as an ambitious task for both the research and industry. The next decades will show if and how these promises are delivered. Likely, it is by supporting, complementing, and relieving healthcare professionals of tedious routines (and not by replacing them) that ML and AI will ‘make the difference.’ To achieve this kind of harmony, we need an interdisciplinary collaboration among healthcare specialists, ML practitioners, and methodologists [87]. To facilitate a dialogue on equal terms, we should improve the overall digital, machine learning, and statistical literacy among medical students [23, 69] and disseminate subject matter knowledge among machine learning practitioners [153]. Lastly, to implement this ambitious project in practice, we have to develop regulatory frameworks and economic incentives [123], establish large-scale data infrastructures [111], and collect high-quality representative datasets [161]. While all of this requires a considerable initial investment, machine-learning-powered medicine is a worthy cause.

References

1. Abdi, A. H., Luong, C., Tsang, T., Allan, G., Nouranian, S., Jue, J., Hawley, D., Fleming, S., et al. (2017). Automatic quality assessment of echocardiograms using convolutional neural networks: Feasibility on the apical four-chamber view, IEEE Transactions on Medical Imaging 36, 6, pp. 1221–1230, doi: 10.1109/TMI.2017.2690836. 2. Ahn, E., Kumar, A., Fulham, M., Feng, D. and Kim, J. (2019). Convolutional sparse kernel network for unsupervised medical image analysis, Medical Image Analysis 56, pp. 140–151, doi: 10.1016/j. media.2019.06.005. 3. Al-masni, M. A., Al-antari, M. A., Choi, M.-T., Han, S.-M. and Kim, T.-S. (2018). Skin lesion segmentation in dermoscopy images via deep full

17

18

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

resolution convolutional networks, Computer Methods and Programs in Biomedicine 162, pp. 221–231, doi: 10.1016/j. cmpb.2018.05.027.

4. Alam, M. Z., Rahman, M. S. and Rahman, M. S. (2019). A random forest based predictor for medical data classification using feature ranking, Informatics in Medicine Unlocked 15, p. 100180, doi: 10.1016/j. imu.2019.100180.

5. Alipanahi, B., Delong, A., Weirauch, M. T. and Frey, B. J. (2015). Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning, Nature Biotechnology 33, 8, pp. 831–838, doi: 10.1038/ nbt.3300, URL https://doi.org/10.1038/nbt. 3300. 6. Amyar, A., Modzelewski, R., Li, H. and Ruan, S. (2020). Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: Classification and segmentation, Computers in Biology and Medicine 126, p. 104037, doi: 10.1016/j.compbiomed.2020. 104037. 7. Arun, N. T., Gaw, N., Singh, P., Chang, K., Hoebel, K. V., Patel, J., Gidwani, M., et al. (2020). Assessing the validity of saliency maps for abnormality localization in medical imaging. URL https://arxiv.org/ abs/2006.00063.

8. Avendi, M., Kheradvar, A. and Jafarkhani, H. (2016). A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI, Medical Image Analysis 30, pp. 108–119. 9. Azar, A. and El-Metwally, S. (2013). Decision tree classifiers for automated medical diagnosis. Neural Computing and Applications, pp. 2387–2403, doi: 10.1007/s00521-012-1196-7. 10. Goodman, B. and Flaxman, S. (2016). European union regulations on algorithmic decision making and a “right to explanation”, ArXiv: 160608813.

11. Badar, M., Shahzad, M. and Fraz, M. M. (2018). Simultaneous segmentation of multiple retinal pathologies using fully convolutional deep neural network, in Medical Image Understanding and Analysis, pp. 313–324.

12. Bai, W., Oktay, O., Sinclair, M., Suzuki, H., Rajchl, M., Tarroni, G., Glocker, B., et al. (2017). Semi-supervised learning for network-based cardiac MR image segmentation, in Medical Image Computing and ComputerAssisted Intervention – MICCAI 2017, pp. 253–260.

13. Baldi, P., Sadowski, P. and Whiteson, D. (2014). Searching for exotic particles in high-energy physics with deep learning, Nature Communications 5, p. 4308, doi: 10.1038/ncomms5308.

References

14. Baltrušaitis, T., Ahuja, C. and Morency, L.-P. (2018). Challenges and applications in multimodal machine learning, in The Handbook of Multimodal Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 2 (Association for Computing Machinery), pp. 17–48, doi: 10.1145/ 3107990.3107993.

15. Baltrušaitis, T., Ahuja, C. and Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy, IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 2, pp. 423–443, doi: 10.1109/tpami.2018.2798607. 16. Barocas, S., Hardt, M. and Narayanan, A. (2019). Fairness and Machine Learning (fairmlbook.org), http://www.fairmlbook. org.

17. Baur, C., Albarqouni, S. and Navab, N. (2017). Semi-supervised deep learning for fully convolutional networks, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2017, pp. 311–319.

18. Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., Nagar, S., Ramamurthy, K. N., Richards, J., Saha, D., Sattigeri, P., Singh, M., Varshney, K. R. and Zhang, Y. (2018). AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias, ArXiv: 1810.01943.

19. Benou, A., Veksler, R., Friedman, A. and Raviv, T. R. (2017). Ensemble of expert deep neural networks for spatio-temporal denoising of contrastenhanced MRI sequences, Medical Image Analysis 42, pp. 145–159, doi: 10.1016/j.media.2017.07.006, URL https://doi.org/10.1016/j. media.2017.07.006. 20. Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., et al. (2018). Deep learning techniques for automatic MRI cardiac multistructures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging 37, 11, pp. 2514–2525, doi: 10.1109/ TMI.2018.2837502. 21. Bishop, C. M. (2006). Pattern Recognition and Machine Learning (Springer-Verlag, Berlin, Heidelberg), ISBN 0387310738.

22. Bobak, C., Titus, A. and Hill, J. (2018). Comparison of common machine learning models for classification of tuberculosis using transcriptional biomarkers from integrated datasets, Applied Soft Computing 74, doi: 10.1016/j.asoc.2018.10.005.

23. Bonvin, R., Buhmann, J., Cortini Jimenez, C., Egger, M., Geissler, A., Krauthammer, M., Schirlo, C., Spiess, C., Steurer, J., Vokinger, K. N., et al. (2022). Studierenden auf den Einsatz von maschinellem Lernen vorbereiten, Schweizerische Ärztezeitung 103, 04, pp. 98–101.

19

20

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

24. Bortsova, G., Dubost, F., Hogeweg, L., Katramados, I. and de Bruijne, M. (2019). Semi-supervised medical image segmentation via learning consistency under transformations, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 (Cham), pp. 810–818. 25. Bovenberg, J., Peloquin, D., Bierer, B., Barnes, M. and Knoppers, B. M. (2020). How to fix the GDPR’s frustration of global biomedical research, Science 370, 6512, pp. 40–42, doi: 10.1126/ science.abd2499.

26. Breiman, L. (2001). Random forests, Machine Learning 45, 1, pp. 5–32, doi: 10.1023/a:1010933404324. 27. Bzdok, D., Altman, N. and Krzywinski, M. (2018). Statistics versus machine learning, Nature Methods 15, 4, pp. 233–234, doi: 10.1038/ nmeth.4642. 28. Caruana, R. (1997). Multitask learning, Machine Learning 28, 1, pp. 41–75, doi: 10.1023/a:1007379606734.

29. Castro, D. C., Walker, I. and Glocker, B. (2020). Causality matters in medical imaging, Nature Communications 11, 1, doi: 10.1038/ s41467020-17478-w. 30. Cha, K. H., Hadjiiski, L., Samala, R. K., Chan, H. P., Caoili, E. M. and Cohan, R. H. (2016). Urinary bladder segmentation in CT urography using deep-learning convolutional neural network and level sets, Medical Physics 43, 4, p. 1882, doi: 10.1118/1.4944498. 31. Chen, C., Qin, C., Qiu, H., Tarroni, G., Duan, J., Bai, W. and Rueckert, D. (2020). Deep learning for cardiac image segmentation: A review, Frontiers in Cardiovascular Medicine 7, p. 25, doi: 10.3389/ fcvm.2020.00025.

32. Chen, L. (2020). Overview of clinical prediction models, Annals of Translational Medicine 8, 4, pp. 71–71, doi: 10.21037/atm.2019. 11.121. 33. Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N., Venugopal, V., Mahajan, V., Rao, P. and Warier, P. (2018). Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study, The Lancet 392, doi: 10.1016/ S01406736(18)31645-3.

34. Chuang, L.-Y., Huang, H.-C., Lin, M.-C. and Yang, C.-H. (2011). Particle swarm optimization with reinforcement learning for the prediction of CpG islands in the human genome, PLoS ONE 6, 6, p. e21036, doi: 10.1371/journal.pone.0021036, URL https://doi. org/10.1371/ journal.pone.0021036.

References

35. Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L. and Prior, F. (2013). The cancer imaging archive (TCIA): Maintaining and operating a public information repository, Journal of Digital Imaging 26, 6, pp. 1045– 1057, doi: 10.1007/s10278-013-9622-7.

36. Codella, N. C. F., Nguyen, Q.-B., Pankanti, S., Gutman, D. A., Helba, B., Halpern, A. C. and Smith, J. R. (2017). Deep learning ensembles for melanoma recognition in dermoscopy images, IBM Journal of Research and Development 61, 4/5, pp. 5:1–5:15, doi: 10.1147/ JRD.2017.2708299. 37. Copeland, B. J. (2018). MYCIN, URL https://www.britannica. com/ technology/MYCIN, Encyclopedia Britannica.

38. Couteaux, V., Nempont, O., Pizaine, G. and Bloch, I. (2019). Towards interpretability of segmentation networks by analyzing deepdreams, in Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support, pp. 56–63. 39. Crawshaw, M. (2020). Multi-task learning with deep neural networks: A survey.

40. Cuocolo, R., Caruso, M., Perillo, T., Ugga, L. and Petretta, M. (2020). Machine learning in oncology: A clinical appraisal, Cancer Letters 481, pp. 55–62, doi: 10.1016/j.canlet.2020.03.032.

41. Dalca, A. V., Yu, E., Golland, P., Fischl, B., Sabuncu, M. R. and Eugenio Iglesias, J. (2019). Unsupervised deep learning for bayesian brain MRI segmentation, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, pp. 356–365. 42. Dean, J. (2021). Introducing pathways: A next-generation AI architecture, URL https://blog.google/technology/ai/ introducingpathways-next-generation-ai-architecture/.

43. Dhungel, N., Carneiro, G. and Bradley, A. P. (2017). A deep learning approach for the analysis of masses in mammograms with minimal user intervention, Medical Image Analysis 37, pp. 114–128, doi: 10.1016/j.media.2017.01.009. 44. Doi, K. (2007). Computer-aided diagnosis in medical imaging: Historical review, current status and future potential, Computerized Medical Imaging and Graphics 31, 4-5, pp. 198–211, doi: 10.1016/j. compmedimag.2007.02.002.

45. Doshi-Velez, F. and Kim, B. (2017). Towards a rigorous science of interpretable machine learning, ArXiv:1702.08608.

21

22

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

46. Elshawi, R., Al-Mallah, M. H. and Sakr, S. (2019). On the interpretability of machine learning-based model for predicting hypertension, BMC Medical Informatics and Decision Making 19, p. 146, doi: 10.1186/ s12911-019-0874-0. 47. Ernst, D., Stan, G.-B., Goncalves, J. and Wehenkel, L. (2006). Clinical data based optimal STI strategies for HIV: a reinforcement learning approach, in Proceedings of the 45th IEEE Conference on Decision and Control (IEEE), doi: 10.1109/cdc. 2006.377527, URL https://doi. org/10.1109/cdc.2006.377527.

48. Ertosun, M. G. and Rubin, D. L. (2015). Probabilistic visual search for masses within mammography images using deep learning, in 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1310–1315, doi: 10.1109/BIBM. 2015.7359868. 49. Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M. and Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks, Nature 542, pp. 115–118, doi: 10.1038/ nature21056.

50. Friedman, J. H. (2002). Stochastic gradient boosting, Computational Statistics & Data Analysis 38, 4, pp. 367–378, doi: 10.1016/s01679473(01)00065-2. 51. Gao, X. W., Hui, R. and Tian, Z. (2017). Classification of CT brain images based on deep learning networks, Computer Methods and Programs in Biomedicine 138, pp. 49–56, doi: 10.1016/j.cmpb. 2016.10.007.

52. Gardner, R. M. and Lundsgaarde, H. P. (1994). Evaluation of user acceptance of a clinical expert system, Journal of the American Medical Informatics Association 1, 6, pp. 428–438, doi: 10.1136/ jamia.1994.95153432. 53. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H. and Crawford, K. (2021). Datasheets for datasets, Communications of the ACM 64, 12, pp. 86–92, doi: 10.1145/3458723.

54. Ghorbani, A., Ouyang, D., Abid, A., He, B., Chen, J. H., Harrington, R. A., Liang, D. H., Ashley, E. A. and Zou, J. Y. (2020). Deep learning interpretation of echocardiograms, npj Digital Medicine 3, p. 10, doi: 10.1038/s41746-019-0216-8.

55. Goecks, J., Jalili, V., Heiser, L. M. and Gray, J. W. (2020). How machine learning will transform biomedicine, Cell 181, 1, pp. 92–101, doi: 10.1016/j.cell.2020.03.022. 56. Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning (MIT Press), http://www.deeplearningbook.org.

References

57. Grewal, M., Srivastava, M. M., Kumar, P. and Varadarajan, S. (2018). Radnet: Radiologist level accuracy using deep learning for hemorrhage detection in ct scans, in 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 281–284, doi: 10.1109/ ISBI.2018.8363574.

58. Guan, H. and Liu, M. (2021). Domain adaptation for medical image analysis: A survey, IEEE Transactions on Biomedical Engineering 69(3), pp. 1173–1185, doi: 10.1109/tbme.2021.3117407. 59. Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, The Journal of the American Medical Association 316(22), pp. 2402–2410, doi: 10.1001/jama. 2016.17216.

60. Hao, K. (2020). Doctors are using AI to triage COVID-19 patients. The tools may be here to stay, MIT Technology Review 27. 61. Harrell, F. (2021). Road map for choosing between statistical modeling and machine learning, URL https://www.fharrell. com/post/stat-ml/, accessed January 2022. 62. Hinton, G. E. (2016). On radiology, URL https://www.youtube. com/ watch?v=2HMPRXstSvQ, accessed January 2022.

63. Hoffman, M. M., Buske, O. J., Wang, J., Weng, Z., Bilmes, J. A. and Noble, W. S. (2012). Unsupervised pattern discovery in human chromatin structure through genomic segmentation, Nature Methods 9, 5, pp. 473–476, doi: 10.1038/nmeth.1937, URL https://doi.org/10.1038/ nmeth.1937.

64. Huang, T.-W., Chen, H.-T., Fujimoto, R., Ito, K., Wu, K., Sato, K., Taki, Y., Fukuda, H. and Aoki, T. (2017). Age estimation from brain MRI images using deep learning, in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp. 849–852, doi: 10.1109/ ISBI.2017.7950650.

65. Hussein, S., Kandel, P., Bolan, C. W., Wallace, M. B. and Bagci, U. (2019). Lung and pancreatic tumor characterization in the deep learning era: Novel supervised and unsupervised learning approaches, IEEE Transactions on Medical Imaging 38, 8, pp. 1777–1787, doi: 10.1109/ TMI.2019.2894349. 66. Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., et al. (2019). CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison, in 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, pp. 590–597.

23

24

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

67. Islam, J. and Zhang, Y. (2017). A novel deep learning based multiclass classification method for Alzheimer’s disease detection using brain MRI data, in Brain Informatics, pp. 213–222. 68. Iwendi, C., Bashir, A. K., Peshkar, A., Sujatha, R., Chatterjee, J. M., Pasupuleti, S., Mishra, R., Pillai, S. and Jo, O. (2020). Covid-19 patient health prediction using boosted random forest algorithm, Frontiers in Public Health 8, p. 357, doi: 10.3389/fpubh. 2020.00357, URL https:// www.frontiersin.org/article/10. 3389/fpubh.2020.00357.

69. James, C. A., Wheelock, K. M. and Woolliscroft, J. O. (2021). Machine learning: The next paradigm shift in medical education, Academic Medicine 96, 7, pp. 954–957, doi: 10.1097/ acm.0000000000003943.

70. James, G., Witten, D., Hastie, T. and Tibshirani, R. (2013). Unsupervised learning, in An Introduction to Statistical Learning with Applications in R (Springer New York), pp. 373–418, doi: 10.1007/978-1-4614-71387 10. 71. Jamieson, T. and Goldfarb, A. (2019). Clinical considerations when applying machine learning to decision-support tasks versus automation, BMJ Quality & Safety 28, 10, pp. 778–781, doi: 10.1136/ bmjqs-2019-009514. 72. Jha, A. K., DesRoches, C. M., Campbell, E. G., Donelan, K., Rao, S. R., Ferris, T. G., Shields, A., Rosenbaum, S. and Blumenthal (2009). Use of Electronic Health Records in U.S. Hospitals, England Journal of Medicine 360, 16, pp. 1628–1638, doi: 10.1056/nejmsa0900592.

73. Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A. and Mark, R. G. (2016). MIMICIII, a freely accessible critical care database, Scientific Data 3, 1, doi: 10.1038/sdata.2016.35, URL https:// doi.org/10.1038/sdata.2016.35.

74. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., et al. (2021). Highly accurate protein structure prediction with AlphaFold, Nature 596, 7873, pp. 583–589, doi: 10.1038/s41586-02103819-2. 75. Kaissis, G. A., Makowski, M. R., Rückert, D. and Braren, R. F. (2020). Secure, privacy-preserving and federated machine learning in medical imaging, Nature Machine Intelligence 2, 6, pp. 305–311, doi: 10.1038/ s42256-020-0186-1.

76. Kallenberg, M., Petersen, K., Nielsen, M., Ng, A. Y., Diao, P., Igel, C., Vachon, C. M., Holland, K., Winkel, R. R., Karssemeijer, N. and Lillholm, M. (2016). Unsupervised deep learning applied to breast density segmentation and mammographic risk scoring, IEEE Transactions on Medical Imaging 35, 5, pp. 1322–1331, doi: 10.1109/TMI.2016.2532122.

References

77. Karun, S., Raj, A. and Attigeri, G. (2019). Comparative analysis of prediction algorithms for diabetes, in Advances in Computer Communication and Computational Sciences (Springer Singapore), pp. 177–187. 78. Kersting, K. (2018). Machine learning and artificial intelligence: Two fellow travelers on the quest for intelligent behavior in machines, Frontiers in Big Data 1, doi: 10.3389/fdata.2018.00006.

79. Khan, M. U., Choi, J. P., Shin, H. and Kim, M. (2008). Predicting breast cancer survivability using fuzzy decision trees for personalized healthcare, in 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5148–5151, doi: 10.1109/IEMBS.2008.4650373. 80. Kharazmi, P., Zheng, J., Lui, H., Wang, Z. J. and Lee, T. K. (2018). A computer-aided decision support system for detection and localization of cutaneous vasculature in dermoscopy images via deep feature learning, Journal of Medical Systems 42, 33, doi: 10.1007/s10916-0170885-2. 81. Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C. and Faisal, A. A. (2018). The artificial intelligence clinician learns Medicine 24, 11, pp. 1716–1720, doi: 10.1038/s41591-018-0213-5, URL https://doi. org/10.1038/s41591-018-0213 5. 82. Lam, C., Yu, C., Huang, L. and Rubin, D. (2018). Retinal lesion detection with deep learning using image patches, Investigative Ophthalmology & Visual Science 59(1), pp. 590–596, doi: 10.1167/ iovs.17-22721.

83. LeCun, Y. and Misra, I. (2021). Self-supervised learning: The dark matter of intelligence, URL https://ai.facebook.com/blog/ selfsupervised-learning-the-dark-matter-of-intelligence/. 84. Li, S., Wei, J., Chan, H.-P., Helvie, M. A., Roubidoux, M. A., Lu, Y., Zhou, C., Hadjiiski, L. M. and Samala, R. K. (2018). Computer-aided assessment of breast density: comparison of supervised deep learning and feature-based statistical learning, Physics in Medicine & Biology 63, 2, p. 025005, doi: 10.1088/ 1361 6560/aa9f87.

85. Li, S., Zhang, C. and He, X. (2020). Shape-aware semi-supervised 3d semantic segmentation for medical images, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, pp. 552–561. 86. Li, Z., Hou, Z., Chen, C., Hao, Z., An, Y., Liang, S. and Lu, B. (2019). Automatic cardiothoracic ratio calculation with deep learning, IEEE Access 7, pp. 37749–37756, doi: 10.1109/ACCESS. 2019.2900053.

25

26

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

87. Lipton, Z. C. (2017). The doctor just won’t accept that! ArXiv: 1711.08037.

88. Lipton, Z. C. (2018). The mythos of model interpretability, Queue 16, 3, pp. 31–57, doi: 10.1145/3236386.3241340.

89. Liu, Q., Yu, L., Luo, L., Dou, Q. and Heng, P. A. (2020a). Semi-supervised medical image classification with relation-driven self-ensembling model, IEEE Transactions on Medical Imaging 39, 11, pp. 3429–3440, doi: 10.1109/TMI.2020.2995518. 90. Liu, Y., Jain, A., Eng, C., et al. (2020b). A deep learning system for differential diagnosis of skin diseases, Nature Medicine 26, pp. 900– 908, doi: 10.1038/s41591-020-0842-3.

91. Lonini, L., Dai, A., Shawen, N., Simuni, T., Poon, C., Shimanovich, L., Daeschler, M., Ghaffari, R., Rogers, J. A. and Jayaraman, A. (2018). Wearable sensors for Parkinson’s disease: which data are worth collecting for training symptom detection models, npj Digital Medicine 64, doi: 10.1038/s41746-018-0071-z. 92. Lopez, C., Tucker, S., Salameh, T. and Tucker, C. (2018). An unsupervised machine learning method for discovering patient clusters based on genetic signatures, Journal of Biomedical Informatics 85, pp. 30–39, doi: 10.1016/j.jbi.2018.07.004.

93. Lotter, W., Diab, A. R., Haslam, B., Kim, J. G., Grisot, G., Wu, E., Wu, K., Onieva, J. O., et al. (2021). Robust breast cancer detection in mammography and digital breast tomosynthesis using an annotationefficient deep learning approach, Nature Medicine, pp. 244–249, doi: 10.1038/ s41591-020-01174-9. 94. Luo, X., Chen, J., Song, T. and Wang, G. (2021). Semi-supervised medical image segmentation through dual-task consistency, in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35(10), pp. 8801– 8809. 95. Madani, A., Moradi, M., Karargyris, A. and Syeda-Mahmood, T. (2018). Semi-supervised learning with generative adversarial networks for chest X-ray classification with ability of data domain adaptation, in 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1038–1042, doi: 10.1109/ ISBI.2018.8363749.

96. Mahmud, M., Kaiser, M. S., Hussain, A. and Vassanelli, S. (2018). Applications of deep learning and reinforcement learning to biological data, IEEE Transactions on Neural Networks and Learning Systems 29, 6, pp. 2063–2079, doi: 10.1109/TNNLS. 2018.2790388.

References

97. Maninis, K.-K., Pont-Tuset, J., Arbeláez, P. and Van Gool, L. (2016). Deep retinal image understanding, in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 (Springer International Publishing, Cham), pp. 140–148. 98. May, M. (2021). Eight ways machine learning is assisting medicine, Nature Medicine 27, 1, pp. 2–3, doi: 10.1038/s41591-020-01197-2. 99. McCarthy, J. (2007). What is artificial intelligence? URL http: //jmc. stanford.edu/artificial-intelligence/what-is-ai/, accessed January 2022. 100. Miao, S., Wang, Z. J. and Liao, R. (2016). A CNN regression approach for real-time 2d/3d registration, IEEE Transactions on Medical Imaging 35, 5, pp. 1352–1363, doi: 10.1109/TMI.2016. 2521800.

101. Miotto, R., Li, L., Kidd, B. A. and Dudley, J. T. (2016). Deep patient: An unsupervised representation to predict the future of patients from the electronic health records, Scientific Reports 6, 1, doi: 10.1038/ srep26094, URL https://doi.org/10.1038/. 102. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D. and Gebru, T. (2019). Model cards for model reporting, in Proceedings of the Conference on Fairness, Accountability, and Transparency (ACM), pp. 220–229, doi: 10.1145/3287560.3287596. 103. Mitchell, T. M. (1997). Machine Learning (McGraw-Hill, New York), ISBN 9780070428072.

104. Mohamed, A. A., Luo, Y., Peng, H., Jankowitz, R. C. and Wu, S. (2018). Understanding clinical mammographic breast density assessment: a deep learning perspective, Journal of Digital Imaging 31(4), pp. 387– 392, doi: 10.1007/s10278-017-0022-2. 105. Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective (The MIT Press), ISBN 0262018020.

106. Ngiam, K. Y. and Khor, I. W. (2019). Big data and machine learning algorithms for health-care delivery, The Lancet Oncology 20, 5, pp. e262–e273, doi: 10.1016/s1470-2045(19)30149-4.

107. Nie, D., Gao, Y., Wang, L. and Shen, D. (2018). ASDNet: Attention based semi-supervised deep networks for medical image segmentation, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, pp. 370–378.

108. Njage, P., Leekitcharoenphon, P. and Hald, T. (2018). Improving hazard characterization in microbial risk assessment using next generation sequencing data and machine learning: Predicting clinical outcomes

27

28

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

in shigatoxigenic Escherichia coli, International Journal of Food Microbiology 292, doi: 10.1016/j.ijfoodmicro.2018. 11.016.

109. Obermeyer, Z. and Emanuel, E. J. (2016). Predicting the future — big data, machine learning, and clinical medicine, New England Journal of Medicine 375, 13, pp. 1216–1219, doi: 10.1056/ nejmp1606181.

110. Outang, D., et al. (2020). Video-based AI for beat-to-beat assessment of cardiac function, Nature 580, pp. 252–256, doi: 10.1038/s41586-0202145-8. 111. Panch, T., Mattie, H. and Celi, L. A. (2019). The “inconvenient truth” about AI in healthcare, npj Digital Medicine 2, 1, doi: 10.1038/s41746019-0155-4.

112. Park, S., Lee, S. M., Lee, K. H., Jung, K.-H., Bae, W., Choe, J. and Seo, J. B. (2020). Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings, European Radiology 30, doi: 10.1007/ s00330-019-06532-x. 113. Pearl, J. (1995). Causal diagrams for empirical research, Biometrika 82, 4, pp. 669–688, doi: 10.1093/biomet/82.4.669.

114. Pearl, J. (2009). Causality (Cambridge University Press), doi: 10. 1017/ cbo9780511803161. 115. Pearl, J. (2019). The seven tools of causal inference, with reflections on machine learning, Communications of the ACM 62, 3, pp. 54–60, doi: 10.1145/3241036.

116. Pereira, S., Meier, R., Alves, V., Reyes, M. and Silva, C. A. (2018). Automatic brain tumor grading from MRI data using convolutional neural networks and quality assessment, in Understanding and Interpreting Machine Learning in Medical Image Computing Applications (Springer International Publishing), pp. 106–114. 117. Pärkkä, J., Ermes, M., Korpipää, P., Mäntyjärvi, J., Peltola, J. and Korhonen, I. (2006). Activity classification using realistic data from wearable sensors, IEEE Transactions on Information Technology in Biomedicine 10(1), pp. 119–128, doi: 10.1109/titb. 2005.856863.

118. Qayyum, A., Qadir, J., Bilal, M. and Al-Fuqaha, A. (2021). Secure and robust machine learning for healthcare: A survey, IEEE Reviews in Biomedical Engineering 14, pp. 156–180, doi: 10.1109/rbme.2020.3013489.

119. Quinlan, J. R. (1986). Induction of decision trees, Machine Learning 1, 1, pp. 81–106, doi: 10.1007/bf00116251. 120. Rahman, T., Chowdhury, M. E. H., Khandakar, A., Islam, K. R., Islam, K. F., Mahbub, Z. B., Kadir, M. A. and Kashem, S. (2020). Transfer learning with deep convolutional neural network (CNN) for pneumonia

References

detection using chest X-ray, Applied Sciences 10, 9, doi: 10.3390/ app10093233, URL https://www.mdpi.com/ 2076-3417/10/9/3233.

121. Raita, Y., Camargo, C. A., Liang, L. and Hasegawa, K. (2021). Big data, data science, and causal inference: A primer for clinicians, Frontiers in Medicine 8, doi: 10.3389/fmed.2021.678047.

122. Rajaraman, S., Sornapudi, S., Alderson, P., Folio, L. and Antani, S. (2020). Analyzing inter-reader variability affecting deep ensemble learning for Covid-19 detection in chest radiographs, PLoS One 15, 11, doi: 10.1371/journal.pone.0242301.

123. Rajkomar, A., Dean, J. and Kohane, I. (2019). Machine learning in medicine, New England Journal of Medicine 380, 14, pp. 1347–1358, doi: 10.1056/nejmra1814259. 124. Rajkomar, A., Hardt, M., Howell, M. D., Corrado, G. and Chin, M. H. (2018). Ensuring fairness in machine learning to advance health equity, Annals of Internal Medicine 169, 12, pp. 866–872.

125. Rajpurkar, P., O’Connell, C., Schechter, A., et al. (2020). CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest X-rays in patients with HIV, npj Digital Medicine 3, doi: 10.1038/ s41746-020-00322-2.

126. Ralha, C., Schneider, H. W., Walter, M. E. and Bazzan, A. L. C. (2010). Reinforcement learning method for BioAgents, in 2010 Eleventh Brazilian Symposium on Neural Networks (IEEE), doi: 10.1109/ sbrn.2010.27, URL https://doi.org/10. 1109/sbrn.2010.27. 127. Ravuri, S., Lenc, K., Willson, M., Kangin, D., Lam, R., Mirowski, P., Fitzsimons, M., et al. (2021). Skilful precipitation nowcasting using deep generative models of radar, Nature 597, 7878, pp. 672–677, doi: 10.1038/ s41586-021-03854-z.

128. Roth, J. A., Battegay, M., Juchler, F., Vogt, J. E. and Widmer, A. F. (2018). Introduction to machine learning in digital healthcare epidemiology, Infection Control & Hospital Epidemiology 39, 12, pp. 1457–1462, doi: 10.1017/ice.2018.265.

129. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence 1, 5, pp. 206–215, doi: 10.1038/s42256-0190048-x.

130. Russell, S. and Norvig, P. (2010). Artificial Intelligence: A Modern Approach, 3rd edn. (Prentice Hall).

131. Sabater, S., Rovirosa, A. and Arenas, M. (2021). In response to Korreman s. et al. radiation oncologists are, above all, medical doctors, Clinical

29

30

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

and Translational Radiation Oncology 28, pp. 116–117, doi: 10.1016/j. ctro.2021.03.005.

132. Sahba, F., Tizhoosh, H. R. and Salama, M. M. (2008). Application of reinforcement learning for segmentation of transrectal ultrasound images, BMC Medical Imaging 8, 1, doi: 10.1186/1471-2342-8-8, URL https://doi.org/10.1186/1471-2342-8-8.

133. Samek, W. and Müller, K.-R. (2019). Towards explainable artificial intelligence, in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (Springer International Publishing), pp. 5–22, doi: 10.1007/978-3-030-28954-6 1.

134. Saria, S. (2018). Individualized sepsis treatment using reinforcement learning, Nature Medicine 24, 11, pp. 1641–1642, doi: 10.1038/ s41591-018-0253-x, URL https://doi.org/10.1038/s41591-0180253-x. 135. Sarkar, S. K., Roy, S., Alsentzer, E., McDermott, M. B. A., Falck, F., Bica, I., Adams, G., Pfohl, S. and Hyland, S. L. (2020). Machine learning for health (ML4H) 2020: Advancing healthcare for all, in E. Alsentzer, M. B. A. McDermott, F. Falck, S. K. Sarkar, S. Roy and S. L. Hyland (eds.), Proceedings of the Machine Learning for Health NeurIPS Workshop, Proceedings of Machine Learning Research, Vol. 136 (PMLR), pp. 1–11. 136. Sathyanarayana, A., Joty, S., Fernandez-Luque, L., Ofli, F., Srivastava, J., Elmagarmid, A., Arora, T. and Taheri, S. (2016). Sleep quality prediction from wearable data using deep learning, JMIR mHealth and uHealth 4, 4, p. e125, doi: 10.2196/mhealth. 6562, URL https://doi.org/10.2196/ mhealth.6562.

137. Sayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., et al. (2019). Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy, Ophthalmology 126, pp. 552–564, doi: 10.1016/j.ophtha.2018.11. 016. 138. Schelb, P., Kohl, S., Radtke, J. P., Wiesenfarth, M., Kickingereder, P., Bickelhaupt, S., et al. (2019). Classification of cancer at prostate MRI: Deep learning versus clinical pi-rads assessment, Radiology 293, 3, pp. 607–617, doi: 10.1148/radiol.2019190938. 139. Schmidhuber, J. (2015). Deep learning in neural networks: An overview, Neural Networks 61, pp. 85–117, doi: 10.1016/j.neunet. 2014.09.003.

140. Schölkopf, B. (2019). Causality for machine learning, ArXiv: 1911.10500.

141. Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A. and Bengio, Y. (2021). Toward causal representation

References

learning, Proceedings of the IEEE 109, 5, pp. 612–634, doi: 10. 1109/ jproc.2021.3058954.

142. Shang, H., Sun, Z., Yang, W., Fu, X., Zheng, H., Chang, J. and Huang, J. (2019). Imaging classification: Evaluation of transfer, multi-task and semisupervised learning, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, pp. 431–439.

143. Shen, L., Margolies, L. R., Rothstein, J. H., Fluder, E., McBride, R. and Sieh, W. (2019). Deep learning to improve breast cancer detection on screening mammography, Scientific Reports, doi: 10. 038/s41598-01948995-4. 144. Sheridan, T. B. and Thompson, J. M. (2018). People versus computers in medicine, in Human Error in Medicine (CRC Press), pp. 141–158.

145. Shilo, S., Rossman, H. and Segal, E. (2020). Axes of a revolution: Challenges and promises of big data in healthcare, Nature Medicine 26, 1, pp. 29–38, doi: 10.1038/s41591-019-0727-5. 146. Sidey-Gibbons, J. A. M. and Sidey-Gibbons, C. J. (2019). Machine learning in medicine: A practical introduction, BMC Medical Research Methodology 19, 1, doi: 10.1186/s12874-019-0681-4.

147. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., et al. (2017). Mastering the game of Go without human knowledge, Nature 550, 7676, pp. 354–359, doi: 10.1038/ nature24270.

148. Stoltzfus, J. C. (2011). Logistic regression: A brief primer, Academic Emergency Medicine 18, 10, pp. 1099–1104, doi: 10.1111/j.15532712.2011.01185.x. 149. Strumbelj, E., Bosnic, Z., Kononenko, I., Zakotnik, B. and Kuhar, C. G. (2010). Explanation and reliability of prediction models: the case of breast cancer recurrence, Knowledge and Information Systems 24(2), pp. 305–324, doi: 10.1007/s10115-009-0244-9. 150. Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction (MIT press).

151. Taghanaki, S. A., Kawahara, J., Miles, B. and Hamarneh, G. (2017). Paretooptimal multi-objective dimensionality reduction deep auto-encoder for mammography classification, Computer Methods and Programs in Biomedicine 145, pp. 85–93, doi: 10.1016/j.cmpb.2017.04.012. 152. The Cancer Imaging Archive (TCIA) (2021). Access data usage stats, URL https://www.cancerimagingarchive.net/stats/, accessed January 2022.

153. Tibshirani, R. and Hastie, T. (2021). A melting pot, Observational Studies 7, 1, pp. 213–215, doi: 10.1353/obs.2021.0012.

31

32

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

154. Umpierrez, G. E. and Klonoff, D. C. (2018). Diabetes technology update: Use of insulin pumps and continuous glucose monitoring in the hospital, Diabetes Care 41, 8, pp. 1579–1589, doi: 10.2337/ dci180002.

155. van Engelen, J. E. and Hoos, H. H. (2019). A survey on semisupervised learning, Machine Learning 109, 2, pp. 373–440, doi: 10.1007/s10994019-05855-6. 156. Wang, D., Zhang, Y., Zhang, K. and Wang, L. (2020a). Focalmix: Semisupervised learning for 3D medical image detection, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3950–3959, doi: 10.1109/CVPR42600. 2020.00401.

157. Wang, Y., Yao, Q., Kwok, J. T. and Ni, L. M. (2021). Generalizing from a few examples, ACM Computing Surveys 53, 3, pp. 1–34, doi: 10.1145/3386252.

158. Wang, Y., Zhao, Y., Therneau, T. M., Atkinson, E. J., Tafti, A. P., Zhang, N., Amin, S., Limper, A. H., Khosla, S. and Liu, H. (2020b). Unsupervised machine learning for the discovery of latent disease clusters and patient subgroups using electronic health records, Journal of Biomedical Informatics 102, p. 103364, doi: 10.1016/j.jbi.2019.103364. 159. Weisberg, S. (2014). Applied Linear Regression, 4th edn. (Wiley, Hoboken NJ).

160. Weston, A. D., Korfiatis, P., Kline, T. L., Philbrick, K. A., Kostandy, P., Sakinis, T., et al. (2019). Automated abdominal segmentation of CT scans for body composition analysis using deep learning, Radiology 290, 3, pp. 669–679, doi: 10.1148/radiol.2018181432. 161. Whang, S. E., Roh, Y., Song, H. and Lee, J.-G. (2021). Data collection and quality challenges in deep learning: A data-centric ai perspective, ArXiv: 2112.06409.

162. Wilkinson, J., Arnold, K. F., Murray, E. J., van Smeden, M., Carr, K., Sippy, R., de Kamps, et al. (2020). Time to reality check the promises of machine learning-powered precision medicine, The Lancet Digital Health 2, 12, pp. e677–e680, doi: 10.1016/s2589-7500(20)30200-4. 163. Worrall, D. E., Wilson, C. M. and Brostow, G. J. (2016). Automated retinopathy of prematurity case detection with convolutional neural networks, in Deep Learning and Data Labeling for Medical Applications, pp. 68–76. 164. Xu, L., Liu, M., Shen, Z., Wang, H., Liu, X., Wang, X., Wang, S., Li, T., Yu, S., Hou, M., Guo, J., Zhang, J. and He, Y. (2020). DW-Net: A cascaded convolutional neural network for apical four-chamber view segmentation in fetal

References

echocardiography, Computerized Medical Imaging and Graphics 80, p. 101690, doi: 10.1016/j.compmedimag.2019.101690.

165. Xuan, P., Sun, C., Zhang, T., Ye, Y., Shen, T. and Dong, Y. (2019). Gradient boosting decision tree-based method for predicting interactions between target genes and drugs, Frontiers in Genetics 10, p. 459, doi: 10.3389/fgene.2019.00459. 166. Yacchirema, D., Puga, J., Palau, C. and Esteve, M. (2018). Fall detection system for elderly people using IoT and big data, Procedia Computer Science 130, pp. 603–610, doi: 10.1016/j.procs.2018.04.110.

167. Yan, Q., Weeks, D. E., Xin, H., Swaroop, A., Chew, E. Y., Huang, H., Ding, Y. and Chen, W. (2020). Deep-learning-based prediction of late agerelated macular degeneration progression, Nature Machine Intelligence 2, pp. 141–150, doi: 10.1038/s42256-020-0154-9.

168. Yu, L., Guo, Y., Wang, Y., Yu, J. and Chen, P. (2017). Segmentation of fetal left ventricle in echocardiographic sequences based on dynamic convolutional neural networks, IEEE Transactions on Biomedical Engineering 64, 8, pp. 1886–1895, doi: 10.1109/ TBME.2016.2628401. 169. Zhang, J., Cui, Y., Jiang, W. and Wang, L. (2015). Blood vessel segmentation of retinal images based on neural network, in Image and Graphics, pp. 11–17.

170. Zhang, J., Gajjala, S., Agrawal, P., Tison, G. H., Hallock, L. A., BeussinkNelson, L., et al. (2018). Fully automated echocardiogram interpretation in clinical practice, Circulation 138, 16, pp. 1623–1635, doi: 10.1161/ CIRCULATIONAHA.118. 034338. 171. Zhang, J. J., Liu, K., Khalid, F., Hanif, M. A., Rehman, S., Theocharides, T., Artussi, A., Shafique, M. and Garg, S. (2019). Building robust machine learning systems, in Proceedings of the 56th Annual Design Automation Conference 2019 (ACM), pp. 1–4, doi: 10.1145/3316781.3323472. 172. Zhang, Y. and Ma, Y. (2019). Application of supervised machine learning algorithms in the classification of sagittal gait patterns of cerebral palsy children with spastic diplegia, Computers in Biology and Medicine 106, pp. 33–39, doi: 10.1016/j.compbiomed.2019.01.009.

173. Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D. P. and Chen, D. Z. (2017). Deep adversarial networks for biomedical image segmentation utilizing unannotated images, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2017, pp. 408–416. 174. Zhao, Q., Honnorat, N., Adeli, E., Pfefferbaum, A., Sullivan, E. V. and Pohl, K. M. (2019). Variational autoencoder with truncated mixture of gaussians for functional connectivity analysis, in Information Processing in Medical Imaging, pp. 867–879.

33

34

Introduction to Machine Learning for Physicians: A Survival Guide for Data Deluge

175. Zhao, Y., Kosorok, M. R. and Zeng, D. (2009). Reinforcement learning design for cancer clinical trials, Statistics in Medicine 28, 26, pp. 3294–3315, doi: 10.1002/sim.3720, URL https://doi. org/10.1002/ sim.3720. 176. Zheng, T., Xie, W., Xu, L., He, X., Zhang, Y., You, M., Yang, G. and Chen, Y. (2017). A machine learning-based framework to identify type 2 diabetes through electronic health records, International Journal of Medical Informatics 97, pp. 120–127, doi: 10.1016/j.ijmedinf.2016.09.014. 177. Zhou, J. and Troyanskaya, O. G. (2015). Predicting effects of noncoding variants with deep learning–based sequence model, Nature Methods 12, 10, pp. 931–934, doi: 10.1038/nmeth.3547, URL https://doi. org/10.1038/nmeth.3547. 178. Zhou, Z.-H. (2017). A brief introduction to weakly supervised learning, National Science Review 5, 1, pp. 44–53, doi: 10.1093/ nsr/nwx106.

179. Zohora, M. F., Tania, M. H., Kaiser, M. S. and Mahmud, M. (2020). Forecasting the risk of type II diabetes using reinforcement learning, in 2020 Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 2020 4th International Conference on Imaging, Vision & Pattern Recognition (icIVPR) (IEEE), doi: 10.1109/ icievicivpr48672.2020.9306653, URL https: //doi.org/10.1109/ icievicivpr48672.2020.9306653. 180. Zong, B., Song, Q., Min, M. R., Cheng, W., Lumezanu, C., Cho, D. and Chen, H. (2018). Deep autoencoding gaussian mixture model for unsupervised anomaly detection, URL https://openreview.net/ forum?id=BJJLHbb0-.

181. Zorman, M., Eich, H. P., Kokol, P. and Ohmann, C. (2001). Comparison of three databases with a decision tree approach in the medical field of acute appendicitis, Studies in Health Technology and Informatics 84, p. 1414–1418.

182. Zou, J. and Schiebinger, L. (2018). AI can be sexist and racist — it’s time to make it fair, Nature 559, 7714, pp. 324–326, doi: 10.1038/d41586018-05707-8. 183. Østvik, A., Smistad, E., Aase, S. A., Haugen, B. O. and Lovstakken, L. (2019). Real-time standard view classification in transthoracic echocardiography using convolutional neural networks, Ultrasound in Medicine & Biology 45, 2, pp. 374–384, doi: 10.1016/j. ultrasmedbio.2018.07.024.

Chapter 2

Data Access and Use

Iñaki Soto-Rey,a* Sebastian Hofmann,a* Holger Storf,b Dennis Kadioglu,b Danny Ammon,c and Michael Storckd

aMedical Data Integration Center, Institute for Digital Medicine, University Hospital Augsburg, Augsburg, Germany bInstitute of Medical Informatics, Data Integration Centre, Goethe University Frankfurt, University Hospital Frankfurt, Frankfurt am Main, Germany cData Integration Center, Jena University Hospital, Jena, Germany dInstitute of Medical Informatics, University of Münster, Münster, Germany [email protected]

As introduced in the Data Integration chapter, medical research bases its fundament on clinical data. Evidence-based medicine and clinical data science go even further and need large datasets to provide useful results. Data sharing becomes therefore a key factor for clinical research. On the other hand, medical data includes sensitive personal and private information, and regulations are needed to define the purposes and users of these data. In this chapter, we introduce the current legislation for data use and sharing, as well as define the methods for secure and law-compliant data access and use. *The authors contributed equally to this work.

Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

36

Data Access and Use

2.1 Legislation Medical research stands and falls with data access. Each jurisdiction governs this differently and a full comparison would go beyond the scope of this book. For Europe, the rules for all member states are laid down in the European General Protection Regulation (GDPR1). But it is also worth looking at the regulations from outside the EU member states, as it is necessary for the import of European data (third country transfer (Articles 44 to 50 GDPR)) that the respective state is considered adequate. Here, we deal with “secondary use” health data collected in the context of a treatment relationship and subsequently used for research (Article 5(1)(b) GDPR). The US Health Insurance Portability and Accountability Act (HIPAA) of 1996 permits secondary use for research [10, 19]. Article 9 GDPR stipulates that medical data enjoys special protection but also allows for the processing of such data subject to the condition of informed consent as required by Article 9(2)(a) GDPR. The valid consent requirement not only provides researchers an avenue to obtain a subject’s data, but the requirement also insulates researchers from potential legal consequences such as criminal charges from the state or civil claims brought by data subjects. Still, the question arises whether this paradigm of self-determination is not simply an illusion for the patient and carte blanche for the researcher [12]. A power imbalance can manifest in multiple scenarios. For example, a patient seeking emergency treatment may not have the time or capacity to understand the consent requirement, and in dire treatment situations, the patient may not be physically or mentally capable of indicating consent at all [17]. In addition, in some circumstances, the researcher can obtain and subsequently process data that does not relate solely to the patient [8]. Other times, the researcher may not have established the purpose for the data request at the time of collection (recital 33) [19]. Patient disadvantages are also conceivable outside the research context. For this reason, the German Code of Criminal Procedure provides that patient data gained with the aid of biobanks can be confiscated for criminal prosecution [11]. In Sweden, such an approach even helped solve the murder of their foreign minister [7].

1 Regulation on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/ EC (Data Protection Directive).

Legislation

It is therefore clear that medical patients often lack the necessary information required for them to provide informed consent as required by Article 9(2)(a). And so, the problem arises – if data researchers need to obtain informed consent for all relevant data, their work would quickly become restricted due to considerable legal uncertainties. Moreover, relying on the paradigm of selfdetermination would allow patients not only to decide the limits of how and why their data can be utilized but whether researchers can even use their data, to begin with, a development that would run contrary to the interests of society [21]. The legislator anticipated this problem, and the GDPR, therefore, considers the Union’s objective of achieving a European Research Area, which explicitly includes public interest studies conducted in the realm of public health (recital 159 and Article 179 TFEU2). The GDPR requires a balance between the guarantee of private life (Art. 7 CFREU3) and the protection of personal data (Article 8 CFREU) on the one hand and the freedom of sciences, which includes the freedom of research (Article 13 CFREU) on the other [22]. The GDPR, therefore, opens the following possibilities of data processing to the researcher.





∑ Anonymization of data so that the data subject can no longer be identified. In such cases, the principles of the GDPR no longer apply (recital 26). ∑ The ability to obtain consent to process the data for entire areas of scientific research (“broad consent”) without being required to inform the data subject as to the particular uses of the data (recital 33). Nevertheless, it should be noted that broad consent requires consent, and therefore the difficulties addressed above remain. ∑ The option to obtain a waiver of consent from the data subject.

The last point may seem surprising at first, but consent is by no means a legal prerequisite for processing personal data in every case. The GDPR, in contrast to the preceding Directive 95/46/EC4 (Article 288 (2) TFEU), is binding, and directly applicable in all Member states. However, the GDPR also contains elements typical of 2 Treaty

on the Functioning of the European Union. of Fundamental Rights of the European Union. 4 Directive on the protection of individuals with regard to the processing of personal data and the free movement of such data. 3 Charter

37

38

Data Access and Use

directives as well, such as the provision allowing for the processing of special categories of personal data for scientific research purposes. Article 9(2)(j) GDPR provides for processing without consent based on Union law or the law of a member state for scientific research purposes, so-called “research exemptions.” To compensate for the potential privacy intrusion and to balance any possible privacy risks emanating from the processed data, Article 89 (1) GDPR requires “appropriate safeguards [...] for the rights and freedoms of the data subject.” This ensures that technical and organizational measures are put in place that guarantees data minimization [23]. Consequently, the data subject rights regulated in Articles 12 to 22 GDPR are weakened by the GDPR and member state regulation in order to uphold and maintain the freedom of data research (Article 89(2) GDPR)). Examples of rights that have been minimized include the right to the erasure of data, as this right has the potential to render impossible or otherwise seriously impair the achievement of the processing objectives outlined in (Article 17(3)(d) GDPR). Similar restrictions apply to purpose limitation (Article 5(1)(b) GDPR), storage limitation (Article 5(1)(e) GDPR), and information requirements (Article 14(5)(b) GDPR). The specter of derogation from these principles by the member states led to variations of the corresponding regulations and thus to new challenges. Researchers thus face a quandary – before data collection has even begun, they must now find out which legal regulations apply to the project in question, which can cause considerable additional work and transaction costs, especially in the case of projects involving multiple member states [19]. Finally, legal fragmentation and uncertainty mean that the GDPR fails to fully meet its objective of simultaneously protecting the rights of data subjects while also promoting research. Due to the lack of a clear and crossmember state regulation for processing data in the sense of Art. 9 GDPR, the GDPR cannot fully meet the balance between protection of personal data and freedom of research [21].

2.2 Data Access

It stands to reason that some conditions must be met before interested researchers can access data outside their institutional

Data Access

organizational and legal domain. On the one hand, there are aspects relating to legal certainty and data protection, as described above. On the other hand, it must be considered to what extent and in what form it is desired to pass on the data.

2.2.1 Governance

The questions “who owns patient data?” and “who may determine their use?” are certainly not easy to answer. In this context, possible parties are the respective physicians or researchers who collect the data, their institutional unit (hospital, university), and, most importantly, the patients themselves. In the process of accessing and using clinical data, the interest of these stakeholders must be taken into account through a comprehensible and transparent decisionmaking process. An effective means for this is the establishment of a central Governance Board or Use and Access Committee (UAC), which assumes the task of data usage decision-making.

2.2.1.1 Roles and rights

The structures and roles integrated into a proposed data usage governance process are outlined below.

Use and Access Committee

The so-called Use and Access Committee is an officially legitimized body defined by rules of procedure. This committee is integrated by voting and non-voting members. These are on the one hand representatives of the participating research institutions and different clinical areas, and on the other hand, representatives of data protection, ethics, medical informatics, biobank, statistics, and patients. The composition and distribution of voting rights can vary. The board meets regularly and makes decisions on data use requests from internal and external parties. If an appropriate structure is in place, decisions can be delegated to appropriate more specific boards.

Domain-specific Governance Boards

If governance structures are already established, it makes sense to integrate them into the decision-making process for relevant

39

40

Data Access and Use

requests. This may be the case in data use requests if only a specific medical field (e.g., cardiology or oncology) is concerned or if other research objects rather than data (e.g., biomaterial) have been requested and a governance board for them already exists.

Ethical Committee and Data Privacy Officer

Both the ethics committee and the data protection officer must be involved in the inquiry process. At best, the clarification takes place before the request reaches the Use and Access Committee. The decisions of the ethics committee and the data protection officer must be considered. At best, their advice is provided and considered before the request reaches the UAC. In any case, representatives of these two institutions may take part as members of the UAC.

Patient Representatives

The involvement of patient representatives in medical research and decision-making processes as described here is increasing and already plays an essential role in several data usage processes.

2.2.1.2 Data access and governance process

The all-encompassing Data Access and Governance Process can vary for the individual institutions depending on the existing structure and local requirements. Nevertheless, the steps can be roughly divided into the following ones:

Inquiry

The process is initiated by a request, e.g., by a researcher of the institution itself or an external researcher or company: “the inquirer.” The ways to do this are fundamentally different, a digitally supported process with standardized forms is recommended. The goal here is to obtain the information necessary for the following steps about the requesting person or institution itself, the background of the request, and initial details about the request or financial aspects.

Feasibility

For the inquirer, it is interesting to know whether his planned project is even possible with a certain partner. For this reason, he first performs a “feasibility request,” which can be carried out

Data Access

with less complex additional information [24]. Sharing these data is legally possible because the inquirer only receives the data that could potentially fit the study criteria and no personal information is being transmitted.

Contract

To secure the agreement conditions and protect them against misuse of the data, a contract is concluded between the stakeholders involved. To reduce the effort to a minimum, in most cases a standardized and legally verified data usage contract can be used as a template.

Provision of Data

After the feasibility has been positively concluded, permission has been granted by the governance board and the general conditions have been agreed upon and a contract has been signed, the data (or other research materials) can be made available to the requesting party. To do this, it is first necessary to identify what exact data is being requested and for what time period. If necessary, a data curation/transformation process must be initiated in order to fit the required quality and technical standards. For the provision, it must be clarified whether the data itself or the result of analysis has to be provided. The form of the desired provision must also be clarified. The transfer of the results must take place in a technically and operationally secure form.

Archive

After the data has been provided in the required manner, it is archived in the exact form. It is important that, just as the data is archived, the information on the preceding process is also archived and linked. The aim is to be able to reactivate this information when required and to present it in a traceable form. This information is called provenance. Definition: Provenance is information about entities, activities, and people involved in producing a piece of data or thing, which can be used to form assessments about its quality, reliability, or trustworthiness [1].

41

42

Data Access and Use

Connection to Data Networks Due to the growing number of medical research networks, some of which have a different focus, there is a growing need to consider the connection of local structures to such networks. In this context, e.g., framework agreements or fast tracks can be coordinated in the governance process. It should also be clarified to what extent the flow of information can be coordinated in order to reduce redundant surveys if necessary. One of the main achievements of the German Medical Informatics Initiative has been the formalization of an agreement on a nationally harmonized data use and access process, which, at an abstract level, divides the steps leading up to the use of data for medical research into three phases, as depicted in Fig. 2.1.

Figure 2.1 Indicated BPMN diagram for the data use and access process of the Medical Informatics Initiative in Germany, Task Force Process Modeling.

2.2.2 Pseudonymization and Anonymization In some use cases, data access may be restricted to a dataset where the included data should not be assignable to the individuals by the user of the dataset. To prevent re-identification of a data subject by the accessible data, the data set can be pseudonymized or anonymized.

2.2.2.1 Pseudonymization

Pseudonymization is one possible data protection procedure, where directly identifiable information like the name or social security

Data Access

number of the individuals in a dataset is replaced by a newly generated code called pseudonym. However, a pseudonymized data subject can be re-identified at least by the person who created the pseudonyms. The pseudonym can be generated by simply using continuous numbers or alphanumeric characters. A more sophisticated way of generating pseudonyms is the use of cryptographic, mathematical functions, or algorithms. When selecting the cryptographic algorithm, however, one should ensure that this algorithm still corresponds to the current state of the art. Personal information like the first character or the first name of the individual should never directly be used as pseudonyms.

2.2.2.2 Anonymization

In contrast to pseudonymization, the anonymization of a data set tries to ensure that the re-identification of an individual subject is no longer possible. However, re-identification of a data subject may be possible over time by breaching a used cryptographic algorithm for anonymization or by using newly developed artificial intelligence methods. Naturally, all directly identifiable information of the individuals has to be deleted in an anonymized dataset. Since certain information of an individual, like age or religion, may be unique in the given dataset, a property called k-anonymity was introduced. K-anonymity means that a data subject cannot be distinguished from at least k-1 other data subjects in the data set referring to one information. If k-anonymity is not given in a data set (k = 1), the corresponding information can be generalized, i.e., changing the concrete age to an age region (18 à “10 ≤ age < 20,” 25 à “20 ≤ age < 30”) or the information can be suppressed completely if it is not possible to generalize the information properly. In order to be effective, k must be ≥ 2. The higher the k, the more securely the data is protected against re-identification, but also the meaningfulness of the data decreases. K-anonymity without applying any other procedures is vulnerable to certain attacks and can be useless in specific data sets. An example can be visualized in the following table:

43

44

Data Access and Use

Table 2.1 Example for a generalization of a dataset, resulting in a dataset with k-anonymity property equals 3 regarding the age information ID

Age

Diagnosis

Age region

Diagnosis

1

21

Melanoma

20 ≤ age < 30

Melanoma

4

78

Diabetes

70 ≤ age < 80

Diabetes

2 3 5 6

72 26 29 74

Stroke

Melanoma Melanoma Dementia

Generalized to

70 ≤ age < 80 20 ≤ age < 30 20 ≤ age < 30 70 ≤ age < 80

Stroke

Melanoma Melanoma Dementia

There are three people aged between 20 and 30 in the original dataset. Then the age is generalized to “20 ≤ age < 30” and the k-anonymity property regarding age is now 3. However, all these subjects have the same diagnosis and anyone who knows, that a specific person between the ages of 20 and 30 is in the dataset also knows that this person has cancer. In order to prevent such information gain, one can additionally apply l-diversity for chosen sensible data items like the diagnosis. This property ensures that at least l distinct values in every group of k data sets are present for the chosen sensible data items. Another approach is the suppression of specific information to protect the individual data, combined with a risk assessment to measure the risk of disclosure of individual data [20].

Differential privacy

Nowadays, differential privacy is one of the strongest methods to control the disclosure of individual information from a given dataset [5]. The basis of this approach is the addition of noise to the data, while the design of the noise is defined by a mathematical definition of database privacy [2, 5]. Algorithms that add random noise to a dataset are called differentially private if the dataset can be changed slightly – add or remove one individual – and the risk of revealing individual information changes only indiscernible [5]. Thus, differential privacy secures the data against several attacks, which other techniques like k-anonymity and l-diversity are prone to [3].

Data Use

2.3 Data Use 2.3.1 Storage as a Specific Use Case Usually, health data is stored in the information systems of a healthcare center. Whenever medical data is to be collected for research, special systems are established. These can either be patient registries or research databases. Both have their specific fields of application, depending on the setting and the requirements.

2.3.1.1 Patient registries

Medical research usually involves the analysis of patient data. In many cases, these data are collected specifically for a single or a group of similar diseases, a so-called registry. The key for such registries is, that a well-defined set of data elements is captured for each patient or participant included in the registry. Software solutions used to build such a registry are called electronic data capture (EDC) systems accordingly. As these data are mostly documented in planned and structured situations, quality control measurements are a common part of the data collection process. Data gathered in a hospital environment is in general checked against administrative, regulatory, and billing requirements. In opposite registry data is validated to meet very specific requirements, depending on the individual research hypothesis, and is then also called qualitycontrolled data. These are especially required to stand up to the requirements of established peer review processes and admission authorities. EDC has to be differentiated on two levels: 1. human vs. machine entry and 2. data provision by professionals vs. patients. Both levels can also have an impact on data quality. Where human entry is in general simple to implement, it increases effort for stakeholders, both in manual data capture and in the required verification. In contrast, machine entry largely excludes typing errors, and the necessary interfaces for the data flow between the source and target system can be complex to implement. On the other hand, higher quality of data can be expected, if a professional documentalist enters and validates the relevant data, as the background of the data collection and the associated quality requirements are known, whereas, if

45

46

Data Access and Use

a patient enters these data, this knowledge is usually not present and therefore a thorough validation done by the patient cannot be expected. Such data provisioned by the patient is also called patientreported outcome and measurement (PROM).

2.3.1.2 Research databases

If data is not to be raised for specific research but, instead elsewhere collected data is to be reused, a so-called research database can be established. Such data collections can be created and continuously filled manually, e.g., as an Excel file, or automated as a data warehouse (see Chapter 3). In addition, it is also possible to use EDC software, in which case one part of the data set can be filled manually and the other part automatically. The scope of data can either be specified with broad and general research targets in mind or can be highly customized and tailored for very specific research hypotheses. While in the first case, the data might be useful for answering rather simple and general research questions, they might be not complete and meaningful enough for complex and highly individual challenges like doing research in the field of rare diseases. In summary, registry data contains usually many details about the individuals and can therefore be used for analyses on the patient level, whereas a data warehouse could be a good source for answering the already mentioned feasibility inquiries. Like with registries, data quality is an important aspect of research databases. As already mentioned, healthcare data is usually not validated with the requirements of research in mind. This means that data quality has to be proven and considered during analysis. A general rule of thumb is garbage in garbage out, i.e., non-qualitycontrolled healthcare data, which has been stored in a research database, is still of low quality, when extracted for use.

2.3.2 Data Sharing

As previously stated, data sharing is essential for successful clinical research. In the context of data sharing, it should be specified what kind of data is being shared, as anonymous data do not fall into data protection restrictive laws and is, therefore “shareable.” A solution to comply with data privacy regulations and still “share” the data are federated (or distributed) analysis, in which the data

References

analysis takes place in the institutions that generate the data and only aggregated (and anonymous) data are shared. Some of the most popular methods for federated analysis are currently the ones included in the OBiBa project [18] and the personal health train [6]. Also, well-known international Programs such as the Observational Health Data Sciences and Informatics (OHDSI) developed a solution for standardizing federated data analysis which includes a set of guidelines and tools, together with a common data model called Observational Medical Outcomes Partnership (OMOP) [16]. From the data privacy perspective, federated analyses would be the preferable solution for data sharing, but this is not always possible as some institutions can contribute to providing data but do not have the resources to perform the analysis themselves. In these cases, the only possible method is centralized data storage. Multi-centric studies and several initiatives around the globe pursue the goal of clinical data sharing across several institutions. The PCORnet, e.g., established a national patient-centered clinical research network in the USA [14]. In Germany, the Medical Informatics Initiative [13] as well as the Network University Medicine [15] aim to create a data sharing framework using data integration centers as data endpoints. Also, the pharmaceutical industry is extremely eager to access clinical data across countries and institutions and invest in several projects to establish these kinds of frameworks, often in cooperation with some countries like in the case of the Innovative Medicine Initiative [9], supporting projects such as the European Clinical Research Infrastructure Network “ECRIN” [4].

References

1. An Overview of the PROV Family of Documents (2013). Retrieved from https://www.w3.org/TR/prov-overview/ (accessed Mar. 30, 2022).

2. Dwork, C., McSherry, F., Nissim, K, Smith, A. (2006). Calibrating Noise to Sensitivity in Private Data Analysis, pp. 265–284: Retrieved from https://link.springer.com/chapter/10.1007/11681878_14 (accessed Mar. 30, 2022). 3. Dwork, C., Roth, A. (2014). The algorithmic Foundations of Differential Privacy, pp. 211–407: Retrieved from https://www.cis.upenn. edu/~aaroth/Papers/privacybook.pdf (accessed Mar. 30, 2022).

47

48

Data Access and Use

4. ECRIN: https://ecrin.org (accessed Mar. 30, 2022).

5. Ficek, J., Wang, W., Chen, H., Dagne, G., Daley, E. (2021). Differential privacy in health research: A scoping review: Retrieved from https:// academic.oup.com/jamia/article/28/10/2269/6333353 (accessed Mar. 30, 2022). 6. GO FAIR, Personal Health Train: Retrieved from https://www.go-fair. org/implementation-networks/overview/personal-health-train/ (accessed Mar. 30, 2022).

7. Hofmann, B. (2009). Broadening consent—and diluting ethics? J Med Ethics, 35, pp. 125–129. 8. Horton, R., Lucassen, A. (2019). Consent and autonomy in the genomics era. Current Genetic Medicine Reports, 7, pp. 85–91.

9. Innovative medicines initiative: Retrieved from https://www.imi. europa.eu/ (accessed Mar. 30, 2022).

10. Kaplan, B. (2016). How should health data be used? Camb Q Healthc Ethics, 25, pp. 312–329. 11. KK-StPO/Hadamitzky § 81 e Rn. 8.

12. Lindner, J. F. (2017). Das Paradoxon der Selbstbestimmung. In Selbst oder bestimmt?: Illusionen und Realitäten des Medizinrechts (Lindner, J. F., ed.). pp. 9–26, Nomos Verlagsgesellschaft mbH & Co. KG, BadenBaden. 13. Medicine Informatics Initiative: Retrieved from https://www. medizininformatik-initiative.de/) (accessed Mar. 30, 2022). 14. National Library of Medicine, Launching PCORnet, a national patientcentered clinical research network (2014): Retrieved from https:// pubmed.ncbi.nlm.nih.gov/24821743/ (accessed Mar. 30, 2022). 15. Network University Medicine: Retrieved from https://www.netzwerkuniversitaetsmedizin.de/ (accessed Mar. 30, 2022).

16. Observational Health Data sciences and informatics (2022): Retrieved from https://www.ohdsi.org/data-standardization/the-commondata-model (accessed Mar. 30, 2022).

17. O’Neill, O. (2003) Some limits of informed consent. Journal of Medical Ethics, 29, pp. 4–7. 18. Open Source Software for Epidemiology, https://www.obiba.org/ (accessed Mar. 30, 2022).

19. Peloquin, D., DiMaio, M., Bierer, B., Barnes, M. (2020). Disruptive and avoidable: GDPR challenges to secondary research uses of data. European Journal of Human Genetics, 28, pp. 697–705.

References

20. Prasser, F., Spengler, H., Bild, R., Eicher, J., Kuhn, K. (2019). Privacyenhancing ETL-processes for biomedical data, pp. 72–81: Retrieved from https://www.sciencedirect.com/science/article/pii/ S1386505618307007 (accessed Mar. 30, 2022). 21. Reichel, J. (2021). Allocation of regulatory responsibilities: Who will balance individual rights, the public interest and biobank research under the GDPR? In: Slokenberga, S., Tzortzatou, O., Reichel, J., et al. GDPR and Biobanking: Individual Rights, Public Interest and Research Regulation across Europe, pp. 421–434, Springer International Publishing, Cham. 22. Rossnagel, A. (2019). Datenschutz in der Forschung. ZD, pp. 157–164.

23. Shabani, M., Chassang, G., Marelli, L. (2021). The impact of the GDPR on the governance of biobank research. In: Slokenberga, S., Tzortzatou, O., Reichel, J., et al.). GDPR and Biobanking: Individual Rights, Public Interest and Research Regulation across Europe, pp. 45–60, Springer International Publishing, Cham.

24. Soto-Rey, I., Trincek, B., Karakoyun, T., Dugas, M., Fritz, F. (2014). Protocol feasibility workflow using an automated multi-country patient cohort system. Studies in Health Technology and Informatics. 205, pp. 985–989. Retrieved from https://pubmed.ncbi.nlm.nih. gov/25160335/ (access Mar. 30, 2022).

49

Chapter 3

Data Integration

Holger Storf,a* Dennis Kadioglu,a* Michael Storck,b Iñaki Soto-Rey,c Sebastian Hofmann,c and Danny Ammond

aInstitute of Medical Informatics, Data Integration Centre, Goethe University Frankfurt, University Hospital Frankfurt, Frankfurt am Main, Germany bInstitute of Medical Informatics, University of Münster, Münster, Germany cMedical Data Integration Center, Institute for Digital Medicine, University Hospital Augsburg, Augsburg, Germany dData Integration Center, Jena University Hospital, Jena, Germany [email protected]

Medical research usually requires data from a cohort of appropriate patients. Collecting such data, even about a single patient, involves retrieving that data from different sources, oftentimes across different institutions. This means that before a researcher can do an all-inclusive analysis, all the data from different sources have to be combined. However, there can be many obstacles to take, both while *The authors contributed equally to this work. Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

52

Data Integration

gathering the required data as well as while combining these data. This whole process is called data integration. Definition: Data Integration

The process of combining data that reside in different sources, to provide users with a unified view of such data [13].

3.1 Introduction The necessity for data integration in medical research as well as health care is constantly rising, e.g., due to increasing personalized patient individual treatment strategies for a specific tumor mutation, or any of the thousands of rare diseases. In both cases, the number of patients as providers of valuable data may be too low, when only taking into account a single healthcare institution. However, the complexity to tackle during data integration can vary significantly from case to case. If we look at a disease-specific patient registry, we can assume, that the data has been collected using the same format and structure, i.e., the filled-out case report form looks exactly the same for every patient included in the registry. The researcher can then analyze this data without further issues. If we instead look at electronic health records of the same patient but, created at different hospitals, we may find, that their contents and structures vary. In this case, the researcher has to harmonize the data first, before the analysis can be run on the combined data. This is due to the fact, that the various IT systems used to collect that data are not interoperable with each other. Definition: Interoperability Interoperability generally refers to the ability of two or more systems or components to exchange information and use the information that has been exchanged [7].

Whenever data integration shall be achieved, the interoperability of the data sources has to be taken into account.

Interoperability

3.2 Interoperability Interoperability between systems means, that information can not only be exchanged but, is also processable in the same way by both sides. This common understanding or common interpretation of data is an important prerequisite for data integration. This common understanding can be achieved by standardizing the representation of information as data. However, since there are several different definitions for interoperability – even though they do not differ fundamentally, of course – according to Van der Veer et al. [17] there are different aspects of interoperability, which have to be achieved by individual measures.

3.2.1 Syntactic and Structural Interoperability

Syntactic Interoperability means, that the data is stored in the same format, i.e., in a tabular format like CSV or a hierarchically structured document like XML. In such cases, the same syntactic rules and therefore the same structure of the whole dataset are applied and processable by both information exchanging systems. In healthcare a multitude of standards exists, which enable information exchange across IT systems from different vendors, sometimes even across different institutions. Well-known and broadly used are those by the Health Level 7 (HL7) organization: HL7 V2 for Messaging, HL7 Clinical Document Architecture (CDA) for the standardized structuring of clinical documents, and finally HL7 Fast Healthcare Interoperability Resources (FHIR) for the modularized information exchange using common web technologies like HTTP.

3.2.2 Semantic Interoperability

Semantic Interoperability ensures, that the exchanged data can not only be processed but, also be interpreted by the receiving system. This is possible due to a common understanding, i.e., data has the same meaning (interpretation) in both the sending and the receiving systems. This usually requires a common terminology (a common “language”), which has to be applied when information is represented as data. Instead of using mostly human-understandable free text, a coding system demands what code to use. A well-known example of such specifications or semantic standards in healthcare

53

54

Data Integration

is the International Statistical Classification of Diseases and Related Health Problems (ICD-10), which contains unique codes for most diseases, e.g., I21.1 for an acute transmural myocardial infarction of the posterior wall. Other examples are Logical Observation Identifiers Names and Codes (LOINC) which contains codes for laboratory results, e.g., 2345-7 for blood glucose, or the Systematized Nomenclature of Medicine – Clinical Terms (SNOMED-CT) which contains codes for nearly any concept or term in healthcare, e.g., 16154003 for a patient. When these standards are to be applied in specific contexts, e.g., the area of rare diseases, there have to be more specific guidelines about what information is required, socalled Common Data Sets. The Set of common data elements for Rare Diseases Registration, for example, specifies 16 items as well as the to-be-used standards like ICD-10 or SNOMED-CT, which should be collected about any patient suffering from one of the thousands of rare diseases.

3.2.3 Process Interoperability

Process Interoperability ensures that data is available at the right time when another system needs it. When for example, healthcare data of a patient shall be used for research, the information that the patient has given appropriate consent is normally needed. Guidelines like those published by the international initiative Integrating Healthcare Enterprises (IHE) generically specify the relevant actors involved in processes and the transactions taking place. These specifications are independent of specific implementations but, wherever applicable and necessary, they refer to already existing syntactic and semantic standards. Formalized process diagrams using the Business Process Modeling Notation (BPMN) help software developers implement their systems appropriately (see Fig. 3.1).

Figure 3.1 Integrating Healthcare Enterprises (IHE) [8].

Extract, Transform, Load Process (ETL Process)

3.3 Extract, Transform, Load Process (ETL Process) 3.3.1 Introduction The process of data integration is in general characterized as a three-step process including extraction of data from one or multiple sources (extract), combination and transformation of the data in the target format (transform), and loading of the data into the target data storage (load).

3.3.2 Extract

First of all, for every data integration, it has to be clear whether the ETL should be a one-time or a continuous process. If it’s a singletime process, the extraction of the data could be done manually; while in a recurring process the extraction should be automated, e.g., through an application programming interface (API). Either way, the extraction of the data requires specific knowledge of the data source systems such as the data structure and the export format of the data. Source systems can vary greatly, and the data can be extracted for example by using manual export functions, file transfer, database management systems, communication servers, or APIs. Furthermore, the export format of the data may follow a common standard like HL7 v2, but also may be proprietary. Therefore, data integration tools often include standard connectors for file reading, database connection, or API calling as well as connectors that directly interpret standardized message formats [11].

3.3.3 Transform

The main task of the transformation process is to adapt the format of the extracted data to the target system. However, it is not sufficient to establish syntactic interoperability, but rather semantic, too (also see Section 3.2). Thus, the data that should be integrated has to be interpreted by the person (or machine) performing the integration. In addition to converting and adjusting units of measurement, the transformation of the data also includes the conversion of information according to nomenclatures, terminologies, or

55

56

Data Integration

classifications such as SNOMED-CT or LOINC. In general, the more structured the extracted data is, the easier it is to transform the data. However, if the data is unstructured or entire free texts are to be integrated, this is no longer possible without the use of natural language processing (NLP). NLP technologies help the computer to understand the meaning of the text or at least extract relevant information from the text blocks. Since current NLP technologies do not reach 100% sensitivity and specificity and also the extracted information is not accurate, the transformation step is also useful as a data quality evaluation method (see Section 3.5) [2, 9].

3.3.4 Load

Loading the transformed data into the target system is the last step of the integration process. The target system can be any possible data storage like a file-based network share or a data warehouse solution. As mentioned in the first step of the process (extract), the load step also varies depending on the purpose of the data integration. If the integration is a one-time process, there are no further problems to solve. Otherwise, there must be a solution for the update of the data and also traceability of changes. Based on the requirements of the target systems, all data can be periodically overwritten with the new data, or only updates to the existing data are written. In the second case, traceability of changes is possible and the target system can be enabled to show the history of the integrated data.

3.4 Data Provisioning and Data Storage

The data in a clinical setting, especially in hospitals, are usually scattered over different systems with their own data storage solutions. Thus, the data cannot easily be used for purposes than they were originally collected for, so-called secondary use. For instance, the use cases of secondary use include quality assurance and clinical research. One solution to overcome the problem of scattered data can be the setup of a platform that stores the data of a hospital across the systems in the same format as proposed by the openEHR community (https://www.openehr.org/). Since modifying documentation systems in the hospital is very difficult and expensive, a data integration approach is nowadays used

Data Provisioning and Data Storage

to combine data across hospital systems and facilitate secondary use [15].

3.4.1 Data Lake

A data lake is a system for data storage where the data is stored in its natural or raw format. All types of data are conceivable in a data lake, from unstructured data in binary format to semi-structured data in files (CSV, XML JSON, etc.) to structured data in relational databases. Also already transformed or analyzed data or even reports can be stored in a data lake. Since the type and the origin of the data in a data lake can be manifold, the management of the information about the data stored in a data lake (the so-called metadata) is essential to keep track of the stored data and the opportunities for its usage [12].

3.4.2 Data Warehouse

A data warehouse is normally a central repository where integrated data from one or more different data sources is stored. These data warehouses are usually loaded through an ETL process. Especially in the area of healthcare informatics, the terms clinical data repository (CDR) or clinical data warehouse are commonly used. For secondary use, the CDR may contain clinical, administrative, trial, and research data. In CDRs the data is structured and metadata information is added to facilitate statistical analysis and data quality measurement. Data already collected in a structured manner like demographics, diagnosis, procedures, and lab results are integrated into most CDR since the integration is quite easy [15]. In contrast to the data lake, the data in a CDR is already transformed and usually semantically annotated in one coordinated data model to provide directly usable data. The data in a data lake has to be transformed afterward for the specific use case, which saves time beforehand and facilitates a more use case-specific transformation [15].

3.4.3 Data Provisioning “On the Fly”

Another approach to facilitate the use of case-specific data provisioning is an integration of data on the fly. With this solution,

57

58

Data Integration

data is not stored in an additional system but made available directly from the primary and secondary healthcare systems via an ETL pipeline. Thus, the data can be specifically transformed according to the requirements of a use case. This approach may be used to integrate data for one-time research purposes or use cases where the data is requested periodically. However, this approach is not feasible to feed permanently available products, like dashboards.

3.5 Data Quality and Data Reliability 3.5.1 Definition and Relevance

Depending on the intended use of certain data, their suitability for the purpose in question can be assessed or quantified. Such statements describe data quality. They can be expressed from different perspectives and have several dimensions [4]. Definition: Data Quality

Data Quality is relative to formally or informally defined quality expectations such as (1) consumer expectations, (2) specifications, or (3) requirements imposed by the usage of data, e.g., to execute certain tasks [4].

Health data can originate from a wide variety of sources – healthcare provider documentation, medical devices, laboratories, biomedical research, citizens, or patients themselves, etc. – and are also used or reused for various purposes. The medical documentation alone that is generated in a healthcare facility already serves several purposes there: It is used as assistance for healthcare, memory aid, means of communication, justification for treatments, service documentation for billing, and more. Therefore, there is a high relevance for data quality in healthcare, but also a high degree of complexity due to the many different types of data and purposes of use. For example, high data quality in billing requires that all billingrelevant diseases are correctly coded and sorted, according to the respective regulations. However, for medical treatment or clinical research, other quality criteria are relevant regarding the patient’s known diseases.

Data Quality and Data Reliability

Processes of data integration and secondary data use are therefore subject to special considerations regarding data quality. A differentiation must be made in this context between (a) data quality considerations at the source system level (“data at rest” level), (b) data quality criteria within the ETL processes (“data in transit” level, cf. Section 3.3), (c) quality of the data for the intended use after data integration (“data in use” level).

For the description of data and thus also for data quality indicators, metadata is often used and added to the actual data. The interoperable representation of health data in technical standards like HL7 FHIR, which offer specific possibilities (or requirements) for the annotation of health data – with metadata for provenance, semantic codings (cf. Section 3.2.2), or special quality indicators – automatically already increases data quality in healthcare. Therefore, interoperability is an important factor for the quality of health data in general. Particularly in the area of reusing EHR (“electronic health record”) data, there are efforts to standardize data quality indicators conceptually, oriented on the overarching categories of conformity, completeness, and plausibility of data [10]. By linking metadata and validation rules, separate metadata repositories can be used to uniformly assess and label the quality of specific health data [9].

3.5.2 FAIR Principles

In 2016, a consortium of scientists and organizations defined basic principles for handling scientifically used data [9]. These principles can be applied to health data and provide a basis for data quality, e.g., in biomedical research. Data complying with the FAIR principles are:

(a) Findable: identifiable and provided with searchable metadata, (b) Accessible: retrievable using standardized communications protocols, (c) Interoperable: represented in standardized formats and references to other data and metadata,

59

60

Data Integration

(d) Reusable: standardized and described with accurate and relevant attributes as well as provenance.

Projects and implementations that enable data use according to the FAIR principles are gathered under the GO FAIR initiative [5]. Examples from healthcare include the implementation of the “Personal Health Train” as a method for FAIR distributed computing [1], and concepts for standardization and analysis of FAIR data related to rare diseases [16].

3.6 Reintegration of Data

Reintegration of data concerns tasks of incorporating data into systems that were already part of the integration process as a source. An important but complex task in the area of health data usage is the reintegration of newly obtained data, e.g., as part of an electronic health record. This refers in particular to the application of proven scientific findings to patient care as part of a “learning health system” [13]. Reintegration of research results into a clinical routine can take place at different levels:

(a) Process level: implement newly developed processes or procedures (b) Information level: obtain new forms of findings or data about patients, e.g., by calculating a newly determined score (c) Data level: feeding back patient-specific findings, e.g., from analysis results, into the patient record. The practical implementation of data reintegration in particular is currently facing various challenges. Primary documentation systems often have to be handled via manufacturer-specific, proprietary interfaces for data inputs. Interoperable accesses are only gradually established, and often primarily for data queries. If data obtained from research processes are to be further used in the context of a patient’s diagnosis and therapy or trigger such treatment, the highly regulated environment of medical devices is encountered, at least in EU countries [14]. This leads to outstanding research being published in scientific journals and code repositories and seldom used in routine care to improve patient diagnosis and treatment.

References

Reintegration of data into patient-managed records (nowadays often implemented via national platforms with interoperable interfaces) raises ethical issues when potential research findings cannot be discussed in advance with health professionals. For these reasons, incorporating the electronic health record into data reintegration processes remains a challenge today [6]. Nevertheless, it is a key factor in supporting the rapid implementation of research results into routine clinical practice using the tools of digitization and interoperability. Therefore, further research is needed and expected in this area in the future. For other aspects of data integration in the context of its use, see Chapter 2.

References

1. Beyan, O., Choudhury, A., van Soest, J., et al. (2020). Distributed analytics on sensitive medical data: the personal health train. Data Intelligence 2 (1-2), pp. 96–107.

2. Denney, M., Long, D., Armistead, M., Anderson, J., Conway, B. (2016). Validating the extract, transform, load process used to populate a large clinical research database: Retrieved from https://www.ncbi.nlm.nih. gov/pmc/articles/PMC5556907/ (accessed Mar. 30, 2022). 3. Etheredge, L. M. (2007). A rapid-learning health system. Health Aff. 26, pp. w107–w118. 4. Fürber, C. (2015). Data quality, In: Data Quality Management with Semantic Technologies (Springer), pp. 20–55. 5. GO FAIR International Support and Coordination Office (GFISCO), GO FAIR. https://www.go-fair.org (accessed Mar. 30, 2022).

6. Holmes, J. H., Beinlich, J., Boland M. R., et al. (2021). Why is the electronic health record so challenging for research and clinical care? Methods Inf Med. 60, pp. 32–48. 7. IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries, in IEEE Std 610 , pp. 1–217, 18 Jan. 1991, doi: 10.1109/IEEESTD.1991.106963. 8. IHE Integrating the Healthcare Enterprise (2017). https://www. ihe.net/uploadedFiles/Documents/QRPH/IHE_QRPH_WP_IHE4ICP. pdf#37 (accessed Mar. 30, 2022). 9. ISO/IEC 11179, Information Technology – Metadata registries (MDR). http://metadata-standards.org/11179/ (accessed Mar. 30, 2022).

61

62

Data Integration

10. Kahn, M. G., Callahan, T. J., Barnard, J., et al. (2016). A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Washington, DC) 4, 1244.

11. Kimball, R., Caserta, J. (2011). The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data. https://www.wiley.com/enus/The+Data+ Warehouse%C2%A0ETL+Toolkit%3A+Practical+Techniques+for +Extracting%2C+Cleaning%2C+Conforming%2C+and+Delivering+ Data-p-9781118079683 (accessed Mar. 30, 2022). 12. Krause, D. D. (2015). Data lakes and data visualization: an innovative approach to address the challenges of access to health care in Mississippi: https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC4731224/ (accessed Mar. 30, 2022).

13. Lapatas, V., Stefanidakis, M., Jimenez, R. C., Via, A., Schneider, M. V. (2015). Data integration in biological research: an overview. J Biol Res (Thessalon) 22(1), p. 9. doi: 10.1186/s40709-015-0032-5. 14. Ludvigsen, K. R., Nagaraja, S., Daly, A. (2021) The intention; Requirements for software as a medical device in EU law. https://ssrn. com/abstract=3825497.

15. MacKenzie, S., Wyatt, M., Schuff, R., Tenenbaum, J., Anderson, N. (2012). Practices and perspectives on building integrated data repositories: results from a 2010 CTSA survey: Retrieved from https://academic. oup.com/jamia/article/19/e1/e119/728077?login=true (access Mar. 30, 2022). 16. Schaaf, J., Kadioglu, D., Goebel, J., et al. (2018). OSSE goes FAIR – Implementation of the FAIR data principles for an open-source registry for rare diseases. Stud Health Technol Inform. 253, pp. 209–213.

17. Van der Veer, H., Wiles, A. (2008). ETSI White Paper No. 3, Achieving Technical Interoperability – the ETSI Approach: Retrieved from https://www.etsi.org/images/files/ETSIWhitePapers/IOP%20 whitepaper%20Edition%203%20final.pdf (accessed Mar. 30, 2022).

Chapter 4

Data Analysis in Genomic Medicine: Status, Challenges, and Developments

Matthias Schlesner

Biomedical Informatics, Data Mining and Data Analytics, Faculty of Applied Computer Science and Medical Faculty, University of Augsburg, Augsburg, Germany [email protected]

4.1 Introduction Next-generation sequencing and other omics techniques have the potential to fundamentally change medicine because they make it possible to interrogate entire molecular layers of cells like genomes, epigenomes, transcriptomes, or proteomes in a short time and at a reasonable cost. This comprehensive characterization of the molecular state of cells and tissues allows it to study the molecular mechanisms of diseases, improve disease classification and diagnostics and develop novel therapies.

Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

64

Data Analysis in Genomic Medicine: Status, Challenges, and Developments

While the first omics-based analyses which have been translated to clinics were microarray-based expression analysis [e.g., 13, 15, 44], in recent years genome analysis gained particular importance. Compared to traditional sequencing techniques, next-generation sequencing (NGS) offers a tremendous increase in throughput combined with a drastic decrease in cost per sequenced base. This makes it possible to perform whole-genome sequencing (WGS) for large cohorts of samples and enables the use of broad sequencing approaches like WES and even WGS as diagnostic tools in clinical settings. Genome sequencing has changed genetic testing from stepwise investigation of a few suspected candidate genes to parallel analysis of large panels of genes, complete exomes, or even complete genomes. With this, it can significantly reduce the time to reach a diagnosis for patients with genetic disorders, which in particular for patients with non-specific phenotypic symptoms often took several years. Several studies have demonstrated the superiority of genome sequencing for molecular diagnostics of genetic disorders over traditional molecular diagnostics methods [e.g., 10, 33, 43, 76]. Similarly, in basic and translational cancer research, large-scale genome sequencing has been quickly adopted as genome alterations play a central role in malignant transformation [145]. Since the first publication of a cancer genome in 2008 [78], NGS formed the basis for substantial advancements in the understanding of the mechanisms underlying the initiation and progression of several cancers [14]. Also for therapeutic decision-making, broad genomic (and transcriptomic) characterizations have demonstrated benefits for the patients and are hence essential parts of precision oncology workflows [53, 157].

4.2 Genome Sequencing in Clinical Applications

In a clinical setting, one way is to sequence genomes in total (WGS). More often, sequencing is restricted to the entirety of proteincoding regions (whole-exome sequencing, WES) or selected genes or genomic regions (panel sequencing). Only WGS covers the

Genome Sequencing in Clinical Applications

entire1 genome, including exons, introns, and intergenic regions, reveals all classes of alterations, and thus provides a comprehensive characterization of the genomic variation. Furthermore, WGS information results in a much higher power to identify genomic patterns like mutational signatures, which reveal information about the mutational processes which have acted on a genome, for example, tumor cells. The identification of mutational signatures and similar patterns might even have therapeutic implications, for example in the case of “BRCAness” which indicates the potential sensitivity of the tumor to PARP inhibitors [26]. A disadvantage of WGS is the higher cost of data generation, data analysis, and longterm storage. This is particularly relevant if very high sequencing coverage is needed to detect variants with low allele frequencies, e.g., to analyze samples with low tumor cell content or to identify mosaic mutations. In such settings, exome sequencing or gene panel sequencing might be a more suitable choice. The current de facto standard for human genome sequencing2 experiments in translational research and clinical applications are paired-end short read sequencing on the Illumina platform. These sequencers produce short reads, typically of 100–250 bp length, which are then mapped to the human reference genome. The short reads of Illumina and related sequencing techniques impose several limitations. With short reads, it is impossible to analyze genetic variation in large repetitive regions, although variations in these regions have been shown to be relevant for a number of diseases [159]. Furthermore, short reads make it challenging to resolve complex structural variations. It has been estimated that in current short read-based WGS datasets only a fraction of the structural variants is detected [16, 173]. As a third limitation, phasing of alleles to resolve haplotypes is not possible directly from short read data. To some extent, this can be compensated by sequencing the genomes of the parents or even computationally through haplotype imputation.

1 Where entire means those parts of the genome which are accessible with current techniques. Highly repetitive parts of the genome are not accessible with current short-read sequencing techniques and are also not included in the commonly used versions of the human reference genome. 2 Sequencing the genome of individuals for which a reference genome is available is sometimes referred to as “resequencing,” to distinguish it from de novo sequencing of organisms where no reference genome is available. Throughout this article, the term “sequencing” is used for the analysis of human genomic sequences.

65

66

Data Analysis in Genomic Medicine: Status, Challenges, and Developments

However, sequencing the parents’ genomes is expensive and often not even possible, and imputation has significant uncertainties and limitations, especially in predicting rare haplotypes. Furthermore, none of these methods can resolve the allelic status of de novo mutations or somatic mutations if they are not in the close neighborhood of haplotype-resolved variants. In recent years, new sequencing technologies developed by Pacific Biosciences (PacBio) and Oxford Nanopore which produce longer reads are gaining increasing importance in medical use cases [51, 110, 116, 132]. Longer reads are particularly useful to analyze repetitive regions, perform de novo assemblies, perform haplotype phasing and decipher complex structural variations. Furthermore, in contrast to Illumina sequencing, Oxford Nanopore and PacBio sequencing both allow the direct detection of modified bases like 5-methyl cytosine and hence can be used to study genetic and epigenetic variation in one run [156, 163]. Oxford Nanopore sequencing enables real-time interference with the sequencing process [91]. Sequencing of a certain DNA molecule can be stopped early in a read, and a new read from another DNA molecule can be started. This selective sequencing allows enriching sequencing data over certain regions of interest, for example, to analyze a gene panel without the need for specialized library preparation [107, 124], which can for example be used to develop flexible molecular diagnostic workflows [123]. Until recently, reliable detection was only possible for genomic variants which are present in a large number of cells and reach a certain allele frequency in a mixed tissue or sample. Mutations that are only present in single cells or minor clones could not be analyzed on a genome-wide scale. This has severely limited the possibilities to analyze somatic mutagenesis in non-clonal tissues. Accordingly, somatic mutations have almost exclusively been studied in cancer, although it has long been acknowledged that somatic mutations play crucial roles in other processes, for example in aging and neurodegenerative diseases [67]. Barcoding of individual DNA molecules allows it to sequence individual DNA molecules multiple times and reduce error rates by building single-molecule consensus data [133]. This approach has been developed further into protocols that sequence copies of both strands of a DNA molecule [66, 135] and build a duplex consensus, which results in a theoretical error

Genome Sequencing in Clinical Applications

rate of less than 10-9 errors per base pair. However, in practice, the error rates are higher because some processes cause artifacts that are not independent on both strands [31]. An improved protocol, called Nanorate sequencing (NanoSeq) addresses these issues and reaches error rates lower than 5 × 10−9 errors per bp [1]. With such a low error rate NanoSeq allows for the detection of mutations in single DNA molecules and hence is suited to study somatic mutations in non-clonal tissues. A related protocol, single-molecule mutation sequencing (SMM-seq), has been reported to enable even lower error rates [100]. Alternatively, also single-cell WGS can be used to study somatic mutations in primary tissues [90, 170]. The ability to determine somatic mutations in non-clonal tissues has already led to a number of new insights, e.g., a distinct pattern of somatic mutations in neurons affected by Alzheimer’s disease [108].

4.2.1 Read Mapping to the Human Reference Genome

Virtually all analyses in the field of human genetics and genomics depend on mapping of the sequencing reads to the human reference genome sequence. A very popular tool for short read alignment is BWA, which uses backward search with the Burrows-Wheeler transformation [86] to achieve high alignment accuracy while keeping a small memory footprint also for large genomes [80]. The first two algorithms of the BWA family, BWA-backtrack and BWA-SW [79] have now largely been replaced by BWA MEM [81], which is suited to align sequences in the range from 70 bp to a few megabases against reference genomes. Already in 2003, the Human Genome Project proclaimed that “an accurate and complete human genome sequence was finished and made available to scientists and researchers” [57]. However, already before it has been noted that it is debatable what “finished” means in the context of eukaryotic genomes, as certain large regions consisting of repetitive sequences had to be excluded as they were not accessible to the sequencing technologies [96]. The first “finished” version of the human genome (referred to as NCBI Build 34 or hg16) and its quickly following successor (NCBI Build 34/hg17, May 2004) covered around 99% of the gene-containing (euchromatic) regions of the human genome and consisted of around 2.851 billion nucleotides [59]. Since then, three further

67

68

Data Analysis in Genomic Medicine: Status, Challenges, and Developments

coordinate-changing updates have followed, with the latest version, GRCh383 having been released in 2013 [136]. The human reference genome as well as reference genomes of some model organisms are maintained by the Genome Reference Consortium (GRC, https:// www.ncbi.nlm.nih.gov/grc). The GRC is regularly releasing patches for the reference genome assembly, which correct errors and add sequences but do not change the genomic coordinates. Because the representation of a reference genome as a single flat string is increasingly imposing limits on different analyses (see below), the GRC has announced that the release of the next coordinate-changing update of the human reference genome (GRCh39) is indefinitely postponed, while new models to represent the human genomic reference are being evaluated [41]. In the community, the adoption of new reference genome versions is a slow process. For most use cases, annotations, databases, and similar resources are required which have to match the coordinate system of the reference genome. These resources either have to be recalculated, or a coordinate transformation (lift-over, [85]) needs to be performed. A lift-over is always a less-than-ideal solution, as with this approach regions that are novel in the new genome version cannot be annotated. Furthermore, not only missing annotations can delay the adoption of new reference genome versions. Updating pipelines, tools, and analysis procedures require significant effort, and for consistency reasons changes in ongoing projects must be avoided or require a complete reanalysis [120]. Therefore, labs that run multiple projects in parallel can usually not switch reference genome versions at one single time point and rather have to use two different versions in parallel. For these reasons, even now, around nine years after the initial release of GRCh38, several studies still use the prior version GRCh37/ hg19 [82]. To reduce the number of incorrectly mapped reads, it is important that the used reference genome is as complete as possible [21, 42]. Sequencing reads which are derived from sequences that are not represented in the reference have a high probability to be erroneously mapped to another place where they can lead to false variant

3 The official designation of the current reference version is GRCh38 (Genome Reference Consortium Human Reference 38). Sometimes, in particular in the context of the UCSC Genome Browser [75], also the name hg38 is used. The previous version was GRCh37 or hg19, respectively.

Genome Sequencing in Clinical Applications

calls. This means that all DNA sequences which are present in the sequenced library should be represented in the reference sequence. Therefore, the human reference genome consists of several different components [57]. First, it contains the assembled chromosomes 1-22, chrX and chrY4 as well as the mitochondrial chromosome chrM. Second, it contains contigs of unlocalized sequences. These sequences are known to belong to a certain chromosome, but with unknown order or orientation. In GRCh38 their identifiers carry the suffix ‘_random’. Third, it contains unplaced sequences for which it is not known from which chromosome they originate. Unplaced sequences have identifiers with the prefix ‘chrU_’. In addition, versions of the human reference genome commonly used for sequencing read mapping contains a contig with the genome sequence of the Epstein-Barr virus (EBV). This contig captures reads from EBV sequences which are present in many human samples, either due to the presence of endogenous EBV in B cells or due to the immortalization of human cells by EBV transformation. To further reduce false alignments, especially in regions with cryptic segmental duplications, the 1000 Genomes Project started in its second phase to include an additional contig into the reference genome, which was at that time based on GRCh37. This contig includes additional unlocalized sequences which consisted mainly of sequenced clones that were discarded by the Human Genome Project and unlocalized scaffolds from the HuRef assembly [42]. These sequences, termed “decoy sequences”, siphon reads which originate from large repetitive regions like the centromeres. A substantial fraction of this “decoy” has been localized in GRCh38: around 70% of reads from the 1000 Genomes project which originally aligned to the decoy sequences when using genome version GRCh37 do now align to regular contigs in GRCh38 [21]. Nevertheless, also for GRCh38 a decoy contig exists (hs38d1) and is often included in the reference genome used for the read alignment. Some parts of the human genome exhibit a very high diversity within the population, which cannot be adequately represented

4 It is common practice to hard-mask the pseudoautosomal regions (PAR) on chrY (i.e., replace their sequence with Ns). The sequence of the PARs in the reference genome is identical on chrX and chrY, and therefore read mapping to these regions would be ambiguous. Due to masking on chrY, reads of the PAR regions are solely mapped to chrX.

69

70

Data Analysis in Genomic Medicine: Status, Challenges, and Developments

on a single linear reference, making alignment to these regions difficult and error-prone. To improve read mapping in these regions, contigs representing alternate haplotypes (ALT contigs) have been introduced into the genome version GRCh38, which represents common variation in highly variable regions like the loci of the human leukocyte antigen (HLA). While some subsequences of the ALT contigs are highly diverged from the primary assembly, most of the ALT contigs are very similar to the primary assembly or to each other. This means that naïve mapping procedures would result in equally good mappings at multiple places. Such mappings are usually considered ambiguous and excluded from further analyses, which would lead to a loss of variant calls in regions for which ALT contigs exist. Therefore, alignment and variant calling using a reference genome with ALT contigs requires adapted processing pipelines. Most current aligners (like all recent versions of bwa mem) are “ALTaware” and can hence be used with a reference sequence containing ALT contigs. Variant calling, however, still requires carefully chosen strategies, especially when variants on the ALT contigs should be considered. Among others, bwa mem alignments need to be postprocessed with a separate script [83]. A tutorial for the popular Genome Analysis Toolkit (GATK; [40]) provides possible workflows for different scenarios [55].

4.2.1.1 Toward a complete human reference genome

The current human reference genome is in two different aspects far from being complete: “difficult” regions are not yet resolved, and human genetic variation is hardly represented. Despite the improvements and refinements over the last 20 years, GRCh38 still contains 151 megabase pairs of unknown sequences distributed throughout the genome [117]. These consist of large completely unresolved regions like the p-arms of acrocentric chromosomes, regions that are represented with artificial models like the centromeric regions, and the unplaced and unlocalized sequences which could not yet be integrated into the primary assembly. Furthermore, there are still errors and artifacts in GRCh38, which altogether can lead to biases in genomic analyses and to the inability to detect medically relevant variants [3, 106, 159]. Recently, the Telomere-to-Telomere (T2T) Consortium presented T2T-CHM13, a 3.055 billion-base pair sequence of a complete haploid human genome

Genome Sequencing in Clinical Applications

which contains gapless assemblies for all human chromosomes except Y [117]. To achieve this, the T2T Consortium has sequenced a complete hydatidiform mole [56]. Complete hydatidiform moles arise either from fertilization of an enucleated egg by sperm or from loss of the maternal complement after fertilization and hence contain only a single set of chromosomes. Assembling a genome from such effectively haploid cells avoids the technical complexity which arises from diploidy. It has been shown that T2T-CHM13 as a reference genome improves read mapping and variant calling after WGS with short read and with long-read technologies [3]. However, the transition to the T2T-CHM13 reference will require adaptation of tools and pipelines as well as databases and similar resources, and for leveraging the full potential of the new reference also improvements in the sequencing technologies are required. The second major limitation of the current reference genome is the inappropriate representation of human genetic variation. Currently, it is common to use a single, monoploid reference structure. GRCh38 is a composite genome that has been assembled from genomic sequences of several different anonymous individuals (https://www.ncbi.nlm.nih.gov/grc/help/faq/#human-referencegenome-individuals) [45]. Around 93% of the primary assembly originates from 11 individuals, with one of these being highly overrepresented, contributing to 70% of the primary assembly [47]. Every sequenced human genome has millions of single nucleotide variants and thousands of structural variants compared to the reference genome [8, 146]. This results in reference bias, and a higher probability to miss alignments or report wrong alignments for reads which contain non-reference sequences, which in turn can cause false negative or false positive variant calls [39, 99]. This is even more of a problem when genomes of individuals with non-European ancestry are sequenced, which exhibit a higher divergence from the current reference genome [105, 139]. Together with the underrepresentation of sequences from individuals with non-European ancestry in genomic variant databases, this leads to a significantly reduced power for risk prediction and diagnosis of genetic diseases in populations of non-European ancestry [98, 139, 140]. Furthermore, since the reference genome is assembled from the genomes of a small number of normal people, it contains a large number of rare variants and also variants that are associated with an

71

72

Data Analysis in Genomic Medicine: Status, Challenges, and Developments

increased risk for diseases [20]. This results in missing potentially disease-associated variation and thus can hamper accurate risk predictions. These issues can be solved when a pan-genome, the representation of all genomic content in a certain species or clade, is used as a reference genome [105, 138]. Pan-genomes, which can be represented as reference genome graphs [23] can improve genotyping accuracy [31, 68, 127]. A recent study has used a haplotype-resolved pan-genome reference to genotype genetic variation and found improvements in particular for large insertions and in repetitive regions [30]. A human pan-genome reference is currently being constructed by the Human Pangenome Project [161], which should provide an accurate and diverse representation of human genomic variation. However, pan-genome-based workflows are computationally much more demanding and not yet as mature as workflows using a linear reference genome [30], hampering their broad adoption in routine analysis workflows.

4.2.2 Variant Detection

Four classes of variants can be identified from NGS data: Single nucleotide variants (SNVs), small insertions and deletions (indels), structural variants (SVs), and copy number aberrations (CNAs) [104]. While the workflows for variant calling have evolved considerably since NGS became broadly applicable more than ten years ago, variant calling is still an area of active research, and visual manual review and validation with orthogonal techniques are common practices in clinical applications [70].

4.2.2.1 Germline variant calling

Germline variants are variants that an individuum has inherited or which arose de novo in the very early stages of embryonic development, and which are hence present in (almost) all cells of the body. The analysis of germline variants is an important tool to diagnose genetic disorders or identify causative genes. While for common genetic disorders, where disease-relevant genes are known a priori, targeted approaches like amplicon or gene panel sequencing are often sufficient to identify disease variants, especially for rare diseases whole exome or WGS might be necessary to detect disease-

Genome Sequencing in Clinical Applications

relevant genomic variants [143]. A number of studies have found a higher diagnostic potential for WGS compared to WES, e.g., [10, 69, 144], and initiatives like the Medical Genome Initiative advocate for implementing WGS as a first-tier test for patients with rare genetic disorders [97]. The analysis of germline variants can also be important for tumor patients as pathogenic variants in cancer susceptibility genes can be associated with the heredity of cancer risk, and certain germline variants can have therapeutic implications. For example, pathogenic variants in the BRCA1 or BRCA2 genes are often associated with the sensitivity of the tumor to PARP inhibitors [92]. Germline variants in TP53 (Li-Fraumeni syndrome) result in an elevated risk for radiation-induced cancers. Hence for these patients, the use of radiotherapy and diagnostic techniques involving ionizing radiation should be kept to a minimum to prevent the development of secondary tumors. A number of benchmarks have been performed to evaluate the performance of germline variant calling pipelines [19, 119, 172], and a working group of the Global Alliance for Genomics and Health (GA4GH) has even published best practices for performing such benchmarks [71]. While the identification of small variants (SNVs and small insertions and deletions) is robust and reliable in “easy” genomic regions, it remains challenging in repetitive or polymorphic regions of the human genome [35]. Since a number of medically relevant genes overlap with such challenging regions, the Genome in a Bottle (GIAB) consortium has released a specific benchmark dataset based on a haplotype-resolved whole genome assembly that covers medically relevant genes overlapping challenging genomic regions [159]. One main finding of this study was that false duplications in the reference assemblies GRCh37 and GRCh38 are a major cause of missed variants in medically relevant genes, and masking these could drastically improve variant recall. Copy number variants and SVs are generally more difficult to detect than small variants. For comprehensive detection of these large variants, WGS is much more powerful than targeted sequencing strategies [88]. However, CNV calling from whole-exome sequencing data is commonly done, and several tools have been developed for this task. Recent benchmarks have detected low concordance between the results produced by different tools, emphasizing that CNV calling from WES data is challenging and not necessarily reliable [38, 46]. Some tools

73

74

Data Analysis in Genomic Medicine: Status, Challenges, and Developments

increase their calling power by using not only the targeted reads but also the nonspecifically captured off-target reads produced during whole-exome sequencing [73, 148]. The detection of structural variants from short read WGS data is even more challenging and currently the accuracy and precision of SV detection are low. A recent review discusses the current developments in SV detection for clinical diagnosis and proposes a roadmap toward accurate and reproducible SV detection for clinical applications [88].

4.2.2.2 Somatic variant calling in cancer

In cancer sequencing studies, it is necessary to distinguish between germline variants and somatic variants, which were acquired later during a lifetime and affect only a single cell or a clone of cells. Tumors are clonal expansions of a single cell, and hence all variants that this cell has acquired before or in the early stages of tumor development appear as somatic variants in all cells of the tumor.5 Those variants are called clonal variants. Somatic variants which are acquired later during tumor evolution are only present in a subset of tumor cells and are commonly referred to as subclonal variants. To discriminate between germline and somatic variants a matched normal tissue from the same patient is usually sequenced. Due to good accessibility most often white blood cells are chosen as matched normal tissue, but any non-clonal tissue from the same patient could be used. It is important that the matched normal sample is not contaminated with tumor cells or tumor DNA, which would lead to a misclassification of somatic variants as germline. Dedicated software packages can be used to detect such a tumor in normal contamination and reclassify wrongly assigned variants [121, 153]. If no matched normal tissue is available, population variation databases like dbSNP [141] or more recent resources like gnomAD [64] can be used to identify likely germline variants [154]. With such an approach, however, complete removal of germline variants from the somatic set is not possible since every individual has private germline variants which are missing from the databases. It should be noted that certain databases like dbSNP also contain well-studied somatic variants, and hence filtering against these

5 With the exception of variants that are lost again, when the affected genomic region is lost later during tumor evolution (or, in exceptional cases, which are reverted in a second mutational event).

Genome Sequencing in Clinical Applications

databases without additional selection criteria should be avoided [61]. In the case of deep sequencing (usually >100×) of tumors with a relatively low tumor cell content ( 40 years for sensors such as Landsat [17] and AVHRR [8], or 20+ years for Moderate Resolution Imaging Spectrometer (MODIS)]. Additionally, many recently developed RS sensors allow for a level of spatial detail and temporal repetition that is unprecedented. Today, satellite-based RS can provide global coverage at a high spatial ( 2.5 [3]. However, the absolute threshold SUVmax > 2.5 depends strongly on the reconstruction method (e.g., the postfilter/post-filter applied) and should therefore be treated with caution. In a retrospective study with 534 subjects, it was shown that – for the classification task to distinguish primary lung lesions from metastatic lung lesions – radiomic features generated from PET images had a significantly better AUROC of 0.91 than radiomic features generated from CT images with an AUROC of 0.70 [9]. In contrast to radiomics, CNNs can be applied to images without pre-processing such as segmentation. CNNs have been successfully applied to the classification task to discriminate suspicious from non-suspicious lung lesions [17]. It has been shown that [18F]FDGPET is a solid image modality for the application of CNNs in this task. However, additional use of CT, maximum intensity projection (MIP) images, and atlas position of the respective lesion as input to the CNN can increase the AUROC slightly from 0.97 to 0.99. Furthermore, the anatomic location of the tumor was predicted with an accuracy of 0.96 for determining the body part, 0.87 for determining the organ, and 0.81 for determining the sub-region of the organ. Tau et al. [19] analyzed 264 patients with a follow-up of 6–43 months and found that a CNN is able to predict positive lymph nodes with an AUROC of 0.80 and distant metastasis at the end of the followup with an AUROC of 0.65. Whole-body total lesion glycolysis (TLG), the summation of individual tumor volumes multiplied by their mean standard uptake value, has been found to be extremely useful for better assessing the prognosis of NSCLC, as it correlates strongly with overall survival. However, it requires the segmentation of all lesions, a task that is very time-consuming. In [4], it was shown that

References

a CNN with a u-net structure is capable of automatically segmenting the lesions resulting in an automated whole-body TLG evaluation that also strongly correlates with the patient’s overall survival.

11.4 Outlook

We have seen how deep neural networks can be successfully applied to specific tasks in molecular imaging and how they can improve diagnostic accuracy, efficiency, and objectivity. The next step to improve machine learning predictions in molecular imaging is to build well-structured, well-annotated image databases, ideally with patient follow-up and outcome, which can be used to train machine learning models. These models could further improve the prognosis, diagnosis, treatment, and prevention of human diseases.

References

1. Alzheimer’s Association (2021). Alzheimer’s disease facts and figures. Alzheimers Dement: 17(3).

2. Balagurunathan, Y. et al. (2019). Quantitative imaging features improve discrimination of malignancy in pulmonary nodules. Scientific Reports 9(1): 1–14. 3. Bianconi, F. et al. (2020). PET/CT radiomics in lung cancer: An overview. Applied Sciences 10(5): 1718.

4. Borrelli, P. et al. (2022). Freely available convolutional neural networkbased quantification of PET/CT lesions is associated with survival in patients with lung cancer. EJNMMI Physics 9(1): 1–10.

5. Chen, S. et al. (2017). Diagnostic classification of solitary pulmonary nodules using dual time 18F-FDG PET/CT image texture features in granuloma-endemic regions. Scientific Reports 7(1): 1–8. 6. Choi, H. et al. (2020). Cognitive signature of brain FDG PET based on deep learning: domain transfer from Alzheimer’s disease to Parkinson’s disease. European Journal of Nuclear Medicine and Molecular Imaging 47(2): 403–412.

7. Cuaron, J., Dunphy, M. and Rimner, A. (2013). Role of FDG-PET scans in staging, response assessment, and follow-up care for non-small cell lung cancer. Frontiers in Oncology 2: 208.

251

252

Neural Networks in Molecular Imaging

8. Khagi, B., and Kwon, G. R. (2020). 3D CNN design for the classification of Alzheimer’s disease using brain MRI and PET. IEEE Access 8: 217830217847. 9. Kirienko, M. et al. (2018). Ability of FDG PET and CT radiomics features to differentiate between primary and metastatic lung lesions. European Journal of Nuclear Medicine and Molecular Imaging 45(10): 1649–1660.

10. Lu, D. et al. and Alzheimer’s Disease Neuroimaging Initiative (2018). Multimodal and multiscale deep neural networks for the early diagnosis of Alzheimer’s disease using structural MR and FDG-PET images. Science Reports 8: 5697. 11. McCulloch, W. S., Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics 5(4): 115–133. 12. Miwa, K. et al. (2014). FDG uptake heterogeneity evaluated by fractal analysis improves the differential diagnosis of pulmonary nodules. European Journal of Radiology 83(4): 715–719.

13. O’Malley, J. P., and Ziessman, H. A. (2020). Nuclear Medicine and Molecular Imaging: The Requisites, e-book. Elsevier Health Sciences. 14. Rosenblatt, F. (1957). The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory.

15. Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1985). Learning internal representations by error propagation. California Univ San Diego La Jolla Inst for Cognitive Science. 16. Shen, T. et al. (2019). Predicting Alzheimer disease from mild cognitive impairment with a deep belief network based on 18F-FDG-PET images. Molecular Imaging 18: 1536012119877285. 17. Sibille, L. et al. (2020). 18F-FDG PET/CT uptake classification in lymphoma and lung cancer by using deep convolutional neural networks. Radiology 294(2): 445–452.

18. Skourt, B. A., El Hassani, A. and Majda, A. (2018). Lung CT image segmentation using deep neural networks. Procedia Computer Science 127: 109–113. 19. Tau, N. et al. (2020). Convolutional neural networks in predicting nodal and distant metastatic potential of newly diagnosed non–small cell lung cancer on FDG PET images. American Journal of Roentgenology 215(1): 192–197.

References

20. Wu, W. et al. (2019). Comparison of prediction models with radiological semantic features and radiomics in lung cancer diagnosis of the pulmonary nodules: a case-control study. European Radiology 29(11): 6100–6108. 21. Zhong, Z. et al. (2018). 3D fully convolutional networks for cosegmentation of tumors on PET-CT images. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

253

Chapter 12

Precision Oncology: Molecular Diversification of Tumor Patients

Sebastian Dintner

General Pathology and Molecular Diagnostics, Medical Faculty, University of Augsburg, Augsburg, Germany [email protected]

Outcomes for cancer patients vary greatly even within the same tumor type, and characterization of molecular subtypes of cancer holds important promise for improving prognosis and personalized treatment. Physicians have always recognized that every patient is unique, and physicians have always tried to tailor their treatments as best they can to individuals. In this chapter, we review current developments in precision oncology, consider the implementation of comprehensive characterization of tumor patients, and discuss the necessary pooling of data to individualize the treatment of today’s tumor patients. Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

256

Precision Oncology: Molecular Diversification of Tumor Patients

12.1 Introduction Cancer is a heterogeneous disease that evolves through many pathways, involving changes in the activity of multiple oncogenes and tumor suppressor genes. The basis for such changes is the vast number and diversity of somatic alterations that produce complex molecular and cellular phenotypes, influencing each tumor’s behavior and response to treatment [15]. Due to the diversity of mutations and molecular mechanisms, outcomes vary greatly. It is therefore important to identify cancer subtypes based on common molecular features and correlate those with outcomes [31, 41]. This will lead to an improved understanding of the pathways by which cancer commonly evolves, as well as better prognosis and personalized treatment. Efforts to distinguish subtypes are complicated by the many kinds of genomic changes that contribute to cancer. While molecular changes are often used to discover subtypes, analysis of a single data type does not typically capture the full complexity of a tumor genome and its molecular phenotypes. For example, point mutations that alter the function of the gene product and point mutations in two different genes may have the same downstream effect, but besides that, a copy number change may be also relevant. Only looking at a point mutation ignores a copy number change, which gets relevant by causing a gene expression change. Therefore, comprehensive molecular subtyping requires the integration of multiple data types. Precision oncology or personalized oncology aims to offer individualized treatment to each cancer patient by applying a comprehensive molecular, cellular, and functional analysis of tumors. We now know that biological properties differ greatly, not only from cancer to cancer but also from patient to patient [17]. One of the core competencies of personalized medicine is to investigate individual tumors in detail and to provide access to customized, personalized treatment to patients. In addition to the identification of new targeted drugs and the definition of rational drug combinations [19, 24], the field also supports the precise use of previously established forms of treatment such as conventional chemotherapy, immunotherapy, radiation therapy, and surgical procedures.

Introduction

The molecular, cellular, and functional data collected in cancer centers at university hospitals and research centers are included in the treatment of patients but are also made available for applied basic research. The experimental work of laboratory scientists leads to a better understanding of different cancers and, in turn, paves the way for new pathogenetically oriented therapeutic strategies.

Figure 12.1 A full OMICS readout. From the genome onwards, gathering all molecular data from cancer cells and comparing it with those from healthy cells can provide researchers and clinicians with valuable insights.

In recent years, high-throughput sequencing technologies provide unprecedented opportunities to depict cancer samples at multiple molecular levels [23]. The integration and analysis of these multi-omics datasets is a crucial and critical step to gaining actionable knowledge in a precision medicine framework. Integrated approaches allow for comprehensive views of genetic, biochemical, metabolic, proteomic, and epigenetic processes underlying disease that, otherwise, could not be fully investigated by

257

258

Precision Oncology: Molecular Diversification of Tumor Patients

single-omics approaches (see Fig. 12.1). Computational multi-omics approaches are based on machine learning techniques and typically aim at classifying patients into cancer subtypes [40], designed for biomarker discovery and drug repurposing [10]. Analysis of datasets generated by multi-omics sequencing requires the development of computational approaches spanning from data integration [52], statistical methods, and artificial intelligence systems to gain actionable knowledge from data. Here we present a descriptive overview of recent multi-omics approaches in oncology, which summarizes the current state-of-art in multi-omics and data analysis, relevant topics in terms of machine learning approaches, and aims of each survey, such as disease subtyping, or patient similarity. We provide an overview of each methodology group, while then focusing on available analysis tools. Key points











∑ The ability to characterize cancers into finer and finer subgroups has been aided enormously by the implementation of next-generation sequencing (NGS), which has permitted the rapid and inexpensive classification of tumors by genomic subtyping. ∑ Increasing numbers of patients are undergoing providerinitiated NGS profiling of their tumors, and new initiatives are being founded to facilitate patient-centered testing. ∑ Oncologists should be familiar with the technical aspects of NGS to facilitate selecting the most appropriate and costeffective testing platform. ∑ Considerations for molecular testing include which tissue type to utilize the timing of profiling in the disease course, the extent of the panel to order, and the degree of clinical annotation reported.

∑ Actionable biomarkers of non-small cell lung cancer make this disease a paradigm for precision oncology at diagnosis of advanced disease, during therapy, and at the time of progression. ∑ Interpretation of molecular data to facilitate best practice remains a challenge; clinical trial participation and sharing of linked molecular/clinical data sets are strongly encouraged.

Genomics

12.2 Genomics Humans display significant phenotypic complexity and diversity beyond other living organisms [57]. In addition to inherited and somatic genomic alterations, diversity is generated through transcriptional and epigenetic regulation, with further contributions by noncoding elements including microRNA, long noncoding RNA, and posttranslational modifications (Fig. 12.2). Individual risk of developing certain cancers can be determined either by heritable mutations with high penetrance or more subtle variants that can contribute to the acquisition of somatic alterations, all of which result in cellular traits that facilitate carcinogenesis. Ensuing cancer hallmarks, such as proliferation, invasion, metastasis, and traits related to treatment such as acquired drug resistance, are realized through additional somatic mutations. Most cancers are associated with a high mutation burden [11], some because of environmental exposure (e.g., smoking in lung cancer) [2] and others as a result of an intrinsic germline susceptibility (e.g., microsatellite instability in hereditary nonpolyposis colorectal cancer) [62] representing a confluence of both germline and somatic genomics. DNA mutations are hallmarks of cancer. Some mutations termed drivers, give tumors a selective growth advantage and promote cancer development. Mutations in the BRCA gene are one example [21]. The knowledge of these drivers can guide treatment decisions. Next, there are passenger mutations, which seem to be important but do not directly drive cancerous growth, as well as other molecular changes at the RNA and protein levels. These mutations all play a role in the deregulation of cell metabolism, stimulation of cell growth, and promotion of metastasis, but their exact contributions are largely unclear. In recent years, NGS has become better at identifying both the mutations in DNA and the changes in gene expression and posttranslational modifications. Here molecular genetic analysis can accurately identify specific actionable genomic variants linked to correspondingly effective molecularly targeted therapy, often with dramatic inversion of tumor growth and survival. Examples

259

260

Precision Oncology: Molecular Diversification of Tumor Patients

include imatinib for c-KIT in gastrointestinal stromal tumor (GIST), vemurafenib for BRAFVal600Glu melanoma, and lorlatinib for ALKrearranged non-small cell lung cancer (NSCLC) [9, 12, 48]. Patient stratification based on genomic biomarker diversification has become an integral component of modern clinical cancer diagnosis and personalized treatment (Table 12.1).

Figure 12.2 Phenotypic diversity is achieved by different layers of omics in humans.

Genomics

Table 12.1 Summary of important biomarkers and their analysis capabilities in precision oncology Type of Biomarker/Methodology

Literature

Examples

[22, 32, 35, 40, 44, 51]

EGFR, ALK, RET, ROS1, NTRK, BRAF, PIK3CA, MET, ERBB2

[8, 30, 47, 49, 50, 54]

ALK, RET, ROS1, NTRK, MET, FGFR

Short nucleotide variants The simplest type of alteration, single-nucleotide variant (SNV), is straightforward and readily measured by NGS. Small insertions and deletions (indels) A; p.Val600Glu. In this unwieldy form, the actual gene reference (spliced mRNA: “NM_004333”) is provided, the “c.” portion indicates the nucleotide change, and the “p.” portion provides the inferred protein change. This representation is unambiguous and provides significant advantages, but it is clearly not easily readable. Move from simple point mutations like BRAF V600E to more complex genomic changes, such as fusions or copy number changes, and the problem becomes confusing. Therefore, it is important to follow the international nomenclature to guarantee comparability in data acquisition [43]. Once a genomic abnormality has been identified, the meaning of the alteration needs to be assessed. Unfortunately, the classification of genomic abnormalities may vary across laboratories and knowledge bases. In contrast to the well-described and accepted categories of pathogenicity for germline variants (a benign, likely benign, variant of uncertain significance (VUS), likely pathogenic, or pathogenic), there is no uniform consensus on how to categorize somatic mutations. There was the simple binary approach applied: is a mutation actionable or nonactionable? This overly broad and subjective concept has evolved, and several systems have recently been proposed to categorize the interpretations of somatic variants [53]. Classifications can also change over time as VUSs are reclassified due to a more useful category or new treatment availabilities. Without additional germline analysis, it cannot be known which variants identified in tumor sequencing are inherited.

Conclusion

Even the most highly specialized oncologist has a hard time keeping up with the latest interpretation of every genomic variant and must constantly go back to primary sources to update their information. This problem is even more acute for those practicing oncologists who are not genomic specialists and may not have the time to keep up with the genomic information. This will remain problematic for the future as omic data accumulation outpaces our ability to transfer it into clinical practice. Oncologists also require time to review and interpret complex test results. A good division of responsibilities and interdisciplinary collaboration between oncologists, pathologists, molecular biologists, and bioinformaticians based on a very wellstructured and comprehensive data set is the essential fundament to meet the care of tumor patients in personalized medicine in the future.

12.6 Conclusion

Multi-omics data from cancer tissue and liquids can help explain their response to treatment and guide the development of therapies that will be more effective for patients. Mutation-associated cancer antigens (neoantigens) are informing the development of personalized anticancer vaccines and cellular immunotherapies, but their efficacy may not just depend on tumor-associated mutations. Differences in the expression of a particular gene or protein levels could also affect treatment response. Multi-omics methods are generating vast amounts of data of many different kinds. Integrating them to obtain biologically meaningful insights is one of the biggest challenges in future medicine. Researchers need sophisticated and robust computational strategies that consider the features of each of the different data modalities. Ultimately, multi-omics could be used in the clinic as a reliable tool to diagnose cancer or determine a patient’s prognosis. For this to happen, researchers will need to be able to gather more information from single cells, or smaller amounts of samples, in less time, and integrate the data in a standardized way. This requires further technological improvements (to decrease sample processing and measurement time, for example) and sharing of

273

274

Precision Oncology: Molecular Diversification of Tumor Patients

integration strategies. As the cost of omics technologies continues to decrease, more labs will be able to perform multi-omics sequencing in samples under various conditions. This also gives us the chance to guarantee the care of tumor patients on a broad basis. Uniform structures and guidelines must be established for this purpose. In the coming years, longitudinal multi-omics profiling of patients, along with their imaging and clinical data, are commonly employed to understand cancer diseases. Due to the current achievements in molecular diagnostics and the increasingly small-scale molecular diversification of disease the original vision of personalized patient care is moving closer.

References

1. Alexandrov, L. B. and Stratton, M. R. (2014). Mutational signatures: the patterns of somatic mutations hidden in cancer genomes. Curr Opin Genet Dev., 24(100), 52–60.

2. Alexandrov, L. B. et al. (2020). The repertoire of mutational signatures in human cancer. Nature, 578(7793), 94–101. 3. Allgäuer, M. et al. (2018). Implementing tumor mutational burden (TMB) analysis in routine diagnostics — a primer for molecular pathologists and clinicians. Transl Lung Cancer Res., 7(6), 703–715.

4. Andersson, A. et al. (2021). Spatial deconvolution of HER2-positive breast cancer delineates tumor-associated cell type interactions, Nat Commun, 12, 6012. 5. André, F. et al. (219). Alpelisib for PIK3CA-mutated, hormone receptor — positive advanced breast cancer, N. Engl. J. Med., 380, 1929–1940.

6. Andrew, N. J. et al. (2021). Adjuvant olaparib for patients with BRCA1or BRCA2-mutated breast cancer, N Engl J Med, 384, 2394–2405.

7. Baselga, J. et al. (1999). Phase II study of weekly intravenous trastuzumab (Herceptin) in patients with HER2/neu-overexpressing metastatic breast cancer, Semin Oncol, 26(4 Suppl 12), 78–83. 8. Bergethon, K. et al. (2012). ROS1 rearrangements define a unique molecular class of lung cancers, J. Clin. Oncol., 30, 863–870.

9. Blanke, C. D. et al. (2008). Phase III randomized, intergroup trial assessing imatinib mesylate at two dose levels in patients with unresectable or metastatic gastrointestinal stromal tumors expressing the kit receptor tyrosine kinase: S0033. J Clin Oncol., 26(4), 626–632.

References

10. Boniolo, F. et al. (2021). Artificial intelligence in early drug discovery enabling precision medicine, Expert Opin Drug Discov., 16(9), 991– 1007. 11. Chan, T. A. et al. (2019). Development of tumor mutation burden as an immunotherapy biomarker: utility for the oncology clinic, Ann Oncol., 30(1), 44–56.

12. Chapman, P. B. et al. (2011). Improved survival with vemurafenib in melanoma with BRAF V600E mutation, N Engl J Med., 364(26), 2507– 2516. 13. Chopra, N. et al. (2020). Homologous recombination DNA repair deficiency and PARP inhibition activity in primary triple negative breast cancer. Nat Commun., 11(1), 2662.

14. Clark, D. F. et al. (2019). Identification and confirmation of potentially actionable germline mutations in tumor-only genomic sequencing, JCO Precis Oncol, 3, PO.19.00076. 15. Dagogo-Jack, I. and Shaw A.T. (2018). Tumour heterogeneity and resistance to cancer therapies, Nature Reviews Clinical Oncology, 15, 81–94. 16. De Bono, J. et al. 2020. Olaparib for metastatic castration-resistant prostate cancer, N Engl J Med, 382, 2091–2102.

17. Dumbrava, E. I., and Bernstam, F. M. (2018). Personalized cancer therapy-leveraging a knowledge base for clinical decision-making, Cold Spring Harb Mol Case Stud., 4(2), a001578.

18. Dumbrava, E. I. et al. (2019). Targeting ERBB2 (HER2) Amplification identified by next-generation sequencing in patients with advanced or metastatic solid tumors beyond conventional indications. JCO Precis Oncol., 3, PO.18.00345. 19. Es, H. A. et al. (2018). Personalized cancer medicine: An organoid approach, Trends in Biotechnology, 36(4), 358–371.

20. Farmer, H. et al. (2005). Olaparib for metastatic breast cancer in patients with a germline BRCA mutation. NEJM, 377(6), 523–533.

21. Ford, D. et al. (1994). Risks of cancer in BRCA1-mutation carriers. Lancet, 19, 343(8899), 692–695. 22. Frampton, G. M. et al. (2015). Activation of MET via diverse exon 14 splicing alterations occurs in multiple tumor types and confers clinical sensitivity to MET inhibitors, Cancer Discov., 5, 850–859.

23. Gagan, J. and Van Allen, E. M. (2015). Next-generation sequencing to guide cancer therapy, Genome Medicine, 7, 80.

275

276

Precision Oncology: Molecular Diversification of Tumor Patients

24. Goodspeed, A. et al. (2016). Tumor-derived cell lines as molecular models of cancer pharmacogenomics, Mol Cancer Res., 14(1), 3–13.

25. Grasso, C. et al. (2015). Assessing copy number alterations in targeted, amplicon-based next-generation sequencing data, J Mol Diagn, 17(1), 53–63. 26. Guo, R. et al. (2020). MET-dependent solid tumours — molecular diagnosis and targeted therapy. Nat Rev Clin Oncol., 17(9), 569–587.

27. Ignatiadis, M. et al. (2021). Liquid biopsy enters the clinic — implementation issues and future challenges, Nat Rev Clin Oncol, 18, 297–312. 28. Islam, S. et al. (2011). Characterization of the single-cell transcriptional landscape by highly multiplex RNA-seq, Genome Res., 21(7), 1160– 1167.

29. Jackson, H. W. et al. (2020). The single-cell pathology landscape of breast cancer, Nature, 578, 615–620. 30. Kohno, T. et al. (2012). KIF5B-RET fusions in lung adenocarcinoma, Nat. Med., 18, 375–377.

31. López-Reig, R. et al. (2019). Prognostic classification of endometrial cancer using a molecular approach based on a twelve-gene NGS panel, Scientific Reports, 9, 18093. 32. Lynch, T. J. et al. (2004). Activating mutations in the epidermal growth factor receptor underlying responsiveness of non-small-cell lung cancer to gefitinib. NEJM, 350(21), 2129–2139.

33. Macosko, E. Z. et al. (2015). Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets, Cell, 161(5), 1202–1214. 34. Mandelker, D. et al. (2018). The emerging significance of secondary germline testing in cancer genomics, J Pathol., 244(5), 610–615. 35. Marchetti, A. et al (2011). Clinical features and outcome of patients with non-small-cell lung cancer harboring BRAF mutations, J. Clin. Oncol., 29, 3574–3579. 36. Martínez-Pérez, E. et al. (2021). Panels and models for accurate prediction of tumor mutation burden in tumor samples, NPJ Precis Oncol., 5, 31.

37. Merker, J. D. et al. (2018). Circulating tumor DNA analysis in patients with cancer: American Society of Clinical Oncology and College of American Pathologists joint review, J. Clin. Oncol., 36, 1631–1641.

References

38. Nishiyama, A. et al. (2020). MET amplification results in heterogeneous responses to osimertinib in EGFR-mutant lung cancer treated with erlotinib, Cancer Sci., 111(10), 3813–3823.

39. Picard, M. et al. (2021). Integration strategies of multi-omics data for machine learning analysis, Computational and Structural Biotechnology Journal, 19, 3735–3746. 40. Planchard, D. et al. (2017). Dabrafenib plus trametinib in patients with previously untreated BRAFV600E-mutant metastatic non-small-cell lung cancer: an open-label, phase 2 trial. Lancet Oncol., 18(10), 1307– 1316.

41. Ramazotti, D. et al. (2018). Multi-omic tumor data reveal diversity of molecular mechanisms that correlate with survival, Nature Communications, 9, 4453. 42. Ray-Coquard, I. et al. (2019). Olaparib plus bevacizumab as first-line maintenance in ovarian cancer, N Engl J Med, 381, 2416–2428.

43. Richards, S. et al. (2015). Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med, 17(5), 405–424. 44. Robichaux, J. P. et al. (2018). Mechanisms and clinical activity of an EGFR and HER2 exon 20-selective kinase inhibitor in non-small cell lung cancer. Nat. Med., 24(5), 638–646.

45. Robson, M. et al. (2017). Olaparib for metastatic breast cancer in patients with a germline BRCA mutation. NEJM, 377(6), 523–533. 46. Sha, D. et al. (2020). Tumor mutational burden as a predictive biomarker in solid tumors. Cancer Discov., 10(12), 1808–1825. 47. Shaw, A. T. et al. (2014). Ceritinib in ALK-rearranged non-small-cell lung cancer. NEJM, 370(13), 1189–1197.

48. Shaw, A. T. et al. (2020), First-line lorlatinib or crizotinib in advanced ALK-positive lung cancer, N Engl J Med., 383(21), 2018–2029. 49. Soda, M. et al. (2007). Identification of the transforming EML4–ALK fusion gene in non-small-cell lung cancer, Nature, 448, 561–566. 50. Solomon, B. J. et al. (2018). Lorlatinib in patients with ALK-positive non-small-cell lung cancer: results from a global phase 2 study. Lancet Oncol., 19(12), 1654–1667.

51. Soria, J. C. et al. (2018). FLAURA Investigators. Osimertinib in untreated EGFR-mutated advanced non-small-cell lung cancer. NEJM, 378(2), 113–125.

277

278

Precision Oncology: Molecular Diversification of Tumor Patients

52. Subramanian, I. et al. (2020). Multi-omics data integration, interpretation, and its application. Bioinformatics and Biology Insights, 14, 1177932219899051. 53. Sukhai, M. A. et al. (2016). A classification system for clinical relevance of somatic variants identified in molecular profiling of cancer, Genet Med, 18, 128–136. 54. Vaishnavi, A. et al. (2013). Oncogenic and drug-sensitive NTRK1 rearrangements in lung cancer. Nat Med, 19(11), 1469–1472.

55. Vega, D. M. et al. (2021). Aligning tumor mutational burden (TMB) quantification across diagnostic platforms: phase II of the Friends of Cancer Research TMB Harmonization Project. Ann Oncol., 32(12), 1626–1636. 56. Wang, C. et al. (2021). Alternative approaches to target Myc for cancer treatment. Sig Transduct Target Ther., 6(1), 117.

57. Sordella, R. et al. (2004). Gefitinib-sensitizing EGFR mutations in lung cancer activate anti-apoptotic pathways, Science, 305(5687), 1163– 1167. 58. Stephan, T. et al. (2022). Darwinian genomics and diversity in the tree of life, PNAS, 119(4), e2115644119.

59. Stephens, P. et al. (2004). Lung cancer: intragenic ERBB2 kinase mutations in tumours, Nature, 431, 525–526.

60. Subramanian, I. et al. (2020). Multi-omics data integration, interpretation, and its application, Bioinformatics and Biology Insights, 14, 1177932219899051. 61. Sukhai, M. A. et al. (2016). A classification system for clinical relevance of somatic variants identified in molecular profiling of cancer, Genet Med, 18, 128–136. 62. Vaishnavi, A. et al. (2013). Oncogenic and drug-sensitive NTRK1 rearrangements in lung cancer, Nat. Med., 19, 1469–1472.

63. Wu, Y. et al. (1999). Association of hereditary nonpolyposis colorectal cancer-related tumors displaying low microsatellite instability with MSH6 germline mutations, Am J Hum Genet., 65(5), 1291–1298.

Chapter 13

Digital Applications in Precision Pathology

Ralf Huss,a,b Johannes Raffler,b Claudia Herbst,a Tina Schaller,a and Bruno Märkla

aPathology and Molecular Diagnostics, University of Augsburg, Augsburg, Germany bDigital Medicine, University Hospital Augsburg, Augsburg, Germany [email protected]

13.1 Introduction The workflow in a histopathology laboratory is well established for decades if not even for centuries. Tissue or cytology specimens are delivered to the lab where well-trained and highly educated laboratory personnel and pathologists process the material and eventually release a report that influences the life of a patient. The histopathology report not only includes the (correct) diagnosis, but also insights into the prognosis and course of a disease and possible best treatment options for an individual patient. Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

280

Digital Applications in Precision Pathology

About 30–40 years ago, automation also entered the otherwise manual handling of samples with a first staining device after standard quality chemicals including commercial antibodies and detection systems became readily available and were affordable for routine and high throughput labs. A more recent disruption in the practice of histopathology is the availability of artificial intelligence and machine learning to assist complex decision-making based on digitized multiplexed and multidimensional images along with big data coming from tumor genetics and molecular pathology.

13.2 Precision Pathology

Precision or personalized cancer medicine is founded on the principle of providing treatment decisions based on the diagnosed cancer type or subtype of the individual patient, a principle that should apply also to all other disease entities. Particularly precision oncology requires the implementation of complex assays capable of identifying druggable targets as accurately and precisely as possible, especially in cancer tissue and its microenvironment or related surrogate (liquid) biomarkers. The specificity and sensitivity of biomarker tests are largely determined by the accuracy and precision of the analytical method used. The quality of the analyzed tissue sample and the method used for performing an assay requires the standardization of the pre-analytical handling of the material and the post-analytical interpretation of the results. The role of pathologists in digital medicine is to take responsibility through developing digital capabilities and user experience to implement and run any type of test robustly and sustainably in routine diagnostic practice. As we move towards an era of digital and computational pathology, the role of pathologists will extend as well as change to include the digitally documented management of more stringent quality control of tissue sample handling and analyses based on even complex biomarker tests that generate a vast body of data to stratify patients also to advanced and combination therapies. With the evolution of precision oncology, the importance of precision diagnostic pathology in guiding patient treatment and care

Precision Pathology

management decisions is growing fast. The potential for harm or at least inadequate treatment must be considered when patients lack access to high‐quality and precision pathology, advanced diagnostic solutions, and expert decision (tumor) boards that are supported by computational solutions. Machine learning resp. Artificial intelligence (ML/AI) is a compelling tool that improves efficiency and provides nextgeneration analyses of complex biomarker and (epi)genomics data. Pathologists must participate in ML/AI development and implementation, especially in precision medicine, or risk being bypassed by non-medical data mining and analysis experts. As automation increases, pathologists should also assess their true value to the healthcare team.

13.2.1 Machine Learning

AI is a powerful technology, but many pathologists do not yet appreciate how it will be applied in their daily clinical work assisting them in their daily and increasing complex workload without any danger to be replaced or substituted. AI-based solutions confirm or refine existing diagnoses commensurate to what is already known and plausible to an existing group of medical experts. Yet pathologists must participate in the development of AI-based solutions to invent and shape the digital future and guide the generation of algorithms through their existing knowledge and experience. ML is a collection of mathematical and computer science techniques for the extraction of relevant data from large data sets and images trained and explained by human experts or in this case by expert pathologists. Therefore, ML can assist the human factor, mostly by performing difficult and tedious tasks. An automated ML method will read multiplex immunohistochemistry (IHC) images always in the same reliable and robust manner, yielding identical results over and over again. This allows global comparability of complex and larger information without significant inter- and intraobserver variability. ML algorithms may include decision trees, Bayes’ networks, clustering solutions, and regression solutions, also depending on whether the solution is rather (weakly) supervised or unsupervised [33, 45]. With the advent of convolutional neural networks in recent years, ML is becoming increasingly accessible

281

282

Digital Applications in Precision Pathology

to researchers from non-computer science backgrounds. However, preparing data for machine learning remains a crucial step that requires hands-on expertise and expert knowledge. One other important topic for the development of robust ML/ AI especially in the context of clinical practice and decision-making is the use of correct input data. Incorrect or inconsistent data and in the case of pathology inadequately processed specimens (preor post-analytical) or images lead to false conclusions. Besides the use of high-quality data, the cleaning or cleansing of data has a high impact on the quality of results. This has to be a part of the standard quality management and pathology lab workflow [22]. Also, graphical shapes and visual representations are now used to allow the understanding of complex data as well as extracted information by non-AI domain experts (e.g., pathologists) to explain the rationale behind MI-based decision rules. Topological data analysis makes it now possible to understand complex MI-based solutions of multidimensional data sets more easily. With the emergence of immune and combination therapies in the field of precision oncology, a multitude of markers (different cell types, receptors, pathway molecules, genetic mutations, etc.) will be potential numerators or denominators of the same equation. The use of topological networks can also intuitively describe spatial relationships between different checkpoint molecules, immune effector cells, or other tissue components [13].

13.2.2 Machine Intelligence

With the advent of modern computing, efforts exist to similarly replace, assist and augment human cognitive and analytic effort, when desirable. It might not always be necessary, but certainly allows for addressing current objectives like to assist making the correct diagnosis and predicting the most effective therapy response. Such an effort attempts to create machine models also for aspects of human intelligence [9]. Machine intelligence (MI) is considered an extension of ML and implies AI. A less ambitious goal is called “narrow AI” or sometimes “symbolic AI” (good old-fashioned AI = GOFAI), which focuses on modeling presumably simpler tasks and supporting medical decision-making. If we succeed with this task, we will transition from

Computational, Algebraic, and Encoded Pathology

narrow AI to broader AI, which we refer to as “MI”. This comprises different layers such as MI and topology. ML is usually not the same thing as AI [25]. To a large degree, AI depends on human experts (“conventional expert knowledge”) to create rule sets or algorithms in order to support decision-making also in medicine. AI’s ability to solve a problem is dependent on having a human expert or pathologist describing a current problem or anticipating a future problem in advance [5]. Furthermore, ML uses stochastic methods sometimes creating its own rules. ML is an attempt to substitute for human experience and expert knowledge in case it is not available or sufficient to solve a relevant problem. MI goes beyond the “black box” nature of ML and includes techniques such as neural networks and DL. The output allows the capture of the full complexity of features and communication networks in heterogeneous and multiplexed tissue specimens and provides explanations of the results for pathologists [2]. In this pursuit, expert pathologists still need to verify the agreement between the MI-based decisions and any ground truth (expert) knowledge. Naturally, painfully acquired human expert knowledge during many years of training and education and the use of MI is an obvious area of conflict. Both approaches converge in the clinical validation of automated rule sets, proving the sustainable correctness of machine-based decisions in precision pathology and medical practice. It remains also legally necessary to confirm any machine-supported diagnosis through a highly skilled and well-trained pathologist and to provide a plausibility check on the accuracy of the computer-assisted decision [3]. There it is desirable that algorithms in healthcare ideally are explainable, dynamic, precise, autonomous, fair, and of course reproducible [30].

13.3 Computational, Algebraic, and Encoded Pathology

Advances in ML have propelled computational pathology (CP) research. MI can do what humans respectively pathologists can do but in a more reliable and standardized way. Today, computer systems approach the diagnostic reality achieved by humans for certain welldefined tasks in pathology. At the same time, pathologists are faced

283

284

Digital Applications in Precision Pathology

with an increased workload both quantitatively (numbers of cases) and qualitatively (the amount and complexity of work per case. The potential of machine learning techniques in pathology ranges from computer-aided support for relatively straightforward tasks to the discovery of innovative prognostic and predictive biomarkers [36]. The most basic applications deal mostly with simple dichotomous detection problems such as the presence of lymph node metastases or not; or counting the density of mitotic or Ki-67 positive tumor cells. They have the potential to increase efficiency and precision in the pathological diagnostic workflow. The first algorithms of this “simple kind” are already commercially available and viable. At the other end of the spectrum, there are models that can assess sub-visual morphological information, potentially playing a role in personalized medicine and precision oncology like cancer pathway analysis [41]. With the increasing complexity of the applications comes an increasing demand for large (big or augmented), well-curated, and clean datasets. This poses challenges for researchers and algorithm developers, as the collection of data is cumbersome and expensive. Still, the potential for CP is large and applications will play a role in the future of pathology [24]. CP or termed by others “algebraic pathology” (AP) leverages mathematical tools and implements data-driven methods as a center for data interpretation in modern tissue diagnosis. The value proposition of CP as well as digital pathology requires working closely with other departments and pathology colleagues. Digital pathology will also foster the training of future computational pathologists, those with both pathology and nonpathology backgrounds, who will eventually decide that CP/AP will serve as an indispensable hub for data-related research in a global health care system. It is still inevitable that a written pathology report contains semantic information in a standardized manner so that it can be used for further ML-assisted analysis. This standardized reporting has also been coined “encoded pathology” to allow the international comparability of texts and the alignment of patient and case information from different subjects such as radiology. Since pathology reports are either biased or contain (tissue) images, it is also necessary to agree on standardized image formats.

Applications of Precision Pathology

13.4 Applications of Precision Pathology With the integration of modern tools such as multiplexing immune phenotyping [10, 12], learning software solutions and machine learning into the routine work of pathologists comes a deeper understanding of the communication network in tissues and reveals the existing tumor heterogeneity that has consequences for various treatment options [6, 20, 22, 44]. According to the understanding of tumor heterogeneity, pathologists will guide oncologists to select the optimal treatment for each cancer patient that will yield the best tumor response. At the end of the last century, Ki67 and Her2/neu were accepted as the first predictive biomarker that required the pathologist to strictly and exclusively quantify cancer cells that (over)expressed those markers in any part of the tumor area [11]. In the meantime, a large number of other biomarkers including overexpressed proteins and molecular aberrations have been identified that guide therapeutic strategies and decisions in many tumor entities [17]. With respect to immunological properties of tumors, it was Galon’s landmark paper in 2012 [15] that demonstrated the combination and simultaneous presence of more than one marker with a defined spatial distribution in different compartments of the cancer tissue [14, 34]. However, this already required a computer-based algorithm and was only possible with (i) a good understanding of cancer immunology; (ii) expert knowledge of the histopathologist reading the case; and (iii) the implementation of an automated image analysis technology. Precision pathology received even higher recognition with the clinical availability and success of immunotherapy. With the advancement of analytical methods like immunohistochemistry (IHC), molecular tools, and computational solutions, immunotherapies make an even greater impact in our clinical practice [28]. Today we have advanced diagnostic tools at hands such as digital imaging for the objective and reproducible assessment of multiple markers on a single cell at a time or on a single tissue slide precisely quantifying the absolute numbers of functionally distinct immune cells as well as their spatial distribution and contextual relationships in various tissue compartments [35]. Different studies have already shown an association between immune cell infiltrates in selected tumor areas and improved outcomes [32, 42].

285

286

Digital Applications in Precision Pathology

The evaluation of PD-1/PD-L1 is currently the diagnostic backbone for the prediction of the response of a checkpoint inhibitor therapy. PD-L1 testing is more complicated than Ki-67 or Her2/neu scoring, due to different antibodies, different testing algorithms, and constantly changing cut-offs. The reading and reporting of PD-L1 scores requires skilled and trained pathologists, also considering the substantial intra-tumor heterogeneity of PD-1 and PD-L1 with prevalent expression in the invasive front. Differences in intratumoral PD-L1 expression are readily recognized in daily practice and documented in the literature [7, 29]. This can also result in differing testing between biopsies and related surgical specimens [23]. Moreover, several authors report a considerable heterogeneity of PD-L1 expression between the primary versus lymph node metastases [26] and distant metastases [27]. Biomarkers reflect the individual tumor immune microenvironment and tumor intrinsic factors like TMB or mismatch-repair (MMR) deficiency associated with the treatment efficacy of anti-PD-1/anti-PD-L1 therapy. Microsatellite instability (MSI) seems to play a tumor agnostic role when pembrolizumab is administered to MSI-positive tumors regardless of the entity [34]. Marabelle et al. have demonstrated the association of TMB with outcomes in patients with advanced solid tumors [31]. Similar results have been obtained in hypermutated tumors [37] and TMB as an indicator for cytolytic activity and prognosis in malignant melanoma and other tumors [8, 38]. However, recent studies have shown the relevance of other components of the immune system that have a significant prognostic value and can be included in therapeutic considerations [16, 40, 43]. While it could be shown that the spatial relationship is a predictive biomarker for the therapeutic use of Ipilimumab, similar algorithms play no role in low- to intermediate prostate cancer [21]. Instead, other contextual dimensions including macrophages have a higher informative value related to prognosis and therapy selection in prostate cancer [39]. These are the factors that influence the adoption of MI and digital biomarker in patient care and also pathology. Helpful resources and practical issues such as guidelines, validation, and accreditation requirement will support the implementation of clinical-grade digital pathology. In the end, however, it is computer science to assist diagnostic decision-making and the practice of pathology.

Digital Image and Data Analysis

13.5 Digital Image and Data Analysis The field of AI and ML algorithms offers a wide array of diagnostic and therapeutic decision support [4]. The primary objective of image analysis in the context of profiling tissue specimens is the ability to accurately quantify and calculate relevant spatial relationships of all immune cells and other immune-related biomarkers [3]. Along with a sophisticated analysis of multiplexed images, the integration of available molecular data from cancer (epi)genome or transcriptome analysis provides an even more granular assessment of a certain cancer type reflecting the situation of an individual cancer patient. However, image analysis cannot be leaned upon unless resolving any inconsistencies caused by pre-analytical variables. Therefore, proper sample handling, processing, staining, and scanning are as important as the curation of data and the development of robust algorithms. The pathologist should perform quality control of all images and tissue processing steps ideally throughout the entire workflow. Image analysis is at the core of digital and computational pathology and requires special expertise and competencies by expert pathologists and active interdisciplinary collaboration with computer and data scientists as well as software engineers [18]. The body of literature and informative textbooks on this topic is steadily growing with the advancement of associated hardware technologies, data storage, and processing capabilities as well as machine and deep learning techniques [3]. The objective of image mining is the discovery and description of features with diagnostic, prognostic, and possibly predictive features relevant to the future practice of digital medicine [11]. The increasing amounts of patientspecific data from images and other (-omics) sources are used to augment the understanding of individual diseases and to generate hypotheses around diagnostic and other biomarkers [1]. Digital and computational pathology have become cornerstones of translational research, transforming tissue pathology-based biomarker strategies for drug development or repurposing [19]. It is a challenge as a part of digital medicine to further integrate advanced data analytics into their decision-making process. The availability of digital tools allows for interpreting the high-dimensional complexity of information in tissue. The implementation, execution, and monetization of digital medicine in the 21st century, will inevitably “call for pathologists as their leaders embracing digital and computational pathology.”

287

288

Digital Applications in Precision Pathology

References 1. Akbar, S., Jordan, L. B., Purdie, C. A., Thompson, A. M., McKenna, S. J. (2015). Comparing computer-generated and pathologist-generated tumour segmentations for immunohistochemical scoring of breast tissue microarrays. Br. J. Cancer 113(7), 1075–1080. 2. Badrinarayanan, V., Kendall, A., Cipolla, R. (2017). SegNet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495. 3. Binnig, G., Huss, R., Schmidt, G. (2018). Tissue Phenomics – Profiling Cancer Patients for Treatment Decisions. Jenny Stanford Publishing Pte. Ltd., Singapore. 4. Campbell, W. S., Foster, K. W., Hinrichs, S. H. (2013). Application of whole slide image markup and annotation for pathologist knowledge capture. J. Pathol. Inform. 4, 2. 5. Campion, F. X., Carlsson, G. (2017): Machine Intelligence for Healthcare, 1st edition.

6. Chen, G. M., Azzam, A., Ding, Y. Y., Barrett, D. M., Grupp, S. A., Tan, K. (2020). Dissecting the tumor-immune landscape in chimeric antigen receptor T-cell therapy: Key challenges and opportunities for a systems immunology approach. Clin. Cancer Res. 26(14), 3505–3513.

7. Cho, J. H., Sorensen, S. F., Choi, Y. L., et al. (2017). Programmed death ligand 1 expression in paired non-small cell lung cancer tumor samples. Clin. Lung Cancer 18(6), e473–e479. 8. Danilova, L., Wang, H., Sunshine, J., et al. (2016). Association of PD-1/ PD-L axis expression with cytolytic activity, mutational load, and prognosis in melanoma and other solid tumors. Proc. Natl. Acad. Sci. U S A 113(48), E7769–E7777.

9. Davies, A., Veličković, P., Buesing, L., et al. (2021). Advancing mathematics by guiding human intuition with AI. Nature 600(7887), 70–74. 10. Decalf, J., Albert, M. L., Ziai, J. (2019). New tools for pathology: A user’s review of a highly multiplexed method for in situ analysis of protein and RNA expression in tissue. J. Pathol. 247(5), 650–661.

11. Dobson, L., Conway, C., Hanley, A., et al. (2010). Image analysis as an adjunct to manual HER-2 immunohistochemical review: A diagnostic tool to standardize interpretation. Histopathology 57(1), 27–38. 12. Du, Z., Lin, J. R., Rashid, R., et al. (2019). Qualifying antibodies for image-based immune profiling and multiplexed tissue imaging. Nat. Protoc. 14(10), 2900–2930.

References

13. Dundar, M., Badve, S., Raykar, V., Jain, R., Sertel, O., Gurcan, M. (2010). A multiple instance learning approach toward optimal classification of pathology slides. International Conference on Pattern Recognition 2010, 2732–2735. 14. Galon, J., Bruni, D. (2019). Approaches to treat immune hot, altered and cold tumours with combination immunotherapies. Nat. Rev. Drug Discov. 18(3), 197–218. 15. Galon, J., Pagès, F., Marincola, F. M., et al. (2012). The immune score as a new possible approach for the classification of cancer. J. Transl. Med. 10, 1.

16. Gibney, G. T., Weiner, L. M., Atkins, M. B. (2016). Predictive biomarkers for checkpoint inhibitor-based immunotherapy. Lancet Oncol. 17(12), e542–e551. 17. Gnjatic, S., Bronte, V., Brunet, L. R., et al. (2017). Identifying baseline immune-related biomarkers to predict clinical outcome of immunotherapy. J. Immunother. Cancer 5, 44.

18. Gurcan, M. N., Boucheron, L. E., Can, A., Madabhushi, A., Rajpoot, N. M., Yener, B. (2009). Histopathological image analysis: A review. IEEE Rev. Biomed. Eng. 2, 147–171.

19. Gutman, D. A., Khalilia, M., Lee, S., et al. (2017). The Digital Slide Archive: A software platform for management, integration, and analysis of histology for cancer research. Cancer Res. 77(21), e75–e78. 20. Handy, C. E., Antonarakis, E. S. (2018). Sipuleucel-T for the treatment of prostate cancer: Novel insights and future directions. Future Oncol. 14(10), 907–917.

21. Harder, N., Athelogou, M., Hessel, H., et al. (2018). Tissue Phenomics for prognostic biomarker discovery in low- and intermediate-risk prostate cancer. Sci. Rep. 8(1), 4470. 22. Huss, R., Coupland, S. E. (2020). Software-assisted decision support in digital histopathology. J. Pathol. 250(5), 685–692.

23. Ilie, M., Long-Mira, E., Bence, C., et al. (2016). Comparative study of the PD-L1 status between surgically resected specimens and matched biopsies of NSCLC patients reveal major discordances: A potential issue for anti-PD-L1 therapeutic strategies. Ann. Oncol. 27(1), 147– 153. 24. Jegou, H., Douze, M., Schmid, C., Pérez, P. (2010). Aggregating local descriptors into a compact image representation, Computer Vision and Pattern Recognition, 13–18.

289

290

Digital Applications in Precision Pathology

25. Kainz, P., Pfeiffer, M., Urschler, M. (2017). Segmentation and classification of colon glands with deep convolutional neural networks and total variation regularization. PeerJ 5, e3874.

26. Kim, M. Y., Koh, J., Kim, S., Go, H., Jeon, Y. K., Chung, D. H. (2015). Clinicopathological analysis of PD-L1 and PD-L2 expression in pulmonary squamous cell carcinoma: Comparison with tumorinfiltrating T cells and the status of oncogenic drivers. Lung Cancer 88(1), 24–33. 27. Kim, S., Koh, J., Kwon, D., et al. (2017). Comparative analysis of PD-L1 expression between primary and metastatic pulmonary adenocarcinomas. Eur. J. Cancer 75, 141–149.

28. Liu, D. (2019). Cancer biomarkers for targeted therapy. Biomark Res 7, 25.

29. Liu, Y., Dong, Z., Jiang, T., et al. (2018). Heterogeneity of PD-L1 expression among the different histological components and metastatic lymph nodes in patients with resected lung adenosquamous carcinoma. Clin. Lung Cancer 19(4), e421–e430. 30. Loftus, T. J., Tighe, P.J., Ozratzgat-Baslanti, T., et al. (2022). Ideal algorithms in healthcare: Explainable, dynamic, precise, fair and reproducible. PLOS Digital Health 1(1). 31. Marabelle, A., Fakih, M., Lopez, J., et al. (2020). Association of tumour mutational burden with outcomes in patients with advanced solid tumours treated with pembrolizumab: prospective biomarker analysis of the multicohort, open-label, phase 2 KEYNOTE-158 study. Lancet Oncol. 21(10), 1353–1365.

32. Masucci, G. V., Cesano A., Hawtin, R., et al. (2016). Validation of biomarkers to predict response to immunotherapy in cancer: Volume I - pre-analytical and analytical validation. J. Immunother. Cancer 4, 76. 33. Meier, A., Nekolla, K., Hewitt, L. C., et al. (2020). Hypothesis-free deep survival learning applied to the tumour microenvironment in gastric cancer. J. Pathol. Clin. Res. 6(4), 273–282. 34. Mlecnik, B., Bindea, G., Angell, H. K., et al. (2016). Integrative analyses of colorectal cancer show immunoscore is a stronger predictor of patient survival than microsatellite instability. Immunity 44(3), 698–711. 35. Patel, S. S., Rodig, S. J. (2020). Overview of tissue imaging methods. Methods Mol. Biol. 2055, 455–465.

36. Rana, A., Lowe, A., Lithgow, M., et al. (2020). Use of deep learning to develop and analyze computational hematoxylin and eosin staining of prostate core biopsy images for tumor diagnosis. JAMA Netw. Open 3(5), e205111.

References

37. Rousseau, B., Foote, M. B., Maron, S. B., et al. (2021). The spectrum of benefit from checkpoint blockade in hypermutated tumors. N. Engl. J. Med. 384(12), 1168–1170. 38. Samstein, R. M., Lee, C. H., Shoushtari, A. N., et al. (2019). Tumor mutational load predicts survival after immunotherapy across multiple cancer types. Nat. Genet. 51(2), 202–206.

39. Sun, Y., Xu, S. (2018). Tumor-associated CD204-positive macrophage is a prognostic marker in clinical Stage I lung adenocarcinoma. Biomed Res Int 2018, 8459193.

40. Surace, M., Rognoni, L., Rodriguez-Canales, J., Steele, K. E. (2020). Characterization of the immune microenvironment of NSCLC by multispectral analysis of multiplex immunofluorescence images. Methods Enzymol. 635, 33–50. 41. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. (2016). Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

42. Taube, J. M., Galon, J., Sholl, L. M., et al. (2018). Implications of the tumor immune microenvironment for staging and therapeutics. Mod. Pathol. 31(2), 214–234. 43. Vasaturo, A., Galon, J. (2020). Multiplexed immunohistochemistry for immune cell phenotyping, quantification and spatial distribution in situ. Methods Enzymol. 635, 51–66.

44. Wickenhauser, C., Bethmann, D., Feng, Z., et al. (2019). Multispectral fluorescence imaging allows for distinctive topographic assessment and subclassification of tumor-infiltrating and surrounding immune cells. Methods Mol. Biol. 1913, 13–31.

45. Xu, Y., Zhu J. Y., Chang, E. I., Lai, M., Tu, Z. (2014). Weakly supervised histopathology cancer image segmentation and classification. Med. Image Anal. 18(3), 591–604.

291

Chapter 14

Computational Pathology

Peter Schüfflera and Wilko Weichertb aInstitute

of Pathology, TUM School of Medicine, Technical University of Munich, Munich, Germany bTUM Department of Informatics, Technical University of Munich, Munich, Germany [email protected]

14.1 Introduction Pathology is currently transitioning from a largely analog to an increasingly digital discipline. Digital pathology not only simplifies the search for and sharing of digitized tissue images as well as their location-independent sign-out, but it also enables the use of artificial intelligence to analyze the images: Computational pathology (CP) is a new area between computer science and pathology with valuable tools for pathologists to render diagnoses reliably and efficiently. In this chapter, we discuss typical examples, opportunities, and hurdles of CP in clinical routine and research. Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

294

Computational Pathology

Artificial intelligence (AI) is omnipresent in our everyday lives and has long since found its way into medicine. For example, AIcontrolled robots carry out complex operations independently [1], or modern imaging methods calculate synthetic images from the inner part of a patient’s body fully automatically simultaneously providing their interpretation, marking suspicious spots of lesions and cancer [20]. In all those examples, the AI ​serves as a tool to solve a specific and narrowly defined task it has been trained for. It only works within the framework and assumptions specified for that task. Therefore, it is called weak AI (in contrast to strong AI, which has generic intellectual abilities at par with humans and is not discussed in this article) [6]. The development of AI for such a specific task requires both digital representative data and performant machine learning algorithms that can learn from these data.

14.2 AI in Pathology is a New Field

AI has emerged in many medical areas but has only recently gained a foothold in pathology. The field of CP, i.e., AI-supported data analysis in pathology, is approximately ten years old [7]. This is no coincidence, as it is also due to the fact that pathology is still a largely analog science, in which hardly any digital image data existed until recently [13, 11]. With the dawning of the digital revolution in pathology and the associated digitization of image data, AI can now step into the field for the first time. The emerging digitalization in pathology is therefore an important factor that will lead to better AI models in the future, as the algorithms can be learned with larger and more comprehensive sets of digital image data. Another key factor for the relatively late adaptation of AI in pathology is the sheer size of the high-resolution tissue scans. While AI in classic applications of computer vision, such as face recognition on a camera, processes images of around 1,000 × 1,000 px in size (= 1 megapixel), in pathology, the algorithms in pathology have to process image dimensions of 150,000 × 200,000 px at 0.25 mpp (microns per pixel), i.e., the 30,000-fold per image in the gigapixel range. Images of such dimensions are not processed “as a whole,” but they must first be divided into many small tiles. Only with

Impact on Clinical Routine

sufficient computing power found in modern data centers, clinically relevant algorithms can be learned from sufficiently large data sets in a reasonable time. The greater availability of digital images and the technical developments in computer hardware, software, and scanners in recent years have reached a critical mass necessary for modern CP with high clinical impact.

14.3 Impact on Clinical Routine

As a diagnostic discipline, pathology is a central component of both medicine and general cancer research. Accordingly, a distinction can be made between two different areas of application for the CP:

Clinical CP develops solutions to automate existing processes in pathology to make them more reproducible, more efficient, or more accurate. It therefore directly benefits pathologists and patients. Examples range from repetitive tasks such as the (automated) counting of positive cancer cells in immunohistochemically stained kidney tissue [17] to diagnostic workflows such as the (automated) detection, quantification, and classification of small and large cancer lesions in prostate biopsies [2, 14, 19]. Clinical CP also includes accompanying technologies to provide digital pathology slides such as automated quality control (QC) of the whole slide images [3]. QC is particularly important for institutes or laboratories that are fully digital and no longer use physical slides. Further, the whole quality management of a laboratory can be supported by AI, for example, as a systematic second opinion to support diagnostic decisions [15]. It is without question that pathology AI must be highly accurate in order to be used clinically. The clinical applicability must therefore be tested for each model. For this, laboratories have the option of using a laboratory-developed test (LDT) that validates the AI ​​for a specific task within the lab using its data and workflows. Additionally, independent organizations provide certificates for AI in pathology, such as the Food and Drug Administration (FDA) in the USA or the CE mark in Europe. In 2021, the FDA authorized for the first time in pathology an AI detecting cancer prostate needle biopsies [4]. The new category of medical devices enables consecutive models to be processed faster in the future.

295

296

Computational Pathology

An important hurdle for AI in pathology is its integration into clinical workflows and laboratory information systems [18]. The existing laboratory IT, hardware, and software are not necessarily prepared for digital pathology, and specific requirements such as high bandwidths, fast storage, and various visualization capabilities of AI​​ in diverse systems have to be fulfilled. This can make investments in a new IT infrastructure necessary. Only a user-friendly integration of digital and AI-supported pathology into the existing laboratory system enables pathologists to use the new technologies efficiently [8–10].

14.4 Impact on Research

The research CP uses machine learning methods to find new properties and patterns in pathology and related data that help to better understand and treat diseases. For example, AI is being used to detect novel biomarkers in H&E-stained images. These markers can be used to subtype and classify diseases, predict survival probabilities or therapeutic success, or conclude on therapyrelevant mutations [5]. Figure 14.1 shows an overview of current models being developed to detect, classify or subtype tumors in specific organs (Fig. 14.1a), detect mutations, or make prognoses for survival and therapy response (Fig. 14.1b). These models are the subject of current research and must be further validated before they ultimately become clinically relevant. CP is a valuable research tool because the AI ​​systematically and reproducibly screens through thousands of tissue images and detects the smallest irregularities. This offers new opportunities in cancer research, both in generating new hypotheses and validating them and will thus further deepen our understanding of the diseases and personalized medicine. A challenge for CP, however, is the generation of suitably large, representative data sets from which the AI ​​can learn a large number of morphologic variations in the tissue. For example, in-house datasets from a single institution can potentially be biased towards the local patient population or towards local laboratory conditions (e.g., the way tissue is cut and stained) and may not be representative and generic for larger populations. This is a general problem in data-driven research and must also be taken into account in CP.

Impact on Research

Figure 14.1 Computational pathology affects all fields of pathology, both in clinical routine (a) and beyond (b). Each cross represents a model that solves a specific problem in cancer (e.g., detection of prostate cancer). The numbers are the references from the review paper for the respective studies [5].

297

298

Computational Pathology

14.5 Structured Reports Are Essential for AI Development Advancing digitalization in pathology is essential to create the large datasets required to develop clinical-grade AI. Prospective scanning in routine diagnostics as well as retrospective scanning of the archives are key to generating valuable data treasures. However, just as much as important is the digital availability of the data associated with the images. This includes, for example, any annotations, molecular pathological and clinical data, outcome data, and pathology reports. While a lot of such data is already stored in a structured way today, pathology reports in particular are often unstructured, hand-written, inconsistent, and therefore difficult to read by a machine. At the same time, the reports are extremely important for the development of AI, since they already contain all the information and annotations relevant to each patient and disease. Structured reporting, i.e., uniform and systematic rendering of pathology reports, is, therefore, an extremely important topic for CP that can often make any further image annotations by pathologists obsolete.

14.6 Datasets at Scale via Federated Learning

As mentioned earlier in this article, AI is capable to learn the manifold variations in human tissue of a specific disease only if it is exposed to a representative data set of adequate size. Distributed computing (e.g., federated learning [12]) is a further alternative to access these required large data sets. In this scenario, the data sets remain in their respective local institutions but are still available to be used to train global AI algorithms. Instead of moving the data to central storage systems, the AI ​​moves from site to site and updates its parameters based on the local data. Only these learned parameters then leave the site, but not the data itself. This technology is the subject of current research and is not yet very widespread in practice, since uniform standards for data formats and interfaces of institutional image data access in digital pathology are still lacking. However, advanced examples of such federated networks can already be found in radiology, e.g., the RACOON network for COVID-19 research [16].

References

14.7 Outlook After the foundations of CP started a decade ago, there are now the first AI models clinically available to support routine diagnostics. With emerging digitalization in pathology, the growing digital image data sets with associated molecular, genetic information, and clinical data, and with more powerful computer hardware and efficient algorithms, AI has grown in the meanwhile to an indispensable tool in pathology, both in routine diagnostics and in research. The hurdles to using AI in clinical practice are no longer only lying in the algorithms themselves, but also in their integration into the practical workflows: How can we integrate AI effectively in our viewers? Is it compatible with existing laboratory information systems or scanner hardware? Do we need additional programs? What are the legal and insurance-related framework conditions when using AI? What is the effective and long-term benefit of AI in pathology? If those questions are answered, AI will soon become an indispensable part of everyday work for many pathologists.

References

1. Aruni, G., Amit, G., and Dasgupta, P. (2018). New surgical robots on the horizon and the potential role of artificial intelligence. Investig. Clin. Urol., 59 (4), 221.

2. Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., and Fuchs, T. J. (2019). Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med., 25 (8), 1301–1309. 3. Campanella, G., Rajanna, A. R., Corsale, L., Schüffler, P. J., Yagi, Y., and Fuchs, T. J. (2018). Towards machine learned quality control: A benchmark for sharpness quantification in digital pathology. Comput. Med. Imaging Graph., 65, 142–151.

4. FDA News Release (2021) FDA authorizes software that can help identify prostate cancer. 5. Echle, A., Rindtorff, N. T., Brinker, T. J., Luedde, T., Pearson, A. T., and Kather, J. N. (2020). Deep learning in cancer pathology: A new generation of clinical biomarkers. Br. J. Cancer, 1–11.

299

300

Computational Pathology

6. Frankish, K., and Ramsey, W. M. (eds.) (2014). The Cambridge Handbook of Artificial Intelligence, Cambridge University Press, Cambridge. 7. Fuchs, T. J., and Buhmann, J. M. (2011). Computational pathology: Challenges and promises for tissue analysis. Comput. Med. Imaging Graph., 35 (7–8), 515–530.

8. Hanna, M. G., Reuter, V.E., Ardon, O., Kim, D., Sirintrapun, S. J., Schüffler, P .J., Busam, K. J., Sauter, J. L., Brogi, E., Tan, L. K., Xu, B., Bale, T., Agaram, N. P., Tang, L. H., Ellenson, L.H., Philip, J., Corsale, L., Stamelos, E., Friedlander, M.A., Ntiamoah, P., Labasin, M., England, C., Klimstra, D. S., and Hameed, M. (2020). Validation of a digital pathology system including remote review during the COVID-19 pandemic. Mod. Pathol., 33 (11), 2115–2127.

9. Hanna, M. G., Reuter, V.E., Samboy, J., England, C., Corsale, L., Fine, S. W., Agaram, N. P., Stamelos, E., Yagi, Y., Hameed, M., Klimstra, D. S., and Sirintrapun, S. J. (2019). Implementation of digital pathology offers clinical and operational increase in efficiency and cost savings. Arch. Pathol. Lab. Med., 143 (12), 1545–1555. 10. Hanna, M. G., Reuter, V. E., Hameed, M. R., Tan, L. K., Chiang, S., Sigel, C., Hollmann, T., Giri, D., Samboy, J., Moradel, C., Rosado, A., Otilano, J. R., England, C., Corsale, L., Stamelos, E., Yagi, Y., Schüffler, P.J., Fuchs, T., Klimstra, D. S., and Sirintrapun, S. J. (2019). Whole slide imaging equivalency and efficiency study: experience at a large academic center. Mod. Pathol., 32 (7), 916–928. 11. Jahn, S.W., Plass, M., and Moinfar, F. (2020). Digital Pathology: Advantages, Limitations and Emerging Perspectives. J. Clin. Med., 9 (11), 3697.

12. Konečný, J., McMahan, B., and Ramage, D. (2015) Federated Optimization: Distributed Optimization Beyond the Datacenter. ArXiv151103575 Cs Math. 13. Pantanowitz, L. (2010). Digital images and the future of digital pathology. J. Pathol. Inform., 1 (1), 15.

14. Pantanowitz, L., Quiroga-Garza, G.M., Bien, L., Heled, R., Laifenfeld, D., Linhart, C., Sandbank, J., Shach, A.A., Shalev, V., Vecsler, M., Michelow, P., Hazelhurst, S., and Dhir, R. (2020). An artificial intelligence algorithm for prostate cancer diagnosis in whole slide images of core needle biopsies: A blinded clinical validation and deployment study. Lancet Digit. Health, 2 (8), e407–e416. 15. Parwani, A. V. (2019). Next generation diagnostic pathology: use of digital pathology and artificial intelligence tools to augment a pathological diagnosis. Diagn. Pathol., 14 (1), 138, s13000-019-0921–2.

References

16. RACOON, Radiological Cooperative Network, Netzwerk Universitätsmedizin. https://www.netzwerk-universitaetsmedizin. de/projekte/racoon - Accessed on 1/17/2022.

17. Schueffler, P. J., Fuchs, T. J., Ong, C. S., Roth, V., and Buhmann, J. M. (2013). Automated analysis of tissue micro-array images on the example of renal cell carcinoma, in Similarity-Based Pattern Analysis and Recognition, Springer, London, pp. 219–246.

18. Schüffler, P. J., Geneslaw, L., Yarlagadda, D. V. K., Hanna, M. G., Samboy, J., Stamelos, E., Vanderbilt, C., Philip, J., Jean, M.-H., Corsale, L., Manzo, A., Paramasivam, N. H. G., Ziegler, J. S., Gao, J., Perin, J. C., Kim, Y. S., Bhanot, U. K., Roehrl, M. H. A., Ardon, O., Chiang, S., Giri, D. D., Sigel, C. S., Tan, L. K., Murray, M., Virgo, C., England, C., Yagi, Y., Sirintrapun, S. J., Klimstra, D., Hameed, M., Reuter, V. E., and Fuchs, T. J. (2021). Integrated digital pathology at scale: A solution for clinical diagnostics and cancer research at a large academic medical center. J. Am. Med. Inform. Assoc., 28 (9), 1874–1884.

19. Ström, P., Kartasalo, K., Olsson, H., Solorzano, L., Delahunt, B., Berney, D. M., Bostwick, D. G., Evans, A. J., Grignon, D. J., Humphrey, P. A., Iczkowski, K. A., Kench, J. G., Kristiansen, G., van der Kwast, T. H., Leite, K. R. M., McKenney, J. K., Oxley, J., Pan, C.-C., Samaratunga, H., Srigley, J. R., Takahashi, H., Tsuzuki, T., Varma, M., Zhou, M., Lindberg, J., Lindskog, C., Ruusuvuori, P., Wählby, C., Grönberg, H., Rantalainen, M., Egevad, L., and Eklund, M. (2020). Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: A population-based, diagnostic study. Lancet Oncol., 21 (2), 222–232. 20. Tushar, F. I., D’Anniballe, V. M., Hou, R., Mazurowski, M. A., Fu, W., Samei, E., Rubin, G. D., and Lo, J. Y. (2022). Classification of multiple diseases on body CT scans using weakly supervised deep learning. Radiol. Artif. Intell., 4 (1), e210026.

301

Chapter 15

Digital Neuropathology

Friederike Liesche-Starnecker,a Georg Prokop,b and Jürgen Schlegelb aPathology and Molecular Diagnostics, University of Augsburg, Augsburg, Germany bDepartment of Neuropathology, Institute of Pathology, Technical University of Munich, Munich, Germany [email protected]

15.1 Introduction: The Roots of Neuropathology Neuropathology is often seen as a specialty in pathology, although its roots lie elsewhere. In the second half of the 19th century, the foundation of neurohistology was given by Camilo Golgi and Santiago Ramon y Cajal. Based on these methodological developments, neurologists and psychiatrists such as Franz Nissl and Alois Alzheimer advanced the development of neuropathology driven by their interest in the pathologies behind neurological diseases [14]. And they, indeed, were not pathologists. Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

304

Digital Neuropathology

The second branch of neuropathology can be traced back to neurosurgeons, including Harvey Cushing and Parcival Bailey, who provided the basis of modern neurooncology [3]. This strong clinical origin is still deep-seated in today’s neuropathologists. However, in most countries, neuropathologists work in pathology. German medical society is one of the rare where neuropathology is a medical discipline on its own, but with only 122 working specialists, it represents one of the smallest professional communities in Germany [7]. It can be beneficial to be part of a small community, though, since everybody knows each other, and channels are pleasantly short. Such precondition offers the possibility for innovation which has made the discipline of neuropathology a pioneer in a diversity of applications in the field of digital medicine.

15.2 Traditional Histology in Future Lights

Nowadays, we use similar methods for diagnostics as the founding fathers of neuropathology. For every diagnostic sample, standard light microscopy is conducted including hematoxylin and eosin (HE) staining and if applicable, complemented by immunohistochemical characterization or electron microscopy. Several traditional classifications are based on qualitative or semi-quantitative assessments and are therefore susceptible to inter- and intraobserver variability. The gold standard for the histopathological staging of Alzheimer’s disease (AD), for example, is based on a qualitative evaluation of the distribution pattern of neurofibrillary tangles suggested by Braak et al. [5]. Due to its robustness and plainness, the Braak staging is widely accepted. Nonetheless, the continuously increasing standards in healthcare and neuroscience demand quantifiable and reproducible examinations. Computational approaches based on machine learning are rising for these tasks as they are capable of highly accurate executions of complex analyses. Signaevsky et al. presented a machine learning–based algorithm using automated quantitative analysis of neurofibrillary tangles in immunohistochemically stained slides [18]. Another group evaluated the burden of amyloid beta in AD patients using convolutional neural networks (CNN) on whole slide images. The output of their pipeline generates heat maps for three different amylopathies in AD patients.

Advanced Neuro-oncologic Diagnostics

The resulting scores showed excellent accordance with established scoring systems [19, 22]. An omnipresent disadvantage of histological analysis is boundedness to one level and two dimensions. This limitation was counteracted by a study group from the Center of Imaging Science of the Johns Hopkins University in Baltimore. They published an approach to link two-dimensional postmortem histology to threedimensional (3D) imaging as magnetic resonance imaging, which can be obtained in living patients. By using a CNN, differences in contrast of the examined stains (for tau, amyloid, and myelin), missing data, and artifacts were accommodated and the 3D distribution of neurofibrillary tangles was mapped in the medial temporal lobe [20]. Advancements in 3D reconstruction are an active research field in neuroscience. With ultrastructural tissue modeling, visualization can become much more detailed. Current tissue models assume axons, for instance, to be perfectly cylindric and neglect diameter variations along them. By performing segmentation based on CNN, Abdollahzadeh et al. detected nanoscopic morphological alterations, including changes in the axonal diameter and the density of myelinated axons, five months after traumatic brain injury in mice [1]. With this kind of information, realistic 3D models of the brain tissue can be built to help investigate pathophysiologic processes underlying diseases of the central or peripheral nervous system.

15.3 Advanced Neuro-oncologic Diagnostics

In the past, revolutionary technology developments have influenced tumor diagnostics. Though, novel technologies do not pursue replacing older methods but give a more complete characterization of the individual tumors. In the last years, profiling the methylation pattern of brain tumors has become a particularly powerful tool in neuropathological diagnostics since, by their methylation profile, tumors of the central nervous system can be identified and classified. Furthermore, some methylation patterns allow conclusions for genetic mutations. Because of its high stability across the tumor, methylation profiling can be performed on small biopsy samples. With included information about 850 thousand methylation spots, such an amount

305

306

Digital Neuropathology

of data can only fully be mastered by machine learning–based algorithms. In 2018, Capper et al. presented their landmark work that comprised the introduction of a brain tumor classifier that is fed with highly resolving data of the tumors’ methylome. The classifier is based on a random forest algorithm that consists of 10,000 binary decision trees. Each tree allocates the tumor to one of the defined methylation classes [12]. After aggregating all scores and calibration steps, the output of the classifier consists of a list of class belonging probabilities. For all included entities, a score between 0 and 1 is assigned. For valid classifications, a threshold value of ≥ 0.9 was suggested [9]. The classifier has found broad application since and has become a favored tool for neuropathologists. A significant diagnostic benefit of using the brain tumor classifier was estimated for 12–14% of all cases [8]. Because of its high diagnostic value, methylation profiling was included in the recently released 5th edition of the classification of brain tumors provided by the World Health Organization [23]. By doing so, the so-called WHO CNS5 moves further in advancing the role of molecular tumor characterization. With the newly included entity “high-grade astrocytoma with piloid features,” integration of the classifier in the diagnostic process is actually necessary, as this tumor exhibits broadly overlapping morphological and molecular features compared to other glial tumors and can only be fully demarcated by its methylation profile [17]. The inclusion of computer-assisted evaluation of extensive molecular data into routine diagnostics is the first step into a new generation of tumor pathology with the opportunity to standardize precise oncologic diagnostics and clinical trial inclusion criteria across different centers [9]. Furthermore, the digital nature of methylation data allows broad accessibility and data exchange and calls for continuative computer-assisted analyses. Despite all possibilities of tumor characterization by means of its genome and epigenome, the morphology should always be a part of comprehensive diagnostics as to date, important tumor characteristics regarding the tumor’s anatomy, e.g., intratumoral heterogeneity are completely neglected in molecular approaches. Heterogeneity, however, is a key feature of the most frequent malignant brain tumor in adults, glioblastoma, and possibly at the root of its dismal prognosis. Based on a work by Verhaak et al., different molecular subtypes of glioblastoma were established [21]. However,

In Situ Microscopy in Real Time

with the detection of subclones that comprise the molecular features of different subtypes, it has become apparent that heterogeneity exists within a single tumor [15, 16]. Molecular intratumoral heterogeneity can be indicated by immunohistochemistry in tissue samples [11]. The evaluation of the grade of heterogeneity is not implemented into routine diagnostics, yet. There is, however, evidence that the extent of heterogeneity is of prognostic relevance [11], which entails attempts to quantify heterogeneity and develop a standardized heterogeneity index. Besides the analog evaluation of digitized histological slides, computational approaches have risen to avoid interobserver variance. The utilization of deep learning algorithms, in particular of CNN, has come to the highest popularity. By CNN-based segmentation of glioblastoma slides, a group of the University of South Australia detected key morphological features that were associated with different mutation profiles and prognoses [24]. Although several studies have proven the powerful capacity of CNN models, there are many challenges to translating their usage into clinical practice, including the organization of standardized file formatting and tissue processing, as well as the availability of widespread computational power, and not least, the protection of data privacy.

15.4 In Situ Microscopy in Real Time

Besides new analytical methods of established morphological techniques including standard staining and immunohistochemistry, new techniques for depicting morphology are in development. Thus, neuropathologists together with neurosurgeons, are the pioneers in using intraoperative confocal laser endomicroscopy (CLE). For this method, fluorescent sodium (FNa), one of the few US Food and Drug Administration-approved fluorescent dyes, is applicated intravenously during surgery. FNa accumulates in areas with disrupted blood-brain barrier. With a probe approximately the size of a pen, the neurosurgeon then records digital black-and-white images at a cellular level. This happens in real time and in situ, thus, without the need for tissue resection [13]. With this technology, new possibilities in intraoperative diagnostics and potentially, therapy can arise. As an alternative to the intraoperative frozen-section

307

308

Digital Neuropathology

procedure, neurosurgeons and neuropathologists analyze the tumor tissue together with the aim to find the optimal resection border. This is of high clinical relevance especially for diffusely infiltrating brain tumors since there is evidence that maximal safe tumor resection can prolong the survival time [6]. The communication is set up in tele(neuro)pathological format allowing the consultation of experts in the field of CLE. A huge advantage of this method is the direct feedback and discussion between neurosurgeon and neuropathologist which avoids sampling errors. A prospective study estimated a mean total imaging time of 5.8 minutes per case for CLE, which included a mean of four location spots per patient [13]. This result represents a noteworthy gain of time compared to a single frozen-section biopsy that requires 20 minutes or more for complete processing and interpretation [2]. As the produced CLE images are all digital, there are already attempts for computer-aided diagnostics. Ziebert et al. successfully predicted the three most commonly encountered brain tumors utilizing deep neuronal networks [25]. To allow a broader implementation and to improve the diagnostic quality, the group Isadyyadzanabadi et al. tried an image style transfer that produced real-time tissue examination by CLE with images more similar to the conventional HE stain [10]. A further advantage of this technology is the possibility to adjust the optimal imaging depth position and use the Z-stack function for sequential imaging. By 3D volumetric reconstruction, shape and anatomical relationships could be reproduced in real time allowing the team of neurosurgeons and neuropathologists to identify, e.g., aberrant vessels indicating neoplastic growth and to understand the tissue’s architecture [4]. Besides the diagnostic obvious application, the method of CLE, in theory, could enable the visualization of living tumor cells. A consequential future possible application could be a specific detection of single tumor cells by their metabolism of dyes in real time. This diagnostic aspect goes along with a therapeutic possibility consisting of a combination of tumor cell detection and elimination by, e.g., lasering single tumor cells. With future comprehensive clinical validation studies, CLE, optionally in combination with machine-learning algorithms, has the potential to be integrated into the clinical workflow as a tool for nearly real-time diagnostics [25].

References

15.5 Conclusion The specialty of neuropathology is in a very exciting transition. Computer-assisted tools expand the scientific but also diagnostic workflow. Even though the algorithms improve day by day, they will not replace neuropathologists in a near future. Rather, they will completely change the requirements directed to neuropathologists. With upcoming computational classifiers, for instance, they have to interpret a variety of scores with verification of plausibility. The integration of multimodal results into one diagnosis can be complex and will become even more challenging with constantly arising new methods.

References

1. Abdollahzadeh, A., Belevich, I., Jokitalo, E., Sierra, A., Tohka, J. (2021). DeepACSON automated segmentation of white matter in 3D electron microscopy. Commun Biol 4: 179. doi: 10.1038/s42003-021-01699-w

2. Ackerman, L. V., Ramirez, G. A. (1959). The indications for and limitations of frozen section diagnosis: A review of 1269 consecutive frozen section diagnoses. Br J Surg 46: 336–350. doi: 10.1002/ bjs.18004619806 3. Bailey, P., Cushing, H. (2006). A classification of the tumours of the glioma group on a histogenetic basis, with a correlated study of prognosis. By Percival Bailey and Harvey Cushing. Medium 8vo, pp. 175, with 108 illustrations, 1926. Philadelphia, London, and Montreal: J. B. Lippincott Company. 21s. net. British Journal of Surgery 14: 554– 555. doi: 10.1002/bjs.1800145540

4. Belykh, E., Patel, A. A., Miller, E. J., Bozkurt, B., Yagmurlu, K., Woolf, E. C., et al. (2018). Probe-based three-dimensional confocal laser endomicroscopy of brain tumors: Technical note. Cancer Manag Res 10: 3109–3123. doi: 10.2147/CMAR.S165980

5. Braak, H., Braak, E. (1991). Neuropathological stageing of Alzheimerrelated changes. Acta Neuropathol 82: 239–259. doi: 10.1007/ BF00308809 6. Brown, T. J., Brennan, M. C., Li, M., Church, E. W., Brandmeir, N. J., Rakszawski, K. L., et al. (2016). Association of the extent of resection with survival in glioblastoma: A systematic review and meta-analysis. JAMA Oncol 2: 1460–1469. doi: 10.1001/jamaoncol.2016.1373

309

310

Digital Neuropathology

7. Bundesärztekammer (2021). Ärztestatistik zum 31. Dezember 2020. Bundesärztekammer, www.bundesärztekammer.de 8. Capper, D., Jones, D. T. W., Sill, M., Hovestadt, V., Schrimpf, D., Sturm, D., et al. (2018). DNA methylation-based classification of central nervous system tumours. Nature 555: 469–474. doi: 10.1038/nature26000 9. Capper, D., Stichel, D., Sahm, F., Jones, D. T. W., Schrimpf, D., Sill, M., et al. (2018). Practical implementation of DNA methylation and copynumber-based CNS tumor diagnostics: the Heidelberg experience. Acta Neuropathol 136: 181–210. doi: 10.1007/s00401-018-1879-y

10. Izadyyazdanabadi, M., Belykh, E., Zhao, X., Moreira, L. B., Gandhi, S., Cavallo, C., et al. (2019). Fluorescence image histology pattern transformation using image style transfer. Front Oncol 9: 519. doi: 10.3389/fonc.2019.00519 11. Liesche-Starnecker, F., Mayer, K., Kofler, F., Baur, S., Schmidt-Graf, F., Kempter, J., et al. (2020). Immunohistochemically characterized intratumoral heterogeneity is a prognostic marker in human glioblastoma. Cancers (Basel) 12. doi: 10.3390/cancers12102964

12. Louis, D. N., Perry, A., Wesseling, P., Brat, D. J., Cree, I. A., FigarellaBranger, D., et al. (2021). The 2021 WHO classification of tumors of the central nervous system: A summary. Neuro Oncol 23: 1231–1251. doi: 10.1093/neuonc/noab106 13. Martirosyan, N. L., Eschbacher, J. M., Kalani, M. Y., Turner, J. D., Belykh, E., Spetzler, R. F., et al. (2016). Prospective evaluation of the utility of intraoperative confocal laser endomicroscopy in patients with brain neoplasms using fluorescein sodium: Experience with 74 cases. Neurosurg Focus 40: E11. doi: 10.3171/2016.1.FOCUS15559

14. Mennel, H. (1997). Schriftenreihe der Deutschen Gesellschaft für Geschichte der Nervenheilkunde, vol 3. Emil Kraepelin und die Neuropathologie. Ein Beitrag zum Spannungsfeld zwischen Natur und Geisteswissenschaften in der Psychiatrie. Nissen, G. Badura, F. 15. Parker, N. R., Hudson, A. L., Khong, P., Parkinson, J. F., Dwight, T., Ikin, R. J., et al. (2016). Intratumoral heterogeneity identified at the epigenetic, genetic and transcriptional level in glioblastoma. Sci Rep 6: 22477. doi: 10.1038/srep22477 16. Patel, A. P., Tirosh, I., Trombetta, J. J., Shalek, A. K., Gillespie, S. M., Wakimoto, H., et al. (2014). Single-cell RNA-seq highlights intratumoral heterogeneity in primary glioblastoma. Science 344: 1396–1401. doi: 10.1126/science.1254257 17. Reinhardt, A., Stichel, D., Schrimpf, D., Sahm, F., Korshunov, A., Reuss, D. E., et al. (2018). Anaplastic astrocytoma with piloid features, a novel

References

molecular class of IDH wildtype glioma with recurrent MAPK pathway, CDKN2A/B and ATRX alterations. Acta Neuropathol 136: 273–291. doi: 10.1007/s00401-018-1837-8

18. Signaevsky, M., Prastawa, M., Farrell, K., Tabish, N., Baldwin, E., Han, N., et al. (2019). Artificial intelligence in neuropathology: Deep learningbased assessment of tauopathy. Lab Invest 99: 1019–1029. doi: 10.1038/s41374-019-0202-4

19. Tang, Z., Chuang, K. V., DeCarli, C., Jin, L. W., Beckett, L., Keiser, M. J., et al. (2019). Interpretable classification of Alzheimer’s disease pathologies with a convolutional neural network pipeline. Nat Commun 10: 2173. doi: 10.1038/s41467-019-10212-1 20. Tward, D., Brown, T., Kageyama, Y., Patel, J., Hou, Z., Mori, S., et al. (2020). Diffeomorphic registration with intensity transformation and missing data: Application to 3D digital pathology of Alzheimer’s disease. Front Neurosci 14: 52. doi: 10.3389/fnins.2020.00052

21. Verhaak, R. G., Hoadley, K. A., Purdom, E., Wang, V., Qi, Y., Wilkerson, M. D., et al. (2010). Integrated genomic analysis identifies clinically relevant subtypes of glioblastoma characterized by abnormalities in PDGFRA, IDH1, EGFR, and NF1. Cancer Cell 17: 98–110. doi: 10.1016/j. ccr.2009.12.020 22. Vizcarra, J. C., Gearing, M., Keiser, M. J., Glass, J. D., Dugger, B. N., Gutman, D. A. (2020). Validation of machine learning models to detect amyloid pathologies across institutions. Acta Neuropathol Commun 8: 59. doi: 10.1186/s40478-020-00927-4

23. WHO Classification of Tumours Editorial Board (2021). WHO Classification of Tumours - Central Nervous System Tumours. 5th edn. International Agency for Research on Cancer, Lyon, France. 24. Zadeh Shirazi, A., McDonnell, M. D., Fornaciari, E., Bagherian, N. S., Scheer, K. G., Samuel, M. S., et al. (2021). A deep convolutional neural network for segmentation of whole-slide pathology images identifies novel tumour cell-perivascular niche interactions that are associated with poor survival in glioblastoma. Br J Cancer 125: 337–350. doi: 10.1038/s41416-021-01394-x

25. Ziebart, A., Stadniczuk, D., Roos, V., Ratliff, M., von Deimling, A., Hanggi, D., et al. (2021). Deep neural network for differentiation of brain tumor tissue displayed by confocal laser endomicroscopy. Front Oncol 11: 668273. doi: 10.3389/fonc.2021.668273

311

Chapter 16

Application of Artificial Intelligence in Gastrointestinal Endoscopy

Alanna Ebigbo, Friederike Prinz, Michael Meinikheim, Markus Scheppach, and Helmut Messmann Department of Gastroenterology, University Hospital Augsburg, Augsburg, Germany [email protected]

Artificial intelligence (AI) has gained tremendous momentum in the medical domain in the past few years. Various AI applications are currently the focus of intensive research with the ultimate goal of improving the quality of patient care. AI can have a wide range of applications in gastrointestinal endoscopy, especially in detecting and classifying neoplastic lesions. In this manuscript, we will discuss and illustrate the current applications of AI in different locations of the gastrointestinal tract.

Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

314

Application of Artificial Intelligence in Gastrointestinal Endoscopy

16.1 Introduction In gastrointestinal (GI) endoscopy, AI can have a wide range of applications. Most AI applications and research studies have focused on malignant or neoplastic GI diseases. The most common examples include the detection and classification of polyps, adenomas, or carcinomas in the colon during screening colonoscopy. For these indications, various devices have already achieved approval by government authorities [Food and Drug Administration (FDA), Conformité Européenne (CE)]. However, AI can be applied to benign or non-neoplastic conditions in the GI tract, such as diagnosing Helicobacter pylori infection. For non-expert endoscopists, some disease entities may be challenging to recognize and interpret correctly. The potential of AI may be the ability to improve the performance of endoscopic procedures on a broad scale. AI applications can be subdivided into tasks or assignments based on clinical challenges which physicians face in everyday practice. AI models can be trained to detect and highlight a region of interest during routine endoscopy. A classic example is the AI-assisted detection of polyps or adenomas during screening colonoscopy or the detection of dysplasia in Barrett’s esophagus. Improved adenoma or dysplasia detection rates could lead to earlier diagnosis of precancerous and early cancerous lesions, thereby improving the prognosis of such diseases. Computer-assisted optical diagnosis has the potential to reduce the quantity of unnecessary random biopsies and can also be applied in the diagnose-and-leave strategy for diminutive colorectal polyps. In addition to detection, AI models can characterize lesions by differentiating, for example, between neoplasia and non-neoplasia. The classification task could also involve other aspects of a lesion’s morphology, such as its invasion depth, which could significantly impact the therapeutic process. AI models have also been applied in the segmentation or delineation of a GI lesion’s outer border. This feature can benefit experts during endoscopic resection and non-experts during targeted biopsies. Most AI models designed for use in the GI tract are based on convolutional neural networks (CNNs). CNNs are deep-learning architectures with deep networks of 100 layers or more. If trained

Esophagus

with a sufficient quantity and diversity of data, these Residual Nets have the potential to reach a very high accuracy in image analysis. The hope is that AI in GI endoscopy will improve the quality of diagnostic and therapeutic procedures and, ultimately, patient care. Quality performance and other tasks that are currently undergoing intensive development include AI-assisted assessment of completeness of mucosa inspection during endoscopy, automated assessment of the level of cleansing at colonoscopy, and grading inflammatory activity in patients with inflammatory bowel disease (IBD). Also, AI support of endoscopic workflows with Natural Language Software (NLS), which involves automated extraction of clinically relevant information from medical records, will go a long way to improve the quality of endoscopic procedures. This chapter will focus on the current applications of AI as a clinical decision support system in the upper and lower GI tract. In addition, we will give an outlook on future perspectives and developments of AI in GI endoscopy.

16.2 Esophagus

The incidence of esophageal cancer in the West, particularly esophageal adenocarcinoma (EAC), is rising [9, 11]. The reason for this trend is the association between EAC and metabolic syndrome, including obesity and diabetes mellitus. The development of AI systems in the esophagus has mainly concentrated on detecting and characterizing early esophageal cancer as well as precancerous conditions such as Barrett’s esophagus (BE). BE is a precancerous condition that can lead to esophageal adenocarcinoma (EAC) [6]. Subtle regions of dysplasia or cancer within Barrett’s epithelium can be challenging to detect and characterize, even for expert endoscopists (Fig. 16.1) [59]. To simplify the detection of dysplastic lesions within BE, the application of acetic acid (AA) has proven to be an easy and accessible tool for endoscopists during endoscopy. When applied to Barrett’s mucosa, it changes color from typical pink to white [33]. The so-called dewhitening sign can indicate the presence of neoplasia [41]. Kandiah et al. developed the Portsmouth acetic acid classification or PREDICT system to detect Barrett’s dysplasia using

315

316

Application of Artificial Intelligence in Gastrointestinal Endoscopy

AA PREDICT improved the sensitivity and negative predictive value of endoscopists significantly [32]. An international working group, the BING working group, proposed another classification system for BE assessment based on narrow-band imaging (NBI). The BING criteria improved the identification of dysplasia and EAC with high levels of accuracy, sensitivity, specificity, and positive and negative predictive values [59]. Various studies of AI systems for detecting and classifying focal Barrett’s lesions have shown excellent results, even in realtime assessment [14–16, 23, 19]. However, novel endoscopic technologies are expected to achieve specific performance measures. The PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) of the American Society for Gastrointestinal Endoscopy described thresholds for Barrett’s evaluation, including specificity of at least 80%, per-patient sensitivity of at least 90%, and a negative predictive value of at least 98% for detecting high-grade dysplasia or EAC compared with the current standard protocol of random biopsies [60].

Figure 16.1 Early esophageal adenocarcinoma within Barrett’s esophagus in white light (a), NBI (b), and magnification endoscopy (c).

A Japanese research group investigated the diagnostic outcome of esophageal cancer using CNNs. The CNN model detected esophageal cancer with a sensitivity of 98% and distinguished successfully between superficial esophageal cancer from advanced cancer with an accuracy of 98% [26]. De Groof et al. developed and validated a deep-learning CAD system to detect neoplastic lesions in BE with an accuracy, sensitivity, and specificity of 88%, 93%, and 83%, respectively. The AI model outperformed even expert endoscopists with accuracy, sensitivity, and specificity of 73%, 74%, and 72%, respectively. A relevant

Esophagus

aspect was to demonstrate the segmentation task by identifying the optimal spot for a biopsy using the AI system [15]. In addition, the CAD system was used in real time and detected neoplastic lesions with high accuracy [14]. A German study group developed an AI system to detect, delineate, and characterize high-grade dysplasia and EAC in Barrett’s esophagus. The system achieved excellent sensitivity and specificity on still images using white-light and NBI [16]. In a follow-up study in a real-time setting, the AI system detected and characterized early neoplasia with an accuracy of 89.9% [19]. A further investigation concentrated on predicting the submucosal invasion depth of esophageal cancer. The differentiation between mucosal and submucosal cancers influences the endoscopic resection modality. In the pilot study, the AI model showed a similar accuracy between the AI model and expert endoscopists [17]. The same study group showed that the AI system detects relatively large lesions (20 mm) and tiny islands of high-grade dysplasia within BE. In particular, they showed that the system detects lesions in white-light endoscopy and other modalities such as narrow-band and texture and color enhancement imaging [18].

Figure 16.2 Heatmaps and AI visualizations of early Barrett’s cancer using multimodal imaging.

The standardized reporting and measurement of the length of BE segments are usually done with the Prague classification. The circular and maximal lengths are reported with the C&M scores. However, the Prague classification is operator-dependent and prone to subjective errors or inaccuracies due to scope positioning. A British study group developed an AI-based method that quantifies BE areas, including BE-islands, by three-dimensional (3D) reconstruction of the esophageal surface, enabling retrospective 3D visualization and post-endoscopic diagnosis [4].

317

318

Application of Artificial Intelligence in Gastrointestinal Endoscopy

Recently, AI systems for the detection and characterization of focal lesions in BE have successfully undergone CE approval. NEC Corporation developed a system that supports physicians during live endoscopy. The system takes a still image as soon as potential neoplasia occurs, places it in the right upper corner of the screen, and provides a heat map of the endoscopic image with a diagnostic prediction. WISE VISION® Endoscopy detected over 90% of Barrett’s dysplasia and is supposed to reduce the overall neoplasia miss rate, especially for non-Barrett’s experts (NEC Corporation, WISE VISION Endoscopy). A second system, developed by Odin Vision and named CADU, also provides information about dysplasia and EAC during endoscopic examinations. The system performs computer-assisted detection and marks the borders to the non-dysplastic area, thereby specifying the resection area (Odin Vision, CADU Artificial Intelligence®). Computer-assisted diagnosis and AI applications in BE could in the future improve early diagnosis of dysplasia and early adenocarcinoma in the esophagus, thereby improving patient prognosis and overall survival.

16.3 Stomach

Current research into AI in gastric endoscopy focuses on the recognition of cancer and its precursors. As this entails pattern recognition on classified and sometimes annotated endoscopic images, CNNs are most often used for this task. After development, these can detect, classify, and delineate gastric mucosal lesions and thus assist the medical practitioner in clinical decision-making. An infection of the stomach with the bacterium Helicobacter pylori (HP) often leads to chronic inflammation of the mucosa and is a prevalent factor in gastric carcinogenesis [13, 52]. Several research groups developed deep-learning algorithms to detect an HP infection on endoscopic still images of the gastric mucosa. These showed impressive performance with a sensitivity and specificity of more than 90% and 80%, respectively, and surpassed less experienced endoscopists in their diagnostic accuracy [29, 50, 61, 62, 84, 89]. The chronic inflammatory process in the gastric mucosa can lead to precursor lesions of cancer, such as mucosal atrophy and

Stomach

intestinal metaplasia. The former condition is characterized by a loss of the mucosal layer’s intricate architecture, which can cause resorption deficiencies such as pernicious anemia. In contrast, the mucosal structure assumes a small or large intestinal phenotype in the latter. These changes hinder the visual detection of existing neoplasms while at the same time increasing the risk of malignant transformation [48]. Zhang et al. [87] developed a neural network, which recognized mucosal atrophy on endoscopic pictures with a sensitivity of 94.5% and a specificity of 94.0%. An algorithm by Yan et al. [83] detected intestinal metaplasia of the stomach with a sensitivity of 91.9% and a specificity of 86.0%. None of the two studies showed a statistically significant difference between the AI algorithm and the individual evaluation by experts. These preclinical data suggest a clinical benefit, especially in endoscopic training, which would have to be substantiated by further clinical research. Stomach cancer is often diagnosed at an advanced state of disease progression, which leads to a poor prognosis [48]. The macroscopic identification of early gastric neoplasias during upper gastrointestinal (GI) endoscopy is challenging since the tumors are hardly distinguishable from inflammatory altered mucosa or the aforementioned precancerous conditions. To date, efforts to ameliorate this obstacle concentrated on higher resolution imaging and optical contrast enhancement (virtual chromoendoscopy). These methods verifiably improved the diagnostic quality and are recommended in national guidelines [48]. Still, today’s diagnostic accuracy is not yet optimal. According to a meta-analysis by Menon et al. [45], early gastric cancer (EGC) was missed in 11.3% of endoscopies that had been performed in the three years preceding the final diagnosis of malignoma. On the other hand, early forms of stomach cancer with infiltration of only the mucosal layer or with the superficial invasion of the submucosal layer of less than 500 µm can, under certain conditions, be resected with curative intention endoscopically. Because of this potential to reduce disease burden, the development of artificial intelligence clinical decision support solutions (AI-CDSS) seems especially important. A neural network by Hirasawa et al. [25] detected EGC with a sensitivity of 92.2% and a positive predictive value (PPV) of 30.6%. In another study by

319

320

Application of Artificial Intelligence in Gastrointestinal Endoscopy

Ikenoyama et al. [27], the same algorithm was compared with the assessment of 67 endoscopists with different levels of experience using an external dataset. The sensitivity of the algorithm and the pooled endoscopists was 58.4% vs. 31.9%, while the PPV and the specificity were 26.0% vs. 46.2% and 87.3% vs. 97.2%. The differences showed statistical significance. In a “mega trial” (>1 000 000 images of >80 000 patients), Luo et al. [42] created an AI algorithm for endoscopic carcinoma detection, which corresponded to expert endoscopists but outperformed less experienced endoscopists in diagnostic accuracy. Further research endeavors aimed at predicting the invasion depth of EGC based on endoscopic images. For this task, the technique of endoscopic ultrasound (EUS) is established in Europe [53]. In Eastern Asia, the standardized analysis of surface and especially vascular patterns of the tumor are also used to estimate tumor ingrowth [1]. An algorithm by Yoon et al. [85] could predict the invasion depth of EGC lesions with a sensitivity of 79.2% and a specificity of 77.8%. Zhu et al. [90] could predict tumor invasion with a sensitivity of 76.5% and a specificity of 95.6% via an AI algorithm, compared to values of 90.8% and 70.7% for the same parameters by experienced endoscopists. The difference in specificity reached statistical significance. While these results show the exciting prediction accuracy of tumor ingrowth by analyzing the visible tumor surface, a head-to-head comparison with the current standard of EUS has not been undertaken yet. Another approach to the improvement of detection rates of gastric carcinomas, but also other abnormalities (e.g., ulcer, angiodysplasia) is the WISENSE system (“wise sense”) by Wu et al. [81, 79]. This deep-learning algorithm is employed to detect in real time the mucosal surface areas not visualized at endoscopy (blind spots). The application can also distinguish EGC from non-neoplastic gastric mucosa with high accuracy of 92.5%. In a prospective clinical trial, WISENSE reduced the rate of blind spots with statistical significance by 15.4% to 5.9%. In 2020, the same research group conducted the first prospective randomized trial to investigate the clinical benefit of AI assistance in the endoscopic diagnostics of gastric cancer. In a tandem approach, where every patient is examined successively with and without the AI system, the rate of missed neoplasias was significantly reduced by 27.3% to 6.1% (relative risk 0.224) [80].

Small Intestine

In summary, deep-learning algorithms show high potential in the detection of stomach cancer and its precursor lesions. Most of the research to date is preclinical. First clinical results suggest a relevant improvement in medical care by the application of AI. At present, none of the above-mentioned AI-CDSS is accredited by the authorities and available for daily practice.

16.4 Small Intestine

The upper (esophagus, stomach, duodenum) and lower (colon, rectum, terminal ileum) parts of the gastrointestinal tract are easily accessible to conventional endoscopy. This does not apply to the jejunum and ileum due to their combined length of approximately five meters and their distance to natural orifices of the body. The preferred non-invasive tool to diagnose small intestinal lesions beyond the reach of conventional scopes is capsule endoscopy which has been brought to perfection over the last 20 years. The procedure involves swallowing a capsule measuring 11 mm × 26 mm with a small camera. As the device travels through the gastrointestinal tract, it takes about 80,000 high-definition pictures and passes naturally via the rectum. Digital images are transmitted to an external recorder, loaded on a computer workstation, and analyzed later by the gastroenterologist. As the small intestinal transit time may be five hours or more, the data analysis is time-consuming. As a target lesion (e.g., a bleeding source) may be visible on no more than 2–3 images, it can easily be overlooked. In this context, the strengths of AI are the standardized recognition of structures (i.e., vessels) and the rapid handling of big data. Thus, an AI application may prove extremely helpful in the automated evaluation of CE images. Tumors of the small intestine, such as adenocarcinomas, neuroendocrine tumors, and malignant stromal tumors, are rare (0.6% of all tumors and 3.3% of all gastrointestinal tumors [63]) and often difficult to diagnose. In one study, Barbosa et al. [7] created a convolutional neural network using CE pictures, which showed a sensitivity of 98.7% and a specificity of 96.6% for the detection of primary small intestinal tumors. The technique of support vector machines was also successfully implemented by other authors [39, 71]. Inoue et al. [28] used conventional endoscopic pictures to create

321

322

Application of Artificial Intelligence in Gastrointestinal Endoscopy

an algorithm that could detect non-ampullary duodenal epithelial tumors with an accuracy of 94.7%. Celiac disease is an autoimmune disorder of the small intestine characterized by an inflammatory response to ingested gluten. Vague symptoms without pathognomonic features make it a chameleon in gastroenterology. The diagnosis is made from serologic testing, the macroscopic and microscopic examination of duodenal mucosa, and clinical improvement following the consumption of a strictly gluten-free diet [58]. Different groups have used AI to improve the detection of villous atrophy, both in conventional endoscopy and CE. Gadermayr et al. [20] combined multi-resolution local binary patterns, a multi-fractal spectrum, and improved Fisher vectors with expert knowledge to predict microscopic villous atrophy from endoscopic images under particular circumstances with an accuracy of 94 to 100%. Other studies also used Fisher’s encoding of CNNs to spot villous atrophy on water-immersed images [77]. Crohn’s disease is another autoimmune disorder that can affect discontinuously every part of the digestive system. A circumscript manifestation confined to the small intestine is possible [54]. A deep-learning algorithm by Klang et al. [34] showed an accuracy of 94–99% for the detection of Crohn’s disease-associated ulcers on CE pictures. AI and in particular CNNs have also demonstrated their usefulness in the image analysis and identification of other abnormalities in the small intestine, such as protruding lesions, mucosal breaks, vessel malformations (angioectasias), and blood content [5, 69]. In summary, AI seems suited to assist a medical practitioner in analyzing small intestinal imaging data. While to date, no application is accredited and confirmatory clinical trials are missing, it can be hypothesized that AI systems may provide convenient support, reduce false-negative findings, and speed up processing time simultaneously.

16.5 Colon

Colorectal cancer is one of the most frequently diagnosed malignancies and the second leading cause of cancer-associated

Colon

deaths [65]. Colorectal polyps are considered precancerous lesions and are associated with an increased risk of colorectal cancer. Therefore, the detection and proper characterization of polyps during colonoscopy are crucial for prognosis. Endoscopic polypectomy prevents the progression of adenomas to carcinomas, thus reducing the risk of colorectal cancer [78, 86]. The adenoma detection rate (ADR) is one way of monitoring the overall performance of colonoscopy [12, 31]. ADR is inversely proportional to the risk for interval colorectal cancer, more advanced disease, and disease outcome [12, 31]. Various expert societies have agreed on a minimum required ADR as a marker of high-quality colonoscopy [57]. Despite these recommendations, the miss rates for adenomas and serrated polyps, range between 9% and 27% and are considerably higher than previously anticipated [88]. In some cases, up to 77.4% of colorectal cancers were attributed to failure to detect or correctly characterize lesions during index colonoscopies [37]. Factors like physician fatigue, inexperience, distraction during procedures, time pressure, or poor bowel preparation might contribute to missed adenomas. In an attempt to eliminate some of these human factors and ultimately improve the outcome for patients, AI systems for colonoscopy have been developed. With deep-learning algorithms, AI systems have focused on colorectal adenoma detection (CADe) and characterization (CADx) by implementing machine learning methods including CNNs and support vector machines (SVM) [51]. CNN uses semi-supervised learning by applying labeled and unlabeled data [43]. SVM uses labeled data exclusively, and this is termed supervised learning. CNN can automatically learn optimum features of the provided dataset, while SVM, on the other hand, needs manual extraction of the given features [3, 73]. CNN outperforms traditional machine learning when large datasets are available, while SVM surpasses CNN when only limited data sets are available [73]. In recent years, CNN has become the standard method for medical image classification. One of the reasons might be that CNN seems to be more robust when it comes to variations in the quality of images [44].

323

324

Application of Artificial Intelligence in Gastrointestinal Endoscopy

16.6 Computer-Aided Detection of Polyps AI systems detect polyps based on pattern recognition and tag them, for example, with bounding boxes during colonoscopy. Initially, several preclinical studies showed the feasibility of adenoma detection based on still images with a sensitivity and specificity of more than 90% [8, 82]. Misawa et al. demonstrated the feasibility of real-time polyp detection on colonoscopy videos with a sensitivity of 90% and a specificity of 63.3% [46]. In a pioneer work, Urban et al. developed an AI polyp detection system based on 8641 labeled images. They were one of the first to demonstrate CADe in real time based on video images of endoscopy procedures with an accuracy of 96.4% [70]. A major concern was the extrapolation of CADe performance on video images to real-life scenarios. As a result, single-center trials conducted by Wang et al. [74–76] and a multicenter randomized trial conducted by Repici et al. [55, 56] in real-life procedures were undertaken and have produced excellent results with significantly higher ADR of AI compared with human endoscopists. Repici et al. observed that regardless of the endoscopists’ level of experience, support by AI seems to be beneficial [55]. However, the significantly higher ADR of AI was limited to adenomas ≤9 mm [73]. It remains unclear whether the detection and removal of diminutive (1–5 mm) polyps will lead to a significantly better patient outcome since only a small percentage present as adenomas or have the potential to progress to colorectal cancer [72]. Nevertheless, systemic reviews and meta-analyses have concluded that endoscopists with the support of AI performed better during colonoscopy [38, 24]. As a limitation, some trials included endoscopist performance below the minimum ADR standard of at least 25%, which professional societies like the European Society of Gastrointestinal Endoscopy and United European Gastroenterology recommend [21, 30, 40, 64].

16.7 Computer-Aided Characterization (CADX) of Polyps

CADx is based on information generated during advanced imaging procedures with magnified virtual chromoendoscopy such as NBI.

Computer-Aided Characterization (CADX) of Polyps

In 2010, Tischendorf et al. conducted a pilot study on 209 polyps [68] and successfully differentiated between neoplastic and nonneoplastic lesions with a sensitivity of 90% and a specificity of 70% [68]. However, in this pilot study, the AI system was inferior to the control group which consisted of two experts. [68] In a follow-up study based on images of 434 polyps, the AI model was superior to non-expert endoscopists and comparable to experts in the classification task of neoplastic vs. non-neoplastic lesions [22]. Takemura et al. approached the characterization task with an SVM algorithm based on pit pattern recognition while in magnification mode [67]. However, this system was semi-automatic and could not provide an immediate output [67]. In a follow-up study on retrospectively collected images, the AI model was able to classify neoplastic lesions (type B-C3 according to Hiroshima classification) with a sensitivity of 97.8% and specificity of 97.9% [66]. After the initial trials, the necessity of preprocessing was a major limitation of the system; a subsequent real-time trial achieved a sensitivity of 93% and a specificity of 93.3% [35]. In contrast to other systems, Byrne and colleagues developed an AI system with overview NBI images [10]. In a retrospective study on unaltered videos, they tested the capacity of their deep convolutional neural network (DCNN) to differentiate diminutive adenomas from hyperplastic lesions. For “high confidence” classifications, their AI system achieved a sensitivity of 98% and a specificity of 83% [10]. Misawa and colleagues attempted a different approach by combining NBI with endocytoscopy. Endocytoscopy is a method that allows an extreme magnification of the observed structure and, therefore, a detailed assessment of the respective tissue [47]. The research group based their AI system on an SVM algorithm that differentiated neoplastic from non-neoplastic lesions with an overall sensitivity of 84.5% and a specificity of 97.6% with a near real-time processing speed of 0.3 seconds [47]. In a follow-up prospective trial consisting of 466 polyps ≤5mm from 325 patients, their system differentiated neoplastic from non-neoplastic lesions in ultramagnifying NBI and methylene blue staining mode with a negative predictive value of more than 90% in real-time [49]. Eventually, after an image-based multicenter comparative analysis where their AI system outperformed 30 endoscopists significantly, their AI system was approved for clinical use [36].

325

326

Application of Artificial Intelligence in Gastrointestinal Endoscopy

CADx systems intend to support the endoscopist while differentiating neoplastic from non-neoplastic polyps. For official approval and clinical implementation of AI systems, the guidelines for PIVI established by the American Society for Gastrointestinal Endoscopy (ASGE) can serve as an orientation for the minimum requirements which these systems need to achieve [2]. For example, PIVI recommends that during the real-time assessment of diminutive polyps in the rectosigmoid colon, new devices should have at least a 90% NPV when the polyp is evaluated as nonneoplastic, and a “diagnose-and-leave” strategy is intended [2]. The PIVI recommendations will preserve and improve the standard and quality of patient care. Implementing AI during colonoscopy harbors a tremendous potential for the patients’ benefit and clinical support for physicians caring for them. The cost-effectiveness of AI systems will ultimately improve the quality of care within the healthcare system. Several AI systems, for example, GI Genius, Endo-AID, or Discovery, have already been approved for clinical use and are commercially available for adenoma detection. Other clinical applications of AI in the colon, including computer-aided assessment of bowel cleansing or automated navigation of withdrawal speed, have been described in research trials and will increase the portfolio of AI tools at the endoscopist’s disposal. AI can provide a second opinion during complex cases and support endoscopists during surveillance colonoscopies. With steadily improving endoscopy systems, the threshold to high-quality data is lower than ever, providing even higher quality data to enhance existing and develop novel and innovative AI systems in the future.

16.8 Conclusion

AI and CAD have shown incredible potential in GI endoscopy and have been applied in all regions of the GI tract. In the future, AI models will assist physicians routinely during basic and complex endoscopic procedures. Cost-effective clinical decision support tools based on deep-learning architectures will improve disease diagnosis, patient care and management, and, ultimately, disease prognosis and overall survival.

References

References 1. Abe, S., et al. (2011), Depth-predicting score for differentiated early gastric cancer. Gastric Cancer, 2011. 14(1): pp. 35–40.

2. Abu Dayyeh, B. K., et al. (2015, ASGE Technology Committee systematic review and meta-analysis assessing the ASGE PIVI thresholds for adopting real-time endoscopic assessment of the histology of diminutive colorectal polyps. Gastrointest Endosc, 2015. 81(3): pp. 502.e1–502.e16. 3. Ahmad, O. F., et al. (2019), Artificial intelligence and computer-aided diagnosis in colonoscopy: Current evidence and future directions. Lancet Gastroenterol Hepatol, 2019. 4(1): pp. 71–80. 4. Ali, S., et al. (2021), A pilot study on automatic three-dimensional quantification of Barrett’s esophagus for risk stratification and therapy monitoring. Gastroenterology, 2021. 161(3): pp. 865–878.e8. 5. Aoki, T., et al. (2021), Automatic detection of various abnormalities in capsule endoscopy videos by a deep learning-based system: a multicenter study. Gastrointest Endosc, 2021. 93(1): pp. 165–173.e1.

6. Arnold, M., et al. (2017), Predicting the future burden of esophageal cancer by histological subtype: International trends in incidence up to 2030. Am J Gastroenterol, 2017. 112(8): pp. 1247–1255.

7. Barbosa, D.J., J. Ramos, and C.S. Lima (2008), Detection of small bowel tumors in capsule endoscopy frames using texture analysis based on the discrete wavelet transform. Annu Int Conf IEEE Eng Med Biol Soc, 2008. pp. 3012–3015. 8. Billah, M., Waheed, S. and Rahman, M.M. (2017), An automatic gastrointestinal polyp detection system in video endoscopy using fusion of color wavelet and convolutional neural network features. Int J Biomed Imaging, 2017. p. 9545920.

9. Bray, F., et al. (2018), Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin, 2018. 68(6): pp. 394–424.

10. Byrne, M. F., et al. (2019), Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut, 2019. 68(1): pp. 94–100.

11. Coleman, H. G., Xie, S.-H. and J. Lagergren, J. (2018), The epidemiology of esophageal adenocarcinoma. Gastroenterology, 2018. 154(2): pp. 390–405.

327

328

Application of Artificial Intelligence in Gastrointestinal Endoscopy

12. Corley, D. A., et al. (2012), Adenoma detection rate and risk of colorectal cancer and death. New England Journal of Medicine, 2014. 370(14): pp. 1298–1306. 13. Correa, P. and M.B. Piazuelo (2012), The gastric precancerous cascade. J Dig Dis, 2012. 13(1): pp. 2–9.

14. de Groof, A. J., et al. (2018), Deep learning algorithm detection of Barrett’s neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video). Gastrointest Endosc, 2020. 91(6): pp. 1242–1250. 15. de Groof, A. J., et al. (2020), Deep-learning system detects neoplasia in patients with Barrett’s esophagus with higher accuracy than endoscopists in a multistep training and validation study with benchmarking. Gastroenterology, 2020. 158(4): pp. 915–929.e4.

16. Ebigbo, A., et al. (2019), Computer-aided diagnosis using deep learning in the evaluation of early oesophageal adenocarcinoma. Gut, 2019. 68(7): pp. 1143–1145. 17. Ebigbo, A., et al. (2021), Endoscopic prediction of submucosal invasion in Barrett’s cancer with the use of artificial intelligence: a pilot study. Endoscopy, 2021. 53(9): pp. 878–883.

18. Ebigbo, A., et al. (2021), Multimodal imaging for detection and segmentation of Barrett’s esophagus-related neoplasia using artificial intelligence. Endoscopy, 2021. 19. Ebigbo, A., et al. (2020), Real-time use of artificial intelligence in the evaluation of cancer in Barrett’s oesophagus. Gut, 2020. 69(4): pp. 615–616.

20. Gadermayr, M., et al. (2016), Computer-aided texture analysis combined with experts’ knowledge: Improving endoscopic celiac disease diagnosis. World J Gastroenterol, 2016. 22(31): pp. 7124–7134. 21. Gong, D., et al. (2020), Detection of colorectal adenomas with a realtime computer-aided system (ENDOANGEL): a randomised controlled study. Lancet Gastroenterol Hepatol, 2020. 5(4): pp. 352–361.

22. Gross, S., et al. (2011), Computer-based classification of small colorectal polyps by using narrow-band imaging with optical magnification. Gastrointest Endosc, 2011. 74(6): pp. 1354–1359.

23. Hashimoto, R., et al. (2020), Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett’s esophagus (with video). Gastrointest Endosc, 2020. 91(6): pp. 1264–1271.e1.

References

24. Hassan, C., et al. (2021), Performance of artificial intelligence in colonoscopy for adenoma and polyp detection: A systematic review and meta-analysis. Gastrointest Endosc, 2021. 93(1): pp. 77–85.e6. 25. Hirasawa, T., et al. (2018), Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer, 2018. 21(4): pp. 653–660.

26. Horie, Y., et al. (2019), Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks. Gastrointest Endosc, 2019. 89(1): pp. 25–32. 27. Ikenoyama, Y., et al. (2021), Detecting early gastric cancer: Comparison between the diagnostic ability of convolutional neural networks and endoscopists. Dig Endosc, 2021. 33(1): pp. 141–150. 28. Inoue, S., et al. (2020), Application of convolutional neural networks for detection of superficial nonampullary duodenal epithelial tumors in esophagogastroduodenoscopic images. Clin Transl Gastroenterol, 2020. 11(3): p. e00154.

29. Itoh, T., et al. (2018), Deep learning analyzes Helicobacter pylori infection by upper gastrointestinal endoscopy images. Endosc Int Open, 2018. 6(2): pp. e139–e144. 30. Kaminski, M. F., et al. (2017), Performance measures for lower gastrointestinal endoscopy: A European Society of Gastrointestinal Endoscopy (ESGE) quality improvement initiative. United European Gastroenterol J, 2017. 5(3): pp. 309–334.

31. Kaminski, M. F., et al. (2010), Quality indicators for colonoscopy and the risk of interval cancer. N Engl J Med, 2010. 362(19): pp. 1795–1803. 32. Kandiah, K., et al. (2018), International development and validation of a classification system for the identification of Barrett’s neoplasia using acetic acid chromoendoscopy: The Portsmouth acetic acid classification (PREDICT). Gut, 2018. 67(12): pp. 2085–2091. 33. Kaufman, H. B. and Harper, D. M. (2004), Magnification and chromoscopy with the acetic acid test. Endoscopy, 2004. 36(8): pp. 748–750; author reply 751. 34. Klang, E., et al. (2020), Deep learning algorithms for automated detection of Crohn’s disease ulcers by video capsule endoscopy. Gastrointest Endosc, 2020. 91(3): pp. 606–613.e2.

35. Kominami, Y., et al. (2016), Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy. Gastrointest Endosc, 2016. 83(3): pp. 643–649.

329

330

Application of Artificial Intelligence in Gastrointestinal Endoscopy

36. Kudo, S. E., et al. (2020), Artificial intelligence-assisted system improves endoscopic identification of colorectal neoplasms. Clin Gastroenterol Hepatol, 2020. 18(8): pp. 1874–1881.e2. 37. le Clercq, C. M., et al. (2014), Postcolonoscopy colorectal cancers are preventable: A population-based study. GUT, 2014. 63(6): pp. 957– 963.

38. Li, J., et al. (2021), Artificial intelligence can increase the detection rate of colorectal polyps and adenomas: A systematic review and metaanalysis. Eur J Gastroenterol Hepatol, 2021. 33(8): pp. 1041–1048. 39. Liu, G., et al. (2016), Detection of small bowel tumor based on multiscale curvelet analysis and fractal technology in capsule endoscopy. Comput Biol Med, 2016. 70: pp. 131–138.

40. Liu, W. N., et al. (2020), Study on detection rate of polyps and adenomas in artificial-intelligence-aided colonoscopy. Saudi J Gastroenterol, 2020. 26(1): pp. 13–19.

41. Longcroft-Wheaton, G., et al. (2013), Duration of acetowhitening as a novel objective tool for diagnosing high risk neoplasia in Barrett’s esophagus: a prospective cohort trial. Endoscopy, 2013. 45(6): pp. 426–432.

42. Luo, H., et al. (2019), Real-time artificial intelligence for detection of upper gastrointestinal cancer by endoscopy: A multicentre, casecontrol, diagnostic study. Lancet Oncol, 2019. 20(12): pp. 1645–1654.

43. Marini, N., et al. (2021), Semi-supervised training of deep convolutional neural networks with heterogeneous data and few local annotations: An experiment on prostate histopathology image classification. Medical Image Analysis, 2021. 73: p. 102165. 44. Maruyama, T., et al. (2018), Comparison of medical image classification accuracy among three machine learning methods. J Xray Sci Technol, 2018. 26(6): pp. 885–893. 45. Menon, S. and Trudgill, N. (2014), How commonly is upper gastrointestinal cancer missed at endoscopy? A meta-analysis. Endosc Int Open, 2014. 2(2): pp. E46–E50.

46. Misawa, M., et al. (2018), Artificial intelligence-assisted polyp detection for colonoscopy: Initial experience. Gastroenterology, 2018. 154(8): pp. 2027–2029.e3.

47. Misawa, M., et al. (2016), Characterization of colorectal lesions using a computer-aided diagnostic system for narrow-band imaging endocytoscopy. Gastroenterology, 2016. 150(7): pp. 1531–1532.e3.

References

48. Moehler, M., et al., Z Gastroenterol, 2019. 57(12): pp. 1517–1632.

49. Mori, Y., et al. (2018), Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: A prospective study. Ann Intern Med, 2018. 169(6): pp. 357–366.

50. Nakashima, H., et al. (2018), Artificial intelligence diagnosis of Helicobacter pylori infection using blue laser imaging-bright and linked color imaging: A single-center prospective study. Ann Gastroenterol, 2018. 31(4): pp. 462–468. 51. Parsa, N. and Byrne, M. F. (2021), Artificial intelligence for identification and characterization of colonic polyps. Therapeutic Advances in Gastrointestinal Endoscopy, 2021. 14: pp. 26317745211014698– 26317745211014698.

52. Parsonnet, J., et al. (1991), Helicobacter pylori infection and the risk of gastric carcinoma. N Engl J Med, 1991. 325(16): pp. 1127–1131. 53. Pei, Q., et al. (2015), Endoscopic ultrasonography for staging depth of invasion in early gastric cancer: A meta-analysis. J Gastroenterol Hepatol, 2015. 30(11): pp. 1566–1573.

54. Preiss, J. C., et al. (2014), [Updated German clinical practice guideline on “Diagnosis and treatment of Crohn’s disease” 2014]. Z Gastroenterol, 2014. 52(12): pp. 1431–1484. 55. Repici, A., et al. (2021), Artificial intelligence and colonoscopy experience: Lessons from two randomised trials. Gut, 2021. 71: pp. 757–765. 56. Repici, A., et al. (2020), Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial. Gastroenterology, 2020. 159(2): pp. 512–520.e7. 57. Rex, D.K., et al. (2006), Quality indicators for colonoscopy. Am J Gastroenterol, 2006. 101(4): pp. 873–885.

58. Rubio-Tapia, A., et al. (2013), ACG clinical guidelines: Diagnosis and management of celiac disease. Am J Gastroenterol, 2013. 108(5): pp. 656–676; quiz 677. 59. Sharma, P., et al. (2016), Development and validation of a classification system to identify high-grade dysplasia and esophageal adenocarcinoma in Barrett’s esophagus using narrow-band imaging. Gastroenterology, 2016. 150(3): pp. 591–598. 60. Sharma, P., et al. (2012), The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on imaging in Barrett’s Esophagus. Gastrointest Endosc, 2012. 76(2): pp. 252–254.

331

332

Application of Artificial Intelligence in Gastrointestinal Endoscopy

61. Shichijo, S., et al. (2019), Application of convolutional neural networks for evaluating Helicobacter pylori infection status on the basis of endoscopic images. Scand J Gastroenterol, 2019. 54(2): pp. 158–163.

62. Shichijo, S., et al. (2017), Application of Convolutional Neural Networks in the Diagnosis of Helicobacter pylori Infection Based on Endoscopic Images. EBioMedicine, 2017. 25: pp. 106–111.

63. Siegel, R. L., Miller, K. D. and Jemal, A. (2018), Cancer statistics, 2018. CA Cancer J Clin, 2018. 68(1): pp. 7–30. 64. Su, J. R., et al. (2020), Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: a prospective randomized controlled study (with videos). Gastrointest Endosc, 2020. 91(2): pp. 415–424.e4. 65. Sung, H., et al. (2021), Global Cancer Statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin, 2021. 71(3): pp. 209–249.

66. Takemura, Y., et al. (2012), Computer-aided system for predicting the histology of colorectal tumors by using narrow-band imaging magnifying colonoscopy (with video). Gastrointest Endosc, 2012. 75(1): pp. 179–185. 67. Takemura, Y., et al. (2010), Quantitative analysis and development of a computer-aided system for identification of regular pit patterns of colorectal lesions. Gastrointest Endosc, 2010. 72(5): pp. 1047–1051.

68. Tischendorf, J. J., et al. (2010), Computer-aided classification of colorectal polyps based on vascular patterns: a pilot study. Endoscopy, 2010. 42(3): pp. 203–207. 69. Tsuboi, A., et al. (2020), Artificial intelligence using a convolutional neural network for automatic detection of small-bowel angioectasia in capsule endoscopy images. Dig Endosc, 2020. 32(3): pp. 382–390.

70. Urban, G., et al. (2018), Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy. Gastroenterology, 2018. 155(4): pp. 1069–1078.e8. 71. Vieira, P. M., et al. (2020), Automatic detection of small bowel tumors in wireless capsule endoscopy images using ensemble learning. Med Phys, 2020. 47(1): pp. 52–63. 72. Vleugels, J. L. A., et al. (2019), Diminutive polyps with advanced histologic features do not increase risk for metachronous advanced colon neoplasia. Gastroenterology, 2019. 156(3): pp. 623–634.e3.

73. Wang, P., E. Fan, and Wang, P. (2021), Comparative analysis of image classification algorithms based on traditional machine learning and deep learning. Pattern Recognition Letters, 2021. 141: pp. 61–67.

References

74. Wang, P., et al. (2020), Effect of a deep-learning computer-aided detection system on adenoma detection during colonoscopy (CADeDB trial): a double-blind randomised study. Lancet Gastroenterol Hepatol, 2020. 5(4): pp. 343–351. 75. Wang, P., et al. (2020), Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study. Gastroenterology, 2020. 159(4): pp. 1252–1261.e5. 76. Wang, P., et al. (2019), Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: A prospective randomised controlled study. Gut, 2019. 68(10): pp. 1813–1819.

77. Wimmer, G., et al. (2018), Fisher encoding of convolutional neural network features for endoscopic image classification. J Med Imaging (Bellingham), 2018. 5(3): p. 034504.

78. Winawer, S. J., et al. (1993), Prevention of colorectal cancer by colonoscopic polypectomy. The National Polyp Study Workgroup. N Engl J Med, 1993. 329(27): pp. 1977–1981. 79. Wu, L., et al. (2019), A deep neural network improves endoscopic detection of early gastric cancer without blind spots. Endoscopy, 2019. 51(6): pp. 522–531. 80. Wu, L., et al. (2021), Effect of a deep learning-based system on the miss rate of gastric neoplasms during upper gastrointestinal endoscopy: a single-centre, tandem, randomised controlled trial. Lancet Gastroenterol Hepatol, 2021. 6(9): pp. 700–708.

81. Wu, L., et al. (2019), Randomised controlled trial of WISENSE, a realtime quality improving system for monitoring blind spots during esophagogastroduodenoscopy. Gut, 2019. 68(12): pp. 2161–2169. 82. Yamada, M., et al. (2019). Development of a real-time endoscopic image diagnosis support system using deep learning technology in colonoscopy. Scientific Reports, 2019. 9(1): p. 14465.

83. Yan, T., et al. (2020), Intelligent diagnosis of gastric intestinal metaplasia based on convolutional neural network and limited number of endoscopic images. Comput Biol Med, 2020. 126: p. 104026. 84. Yasuda, T., et al. (2020), Potential of automatic diagnosis system with linked color imaging for diagnosis of Helicobacter pylori infection. Dig Endosc, 2020. 32(3): pp. 373–381.

85. Yoon, H. J., et al. (2019). A lesion-based convolutional neural network improves endoscopic detection and depth prediction of early gastric cancer. J Clin Med, 2019. 8(9).

333

334

Application of Artificial Intelligence in Gastrointestinal Endoscopy

86. Zauber, A. G., et al. (2012). Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths. N Engl J Med, 2012. 366(8): pp. 687–696.

87. Zhang, Y., et al. (2020), Diagnosing chronic atrophic gastritis by gastroscopy using artificial intelligence. Dig Liver Dis, 2020. 52(5): pp. 566–572.

88. Zhao, S., et al. (2019), Magnitude, risk factors, and factors associated with adenoma miss rate of tandem colonoscopy: A systematic review and meta-analysis. Gastroenterology, 2019. 156(6): pp. 1661–1674. e11.

89. Zheng, W., et al. (2019), High accuracy of convolutional neural network for evaluation of helicobacter pylori infection based on endoscopic images: Preliminary experience. Clin Transl Gastroenterol, 2019. 10(12): p. e00109. 90. Zhu, Y., et al. (2019), Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy. Gastrointest Endosc, 2019. 89(4): pp. 806–815.e1.

Part IV

Digital Therapeutics

Chapter 17

Digital Transformation Processes in Acute Inpatient Care in Germany

Andreas Mahler and Kerstin Lamers

Nursing Science and Development, University Hospital Augsburg, Augsburg, Germany [email protected], [email protected]

Within an acute inpatient maximum care provider, nursing is quantitatively the largest occupational group. Nursing, therefore, plays a decisive role in the digitalization of existing treatment and care processes. However, current studies show that nurses muster little affinity for digital applications and technologies. Since the enforcement of the Hospital Future Act (HFA/ Krankenhauszukunftsgesetz), hospitals have been obliged to embark on the ambitious path toward digitalization. If one reflects on the current degree of digitalization in the context of nursing, digitalization is usually reduced to documenting patient data in computer systems. This presents a mere transfer from paper to a digital medium without increasing functionality. However, through Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

338

Digital Transformation Processes in Acute Inpatient Care in Germany

research and cooperation with technical partners, products can already be used today that can effectively reduce the daily workload of nurses, both in terms of time and physical labor. This is described using the example of sensory support for determining and visualizing the contact pressure. Another challenge is to familiarise nurses with new technologies and to promote and coordinate their sensible use so that digitalization is perceived as a means to ease the workload rather than a burden. Finally, the perspective of nursing science in the context of digital transformation within the setting of a university hospital will also be taken into account. Here, the current developments are seen as an opportunity with regard to the resulting possibilities in data collection and evaluation as well as applied nursing research.

17.1 Introduction

With the entry into force of the HFA in autumn 2020, the way forward for the German stakeholders was clear. The federal government represented through the Federal Social Security Office places a clear focus on digitalization within the hospital landscape. With a budget of up to 4.3 billion Euros, inpatient care in Germany is to undergo an unprecedented digital upgrade. The aim seems to be to generate a common benchmark within care, as the levels of digitalization vary strongly between the individual facilities [2]. The focus of the HFA lies primarily in professional care. What holds many opportunities also entails a certain risk. Many hospitals, which for years remained dormant in the context of digitalization, are now experiencing a long-awaited initiative boost. When comparing the degree of digitalization in the clinical context with other areas of social life, it becomes clear that progress has only made limited inroads in the relevant institutions [11]. The use of individual technical devices is mainly concentrated on desktop PCs. In an explorative study by Bräutigam, 85% of the questioned nurses stated that they use these systems at their workplace. In the same survey, however, digital ward round trolleys or tablets were only used by slightly more than 10% [4]. To classify digitalization

Approaches to Implementation

correctly, an apriori differentiation must be made. In its brochure “Digitalization of Care”, the Healthy Care Campaign (Offensive Gesund Pflegen) has subdivided digitalization into the aspects of electronic care documentation, technical assistance systems, telecare/ telemedicine, and robotics [14]. It makes sense to add technologies that support communication and learning to this categorization [12]. Up to now, technologisation has primarily involved a switch from analog media, such as paper documentation, to digital media. However, if a more in-depth digitalization is to take place in the clinics, fundamental changes must take place. The processes of the “old,” analog work environment must be questioned and adapted to the current technical possibilities. This will impact the way nurses think and act and further affect their self-conception [4]. How these effects become evident will be outlined below taking best practice models into account.

17.2 Approaches to Implementation

Nurses have long been aware that digitalization will fundamentally alter our society and our work environment and that this change is already actively taking place. At the same time, this development presents them with unprecedented challenges. Digitalization will presumably bring about massive changes in work structures, especially in the area of human-machine interaction that is elementary for professional care. In addition, newly emerging intersectoral structures must be dealt with. So far, there is little empirical data on the impact of digital technology in hospitals. Whether technical innovations improve the quality of care, or whether they have a direct influence on issues such as time, the substitution of work, heteronomy, and de-professionalization should be continuously and systematically monitored and recorded from their introduction onward. Implementation strategies play a key role here [4]. The implementation of digital innovations in the work environment of a hospital is a fundamentally complex undertaking for which various approaches must be considered and taken into account. Measures of successful implementation include the acceptance of digital technologies by the staff and the resulting use of digital technologies within the changed work processes.

339

340

Digital Transformation Processes in Acute Inpatient Care in Germany

For the success of digital transformation, it is crucial to link the long-term vision with short-term interventions in a comprehensible way. Consequently, the statement “If you digitize a bad process, you have a bad digital process” applies [6].

17.2.1 Top-Down

Digitalization goals are set by the upper management level. Examples of application through a top-down approach can be found in many companies. In the case of a hospital, the following scenario is possible. A working group consisting of experts from the fields of nursing, medicine, and business is appointed. These experts are simultaneously members of a steering committee for decisionmaking. The group provides all guidelines, information, plans, and financial processes. The group should clearly communicate all expectations to the user. With this approach, ambiguity is often the cause of failure. The upper management should be as specific as possible in communicating their expectations. Process formality is particularly relevant to this approach. Using this method, software for duty scheduling and accounting was implemented at Augsburg University Hospital in the mid2000s. Special attention was paid to the accounting function and interoperability with the existing personnel management software. However, the planning component was insufficiently considered in the selection process. After the introduction, obvious deficiencies quickly became apparent. It was not possible to map the needs of the ward managers who were in charge of the work schedules. While it was possible to divide the working hours of the staff into a shift system, it was not possible to allocate shifts to specific areas of work, e.g., the emergency department or operating theatre. This task still had to be done in MS Excel. This additional effort caused by the transfer between the systems regularly led to errors and additional work for the users. As a result, the planning software was changed after a few years.

17.2.2 Bottom-Up

At the core of this approach is the premise of participation. All actors involved in the process of digitalization, but especially the

Approaches to Implementation

users, should be heard in the context of planning and development. The adaptation requests of the individual departments and thus of the employees using the technology need to be taken into account. This especially impacts the acceptance and use of jointly implemented projects. From a practical point of view, this can be achieved, for example, through the integration of hands-on care in product presentations, participation in working group meetings, or a development partnership with manufacturers and providers of IT systems. Nevertheless, a high effort is to be assumed with regard to the adaptations of the modules and systems to be made.

Figure 17.1 Advantages and disadvantages of the top-down or bottom-up approaches.

Which approach is particularly successful in the long term should be decided on a case-by-case basis, taking into account the advantages and disadvantages (see Fig. 17.1). The higher-level project management is responsible for creating an ideal balance between the approaches mentioned.

17.2.3 The Holistic Approach

The holistic approach stands in contrast to the very strong organizational or operational approaches already mentioned. This approach corresponds most closely to a paradigm that runs parallel to implementation.

341

342

Digital Transformation Processes in Acute Inpatient Care in Germany

The focus lies on a holistic as well as purposeful approach. All those involved, as well as the innovations to be introduced, are not viewed in isolation, but as parts of a whole. The basic assumption of holism, that everything is interconnected, is virtually obligatory for the embedding of new systems and processes in the context of digitalization. Awareness that everything interacts in a digital world can help to make a very consensual decision. In a hospital setting, a wide variety of people and thus also ways of life, values, and norms come together embedded in a field with high responsibilities. This circumstance becomes very clear in the context of complex hospital information systems (HIS). Whereas in the past these were pure administration tools, today HIS are data repositories with extensive connections that only provide added value for the user, when the individual components smoothly interact with each other. So it is not about the best application from the point of view of the individual departments, but about the integration of applications into a digital “clinic world.” The need for such interoperability becomes very clear, for example, in the premedication of patients before surgeries. The medicines to be administered are noted in the patient’s chart. However, these might be supplemented or modified in the course of anesthetic preparation. In order to ensure correct implementation on the part of the nurses, they must be able to view the information in the context of the medication. Here, it is crucial that the different parts of the documentation must interact with each other. This is only possible if the patient’s care process is the guiding principle for the software requirements and not the preferences of individual departments.

17.3 Direct Care Perspective

Direct care is subject to an enormous amount of documentation on a daily basis [5]. The extent of documentation that the nursing profession has had to deal with for years is unknown to society at large. Winkler et al. precisely quantify this effort. They assume that each patient requires about 15 minutes of documentation time per day in the hospital. If this is added to the average number of patients to be cared for in the average inpatient area, with a current ratio of 10:1, a total time of 150 minutes results. Nurses, therefore, need onethird of their working time for documentation. Based on the study by

Direct Care Perspective

Winkler et al. from 2006, it can be assumed that this calculated time will increase steadily due to the increasing complexity of patients’ and legal requirements [18]. Synchronously to this development, many hospitals are currently converting from handwritten to digital documentation. In this context, the digitalization process is a potential confounder for many nurses and results in very low acceptance. Since hands-on care usually entails only one point of contact with digitalization in documentation, it can be seen as the task of the upper management and science to reduce the inhibiting factors here and to condition them positively. How can such an undertaking succeed?

17.3.1 Digitalization in the Context of Care Relief

With the Nursing Staff Strengthening Act (NSSA/ Pflegepersonalstärkungsgesetz), which came into force in 2019, it was decided that a large part of the nursing staff costs currently reimbursed via the Diagnosis Related Groups (DRG) will be removed from the DRG system and reimbursed separately in future. The aim is to appropriately fund nursing staff costs independent of the case rates. For this purpose, a hospital-specific nursing budget based on the principle of self-cost recovery will be used. The designated care budget also finances so-called care-relieving measures. These include, for example, workflow improvements when tasks can be transferred from nursing staff to other professional groups [13]. In addition to the autonomous nursing tasks according to the German Nursing Professions Act § 4 (Pflegeberufegesetz/NPA), the nursing and functional service of a maximum care provider also fulfill various other minor tasks These additional tasks are perceived as time-consuming and interruption-sensitive by the nursing staff working on patients. They have an unsatisfactory effect on the daily work routine, the quality of the service, and, in particular, on the holistic character of care situations. The employment of autonomous financial means can prospectively implement the use of practice-centered technologies in a care-relieving context. In this way, acceptance of the digitalization process in care can be increased.

343

344

Digital Transformation Processes in Acute Inpatient Care in Germany

This can be exemplified by the use of an electronic pressure measurement foil in direct nursing care. Promoting the patient’s movement during the acute inpatient stay is of great importance in many respects. Experience shows that continuous changes in position have a positive effect on the patient’s physical condition. Fundamental influences on the cardiovascular system, perception, and sensation can be observed. Particularly on the prophylactic level, the targeted mobilization of the patient can counteract complications, such as the development of a pressure sore or pneumonia. This mobilization of the patient, which is sometimes necessary at high frequency, is considered one of the core activities of professional nursing. Currently, nursing care can only provide snapshot impressions of a patient’s mobility and positioning. The practice is executed as follows: Classically, nursing measures are to be considered in units. They take place in a particular rhythm, during the so-called “sighting rounds,” performed in varying time spans depending on the patient’s status and specialist area. For high-risk patients, mobilization measures are carried out up to twelve times a day. Even with this narrow interval, the patient is without observation between the individual interventions. There is no valid way for the nurse to track the changes in movement and pressure during this time frame. The nurse, therefore, re-positions the patient at regular intervals, usually according to the same pattern. The possibility of continuous monitoring of the changes in movement and pressure between mobilizations can be used to specifically control and systematically counteract a distortion of information. With the current practice, the detection of a potentially existing risk is not automated on the basis of given movement profiles and is documented by the nurse in the HIS after the intervention. Thus, movement promotion is currently still mostly practiced without the use of assistive technologies and is experienced as extremely time-consuming depending on the patient’s status and the caregiver’s resources. The degree of immobility of the patient takes on highly individual forms and carries information in the nursing-medical context. The patient’s intrinsic factors, e.g., blood circulation, positional preferences, weight, etc., have an impact on the patient’s contact pressure.

Direct Care Perspective

The use of technologies in nursing can not only be assumed to reduce the workload multifactorially, but also to generate measurable data for science, management, and, as a result, the introduction of targeted interventions to improve the quality of nursing care (see Fig. 17.2).

Figure 17.2 Use of innovative technology in nursing practice.

Based on statements made by employees of the care service in various institutions, they currently see themselves at a turning point in the transformation process. Past introductions of digital innovations were perceived by the staff as follows: “We work for the digital system.” Now they are in transition to the conviction that “the digital system is working for us.” The latter motivation must now be continuously supported and maintained by both management and academia.

345

346

Digital Transformation Processes in Acute Inpatient Care in Germany

17.4 Care Management Perspective Digitalization in care is currently equated with care documentation, as described above, but the effects can also be felt at the management level. Agile management methods are being introduced, modern ways of communicating with the nursing staff emerge and requirements and possibilities are also changing in staff qualification. In the past, high-tech equipment was only used in intensive care, but now it is increasingly being utilized in other settings as well. Participation in the development and implementation of new, digital technologies, with regard to the specifications and regulations of data protection and refinancing, is seen as a central challenge of care management [9].

17.4.1 Sector-Specific Challenges

The low level of digitalization in the healthcare system in Germany already mentioned in the introduction can be attributed to structural and personnel problems. The structural problems include legislation in the area of data protection, the financing models of the clinics, as well as the intra-clinical sectoral division as inhibiting factors. The personnel challenges are to be considered at various levels, especially in nursing performance.

17.4.1.1 Structural difficulties

With regard to structural problems, only the intra-clinical obstacles are highlighted here, as there is little direct influence that nursing management can exert on the legal framework and the technical infrastructure of telematics. Intra-hospital structures are characterized by a high degree of autonomy and a resulting differentiation of various professions. This results in coordination and communication problems. Essentially, four subsystems have emerged in the clinical context. The medical profession is isolated in its disciplines. The nurses are entrusted with the treatment and care of patients regardless of their specialty, but they also coordinate the associated processes. Another subsystem is the upper management, whose core task is external communication with partners, and finally controlling, whose main task is the overarching management of the hospital’s departments

Care Management Perspective

[1]. Analogous to the subsystems, the needs of the professional groups differ. As a result, software systems are adapted to a specific area of application.

17.4.1.2 Professional difficulties

In a study conducted in 2018 by Wörwag and Cloots, with 1501 participants from various sectors, it was shown that in the area of healthcare professions, only 26–28% of respondents believe in the creation of creative flexibility through digitalization. Particularly interesting is the fact that in the healthcare sector, creativity and the ability to innovate are only seen as relevant by 26% of employees. But especially when working with vulnerable groups, this room for development could be used to individually design interventions. In the same study, statements are also made regarding gender distribution. The authors conclude that the fear of an increase in routine work is more pronounced among women than among men. If one takes into account that the rate of women in nursing is currently at 80%, the general sentiment with regard to the challenges of digitalization in the field of nursing is clear [19]. This is aggravated by a low level of technological competence due to the lack of qualifications and experience, which means that the evidence of the benefits of technologies for nursing activities could hardly be proven [15]. In the future, it will therefore be necessary to create a vision of human digital work, taking into account the needs of patients and carers [19]. In addition, employee satisfaction is a decisive factor if and how digital transformation can be implemented. This means that employees must be convinced of the added value of the change. Employee satisfaction is also directly related to employee motivation. The higher the latter, the better the chances of success. If it is possible to explain the benefits to the employees, they are more likely to actively participate in the implementation [8].

17.4.2 Communication in the Care Team

Due to the Covid-19 pandemic, cooperation with digital tools has become part of everyday life in clinics. Video conferences, digital congresses, and further training courses are now part of everyday life. These methods are also used in professional groups that work in direct patient care.

347

348

Digital Transformation Processes in Acute Inpatient Care in Germany

Meetings In the analogous past, participation in so-called ward meetings was only possible in person and on-site. Due to shift work and the high rate of part-time employees, the attendance rate in meetings was usually only 20%, depending on the team. If decisions had to be made in the meeting, only the opinion of a few was heard and the majority of the team was left out. Since the beginning of the corona pandemic, these meetings have been held in digital format at Augsburg University Hospital. An almost doubling of the number of participants can be observed. The digital format creates the possibility of participation at short notice, it is not necessary to organize childcare for this time and the time factor for commuting to the workplace is eliminated. A higher participation rate also means that decisions are made by a larger part of the care team and therefore a higher acceptance can be assumed. The possibility of passing on information is also significantly improved by the low-threshold access to the meetings.

17.4.3 Qualification of Employees

What applies to meetings also applies to the same extent to the qualification of staff in the clinical setting. Through homeschooling their children, most households are equipped with technical appliances, such as laptops or tablets. With the introduction of electronic nursing documentation in the intensive care units at the Augsburg University Hospital, online teaching was used on a large scale for the first time. In this context, the participants also increasingly acquired subject and methodological competencies and the format could thus be developed from a digital lecture to an interactive exchange with the participants. Due to the possibility of training a larger number of participants at the same time, the number of events could be reduced, and thus the qualification of approx. 250 nurses took place within a few weeks. The feedback and the learning successes can also be assessed as positive throughout and there are no negative deviations compared to the departments that were conventionally qualified before the Covid-19 pandemic. Theoretical basics are acquired digitally in advance. Blended learning is therefore also gaining in importance in the nursing service and it is the task of nursing management to demand and promote this form of qualification.

Perspective of Nursing Science

Not only do digital training methods open up further perspectives, but also the possibility of data generation, evaluation, and the resulting derivation of content, as described below.

17.5 Perspective of Nursing Science

Nursing science at Augsburg University Hospital sees its primary mission as a continuous theory-practice transfer. A perpetual field of tension between legal and scientific requirements and the realistically possible implementation in practice is thus created. In some places, completely contrary to the perspective of direct care, nursing science obviously benefits greatly from the advancing development of analog nursing and treatment documentation toward digital comprehensively usable data. In order to be able to guarantee evidence-based and constantly developing nursing care, the nursing science sub-areas of nursing research and quality assurance are elementary. In the following, digitalization in these fields will be discussed in two projects.

17.5.1 The Evaluability of Data: New Ways and Opportunities for Care

Without a data basis, no evaluability and no generation of measures are feasible. The transformation process of documentation has made a completely new set of data available. In the current development, those involved in nursing science, and thus in many cases also in quality assurance, must understand the task of dealing with this data responsibly and efficiently. Digitalization is considered a success for improving the quality of care, economic efficiency, and transparency. For care and treatment documentation, digitalization offers new possibilities in the context of evaluation and analysis. There are numerous reasons for this. For one, it has to provide and access data on a decentralized or mobile basis. The times of difficult access, waiting times and poor readability of data can be regarded as a thing of the past. The usability of data no longer refers only to revenue-relevant diagnoses and procedures, but also to entirely qualitative approaches, such as conducting a document analysis using the nursing progress report.

349

350

Digital Transformation Processes in Acute Inpatient Care in Germany

17.5.2 Nursing Research As already mentioned, the digitalization of nursing and treatment documentation also offers various possibilities in the context of research creating a versatile database. An example of utilizing this data is a study conducted at the Augsburg University Hospital in 2021, which will be presented below. By means of routinely conducted monitoring within the framework of the external quality assurance of pressure ulcer prophylaxis, an increased relevance for the phenomenon of long lie could be shown. Triggers for the identification of the data were the forms for the digital wound documentation and the patient complex measures score. Initially, a subjectively assessed accumulation of the number of long lie cases was shown. The cases with indicated recumbency trauma or with indications of such were then isolated and statistically recorded using Excel. The researcher was then able to carry out a monocentric retrospective document analysis of the digital patient files of 63 patients affected by recumbency trauma who were undergoing inpatient treatment at Augsburg University Hospital, using the HIS. The evaluation was carried out according to a mixed-method design. Through this research, insights could be gained into the nursingspecific aspects of the patient groups who are admitted to the hospital with a long lie. Particular interest was paid to the indicators that can serve to develop a diagnosis by the nurses. Isolation of these indicators and the associated cases would not have been possible to this extent without the provision of nursing and treatment documentation in digital form. These findings on the detection of recumbency trauma made the serious effects on the patients clear and contributed to the early detection of this phenomenon. Above all, this contributes to the improvement of the cross-sectoral care situation of those affected [16].

17.5.2.1 Nursing indicators as a clinical and cross-sectoral management tool

The collection of key figures can objectify the often prevailing subjective perception. Furthermore, they can indicate completely new phenomena. Through a systematic or repeated collection of indicators, the following changes can be detected:

Perspective of Nursing Science



∑ Unintended changes indicate systemic developments. ∑ Intended changes can shed light on the effectiveness of a planned measure.

Access to such data can enable the leadership of a hospital to plan strategic goals and make its successes visible [10]. Indicators should enable an efficient discourse on leadership and strategic goals. At the same time, indicators promote a learning organization, for learning organizations are successful organizations [17]. Nurses are the largest professional group in the health sector and are responsible for a large part of the operating costs. Yet their contribution is often invisible to those in charge and undervalued by many decision-makers. “Often, key figures of nursing – such as nursing-sensitive outcomes – are not present in databases and thus are not included in business management considerations. Nursing leadership should present comprehensive and accurate information about nursing performance and its benefits in the context of an institution’s strategic goals” [7].

Primarily financial or revenue-relevant facts such as medical diagnoses and staff absences are depicted. This type of thinking falls short of reality. The long-term paradigm, which seems to be lacking in many areas of the health care system, also comes into play in the context of nursing indicators.

17.5.2.2 Imagine the given scenario

A patient, who develops a pressure ulcer during his hospital stay, is initially perceived as routinely billable. The fact that this pressure ulcer, possibly traceable to improperly performed care due to insufficient qualitative and quantitative staffing, now harbors an increased risk potential for a need for care. This implies a directly proportional higher financial burden for the health care system. It is therefore not only a matter of presenting the above-mentioned key figures but a matter of making them interpretable by inferential statistical methods. In order to make the above-mentioned facts transparent, a dashboard is being created within the framework of a project at the University Hospital Augsburg.

351

352

Digital Transformation Processes in Acute Inpatient Care in Germany

The development of the dashboard is a multi-stage process as shown in Fig. 17.3.

Figure 17.3 Multi-stage development process of a care dashboard.

The following indicators, among others, are set when compiling an evidence-based set of indicators (see Fig. 17.4).

Figure 17.4 Example of a key figure set for nursing care.

The integration of the key figures into the care performance described above will become more and more central in the future, as business management considerations continue to increase in the institutions of the health care system. In order to demonstrate the quality and efficiency of nursing, the creation of corresponding digital structures is indispensable. There is enormous potential here, especially in the utilization of digitally documented data within electronic nursing documentation. The upper management level of nursing is called upon to deal proactively with key figures and present their qualitative and economic relevance and derive measures from them.

Prospects

At this point, however, the focus is not only on making the key figures themselves available in the context of the digital transformation process but above all on application-based automated processing, aggregation, and visualization in real time. Modern applications make it possible to transform the classic reporting system into an interactive and, at the same time, lowthreshold, usable control instrument, for nurses and ward managers alike.

17.6 Prospects

Against the backdrop of demographic change and the accompanying shortage of nursing professionals, the digital transformation process is seen as an important factor in relieving the already strained institutionalized care system. Nevertheless, digitalization has become more of compensation rather than a sustainable and globalized optimization. Nursing work is characterized by a high degree of individuality and agility. Technologising these attributes is currently neither possible nor intended by the focused professional group. This character must be regarded as the first realization for the preservation and further development of the nursing profession. In addition to this critical view, digitalization also gives rise to several other perspectives for professional nursing in the clinical environment. The aspect of cognitive support, in the form of data collection and automated evaluation up to the resulting recommendation for action, offer decision-making aids in everyday nursing care. Through the further development of artificial intelligence (AI), a multitude of application possibilities with high practical relevance will arise in the future. Robotics and complex assistive systems can significantly reduce physical labor. As a result, digitalization can have a positive influence on employee health. In addition, digitalization creates possibilities for recording and evaluating quality indicators and key figures. By means of the objective determination of the patient’s care needs (degree of complexity, access frequency, and number) and the resulting staff commitment time for the nursing professionals deployed, this can serve as a basis for the prospective deployment of staff, which in turn falls within the competence area of nursing management. For this purpose, care and planning data must be correlated.

353

354

Digital Transformation Processes in Acute Inpatient Care in Germany

Only through the symbiosis of science, management and practice can the digital transformation in nursing succeed in the long term. It can be assumed that positive effects on the framework conditions of the clinical actors and patient care will result.

References

1. Baller, G. (2017). Kommunikation im Krankenhaus. Springer Gabler: Berlin.

2. Bundesgesundheitsministerium (2021a): Krankenhauszukunftsgesetz für die Digitalisierung von Krankenhäusern. Retrieved from https://www.bundesgesundheitsministerium.de/ krankenhauszukunftsgesetz.html on 25.10.2021. 3. Bundesgesundheitsministerium (2021b). Pflegepersonaluntergrenzen. Retrieved from https:// w w w. b u n d e s g e s u n d h e i t s m i n i s te r i u m . d e / t h e m e n / p f l e g e / pflegepersonaluntergrenzen.html on 23.12.2021.

4. Bräutigam, C., Enste, P., Evans, M., Hilbert, J., Merkel S., Öz F. (2017). Digitalisation in hospitals. More technology - better work? Hans Böckler Foundation. 5. Deutscher Pflegerat e.V. (2011): Press release. DPR welcomes proposals to reduce bureaucracy in nursing. Retrieved from https:// www.verbaende.com/news.php/DPR-begruesst-Vorschlaege-zurEntbuerokratisierung-in-der-Pflege?m=75909 on 25.10.2021. 6. Dirks, T. (2015). SZ Economic Summit 2015 in Berlin.

7. Dubois, C. A., D’ Amour, D., Pomey, M. P., Girard, F., and Brault, I. (2013): Conceptualizing performance of nursing care as a prerequisite for better measurement: A systematic and interpretive review. BMC Nursing, 12(1), 1. 8. Eierdanz, F. (2020). Mitarbeiterzufriedenheit im Rahmen digitaler Transformationsprozesse, in: Lahm, A. (ed.): Digitalisierung der Pflege. In Support of Better Work Organisation. Springer: Berlin, pp. 85–95.

9. Glaab, B., Fischer, U. (2020). Digitalisierung und intelligente Technik in der Pflege - Auswirkungen auf das Pflegemanagement: 25 Jahre Pflege studieren-Über Umwege und neue Horizonte. De Gruyter Oldenbourg, pp. 202–207. 10. Henri, J.F. (2006). Management control systems and strategy: A resource-based perspective. Accounting, Organizations and Society, 31(6), 529–558.

References

11. Klauber, J. (2019). Krankenhaus-Report Krankenhaus. Springer: Berlin.

2019:

Das

digitale

12. Kubek, V. (2020). Digitalisierung in der Pflege: Überblick über aktuelle Ansätze, in: Lahm, A. (ed.): Digitalisierung der Pflege. In Support of Better Work Organisation. Springer: Berlin, pp. 15–20.

13. Penter, V. (2020). P wie Pflegebudget. Retrieved from https://kugesundheitsmanagement.de/2020/08/11/p-wie-pflegebudget/ on 25.10.2021. 14. Rösler, U. (2018). Digitalisierung in der Pflege, in: Wie intelligente Technologien die Arbeit professionell Pflegender verändern, 1. Jg.

15. Weidner, F. (2019). Digitalisierung für die Pflege - Wege zur Alltagsund Nutzer/innenorientierung, in: Soziale Arbeit in der digitalen Transformation, Archiv für Wissenschaft und Praxis der sozialen Arbeit, 2, pp. 50–60.

16. Wiedemann, L. (2021). Entwicklung einer potenziellen Pflegediagnose zum Phänomen Liegetrauma. Presentation using the NANDA International nursing classification system on the basis of a document analysis in the acute inpatient setting. Unpublished thesis. Catholic Foundation University Munich. 17. Widener, S. K. (2007). An empirical analysis of the levers of control framework. Accounting, Organizations and Society, 32, 757–788.

18. Winkler, P., Rottensteiner, I., Pochobradsky, E., Riess, G. (2006). Österreichischer Pflegebericht. Commissioned by the Federal Ministry of Health and Women, ÖBIG, Vienna. 19. Wörwag, S. (2020). Was bringt die Digitalisierung der Arbeit: Raus aus der Routine oder rein in neue Regelabhängigkeiten: Human Digital Work - Eine Utopie?, Springer: Wiesbaden, pp. 127–147.

355

Chapter 18

Digital Medicine in Neurology

Antonios Bayas, Monika Christ, and Markus Naumann Department of Neurology, Medical Faculty, University of Augsburg, Augsburg, Germany [email protected]

Recent advances in the field of digital medicine have already led to improvements in the identification, phenotyping, monitoring, and treatment of a wide range of neurological disorders. With a focus on multiple sclerosis (MS) and Parkinson’s disease (PD), this chapter aims to present already established and potential applications, perspectives, and challenges of implementing digital medicine in neurology.

18.1 Introduction

Neurological disorders contribute to 4–5% of the global burden of diseases and are increasing in prevalence [33]. They are characterized by structural, biochemical, or electrical abnormalities Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

358

Digital Medicine in Neurology

of the brain, spinal cord, and nerves. In 2014, nine of the most common neurological disorders in the United States, dementia, chronic low back pain, stroke, traumatic brain injury, migraine headache, epilepsy, MS, traumatic spinal cord injury, and PD, cost the US economy $790 billion [24]. Biomarkers are objectively measured indicators of physiologic or pathologic processes or pharmacologic responses to therapeutic intervention, they may be diagnostic, prognostic, predictive, or treatment response markers [15]. For individualized patient care, the collection of data is essential for correct diagnosis, treatment planning, treatment efficacy, and safety monitoring. For diagnostic tools, gathering a great quantity of data may lead to a more accurate and earlier diagnosis with a potentially high impact on disease outcomes, especially in neurological disorders. For patient monitoring, digital biomarkers, either collected in given time intervals or continuously, are of particular importance. Artificial intelligence models together with high-quality clinical data will result in improved prognostic and diagnostic models facilitating clinical decision tools [55]. This will support neurologists in making an early diagnosis and improving patient care. For processing and using electronic data, digital platforms are vital for the exchange of health information between patients, practitioners, and caregivers [74]. Digital therapeutics, appropriate to support the care of several neurological disorders, is a section of digital health defined by the Digital Therapeutics Alliance as “delivering evidence-based therapeutic interventions to patients that are driven by software to prevent, manage, or treat a medical disorder or disease. They are used independently or in concert with medications, devices, or other therapies to optimize patient care and health outcomes” [1]. In recent years, artificial intelligence has gained access to a variety of neurological disorders, like stroke, epilepsy, MS, dementia, and movement disorders. For example, in epilepsy machine-learning applications include diagnosis of epilepsy, psychogenic non-epileptic seizures (PNES), subtypes of epilepsy, the prevention of sudden unexpected death in epilepsy, and reducing inter-observer variability of EEG interpretation [54]. For example, 20% of epilepsy patients referred to a tertiary center are eventually diagnosed with PNES using video

Introduction

electroencephalography resulting in numerous hospitalizations and inadequate treatment [54]. In a prospective study, a machinelearning approach was able to perform an individual classification of PNES from controls with a mean accuracy of 74.5% [70]. With an aging population, dementia will be a major challenge for the healthcare system as well as for society. Early detection may lead to early treatment in the future, hopefully with more effective therapeutic interventions. For identifying mild cognitive impairment and dementia, a recent review revealed that digital cognitive tests showed good performances. Most of the digital cognitive tests were shown to have comparable diagnostic performances with paper-andpencil tests. However, all the digital tests only had a few validation studies [10]. For migraine, several app-based headache diaries are available to record, e.g., the frequency and intensity of headaches, or to identify headache triggers. Studies have shown that electronic headache diaries are a reliable method of data collection, preferred to paper headache diaries by patients [66]. In recent years, numerous studies have also investigated the use of digital behavioral therapies (such as relaxation, biofeedback, and cognitive behavioral therapy) in adults and children with migraine. A systemic review revealed moderate to high acceptance of the use of these technologies, but only a minority of studies demonstrated a statistically significant reduction in symptoms [48]. Recent studies have focused on the diagnosis of migraine by applying machine learning on functional magnetic resonance imaging (MRI) or morphometric MRI data to examine the functional connectivity of different brain regions. They were able to distinguish migraine patients from control subjects with an accuracy of 86% [47]. Another broad field of digital medicine application is the diagnosis and treatment of stroke. In addition to numerous educational applications that can improve knowledge about the signs and symptoms of stroke, several mobile applications for primary and secondary prevention exist. Various studies, e.g., demonstrate a significant improvement in cardiovascular risk factors by using app-based applications [21, 64]. Based on strong scientific evidence, telemedical tools, e.g., remote image viewing and clinical evaluation are widely used in emergency medicine to select candidates for thrombolysis or thrombectomy. New applications

359

360

Digital Medicine in Neurology

use artificial intelligence, machine learning, and automated stroke imaging analysis for the early identification of acute stroke and prompt intervention to reduce morbidity and mortality. For reducing stroke-related disability, several studies have assessed the efficacy of remotely supervised or self-administered telerehabilitation and showed significant improvements in functional outcomes after stroke [64]. In the following, the potential impact and application of digital medicine in two neurological conditions, MS and PD is described in more detail.

18.2 Multiple Sclerosis and Parkinson’s Disease 18.2.1 Multiple Sclerosis

MS is a chronic neurological disease of the central nervous system (CNS) affecting more than two million people worldwide. The exact etiology of MS is still unclear, although there are compelling arguments for (auto)immunopathogenesis [30, 31]. According to the current dogma, MS is caused by a combination of neuroinflammation and neurodegeneration. Various functional neurological systems can be affected in MS, resulting in cognitive, visual, cerebellar, motor, and sensory symptoms. MS affects people over a wide age range, starting as pediatric MS in 3–5% of patients before the age of 18. Treatment of MS is based on (a) treatment of relapses, (b) immunoprophylactic disease-modifying drugs (DMDs), and (c) treatment of symptoms like spasticity. The course of MS can vary from benign to highly aggressive. Similarly, clinical, radiological, or genetic pathogenesis is extremely heterogeneous. MS is still incurable, but therapeutic options, particularly for the treatment of relapsing MS (RMS), have evolved since the approval of the first interferon-beta more than 25 years ago. Currently, more than 10 drug classes are approved for the immunotherapy of MS, some of which are considered moderately effective but relatively safe and others that are considered highly effective but associated with potentially more severe adverse effects [13]. The therapeutic goal is early and complete freedom from disease activity, as this has a major impact on long-term disability and quality of life.

Multiple Sclerosis and Parkinson’s Disease

The challenge in clinical decision-making is to select an appropriate treatment for the individual patient and weigh the benefits and risks of treatment avoiding disability as well as overtreatment. The aim of prognosis research in MS is to improve future health outcomes in MS by predicting as accurately as possible diseaserelated outcomes, identifying relevant prognostic factors associated, and targeting treatment to patient subgroups regarding the risk of outcome [28, 29, 58, 65]. In MS, prognostic factors include age, body mass index, sex, smoking, and disease duration [9]. Diagnostic tools for prognosis prediction and monitoring disease activity include MRI and optical coherence tomography (OCT) [50, 61]. In a complex disease like MS, a combination of factors for long-term prognosis is necessary. Currently, numerous studies have shown prognostic factors in people with MS (pwMS) subgroups and cohorts. Despite enormous efforts, a single prognostic factor or a combination of disease characteristics to predict the prognosis in the individual patient is lacking. MS symptoms can occur and increase over time either caused by relapses or chronic progression. In clinical practice it is crucial to monitor the disease course by regular neurologic examination, gathering patient-reported outcomes, functional tests, MRI, and laboratory evaluation among others. Thus, for monitoring the disease course, a large amount of data must be recorded continuously, assessed, and analyzed. The early identification of responders and non-responders to DMDs is crucial for potential treatment adjustments. Collecting these data is time-consuming and requires personnel and device resources. Using digital technology devices can facilitate this process by the collection of digital biomarkers. In MS they can be subdivided into diagnostic (e.g., oligoclonal bands in cerebrospinal fluid), prognostic (e.g., neurofilament light), predictive (e.g., MRI), disease activity (e.g., clinical parameters), and treatment response (e.g., clinical assessment, MRI) biomarkers [15]. Several digital devices can help to collect data in given defined intervals or continuously, and thus enable regular and close monitoring with the option to react rapidly to changes in the disease course or symptoms. Digital biomarkers are objective, quantifiable physiological and behavioral data measured and collected by digital devices and collected by, e.g., portables and wearables [15].

361

362

Digital Medicine in Neurology

18.2.1.1 Monitoring MS symptoms and disease course Digital biomarkers can be obtained as clinical digital biomarkers and from digitalized diagnostic procedures, e.g., OCT.

Clinical digital biomarkers

MS as a disorder of the CNS can affect a variety of functional systems, like visual and motor function. In the following, possible applications for assessing MS disease evaluation and surveillance are presented. Often pwMS have three-monthly to half-yearly or yearly clinical routine visits with neurologic examinations and regular MRI, limiting timely insight into symptom progression. Since MS affects multiple functional systems, a range of validated clinical tests are necessary and already used in clinical practice for disease monitoring: the Expanded Disability Status Scale (EDSS) measures disability covering all functional systems but weighs some important like cognition only poorly. The Symbol Digit Modalities Test (SDMT) measures mental processing speed [53], dexterity is measured by the 9-hole peg test (9-HPT) [18], and mobility by the timed 25-foot walk (T25FW) mobility [51]. For monitoring neurological function by digital biomarkers, various smartphone apps have been developed in recent years including the Floodlight app (Roche, Switzerland), Konectom(TM) (Biogen, Cambridge, MA, USA), MSCopilot® (Ad Scientiam, Paris, France) or MS Sherpa (Orikami, Nijmegen, The Netherlands) [15, 68]. Smartphones as well as smart watch sensors, e.g., accelerometers, gyroscopes, and gesture sensors, are well suitable for quantifying relevant neurological functions. Thus, regular and continuous monitoring of neurological function by patients themselves is feasible [67]. However, to enable disease monitoring and communication between patients and healthcare professionals, with potential consequences for treatment decisions, it is necessary to transmit data to the healthcare team that may be impeded by network obstacles or data protection. In a 24-week study the performance characteristics of the Floodlight Proof-of-Concept (PoC) app., cognition (electronic SDMT), upper extremity function (Pinching Test, Draw a Shape Test), and gait and balance (Static Balance Test, U-Turn Test, Walk Test, Passive Monitoring) were assessed by smartphone-based active tests and passive monitoring. In 76 pwMS and 25 healthy controls,

Multiple Sclerosis and Parkinson’s Disease

the Floodlight PoC app captured reliable and clinically relevant measures of functional impairment in MS, supporting its potential use in clinical research and practice [49]. In another study, publicly available data from the Floodlight Open study, which collected smartphone-based test data from self-declared persons with MS with a number of different tests implemented in the Floodlight Open app were analyzed. Study results suggest that strong longterm practice effects on cognitive and dexterity functions must be taken into account for identifying disease-related changes in these domains, especially in the context of personalized health and in studies without a comparator arm [72]. Impaired lower extremity function resulting in gait abnormality belongs to the most frequent and visible MS symptoms, caused by a variety of disease-related events such as pyramidal, cerebellar or sensory dysfunction [19]. Different tools, among them wearable sensors, are available for assessing gait impairment in MS, ranging from standardized clinical measures, timed measures, patientreported outcomes, observational gait analysis, instrumented walkways, or three-dimensional gait analysis. The advantages and disadvantages of various gait assessment methods have been reviewed recently [15]. A recent study in 102 pwMS and 22 healthy controls investigated if sensor-based gait analysis with a foot-worn sensor-based gait analysis system can detect gait impairments in pwMS. Correlation analysis showed that sensor-based gait analysis objectively supported the clinical assessment of gait abnormalities even in the lower stages of MS, especially when walking at a fast speed, with stride length and gait speed as the most clinically relevant gait measures [19]. In clinical practice, the transition from relapsing-remitting MS (RRMS) to secondary progressive MS (SPMS), as well as the progression of people with SPMS is often difficult to assess, but crucial for treatment decisions. In many patients, deterioration of gait is the prevailing symptom. Monitoring gait characteristics is therefore an essential component of disease evaluation. Various smartphone apps and fitness trackers (Fitbit, Polar, Apple Watch, Garmin, and others) are commercially available and widely used in daily activities. As a limitation, wearable sensors collect a smaller number of gait variables compared to non-wearable research

363

364

Digital Medicine in Neurology

systems, but can be used in community settings and can provide real-time feedback to patients and clinicians [62]. A study by Block et al. investigated if remote step count monitoring using an accelerometer (Fitbit Flex) could enhance MS disability assessment. Lower average daily step count was associated with a greater disability on the EDSS and step count demonstrated moderate-strong correlations with other walking measures [7]. Over half of pwMS suffer from balance deficits over the disease course, they are even common in early disease stages [26, 43]. Since balance is often subject to personal estimates, objective and quantifiable measurements are needed. As for other neurological deficits, various clinic-/research-based tests are available. For example, in the clinic balance can be measured by static posturography and body sensors. For everyday measurements by pwMS themselves, smartphone apps include tests for balance and gait assessments, e.g., the Health App (Apple Distribution International) capturing bipedal stance and asymmetric gait. Approximately 40–60% of pwMS report cognitive dysfunction that may already be present at disease manifestation, sometimes even before [40]. Only a few cognitive domains are affected, with prominent deficits in information processing speed and episodic memory, and less frequently in executive functions, including verbal fluency and word list generation. Cognitive deficits are usually underestimated during routine clinical visits, mainly because self-reports are not a valid measure of cognitive ability and the use of cognitive tests does not prevail [59]. A number of tests of cognition have been developed for clinical use in MS, including the Brief Repeatable International Cognitive Assessment for MS, the Brief Repeatable Battery of Neuropsychological Tests, and the Minimal Assessment of Cognitive Function in MS [15]. Monitoring of cognitive function can be done by pwMS by smartphone apps, e.g., with the Floodlight app, as described above. In a prospective study, smartphone SDMT achieved slightly higher correlations with cognitive subscores of neurological examinations and with brain injury measured by MRI than traditional SDMT [56]. Patient-reported outcomes (PROs) focusing on various MS aspects like neurologic functions and quality of life have many potential advantages, such as facilitating communication between patients and physicians, assessing changes in disease status, screening for

Multiple Sclerosis and Parkinson’s Disease

unidentified symptoms as well as monitoring the safety and efficacy of medications [23]. Online administration of PROs has been shown to be equivalent to using paper questionnaires [36].

Automated imaging and diagnostic tools, digitalized diagnostic procedures

MRI of the brain and spinal cord is the most important technique for diagnosing MS and tracking disease evolution. Conventional manual methods for assessing lesion development are time-consuming, subjective, and error-prone. Thus, the development of automated techniques for the detection and quantification of the development of MS lesions is a major challenge and has been addressed by many [39]. Deep-learning techniques have achieved substantial progress in the lesion segmentation task by learning complex lesion representations directly from images. State-of-the-art automatic statistical and deep-learning MS segmentation methods have been reviewed recently, and current and future clinical applications have been discussed [42]. MRI may also be used for prognosis evaluation. For predicting the occurrence of a second clinical attack leading to the diagnosis of clinically definite MS, machine-learning techniques have successfully been used on the basis of a single patient’s lesion features and clinical/demographic characteristics [73]. Visual function is often affected in MS, manifesting as reduced visual acuity or color vision. OCT is a noninvasive method routinely used to examine the retina. OCT is also used to study the retina in patients with optic neuritis (ON), a frequent manifestation of MS. In a prospective study, the odds of finding retinal atrophy in subjects with MS were more than 17 times greater than in non-MS subjects [25]. The usefulness of monitoring peripapillary retinal nerve fiber layer (pRNFL) thickness by OCT for the prediction of the risk of disability worsening was evaluated in a multicentre cohort study. In patients with clinically isolated syndrome, relapsing-remitting MS, and progressive MS, pRNFL thickness and macular volume were assessed once at study entry (baseline) by OCT and were calculated as the mean value of both eyes without optic neuritis for patients without a history of optic neuritis or the value of the non-optic neuritis eye for patients with previous unilateral optic neuritis. The

365

366

Digital Medicine in Neurology

association of pRNFL thickness or macular volume at baseline in eyes without optic neuritis with the risk of subsequent disability worsening by use of proportional hazards models that included OCT metrics and age, disease duration, disability, presence of previous unilateral optic neuritis, and use of disease-modifying therapies as covariates was estimated. Patients with a pRNFL of less than or equal to 87 μm or less than or equal to 88 μm had double the risk of disability worsening at any time after the first and up to the third years of follow-up, and the risk was increased by nearly four times after the third and up to the fifth years of follow-up. Thus, OCT as a widely available diagnostics could predict disability evolution in this cohort [44].

18.2.1.2 Digital treatment support

Digital data may also be used to monitor and support MS treatment. For example, Lang et al. implemented PatientConcept, a CE-certified, ID-associated multilingual software application to promote patienttailored management of MS. Since its implementation in 2018, about 3000 MS patients have used the app that maps risk management plans of all current DMDs. It also enables continuous monitoring of various PROs [35]. A recent review has conducted a systematic search to identify all relevant published and unpublished clinical trials in the English language, in which digital technology was applied. The authors found 10 ongoing studies out of 35 using digital technology as the main intervention. Digital technology was used as the main intervention for rehabilitation and psychotherapy [3]. In MS, like in other chronic disorders, adherence to medication, like immunotherapies, is a major issue. Thirteen to 72% of patients do not adhere to disease-modifying MS treatments, and poor adherence or treatment gaps are associated with a higher rate of relapse [6]. Whether telehealth technologies could help to monitor or improve adherence has been addressed by a number of studies, but results have been uneven [22, 45, 60].

18.2.1.3 Prediction of disease activity and treatment decisions

Ideally, prognostic models are developed and validated using large, high-quality datasets with pwMS representative of the population to which the model will later be applied. There are several attempts to establish prognostic models for MS.

Multiple Sclerosis and Parkinson’s Disease

ProVal-MS, as an example, is a prospective, non-interventional, diagnostic phase II cohort study validating a multidimensional treatment decision score, established in a separate retrospective MS study cohort. ProVal-MS is aiming to predict the 24-month outcome in a minimum of 250 untreated patients with early relapsing-remitting MS or clinically isolated syndrome patients in five academic centers in Germany within and associated with the DIFUTURE consortium (https://www.medizininformatik-initiative.de/de/konsortien/ difuture). ProVal-MS makes the effort to harmonize data structures at the participating sites to follow a common MS core dataset which was agreed upon by all partners. This approach makes it possible to transfer routine data to the study database where it can be enriched with study-specific data captured outside of the clinical routine.

18.2.2 Parkinson’s Disease

PD is a progressive neurodegenerative disorder with a prevalence of 0.3% in industrialized countries. Incidence increases with age, peaking between age 85 and 89, PD is rarely seen in individuals younger than 40 years [5, 27]. The pathological mechanism is not fully understood, with both genetic and environmental factors contributing. Structurally, an important finding in PD is the progressive degeneration of neurons in the substantia nigra, and pars compacta. The clinical spectrum is variable and includes motor and non-motor symptoms [27]. Current criteria define PD by the presence of bradykinesia combined with either rest tremor (4–6 Hz), rigidity, or both [8]. Another typical finding in PD is postural instability. Non-motor symptoms include cognitive decline, mental symptoms (especially depression and anxiety), dysautonomia (including dysregulation of blood pressure and/or temperature, bladder and bowel dysfunction, and sexual dysfunction) and sleep disturbance [27]. The non-motor symptoms often precede motor symptoms for years. In this so-called prodromal phase, especially anosmia (in 90% of PD patients), rapid eye movement sleep behavior disorder, and autonomic symptoms (e.g., constipation, orthostatic hypotension) are described [5, 27]. The diagnosis of PD is mainly clinically based. Due to the heterogeneity of the clinical picture, diagnosis of PD is often challenging but essential for prognosis and treatment decisions,

367

368

Digital Medicine in Neurology

especially in differentiation to atypical Parkinsonian syndromes (including progressive supranuclear palsy (PSP), multiple system atrophy (MSA), corticobasal degeneration (CBD), and dementia with Lewy bodies (DLB)) or other differential diagnoses (e.g., druginduced parkinsonism, vascular parkinsonism) [27]. To support the clinical diagnosis, only limited imaging techniques are available in clinical practice: Presynaptic dopamine transporter SPECT (DAT-SPECT) may detect nigrostriatal deficiency in otherwise unclear parkinsonism or tremor syndrome (98–100% sensitivity and specificity) [5] and fluorodeoxyglucose positron emission tomography (FDG-PET) can differentiate between typical (PD) and atypical neurodegenerative Parkinsonian syndromes. CSF or blood biomarkers for diagnosis or prognostic assessment in PD are not available so far. The pharmacological treatment options in PD are only symptomatic, no causal pharmacologic treatments are available. Besides pharmacological treatment, deep brain stimulation (DBS) has become an established treatment in patients with motor fluctuation, complications of medication, medication-refractory tremors, and dyskinesias. To slow down disability accrual, maintain functionality, and reduce the risk of falls non-pharmacologic treatments (e.g., physiotherapy, ergotherapy, speech therapy) are necessary [5]. As mentioned before, PD has a wide clinical spectrum with different clinical courses and prognoses. Until today, radiological, clinical, or validated laboratory biomarkers to predict the clinical course are lacking [5].

18.2.2.1 Monitoring of symptoms and activities of daily living Motor symptoms, especially gait disturbances, are the most common symptoms in PD patients and they significantly affect activities of daily living and thus the quality of life. The assessment of the severity of motor symptoms, therefore, plays an important role in the diagnosis and evaluation of the severity of the disease. In clinical practice, this is usually assessed by the Movement Disorder Unified Parkinson Disease Rating Scale (MDS-UPDRS) and the Hoehn and Yahr scale. These rating instruments, however, show high interrater variability and often fail to capture symptom fluctuation [11].

Multiple Sclerosis and Parkinson’s Disease

Wearable sensor devices are therefore a promising approach to measuring symptom burden in daily life (e.g., nocturnal hypokinesia, freezing, falls) and can obtain reliable longitudinal information, especially by covering real-life conditions. As symptoms can fluctuate extensively, motor symptoms like freezing, dyskinesias, ON and OFF phases, and falls due to postural instability may not be present at patient visits. Wearables can therefore help to provide additional information for treatment decisions. A recent systematic review by Ancona et al. [2] of 26 published reports (years 2008 to 2020) regarding wearables for home-based assessment in PD revealed that most of the used wearables include sensors for accelerometric and/or gyroscopic measures. Motor items assessed, were bradykinesia, tremor, motor fluctuations, postural instability, axial symptoms, and gait disturbances. Thereby it was possible to discriminate between PD patients and controls and to detect postural abnormalities with high sensitivity (99.3%) and specificity (76.4%) and thereby distinguish between PD fallers and PD non-fallers. For validation of sensor data, matching with MDS-UPDRS and/or motor diary and/or questionnaires and/ or video monitoring was performed. Most of the reports revealed high accuracy and reliability of wearables in detecting abnormal movements under study conditions. However, some studies demonstrated that systems designed for high user comfort and therefore good compliance resulted in significantly lower data quality in daily use. In another systematic review by Channa et. al. [11], different types of wearables in PD were analyzed, and it was concluded that the most suitable wearable sensors to detect gait disturbance are wearable insoles, calling for further large-scale studies to validate the findings. In comparison, no established wearables monitoring the often less noted but also the quality of life compromising nonmotor symptoms of PD are available. As non-motor symptoms usually precede motor symptoms of PD by years, they are of high interest for early diagnosis. Areas covered by the research, include among others monitoring for blood pressure, inertial sensors for nocturia, actigraphy for sleep and sleepiness, and smart belts for gastrointestinal symptoms [69]. Despite the promising results of data collection by wearables and subsequent analysis, large-scale clinical use has been limited.

369

370

Digital Medicine in Neurology

Furthermore, as most studies focus on single sensor-based approaches, perspectives regarding the possible advantages of the combination of multi-sensoric data assessment are lacking. Additionally, apart from collecting real-time (passive) data outside clinical settings, app-based systems also offer to include selfreported (active) data from diaries, questionnaires, tests, and PROs and so can help to close gaps in longitudinal information assessment [38].

18.2.2.2 Digital treatment support

For advancing current PD treatments, various digital treatment options have been developed in recent years and at least to some extent already validated but lack widespread usage in clinical practice. They range from app-based solutions to telemedicine applications, digital therapeutic platforms, and optimized DBS. Also, self-care monitoring and self-care management are important for individual PD coping. In a systemic review, Lee et al. [37] identified 17 studies regarding mobile applications for self-care in PD from 2013 to 2020, including three randomized controlled trials. The apps included functions like motor and non-motor symptom data collection (manually through PROs or diaries; or through sensors), and self-management functions like reminders (e.g., medication intake) or user interactions (e.g., games). The effects of using the app were measured through changes in symptoms or activity levels. Despite apps for PD being specifically developed and demonstrating high levels of adherence, the positive effects were mainly in the field of self-monitoring, but not self-management and quality of life improvement. For example, a promising approach is the result of a proof-of-concept study using the MedRhythms tool, connecting real-time walking cadence analysis of a patient with individual music playing, and providing direct feedback with real-time modulation of gait. This “rhythmic locomotor training program” has already been proved in post-stroke patients [32], but might also be transferred to PD patients [17]. Wearables may also be used for treatment optimization, e.g., regarding symptom fluctuations. In an L-Dopa-response study [57] treatment response and side effects were analyzed by using wearable tools for daily living, enabling physicians to modify medication through the in-clinic decision. However, in this study,

Multiple Sclerosis and Parkinson’s Disease

the at-home environment was simulated in a hospital to validate the data collected by wearables against UPDRS scores, limiting the transfer of results to real-life conditions. Further applications of digitally supported treatment include health exercise programs through mobile health apps, leading to promising results regarding improved physical activity, especially for less active PD patients [14]. Due to the broad spectrum of symptoms and the related disability, there is a need for multidisciplinary expert teams for better and individual care, also to prevent the loss of autonomy. Video conferencing systems with data visualization and data exchange options for remote delivery of multispecialty care can result in better care and empower patients to monitor their disease [41]. DBS is a surgical treatment option for PD, but the selection of possible candidates, the optimal placement of electrodes, and the optimization of stimulation patterns during the disease course under real-life conditions are still challenging. Various studies based on machine-learning strategies have shown the impressive potential of digital medicine in this field, however still in the early stages [71].

18.2.2.3 Automated diagnosis, prediction of disease severity, and treatment decisions

As already mentioned, the current diagnosis of PD mainly relies on clinical findings being prone to subjective (mis-)interpretation. Digital biomarkers as well as imaging – through machine learning – may facilitate early and correct diagnosis. In recent years, several studies on automated imaging in parkinsonism have been published [4, 12, 52]. For example, Automated Imaging Differentiation in Parkinsonism (AID-P) was a multi-site study including 1002 subjects (511 with PD, 84 with MSA, 129 with PSP, and 278 healthy controls) to analyze the accuracy of a machine-learning application based on diffusion MRI and the MDS-UPDRS. As a result, it was possible to distinguish between PD and Parkinsonian syndromes with high accuracy, sensitivity, and specificity. A limitation of the study was that no patients with other Parkinsonian syndromes like CBD or DLB were included. Applications of machine learning in PD diagnosis were the object of a systemic review of 209 publications from 2009 until 2021 [46], including among others analyses of handwriting patterns, voice

371

372

Digital Medicine in Neurology

recordings, movements, neuro-imaging (SPECT, PET, MRI), CSF, and serum data, partially in combination. The review compared different machine-learning models in terms of accuracy of detecting PD against healthy controls or atypical Parkinsonian syndromes. In general, most models showed a comparably good prediction and accuracy, well above chance. For example, for voice recording, 49 studies showed an average accuracy of 90.9% (standard deviation (SD) 8.6); for movement 43 studies 89.1% (SD 8.3); for MRI 32 studies 87.5% (SD 8.0); for handwriting patterns 15 studies 87% (SD 6.3); for SPECT 12 studies 94.4% (SD 4.2); for PET 4 studies an average accuracy of 85.6% (SD 6.6). Especially algorithms of the support vector machine, neural networks, and ensemble learning proved to deliver the highest classification accuracy, highlighting the potential of data-derived diagnostic performance. However, as a limitation only some of the technical approaches in the study were validated in clinical settings, stressing the importance of further research. As another restriction, the impact of quality of training material for machine-learning output quality was not included. Most currently, “deep phenotyping” of PD as another approach to deep learning has been propagated as the most promising effort in identifying PD biomarkers in a current review [16]. Deep phenotyping or multimodal phenotypic approaches draw from a broad range of digital and non-digital data. Gathering data through the combination of a broad variety of sources, e.g., wearable derived data, classical rating scales for motor function, mood, sleep, and other sources like blood analyses or imaging might facilitate early diagnosis. Data integration and data sharing will be essential for feeding machine or deep-learning algorithms. For example, the following projects for PD-related collection of standard datasets for further development of biomarkers and/or prognostic indicators, focus on clinical as well as pre-clinical areas: Besides MS, the DIFUTURE consortium also collects standardized clinical data for PD at different sites in Germany, providing the infrastructural basis ensuring privacy, IT security, and data protection [34]. Intelligent Parkinson eaRly detectiOn Guiding NOvel Supportive InterventionS (i-PROGNOSIS) focuses on the non-clinical collection of data through personal mobile devices, with the aim of tracking the transition from healthy to PD for future machine learning

Figure 18.1 Overview of applications of digital medicine in neurology. CSF, cerebrospinal fluid; DBS, deep brain stimulation; OCT, optical coherence tomography; PROs, patient-reported outcomes.

Multiple Sclerosis and Parkinson’s Disease 373

374

Digital Medicine in Neurology

(https://cordis.europa.eu/project/id/690494/de). Third, the Parkinson’s Disease Digital Biomarker DREAM Challenge uses two benchmarked crowd-sourced data sets from a remote smartphonebased study (mPower) and a multi-wearable clinical study that included symptom severity assessment by trained clinicians (L-dopa Response Study) to predict performance for PD status through machine-learning application. With their method, they were able to predict the presence of PD and the severity of the symptoms of tremor, dyskinesia, and bradykinesia with high performance [63]. Despite all achievements, further efforts are necessary to implement digital medicine in PD, especially for its use in largescale clinical practice. A European initiative embedding this into a dedicated ethical, legal, and social aspects-framework (ELSA) is DIGIPD (Validating DIGItal biomarkers for better-personalized treatment of PD). It aims at providing physicians, pharmaceutical companies, and other healthcare professionals to deliver groupings of PD patients in comparable disease progression to develop and optimize individual and early treatment options [20].

18.3 Summary and Outlook

Digital medicine in neurology has markedly developed in recent years: Longitudinal data collection, collecting big data, also by including a variety of sensor types and apps, implementing platforms for telemedicine and machine-learning applications for various clinical situations has a huge potential for facing diagnostic challenges and improving patient care. Figure 18.1 depicts the highly interdependent, already implemented, and soon-to-be implemented areas for digital medicine in neurology. Future challenges include large-scale validations, data protection, acceptance, availability, and expenses of technology, and securing data quality.

References

1. Abbadessa, G., Brigo, F., Clerico, M., Mercanti, S. de, Trojsi, F., Tedeschi, G., et al. (2022). Digital therapeutics in neurology. J Neurol. 3, pp. 1209–1224.

References

2. Ancona, S., Faraci, F. D., Khatab, E., Fiorillo, L., Gnarra, O., Nef, T., et al. (2022). Wearables in the home-based assessment of abnormal movements in Parkinson’s disease: A systematic review of the literature. J Neurol. 1, pp. 100–110. 3. Angelis, M. de, Lavorgna, L., Carotenuto, A., Petruzzo, M., Lanzillo, R., Brescia Morra, V., et al. (2021). Digital technology in clinical trials for multiple sclerosis: Systematic review. J Clin Med. 11, p. 2328.

4. Archer, D. B., Bricker, J. T., Chu, W. T., Burciu, R. G., McCracken, J. L., Lai, S., et al. (2019). Development and validation of the automated imaging differentiation in parkinsonism (AID-P): A multicentre machine learning study. Lancet Digit Health. 5, e222–e231. 5. Armstrong, M. J., and Okun, M. S. (2020). Diagnosis and treatment of Parkinson disease: A review. JAMA. 6, pp. 548–560.

6. Bayas, A., Ouallet, J. C., Kallmann, B., Hupperts, R., Fulda, U., and Marhardt, K. (2015). Adherence to, and effectiveness of, subcutaneous interferon β-1a administered by RebiSmart® in patients with relapsing multiple sclerosis: Results of the 1-year, observational SMART study. Expert Opin Drug Deliv. 8, pp. 1239–1250. 7. Block, V. J., Lizée, A., Crabtree-Hartman, E., Bevan, C. J., Graves, J. S., Bove, R., et al. (2017). Continuous daily assessment of multiple sclerosis disability using remote step count monitoring. J Neurol. 2, pp. 316–326.

8. Bloem, B. R., Okun, M. S., and Klein, C. (2021). Parkinson’s disease. Lancet. 10291, pp. 2284–2303. 9. Briggs, F. B. S., Thompson, N. R., and Conway, D. S. (2019). Prognostic factors of disability in relapsing remitting multiple sclerosis. Mult Scler Relat Disord, pp. 9–16. 10. Chan, J. Y. C., Yau, S. T. Y., Kwok, T. C. Y., and Tsoi, K. K. F. (2021). Diagnostic performance of digital cognitive tests for the identification of MCI and dementia: A systematic review. Ageing Res Rev, p. 101506. 11. Channa, A., Popescu, N., and Ciobanu, V. (2020). Wearable solutions for patients with Parkinson’s disease and neurocognitive disorder: A systematic review. Sensors (Basel). 9, p. 2713.

12. Chougar, L., Pyatigorskaya, N., Degos, B., Grabli, D., and Lehéricy, S. (2020). The role of magnetic resonance imaging for the diagnosis of atypical parkinsonism. Front Neurol, p. 665. 13. Comi, G., Radaelli, M., and Soelberg Sørensen, P. (2017). Evolving concepts in the treatment of relapsing multiple sclerosis. Lancet. 10076, pp. 1347–1356.

375

376

Digital Medicine in Neurology

14. Crotty, G. F., and Schwarzschild, M. A. (2020). Chasing protection in Parkinson’s disease: Does exercise reduce risk and progression? Front. Aging Neurosci., p. 186.

15. Dillenseger, A., Weidemann, M. L., Trentzsch, K., Inojosa, H., Haase, R., Schriefer, D., et al. (2021). Digital biomarkers in multiple sclerosis. Brain Sci. 11, p. 1519. 16. Dorsey, E. R., Omberg, L., Waddell, E., Adams, J. L., Adams, R., Ali, M. R., et al. (2020). Deep phenotyping of Parkinson’s disease. J Parkinsons Dis. 3, pp. 855–873. 17. Ellis, T. D., and Earhart, G. M. (2021). Digital therapeutics in Parkinson’s disease: Practical applications and future potential. J Parkinsons Dis. s1, S95–S101.

18. Feys, P., Lamers, I., Francis, G., Benedict, R., Phillips, G., LaRocca, N., et al. (2017). The Nine-Hole Peg Test as a manual dexterity performance measure for multiple sclerosis. Mult Scler. 5, pp. 711–720. 19. Flachenecker, F., Gaßner, H., Hannik, J., Lee, D.-H., Flachenecker, P., Winkler, J., et al. (2019). Objective sensor-based gait measures reflect motor impairment in multiple sclerosis patients: Reliability and clinical validation of a wearable sensor device. Mult Scler Relat Disord, p. 101903. 20. Fröhlich, H., Bontridder, N., Petrovska-Delacréta, D., Glaab, E., Kluge, F., Yacoubi, M. E., et al. (2022). Leveraging the potential of digital technology for better individualized treatment of Parkinson’s disease. Front. Neurol. 13, p. 788427.

21. Fruhwirth, V., Enzinger, C., Weiss, E., Schwerdtfeger, A., Gattringer, T., and Pinter, D. (2019). Apps in der Sekundärprävention nach Schlaganfall. Wiener medizinische Wochenschrift. 170, p. 9735. 22. Golan, D., Sagiv, S., Glass-Marmor, L., and Miller, A. (2020). Mobile phone-based e-diary for assessment and enhancement of medications adherence among patients with multiple sclerosis. Mult Scler J Exp Transl Clin. 3, 2055217320939309.

23. Golan, D., Sagiv, S., Glass-Marmor, L., and Miller, A. (2021). Mobilephone-based e-diary derived patient reported outcomes: Association with clinical disease activity, psychological status and quality of life of patients with multiple sclerosis. PLoS One. 5, e0250647.

24. Gooch, C. L., Pracht, E., and Borenstein, A. R. (2017). The burden of neurological disease in the United States: A summary report and call to action. Ann Neurol. 4, pp. 479–484.

References

25. Green, A. J., McQuaid, S., Hauser, S. L., Allen, I. V., and Lyness, R. (2010). Ocular pathology in multiple sclerosis: Retinal atrophy and inflammation irrespective of disease duration. Brain. 6, pp. 1591– 1601. 26. Gunn, H. J., Newell, P., Haas, B., Marsden, J. F., and Freeman, J. A. (2013). Identification of risk factors for falls in multiple sclerosis: A systematic review and meta-analysis. Phys Ther. 4, pp. 504–513. 27. Hayes, M. T. (2019). Parkinson’s disease and parkinsonism. Am J Med. 7, pp. 802–807. 28. Hemingway, H., Croft, P., Perel, P., Hayden, J. A., Abrams, K., Timmis, A., et al. (2013). Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes. BMJ, e5595.

29. Hingorani, A. D., van der Windt, D. A., Riley, R. D., Abrams, K., Moons, K. G. M., Steyerberg, E. W., et al. (2013). Prognosis research strategy (PROGRESS) 4: Stratified medicine research. BMJ, e5793. 30. Hohlfeld, R., Dornmair, K., Meinl, E., and Wekerle, H. (2016). The search for the target antigens of multiple sclerosis, part 1: Autoreactive CD4+ T lymphocytes as pathogenic effectors and therapeutic targets. Lancet Neurol. 2, pp. 198–209. 31. Hohlfeld, R., Dornmair, K., Meinl, E., and Wekerle, H. (2016). The search for the target antigens of multiple sclerosis, part 2: CD8+ T cells, B cells, and antibodies in the focus of reverse-translational research. Lancet Neurol. 3, pp. 317–331.

32. Hutchinson, K., Sloutsky, R., Collimore, A., Adams, B., Harris, B., Ellis, T. D., et al. (2020). A music-based digital therapeutic: Proof-of-concept automation of a progressive and individualized rhythm-based walking training program after stroke. Neurorehabil Neural Repair. 11, pp. 986–996. 33. Janca, A., Aarli, J. A., Prilipko, L., Dua, T., Saxena, S., and Saraceno, B. (2006). WHO/WFN Survey of neurological services: A worldwide perspective. J Neurol Sci. 1, pp. 29–34. 34. Kohlbacher, O., Mansmann, U., Bauer, B., Kuhn, K., and Prasser, F. (2018). Data integration for future medicine (DIFUTURE): An architectural and methodological overview. Methods Inf Med. 57, e57–e65.

35. Lang, M., Rau, D., Cepek, L., Cürten, F., Ringbauer, S., and Mayr, M. (2021). An ID-associated application to facilitate patient-tailored management of multiple sclerosis. Brain Sci. 8, p. 1061.

36. Lavorgna, L., Miele, G., Petruzzo, M., Lanzillo, R., and Bonavita, S. (2018). Online validation of the Italian version of the patient determined

377

378

Digital Medicine in Neurology

disease steps scale (PDDS) in people with multiple sclerosis. Mult Scler Relat Disord, pp. 108–109.

37. Lee, J., Yeom, I., Chung, M. L., Kim, Y., Yoo, S., and Kim, E. (2022). Use of mobile apps for self-care in people with Parkinson disease: Systematic review. JMIR Mhealth Uhealth. 1, e33944. 38. Little, M. A. (2021). Smartphones for remote symptom monitoring of Parkinson’s disease. J Parkinsons Dis. s1, S49–S53.

39. Lladó, X., Ganiler, O., Oliver, A., Martí, R., Freixenet, J., Valls, L., et al. (2011). Automated detection of multiple sclerosis lesions in serial brain MRI. Neuroradiology. 8, pp. 787–807. 40. Lovera, J., and Kovner, B. (2012). Cognitive impairment in multiple sclerosis. Curr Neurol Neurosci Rep. 5, pp. 618–627.

41. Luis-Martínez, R., Monje, M. H. G., Antonini, A., Sánchez-Ferro, Á., and Mestre, T. A. (2020). Technology-enabled care: Integrating multidisciplinary care in Parkinson’s disease through digital technology. Front. Neurol., p. 575975.

42. Ma, Y., Zhang, C., Cabezas, M., Song, Y., Tang, Z., Liu, D., et al. (2022). Multiple sclerosis lesion analysis in brain magnetic resonance images: Techniques and clinical applications. IEEE J Biomed Health Inform. 43. Martin, C. L., Phillips, B. A., Kilpatrick, T. J., Butzkueven, H., Tubridy, N., McDonald, E., et al. (2006). Gait and balance impairment in early multiple sclerosis in the absence of clinical disability. Mult Scler. 5, pp. 620–628. 44. Martinez-Lapiscina, E. H., Arnow, S., Wilson, J. A., Saidha, S., Preiningerova, J. L., Oberwahrenbrock, T., et al. (2016). Retinal thickness measured with optical coherence tomography and risk of disability worsening in multiple sclerosis: a cohort study. Lancet Neurol. 6, pp. 574–584. 45. Marziniak, M., Brichetto, G., Feys, P., Meyding-Lamadé, U., Vernon, K., and Meuth, S. G. (2018). The use of digital and remote communication technologies as a tool for multiple sclerosis management: Narrative review. JMIR Rehabil Assist Technol. 1, e5. 46. Mei, J., Desrosiers, C., and Frasnelli, J. (2021). Machine learning for the diagnosis of Parkinson’s disease: A review of literature. Front. Aging Neurosci., p. 633752.

47. Messina, R., and Filippi, M. (2020). What we gain from machine learning studies in headache patients. Front. Neurol., p. 221. 48. Minen, M. T. (2016). Electronic behavioral interventions for headache: A systematic review. J Headache Pain. 1, pp. 1–20.

References

49. Montalban, X., Graves, J., Midaglia, L., Mulero, P., Julian, L., Baker, M., et al. (2022). A smartphone sensor-based digital outcome assessment of multiple sclerosis. Multiple sclerosis. 4, pp. 654–664. 50. Montolío, A., Cegoñino, J., Garcia-Martin, E., and Del Pérez Palomar, A. (2022). Comparison of machine learning methods using spectralis OCT for diagnosis and disability progression prognosis in multiple sclerosis. Ann Biomed Eng. 5, pp. 507–528. 51. Motl, R. W., Cohen, J. A., Benedict, R., Phillips, G., LaRocca, N., Hudson, L. D., et al. (2017). Validity of the timed 25-foot walk as an ambulatory performance outcome measure for multiple sclerosis. Mult Scler. 5, pp. 704–710.

52. Papathoma, P.-E., Markaki, I., Tang, C., Lilja Lindström, M., Savitcheva, I., Eidelberg, D., et al. (2022). A replication study, systematic review and meta-analysis of automated image-based diagnosis in Parkinsonism. Sci Rep. 1, p. 2763.

53. Parmenter, B. A., Weinstock-Guttman, B., Garg, N., Munschauer, F., and Benedict, R. H. B. (2007). Screening for cognitive impairment in multiple sclerosis using the Symbol digit Modalities Test. Mult Scler. 1, pp. 52–57. 54. Patel, U. K., Anwar, A., Saleem, S., Malik, P., Rasul, B., Patel, K., et al. (2019). Artificial intelligence as an emerging technology in the current care of neurological disorders. J Neurol. 5, pp. 1623–1642. 55. Pedersen, M., Verspoor, K., Jenkinson, M., Law, M., Abbott, D. F., and Jackson, G. D. (2020). Artificial intelligence for clinical decision support in neurology. Brain Commun. 2, fcaa096. 56. Pham, L., Harris, T., Varosanec, M., Morgan, V., Kosa, P., and Bielekova, B. (2021). Smartphone-based symbol-digit modalities test reliably captures brain damage in multiple sclerosis. npj Digit. Med. 1, p. 36.

57. Pulliam, C. L., Heldman, D. A., Brokaw, E. B., Mera, T. O., Mari, Z. K., and Burack, M. A. (2018). Continuous assessment of levodopa response in Parkinson’s disease using wearable motion sensors. IEEE Trans Biomed Eng. 1, pp. 159–164. 58. Riley, R. D., Hayden, J. A., Steyerberg, E. W., Moons, K. G. M., Abrams, K., Kyzas, P. A., et al. (2013). Prognosis research strategy (PROGRESS) 2: Prognostic factor research. PLoS Med. 2, e1001380.

59. Rocca, M. A., Amato, M. P., Stefano, N. de, Enzinger, C., Geurts, J. J., Penner, I.-K., et al. (2015). Clinical and imaging assessment of cognitive dysfunction in multiple sclerosis. Lancet Neurol. 3, pp. 302–317.

379

380

Digital Medicine in Neurology

60. Roche, J., McCarry, Y., and Mellors, K. (2014). Enhanced patient support services improve patient persistence with multiple sclerosis treatment. Patient Prefer Adherence. 8, pp. 805–811. 61. Rotstein, D., and Montalban, X. (2019). Reaching an evidence-based prognosis for personalized treatment of multiple sclerosis. Nat Rev Neurol. 5, pp. 287–300. 62. Shanahan, C. J., Boonstra, F. M. C., Cofré Lizama, L. E., Strik, M., Moffat, B. A., Khan, F., et al. (2017). Technologies for advanced gait and balance assessments in people with multiple sclerosis. Front. Neurol., p. 708.

63. Sieberts, S. K., Schaff, J., Duda, M., Pataki, B. Á., Sun, M., Snyder, P., et al. (2021). Crowdsourcing digital health measures to predict Parkinson’s disease severity: The Parkinson’s Disease Digital Biomarker DREAM Challenge. npj Digit. Med. 1, p. 53. 64. Silva, G. S., and Schwamm, L. H. (2021). Advances in stroke: Digital health. Stroke. 1, pp. 351–355.

65. Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., et al. (2013). Prognosis research strategy (PROGRESS) 3: Prognostic model research. PLoS Med. 2, e1001381. 66. Stubberud, A., and Linde, M. (2018). Digital technology and mobile health in behavioral migraine therapy: A narrative review. Curr Pain Headache Rep. 10, p. 66.

67. Tanigawa, M., Stein, J., Park, J., Kosa, P., Cortese, I., and Bielekova, B. (2017). Finger and foot tapping as alternative outcomes of upper and lower extremity function in multiple sclerosis. Mult Scler J Exp Transl Clin. 1, 2055217316688930.

68. van Beek J, Freitas R, Bernasconi C, et al. (2019; Seattle, Washington, USA). FLOODLIGHT Open - A global, prospective, open-access study to better understand multiple sclerosis using smartphone technology. Presented at: Annual Meeting of the Consortium of Multiple Sclerosis Centers (CMSC), https://cmsc.confex.com/cmsc/2019/mediafile/ Handout/Paper5900/QOL10_CMSC%202019%20FLOODLIGHT%20 Open%20Poster_Global_van%20Beek%20et%20alFINA.pdf. 69. van Wamelen, D. J., Sringean, J., Trivedi, D., Carroll, C. B., Schrag, A. E., Odin, P., et al. (2021). Digital health technology for non-motor symptoms in people with Parkinson’s disease: Futile or future? Parkinsonism & Related Disorders, pp. 186–194.

70. Vasta, R., Cerasa, A., Sarica, A., Bartolini, E., Martino, I., Mari, F., et al. (2018). The application of artificial intelligence to understand the pathophysiological basis of psychogenic nonepileptic seizures. Epilepsy Behav, pp. 167–172.

References

71. Watts, J., Khojandi, A., Shylo, O., and Ramdhani, R. A. (2020). Machine learning’s application in deep brain stimulation for Parkinson’s disease: A review. Brain Sci. 11, p. 809.

72. Woelfle, T., Pless, S., Wiencierz, A., Kappos, L., Naegelin, Y., and Lorscheider, J. (2021). Practice effects of mobile tests of cognition, dexterity, and mobility on patients with multiple sclerosis: Data analysis of a smartphone-based observational study. J Med Internet Res. 11, e30394.

73. Wottschel, V., Alexander, D. C., Kwok, P. P., Chard, D. T., Stromillo, M. L., Stefano, N. de, et al. (2015). Predicting outcome in clinically isolated syndrome using machine learning. Neuroimage Clin., pp. 281–287. 74. Ziemssen, T., Kern, R., Voigt, I., and Haase, R. (2020). Data collection in multiple sclerosis: The MSDS approach. Front. Neurol., p. 445.

381

Chapter 19

Neurorehabilitation Medicine

Andreas Bender,a,b Thomas Platz,c,d and Andreas Straubeb

aTherapiezentrum Burgau, Hospital for Neurorehabilitation, Burgau, Germany bDepartment of Neurology, Ludwig-Maximilians-University of Munich, Munich, Germany cNeurorehabilitation Research Group, University Medical Centre, Greifswald, Germany dBDH-Clinic Greifswald, Institute for Neurorehabilitation and Evidence-Based Practice, University of Greifswald, Greifswald, Germany [email protected]

19.1 Introduction Digital medicine is rapidly entering the field of neurorehabilitation in several distinct indications and clinical applications. This chapter aims at providing an overview of current trends and developments in this field without attempting to cover all aspects exhaustingly. Examples of digital medicine applications in neurorehabilitation range from therapeutic indications, such as robot-assisted gait or arm training, digital telerehabilitation, and virtual reality in Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

384

Neurorehabilitation Medicine

rehabilitation to diagnostic and prognostic indications. In the latter, methods of machine learning and artificial intelligence are applied to large datasets of biomarkers to predict current health states, which are not evident in routine clinical assessment, and to predict future clinical outcomes in an effort to improve long-term prognosis. Also, methods of digital medicine can be used as medical appliances and long-term assistive devices to improve functioning, activities, and participation in the life, such as eye-tracking as a means for communication. The most evidence for the effectiveness of digital medicine in neurorehabilitation stems from the post-acute rehabilitative care of stroke patients. This patient population represents the largest cohort in the field of neurorehabilitation and has an urgent clinical need for interventions aimed at improving quality of life, functioning, activities, and participation in all aspects of post-stroke life. Stroke is a leading cause of long-term morbidity and disability worldwide, with a negative effect on participation and quality of life [8]. As it is projected to become even more prevalent and relevant for the future of health care systems, we focus this review on applications of digital medicine in the field of stroke rehabilitation and here, especially on rehabilitation of motor function. It is important to note though, that digital medicine in neurorehabilitation is relevant to many other aspects of rehabilitation and body functions, ranging from speechand-language therapy to cognitive training.

19.2 Therapeutic Potential of Digital Medicine in Neurorehabilitation 19.2.1 Virtual Reality Rehabilitation

Virtual reality (VR) in general refers to an interface between a user and a computer that generates the simulation of a real-world scenario and enables the user to participate in that simulation and exert control over certain aspects of it (for review, see [10, 20]). The interaction with the simulated world can comprise several sensory canals, such as visual and auditory. VR devices can be subdivided into immersive systems with full integration of the user into the virtual world by means of head-mounted displays or large screen

Therapeutic Potential of Digital Medicine in Neurorehabilitation

projections and into semi- or non-immersive systems, where a computer screen is used to display the virtual environment (for example, Fig. 19.1) [18]. If VR hardware is used to let the user, i.e., the patient, participate in a game, where he has to achieve specific aims and is rewarded with an increasing score, this is referred to as gamification or exergames [42].

Figure 19.1 Example of immerse VR system for neurorehabilitation based on a head-mounted display and camera-based motion detection (A). Therapeutic options comprise motor tasks (B), cognitive tasks, as well as tasks tailored to mimic activities of daily living, such as cutting vegetables in a virtual kitchen (C). Images reproduced with permission by CUREOSITY GmbH, Düsseldorf, Germany, www.cureosity.de.

VR systems have been used for rehabilitation purposes for many years now and there is accumulating evidence that they may provide added value to recovery from the neurological disease [2, 3, 10, 18, 20, 25]. To further increase the immersion into VR rehabilitation protocols, VR system can be combined with robotic rehabilitation

385

386

Neurorehabilitation Medicine

devices, to provide further sensory feedback and additional movement repetition [26]. VR systems for rehabilitation may either be used within an onsite therapeutic session in a single therapy or group therapy setting, where a therapist can control and adjust the VR software as well as the patients’ movements. Or it may be used as part of a telerehabilitation program, where the patient and therapist are at different locations, or even as a standalone system in the home setting. Potential advantages of the VR rehabilitation approach over conventional rehabilitation may be:

(a) Increasing motivation for training by providing a visually attractive gaming context (b) Increasing rehabilitation dose by providing options for selftraining (c) Providing rehabilitation services to areas with few onsite rehabilitation facilities if combined with telerehabilitation (d) Providing an enriched rehabilitation environment

Disadvantages of VR-based rehabilitation may be a reduction in true human interpersonal contact between therapist and patient as well as a lack of true sensory and proprioceptive input from moving limbs. Risks and adverse events associated with the technique may be transient nausea and dizziness as well as headache [25]. In theory, falling or trauma because of contact of moving limbs with the true, i.e., the non-virtual environment may occur. VR systems have mainly been used for rehabilitation of upper limb function and activities after stroke but also for neuropsychological functions. Evaluated outcomes range from motor function with various tests, balance, executive function, attention, spasticity, quality of life (QOL), visual perception, and depression to pain [10]. The study populations were heterogenous with respect to chronicity (acute, subacute, chronic), the severity of the stroke (mild, moderate, severe), VR systems used (immersive, non- or partially immersive), types of co-therapy (none, classical physio- and occupational therapy, in combination with robotic training, in combination with transcranial direct current stimulation, tDCS), as well as setting (inpatient, outpatient, homecare). Typically, VR interventions in studies range in duration from 2 to 12 weeks with a daily dose of 20 to 60 minutes of VR therapy.

Therapeutic Potential of Digital Medicine in Neurorehabilitation

Several systematic reviews have examined the effects of VR rehabilitation on stroke outcomes. A 2017 Cochrane Review identified 72 trials on VR for stroke rehabilitation with a total of 2,470 patients [25]. The quality of the evidence was mostly low. VR did not result in significantly improved upper limb function, gait speed, or balance compared to dose-equivalent conventional therapy. VR training in addition to conventional therapy, i.e., higher rehabilitation dose, led to a statistically significant difference between groups for arm- and hand function in favor of the VR group. VR was however associated with slightly improved activities of daily living (ADL) compared to usual care. Another recent systematic review and meta-analysis including 20 clinical trials found improved upper limb motor function with a rather robust effect size compared to conventional therapy [10]. There was also a moderate effect on improving functional independence, measured with the FIM, compared to conventional care. Karamians and colleagues also performed a systematic review and meta-analysis on VR and gaming-based interventions and identified 38 suitable studies that met their inclusion criteria [20]. Overall, VR and gaming were associated with increased upper limb functions and activities compared to conventional rehabilitation methods. Further subgroup analyses showed that the impact of VR interventions was greater in acute and subacute patients compared to chronic patients and that VR interventions including gaming were more effective than VR without gaming. The positive effects of VR interventions on upper limb function were replicated by another meta-analysis, which provided additional evidence, that VR can also lead to improved lower limb function, gait velocity, balance, and overall improved ambulation [55]. It is difficult to establish the efficacy of VR rehabilitation if VR training is in addition to conventional rehabilitation without control groups, which are matched for the overall therapy dose. While it is not feasible to provide true Sham-VR rehabilitation as a control group, the control group should receive non-VR training at the same dose as the true VR group. Otherwise, such studies would study the effect of additional therapy but not a specific VR training effect. Several studies have adopted a study design with dosematched control groups. A small study found better upper extremity

387

388

Neurorehabilitation Medicine

functional recovery in the VR group compared to the occupational therapy control group in subacute stroke patients [31]. In this study, mirror therapy – an evidence-based treatment concept in stroke rehabilitation – was incorporated into the VR treatment program. It relies on activating mirror neurons by observing how a patient’s paretic arm is seemingly moving, while in reality, only the nonaffected arm is moving [27, 33]. In conventional rehabilitation, the patient observes a mirror giving the impression that the unaffected arm is moving while indeed looking at the mirror image of the nonparetic arm moving. VR rehabilitation is uniquely suited to stimulate mirror neurons in a fully immersive virtual environment when the patient’s avatar is moving both arms and hands, while the patient is only moving the unaffected limb. VR rehabilitation may also be effective if commercially available, mass production gaming systems and additions like balance boards are used instead of high-end medical products [29]. Combining VR rehabilitation with additional rehabilitation devices, such as robots for gait- or arm-training or wearable devices that provide motion-sensitive sensors for additional input to the VR software, such as special smart gloves, may be of additional value. Four weeks of non-immersive VR training with a smart glove for 30 minutes per day improved hand function and ADL when applied in addition to standard rehabilitation measures [36]. Fully immersive VR systems in combination with robotic arm training with an exoskeleton are providing similar improvements in arm activities as conventional occupational therapy [22]. In this SMARTS2 study, stroke patients navigated a dolphin freely in the ocean, exploring its environment and performing different tasks, such as fighting with a shark or eating fish, or just playing in a fully immersive VR room. The dolphin was controlled with the paretic arm, which was supported by the ARMEO Power® robotic training system. Even though robot-assisted VR training was not superior to the same dose of conventional therapy, it can be performed with less one-on-one supervision by a therapist or even in a group therapy session, so therapy becomes more efficient. In addition to measures of improved motor function, studies showed improved QOL, as measured with the EuroQOL, when conventional rehabilitation is enriched by added VR [39].

Therapeutic Potential of Digital Medicine in Neurorehabilitation

Outside the field of motor rehabilitation, VR has been established for the rehabilitation of patients with disorders of balance and vertigo [17, 54]. In some programs, patients are asked to move the center of gravity of their bodies within the limits provided by the program and so to train the control of their body sway. Also, VR plays an increasingly important role in the field of rehabilitation of balance and gait in Parkinson´s disease, where evidence suggests, that VR is superior to conventional physical therapy [12]. Apart from indications to improve motor function, balance, or gait, VR-augmented rehabilitation is currently being evaluated for improvement of unilateral spatial neglect [6, 34] or aphasia [16]. There is scientific level I evidence that VR rehabilitation can improve independence in the ADL as well as motor function and activity when it is added to the rehabilitation of stroke patients. VR rehabilitation may also be effective in the improvement of non-motor deficits and patient populations apart from acute or chronic stroke.

19.2.2 Telerehabilitation

Telerehabilitation refers to a rehabilitation setting, where professional rehabilitation providers and patients are at separate locations and where modern information technology is used to instruct and supervise patients during training. Telerehabilitation may be a more convenient and less expensive way of providing rehabilitation than the usual in-person setting, especially for patients, who do not have medical indications for staying in a rehabilitation hospital. It has the additional advantage of less disruption of normal day routines since patients can stay at home. It is suggested that in certain settings, telerehabilitation may not be inferior to conventional in-person rehabilitation [24]. Telerehabilitation may also decrease the burden of inpatient hospital care, which is especially relevant in periods of high hospital strain as in the case of the COVID-19 pandemic [44, 47]. Telerehabilitation may also be a cost-effective option for remote or underserviced areas [46]. A meta-analysis of meta-analyses found that telerehabilitation was not inferior to standard rehabilitation services for the indications of cardiorespiratory and musculoskeletal disease and

389

390

Neurorehabilitation Medicine

potentially even slightly superior in the case of neurological disease regarding physical functioning [44]. One point of criticism of telerehabilitation is that compliance may be low due to lack of social interaction. It is very interesting that even this feature of social interaction can be incorporated into telerehabilitation programs. A small study used a combination of telerehabilitation with VR gamification rehabilitation and compared the compliance and efficacy of single-user versus multiple-user therapy sessions [48]. The impression of social interaction in the group therapy setting was created by having the avatars of different patients interacting in the virtual world by for example playing a ball game. Overall, compliance with this rehabilitation program was very high. Results suggested that the group setting may indeed be beneficial over the single-user setting. A very smart recent trial compared different forms of homebased VR/gaming versus intensive face-to-face therapy provided by therapists in community-dwelling stroke patients in a chronic disease phase, i.e., several years after their stroke [14]. The intensive faceto-face treatment group without VR/gaming received constrained induced movement therapy (CIMT), which is an evidence-based method of mass motor practice. The home-based, tele-rehabilitated VR/gaming group showed similar motor function improvement as the intensive in-center CIMT group but required only 20% of the therapist resources, required for the in-person therapy. In addition, VR/gaming was more effective than classical low-intensity rehabilitative training, which often is the standard in chronic stroke patients. In this randomized trial, a low-cost VR/gaming system was used, which relies on movement detection by a camera and transfer of the detected patient´s movement onto an avatar. This is another excellent example that VR equipment does not necessarily have to be associated with high cost. Telerehabilitation is not limited to motor rehabilitation but is also suitable for speech and language rehabilitation, for example in stroke patients with aphasia, who were successfully provided with a tablet-based VR rehabilitation system to improve language functions [28]. Rehabilitation of Parkinson’s disease (PD) and multiple sclerosis are further examples of the successful use of telerehabilitation approaches [41, 49]. In the case of PD, additional feedback information for healthcare professionals may be provided using wearables, in order to track tremor amplitudes and physical activity.

Therapeutic Potential of Digital Medicine in Neurorehabilitation

There is increasing evidence that telerehabilitation may be an effective form of rehabilitation, especially in situations of low inor outpatient treatment capacity for patients with sufficient ADL independence to live at home. Telerehabilitation can be successfully combined with VR- and gamification systems.

19.2.3 Robotic Therapy in Neurorehabilitation

Robotic therapy in motor rehabilitation refers to microprocessorcontrolled devices that support patients with paresis by enabling joint- and limb movements through electric actuators. These often electromechanical devices are typically used for rehabilitation of arm- and hand function or for gait, even though there is an increasing number of robots to address other needs, such as e.g. animal-assisted therapy [5]. Electromechanical gait training is probably the most common and most available form of robotic motor rehabilitation. Three different types of devices can be distinguished: (i) stationary endeffector devices, where leg movement is provided by moving foot plates onto which patients are strapped, (ii) stationary exoskeleton devices, where the complete limb is fixed to a robotic skeleton, and (iii) mobile exoskeletons, where the limbs are attached to a robotic skeleton and the whole system is capable of “walking,” i.e., covering a certain distance on the ground (for examples, see Fig. 19.2). Overall, end-effector devices, such as in Fig. 19.2A, seem to be more effective for rehabilitation of gait than exoskeletons. In general, electromechanical gait training is quite effective with the number 7, needed to treat an additional positive outcome. This means that for every seven patients, who are unable to walk, for example, after stroke, and who receive such training, the walking capability of one additional patient will recover [30]. Stationary robotic gait training devices are often combined with visual feedback on the weight and pressure distribution of the feet as a form of biofeedback to improve the symmetry of the gait cycle. They typically also contain forms of VR technology to provide the patient with the impression of actually walking through landscapes to increase motivation and provide context for the walking task.

391

392

Neurorehabilitation Medicine

Figure 19.2 Examples of robot-assisted therapy in neurorehabilitation. (A) End-effector device for electromechanical gait training; (B) stationary exoskeleton device for electromechanical gait training; (C) exoskeleton device for electromechanical arm training in combination with exergaming (screen); (D) mobile exoskeleton for electromechanical overground gait training. Images provided with permission by Therapiezentrum Burgau (A, C), https://www. hocoma.com/de/(B), and www.eksobionics.com (D).

Robot-assisted gait therapy, especially if high doses of training are applied, not only increases functional ambulation but also improves balance [51]. Mobile robotic devices for overground gait rehabilitation provide the benefit of bringing this form of training to outpatient- or home-

Therapeutic Potential of Digital Medicine in Neurorehabilitation

based rehabilitation. A recent study showed that daily training with a bionic leg orthosis for 10 weeks improved leg function and walking capacity compared to equivalent doses of conventional physiotherapy in chronic stroke patients [52]. Robot-assisted rehabilitation of upper extremity motor impairment, which is also often combined with gamification and VR, was recently shown to be superior to dose-matched conventional rehabilitation in certain patient subgroups after stroke [53]. Again, also for the upper extremities, end-effector devices seemed to be more efficient than exoskeleton devices. In general, effect sizes for the superiority of robotic training are small though. Advantages of robotic rehabilitation devices over conventional therapy may be:

∑ Increase in the number of movement repetitions as the key principle of motor rehabilitation ∑ Increase in motivation by means of VR/gamification in technology-prone patients ∑ Decreasing physical strain on therapists ∑ Increasing therapy efficiency by enabling group therapy sessions

Disadvantages may be, that more time is needed for setting up the robotic rehabilitation device for each patient and the oftenenormous costs of such systems, which are rarely reimbursed by health insurance. Another option is the combination of exoskeletons with functional muscle stimulation in order to activate spinal neuronal mechanisms in order to re-establish gait or pedaling patterns [45]. Recently, also completely implantable functional electrostimulation (FES) systems were tested for the rehabilitation of gait in stroke patients, showing preliminary evidence that they may be beneficial for improving walking endurance and walking speed [19]. Robotic training devices are effective for rehabilitation of upper and lower extremity motor function with comparably low numbers needed to treat (grade I level of evidence). By increasing the number of movement repetitions, they may promote recovery of function more effectively and more efficiently than conventional therapy. Robotic training may be combined with VR technology to provide additional motivation and context to patients.

393

394

Neurorehabilitation Medicine

19.2.4 Humanoid Robot Assistance in Neurorehabilitation Stroke-related disabilities (“neuro-disabilities”) are increasing significantly worldwide, a trend that will continue in the years to come (GBD, 2021). At the same time, an evidence-based neurorehabilitative treatment that is adequate both in terms of contents and “dosage” (intensity) can sustainably reduce disability and restore independence in everyday life. Both the specificity of training, i.e., targeting the deficient brain function, and sufficiently repetitive and challenging training schedules that are neither too demanding, nor not challenging enough for an individual stroke survivor at her or his current status of recovery, are needed for prolonged times to promote functional recovery after brain damage. Indeed, such therapies have been identified by neurorehabilitation research evidence including clinical trials and meta-analyses [37]. In many areas of brain dysfunction, e.g., motor functions, perception, language, or cognition evidence-based practice recommendations guide the way to achieve at least partial recovery by structured training that frequently involves daily therapeutic sessions over weeks to months. Such therapy requires specifically trained qualified therapeutic staff that continuously instructs patients what and how to do, gives personalized feedback, encourages patients to sustain practicing functions they are not “good at,” and supports motivation. But how can this be achieved if the demand grows, but the human resources presumably cannot be enlarged to the extent needed? Humanoid robots as socially interactive therapy assistants in neurorehabilitation have the potential to expand the scope of action of human practitioners (e.g., therapists). In principle, therapeutic activities are planned co-ordinations (joint actions of therapists with patients following goals) with success criteria that are both internal to that coordination, i.e., implementation of training schedules as well as promotion of patients’ motivation and adherence to the therapy applied, and external to that coordination, i.e., the intended clinical outcome [32].

Therapeutic Potential of Digital Medicine in Neurorehabilitation

Accordingly, there are these two focuses of therapeutic activities, one related to the therapeutic process itself, and one related to the intended clinical outcome. In the context of neurorehabilitation, therapists frequently have the task of “managing” the therapeutic process in a way that the (active) components of the applied therapy promoting the intended recovery of brain functions are realized, a task that requires knowledge about specific brain functions, clinical and therapeutic knowledge, and affords the active and ongoing integration of individual aspects relevant for the therapy and a personalized socially interactive set-up. If humanoid robots could successfully be used for digital medicine applications in neurorehabilitation achieving these characteristics of therapist-led interventions, and training with them would be similarly clinically effective and acceptable for stroke survivors, and various neuro-disabilities addressable by such training could be reduced or even avoided. Until more recently, this has only been a scientific concept. Such humanoid robot applications for therapy in neurorehabilitation did not exist. And for their development, a number of scientific questions need to be addressed requiring a genuine inter-professional approach. The multi-professional research network evidence-based robot assistance in neurorehabilitation, E-BRAiN (www.ebrainscience.de), was set forth by Thomas Platz and colleagues to answer some of the related questions (Fig. 19.3).

Figure 19.3 The humanoid robot gives specific training instructions with a natural way of gesturing (copyright Thomas Platz, 2022).

395

396

Neurorehabilitation Medicine

Within the E-BRAiN concept, digital neurorehabilitative treatment (for arm motor and neuro-visual functions) using a humanoid robot is developed collaboratively by clinicians and computer scientists [13]. Since a humanoid robot has a “human-like” appearance and interacts with speech and mimics, its activity as a “therapeutic assistant” has the potential to be designed to mimic human therapeutic interaction comprehensively [38]. Accordingly, the therapeutic system E-BRAiN uses therapeutic goal-related information, training specifications, audiovisual instructions, verbal and graphical training-specific feedback with positive social stimuli, and both a general therapeutic and training-specific complex dialogue structure leading through a therapeutic session from “welcome, Mrs. …” to “I’m looking forward to seeing you for our next session tomorrow morning at 10.” First patient experiences using the E-BRAiN system had been positive while a more formal evaluation is underway [38].

19.3 Diagnostic and Prognostic Potential of Digital Medicine in Neurorehabilitation

Another well-established application of digital medicine in the field of neurorehabilitation is the assessment of unresponsive patients following acute severe brain injuries, such as traumatic brain injury (TBI), hypoxic-ischemic encephalopathy due to cardiac arrest, or subarachnoid hemorrhage (SAH). Here, methods of machine learning and artificial intelligence (ML/AI) are applied to large sets of biological data, stemming for example from high-density EEG recordings (HD-EEG) or from neuroimaging methods, such as functional MRI (fMRI) or FDG-PET (for examples, see Fig. 19.4). The aim of such techniques is to find signatures of preserved consciousness in clinically unconscious patients. If patients have their eyes open but do not respond to their environment and do not react purposefully, their clinical state is referred to as the UWS; traditional terminology: vegetative state) [23]. It has been increasingly becoming clear though, that 10–20% of such patients may indeed be conscious and reactive to their environment, without being able to clinically show it due to a lack of behavioral motor options [35, 40]. It is important to identify such patients, who suffer from Cognitive-Motor-Dissociation (CMD), because they may have a

Diagnostic and Prognostic Potential of Digital Medicine in Neurorehabilitation

better prognosis and outcome than patients in true UWS. Because clinical evaluation alone is not sufficient to detect such “hidden” consciousness, various high-tech methods have been developed to assist in establishing the correct diagnosis [4, 21]. This is extremely important, since the rates of misdiagnoses in this group of patients are still unacceptably high, ranging up to 40% of UWS cases [50].

Figure 19.4 Example of methods based on machine learning to detect consciousness in patients with severe disorders of consciousness. (A) Bioelectrical brain signals are detected with high-density (HD) EEG systems, here with 256 electrodes. (B) Times-series HD-EEG signals are used to determine the connectivity of different brain areas, depicted by different color lines, showing the strength of connections. (C) Results from (B) can be compared to known populations with different disease states. Individual patients are classified based on connectivity/coherence; in this example, the individual patient (red arrow) is classified as MCS with more intensive connectivity than patients in unresponsive wakefulness syndrome (UWS). (D) FDG-PET imaging of a patient in a minimally conscious state (MCS), showing areas of preserved (warm colors) or decreased (cold colors) brain glucose metabolism. Statistics are derived from comparison to known patient cohorts.

397

398

Neurorehabilitation Medicine

Looking at the example of HD-EEG, it becomes evident, why ML/AI methods are necessary to analyze the data. We use an HDEEG system with 256 electrodes to evaluate unresponsive patients. These are an order of magnitude more electrodes than routine clinical EEG and are not fully analyzable by a clinical EEG expert alone. 10 minutes of HD-EEG with 256 Channels at a sampling rate of 1000 Hz amount to more than 150 million data points for a single recording in a single patient. ML/AI approaches though are capable of analyzing such individual big data sets, classifying patients into either UWS or better than UWS, i.e., at least being MCS, based on for example metrics of brain connectivity and coherence. Currently, large international, multicenter, prospective clinical observational trials are underway to establish, whether digital medicine–assisted analyses of such multimodal and multidimensional data in patients with disorders of consciousness (DoC) are suitable to identify patients with covert consciousness, who may eventually have a better prognosis (for example, the PerBrain study, ClinicalTrials.gov Identifier: NCT04798456).

19.4 Digital Medicine as a Long-Term Medical Aid in and after Neurorehabilitation

Despite extensive neurorehabilitation efforts, many patients continue to experience neurological deficits, which lead to disability, reduced participation in life as well as in the impaired QOL. This is especially relevant for patients with initially very severe forms of brain injury or neurologic disease. Digital Medicine can ameliorate the consequences of sustained neurologic impairment in the form of medical aids and equipment during neurorehabilitation as well as in the long-term care setting. Examples of such technologies are eye-tracking-based environmental control and communication systems, robotic walking aids in patients with spinal cord injuries, or brain-computer interfaces (BCI) for communication or for controlling prosthetic devices. Many of these approaches are still in the developmental research phase and are not readily available for routine use, though. Eye movements can be either detected by camera-based eyetracking devices or by direct measurement of the bioelectrical signal

Conclusion

from the moving eye with the use of surface electrodes located on the skin in proximity to the eye. These signals can be used to control a motorized wheelchair for the purpose of mobility, to control a robotic arm to assist in ADL, or to control a virtual keyboard to write and produce speech [1, 7, 9, 43]. BCI systems based on using detecting surface bioelectrical brain activity by means of electrodes for establishing a channel for communication in patients with severe motor impairment are currently the most advanced and established in clinical use. Such patients may suffer from near complete or complete lack of motor control due to brain-stem infarction or neurodegenerative disease, especially motoneuron disease. Using BCI technology to operate spellers and dictionary functions, it is possible to achieve a speed of about 30 characters per minute with high accuracy beyond 90% [15]. BCI is a quickly expanding field for patients with neurologic impairment, but many technologies and approaches are still in the proof-of-concept phase and relatively far away from routine clinical use. Eye-tracking-based environmental control systems for communication or wheelchair control are an exception in this regard because ready-to-use commercial systems are available and are sometimes even reimbursed by health insurance. They can provide severely disabled neurological patients with an increase in independence in the ADL as well as in participation in life. People with paraplegia due to spinal cord injury may achieve a level of independent walking proficiency at home or in the community which allows for functional ambulation with reduced walking speed, by means of a powered exoskeleton [11]. These systems may also even be reimbursed by health insurance in some countries.

19.5 Conclusion

In conclusion, digital medicine has already a great impact on the field of neurorehabilitation. Methods of VR rehabilitation, telerehabilitation, and robot-assisted therapy and their combinations have already become part of the arsenal of routine neurorehabilitative care, and it is expected that many more applications and techniques will enter standard care in the near future. Scientific research accumulates

399

400

Neurorehabilitation Medicine

growing evidence that such methods are effective and efficient in addition to standard neurorehabilitative care. It remains a challenge to implement these digital medicine approaches into standard neurorehabilitative care and further research is needed, on how this may be best achieved. Methods of machine learning and artificial intelligence have also entered the field of neurorehabilitation, for example, in improving diagnosis and prognosis in unresponsive patients with severe DoC, based on large data sets from neurophysiology or functional neuroimaging. Finally, methods of digital medicine are used to improve independence in ADL, QOL, and participation in neurologic patients following neurorehabilitation in the form of long-term assistive technology, such as communication or mobility devices, based on eye-tracking.

Acknowledgments

The research project “E-BRAiN – Evidenz-based Robot Assistance in Neurorehabilitation” is supported by the European Social Fund (ESF), reference: ESF/14-BM-A55-0001/19-A02, and the Ministry of Education, Science and Culture of Mecklenburg-Vorpommern, Germany. This work was further supported by the BDH Bundesverband Rehabilitation e.V. (charity for neuro-disabilities) by a non-restricted personal grant to TP; the sponsors had no role in the decision to publish or any content of the publication. The research project “PerBrain – A Multimodal Approach to Personalized Tracking of Evolving State-Of-Consciousness in BrainInjured Patients” (ClinicalTrials.gov Identifier: NCT04798456) is supported by the ERA PerMed JTC 2019-101 and in Germany by the Federal Ministry for Education and Research (BMBF; #01KU2003) with a grant to AB.

References

1. Antoniou, E., Bozios, P., Christou, V., Tzimourta, K. D., Kalafatakis, K., Tsipouras M. G., Giannakeas, N., & Tzallas, A. T. (2021). EEG-based eye movement recognition using brain-computer interface and random forests. Sensors (Basel). 21, 2339.

References

2. Aramaki, A. L., Sampaio, R. F., Reis, A. C. S., Cavalcanti, A., & Dutra, F. (2019). Virtual reality in the rehabilitation of patients with stroke: an integrative review. Arq Neuropsiquiatr. 77, 268–278. 3. Arienti, C., Lazzarini, S. G., Pollock, A., & Negrini, S. (2019). Rehabilitation interventions for improving balance following stroke: an overview of systematic reviews. PLoS One. 14, e0219781. 4. Bender, A., Jox, R. J., Grill, E., Straube, A., & Lule, D. (2015). Persistent vegetative state and minimally conscious state: a systematic review and meta-analysis of diagnostic procedures. Dtsch Arztebl Int. 112, 235–242. 5. Burton, A. (2013). Dolphins, dogs, and robot seals for the treatment of neurological disease. Lancet Neurol. 12, 851–852.

6. Choi, H. S., Shin, W. S., & Bang, D. H. (2021). Application of digital practice to improve head movement, visual perception and activities of daily living for subacute stroke patients with unilateral spatial neglect: preliminary results of a single-blinded, randomized controlled trial. Medicine (Baltimore). 100, e24637.

7. Cojocaru, D., Manta, L. F., Pană, C. F., Dragomir, A., Mariniuc, A. M., & Vladu, I. C. (2021). The design of an intelligent robotic wheelchair supporting people with special needs, including for their visual system. Healthcare (Basel). 10(1), 13. 8. Collaborators, G. B. D. S. (2021). Global, regional, and national burden of stroke and its risk factors, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Neurol. 20, 795–820.

9. Dahmani, M., Chowdhury, M. E. H., Khandakar, A., Rahman, T., AlJayyousi, K., Hefny, A., & Kiranyaz, S. (2020). An intelligent and lowcost eye-tracking system for motorized wheelchair control. Sensors (Basel). 20. 10. Domínguez-Téllez, P., Moral-Muñoz, J. A., Salazar, A., Casado-Fernández, E., & Lucena-Antón, D. (2020). Game-based virtual reality interventions to improve upper limb motor function and quality of life after stroke: systematic review and meta-analysis. Games Health J. 9, 1–10. 11. Esquenazi, A., Talaty, M., Packel, A., & Saulino, M. (2012). The ReWalk powered exoskeleton to restore ambulatory function to individuals with thoracic-level motor-complete spinal cord injury. Am J Phys Med Rehabil. 91, 911–921. 12. Feng, H., Li, C., Liu, J., Wang, L., Ma, J., Li, G., Gan, L., Shang, X., & Wu, Z. (2019). Virtual reality rehabilitation versus conventional physical therapy for improving balance and gait in Parkinson’s disease patients: a randomized controlled trial. Med Sci Monit. 25, 4186–4192.

401

402

Neurorehabilitation Medicine

13. Forbrig, P., Bundea, A., Pedersen, A. L., & Platz, T. (2020). Digitalization of training tasks and specification of the behaviour of a social humanoid robot as coach, in 8th IFIP WG 132 International Working Conference, HCSE 2020, Eindhoven, The Netherlands, pp. 45–57. 14. Gauthier, L. V., Nichols-Larsen, D. S., Uswatte, G., Strahl, N., Simeo, M., Proffitt, R., Kelly, K., Crawfis, R., Taub, E., Morris, D., Lowes, L. P., Mark, V., & Borstad, A. (2022). Video game rehabilitation for outpatient stroke (VIGoROUS): a multi-site randomized controlled trial of in-home, selfmanaged, upper-extremity therapy. EClinicalMedicine. 43, 101239.

15. Gembler, F. W., Benda, M., Rezeika, A., Stawicki, P. R., & Volosyak, I. (2020). Asynchronous c-VEP communication tools-efficiency comparison of low-target, multi-target and dictionary-assisted BCI spellers. Sci Rep. 10, 17064. 16. Grechuta, K., Rubio Ballester, B., Espín Munne, R., Usabiaga Bernal, T., Molina Hervás, B., Mohr, B., Pulvermüller, F., San Segundo, R., & Verschure, P. (2019). Augmented dyadic therapy boosts recovery of language function in patients with nonfluent aphasia. Stroke. 50, 1270–1274. 17. Heffernan, A., Abdelmalek, M., & Nunez, D. A. (2021). Virtual and augmented reality in the vestibular rehabilitation of peripheral vestibular disorders: systematic review and meta-analysis. Sci Rep. 11, 17843. 18. Henderson, A., Korner-Bitensky, N., & Levin, M. (2007). Virtual reality in stroke rehabilitation: a systematic review of its effectiveness for upper limb motor recovery. Top Stroke Rehabil. 14, 52–61.

19. Kang, G. E., Frederick, R., Nunley, B., Lavery, L., Dhaher, Y., Najafi, B., & Cogan, S. (2021). The effect of implanted functional electrical stimulation on gait performance in stroke survivors: a systematic review. Sensors (Basel). 21.

20. Karamians, R., Proffitt, R., Kline, D., & Gauthier, L. V. (2020). Effectiveness of virtual reality- and gaming-based interventions for upper extremity rehabilitation poststroke: a meta-analysis. Arch Phys Med Rehabil. 101, 885–896.

21. Kondziella, D., Bender, A., Diserens, K., van Erp, W., Estraneo, A., Formisano, R., Laureys, S., Naccache, L., Ozturk, S., Rohaut, B., Sitt, J. D., Stender, J., Tiainen, M., Rossetti, A. O., Gosseries, O., Chatelle, C., & Ean Panel on Coma, Disorders of Consciousness. (2020). European Academy of Neurology guideline on the diagnosis of coma and other disorders of consciousness. Eur J Neurol. 27, 741–756.

References

22. Krakauer, J. W., Kitago, T., Goldsmith, J., Ahmad, O., Roy, P., Stein, J., Bishop, L., Casey, K., Valladares, B., Harran, M. D., Cortes, J. C., Forrence, A., Xu, J., DeLuzio, S., Held, J. P., Schwarz, A., Steiner, L., Widmer, M., Jordan, K., Ludwig, D., Moore, M., Barbera, M., Vora, I., Stockley, R., Celnik, P., Zeiler, S., Branscheidt, M., Kwakkel, G., & Luft, A. R. (2021). Comparing a novel neuroanimation experience to conventional therapy for highdose intensive upper-limb training in subacute stroke: the SMARTS2 randomized trial. Neurorehabil Neural Repair. 35, 393–405. 23. Laureys, S., Celesia, G. G., Cohadon, F., Lavrijsen, J., Leon-Carrion, J., Sannita, W. G., Sazbon, L., Schmutzhard, E., von Wild, K. R., Zeman, A., & Dolce, G. (2010). Unresponsive wakefulness syndrome: a new name for the vegetative state or apallic syndrome. BMC Med. 8, 68.

24. Laver, K. E., Adey-Wakeling, Z., Crotty, M., Lannin, N. A., George, S., & Sherrington, C. (2020). Telerehabilitation services for stroke. Cochrane Database Syst Rev. 1, Cd010255.

25. Laver, K. E., Lange, B., George, S., Deutsch, J. E., Saposnik, G., & Crotty, M. (2017). Virtual reality for stroke rehabilitation. Cochrane Database Syst Rev. 11, Cd008349. 26. Manuli, A., Maggio, M. G., Latella, D., Cannavò, A., Balletta, T., De Luca, R., Naro, A., & Calabrò, R. S. (2020). Can robotic gait rehabilitation plus Virtual Reality affect cognitive and behavioural outcomes in patients with chronic stroke? A randomized controlled trial involving three different protocols. J Stroke Cerebrovasc Dis. 29, 104994. 27. Mao, H., Li, Y., Tang, L., Chen, Y., Ni, J., Liu, L., & Shan, C. (2020). Effects of mirror neuron system-based training on rehabilitation of stroke patients. Brain and Behav. 10, e01729.

28. Maresca, G., Maggio, M. G., Latella, D., Cannavò, A., De Cola, M. C., Portaro, S., Stagnitti, M. C., Silvestri, G., Torrisi, M., Bramanti, A., De Luca, R., & Calabrò, R. S. (2019). Toward improving poststroke aphasia: a pilot study on the growing use of telerehabilitation for the continuity of care. J Stroke Cerebrovasc Dis. 28, 104303.

29. Marques-Sule, E., Arnal-Gómez, A., Buitrago-Jiménez, G., Suso-Martí, L., Cuenca-Martínez, F., & Espí-López, G. V. (2021). Effectiveness of nintendo Wii and physical therapy in functionality, balance, and daily activities in chronic stroke patients. J Am Med Dir Assoc. 22, 1073– 1080.

30. Mehrholz, J., Thomas, S., Werner, C., Kugler, J., Pohl, M., & Elsner, B. (2017). Electromechanical-assisted training for walking after stroke. Cochrane Database Syst Rev. 5, Cd006185.

403

404

Neurorehabilitation Medicine

31. Mekbib, D. B., Debeli, D. K., Zhang, L., Fang, S., Shao, Y., Yang, W., Han, J., Jiang, H., Zhu, J., Zhao, Z., Cheng, R., Ye, X., Zhang, J., & Xu, D. (2021). A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke. Ann N Y Acad Sci. 1493, 75–89.

32. Michael, J., McEllin, L., & Felber, A. (2020). Prosocial effects of coordination: what, how and why? Acta Psychol (Amst). 207, 103083. 33. Nogueira, N., Parma, J. O., Leao, S., Sales, I. S., Macedo, L. C., Galvao, A., de Oliveira, D. C., Murca, T. M., Fernandes, L. A., Junqueira, C., Lage, G. M., & Ferreira, B. P. (2021). Mirror therapy in upper limb motor recovery and activities of daily living, and its neural correlates in stroke individuals: a systematic review and meta-analysis. Brain Res Bull. 177, 217–238.

34. Ogourtsova, T., Souza Silva, W., Archambault, P. S., & Lamontagne, A. (2017). Virtual reality treatment and assessments for post-stroke unilateral spatial neglect: a systematic literature review. Neuropsychol Rehabil. 27, 409–454.

35. Pan, J., Xie, Q., Qin, P., Chen, Y., He, Y., Huang, H., Wang, F., Ni, X., Cichocki, A., Yu, R., & Li, Y. (2020). Prognosis for patients with cognitive motor dissociation identified by brain-computer interface. Brain. 143, 1177– 1189. 36. Park, Y. S., An, C. S., & Lim, C. G. (2021). Effects of a rehabilitation program using a wearable device on the upper limb function, performance of activities of daily living, and rehabilitation participation in patients with acute stroke. Int J Environ Res Public Health. 18, 5524.

37. Platz, T. (2019). Evidence-based guidelines and clinical pathways in stroke rehabilitation-an international perspective. Front Neurol. 10, 200. 38. Platz, T., Seidel, J., Muller, A., Goldmann, C., & Pedersen, A. L. (2021). THERapy-related InterACTion (THER-I-ACT) in rehabilitation: instrument development and inter-rater reliability. Front Neurol. 12, 716953. 39. Rodríguez-Hernández, M., Criado-Álvarez, J. J., Corregidor-Sánchez, A. I., Martín-Conty, J. L., Mohedano-Moriano, A.. & Polonio-López, B. (2021). Effects of virtual reality-based therapy on quality of life of patients with subacute stroke: a three-month follow-up randomized controlled trial. Int J Environ Res Public Health. 18. 40. Schiff, N. D. (2015). Cognitive motor dissociation following severe brain injuries. JAMA Neurol. 72, 1413–1415. 41. Sevcenko, K., & Lindgren, I. (2022). The effects of virtual reality training in stroke and Parkinson’s disease rehabilitation: a systematic review and a perspective on usability. Eur Rev Aging Phys Act. 19, 4.

References

42. Skjaeret, N., Nawaz, A., Morat, T., Schoene, D., Helbostad, J. L., & Vereijken, B. (2016). Exercise and rehabilitation delivered through exergames in older adults: an integrative review of technologies, safety and efficacy. Int J Med Inform. 85, 1–16.

43. Sunny, M. S. H., Zarif, M. I. I., Rulik, I., Sanjuan, J., Rahman, M. H., Ahamed, S. I., Wang, I., Schultz, K., & Brahmi, B. (2021). Eye-gaze control of a wheelchair mounted 6DOF assistive robot for activities of daily living. J Neuroeng Rehabil. 18, 173. 44. Suso-Martí, L., La Touche, R., Herranz-Gómez, A., Angulo-Díaz-Parreño, S., Paris-Alemany, A., & Cuenca-Martínez, F. (2021). Effectiveness of telerehabilitation in physical therapist practice: an umbrella and mapping review with meta-meta-analysis. Phys Ther. 101.

45. Szecsi, J., Schlick, C., Schiller, M., Pollmann, W., Koenig, N., & Straube, A. (2009). Functional electrical stimulation-assisted cycling of patients with multiple sclerosis: biomechanical and functional outcome – a pilot study, J Rehabil Med. 41, 674–680.

46. Tchero, H., Tabue Teguo, M., Lannuzel, A., & Rusch, E. (2018). Telerehabilitation for stroke survivors: systematic review and metaanalysis. J Med Internet Res. 20, e10867. 47. Tenforde, A. S., Borgstrom, H., Polich, G., Steere, H., Davis, I. S., Cotton, K., O’Donnell, M., & Silver, J. K. (2020). Outpatient physical, occupational, and speech therapy synchronous telemedicine: a survey study of patient satisfaction with virtual visits during the COVID-19 pandemic. Am J Phys Med Rehabil. 99, 977–981. 48. Thielbar, K. O., Triandafilou, K. M., Barry, A. J., Yuan, N., Nishimoto, A., Johnson, J., Stoykov, M. E., Tsoupikova, D., & Kamper, D. G. (2020). Home-based upper extremity stroke therapy using a multiuser virtual reality environment: a randomized trial. Arch Phys Med Rehabil. 101, 196–203. 49. Truijen, S., Abdullahi, A., Bijsterbosch, D., van Zoest, E., Conijn, M., Wang, Y., Struyf, N., & Saeys, W. (2022). Effect of home-based virtual reality training and telerehabilitation on balance in individuals with Parkinson disease, multiple sclerosis, and stroke: a systematic review and meta-analysis. Neurol Sci. 43, 2995–3006.

50. Wang, J., Hu, X., Hu, Z., Sun, Z., Laureys, S., & Di, H. (2020). The misdiagnosis of prolonged disorders of consciousness by a clinical consensus compared with repeated coma-recovery scale-revised assessment. BMC Neurol. 20, 343.

405

406

Neurorehabilitation Medicine

51. Wang, L., Zheng, Y., Dang, Y., Teng, M., Zhang, X., Cheng, Y., Zhang, X., Yu, Q., Yin, A., & Lu, X. (2021). Effects of robot-assisted training on balance function in patients with stroke: a systematic review and meta-analysis. J Rehabil Med. 53, jrm00174.

52. Wright, A., Stone, K., Martinelli, L., Fryer, S., Smith, G., Lambrick, D., Stoner, L., Jobson, S., & Faulkner, J. (2021). Effect of combined home-based, overground robotic-assisted gait training and usual physiotherapy on clinical functional outcomes in people with chronic stroke: a randomized controlled trial. Clin Rehabil. 35, 882–893. 53. Wu, J., Cheng, H., Zhang, J., Yang, S., & Cai, S. (2021). Robot-assisted therapy for upper extremity motor impairment after stroke: a systematic review and meta-analysis. Phys Ther. 101.

54. Xie, M., Zhou, K., Patro, N., Chan, T., Levin, M., Gupta, M. K., & Archibald, J. (2021). Virtual reality for vestibular rehabilitation: a systematic review. Otol Neurotol. 42, 967–977.

55. Zhang, B., Li, D., Liu, Y., Wang, J., & Xiao, Q. (2021). Virtual reality for limb motor function, balance, gait, cognition and daily function of stroke patients: a systematic review and meta-analysis. J Adv Nurs. 77, 3255–3273.

Chapter 20

Digital Psychiatry

Sophie-Kathrin Kirchner,a,b Anna Hirschbeck,a,b and Irina Papazovaa,b

aPsychiatry and Psychotherapy, Faculty of Medicine, University of Augsburg, Augsburg, Germany bAugsburg District Hospital, Augsburg, Germany [email protected]

20.1 Entering a New World of Mental Health Care Revolutionary innovations are based on two things: new technology and a fresh mindset [52]. Technological innovations can unleash a cascade of new developments. In 1953, Watson and Crick published an article describing the structure of deoxyribose nucleic acid (DNA) [76]. Based on this discovery, new technologies evolved such as the polymerase chain reaction (PCR), DNA sequencing, cloning and genome editing methods like CRISPR/Cas. All these technologies lead to a tremendous advance in cancer research and culminated in the success of developing a vaccine against SARS-CoV-2 in 2020 that could eventually break the spell of the COVID-19 pandemic [16]. Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

408

Digital Psychiatry

Hopes were high that the discoveries and innovations in genetics would also revolutionize the field of psychiatry. Yet, innovations translating into actually applied diagnostic and treatments failed. In 2015, Tom Insel, the former director of the National Institute of Mental Health (NIMH) and known for its research on the genetics of schizophrenia, turned his back on the NIMH and joined the Life Science division of Google X (now Verily Life Sciences). Nowadays, artificial intelligence (AI) and machine-learning (ML) seem to be the gold-digging field of revolutionary progress in mental health and neuroscience. And hopes are high again. In 2019, Elon Musk disclosed his goal to create a brain-machine interface with the ability to achieve a “full symbiosis with artificial intelligence.” Companies and universities, such as Kernel, International Business Machines Corporation (IBM), John Hopkins University, Massachusetts Institute of Technology (MIT), Meta (former Facebook) and BrainGate, are investing money and manpower in order to develop brain-machine interfaces. Almost all big tech companies focus on natural language processing (NLP) in order to thoroughly understand human speech, analyze linguistic patterns and decode human minds. All these advances can be used for mental healthcare and will change the way of diagnosis and treatment for patients suffering from mental disorders. From today’s patient perspective, however, these developments still seem light years away. But are they really? Or are we already at the verge of entering a new world of mental healthcare? Our everyday life is already very permeated and interwoven with modern technology. During the Covid-19 pandemic, social distancing became mandatory and had to be bridged by modern technology; and hence the everyday application of digitalization accelerated. And so did digital mental healthcare. Patients and their therapists were not allowed to see each other in person, so videobased therapy became a common tool to continue therapy. More people and especially children suffered from mental distress and started to use mindfulness and psychological support smartphone apps to ease their discomfort [64]. Since the wait for psychiatric treatment can last weeks to months, health insurers, who had to manage the limited resources of direct patient care, discovered the added value of digital applications and promoted “apps on

Diagnosis and Prevention, Prognosis, and Treatment Selection

prescription” [25]. Nowadays, a variety of internet and mobilebased applications exist. They are usually based on the principal of behavioral psychotherapy and can bridge the gap between the need for treatment and the capacity to deliver. ML based algorithms can collect and comprehensively process the information from imaging, genetic analysis, and phenotyping in a way that can help the therapist to diagnose, predict the course of a disease, and thus select the right treatment option. In this chapter, we would like to introduce the principals of digital diagnosis and prevention and new paradigms of digital treatment of mental disorders.

20.2 Diagnosis and Prevention, Prognosis, and Treatment Selection

The quest to objectify the process of diagnosis, prognosis and treatment selection have been challenged by the individual nature of mental illness. Psychiatric disorders are the result of a complex interplay of genetic, environmental, behavioral, biological and psychological factors and thus heterogeneous in their origin and manifestation [69]. Classical statistical paradigms are insufficient to address this heterogeneity, since there are often limited to group level analysis [10, 18]. ML, in contrast, can process and integrate big multidimensional data and make multi-outcome predictions on a single patient level, and thus enable a shift from trial-anderror mental health care [56] towards precision psychiatry [10]. In research studies, the use of ML has been applied for the search of biomarkers, diagnosis, prognosis and outcome prediction [18]. Supervised ML algorithms are most frequently used in the field of psychiatry [28]. However, unsupervised ML [78] as well as deep learning algorithms are gaining more popularity [67]. NLP has also a very promising application in psychiatry, since it is capable of detecting emotions [11], suicide intention [15] or mood changes [44] upon social media posts. In Table 20.1, you can find an overview of the most used AI approaches in mental health. We would like to stress that the listed applications are just examples, and that no AI technique is exclusive to just one problem.

409

410

Digital Psychiatry

Table 20.1 Definition of frequently used AI techniques and examples of their applications in psychiatry (based on [18, 28, 34, 60]) Examples of Application Diagnosis, prognosis, outcome prediction, treatment selection, treatment support, treatment (reinforcement ML) Diagnostic and Algorithms that learn Supervised search for biomarker: on labeled data and machine classifying study then predict the label of learning unlabeled data (e.g., binary/ participants as psychiatric patients (e.g., multiclass classification) schizophrenia) or as healthy controls upon neuroimaging data (e.g., Ref. [4]) Identifying biomarkers Semi-supervised Algorithms that learn on both labeled and unlabeled in patients with machine uncertain diagnosis data learning (e.g., mild cognitive impairment [23]) Improving of diagnostic Algorithms that learn on Unsupervised criteria: identifying unlabeled data and try to machine disorder subtypes identify the underlying learning according to symptom structure (e.g., dimension similarity reduction) AI Technique Machine learning

Deep learning

Description Algorithms that are capable of learning, i.e., of improving and adjusting using previous experiences

ML algorithms that learn on raw data upon a multilayer structure based in artificial neuronal networks to discover the hidden structure of the data Natural language AI techniques focused processing on processing and understanding of human language

Outcome prediction: e.g., upon electronic healthcare records (EHRs) [61]

Suicide prevention: e.g., detecting suicide intention upon social media posts [15]

Diagnosis and Prevention, Prognosis, and Treatment Selection

The digital progress provides us also with new sources of data. For instance, situational and physiological factors of a psychiatric episode (e.g., panic attack) could be assessed via the sensors of smartphones and smartwatches (“wearables”). Moreover, patients can describe their emotions, thoughts and behavior during or immediately after the psychiatric episode via smartphone apps (ecological momentary assessments, EMAs [62]). The usage of EMAs would ease, contextualize and objectify patients’ symptom reports which are often retrospective [62], and could thus facilitate the diagnostic process. Simultaneously, clinicians’ notes and documentation are also being digitalized by the implementation of electronic health care records (EHRs). EHRs could not only improve the quality of care [68], but also provides researchers with a big naturalistic data base for the detection of socio-demographic, psychopathological markers that could improve diagnosis, predict treatment outcome and prevent suicide. Indeed, research showed that application of NLP to unstructured clinical notes and correspondents with patients could predict suicide ideation and attempts with precision above 80% [22]. The applications of digital technologies in psychiatry are numerous and include various data domains. Below, we would illustrate some of the possibilities in clinical practice upon an example case (see Box 20.1). We refer to review articles for a more thorough examination of applications of AI and digital technologies for diagnosis, prevention, prognosis, and outcome prediction [e.g., 4, 12, 18, 28]. We stress that the presented applications are based on research results and are just possibilities. To the best of our knowledge, however, none of these applications are thoroughly investigated for clinical work and part of clinical practice guidelines. Box 20.1 In the doctor’s office: Ms. M

Ms. M (age 21) is a last year college student who lives with friends. In the last two months, she has been feeling increasingly sad and lethargic: she has been having difficulties to get out of bed and to visit lectures or spend time with friends. The symptoms started with the break-up with her boyfriend, who she describes as a “cheater” and a “liar.” She has currently no contact with him, but spends 1 h/ day scrolling through his social media in the search of “hints” of his cheating. Her mood worsened with the start of her final exams

411

412

Digital Psychiatry

5 weeks ago. She spent the majority of time in the library learning and stopped doing her regular exercise (jogging). In the last week, Ms. M has been having trouble sleeping: she lies and bed and ruminates about “fixing her life.” She also reports about mental illness in her family, her mother suffers from bipolar disorder since the age of 24. In her manic states, her mother experienced psychotic symptoms (hearing voices, paranoia).

20.2.1 Diagnosis and Prevention First, Ms. M’s physician would assign a diagnosis that best describes her complaints. Currently, this process relies mostly on patients’ verbal reports and the assessment of the clinician. In the case of Ms. M, her sadness, loss of interest in jogging, rumination and sleeping problems are consistent with the diagnosis of depression. However, given her family history, the beginning of a bipolar or a psychotic disorder could not be ruled out, since both disorders have a strong genetic component and are often proceeded by a depressive episode [48]. Currently, the doctor would diagnose Ms. M with depression and observe the possible development of manic and/or psychotic symptoms. In the future, this process might become accelerated by ML technologies. This would be especially relevant for mental illnesses such as bipolar disorder, where misdiagnosis rates are up to 70% [30]. Redlich [50] showed that application of ML classification to brain images to differentiate between depression and bipolar disorder reduces the misdiagnosis rate to 31%. Similar results were achieved upon a low-cost self-report online questionnaire [70]. Moreover, a review of AI algorithms used to investigate neuroimaging in schizophrenia, Alzheimer’s disease, mild cognitive impairment, autism, and attention-deficit hyperactivity disorder showed the advantages of single subject prediction [4]. Regarding psychotic disorders, a large body of research uses ML to detect patterns of brain structure and function specific to schizophrenia. For instance, the PRONIA research group (https://www.pronia.eu/) has focused on classification and prognosis of outbreak of psychotic disorder in individuals at high clinical risk. Recently, they demonstrated that combination of clinical assessment and ML could achieve the most accurate prediction for the transition to psychosis in individuals at high risk [54].

Diagnosis and Prevention, Prognosis, and Treatment Selection

In means of public health, the diagnosis and prevention of psychiatric disorders could also be improved by implementing AI to social media data [60]. For instance, supervised ML of over 40,000 Instagram photos of 166 participants could identify predictive markers for depression [51]. Moreover, compared to unassisted general practitioners [46], the algorithm could detect depression in patients with a higher accuracy [51]. NLP to social media could be also used to detect and prevent suicide intent [15]. Indeed, this application is not hypothetical anymore since social media providers such as Facebook [27] announced they would screen their users’ posts aiming suicide prevention. Despite the promising results, there are a lot of methodological and ethical issues to be addressed before these techniques could be implemented in clinical praxis (e.g., [18]). In our example, Ms. M would be diagnosed according to the current criteria and clinical practices with a depressive episode.

20.2.2 Prognosis, Treatment Selection, and Outcome Prediction

After diagnosing Ms. M.’s disorder, her physician would inform her about the disorder and its possible course and treatment options (psychoeducation). In the field of psychiatry, prognosis is mostly based on group statistics and quite inaccurate at the first onset of the disorder [18]. A more personalized prognosis would be crucial for treatment selection and would support patients in their clinical decisions. AI models, with their ability to simultaneously analyze complex multimodal data, have the potential to improve prognosis estimates. For instance, Koutsouleris et al. [39] predicted functional outcomes in schizophrenia patients based on psychosocial variables with accuracy of over 70%. Application of ML have been shown to improve long-term prognosis also in depression, post-traumatic stress disorder and Alzheimer’s disease [60]. After the psychoeducation, Ms. M would receive a treatment, which in the case of depression normally consists of medication combined with psychotherapy. Unfortunately, 30–50% of patients do not respond to antidepressants initially and further treatment steps are needed until the patients achieve remission [56]. This trialand-error approach can be tiring and also reduce patients’ treatment adherence. So currently, Ms. M. would be prescribed a medication

413

414

Digital Psychiatry

with the chance of 50%–70% to be effective at the first trial. Her doctor would not be really able to know, which antidepressant would work immediately, so he would just guess. AI technologies could improve this process by using genetic, neurobiological, EHR, social media or cognitive testing data to predict how good a patient would respond to different medications, psychotherapies or brain stimulation techniques to treat depression, psychosis or anxiety [12]. For instance, Chekroud et al. [14] were able to define 25 factors that could predict the remission after the antidepressant citalopram with accuracy above 60%, i.e., 30% improvement than the base rate. Further research to validate and expand these findings was conducted [13, 14]. Most importantly, the research results were translated in the practice with the web service SpringHealth (https://springhealth.com/), where patients and primary health care providers are assisted in their clinical decision. In the field of psychotherapy, there is also growing research on predicting which treatments or which key features would lead to the best possible outcome. Despite some evidence of promising algorithms, most models proved to be unfeasible [1, 43]. Noticeably, this research area is in its early stages and studies with larger samples are still much needed [12]. In our case, Ms. M. would be treated according to current clinical guidelines with a combination of an antidepressant and cognitive behavioral therapy.

20.3 Digital Treatment Options for Mental Health

Not only diagnosis and prevention but also psychological treatment of mental illness has undergone a radical transformation in recent years. This might be driven primarily by the universal availability and almost daily use of digital technologies for a variety of functions and activities [21]. Telemedical applications, e.g., internet programs helped to maintain the psychiatric care during difficult times of the COVID-19 pandemic [45] and are also establishing themselves as an important pillar of psychiatric care.

Digital Treatment Options for Mental Health

Box 20.2 In the doctor’s office: Ms. M.

You have already met Ms. M. (21 years old) and learned some important details about her current mental and physical condition. After an extensive medical check, the doctor diagnosed depression and prescribed an antidepressant as well as cognitive behavioral therapy sessions. In addition, Ms. M. wanted to avoid contact due to the current corona situation because she was afraid of getting infected. Consequently, the doctor and therapist were looking for a suitable digital treatment option that could help her in the best possible way.

In order to choose good-quality treatment options for Ms. M., we would like to address a few critical points to consider when choosing good-quality applications. The following listed applications are just examples hereby and none of these therapy options is exclusive to just one problem.

20.3.1 Psychotherapy at a Distance (Text-Based and Video-Based Treatment)

Ms. M. could get offered psychotherapy at a distance. Text-based therapy via e-mail offers possible methods to teach therapeutic contents which are individually adapted to each patient. The resulting communication takes place at a spatial distance and with a time delay. In contrast to the standardized and structured approaches described above, the specific design (e.g., text length, frequency of e-mails) is usually left to the therapist [9]. Manuals for internet-based applications for depressive patients can provide guidance [75]. Chat therapy is also text-based and takes place at distance. In contrast to e-mail therapies they do not take place with a time delay, but in real-time [29]. Weekly sessions often take 50 to 55 minutes [37]. However, the information exchange is more difficult because the message transmission is time-delayed. This means, therapist and patient have to wait until the new message has arrived. Thus, this may result in a broken flow of conversation [9] and may have a demotivating effect on the therapy. Therapy via videoconference which – in contrast to e-mail and chat therapy – enables real-time verbal communication and is more

415

416

Digital Psychiatry

similar to traditional face-to-face treatment [7]. The communication is synchronous and in real time. Videoconferences were hardly established in standard care before 2020. However, since the beginning of the COVID-19 pandemic, we have noticed that video conferences have made major contribution to maintaining care of mental illness [45]. Therapists were found to be flexible and open towards this treatment method. The evaluation of the effectiveness, advantages and disadvantages varied highly, but the importance of enabling video treatment in the current situation has been confirmed [63].

20.3.2 Internet- and Mobile-Based Psychological Interventions

In order to be able to offer suitable treatment options despite the current situation, the therapist suggests further digital Internet- and mobile-based interventions to Ms. M. Internet- and mobile-based interventions offer a wide range of possible forms of treatment and thus offer great potential to support the mental healthcare system. They are used in information transfer, from prevention and self-help through treatment and aftercare [38]. Many studies verify the effectiveness of these interventions and for several mental illness [36, 40, 53]. As a result, they are already part of regular clinical care in some European countries such as the Netherlands, Great Britain and Sweden [8].

20.3.2.1 Types

Due to this very dynamic topic, we can only provide a brief insight into potential Internet- and mobile-based interventions. Current applications are often based on the methods of cognitive behavioral therapy. Due to high self-management components and with the help of psychoeducational elements, they provide information about mental illnesses and can promote the empowerment and selfdetermination of affected people [42]. The most common form of Internet-based interventions are self-management programs. These are either worked through independently or in a guided form. “Guided” means supervised or based on regular feedback by treating professionals. The extent of guidance may vary: In some interventions, mainly the usage of

Digital Treatment Options for Mental Health

the program is motivated. In other interventions, the processing of therapeutic content (e.g., feedback on exposure protocols or specific exercises to reduce avoidance behaviors for anxiety) is targeted. Some self-management interventions can also be used independently and without professional guidance. These programs are called “unguided” Internet-based interventions. A third option are self-management interventions with a combination of direct face-to-face contact with the therapist. They are called “mixed” interventions or combined treatment [19]. Scientists from the Australian National University (Australia) developed the online intervention moodgym (https://moodgym. de) over 15 years ago. Now, moodgym has over a million registered users worldwide since its launch in 2001. Moodgym was developed to prevent and relieve mild to moderately severe depressive symptoms. The program is based on cognitive behavioral therapy techniques and is divided into five building blocks: 1. Feelings 2. Thoughts 3. Developing alternative thoughts 4. Dealing with stress 5. Relationships [77]. Moodgym has already been scientifically tested in a randomized and controlled study with over 600 participants with mild and moderate depression. Significant improvements in depressive symptoms were observed in the intervention group [42]. The Moderated Online Social Therapy (MOST+) platform (https://most.org.au/) is another web-based therapy system for help-seeking young people experiencing mental ill-health. MOST+ merges 1. interactive web-therapy 2. peer-to-peer web-based social networking 3. peer moderator support, 4. clinical moderation and 5. on demand web chat with registered clinicians [3]. In addition to Internet-delivered and computerized treatments, smartphones have also quickly become a driving force of digital health. They are compact and wireless. But most importantly, their sensors allow new data capture (passive and active) and graphical power for retrieval of individualized interventions [71]. Smartphones are no longer just used for making calls or texting. Primarily using their own devices, people use social media – frequently accessed via smartphone apps – for global networking, information exchange and application of health resources [71]. While many relate to the popular conception that screen time and social media use is detrimental to mental health, recent literature shows a more nuanced picture [17, 33]. For example, during the COVID-19

417

418

Digital Psychiatry

pandemic, social media have been a source of social support for many who have been socially isolated and lonely. But furthermore, social media can also be used as a therapeutic tool. For example, the PRIME app is designed to improve mood and motivation in young people with schizophrenia through the promotion of functional recovery and the mitigation of negative symptoms [58]. Through motivational coaching, PRIME delivers text (SMS-based) messages from trained therapists, contains individualized goal setting and enables social networking via direct peer-to-peer messaging to capture and reinforce rewarding experiences and goal achievements [57]. A further self-management and popular smartphone app called Calm was developed to deliver mindfulness meditation in order to reduce stress and improve mindfulness and self-compassion. Huberty et al. (2019) aimed to test the initial efficacy and sustained effects of Calm on stress, mindfulness and self-compassion for 8 weeks in college students with elevated stress levels. Significant differences in all outcomes were observed in the intervention group. Further, the majority of students reported that Calm was helpful to reduce stress and stated they would use the app in the future [32]. Box 20.3 In the doctor’s office: Ms. M

Based on the given diagnosis and information, we suggest Ms. M the web-based online program moodgym to treat her depressive symptoms. In order to improve her sleep patterns and to enhance her mindfulness as well as her self-satisfaction, we recommend the smartphone app Calm.

20.3.2.2 Evidence Effective Internet- and mobile-based interventions have now been scientifically reviewed and evaluated for a large number of mental illnesses. So there is substantial evidence especially for the treatment of anxiety disorders, depression, insomnia, post-traumatic stress disorder and obsessive-compulsive disorder [65]. Especially for depression, the results of a current meta-analysis show that both “guided” and “unguided” online interventions can improve the symptoms. In the short term, users benefited more from a guided intervention, i.e., with included professional feedback. In the

Digital Treatment Options for Mental Health

longer term, however, the effects of guided and unguided intervention were similar [35]. Most of these interventions are targeting young and older adults (approx. 18–65 years) [65]. With regards to existing programs, the results of a systematic review present that user acceptance of online interventions for the treatment of depressive symptoms is predominantly high to very high [55]. Although mental health apps are flooding the consumer market, yet very few studies have examined their effectiveness [41], and many apps do not follow evidence-based guidelines and principles [49].

20.3.2.3 Quality assurance

The range of available digital health application is enormous and confusing. In recent years, there has been a surge in demand especially in the use of smartphone apps to treat and support mental health disorders. This increase has been further intensified since the COVID-19 pandemic [72]. Several reports indicate that nearly 320,000 health apps [2] are available of which more than 10,000 are mental health apps [73]. As we already mentioned, most of these health apps are not evidence based and have not been validated in clinical trials, underlining concerns about the effectiveness and safety of publicly available application. This is worrying, as people looking for a digital support service usually search on social networks and without the aid of a professional [59]. Considering these points, it is also very difficult for psychological and psychiatric professionals (e.g., medical doctors, psychologists, therapists) to select a suitable digital health application from the high range of Internet- and mobile-based online interventions. Meanwhile, portals have been developed to provide clear and scientific information. One example is the “Mobile Health App Database” (MHAD: http://www.mhad.science), containing over 1,000 apps which have been evaluated by now [66]. In Germany, a task force of the DGPPN (Deutsche Gesellschaft für Psychiatrie und Psychotherapie, Psychosomatik und Nervelheilkunde e.V.) and the DGP (Deutsche Gesellschaft für Psychologie) defined key points for the quality assessment of suitable interventions. Theses quality criteria include 1. indication 2. description of the intervention 3. qualification 4. effectiveness 5. patient safety 6. data protection and data security 7. costs 8. integration into care [38].

419

420

Digital Psychiatry

20.4 Limitations and Challenges for the New World of Mental Health 20.4.1 Efficacy vs. Effectiveness As pointed out, we are facing a heterogeneous landscape of lifestyle apps, web-based wellbeing programs and prescriptible evidencebased digital treatment options [71]. There is an increasing emphasis on reviewing digital health applications for their user feasibility and efficacy. Efficacy of Internet and mobile-based apps is increasingly tested in randomized controlled trials (RCT). Yet, it is questionable if digital interventions can be reviewed by RCTs originally developed for pharmaceutical trials, as software is constantly changing and improving in order to meet adoption and engagement [31]. Perhaps a new study design is needed that can monitor efficacy continuously during use. Furthermore, efficacy does not automatically translate into effectiveness [31]. Effectiveness proves that healthcare applications are not only successful under research conditions but can also prevail in patients’ lives and can be implemented in the existing healthcare systems. These real-world implementation efforts often fail because of the lack of knowledge of the application and the lack of use. So we still have to deal with a large research to practice gap starting with failures at many points, including design, research methods, and implementation approaches [47]. So, the next step will be, to select the most promising digital healthcare tools for diagnosis and treatment and from there on, we have to accompany these elaborately developed digital healthcare interventions from the research area into the actual application and the actual benefit for patients under real-world conditions.

20.4.2 Data Safety and Legal Concerns

In recent years, we have repeatedly observed that technological progress occurs at a different pace than its legal regulation. We could also observe that legal regulation depends very much on the country in question. In the US, the Health Insurance Portability and Accountability Act of 1996 (HIPAA) is supposed to protect healthcare-related data from fraud and theft [6]. But many digital healthcare innovations do not

Limitations and Challenges for the New World of Mental Health

fall under the HIPAA regulation, as HIPAA only applies to ‘covered entities’ (which are healthcare providers, healthcare clearinghouses, or health plans), ‘business associates’, or subcontractors of business associates [26]. HIPAA does not apply when personal data in shared in an app in a different context. The Federal Drug Administration (FDA) has been granted the authority of medical devices including digital healthcare interventions. But the FDA declined to regulate the large portion of arguably healthcare-related apps that fall into a ‘general wellness’ category [5, 20]. So, many applications do neither protect healthcare information nor are they federally approved. In Europe, data security has become a particular concern of the European Union (EU) and has led to the comprehensive General Data Protection Regulation (GDPR) [81]. Digital healthcare interventions must furthermore comply with the Medical Devices Act (MDA) before they are considered safe and prescriptible [80]. Data protection is essential in digital mental healthcare applications, as highly sensitive data is collected and exchanged [5]. By mental healthcare applications, people in crisis are encouraged to share very private and intimate information. These people are particularly vulnerable in their situation of crisis, which is why a particularly high level of protection is needed. State legislatures are aware of this responsibility. But these regulations have not been homogenized yet and differ tremendously between countries. Institutions that develop mental health care tools for diagnosis and treatment are not transparently controlled, nor can they advance their innovations in a simple and low-impact way. It will be an important task in the future to ensure that digital healthcare applications are simple yet safe for direct patient use. In addition, therapists and patients need to be trained in the new forms of diagnosis and treatment and made aware of their potential problems. Similar to the standards of pharmacotherapy, the therapists must have a profound knowledge of the prescribed product. Therapists must be able to inform and educate about the efficacy but also the side effects of the product. To ensure this, the developers of the innovations should work more closely with the therapists and impart this knowledge. In addition, it is necessary to integrate knowledge of new diagnostic and therapeutic options into the curricula of medical and psychological training. Based on this, it must be legally regulated who is held responsible when treatment errors occur.

421

422

Digital Psychiatry

20.4.3 Ethical Challenges In a traditional treatment setting, the patient reports his or her symptoms to the therapist. Based on this information, the therapist can make a diagnosis and select an appropriate treatment option. The relationship between patient and therapist in this context is always reciprocal [24]. Every question is itself an intervention. When a therapist asks if a patient is thinking about suicide, it is not only a question, but also a sign of acknowledging the worries and offering help. The use of digital innovations in mental healthcare will function differently and therefore requires special caution. People answer questions differently when questioned by a human counterpart or a machine [74]. It starts with the collection of information. Some dare to reveal more about themselves in the quasi-anonymous space of the digital environment; others are more reluctant to be questioned by a machine. It is essential to bear this difference in mind when recording the symptoms and deducing a diagnosis. Complicating matters further, digital phenotyping captures not only active data but also passive data. This results in the collection of thousands of data endpoints per patient and a highly complex set of information. Currently, we are still in the process of understanding how to interpret such complex information. ML can help us to interpret some of the information but only based on the already existing training data sets. By collecting passive data, we intrude for the first time into a very private territory of the patient, into his/her home and his/her normality. Until now, the sovereignty of sharing information was subject to the patient. This is now being changed, and it is the responsibility of the therapist to deal with it sensibly. In the ‘psychiatrization of everyday life’ lies the risk of dissolving the boundaries between healthy and sick [24]. There is a danger that a human crisis will be misinterpreted as a disorder; and vice versa that disorder dismantled in high-functioning behavior will be overseen. From an ethical point of view, it is necessary that we see the subjective burden of the patient as defining the disorder and that the sheer symptomatology is not enough to distinguish between mentally sick and healthy. It is important to maintain this paradigm and diagnose disorders in a patient-centered manner rather than based on deviant behaviors that are detectable under new

References

circumstances. It is essential to maintain our patients’ boundaries. Non-sharing of information must be accepted and receiving intimate information requires the utmost sensitivity. Furthermore, we must ensure that digital healthcare applications are available to patients without discrimination and exclusion. Today, digital exclusion affects poorer, older and less educated people. Algorithms and apps do not always take sufficient account of different ethnic groups and may underrepresent minorities [79]. Hence, healthcare systems diversifying their treatment options by digital means, have to make the according changes to deliver everyone fair and equal treatment options.

20.5 Conclusion

Technology is on the verge of a new digital diagnostics and treatment. This will change mental healthcare fundamentally. It is yet too early to make a final statement about the digitalization of psychiatry. As technology advances, there will great opportunities to offer more and better support to patients suffering from mental problems. Through digitalization, we will be able to increase the capacity of the help system and reach more and distant people. Yet the field of digital mental health care is still very disparate and has many pitfalls. In the years coming, it will be our task to select the best diagnostic and treatment options and perfect them to a safe and effective level for the benefit of our patients.

References

1. Aafjes-van Doorn, K., Kamsteeg, C., Bate J., Aafjes, M. (2021). A scoping review of machine learning in psychotherapy research. Psychother Res. 31:92–116. https://doi.org/10.1080/10503307.2020.1808729.

2. Aitken, M., Clancy, B., Nass, D. (2017). The Growing Value of Digital Health: Evidence and Impact on Human Health and the Healthcare System. Report, US, IQVIA Institute for Human Data Science. 3. Alvarez-Jimenez, M., Rice, S., D’Alfonso, S., et al. (2020). A novel multimodal digital service (Moderated Online Social Therapy+) for help-seeking young people experiencing mental ill-health: pilot evaluation within a national youth e-mental health service. J Med Internet Res. 22:e17155. https://doi.org/10.2196/17155.

423

424

Digital Psychiatry

4. Arbabshirani, M. R, Plis, S., Sui, J., Calhoun, V. D. (2017). Single subject prediction of brain disorders in neuroimaging: promises and pitfalls. Neuroimage. 145:137–165. https://doi.org/10.1016/j. neuroimage.2016.02.079. 5. Armontrout, J., Torous, J., Fisher, M., et al. (2016). Mobile mental health: navigating new rules and regulations for digital tools. Curr Psychiatry Rep. 18:91. https://doi.org/10.1007/s11920-016-0726-x. 6. Atchinson, B., Fox, D. (1997). The politics of the Health Insurance Portability and Accountability Act. Health Aff. 16:146–150. https:// doi.org/10.1377/HLTHAFF.16.3.146.

7. Barak, A., Klein, B., Proudfoot, J. G. (2009). Defining internet-supported therapeutic interventions. Ann Behav Med. 38:4–17. https://doi. org/10.1007/s12160-009-9130-7. 8. Berger, T. (2013). Internet-based treatments: experiences from Sweden. An interview with Gerhard Andersson. Verhaltenstherapie. 23:211–214. https://doi.org/10.1159/000188079. 9. Berger, T. (2015). Internetbasierte Interventionen bei psychischen Störungen. Hogrefe Verlag. 10. Bzdok, D., Meyer-Lindenberg, A. (2018). Machine learning for precision psychiatry: opportunities and challenges. Biol Psychiatry Cogn Neurosci Neuroimaging. 3:223–230. https://doi.org/10.1016/j. bpsc.2017.11.007.

11. Calvo, R. A., Milne, D. N., Hussain, M. S., Christensen, H. (2017). Natural language processing in mental health applications using nonclinical texts. Nat Lang Eng. 23:649–685. https://doi.org/10.1017/ S1351324916000383. 12. Chekroud, A. M., Bondar, J., Delgadillo, J., et al. (2021). The promise of machine learning in predicting treatment outcomes in psychiatry. World Psychiatry. 20:154–170. https://doi.org/10.1002/wps.20882.

13. Chekroud, A. M., Gueorguieva, R., Krumholz, H. M., et al. (2017). Reevaluating the efficacy and predictability of antidepressant treatments. JAMA Psychiatry. 74:370–378. https://doi.org/10.1001/ jamapsychiatry.2017.0025. 14. Chekroud, A. M., Zotti, R. J., Shehzad, Z. et al. (2016). Cross-trial prediction of treatment outcome in depression: a machine learning approach. Lancet Psychiatry. 3:243–250. https://doi.org/10.1016/ S2215-0366(15)00471-X. 15. Coppersmith, G., Leary, R., Crutchley, P., Fine, A. (2018). Natural language processing of social media as screening for suicide risk.

References

Biomed Inform Insights. 10:1178222618792860. org/10.1177/1178222618792860.

https://doi.

16. Corbett, K. S., Edwards, D. K., Leist, S. R., et al. (2020). SARS-CoV-2 mRNA vaccine design enabled by prototype pathogen preparedness. Nature. 586:567–571. https://doi.org/10.1038/s41586-020-2622-0.

17. Davidson, B., Shaw, H., Ellis, D. A. (2020). Fuzzy constructs: the overlap between mental health and technology ‘Use’. PsyArXiv:10.31234. https://doi.org/10.31234/osf.io/6durk. 18. Dwyer, D. B., Falkai, P., Koutsouleris, N. (2018). Machine learning approaches for clinical psychology and psychiatry. Annu Rev Clin Psychol. 14:91–118.

19. Ebert, D. D., Van Daele, T., Nordgreen, T., et al. (2018). Internet- and mobile-based psychological interventions: applications, efficacy, and potential for improving mental health a report of the EFPA e-health taskforce. Eur Psychol. 23:167–187.

20. Elenko, E., Speier, A., Zohar, D. (2015). A regulatory framework emerges for digital medicine. Nat Biotechnol. 33:697–702. https://doi. org/10.1038/nbt.3284. 21. Fairburn, C. G., Patel, V. (2017). The impact of digital technology on psychological treatments and their dissemination. Behav Res Ther. 88:19–25. https://doi.org/10.1016/j.brat.2016.08.012.

22. Fernandes, A. C., Dutta, R., Velupillai, S., et al. (2018). Identifying suicide ideation and suicidal attempts in a psychiatric clinical research database using natural language processing. Sci Rep. 8:7426. https:// doi.org/10.1038/s41598-018-25773-2.

23. Filipovych, R., Davatzikos, C. (2011). Semi-supervised pattern classification of medical images: application to mild cognitive impairment (MCI). NeuroImage. 55:1109–1119. https://doi. org/10.1016/j.neuroimage.2010.12.066.

24. Fuchs, T. (2021). Digitalized psychiatry : critical considerations on a new paradigm. Nervenarzt. 92:1149–1154. https://doi.org/10.1007/ s00115-021-01188-9.

25. Geier, A. S. (2021). Digitale Gesundheitsanwendungen (DiGA) auf dem Weg zum Erfolg – die Perspektive des Spitzenverbandes Digitale Gesundheitsversorgung. Bundesgesundheitsbl. 64:1228–1231. https://doi.org/10.1007/s00103-021-03419-5. 26. Glenn, T., Monteith, S. (2014). Privacy in the digital world: medical and health data outside of HIPAA protections. Curr Psychiatry Rep. 16:494. https://doi.org/10.1007/s11920-014-0494-4.

425

426

Digital Psychiatry

27. Gomes de Andrade, N. N., Pawson, D., Muriello, D., et al. (2018). Ethics and artificial intelligence: suicide prevention on Facebook. Philos Technol. 31:669–684. https://doi.org/10.1007/s13347-018-0336-0.

28. Graham, S., Depp, C., Lee, E. E., et al. (2019). Artificial intelligence for mental health and mental illnesses: an overview. Curr Psychiatry Rep. 21:116. https://doi.org/10.1007/s11920-019-1094-0. 29. Hintenberger, G. (2009). Der Chat als neues Beratungsmedium, in Handbuch Online-Beratung, Vandenhoeck & Ruprecht GmbH & Co KG, Göttingen, pp. S.69–S.78. 30. Hirschfeld, R. M. A., Lewis, L., Vornik, L. A. (2003). Perceptions and impact of bipolar disorder: how far have we really come? Results of the national depressive and manic-depressive association 2000 survey of individuals with bipolar disorder. J Clin Psychiatry. 64:14089.

31. Hirschtritt, M. E., Insel, T. R. (2018). Digital technologies in psychiatry: present and future. Focus (Am Psychiatr Publ). 16:251–258. https:// doi.org/10.1176/appi.focus.20180001.

32. Huberty, J., Green, J., Glissmann, C., et al. (2019). Efficacy of the mindfulness meditation mobile app ”calm“ to reduce stress among college students: randomized controlled trial. JMIR Mhealth Uhealth. 7:e14273. https://doi.org/10.2196/14273.

33. Jensen, M., George, M., Russell, M., Odgers, C. (2019). Young adolescents’ digital technology use and mental health symptoms: little evidence of longitudinal or daily linkages. Clin Psychol Sci. 7:1416–1433. https:// doi.org/10.1177/2167702619859336. 34. Jordan, M. I., Mitchell, T.M. (2015). Machine learning: trends, perspectives, and prospects. Science. 349:255–260. https://doi. org/10.1126/science.aaa8415.

35. Karyotaki, E., Efthimiou, O., Miguel, C., et al. (2021). Internet-based cognitive behavioral therapy for depression: a systematic review and individual patient data network meta-analysis. JAMA Psychiatry. 78:361–371. https://doi.org/10.1001/jamapsychiatry.2020.4364.

36. Karyotaki, E., Riper, H., Twisk, J., et al. (2017). Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry. 74:351–359. https://doi.org/10.1001/ jamapsychiatry.2017.0044. 37. Kessler, D., Lewis, G., Kaur, S., et al. (2009). Therapist-delivered internet psychotherapy for depression in primary care: a randomised controlled trial. Lancet. 374:628–634. https://doi.org/10.1016/ S0140-6736(09)61257-5.

References

38. Klein, J. P., Knaevelsrud, C., Bohus, M., et al. (2018). Internet-based selfmanagement interventions: quality criteria for their use in prevention and treatment of mental disorders. Nervenarzt. 89:1277–1286. https://doi.org/10.1007/s00115-018-0591-4. 39. Koutsouleris, N., Kahn, R. S., Chekroud, A. M., et al. (2016). Multisite prediction of 4-week and 52-week treatment outcomes in patients with first-episode psychosis: a machine learning approach. Lancet Psychiatry. 3:935–946. https://doi.org/10.1016/S22150366(16)30171-7. 40. Kuester, A., Niemeyer, H., Knaevelsrud, C. (2016). Internet-based interventions for posttraumatic stress: a meta-analysis of randomized controlled trials. Clin Psychol Rev. 43:1–16. https://doi.org/10.1016/j. cpr.2015.11.004.

41. Leigh, S., Flatt, S. (2015). App-based psychological interventions: friend or foe? Evid Based Ment Health. 18:97–99. https://doi.org/10.1136/ eb-2015-102203. 42. Löbner, M., Pabst, A., Stein, J., et al. (2018). Computerized cognitive behavior therapy for patients with mild to moderately severe depression in primary care: a pragmatic cluster randomized controlled trial (@ktiv). J Affect Disord. 238:317–326. https://doi.org/10.1016/j. jad.2018.06.008. 43. Lorenzo-Luaces, L., Peipert, A., De Jesús-Romero, R., et al. (2021). Personalized medicine and cognitive behavioral therapies for depression: small effects, big problems, and bigger data. J Cogn Ther. 14:59–85. https://doi.org/10.1007/s41811-020-00094-3.

44. Low, D. M., Rumker, L., Talkar, T., et al. (2020). Natural language processing reveals vulnerable mental health support groups and heightened health anxiety on reddit during COVID-19: observational study. J Med Internet Res. 22:e22635. https://doi.org/10.2196/22635.

45. Meyer-Lindenberg, A. (2021). Digital life in a networked world: opportunities and risks for psychiatry. Nervenarzt. 92:1130–1139. https://doi.org/10.1007/s00115-021-01203-z. 46. Mitchell, A. J., Vaze, A., Rao, S. (2009). Clinical diagnosis of depression in primary care: a meta-analysis. Lancet. 374:609–619. https://doi. org/10.1016/S0140-6736(09)60879-5.

47. Mohr, D. C., Riper, H., Schueller, S. M. (2018). A solution-focused research approach to achieve an implementable revolution in digital mental health. JAMA Psychiatry. 75:113–114. https://doi.org/10.1001/ jamapsychiatry.2017.3838.

427

428

Digital Psychiatry

48. Murray, R. M., Sham, P., Van Os, J., et al. (2004). A developmental model for similarities and dissimilarities between schizophrenia and bipolar disorder. Schizophr Res. 71:405–416. https://doi.org/10.1016/j. schres.2004.03.002.

49. Nicholas, J., Larsen, M. E., Proudfoot, J., Christensen, H. (2015). Mobile apps for bipolar disorder: a systematic review of features and content quality. J Med Internet Res. 17:e198. https://doi.org/10.2196/ jmir.4581. 50. Redlich, R., Almeida, J. R., Grotegerd, D., et al. (2014). Brain morphometric biomarkers distinguishing unipolar and bipolar depression: a voxel-based morphometry–pattern classification approach. JAMA Psychiatry. 71:1222–1230. https://doi.org/10.1001/ jamapsychiatry.2014.1100. 51. Reece, A. G., Danforth, C. M. (2017). Instagram photos reveal predictive markers of depression. EPJ Data Sci. 6:15. https://doi.org/10.1140/ epjds/s13688-017-0110-z.

52. Ringberg, T., Reihlen, M., Rydén, P. (2019). The technology-mindset interactions: leading to incremental, radical or revolutionary innovations. Ind Mark Manag. 79:102–113. https://doi.org/10.1016/j. indmarman.2018.06.009. 53. Riper, H., Blankers, M., Hadiwijaya, H., et al. (2014). Effectiveness of guided and unguided low-intensity internet interventions for adult alcohol misuse: a meta-analysis. PLoS One. 9:e99912. https://doi. org/10.1371/journal.pone.0099912.

54. Rosen, M., Betz, L. T., Schultze-Lutter, F., et al. (2021). Towards clinical application of prediction models for transition to psychosis: a systematic review and external validation study in the PRONIA sample. Neurosci Biobehav Rev. 125:478–492. https://doi.org/10.1016/j. neubiorev.2021.02.032.

55. Rost, T., Stein, J., Löbner, M., et al. (2017). User acceptance of computerized cognitive behavioral therapy for depression: systematic review. J Med Internet Res. 19:e309. https://doi.org/10.2196/ jmir.7662.

56. Rush, A. J., Trivedi, M. H., Wisniewski, S. R., et al. (2006). Acute and longer-term outcomes in depressed outpatients requiring one or several treatment steps: a STAR*D report. AJP 163:1905–1917. https://doi.org/10.1176/ajp.2006.163.11.1905. 57. Schlosser, D., Campellone, T., Kim, D., et al. (2016). Feasibility of PRIME: a cognitive neuroscience-informed mobile app intervention to enhance motivated behavior and improve quality of life in recent

References

onset schizophrenia. JMIR Res Protoc. 5:e77. https://doi.org/10.2196/ resprot.5450.

58. Schlosser, D. A., Campellone, T. R., Truong, B., et al. (2018). Efficacy of PRIME, a mobile app intervention designed to improve motivation in young people with schizophrenia. Schizophr Bull. 44:1010–1020. https://doi.org/10.1093/schbul/sby078. 59. Schueller, S. M., Neary, M., O’Loughlin, K., Adkins, E. C. (2018). Discovery of and interest in health apps among those with mental health needs: survey and focus group study. J Med Internet Res. 20:e10141. https:// doi.org/10.2196/10141. 60. Shatte, A. B. R., Hutchinson, D. M., Teague, S. J. (2019). Machine learning in mental health: a scoping review of methods and applications. Psychol Med. 49:1426–1448. https://doi.org/10.1017/S0033291719000151.

61. Shickel, B., Tighe, P. J., Bihorac, A., Rashidi, P. (2018). Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 22:1589– 1604. https://doi.org/10.1109/JBHI.2017.2767063.

62. Shiffman, S., Stone, A. A., Hufford, M. R. (2008). Ecological momentary assessment. Annu Rev. Clin Psychol. 4:1–32. https://doi.org/10.1146/ annurev.clinpsy.3.022806.091415. 63. Simpson, S., Richardson, L., Pietrabissa, G., et al. (2021). Videotherapy and therapeutic alliance in the age of COVID-19. Clin Psychol Psychother. 28:409–421. https://doi.org/10.1002/cpp.2521.

64. Sorkin, D. H., Janio, E. A., Eikey, E. V., et al. (2021). Rise in use of digital mental health tools and technologies in the United States during the COVID-19 pandemic: survey study. J Med Internet Res. 23:e26994. https://doi.org/10.2196/26994. 65. Stein, J., Röhr, S., Luck, T., et al. (2018). Indication and evidence of internationally developed online coaches as intervention for mental illness: a meta-review. Psychiatr Prax. 45:7–15. https://doi. org/10.1055/s-0043-117050. 66. Stoyanov, S. R., Hides, L., Kavanagh, D. J., et al. (2015). Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 3:e27. https://doi.org/10.2196/mhealth.3422.

67. Su, C., Xu, Z., Pathak, J., Wang, F. (2020). Deep learning in mental health outcome research: a scoping review. Transl Psychiatry. 10:1–26. https://doi.org/10.1038/s41398-020-0780-3. 68. Takian, A., Sheikh, A., Barber, N. (2012). We are bitter, but we are better off: case study of the implementation of an electronic health record

429

430

Digital Psychiatry

system into a mental health hospital in England. BMC Health Serv Res. 12:484. https://doi.org/10.1186/1472-6963-12-484.

69. Tandon, N., Tandon, R. (2019). Using machine learning to explain the heterogeneity of schizophrenia. Realizing the promise and avoiding the hype. Schizophr Res. 214:70–75. https://doi.org/10.1016/j. schres.2019.08.032.

70. Tomasik, J., Han, S. Y. S., Barton-Owen, G., et al. (2021). A machine learning algorithm to differentiate bipolar disorder from major depressive disorder using an online mental health questionnaire and blood biomarker data. Transl Psychiatry. 11:1–12. https://doi. org/10.1038/s41398-020-01181-x. 71. Torous, J., Bucci, S., Bell, I. H., et al. (2021). The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. 20:318–335. https:// doi.org/10.1002/wps.20883.

72. Torous, J., Jän Myrick, K., Rauseo-Ricupero, N., Firth, J. (2020). Digital mental health and COVID-19: using technology today to accelerate the curve on access and quality tomorrow. JMIR Ment Health. 7:e18848. https://doi.org/10.2196/18848.

73. Torous, J., Roberts, L. W. (2017). Needed innovation in digital health and smartphone applications for mental health: transparency and trust. JAMA Psychiatry. 74:437–438. https://doi.org/10.1001/ jamapsychiatry.2017.0262. 74. Tsvetkova, M., Yasseri, T., Meyer, E. T., et al. (2017). Understanding human-machine networks: a cross-disciplinary survey. ACM Comput Surv. 50:12:1–12:35. https://doi.org/10.1145/3039868. 75. Vernmark, K., Lenndin, J., Bjärehed, J., et al. (2010). Internet administered guided self-help versus individualized e-mail therapy: a randomized trial of two versions of CBT for major depression. Behav Res Ther. 48:368–376. https://doi.org/10.1016/j.brat.2010.01.005.

76. Watson, J. D., Crick, F. H. (1953). Molecular structure of nucleic acids: a structure for deoxyribose nucleic acid. Nature. 171:737–738. https:// doi.org/10.1038/171737a0.

77. Weitzel, E. C., Quittschalle, J., Welzel, F. D., et al. (2021). E-mental health and healthcare apps in Germany. Nervenarzt. 92:1121–1129. https:// doi.org/10.1007/s00115-021-01196-9. 78. Zeng, L.-L., Shen, H., Liu, L., Hu, D. (2014). Unsupervised classification of major depression using functional connectivity MRI. Hum Brain Mapp. 35:1630–1641. https://doi.org/10.1002/hbm.22278.

References

79. Zou, J., Schiebinger, L. (2018). AI can be sexist and racist: it’s time to make it fair. Nature. 559:324–326. https://doi.org/10.1038/d41586018-05707-8. 80. 2017 Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/ EEC (Text with EEA relevance). 81. 2019 European Parliament resolution of 12 February 2019 on the implementation of the Cross-Border Healthcare Directive (2018/2108(INI).

431

Chapter 21

Digital Neurosurgery

Björn Sommer and Ehab Shiban

Neurosurgery, University Hospital Augsburg, Augsburg, Germany [email protected]

The neurosurgical field has been more and more digitalized over the past two decades. As imaging modalities are more readily available, the possibilities seem endless. This chapter should provide the reader with an overview of the history of digitalization in neurosurgery, the main fields of application, its advantages and drawbacks, and a glimpse into future projects.

21.1 Introduction

The first use of digital technologies in neurosurgery regards the field of preoperative imaging. With the development of computer-assisted tomography (CAT) by Sir Godfrey Newbold Hounsfield in 1973 [10], multiple plain X-ray measurements are taken from different angles and are computed to create cross-sectional images of different Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

434

Digital Neurosurgery

tissues. From this point onward, it was possible to non-invasively investigate organ systems three-dimensionally (3D). The methods of electronic data processing and storage as well as the resolution and quality of the imaging scanners improved drastically. Intraoperative acquisitions of images from tumors using ultrasound were first described by the radiologist Michael H. Reid from the University of California, Sacramento, in 1978 [26]. He visualized a cystic astrocytoma of the cervical cord. The ability to resonate protons using a magnetic field was investigated by many researchers. Two- or three-dimensional images of soft tissues could be created due to the variations in water contents and proton relaxation times. Paul C. Lauterbur published the first nuclear magnetic resonance (MR) image using magnetic field gradients and his related theory in 1973 [15]. Regarding the digitalization of imaging, these three techniques can be considered the pioneer events building the foundation of intraoperative real-time imaging. Before this era, anatomical landmarks and low-resolution analog radiographs were used in combination with frame-based constructions to maneuver and localize brain lesions. The inaccuracy led to possible harm to healthy brain structures with devastating effects for the patient. The transition from macro- to microsurgery was one important step in high-precision surgery. From then on, digital development was more and more integrated into clinical practice. Computed images were transferred and virtually projected onto the surface of the patient’s skin surface. The next developmental step was the intraoperative acquisition of CT or MR images, which could be used as a resection control. An update of navigational data with intraoperative data could aid in surgical planning, and give an immediate response to the extent of resection or even shifting of eloquent brain structures and fiber tracts due to the loss of CSF, brain parenchyma, or bleeding. Threedimensional data sets that can be used for the virtual reconstruction of the operating field are not only used in brain surgery but also in complex operations on the spinal column. The transition of these applications to the field of intraoperative robotics and virtual realityaugmented real-time surgery is the high-end application of digital neurosurgery up to date.

Intraoperative Imaging

The following sections highlight the different themes and give an overview of the techniques with examples of clinical application, advantages, and pitfalls as well as cost-effectiveness analysis.

21.2 Intraoperative Imaging

21.2.1 Intraoperative Ultrasound One of the most versatile instruments of imaging is ultrasound. For neurosurgeons, the opportunity to create real-time images of dynamic (e.g., blood flow, CSF flow) or static (e.g., tumor, intracranial hemorrhage) of the CNS is more than welcome. The first applications of intraoperative ultrasound (ioUS) date back to the 1970s and were restricted because of poor resolution and lack of specific training [26]. Technological advancement brought forth powerful engines to convert frequency shifts into dynamic images that were capable to be fused with preoperative imaging. The simultaneous visualization of blood flow, parenchyma, and hemodynamic circulatory parameters are ideal for the assessment of brain micro- and macro-circulation using contrast–enhancing ultrasound (CEUS) in intracerebral vascular malformations or feeding/draining vessels of brain tumors [6, 25].

Figure 21.1 Intraoperative ultrasound merged with MRI and fiber tracts [4].

435

436

Digital Neurosurgery

The combination of intraoperative ultrasound (ioUS) and neuronavigation was first successfully applied in 1993 in 7 patients with low-grade astrocytomas, chronic intracerebral hematoma, and porencephalic cysts [13]. With iopUS, flaws like brain shift can be corrected by the acquisition of intraoperative images, which was very helpful during the resection of extensive tumor masses. In glioma surgery, ioUS is used as a complementary tool to verify the extent of resection and surgical guidance [17]. An example of functional imaging techniques and intraoperative ultrasound is given in Fig. 21.1 [4].

21.2.2 Intraoperative Computed Tomography

After the development of the first CT scanner, it was a natural desire to move the whole system into the OR. Most of the scanners are mounted on rails that are embedded in the OR floor, and the gantry is moved to the patient. Ceiling-mounted systems became available in 2013 (Fig. 21.2). In 1979, the first glioma resection with intraoperative CT (iopCT) resection control was performed [30]. Since those, the image quality, radiation exposure, scanning times, and mass of the gantry improved dramatically. Due to the poorer imaging of tissues, working fields of iopCT included guidance for navigation, the extent of gross-total resection of skull-base tumors, vascular lesions including vessel patency or cerebral perfusion imaging (CT-angiography), or spine surgery [28]. Especially the position control of spinal instrumentation screws is one major and widely used application of intraoperative CT imaging, although the revision rate due to intraoperative detection of navigated screw misplacement ranges around 1–2% [9]. Due to the limited space in the OR, different concepts of mounting the gantry have been developed. At the Department of Neurosurgery, University Hospital of Augsburg, we installed our 64-slice scanner (Somatom Definition AS, Siemens) as a ground-based system with floor-embedded rails in October 2015. So far, over 500 operations with more than 1500 intraoperative CT scans were performed in our institution (Fig. 21.3).

Intraoperative Imaging

Figure 21.2 First ceiling-mounted CT (IMRIS Visius).

Next-generation iopCTs are taking up a minimum of space (1.5 m2) with a wide gantry opening. They are integrated into the OR table (Fig. 21.4).

437

438

Digital Neurosurgery

Figure 21.3 Intraoperative 64-slice CT, Department of Neurosurgery, University Hospital of Augsburg, Germany.

Figure 21.4 Airo, Brainlab, Munich, Germany (https://www.brainlab.com/de/ news/airo/) (Accessed: 6-February-2022).

Intraoperative Imaging

21.2.3 Intraoperative Magnetic Resonance Imaging The first intraoperative MRI system had a “double donut design” and a field strength of 0.5T in Brigham and Women’s Hospital, Harvard Medical School, Boston, U.S.A., and was installed in 1994 [19]. A tworoom concept using a 0.2T scanner was introduced in Erlangen and Heidelberg, Germany in 1996 [7]. The first brain tumor resection with intraoperative MRI resection control was in 1996 at Brigham & Women’s. In Germany, Heidelberg and Erlangen were the first University Hospitals to perform such operations [1, 7]. The first intraoperative 1.5T “high-field” scanner was installed in Erlangen in 2002. Up to date, three centers exist where “next-generation” 3T-iopMRI have been installed in Germany: Munich (Technical University of Munich) February 1st, 2018, Dresden (University Hospital) August 22nd, 2018, and Ulm (Bundeswehrkrankenhaus) June 24th, 2013. The benefits of intraoperative MRI resection control regarding the completeness of resection have been underlined by an RCT [29]. Updating neuronavigational data through intraoperative imaging and correcting the errors without elaborate and time-consuming procedures is one fine advantage of intraoperative imaging, and brain surgery, especially MR imaging. In 1997, the first intraoperative update of navigational data for guidance systems was reported by the Heidelberg group [35]. As 1.5T scanners became commercially available, this technique has been applied in brain tumor surgery, epilepsy surgery, and stereotactic surgery in combination with other imaging techniques such as functional MR imaging, magnetoencephalography, PET imaging, or MR spectroscopy [24, 27, 32]. Especially in epilepsy surgery, where the hypothetical epileptogenic focus can be localized using invasive electrode measurements, advances in digital data processing contribute to surgical accuracy. To identify possible electrode displacement in epilepsy surgery, the first reports demonstrate the benefit of merging intraoperative data to reconstruct 3D images using methods such as the volume rendering technique [31].

439

440

Digital Neurosurgery

Figure 21.5 Intraoperative image of invasive grid electrode recording (left) with 3D reconstruction using the volume rendering technique (right).

21.3 Neuronavigation 21.3.1 Intraoperative Navigation: Brain To a surgeon, the possibility to receive immediate feedback on the exact localization of his working field is essential for successful epilepsy. Even though profound knowledge of anatomical structures and landmarks is mandatory for each operation, the interindividual variability and exceptions could lead to misinterpretation of the actual situation. Thus, several image-guided navigation systems have been developed. One can distinguish between frame-based and frameless devices. The most commonly frame-based system used is the stereotaxy frame of Leksell, which was introduced in 1949. It is based on the “principle of center of the arc,” where three polar coordinates (depth, a.p., and angle) are used to localize the target point [16]. One key neurosurgical procedure is the implantation of deep brain stimulation electrodes in important, deep-seated brain nuclei for the treatment of movement disorders, e.g., Parkinson’s disease. The target point (ncl. subthalamicus, globus pallidus internus) has a volume of about 200–500 mm3, which is reached over a distance of 10 cm from the brain surface and more [34]. Precision and accuracy have to be guaranteed, which can be achieved via a framebased system between 1 to 3 mm [5]. Limitations of frame-based systems are their reliance on pure preoperative data, installation on the patient’s head, its weight, unwieldiness, and missing ability to correct inaccuracies during the operation.

Neuronavigation

Frameless systems, on the other hand, seem to fulfill all the features that frame-based are lacking. The first attempts to implement frameless systems into neuronavigation of the brain were made back in 1993 [27]. The point-pair registration used defined landmarks that were put on the patient’s skin surface, “fiducials” which had to be non-colinear to allow reconstruction of 3D coordinates. These fiducials were acquired via CT or MRI, digitally processed with neuronavigation software, and electronically compared to the real localization point in space when the patient´s head was fixed in the head holder or clamp. Alternatively, the surface contouring can be scanned, and the multiple random points are digitally converted and compared with the radiographic surface. Flaws of frameless navigation systems are the fragility of the reference array, the quality of preoperative imaging as a reference of all digital image merging processes, the occurrence of intraoperative brain shift due to the removal of CSF of brain mass, and the co-registration process with adjustment of anatomical landmarks itself [23, 34]. Nevertheless, frameless neuronavigation with the implementation of functional MRI data has become a standard worldwide, especially in lesions in or adjacent to eloquent brain areas [22].

Figure 21.6 Intraoperative neuronavigation of a patient with postcentral glioblastoma (red) and 3D-tractography of the pyramidal tracts (pink).

441

442

Digital Neurosurgery

21.3.2 Intraoperative Navigation: Spine The spinal column has always been considered a challenge for orthopedic surgeons, trauma surgeons, and neurological surgeons alike. Compared to the fixation of traumatic fractures of the extremities, the proximity to the spinal cord, its nerve radices, and associated blood vessels are even more challenging in terms of reconstruction and preservation of function. After the invention of X-ray fluoroscopy, the placement of screws, wires, and plates for osteosynthesis improved dramatically in spine surgery regarding precision and complication rate. However, especially in surgery of the cervical and thoracic spine, deformities, or revision surgery, intraoperative X-ray quality was not sufficient enough for safe placement of the screw-based fixation alone. Besides, frequent X-ray imaging led to a high dosage of radiation exposure for both the patient and the surgical team.

Figure 21.7 Spinal navigation-guided insertion of a pedicle screw in Th12 with a 3D image of the navigation clamp on the spinal process of vertebra L4.

Based on the experience with other intraoperative devices of imaging-guided navigation, the first report of stereotaxy-guided

Neuronavigation

insertion of pedicle screws in the lumbar spine was performed in 1994 [21]. From this point onward, the technical development of key components such as optical localizers, higher speed of computing processes, and advancement in imaging systems lead to the CTbased imaging systems that were used in the operating theater. The first structured and systematic report on the utility and accuracy of intraoperative CT and navigation was published in 2000 [8]. Following the fixation of the navigation clamp which is equipped with imaging reflectors, the initial intraoperative CT scan is performed. This data set provided the basis for the matching of virtual and real data of the spinal structures. After co-registration and referencing, tools such as the awl and insertion burr hole can be registered to burr a canal along the optimal trajectory (Fig. 21.7). After screw placement through this canal, the iop CT scan identifies misplacement or fractures along the bony canal of the screw pathway. By this method, only around 1% of all CT-guided screws are misplaced and can be revised at once. Thus, revision surgery and additional radiation dosage can be avoided [11, 36].

Figure 21.8 The “MRXO suite,” Tokai University School of Medicine, Kanagawa, Japan [18].

443

444

Digital Neurosurgery

21.3.3 Hybrid Operating Rooms Summarizing all the technical applications of Sections 21.3.1 to 21.3.2, they merge into a multiple-room concept that combines every element of state-of-the-art surgery in one area. The first of these hybrid operating theaters in the field of neurosurgery incorporating CT, MRI, X-ray, and angiography (the “MRXO suite”) was inaugurated in February 2006 at the Tokai University School of Medicine, Kanagawa, Japan [18]. One famous evolution of this hybrid room is the 5700 squarefoot three-room Advanced Multimodality Image-Guided Operating (AMIGO) suite located inside the Brigham and Women’s Hospital of Havard Medical School, Boston, Massachusetts, U.S.A., in April 2012 (Fig. 21.9).

Figure 21.9 The “AMIGO suite,” Brigham and Women’s Hospital of Havard Medical School, Boston, Massachusets, U.S.A. (https://ncigt.org/AMIGO).

21.4 Robotics The first robotic systems used in medicine were adaptations from the automobile. From the initial introduction of a robotic arm by Unimate in 1961, another prominent example is the PUMA (Programmable Universal Machine for Assembly) developed by Victor Schreimann from Standford University for General Motors in the 80s, which was used in 1985 for the first stereotactic CT-guided biopsy in neurosurgery [14, 20]. Simultaneously, the National Air and Space Administration (NASA) became interested in remote surgery.

Robotics

Since then there was a rapid development of robotic systems, and their inherent technology, especially man-machine interfaces and computing. Current robot classifications include remote manipulator (“Master-Slave”) systems, and passive, semi-active and active medical robots. One of the most successful commercially available remote manipulators is the DaVinci system from Intuitive Surgical, Sunnyvale, CA, U.S.A. (Fig. 21.10).

Figure 21.10 The remote manipulator DaVinci, Xi system (https://www. intuitive.com/de-de/products-and-services/da-vinci/systems).

Figure 21.11 CIRQ (https://www.brainlab.com/de/chirurgie-produkte/ digitale-wirbelsaeulenchirurgie/cirq-robotics/).

445

446

Digital Neurosurgery

Passive systems are robotic arms such as the first PUMA system with various degrees of freedom in their joints that can be moved into the desired position under the control of neuronavigation (Fig. 21.11). Semi-active systems are controlled by the surgeon, but give, e.g., haptic or visual feedback about the surgical procedure. The range of motion is confined to previously defined boundaries. In neurosurgery, systems such as the ROSA ONE Brain System (Zimmer Biomet) or the Mazor X Stealth Edition (Medtronic) platforms require presurgical planning with, e.g., trajectories needed to place a stereotactic electrode or pedicle screw.

21.4.1 Augmented Reality

When three-dimensional, virtual objects are superimposed on real objects, one speaks of an augmentation of reality, or “augmented reality (AR).” AR devices can be used in spine surgery to aid the placement of bone screws screw [12]. The first successfully AR-guided instrumentation of pedicle screws was performed on February 16th, 2021 in The John Hopkin’s Hospital, Baltimore, ML, U.S.A. Other applications are the visualization of brain tumors, complex vascular pathologies, or fiber tracts. To underline the benefit of AR, one study compared two groups that treat an intracerebral aneurysm by microsurgical clipping. The first group did standard presurgical planning using 2- and 3-dimensional imaging. The second group used AR to virtually perform the craniotomy, dural opening, and finally the virtual clipping of the aneurysm. The latter performed the surgical procedure successfully with a time sparing of 80 minutes due to the presurgical AR training [33]. The Magic Leap goggles are capable of displaying organ systems in virtual reality, which projects a 3D image on the visual field and lenses of the person wearing the headset. From a neurosurgical perspective, it can visualize complex anatomical relationships and aid in surgical planning of challenging lesions (Fig. 21.12). Operating microscopes are the workhorse of microneurosurgery. With the evolution of digitalization, 3D images, or AR images such as supplying blood vessels can be overlaid on the white light image with modern microscope systems (Fig. 21.13). Exoscopes such as the

Robotics

ORBEYE (Olympus Europa SE & Co. KG, Hamburg, Germany) deliver high-resolution images and give the neurosurgeon more freedom during surgery with extended maneuverability and an increased field of vision.

Figure 21.12 Combination of an interactive anatomy viewer and AR with the Magic Leap googles (https://www.brainlab.com/surgery-products/overviewplatform-products/mixed-reality-applications/mixed-reality-viewer/).

Figure 21.13 AR image of intracranial arteries (Leica AERVO https://www. leica-microsystems.com/products/surgical-microscopes/p/arveo/ 2018, AANS congress debut).

447

448

Digital Neurosurgery

References 1. Albert, F. K., Forsting, M., Sartor, K., Adams, H. P., and Kunze, S. (1999). Early postoperative magnetic resonance imaging after resection of malignant glioma: objective evaluation of residual tumor and its influence on regrowth and prognosis. Neurosurgery, 34:45–61. 2. Au, K. L., Wong, J. K., Tsuboi, T., Eisinger, R. S., Moore, K., Lopes J. M., et al. (2021). Globus pallidus internus (GPi) deep brain stimulation for Parkinson’s disease: expert review and commentary. Neurol Ther, 10:7–30.

3. Barnett, G. H., Kormos, D. W., Steiner, C. P., Weisenberger, J. (1993). Use of a frameless, armless stereotactic wand for brain tumor localization with two-dimensional and three-dimensional neuroimaging. Neurosurgery, 33:674–678.

4. Bastos, D. C., Juvekar, P., Tie, Y., Jowkar, N., Pieper, S., Wells, W. M., Bi WL, Golby, A., Frisken, S., Kapur, T. (2021). Challenges and opportunities of intraoperative 3D ultrasound with neuronavigation in relation to intraoperative MRI. Front Oncol, 11:656519. 5. Burchiel, K. M., McCartney, S., Lee, A., Raslan, A. M. (2013). Accuracy of deep brain stimulation electrode placement using intraoperative computed tomography without microelectrode recording. J Neurosurg, 119:301–306.

6. Della Peppa, G. M., Di Bonaventura, R., Latour, K., Sturiale, C. L., Marchese, E., Puca, A., et al. (2021). Combined use of color Doppler ultrasound and contrast-enhanced ultrasound in the intraoperative armamentarium for arteriovenous malformation surgery. World Neurosurg, 147:150–156. 7. Fahlbusch, R. (2011). Development of intraoperative MRI: a personal journey. Acta Neurochir Suppl, 109:9–16.

8. Haberland, N., Ebmeier, K., Grunewald, J. P., Hliscs, R., Kalff, R. L. (2000). Incorporation of intraoperative computerized tomography in a newly developed spinal navigation technique. Computed Aided Surg, 5:8–27.

9. Hagan, M. J., Syed, S., Leary, O. P., Persad-Paisley, E. M., Lin, Y., et al. (2022). Pedicle screw placement using intraoperative computed tomography and computer-aided spinal navigation improves screw accuracy and avoids postoperative revisions: single-center analysis of 1400 pedicle screws. World Neurosurg, Jan 3; S1878–8750(21)019574. doi: 10.1016/j.wneu.2021.12.112. Online ahead of print.

References

10. Hounsfield, G. N. (1973). Computerized transverse axial scanning (tomography), Br J Radiol, 46:1016–1022.

11. Ishak, B., Younsi, A., Wieckhusen, C., Slonczewski, P., Unterberg, A. W., Kiening, K. L. (2019). Accuracy and revision rate of intraoperative computed tomography point-to-point navigation for lateral mass and pedicle screw placement: 11-year single center experience in 1054 patients. Neurosurg Rev, 42:895–890. 12. Kalfas, I. H. (2021). Machine vision navigation in spine surgery. Front Surg., 8:640554.

13. Koivukangas, J., Louhisalmi, Y., Alakuijala, J., Oikarinen, J. (1993). Ultrasound-controlled neuronavigator-guided brain surgery. J Neurosurg, 79:36–42. 14. Kwoh, Y. S., Hou, J., Jonckheere, E. A., Hayati, S. (1988). A robot with improved absolute accuracy for CT guided stereotactic brain surgery. IEEE Trans Biomed Eng, 35:153–160.

15. Lauterbur, P. C. (1973). Image formation by induced local interactions: examples employing nuclear magnetic resonance, Nature. 242:190– 191. 16. Leksell, LA. (1949) Stereotactic apparatus for intracerebral surgery. Acta Chir Scand, 99:229–233.

17. Mahboob, S. O., Mcphillips, R., Qiu, Z., Jiang, Y., Meggs, C., Schiavone, G. et al. (2016). Intraoperative ultrasound (IoUS) guided resection of gliomas; a meta-analysis and review of the literature. World Neurosurg, 92:255–263. 18. Matsumae M, Koizumi J, Fukuyama H, et al. (2007). World’s first magnetic resonance imaging/x-ray/operating room suite: a significant milestone in the improvement of neurosurgical diagnosis and treatment. J Neurosurg, 107:266–273.

19. Mislow, J. M., Golby, A. J., Black, P. M. (2009). Origins of intraoperative MRI: Neurosurg Clin N Am. 20:137–146.

20. Moran, M. E. (2007). Evolution of robotic arms. J Robot Surg, 1:103– 111.

21. Murphy, M. A., McKenzie, R. L., Kormos, D. W., Kalfas, I. H. (1994). Frameless stereotaxis for the insertion of lumbar pedicle screws: a technical note, J. Clin. Neuroscience, 1:257–260. 22. Nimsky C, Ganslandt O, Hastreiter P, Wang R, Benner T, Sorensen AG, et al. (2005). Preoperative and intraoperative diffusion tensor imaging– based fiber tracking in glioma surgery. Neurosurgery, 56:178–185.

449

450

Digital Neurosurgery

23. Nimsky, C., Ganslandt, O., Cerny, S., Hastreiter, P., Greiner, G., Fahlbusch, R. (2000). Quantification of, visualization of, and compensation for brain shift using intraoperative magnetic resonance imaging. Neurosurgery, 47:1070–1079. 24. Nimsky, C., Ganslandt, O., Hastreiter, P., Wang, R., Sorensen A. G., Fahlbusch, R. (2007). Preoperative and intraoperative diffusion tensor imaging-based fiber tracking in glioma surgery. Neurosurgery, 61(1 Suppl):178–185.

25. Prada, F., Del Bene, M., Mauri, G., Lamperti, M., Vailati, D., et al. (2018). Dynamic assessment of venous anatomy and function in neurosurgery with real-time intraoperative multimodal ultrasound: technical note. Neurosurg Focus, 45:E6. 26. Reid, M. H. (1978). Ultrasonic visualization of a cervical cord cystic astrocytoma, AJR Am J Roentgenol, 131:907–908. 27. Roessler, K., Sommer, B., Merkel, A., Rampp, S., Gollwitzer, S., Hamer, H., Buchfelder, M. (2016). A frameless stereotactic implantation technique for depth electrodes in refractory epilepsy utilizing intraoperative MRI imaging. World Neurosurg., 94:206–210. 28. Schichor, C., Terpolilli, N., Thorsteinsdottir, J., Tonn, J. C. (2017). Intraoperative computed tomography in cranial neurosurgery. Neurosurg Clin N Am., 82:595–602. 29. Senft, C., Bink, A., Franz, K., Vatter, H.m Gasser, T., Seifert, V. (2011). Intraoperative MRI guidance and extent of resection in glioma surgery: a randomised, controlled trial. Lancet Oncol, 12:997–1003. 30. Shalit MN, Israeli Y, Matz S, Cohen M (1979). Intra-operative computerized axial tomography. Surg Neurol, 11:382–384.

31. Sommer, B., Rampp, S., Doerfler, A., Stefan, H., Hamer, H. M., Buchfelder, M., Roessler, K. (2018). Investigation of subdural electrode displacement in invasive epilepsy surgery workup using neuronavigation and intraoperative MRI. Neurol Res, 40:811–821. 32. Sommer, B., Roessler, K., Rampp, S., Hamer, H. M., Blumcke, I., Stefan, H., Buchfelder, M. (2016). Magnetoencephalography-guided surgery in frontal lobe epilepsy using neuronavigation and intraoperative MR imaging. Epilepsy Research, 126:26–36. 33. Steinecke, T. C., Barbery, D. (2021). Microsurgical clipping of middle cerebral artery aneurysm: preoperative planning using virtual reality to reduce procedure time. Neurosurg Focus, 51:E12.

References

34. Steinmeier, R., Rachinger, J., Kaus, M. Ganslandt, O., Huk, W., Fahlbusch, R. (2000). Factors influencing the application accuracy of neuronavigation systems. Stereotact Funct Neurosurg, 75:188–202. 35. Wirtz, C. R., Bonsanto, M. M., Knauth, M., Tronnier, V. M., Albert, F. K., Staubert, A., Kunze, S. (1997). Intraoperative magnetic resonance imaging to update interactive navigation in neurosurgery: method and preliminary experience. Comput Aided Surg, 2:172–179.

36. Zausinger, S., Scheder, B., Uhl, E., Heigl, T., Morhard, D., Tonn, J.C. (2009). Intraoperative computed tomography with integrated navigation system in spinal stabilizations. Spine (Phila Pa 1976), 15: 2919–2926.

451

Chapter 22

Digital Surgery: The Convergence of Robotics, Artificial Intelligence, and Big Data

Philipp Jawny and Michael Beyer

Department of Cardiothoracic Surgery, University Hospital Augsburg, Augsburg, Germany [email protected]

22.1 Introduction Modern surgery essentially distinguishes between open and minimally invasive surgical techniques. These two paradigms were created by the implementation of the latest procedural and technical innovations of their respective times. Alongside the development of robot-assisted surgery and the increasingly accelerating digital transformation of the entire healthcare system in recent years the term “digital surgery” is also currently emerging. Although its definition is still quite variable, it can Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

454

Digital Surgery

be defined as the convergence of technology, artificial intelligence, and real-time data in the operating room. The digitalization of surgical processes promises no less than a further paradigm shift toward data-driven and technically highly precise operations. Since surgery occupies a unique role in medicine due to the special nature of motor implementation of therapeutic measures very individual challenges as well as unique opportunities arise. Accordingly, this chapter aims to elaborate on the current situation of modern surgery and its digital infrastructure. In this context, the existing equipment, latest developments, and possible potentials will be presented. Furthermore, the importance of robotics, artificial intelligence, and big data will be evaluated. Special attention will be given to clinical applications as well as the future role of future surgeons in a progressing digital working environment.

22.2 State-of-the-Art Surgery

The development of today’s surgery ultimately is the result of a series of groundbreaking achievements in medical history. Its foundation was laid during the 19th century by the advancements of aseptic operative conditions [52] and general anesthesia [30] Thus, open-invasive surgical procedures could be progressively established in the further course of time. The horizon of what was surgically feasible was continuously expanded, supported by constant technical innovation. The first endoscopes were developed and used experimentally in surgery as early as the beginning of the 20th century. However, it was not until the introduction of digital camera technology in the 1970s that their use in clinical practice increased [41]. During the 1980s, the term minimally invasive surgery finally developed and initiated a paradigm shift with the primary goal of reducing intraoperative trauma in all surgical specialties [24]. Subsequently and especially in recent year, minimally invasive surgery has been interpreted more and more in a therapeutically holistic way. Therefore, the focus is currently on the most minimally invasive, patient-specific surgical intervention, accompanied by gentle anesthesia and early postoperative functional rehabilitation.

Digital Infrastructure in the Surgical Environment

Since the 1990s minimally invasive surgery has also been complemented by robot-assisted surgery, which follows the already existing paradigms but adds technically high-precision instruments, following a digital input of the surgeon in charge [20]. These two paradigms, open and minimally invasive surgery, still characterize all surgical activity today. The number of operations performed worldwide has increased continuously in recent years. Currently, more than 310 million surgical procedures are performed worldwide every year [57, 58]. Morbidity is estimated to be about 15 % (> 45 million complications) and mortality about 0.5% (>1.5 million deaths) [51]. Accordingly, the goal of further development of modern surgical techniques is always the minimization of mortality and morbidity, as well as the optimization of patients’ postoperative quality of life.

22.3 Digital Infrastructure in the Surgical Environment

As already mentioned, further development of surgery has always been closely accompanied by technical and instrumental innovation, which is why surgical departments have also always been among the most technologically advanced working areas of a modern hospital. With the accelerating digitalization of the entire healthcare system, the surgical working environment has changed to a large extent in recent years. Compared with other industries, however, the implementation of modern digital infrastructure in medicine is lagging [2]. Depending on the respective specialty, highly individual requirements must be met. If this can be achieved, however, there are also far-reaching and unique potentials for better patient care. For a better understanding, the current technical infrastructure of elementary working areas within the surgical workflow will be elaborated and current innovations, as well as their potential, will be presented.

22.3.1 The Modern Operating Room

The operating room is the key location for all surgical specialties. Its space and equipment provide the fundamental infrastructure for surgical procedures and must meet a wide range of requirements,

455

456

Digital Surgery

some of which are highly specific. Primarily, aseptic working conditions and the possibility of providing general anesthesia with full access to the patient must be ensured. In addition, the operating room serves as a technical and spatial interface for medical devices. As an information hub, it must also make all relevant pre- and intraoperatively collected data available to the entire surgical team at all times [29]. Furthermore, when designing efficient operating theaters, factors such as noise levels and ergonomics must be considered, as these can have an impact on the operative quality [3, 33]. A modern operating room accordingly always targets the optimized effectiveness of the surgical workflow, consisting of operative quality, productivity, and patient safety [3]. To achieve this goal, the digitalization process must focus primarily on data acquisition and processing, as well as communication and networking. The current gold standard for new surgical facilities is an integrated operating room, which allows the connection of multiple medical devices from different manufacturers and the aggregation of comprehensive data [27]. Digital communication and networking within the operating room currently take place mainly via wired connections. Because of the steadily growing number of integrable systems, this communication standard is increasingly reaching its limits. However, modern wireless communication standards such as 5G already meet the requirements to take over a large part of the necessary data transmission [13], while helping to create a quiet and ergonomic working environment at the same time. The cognitive or smart operating room of the future will ultimately represent a spatial and digital platform for seamless integration of customizable hardware and software solutions. With its sensor technology, real-time analysis of integrated systems, and the use of artificial intelligence, the operating room will become an intelligent assistant [27]. Individually optimized working conditions can be created automatically and adapted to the surgical procedure and the surgical team in charge. Sources of error within the workflow can be detected and avoided at an early stage, processes can be optimized on a data-driven basis, and interaction with increasingly complex devices can be simplified by central control elements or even voice assistants [35].

Digital Infrastructure in the Surgical Environment

22.3.2 Preoperative Planning and Intraoperative Decision-Making Preoperative planning is one of the most important factors for effective surgery. It improves surgical quality, productivity, and patient safety [44]. Medical imaging plays a decisive role in this process. At the moment, mainly two-dimensional (2D) imaging techniques such as radiographs, computed tomography, and magnetic resonance imaging are used for this purpose [5]. Thanks to the computer-aided calculation of multiple 2D image layers, it is also possible to create three-dimensional (3D) reconstructions, which can be used as a digital model or, through modern 3D printing, as a physical replica for preoperative planning [6]. Initial studies have already demonstrated that the computed models have satisfactory accuracy in mapping patient-specific anatomy [31] and thus can help to improve understanding, especially of complex and variable, anatomical structures [25]. While these models, if available, are currently mainly used preoperatively, they could contribute to improved intraoperative navigation and even more patient-specific surgery by being available intraoperatively, for example through virtual or augmented reality solutions. In addition to the visualization of structural conditions, the collection and interpretation of patient-specific data both pre and intraoperatively also plays an important role in surgical decisionmaking. While preoperative decision-making toward a surgical intervention follows evidence-based guidelines and patient-specific risk evaluations by means of various scoring systems to a large extent, intraoperative decision-making processes are subject to a certain degree of uncertainty due to missing or subjective data [19]. It remains a challenge for every surgeon to make prompt and critical decisions for their patients during a procedure on a caseby-case basis. Yet a successful postoperative outcome is dependent on a complex combination of factors, including patient and surgical characteristics [17, 21]. The extensive use of digital decision-making systems and focus on qualitative and quantitative data collection will further promote

457

458

Digital Surgery

the objectifiability of surgical decision-making and thus provide new insights. By interpreting large amounts of data, surgery will continue to develop individual patient-specific therapeutic approaches. Supported by relevant data, as well as available computing power, digital decision-making systems will become more pervasive at every stage of a patient’s care [56].

22.3.3 Documentation and Reporting

The documentation and reporting of surgical procedures must be fundamentally changed to reflect these developments. During surgery, existing data is interpreted, new data is collected, decisions are made, and therapeutic measures are performed. All this information must be objectively documented to be further processable. However, a large portion of this data is either subjective, only partially stored, or not stored at all. For example, intraoperative decisions and deviations from standardized procedures are only subjectively recorded in the surgeon’s operative report, while digitally collected objective measurements, such as intraoperative monitoring, are sometimes still recorded manually or as occasional digital snapshots. The majority of the surgical video is either not recorded or is stored for a limited time before being discarded at a later date [7]. To enable the evolution toward data-driven and patient-specific surgery in the first place, documentation and reporting must be understood not so much in terms of team communication and legal protection. Rather, it should primarily serve the objective and interoperable collection of data and thus the optimization of therapy. Automated and digital documentation solutions, therefore, aim to create a uniform and interoperable database, but also make a significant contribution to reducing documentation effort for the responsible personnel and thereby increase productivity. Complete data storage during surgical procedures can quickly result in several hundred gigabytes of data per procedure. If these data sets are converted into nationally or even internationally standardized formats, breakthrough insights and a new level of data transparency can be achieved [19].

Digital Infrastructure in the Surgical Environment

22.3.4 Surgical Education and Training Against the backdrop of increasingly complex interventions and under the steadily growing pressure of the current shortage of personnel and junior staff, surgical education and training are also undergoing relevant changes. The one-to-one master-apprentice training that has been established for a long time is increasingly reaching its limits. Further training paradigms such as “see one, do one, teach one” no longer meet current quality demands. Under these circumstances, a pragmatic reduction according to a “do one, teach one” principle is being discussed [42, 49]. Nevertheless, it is undisputed that the experience of a surgeon is one of the most important factors influencing the outcome of the patient [5, 45]. Compared to other specialties, surgical education and training must meet very diverse and specific requirements. First of all, basic and later very specific medical knowledge must be imparted. Subsequently, technical skills must be trained and developed in a targeted manner. In addition, technical knowledge, and the handling of a wide range of instruments as well as diagnostic and therapeutic equipment must be taught. Last but not least, soft skills such as dealing with emergencies and stress, as well as leadership and communication within the team and with patients must also be developed. In order to ensure that all this is achieved to an adequate degree, clear objectives must be defined. Learning contents and scenarios should be available in a scalable format. Learning progress should be individually assessable in order to make further learning paths target-oriented and individualizable. Students and young surgeons already have access to multimedia content in the form of text, video, or audio online on-demand [53]. Modern knowledge databases accessible via multiple devices are gradually replacing their printed predecessors. Digital congress contributions, lecture series, and podcasts enliven scientific discussion. Simulation-based learning tools are becoming more and more immersive through the adoption of the latest technologies, allowing for an increasingly successful transfer of knowledge [14]. But also, intraoperative mentoring can be realized beyond the borders of one’s institution on an international level thanks to digital streaming technologies [4].

459

460

Digital Surgery

If the advancement of surgical education continues at the current pace, young surgeons will have on-demand access to global educational content as well as skill training of varying experience levels at any time. Communication and interaction with leading international experts will also become easier and available across the board. In the future, the old paradigm of “see one, do one, teach one” could be replaced by “see as many as you like whenever you want, do as many as you need to master required skills, and become a teacher in a scalable digital environment of surgical education.”

22.4 Digital Revolution of Surgery

The examples just mentioned clearly demonstrate how far-reaching and promising the changes of a digital working environment in surgery can be. However, digitalization will not only revolutionize the surgical working environment but also surgical therapy itself in the medium term. Following the achievements of open and minimally invasive surgery, digital surgery will shape the next phase of innovation through the combination of robotics, artificial intelligence, and objective real-time data. At this point, we are at the beginning of this development.

Figure 22.1 Schematic illustration of surgical innovation over time through adaptation of new paradigms (illustration created by the author, adapted from [19]).

Digital Revolution of Surgery

With the technical achievements of robot-assisted surgery, the increasing computing power for advanced artificial intelligence algorithms, and the growing availability of relevant real-time data, we are at another turning point in the history of surgery. By combining these technologies, the entire process of surgical therapy can be implemented digitally for the first time. To understand the fundamental relevance of this development, the input-process-output model of systems analysis and software engineering can be used to describe digital surgery in its most basic structure [59]. Accordingly, big data stands for the abundance of general as well as patient- and procedure-specific data and thus for the digital input. Artificial Intelligence algorithms are able to process these extensive amounts of data in real-time. Finally, modern robotic platforms perform the output in the form of technically precise and digitally controlled motor actions. Digital surgery will ultimately be a datadriven interplay of these three technologies and is also referred to as “cognition-guided surgery” [38]. In the following, we will further explore these technologies and their relevance in the surgical context.

22.4.1 Surgical Robotics

Robotics as a scientific discipline currently has no uniform definition. Rather, it is understood as an interdisciplinary field of study of engineering and computer science with a focus on the robot’s task domain [43]. In the context of surgery, robots are digital platforms for interaction with patients. They integrate both the system for perception as well as manipulation and provide a digital interface between patient and surgeon during the surgical procedure. In recent years, robot-assisted surgery has become widely adopted and has proven that it can overcome the intrinsic limitations of surgical endoscopy. Key advantages include high-resolution stereoscopic optics, stable and surgeon-controlled camera guidance, optimized ergonomics, and improved freedom of movement as well as motion scaling [1]. Nevertheless, the necessary evidence to actually establish robot-assisted surgery as a standard of care is currently still lacking [11].

461

462

Digital Surgery

Experts from all surgical disciplines agree that future technical innovation will largely focus on robotic surgery, which will lead to significant treatment benefits in the medium term. Visualization and cognition, as well as the technically highly precise instrumentation of robotic systems, but also the possibilities of future autonomy play a decisive role in this regard.

22.4.1.1 Visualization and cognition

The most important factor in intraoperative perception is visualization. The visual impression of anatomy as well as critical structures enables surgeons to navigate accurately and without complications in the respective surgical field. It is already known from conventional intraoperative endoscopy in the context of minimally invasive surgery that three-dimensional (3D) visualization improves surgical performance compared to higher-resolution two-dimensional (2D) imaging [54]. Modern robotic systems therefore usually integrate 3D optics. Higher resolutions and the associated sharper representation of the surgical field offer additional potential for optimization in the future. In recent years, new imaging techniques beyond the human color and perception spectrum have also been developed and integrated into robotic platforms. These include the now well-established near-infrared (NIR) fluorescence imaging using indocyanine green (IGC) [12], as well as laser speckle contrast imaging and narrowband imaging (NBI), which can be used to visualize perfusion or ventilation areas as well as vascular structures, to name a few [8]. This way resection margins can be optimized according to patientspecific areas of perfusion or ventilation [9], or metastases can be specifically targeted [34]. Such tools and their future iterations will provide surgeons with additional capabilities and data respectively beyond human perception. In addition to visualization, especially haptics represents an important cognition during a surgical procedure. By perceiving surface textures, temperature, tension, pressure, and other stimuli an experienced surgeon is able to assess the structural integrity and localization of anatomical structures and use his measures accordingly in a targeted manner [10]. Digitally recreating these sensory impressions of multiple human mechanoreceptors presents an enormous challenge [15]. However, should it succeed, it will

Digital Revolution of Surgery

enable an increasingly immersive interaction with the patient via robotic systems. Last but not least, the internal sensors of robotic platforms represent another important data set that has not really been researched yet. For the first time in surgical history, it provides objective data on intraoperative movement patterns and thus individual surgical techniques. If further studies identify a significant correlation between different techniques and operative outcomes, this could lead to far-reaching standardization of surgery in the long term. The implementation of such findings could target surgical training or pave the way for optimized autonomous systems.

22.4.1.2 Advanced instruments

Apart from the possibilities of visualization and cognition, surgical robotics primarily enables technically precise manipulation of tissues, thanks to its advanced instruments. The advantages of robotic instruments primarily relate to their superior dexterity and precision through improved freedom of movement, ergonomics, and articulation in seven axes [37]. By separating the surgeon’s input and the motor output, there are no longer any human limits to the technical development of robotic instruments in terms of shape, size, and ergonomics. As a first step, this will lead to increasing the miniaturization of instruments while simultaneously increasing their precision. After the first years of robotic surgery, when we mainly saw instruments that were ultimately digital replicas of endoscopic instruments, we can currently observe the development of nonlinear robotic instruments [26]. These allow both the use of multiple instruments via a single access port, as well as tissue-sparing use within complex anatomic structures. Further developments in robotic instrumentation cannot be foreseen in the end, but they will almost certainly change the nature of surgery significantly.

22.4.1.3 Autonomous robotic systems

While the technical systems for perception and interaction with the patient are constantly evolving, the question of the autonomy of the machine is inherent in the nature of robotics. In the best case, autonomy could standardize surgical techniques and thus increase their effectiveness.

463

464

Digital Surgery

Initial experimental applications have shown that automated interpretation of visual and sensory data can both increase precision and reduce error [46]. Furthermore, other sources of error such as stress and fatigue can be eliminated, especially during lengthy surgical procedures [40]. However, as machines become increasingly autonomous, new questions arise, especially legal, regulatory, and ethical questions, to which no answers yet exist [40]. The technical foundation for possible autonomy resides in the integration of artificial intelligence into future robotic platforms in order to mature them into intelligent assistants.

22.4.2 Artificial Intelligence

The term artificial intelligence describes algorithms that give computers the ability to think and perform cognitive functions such as problem-solving, object and word recognition, and decisionmaking [23]. A wide range of concepts is subsumed under the topic of artificial intelligence. For surgical applications, the primary benefit of these technologies in the foreseeable future will be the computerassisted optimization of human performance [22]. For this purpose, four main areas of focus, namely machine learning, natural language processing, artificial neural networks, and computer vision are of primary importance.

22.4.2.1 Machine learning

Machine learning describes the ability of machines to make predictions or perform tasks based on input and output data without being specifically programmed to do so [22]. It is particularly useful for detecting subtle patterns in large data sets that would not have been apparent using traditional statistical methods or human perception. Due to a large number of pre-, intra-, and postoperative data streams in modern surgery, this technology will become highly relevant in the surgical environment. By incorporating multiple perioperative data sources such as diagnoses, treatments, and laboratory measurements into a machine learning algorithm one was able to outperform logistic regression for the prediction of surgical site infections [47].

Digital Revolution of Surgery

22.4.2.2 Natural language processing Natural language processing enables the machine interpretation of human language, both written and spoken. Modern systems are no longer only able to recognize individual words but can also perceive correlations in terms of semantics and syntax [39]. Particularly in the case of subjective forms of documentation such as surgical reports, this technology can help to improve the available quantity and quality of data by systematically standardizing the processing of both retrospective and prospective documentation. Analysis of operative reports and ongoing documentation, for example, has enabled the creation of a predictive model regarding postoperative anastomotic insufficiency in patients undergoing colorectal resection [48]. Natural language processing is also used to automate case encoding based on written documentation and thus make the clinical workflow more efficient. This method is shown to generate reliable data similar to or better than human processing [18].

22.4.2.3 Artificial neural networks

Artificial neural networks represent a further subgroup of machine learning. Input signals are processed in levels of simple computational steps and further processed with weighting adjusted in the course, while the network learns the relationship between input and output with increasing data volume. This way, traditional risk prediction models can be significantly outperformed [22]. Faster risk prediction with higher accuracy may significantly improve patient care in this regard. In combination with real-time intraoperative data, surgical techniques could be improved to minimize postoperative complications. For example, artificial neural networks combined with other machine learning approaches were already able to predict inhospital mortality after open abdominal aortic aneurysm repair with 95.4% accuracy by using multiple patient-specific clinical variables [36].

22.4.2.4 Computer vision

Computer vision describes the ability of machines to interpret the content of images and videos [50]. Since surgical interventions are

465

466

Digital Surgery

significantly influenced by the perception of visual impressions, there is tremendous potential in the application of this subfield of artificial intelligence. Endoscopic videos from minimally invasive or robotic surgery are particularly suitable for intraoperative real-time analysis due to the immediate digital availability of the video signal and the ideally stable field of view. Image classification is used to identify surgical steps, for example, as well as to detect missing and unexpected steps. Object recognition enables the identification of anatomical or pathological structures or surgical instruments within the field of view. Semantic segmentation refines this capability by recognizing the respective structures with pixel precision [28].

22.4.3 Big Data

All these systems are significantly dependent on sufficient data quality and quantity. In this context, the term big data is almost ubiquitous today but has no uniform definition. It is often used to describe practical processes such as the collection, organization, and modeling of integrated data [16]. The processing of big data in surgery is also called surgical data science, which is an area of biomedical data science but has unique characteristics due to its focus on procedural data. It refers to the patient, all effectors manipulating the patient, sensors to sense patient- and procedurerelated data, and medical domain knowledge [32]. While much of this data is already perceived and interpreted in clinical practice, it is often not adequately stored. Scientific processing or even processing by the above-mentioned artificial intelligence algorithms is therefore only possible to a limited extent. However, especially in a complex field such as surgery, with multiple influencing factors and stakeholders, a fully comprehensive data set is of central importance in order to be able to make patient-specific decisions at any time of treatment.

22.5 Problems and Challenges

While technical development in all of the aforementioned areas is constantly accelerating and scientific research is delivering

Problems and Challenges

promising results in experimental settings, fundamental problems and challenges must first be overcome in order to achieve convergence of all technologies and thereby enable widespread integration into clinical practice.

22.5.1 Technical Infrastructure and Interoperability

Data-driven medicine or surgery can only be successfully implemented if diagnoses, clinical findings, measurements, and treatments are actually treated as data. In other words, they must be collected in a standardized manner, objectively recorded, and systematically stored and processed. This requires a holistic, standardized, and interoperable digital infrastructure of at least the individual hospital, rather than the entire healthcare system. In the process, data collection should be objectified and automated as often as possible. The current practice of manually transferring measured values from one system to another not only leads to unnecessary data duplications but also represents an additional source of error. However, due to the often very specific requirements across several medical departments, isolated solutions are indispensable on both the software and hardware sides. This makes open and structured data even more important in order to achieve interoperability across different devices. The stakeholders of the treatment process must exert a certain pressure on the medical device industry to accept open and interoperable data standards or to introduce them in a mandatory manner. Because in the end, the patient or respectively the treatment team, which is bound to confidentiality, will always remain the owner of the collected data, not only from a therapeutic but also from a data protection perspective.

22.5.2 Technical Expertise

As the digital infrastructure and its requirements rapidly evolve, a certain level of technical understanding becomes a mandatory requirement for all users within the healthcare system. The lack of appropriate skills among users currently results in existing digital systems being utilized incompletely, incorrectly, or not at all.

467

468

Digital Surgery

However, not only the daily use but also the implementation of new solutions, as well as the further development of existing systems in line with therapeutic demand, requires a certain understanding of the structure and capabilities of digital infrastructure. Accordingly, basic technical knowledge should become an obligatory component of every medical education. In particular, the stakeholders responsible for therapeutic measures must gain additional knowledge in order to be able to help shape further technical development in line with clinical requirements.

22.6 Future of Surgery

The digitization of medicine and consequently of surgery is indisputably the task of our generation. Robotics, artificial intelligence, and big data will fundamentally change the way we practice surgery in the future [55]. The intention is to assist the surgeon not only during the operation but also during the whole patient process by giving the relevant information at the right time and in the right place [27]. While surgical robotics primarily provides new objective data and enables highly precise therapeutic options based on digital input, modern methods of artificial intelligence are able to comprehensively analyze situations on the basis of extensive and high-quality data, provide digital decision support, or even make decisions themselves. Big data in surgery ultimately represents the interface between these two technologies and allows further patient-, procedure-, or specialty-specific data to be integrated as well. In summary, future innovations in data-driven and patientcentered surgery will not be achieved by improvements in just one of these three areas, but only through a seamless interaction of all technologies.

22.6.1 The Need for Interprofessional Teams

Overcoming the problems and challenges and leveraging the potential of an increasingly complex digital surgical working environment requires strong interdisciplinary collaboration [27]. This is no

References

longer limited to therapeutic or nursing disciplines. Engineers and computer scientists must also be closely integrated into the treatment processes. Not just to provide technical troubleshooting support, but more importantly to use their expertise and analytical skills in a targeted way to optimize processes. In medicine, and especially in surgery, highly individual and patient-specific issues will always arise, despite or perhaps even because of increasing standardization and specialization. Accordingly, all stakeholders in digital surgery must be available onsite in order to be able to react quickly and in a targeted manner if necessary and to gradually develop the relevant processes.

22.6.2 The Future Role of Surgeons

Within this multidisciplinary team, a surgeon must continue to take the leadership role, being responsible for therapeutic measures. At the same time, medical expertise alone is no longer sufficient to fulfill this function. The surgeon of tomorrow must not only be able to plan, perform and follow up on highly complex surgical procedures but must also understand the basic principles, capabilities, and limits of data science and digital technologies and actively contribute to the development and clinical application of future technologies [32]. Supported by integrated digital assistance systems, surgeons will then be able to overcome the limitations of human perception and motor skills. Redundant tasks can be taken over by partially autonomous machines. Yet surgeons will not be replaced by machines that use technologies such as robotics, artificial intelligence, and big data in the foreseeable future. However, one thing seems certain: surgeons who utilize these technologies for the benefit of their patients will replace those who do not.

References

1. Ballantyne, G. H., Moll, F. (2003). The da Vinci telerobotic surgical system: the virtual operative field and telepresence surgery. Surg Clin N Am. 83(6), 1293–1304. 2. Beaulieu, M., Bentahar, O. (2021). Digitalization of the healthcare supply chain: a roadmap to generate benefits and effectively support healthcare delivery. Technol Forecast Soc Change. 167, 120717.

469

470

Digital Surgery

3. Bharathan, R., Aggarwal, R., Darzi, A. (2013). Operating room of the future. Best Pract Res Clin Obstet Gynaecol. 27(3), 311–322.

4. Bilgic, E., Turkdogan, S., Watanabe, Y., Madani, A., Landry, T., Lavigne, D., et al. (2017). Effectiveness of telementoring in surgery compared with on-site mentoring: a systematic review. Surg Innov. 24(4), 379– 385. 5. Birkmeyer, J. D., Finks, J. F., O’Reilly, A., Oerline, M., Carlin, A. M., Nunn, A. R., et al. (2013). Surgical skill and complication rates after bariatric surgery. N Engl J Med. 369(15), 1434–1442.

6. Bücking, T. M., Hill, E. R., Robertson, J. L., Maneas, E., Plumb, A. A., Nikitichev, D. I. (2017). From medical imaging data to 3D printed anatomical models. PLOS ONE. 12(5), e0178540.

7. Chadebecq, F., Vasconcelos, F., Mazomenos, E., Stoyanov, D. (2020). Computer vision in the surgical operating room. Visc Med. 36(6), 456– 462.

8. Chalopin, C., Maktabi, M., Köhler, H., Cervantes-Sanchez, F., Pfahl, A., Jansen-Winkeln, B., et al. (2021). Intraoperative imaging for procedures of the gastrointestinal tract, in Innovative Endoscopic and Surgical Technology in the GI Tract (Horgan, S., Fuchs, K.-H., eds), Springer International Publishing, Cham, pp. 365–379. 9. Chiow, A. K. H., Rho, S. Y., Wee, I. J. Y., Lee, L. S., Choi, G. H. (2021). Robotic ICG guided anatomical liver resection in a multi-centre cohort: an evolution from “positive staining” into “negative staining” method. HPB (Oxford), 23(3), 475–482. 10. Culbertson, H., Schorr, S. B., Okamura, A. M. (2018). Haptics: the present and future of artificial touch sensation. Annu Rev Control Robot Auton Syst. 1(1), 385–409.

11. Damle, A., Damle, R. N., Flahive, J. M., Schlussel, A. T., Davids, J. S., Sturrock, P. R., et al. (2017). Diffusion of technology: trends in roboticassisted colorectal surgery. Am J Surg. 214(5), 820–824. 12. Daskalaki, D., Aguilera, F., Patton, K., Giulianotti, P. C. (2015). Fluorescence in robotic surgery. J Surg Oncol. 112(3), 250–256. 13. Dohler, M. (2021). 5G networks, haptic codecs, and the operating theatre, in Digital Surgery (Atallah, S., ed), Springer, Cham, pp. 71–86.

14. Dyke, C., Franklin, B. R., Sweeney, W. B., Ritter, E. M. (2021). Early implementation of Fundamentals of Endoscopic Surgery training using a simulation-based mastery learning curriculum. Surgery. 169(5), 1228–1233.

References

15. El Rassi, I., El Rassi, J.-M. (2020). A review of haptic feedback in teleoperated robotic surgery. J Med Eng Technol. 44(5), 247–254.

16. Favaretto, M., De Clercq, E., Schneble, C. O., Elger, B. S. (2020). What is your definition of Big Data? Researchers’ understanding of the phenomenon of the decade. PLOS ONE. 15(2), e0228987.

17. Franovic, S., Kuhlmann, N. A., Pietroski, A., Schlosser, C. T., Page, B., Okoroha, K. R., et al. (2021). Preoperative patient-centric predictors of postoperative outcomes in patients undergoing arthroscopic meniscectomy. Arthroscopy. 37(3), 964–971.

18. Friedman, C., Shagina, L., Lussier, Y., Hripcsak, G. (2004). Automated encoding of clinical documents based on natural language processing. J Am Med Inform Assoc. 11(5), 392–402.

19. Fuerst, B., Fer, D. M., Herrmann, D., Kilroy, P. G. (2021). The vision of digital surgery, in Digital Surgery (Atallah, S., ed), Springer, Cham, pp. 11–23. 20. George, E. I., Brand, T. C., LaPorta, A., Marescaux, J., Satava, R. M. (2018). Origins of robotic surgery: from skepticism to standard of care. JSLS. 22(4), e2018.00039.

21. Geraci, T. C., Chang, S. H., Shah, S. K., Kent, A., Cerfolio, R. J. (2021). Postoperative air leaks after lung surgery: predictors, intraoperative techniques, and postoperative management. Thorac Surg Clin., 31(2), 161–169. 22. Hashimoto, D. A., Ward, T. M., Meireles, O. R. (2020). The role of artificial intelligence in surgery. Adv Surg. 54, 89–101.

23. Hashimoto, D. A., Rosman, G., Rus, D., Meireles, O. R. (2018). Artificial intelligence in surgery: promises and perils. Ann Surg. 268(1), 70–76.

24. Himal, H. S. (2002). Minimally invasive (laparoscopic) surgery. Surg Endosc Other Interv Tech. 16(12), 1647–1652.

25. Javan, R., Herrin, D., Tangestanipoor, A. (2016). Understanding spatially complex segmental and branch anatomy using 3D printing: liver, lung, prostate, coronary arteries, and circle of Willis. Acad Radiol. 23(9), 1183–1189.

26. Keller, D., Atallah, S., Seela, R., Seeliger, B., Parra-Davila, E. (2021). Nonlinear robotics in surgery, in Digital Surgery (Atallah, S., ed), Springer, Cham, pp. 285–310. 27. Kenngott, H. G., Apitz, M., Wagner, M., Preukschas, A. A., Speidel, S., Müller-Stich, B. P. (2017). Paradigm shift: cognitive surgery. Innov Surg Sci. 2(3), 139–143.

471

472

Digital Surgery

28. Kitaguchi, D., Takeshita, N., Hasegawa, H., Ito, M. (2022). Artificial intelligence‐based computer vision in surgery: recent advances and future perspectives. Ann Gastroenterol Surg. 6(1), 29–36. 29. Koninckx, P. R., Stepanian, A., Adamyan, L., Ussia, A., Donnez, J., Wattiez, A. (2013). The digital operating room and the surgeon. Gynecol. Surg. 10(1), 57–62.

30. Larson, M. D. (2011). History of anesthesia, in Basic of Anesthesia 6th ed (Miller R, Pardo Jr MC, ed), Elsevier Saunders, 3–10.

31. Luzon, J. A., Andersen, B. T., Stimec, B. V., Fasel, J. H. D., Bakka, A. O., Kazaryan, A. M., et al. (2019). Implementation of 3D printed superior mesenteric vascular models for surgical planning and/or navigation in right colectomy with extended D3 mesenterectomy: comparison of virtual and physical models to the anatomy found at surgery. Surg Endosc. 33(2), 567–575. 32. Maier-Hein, L., Eisenmann, M., Sarikaya, D., März, K., Collins, T., Malpani, A., et al. (2022). Surgical data science: from concepts toward clinical translation. Med Image Anal. 76, 102306.

33. McLeod, R., Myint-Wilks, L., Davies, S. E., Elhassan, H. A. (2021). The impact of noise in the operating theatre: a review of the evidence. Ann R Coll Surg Engl. 103(2), 83–87. 34. Mehdorn, A. S., Beckmann, J. H., Braun, F., Becker, T., Egberts, J. H. (2021). Usability of indocyanine green in robot-assisted hepatic surgery. J Clin Med. 10(3), 456.

35. Miehle, J., Ostler, D., Gerstenlauer, N., Minker, W. (2017). The next step: intelligent digital assistance for clinical operating rooms. Innov Surg Sci. 2(3), 159–161.

36. Monsalve-Torra, A., Ruiz-Fernandez, D., Marin-Alonso, O., SorianoPayá, A., Camacho-Mackenzie, J., Carreño-Jaimes, M. (2016). Using machine learning methods for predicting in hospital mortality in patients undergoing open repair of abdominal aortic aneurysm. J Biomed Inform. 62, 195–201.

37. Morrell, A. L. G., Morrell-Junior, A. C., Morrell, A. G., Mendes, J. M. F., Tustumi, F., De-Oliveira-E-Silva, L. G., et al. (2021). The history of robotic surgery and its evolution: when illusion becomes reality. Rev Col Bras Cir. 48, e20202798. 38. Müller-Stich, B., Wagner, M., Schulze, A., Bodenstedt, S., MaierHein, L., Speidel, S., et al. (2022). „Cognition-Guided Surgery“: computergestützte intelligente Assistenzsysteme für die onkologische Chirurgie. Forum. 37(1), 32–37.

References

39. Nadkarni, P. M., Ohno-Machado, L., Chapman, W. W. (2011). Natural language processing: an introduction. J Am Med Inform Assoc. 18(5), 544–551.

40. O’Sullivan, S., Nevejans, N., Allen, C., Blyth, A., Leonard, S., Pagallo, U., et al. (2019). Legal, regulatory, and ethical frameworks for development of standards in artificial intelligence (AI) and autonomous robotic surgery. Int J Med Robot. 15(1), e1968. 41. Peter, S. D. S., Holcomb, III, G. W. (2008). History of Minimally Invasive Surgery, Elsevier Health Sciences, Philadelphia. 42. Picarella, E. A., Simmons, J. D., Borman, K. R., Replogle, W. H., Mitchell, M. E. (2011). “Do one, teach one” the new paradigm in general surgery residency training. J Surg Educ. 68(2), 126–129. 43. Redfield, S. (2019). A definition for robotics as an academic discipline. Nat Mach Intell. 1(6), 263–264.

44. Salb, T., Weyrich, T., Dillmann, R. (1999). Preoperative Planning and Training Simulation for Risk Reducing Surgery. International Training and Education Conference (ITEC), Citeseer. 45. Scally, C. P., Varban, O. A., Carlin, A. M., Birkmeyer, J. D., Dimick, J. B. (2016). Video ratings of surgical skill and late outcomes of bariatric surgery. JAMA Surg. 151(6), e160428.

46. Shademan, A., Decker, R. S., Opfermann, J. D., Leonard, S., Krieger, A., Kim, P. C. W. (2016). Supervised autonomous robotic soft tissue surgery. Sci Transl Med. 8(337), 337ra364–337ra364.

47. Soguero-Ruiz, C., Fei, W. M. E., Jenssen, R., Augestad, K. M., Álvarez, J.L. R., Jiménez, I. M., et al. (2015). Data-driven temporal prediction of surgical site infection. AMIA Annu Symp Proc. 2015, 1164–1173.

48. Soguero-Ruiz, C., Hindberg, K., Rojo-Álvarez, J. L., Skrøvseth, S. O., Godtliebsen, F., Mortensen, K., et al. (2016). Support vector feature selection for early detection of anastomosis leakage from bag-ofwords in electronic health records. IEEE J Biomed Health Inform. 20(5), 1404–1415. 49. Stevens, R., Davies, M. (2012). “Do one, teach one”: the new paradigm in general surgery residency training. J Surg Educ. 69, 135–136.

50. Szeliski, R. (2010). Computer Vision: Algorithms and Applications. Springer Science & Business Media.

51. The International Surgical Outcomes Study Group. (2016). Global patient outcomes after elective surgery: prospective cohort study in 27 low-, middle- and high-income countries. Br J Anaesth. 117(5), 601–609.

473

474

Digital Surgery

52. Thurston, A. J. (2000). Of blood, inflammation and gunshot wounds: the history of the control of sepsis. Aust N Z J Surg. 70(12), 855–861. 53. Tokuno, J., Fried, G. M. (2022). Digital education in general thoracic surgery: a narrative review. Ann Thorac Surg.

54. Wahba, R., Datta, R., Busshoff, J., Bruns, T., Hedergott, A., Gietzelt, C., et al. (2020). 3D versus 4K display system – influence of “State-ofthe-art”-display technique on surgical performance (IDOSP-study) in minimally invasive surgery: a randomized cross-over trial. Ann Surg. 272(5), 709–714. 55. Wall, J., Krummel, T. (2020). The digital surgeon: how big data, automation, and artificial intelligence will change surgical practice. J Pediatr Surg. 55s, 47–50.

56. Ward, T. M., Meireles, O. (2021). The cognitive revolution, in Digital Surgery (Atallah, S., ed), Springer, Cham, pp. 1–9. 57. Weiser, T. G., Haynes, A. B., Molina, G., Lipsitz, S. R., Esquivel, M. M., Uribe-Leitz, T., et al. (2015). Estimate of the global volume of surgery in 2012: an assessment supporting improved health outcomes. Lancet. 385, S11. 58. Weiser, T. G., Haynes, A. B., Molina, G., Lipsitz, S. R., Esquivel, M. M., Uribe-Leitz, T., et al. (2016). Size and distribution of the global volume of surgery in 2012. Bull World Health Organ. 94(3), 201–209F.

59. Zelle, J. M. (2004). Python Programming: An Introduction to Computer Science, Franklin, Beedle & Associates, Inc.

Chapter 23

Digital Urology

Alexander Buchner

Department of Urology, Ludwig-Maximilians-University of Munich, Munich, Germany [email protected]

23.1 Introduction During the last years, there was continuous and ongoing progress in new digital methods and tools in urology, leading to substantial improvements in diagnosis, prognosis assessment, telemedical care, and therapy. The following chapter will focus on the topics of telemedicine, robotics, and on the emerging role of artificial intelligence in the field of urology.

23.2 Telemedicine

Telemedicine implementations include televisitations, teleconsultation, patient monitoring, and more. While telemedicine is a rapidly growing sector of the healthcare industry, the use of telemedicine Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

476

Digital Urology

in urology is not very well defined. The evidence base for the use of telemedicine is not robust. However, institutions and physician groups are testing urological telemedicine in a number of different settings [13]. The 2019 SARS-CoV-2 pandemic has caused increased interest in the application of telemedicine in health care, also in urology. The pandemic showed that cancer patients have a significantly higher risk of adverse events and poorer outcomes from COVID-19 [28]. There are many patients with urological tumors (e.g., prostate cancer, bladder cancer, kidney cancer), and these vulnerable uro-oncological patients are willing to engage in televisitations but not in all scenarios. The existing personal patient-physician relationship is crucial for the patients and has to be balanced with the application of telemedicine during the pandemic and later [41]. There is a need for guidelines in the field of telemedicine. In 2020, the European Association of Urology released recommendations for the application of telemedicine. The authors concluded that telemedicine facilitates specialized urological clinical support at a distance, solves problems of limitations in mobility, reduces unnecessary visits to clinics, and is useful for reducing the risk of viral transmission in the current COVID-19 outbreak. The authors confirm that there may also be reasons for the continued use of telemedicine beyond the pandemic [43]. Scientific literature shows that there is an increasing number of studies about telemedicine in urology, including many prospective studies and randomized clinical trials. Published data indicate that telemedicine has been implemented successfully in several common clinical scenarios, including decision-making after diagnosis of nonmetastatic prostate cancer, follow-up care of patients after curative treatment of prostate cancer, the initial diagnosis of hematuria, diagnosis and follow-up care of urinary stones and urinary tract infections, initial evaluation of patients with incontinence, administration of behavioral therapies and more [37]. Furthermore, the field of sexual medicine is particularly primed for telemedicine because many men feel uncomfortable bringing up their sexual problems in person. They often feel embarrassed about their sexual dysfunction and avoid seeking medical help in a traditional setting. Telemedicine allows these patients to talk about extremely personal and upsetting topics (e.g., erectile dysfunction, premature ejaculation) in a comfortable and familiar environment [7].

Telemedicine

To illustrate the variety of telemedicine applications in urology, some examples are described in the following text. The use of wearables to measure body functions objectively is becoming more common in recent years with the idea of the quantified self, where individuals can track and store measurable health parameters. Physical activity monitors are being used in chronic conditions including chronic obstructive pulmonary disease, congestive heart failure, diabetes mellitus, and obesity. There is evidence that activity monitoring wearables are a promising tool for oncological practice [6]. Since many patients in urology are cancer patients with often long-term therapy in an ambulant setting, such devices can help to assess the general condition of these patients. A prospective study on patients undergoing cystectomy for bladder cancer used an activity tracker to monitor heart rate, steps, and sleep data for 30 days postoperatively. A higher step count was significantly associated with reduced odds of an adverse event [46]. In another recent study, a wearable urinary management system was developed to analyze urinary time and interval data detected through patient-worn smart bands, and the results of the analysis were presented through a webbased visualization to enable monitoring and appropriate feedback for urological patients. This shows that also behaviors such as urination can be monitored objectively, enabling support for clinical diagnostic assistance [14]. Smartphone apps are being widely used for various purposes for many years. There are more and more apps available for medical purposes, also in urology. In a randomized controlled trial on women with stress urinary incontinence, one patient group used an app with a treatment program focused on pelvic floor muscle training, and information about incontinence and lifestyle factors. After three months of follow-up, the app group reported significant improvements in symptom severity and quality of life, compared to the control group [3]. Direct-to-consumer (DTC) telemedicine services are increasingly used. Patients can access a licensed physician online from a smartphone or computer, often 24 hours a day. A cross-sectional observational study on more than 20,000 patients evaluated the management of urinary tract infections in a large DTC telemedicine platform. The authors concluded that management of urinary tract infections via DTC telemedicine appears to be appropriate for average-risk patients, and most are able to self-diagnose. Most

477

478

Digital Urology

patients in the study received guideline-concordant care. Therefore, DTC telemedicine offers convenient, low-cost care that is generally appropriate [40]. The performance of a virtual clinic for patients with uncomplicated acute ureteric colic was evaluated in a prospective study on more than 1,000 cases. Patients were referred in real-time by clinicians using an electronic referral method. Referrals were reviewed every weekday by a urologist or specialist urology stone nurse. A virtual clinic telephone consultation was performed by a specialist nurse or consultant urologist via the patient’s personal mobile cellular telephone or landline number. The median time for a virtual clinic decision was two days. In total, 34.4% of the patients were discharged directly or after a further virtual clinic encounter. A further 48.4% proceeded to a face-to-face clinic and 17.2% to surgical intervention. This study demonstrated significant improvements in a virtual clinic compared to traditional management pathways regarding the clinical outcome, safety, and positive fiscal and environmental savings [10]. Another prospective study established a tele-urology program for managing hematuria consultations, including a survey of patient attitudes and satisfaction with such a program. Following initial referral, patients were contacted and scheduled for a dedicated telephone appointment consisting of a structured interview performed by a physician resident. After each interview, the physician scheduled upper tract imaging, cystoscopy, and additional studies, if indicated. Patients reported high acceptance and satisfaction with telephone clinics as a mechanism for expedited hematuria evaluation. The authors concluded that telephone appointments have the potential to positively impact healthcare access and productivity [44]. In summary, telemedicine has the potential to become an important factor in urological healthcare, especially as teleconsultation, DTC telemedicine services, specialized smartphone apps, and wearables for patient monitoring. One limitation is that some patients, especially at a higher age, might experience difficulties to use the new technology. Therefore, telemedicine services should always be an option in the future, but not mandatory.

Robotics

23.3 Robotics While robots currently are commonplace in the industry, their use in medicine started relatively recently. Urology accounted soon for the largest single-specialty use of robot-assisted surgery [24]. The daVinci platform (Intuitive Surgical, Sunnyvale, CA) dominated the world of robotic surgery during the last few years [26]. For two decades, the CyberKnife radio-surgical robotic system (Accuray, Madison, WI) is available and it is useful for the radiation therapy of cancers, tumors, and other lesions [45].

23.3.1 Robot-Assisted Surgery

The initial daVinci system received approval by the Food and Drug Administration (FDA) for use in the United States in the year 2000 [45]. According to the manufacturer’s website (Intuitive Surgical), through 2020 there were more than 8.5 million procedures performed with this system, and nearly 6000 daVinci systems were installed worldwide in 67 countries. This robotic platform has a central arm with a binocular lens for 3D vision on a cart, and additional arms that can carry a variety of surgical instruments. The daVinci system was the first to be used for a cholecystectomy (Belgium 1997), and for a mitral valve replacement in the following year [26]. In the field of urology, the first robot-assisted radical prostatectomy was performed in 2000 in Frankfurt, Germany. The first robot-assisted Anderson-Hynes pyeloplasty was performed in Austria in 2002, and the first robot-assisted radical cystectomy was performed in 2003 in Frankfurt [30]. There is a continuous technical evolution of the robotic platform. Currently, most installed systems access the patient’s body through multiple ports (for the camera and the surgical instruments). In 2018, the FDA approved the daVinci SP, a single-port system in which three fully wristed surgical arms and a camera reside within a single 25 mm port. Several studies evaluated this new system for radical prostatectomy in comparison to the multiport systems. There is evidence that single-port robot-assisted radical prostatectomy is a feasible alternative to the traditional multiport procedure with potential benefits in pain management and length of stay [20].

479

480

Digital Urology

During the last years, robotic surgical platforms such as the daVinci system have been installed worldwide, becoming part of the standard of care in common procedures such as radical prostatectomy [16, 31, 38]. Several studies evaluated the results of robotic surgery in comparison to the traditional open surgical approach. For example, a randomized clinical trial (evidence level 1) conducted in Australia found that robot-assisted radical prostatectomy and open radical retropubic prostatectomy yielded similar functional outcomes at 24 months. The authors concluded that clinicians and patients should view the benefits of a robotic approach as being largely related to its minimally invasive nature [11]. Similarly, a recent review comparing robot-assisted vs. open radical prostatectomy summarized that there is evidence in the literature showing improved perioperative outcomes in robot-assisted prostatectomy but no clear benefit on functional and oncological outcomes [12]. Other surgical procedures in urology besides prostatectomy where the robotic approach is successfully used include Anderson-Hynes pyeloplasty for treatment of uteropelvic junction obstruction, nephrectomy, adrenalectomy, and cystectomy [33, 45]. Performing surgery with a robotic system is significantly different from “classical” surgery. It is difficult to integrate robot-assisted surgery skills into urological training programs. Several high-quality commercially available simulators have been developed for the safe training of these skills. There is strong evidence in a randomized controlled trial that supports the usefulness of simulation training for robot-assisted surgery [15]. Consequently, there are growing calls to incorporate such simulators into the urological training curriculum to shorten the learning curve [30]. The European Association of Urology (EAU) Robotic Urology Section (ERUS) developed a training program for robot-assisted radical prostatectomy in 2015. This program includes theoretical training, live case observation, table assistance, laboratory exercises, and modular console training [36].

23.3.2 Radiosurgery

In the year 2001, the CyberKnife radio-surgical robotic system received approval from the FDA [45]. This system undergoes a continuous evolution; the principle of function is always the same: a linear accelerator mounted on a robotic manipulator delivers many

Robotics

independently targeted treatment beams with high precision under continuous X-ray guidance. If necessary, tracking of the therapy target is assisted by the placement of gold markers. For example, this allows correct and exact radiation placement on a moving target such as a lung tumor which naturally moves during respiration. Applications of CyberKnife include treatment of the brain, spine, lung, prostate, liver, head and neck, and other extracranial sites [22]. Non-surgical radiation therapy of the brain (e.g., treatment of brain metastases) is called stereotactic radiosurgery (SRS). When SRS is used to treat other sites, it is called stereotactic body radiotherapy (SBRT). In the field of urology, SBRT with CyberKnife is performed in patients with prostate cancer. A multi-center study on 309 patients found minimal toxicity and a favorable disease-free rate (superior to data from historical controls) [35]. Patients on anticoagulant/ antiplatelet medications (which is not rare in this age group) are at a high risk of bleeding following external beam radiation therapy for localized prostate cancer. A study on such patients found that the radiotherapy was well tolerated. High-grade bleeding toxicities were uncommon and resolved with time. The authors concluded that baseline anticoagulation usage should not be considered a contraindication to prostate SBRT [39]. The CyberKnife system can also be used to treat renal tumors. A prospective case-control study on 40 patients with renal tumors and an indication for nephrectomy and subsequent hemodialysis were treated with CyberKnife as a one-time outpatient procedure. The local tumor control after 9 months was 98%, the toxicity was low, and the renal function remained stable. Nephrectomy could be avoided in all cases [47]. Patients with advanced renal cell carcinoma often present with pulmonary metastases that are difficult to treat. In a study on 50 patients with pulmonary metastases robotic radiosurgery with CyberKnife was tested and was found to be a safe and highly effective treatment option in this patient group [42]. Renal cell carcinoma is a frequent source of brain metastases. In another study on 66 patients with a total of 207 brain lesions CyberKnife was used for treatment (SRS). Local Control rates were 84%, 94%, and 88% for SRS only, for neurosurgery and SRS, and for whole-brain radiotherapy and additional SRS, respectively. The authors concluded that stereotactic

481

482

Digital Urology

radiosurgery is a safe and effective treatment option in patients with brain metastases from renal cell carcinoma [21]. Taken together, robotic radiosurgery is a promising additional therapy option for a patient with urological tumors, for the treatment of primary tumors and metastases.

23.4 Artificial Intelligence

Artificial intelligence (AI) and machine learning (ML) are applied to numerous areas in the modern world. These approaches are useful to identify patterns and connections in complex data, and of course, there are many possible benefits when using AI in medicine. For example, AI can help to extract meaningful information from medical databases; furthermore, AI can assist in diagnosis, outcome assessment, and therapeutic decision-making. Several urological applications of AI in this field are presented in the following text. Medical information for clinicians and scientists is available in a multitude of databases that contain data about scientific literature, clinical trials, patents, biological data (genomics, transcriptomics, proteomics, etc.), and more. It is often difficult to extract meaningful, comprehensive, and “targeted” results from these sources manually. There is an urgent need for more sophisticated queries, and several AI projects try to help in overcoming this information overload. A recent review demonstrates that by applying natural language processing and ML algorithms, validated and optimized AI leads to a speedier, more personalized, efficient, and focused information search on the topic of urological cancer compared with traditional methods. The authors show the use of the AI-based Dimensions database (https://www.dimensions.ai) that integrates research information from numerous different sources and presents search results as an interactive visualization. The dashboard is updated as the user selects specific data of interest for a more focused analysis [48]. These kinds of tools will become more and more important to assist in clinical routine and research. Some peer-reviewed publications employ AI-based tools to support the development of uro-oncological guidelines [34]. Even high-volume databases do not cover all patient characteristics and drawn results may be limited. A new viable

Artificial Intelligence

automated solution is ML based on deep neural networks applied to natural language processing (NLP), extracting detailed information from narratively written (e.g., pathologic radical prostatectomy) electronic health records (EHRs). A study on a prostatectomy database used state-of-the-art NLP techniques to train an industrystandard language model for pathologic EHRs by transfer learning. The cumulative agreement between the automatically extracted data and the manually curated gold-standard data was 91%. The results indicate that precise and efficient data management from narrative documentation for clinical research is possible using AI methods [27]. Not only clinicians and researchers can use an AI approach. Also, patients can retrieve meaningful information from AI systems that can, for example, help them to choose a specific therapy. Data from a large clinical registry of men diagnosed with prostate cancer and ML methods were used to develop and validate a model that informs patients of treatment options chosen by men with similar clinical and histopathological characteristics. This web-based platform (called askMUSIC) generates accurate predictions for most prostate cancer treatments. The AI model exhibited excellent discrimination between treatment options (active surveillance, prostatectomy, radiotherapy, androgen deprivation, etc.) in the validation cohort. This tool can help patients to understand decisions made by similar men and therefore, it can help them to make the optimal decision for their case [4]. Another emerging field of application for AI in medicine is radiomics. ML means pattern recognition. Imaging in radiology delivers a multitude of data that contain meaningful patterns where AI can assist in a valid evaluation. Radiomic analysis is defined as the computational extraction of features from radiographic images for quantitatively characterizing disease patterns [1]. An example of radiomics in urology is a study to evaluate the performance of MRI (magnetic resonance imaging)–based radiomic features in identifying the presence or absence of clinically significant prostate cancer in patients on active surveillance (AS). As a result, radiomic features appeared to be more relevant for the identification of clinically relevant prostate cancer than the routine assessment by the radiologist. If these findings can be confirmed in subsequent studies, radiomics can help to identify which patients are candidates for AS,

483

484

Digital Urology

and could allow noninvasive monitoring of disease grade and stage for patients that are already on AS [1]. In a study on 721 men with metastatic castration-resistant prostate cancer, the prognostic value of the automated Bone Scan Index (aBSI) was evaluated. This is a quantitative assessment of bone scan data that represents the tumor burden of the skeleton. Artificial neural networks are used to detect and classify metastatic hot spots in bone scans, and commercial systems are available. The aBSI was significantly associated with overall survival and remained independently associated with overall survival in a multivariable survival model [2]. Another example of radiomics in urology is a recent meta-analysis of eight studies with a total of 860 patients that applied ML techniques to the prediction of muscle-invasive bladder cancer (MIBC) based on preoperative imaging with MRI or CT (computed tomography). The summary estimates for sensitivity and specificity in predicting MIBC were 82% and 81%, respectively. This indicates a high diagnostic performance of radiomics in the staging of bladder cancer [23]. The next step in utilizing AI for an optimized individual risk assessment is the combination of imaging biomarkers (radiomics) with biological biomarkers. This approach was used to create a “radiogenomics” signature for predicting the progression-free interval in bladder cancer patients. In a study on 62 patients with urothelial carcinoma of the bladder, radiomics features, clinical data, and gene expression profiles were combined to develop prognostic signatures with excellent predictive ability [29]. In addition to the analysis of radiologic images, image data from histopathological samples can also be used for evaluation with ML methods. Tissue Phenomics is the discipline of mining tissue images to identify patterns that are related to clinical outcomes providing potential prognostic and predictive value. In a study on tissue samples (immunohistochemical staining) from prostate cancer patients with prostatectomy, this approach was applied to automatically identify tissue-based biomarkers with significant prognostic value. Recurrence prediction yielded accuracies up to 83% thereby clearly outperforming prediction based on the Gleason score [18]. Another study used an AI algorithm called the multiview boosting method to detect prostate cancer on digitized pathology images with different resolutions. The tissue specimen images were segmented using intensity- and texture-based features.

Artificial Intelligence

Using the segmentation results, a number of morphological features from lumens and epithelial nuclei were computed to characterize the tissues. Tissue characteristics from differing resolutions were combined to achieve accurate cancer detection (the area under the curve was ≥0.97 in various settings) [25]. Another promising application of AI-based image analysis besides radiomics and digital pathology is the analysis of optical tissue features, maybe in the future performed directly during patient examination or surgery. A study evaluated shortwave infrared Raman spectroscopy to differentiate between benign and malignant renal tissue (clear-cell renal cell carcinoma). The AIbased classification of the tissue samples had very good accuracy (area under curve = 0.94). This approach has the potential to be used as a diagnostic tool and for intraoperative guidance during partial nephrectomy [17]. Albeit many applications of ML in urology are in uro-oncology, the application of AI is also being investigated in benign urological diseases. One example is urolithiasis, where shockwave lithotripsy is a widely accepted treatment option. A study used five commonly used ML models to predict the success of lithotripsy, based on the three-dimensional texture analysis features of each kidney stone, together with clinical features (body mass index, initial stone size, and skin-to-stone distance). The best model had good predictive power for lithotripsy success (area under curve value 0.85) and clearly outperformed the predictive power of the clinical features alone [32]. With the emerging role of robotic surgery, there is not only a need for valid evaluation of surgical skills and performance but also high-quality surgical footage from the robot than can be applied to evaluate surgical performance automatically through AI techniques. For example, a study developed and validated an automated assessment of surgical performance (AASP) system for an objective and computerized assessment of pelvic lymph node dissection as an integral part of robot-assisted radical cystectomy. The quality of lymph node clearance was assessed based on the features derived from the video footage. The automated scores were compared to the validated pelvic lymphadenectomy appropriateness and completion evaluation (PLACE) scoring rated by a panel of expert surgeons. The accuracy of predicting the expert-based PLACE scores was 83%

485

486

Digital Urology

[5]. Therefore, AI methods could be a promising tool for objective assessment of surgical skills in the future. There is a big variety of AI applications in urology, including diagnosis, outcome prediction, treatment planning, and surgical skill assessment, but usually as research projects [8, 9]. Today, real-world implementation of AI technologies into existing clinical workflows is not yet a reality, but AI has the potential to support more accurate clinical decision-making. It can be anticipated that the fields with the earliest integration of AI are those with a strong image-based or visual component that is amenable to automated analysis or diagnostic prediction, including radiology and pathology [19].

23.5 Conclusion

Digitalization is rapidly and successfully evolving in the field of urology. Telemedicine will be a standard part of urological care in the future, in form of teleconsultations and the use of apps and wearables to monitor health status and body functions, e.g., during long-term therapy. Robot-assisted surgery and robotic radiosurgery have already been established as standard options for several indications. AI applications are very promising to assist in diagnosis and therapeutic decision-making (e.g., selection of the optimal therapy for an individual patient), and also to improve knowledgemanagement for research and evidence-based patient consultation.

References

1. Algohary, A., Viswanath, S., Shiradkar, R., Ghose, S., Pahwa, S., Moses, D., Jambor, I., Shnier, R., Bohm, M., Haynes, A. M., Brenner, P., Delprado, W., Thompson, J., Pulbrock, M., Purysko, A. S., Verma, S., Ponsky, L., Stricker, P., Madabhushi, A. (2018). Radiomic features on MRI enable risk categorization of prostate cancer patients on active surveillance: preliminary findings. J. Magn. Reson. Imaging, 48, 818–828. 2. Armstrong, A. J., Anand, A., Edenbrandt, L., Bondesson, E., Bjartell, A., Widmark, A., Sternberg, C. N., Pili, R., Tuvesson, H., Nordle, O., Carducci, M. A., Morris, M. J. (2018). Phase 3 assessment of the automated bone scan index as a prognostic imaging biomarker of overall survival in men with metastatic castration-resistant prostate cancer: a secondary analysis of a randomized clinical trial. JAMA Oncol., 4, 944–951.

References

3. Asklund, I., Nystrom, E., Sjostrom, M., Umefjord, G., Stenlund, H., Samuelsson, E. (2017). Mobile app for treatment of stress urinary incontinence: a randomized controlled trial. Neurourol. Urodyn., 36, 1369–1376. 4. Auffenberg, G. B., Ghani, K. R., Ramani, S., Usoro, E., Denton, B., Rogers, C., Stockton, B., Miller, D. C., Singh, K., Michigan Urological Surgery Improvement Collaborative. (2019). askMUSIC: leveraging a clinical registry to develop a new machine learning model to inform patients of prostate cancer treatments chosen by similar men. Eur. Urol., 75, 901–907.

5. Baghdadi, A., Hussein, A. A., Ahmed, Y., Cavuoto, L. A., Guru, K. A. (2019). A computer vision technique for automated assessment of surgical performance using surgeons’ console-feed videos. Int. J. Comput. Assist. Radiol. Surg., 14, 697–707. 6. Beg, M. S., Gupta, A., Stewart, T., Rethorst, C. D. (2017). Promise of wearable physical activity monitors in oncology practice. J. Oncol. Pract., 13, 82–89.

7. Brimley, S., Natale, C., Dick, B., Pastuszak, A., Khera, M., Baum, N., Raheem, O. A. (2021). The emerging critical role of telemedicine in the urology clinic: a practical guide. Sex. Med. Rev., 9, 289–295. 8. Chen, A. B., Haque, T., Roberts, S., Rambhatla, S., Cacciamani, G., Dasgupta, P., Hung, A. J. (2022). Artificial intelligence applications in urology: reporting standards to achieve fluency for urologists. Urol. Clin. North Am., 49, 65–117. 9. Chen, J., Remulla, D., Nguyen, J. H., Aastha, D., Liu, Y., Dasgupta, P., Hung, A. J. (2019). Current status of artificial intelligence applications in urology and their potential to influence clinical practice. BJU Int., 124, 567–577. 10. Connor, M. J., Miah, S., Edison, M. A., Brittain, J., Smith, M. K., Hanna, M., El-Husseiny, T., Dasgupta, R. (2019). Clinical, fiscal and environmental benefits of a specialist-led virtual ureteric colic clinic: a prospective study. BJU Int., 124, 1034–1039.

11. Coughlin, G. D., Yaxley, J. W., Chambers, S. K., Occhipinti, S., Samaratunga, H., Zajdlewicz, L., Teloken, P., Dunglison, N., Williams, S., Lavin, M. F., Gardiner, R. A. (2018). Robot-assisted laparoscopic prostatectomy versus open radical retropubic prostatectomy: 24-month outcomes from a randomised controlled study. Lancet Oncol., 19, 1051–1060. 12. Dell’Oglio, P., Mottrie, A., Mazzone, E. (2020). Robot-assisted radical prostatectomy vs. open radical prostatectomy: latest evidences on perioperative, functional and oncological outcomes. Curr. Opin. Urol., 30, 73–78.

487

488

Digital Urology

13. Ellimoottil, C., Skolarus, T., Gettman, M., Boxer, R., Kutikov, A., Lee, B. R., Shelton, J., Morgan, T. (2016). Telemedicine in urology: state of the art. Urology, 94, 10–16. 14. Eun, S. J., Lee, J. Y., Jung, H., Kim, K. H. (2021). Personalized urination activity management based on an intelligent system using a wearable device. Int. Neurourol. J., 25, 229–235.

15. Feifer, A., Al-Ammari, A., Kovac, E., Delisle, J., Carrier, S., Anidjar, M. (2011). Randomized controlled trial of virtual reality and hybrid simulation for robotic surgical training. BJU Int., 108, 1652–1656; discussion 1657. 16. George, E. I., Brand, T. C., LaPorta, A., Marescaux, J., Satava, R. M. (2018). Origins of robotic surgery: from skepticism to standard of care. JSLS, 22, e2018.00039. 17. Haifler, M., Pence, I., Sun, Y., Kutikov, A., Uzzo, R. G., Mahadevan-Jansen, A., Patil, C. A. (2018). Discrimination of malignant and normal kidney tissue with short wave infrared dispersive Raman spectroscopy. J. Biophotonics, 11, e201700188.

18. Harder, N., Athelogou, M., Hessel, H., Brieu, N., Yigitsoy, M., Zimmermann, J., Baatz, M., Buchner, A., Stief, C. G., Kirchner, T., Binnig, G., Schmidt, G., Huss, R. (2018). Tissue Phenomics for prognostic biomarker discovery in low- and intermediate-risk prostate cancer. Sci. Rep., 8, 4470. 19. He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., Zhang, K. (2019). The practical implementation of artificial intelligence technologies in medicine. Nat. Med., 25, 30–36.

20. Hinojosa-Gonzalez, D. E., Roblesgil-Medrano, A., Torres-Martinez, M., Alanis-Garza, C., Estrada-Mendizabal, R. J., Gonzalez-Bonilla, E. A., Flores-Villalba, E., Olvera-Posada, D. (2022). Single-port versus multiport robotic-assisted radical prostatectomy: a systematic review and meta-analysis on the da Vinci SP platform. Prostate, 82, 405–414. 21. Ippen, F. M., Mahadevan, A., Wong, E. T., Uhlmann, E. J., Sengupta, S., Kasper, E. M. (2015). Stereotactic radiosurgery for renal cancer brain metastasis: prognostic factors and the role of whole-brain radiation and surgical resection. J. Oncol., 2015, 636918. 22. Kilby, W., Dooley, J. R., Kuduvalli, G., Sayeh, S., Maurer, C. R., Jr. (2010). The CyberKnife robotic radiosurgery system in 2010. Technol. Cancer Res. Treat., 9, 433–452.

23. Kozikowski, M., Suarez-Ibarrola, R., Osiecki, R., Bilski, K., Gratzke, C., Shariat, S. F., Miernik, A., Dobruch, J. (2022). Role of radiomics in the prediction of muscle-invasive bladder cancer: a systematic review and meta-analysis. Eur. Urol. Focus, 8, 728–738.

References

24. Kumar, R., Hemal, A. K. (2005). Emerging role of robotics in urology. J. Minim. Access Surg., 1, 202–210.

25. Kwak, J. T., Hewitt, S. M. (2017). Multiview boosting digital pathology analysis of prostate cancer. Comput. Methods Programs Biomed., 142, 91–99. 26. Lane, T. (2018). A short history of robotic surgery. Ann. R. Coll. Surg. Engl., 100, 5–7.

27. Leyh-Bannurah, S. R., Tian, Z., Karakiewicz, P. I., Wolffgang, U., Sauter, G., Fisch, M., Pehrke, D., Huland, H., Graefen, M., Budaus, L. (2018). Deep learning for natural language processing in urology: state-of-the-art automated extraction of detailed pathologic prostate cancer data from narratively written electronic health records. JCO Clin. Cancer Inform., 2, 1–9. 28. Liang, W., Guan, W., Chen, R., Wang, W., Li, J., Xu, K., Li, C., Ai, Q., Lu, W., Liang, H., Li, S., He, J. (2020). Cancer patients in SARS-CoV-2 infection: a nationwide analysis in China. Lancet Oncol., 21, 335–337.

29. Lin, P., Wen, D. Y., Chen, L., Li, X., Li, S. H., Yan, H. B., He, R. Q., Chen, G., He, Y., Yang, H. (2020). A radiogenomics signature for predicting the clinical outcome of bladder urothelial carcinoma. Eur. Radiol., 30, 547–557. 30. MacCraith, E., Forde, J. C., Davis, N. F. (2019). Robotic simulation training for urological trainees: a comprehensive review on cost, merits and challenges. J. Robot. Surg., 13, 371–377.

31. Makarov, D. V., Yu, J. B., Desai, R. A., Penson, D. F., Gross, C. P. (2011). The association between diffusion of the surgical robot and radical prostatectomy rates. Med. Care, 49, 333–339. 32. Mannil, M., von Spiczak, J., Hermanns, T., Poyet, C., Alkadhi, H., Fankhauser, C. D. (2018). Three-dimensional texture analysis with machine learning provides incremental predictive information for successful shock wave lithotripsy in patients with kidney stones. J. Urol., 200, 829–836.

33. Mastroianni, R., Ferriero, M., Tuderti, G., Anceschi, U., Bove, A. M., Brassetti, A., Misuraca, L., Zampa, A., Torregiani, G., Ghiani, E., Giannarelli, D., Guaglianone, S., Gallucci, M., Simone, G. (2022). Open radical cystectomy versus robot-assisted radical cystectomy with intracorporeal urinary diversion: early outcomes of a single center randomised controlled trial. J. Urol., 207, 982–992.

489

490

Digital Urology

34. McDonald, S., Elliott, J. H., Green, S., Turner, T. (2019). Towards a new model for producing evidence-based guidelines: a qualitative study of current approaches and opportunities for innovation among Australian guideline developers. F1000Res, 8, 956.

35. Meier, R. M., Bloch, D. A., Cotrutz, C., Beckman, A. C., Henning, G. T., Woodhouse, S. A., Williamson, S. K., Mohideen, N., Dombrowski, J. J., Hong, R. L., Brachman, D. G., Linson, P. W., Kaplan, I. D. (2018). Multicenter trial of stereotactic body radiation therapy for low- and intermediate-risk prostate cancer: survival and toxicity endpoints. Int. J. Radiat. Oncol. Biol. Phys., 102, 296–303. 36. Mottrie, A., Novara, G., van der Poel, H., Dasgupta, P., Montorsi, F., Gandaglia, G. (2016). The European association of urology robotic training curriculum: an update. Eur. Urol. Focus, 2, 105–108.

37. Novara, G., Checcucci, E., Crestani, A., Abrate, A., Esperto, F., Pavan, N., De Nunzio, C., Galfano, A., Giannarini, G., Gregori, A., Liguori, G., Bartoletti, R., Porpiglia, F., Scarpa, R. M., Simonato, A., Trombetta, C., Tubaro, A., Ficarra, V., Research Urology Network. (2020). Telehealth in urology: a systematic review of the literature. how much can telemedicine be useful during and after the COVID-19 pandemic? Eur. Urol., 78, 786– 811. 38. Oberlin, D. T., Flum, A. S., Lai, J. D., Meeks, J. J. (2016). The effect of minimally invasive prostatectomy on practice patterns of American urologists. Urol. Oncol., 34, 255 e1–255 e5.

39. Pepin, A., Shah, S., Pernia, M., Lei, S., Ayoob, M., Danner, M., Yung, T., Collins, B. T., Suy, S., Aghdam, N., Collins, S. P. (2021). Bleeding risk following stereotactic body radiation therapy for localized prostate cancer in men on baseline anticoagulant or antiplatelet therapy. Front. Oncol., 11, 722852.

40. Rastogi, R., Martinez, K. A., Gupta, N., Rood, M., Rothberg, M. B. (2020). Management of urinary tract infections in direct to consumer telemedicine. J. Gen. Intern. Med., 35, 643–648. 41. Rodler, S., Apfelbeck, M., Schulz, G. B., Ivanova, T., Buchner, A., Staehler, M., Heinemann, V., Stief, C., Casuscelli, J. (2020). Telehealth in urooncology beyond the pandemic: toll or lifesaver? Eur. Urol. Focus, 6, 1097–1103. 42. Rodler, S., Gotz, M., Mumm, J. N., Buchner, A., Graser, A., Casuscelli, J., Stief, C., Furweger, C., Muacevic, A., Staehler, M. (2022). Image-guided robotic radiosurgery for the treatment of lung metastases of renal cell carcinoma-a retrospective, single center analysis. Cancers (Basel), 14, 356.

References

43. Rodriguez Socarras, M., Loeb, S., Teoh, J. Y., Ribal, M. J., Bloemberg, J., Catto, J., N’Dow, J., Van Poppel, H., Gomez Rivas, J. (2020). Telemedicine and smart working: recommendations of the European association of urology. Eur. Urol., 78, 812–819.

44. Safir, I. J., Gabale, S., David, S. A., Huang, J. H., Gerhard, R. S., Pearl, J., Lorentz, C. A., Baumgardner, J., Filson, C. P., Issa, M. M. (2016). Implementation of a tele-urology program for outpatient hematuria referrals: initial results and patient satisfaction. Urology, 97, 33–39. 45. Shah, J., Vyas, A., Vyas, D. (2014). The history of robotics in surgical specialties. Am. J. Robot Surg., 1, 12–20.

46. Slade, A. D., Cardinal, J. R., Martin, C. R., Presson, A. P., Allen, C. D., Lowrance, W. T., Dechet, C. B., O’Neil, B. B. (2021). Feasibility of wearable activity trackers in cystectomy patients to monitor for postoperative complications. Curr. Urol., 15, 209–213.

47. Staehler, M., Bader, M., Schlenker, B., Casuscelli, J., Karl, A., Roosen, A., Stief, C. G., Bex, A., Wowra, B., Muacevic, A. (2015). Single fraction radiosurgery for the treatment of renal tumors. J. Urol., 193, 771–775. 48. Stenzl, A., Sternberg, C. N., Ghith, J., Serfass, L., Schijvenaars, B. J. A., Sboner, A. (2022). Application of artificial intelligence to overcome clinical information overload in urological cancer. BJU Int., 130, 291–300.

491

Chapter 24

Digitalization in Anesthesiology and Intensive Care

Philipp Simon,a Ludwig Christian Hinske,b,c and Axel Hellera aClinic

for Anesthesiology and Operative Intensive Care Medicine, University Hospital Augsburg, Augsburg, Germany bInstitute for Digital Medicine, University Hospital Augsburg, Augsburg, Germany cClinic for Anesthesiology, Ludwig-Maximilians-University Hospital, Munich, Germany [email protected]

24.1 Introduction Knowledge management in anesthesiology, intensive care, and emergency medicine has received a considerable boost in the past 15 years through the introduction of automated measurement data transfer into electronic protocols and patient records. On the one hand, this provides an enormous opportunity to make data available in larger networks and cooperation, and on the other hand, to make predictions based on corresponding analyses that are not readily Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

494

Digitalization in Anesthesiology and Intensive Care

available to physicians due to their complexity. One example of this is telemetric applications in intensive care and emergency medicine, where intensive care units at lower levels of care receive decision support from maximum care providers. In emergency medicine, physician consultation services can be brought to the emergency scene when an emergency physician is not physically available there. All of these applications have led to quality improvements in patient care. In parallel, cost savings have even been demonstrated in some scenarios. One challenge remains the structuring and annotation of the data already during input in order to ensure the best possible evaluability and further processing here.

24.2 The Perioperative Process: Digitalization in Anesthesiology and Critical Care

The perioperative process defines all medical interventions before (pre-), during (intra-), and after (postoperative) a surgical procedure [18]. In the preoperative setting, the patient is evaluated by the surgeon and the anesthetist, leading to an initial estimate. Based on this estimate, the patient will either be scheduled for an operative intervention or requires additional medical attention first. After the operation, the patient usually requires medical attention in a Post-Anesthesia Care Unit (PACU), before being discharged (in case of ambulatory care), admitted to a regular ward, or in case of necessity of continuous medical attention or invasive procedures to an intensive care unit (ICU). Therefore, anesthesiology and critical care medicine are tightly connected fields that share important features regarding the importance of high-resolution data evaluation during the care process. In both fields, patients are often subject to extreme physiological situations that need medical intervention in order to maintain the patients’ homeostasis to the best possible degree. Therefore, tight and continuous monitoring of vital parameters as well as additional relevant variables of the perioperative care process is a central necessity to detect hemodynamic or respiratory instabilities and react accordingly. Such events can occur at any time during the perioperative process, due to the patients’ health conditions, intraoperative events such as acute blood loss, clamping of vessels,

Digital Anesthesiology and Intensive Care in Research and Education

triggering of physiological reflexes, or homeostatic imbalances. Depending on the type of surgery, the indication, comorbidities, and intraoperative events, patients might require prolonged attention. It was for this reason that the first ICU was established, a highly successful concept that immediately spread to other fields in medicine, specializing in patients where the physiological process is severely impaired and requires continuous attention [4]. Implementation of multiparameter early warning scores (MEWS) on normal care units, combining a range of physiologic parameters into a summated score, demonstrated a significant reduction in the incidence of cardiac arrest [23]. However, only 68% protocol compliance was achieved with manual MEWS paper curves [13]. One potential solution to this problem is MEWS-based electronic automated vital sign monitoring systems [6], which have been shown to increase survival during ward emergencies and reduce the time required to measure and record vital signs [3]. Follow-up studies with connection to paging devices also found reductions in both cardiac arrest and unplanned intensive care admissions in a high-risk surgical cohort. When using these systems, patient deterioration is more frequently detected by monitor alarms than by staff observation, and there is increased availability of physiologic data when MET arrives [6].

24.3 Digital Anesthesiology and Intensive Care in Research and Education

The digitalization process in anesthesiology and intensive care medicine also creates new opportunities in research and teaching. AIMS (anesthesia information management systems) or also known as PDMS (patient data management systems) are increasingly finding their way into anesthesiology and intensive care medicine. Increasing networking and digitization enables the seamless recording of all accruing data and thus comprehensive documentation in everyday clinical practice. In addition, ecological aspects are playing an increasingly important role and are leading to initiatives such as the “paperless hospital.” Although such digital PDMS originated in anesthesiology and intensive care, they are growing out of their infancy and are just beginning to triumph throughout the hospital.

495

496

Digitalization in Anesthesiology and Intensive Care

Provided that they are compatible, a new digital world of medical data is emerging, enabling a new form of health services research. Digitalization is giving simulation unimagined opportunities for training – and even educating – medical personnel. That is why simulation centers are widely used in anesthesia, but less so in critical care medicine. Various forms of simulation trainers exist from mannequins for training intubation to entire simulation operating rooms. This has become an integral part of education and training as well as an indispensable part of student teaching [7]. In this context, digital emergency checklists were also developed and made available free of charge by the German Federal Association of Anesthesiologists [19]. Another area of research is large patient databases, which are not only used to gain immediate insights into the course of illnesses or to make forecasts [21]. Rather, these data can also be used to simulate measures that cannot be carried out on the patient himself [2].

24.4 Fair Anesthesia: Data Sharing and Open Science

Figure 24.1 Overview of the three most popular open access ICU databases. eICU contains the largest number of patients and admissions, covering more than 250 hospitals across the USA. However, both MIMIC and Amsterdam UMCDB, even though single-hospital databases, contain more high-resolution data.

Fair Anesthesia: Data Sharing and Open Science

In the last couple of years, science has made a giant leap toward reproducibility, which is tightly linked to data availability. One of the earliest efforts to make data available to the public to leverage scientific collaboration is MIMIC: the Medical Information Mart for Intensive Care [9]. MIMIC is the result of a collaboration between academia (Massachusetts Institute of Technology), industry (Philips Medical Systems), and clinical medicine (Beth Israel Deaconess Medical Center) that was founded in 2003. Its vision was the utilization of data that was generated during patient care. As such, its initial version contained three types of data: clinical data retrieved from charts, high-resolution physiological data, such as arterial blood pressure waveform data, and survival status obtained from Social Security Administration Death Master Files (Critical Data 2016). Importantly, after anonymization, these data were made available to the public, such that biomedical research could advance to support patient care in this highly critical setting [9]. MIMIC is now available in its 4th version, and has added a significant number of data types, including imaging and emergency department data prior to ICU admission [8, 24]. However, MIMIC is more than just its data. The MIT Critical Data team established the concept of ICU datathons, a combination of the word data and marathon [16]. This format is characterized by interdisciplinary teams of both experts and enthusiasts from the medical as well as the data science world that get together for a weekend and work on relevant medical questions, competing with each other for the best project, analysis, and result [1]. This concept has become extremely popular and has become a regular event for the European Society of Intensive Care Medicine. MIMIC has also been the basis for important work on artificial intelligence in the ICU. Based on MIMIC’s success, the team around MIMIC was then able to convince Philip’s Medical Systems to share their database of ICU monitoring data as well. This database called eICU includes data from more than 250 hospitals across the United States of America and is also freely accessible [20]. MIMIC and eICU have greatly contributed to the advancement of knowledge in the field of intensive care. Komorowski and his colleagues drew attention to the field of reinforcement learning in intensive care by demonstrating that this type of machine learning might be used to support hemodynamic therapy decision-making [11]. MIMIC

497

498

Digitalization in Anesthesiology and Intensive Care

was also part of the basis for the development of the Hypotension Prediction Index© [5]. Even though MIMIC and eICU provide a plethora of exciting digital data for researchers interested in intensive care medicine, they cannot sufficiently represent patients and clinics from around the world. Recently, European institutions have started to follow a similar direction. A team from the University Medical Center Amsterdam was able to release a similar database despite the strict regulations that the General Data Protection Regulation imposes [22]. For perioperative data, there is currently only one database openly available. VitalDB contains 6388 surgical patients, including high-quality biosignal waveform data as well as more than 60 surgery-related clinical variables [25]. As a centralized approach, in 2008 the Multicenter Perioperative Outcomes Group (MPOG) was founded as a consortium of now more than 100 collaborators representing 51 hospitals mainly from the United States. The idea is to collect EHR and AIMS data to support research in the perioperative setting. Data access can be requested for a well-defined subset of the data tailored to answer a specific research question. A similar approach is followed by National Anesthesia Outcomes Registry (NACOR). NACOR started collecting data starting in 2010 with a focus on benchmarking and quality control. This is especially valuable since randomized controlled trials are especially challenging in the perioperative setting, because a) many times deviations from the standard protocol might raise ethical concerns and b) due to high-security standards and consequently low incidence rates, an unfeasible number of patients would need to be recruited to reliably prove intervention effects [12].

24.5 Clinical Decision Support

In order to promote digitization in medicine, the German government initiated the medical informatics initiative in 2018 with the vision to link research and healthcare in a more targeted manner. In the end, five consortia with different concepts and a wide variety of projects emerged nationwide with the goal of advancing digitization in research and care. Data integration centers were established at the university level to link the wealth of data generated daily in patient

Clinical Decision Support

care, standardize it, and make it usable for research and care, with the aim of enabling more personalized patient care in all clinical areas. One example from the first funding period is the development of an app that provides real-time support to improve the detection of acute lung failure in intensive care units so that therapy can be initiated more quickly [15]. In this context, the current technical possibilities of digitalization are used to recognize acute disease states more quickly and to initiate adequate therapy. Another development in the field of clinical decision support is the digital linking of numerous measurement parameters in hightech areas such as the operating room or in intensive care, where many vital functions are monitored in real time. The focus here is on making these numerous measured values from a wide variety of organ functions available to the treating physician in real time, bundled on a device so that a multi-functional monitoring platform is created that can support faster decision-making processes through decision support based on algorithms to be developed using machine learning [15]. Ideally, this will allow potential problems to be identified sooner in the future, before any potential deterioration in the patient’s condition. The German healthcare system faces the challenge of ensuring high-quality, nationwide healthcare in the face of an increasing shortage of medical and nursing staff in the future. As part of digitization, particularly in intensive care medicine, telemedicine cooperation structures can help make expert knowledge available around the clock in underserved regions and improve treatment quality cost-effectively and sustainably. In the fields of teleintensive medicine and tele-emergency medicine, positive results in the care of critically ill patients have been demonstrated by numerous international studies and also Germany-wide projects. In anesthesia, supplementary tele-consultations offer the possibility of providing specialist supervision from preoperative risk evaluation to post-anesthesiologic care as needed and without delay. In pain management, telemedicine can also help support timely and individualized care [14]. In addition, the establishment of an “Anesthesiology Control Tower” represents further achievement in the field of perioperative clinical decision support. Especially in anesthesiology workplaces

499

500

Digitalization in Anesthesiology and Intensive Care

with which a wide variety of data can be bundled in real time and made available to a special team [17], beginning with patient-related factors and vital parameters recorded during the operation, as well as information about the operation and static and dynamic risk factors such a system leverages clinical care. In this way, especially in critical situations, a small number of practitioners can be supported in decision-making at the operating table by increasingly specialized teams. This concept, however, is relatively new and requires further research and evaluation before it possibly becomes routine in the future and finds its way into everyday life [10].

Figure 24.2 LCH: Decision support concept of the anesthesiology control tower. Modified after Ref. [10].

24.6 Perspective and Vision Digitization should be seen as a great opportunity in anesthesiology and intensive care medicine, enabling the practitioner to make decisions in less time and also to adapt to the therapy more quickly by recognizing changes. Especially in anesthesiology and intensive care medicine, the abundance of data generated every second by monitoring patients provides a very good basis for achieving automation with the latest methods of informatics in order to 1. enable faster detection of changes in the process as well as the occurrence of acute situations, 2. provide the practitioner with real-time decision support, 3. offer a more comprehensive basis for therapy decisions, and 4. lead to a shortening of decision-making

References

processes. In addition, digitization is an opportunity to simplify and bundle necessary documentation. As a further perspective, this paves the way for further specification and more patient-adapted individual decisions in therapies (ventilation, medication, etc.) based on patient-specific data in the operating room as well as in intensive care.

References

1. Aboab, J., Celi, L. A., Charlton, P., Feng, M., Ghassemi, M. Dominic, C., et al. (2016). A ‘Datathon’ model to support cross-disciplinary collaboration. Science Translational Medicine 8(333): 333ps8. 2. Bartenschlager, C. C., Brunner, J. O., et al. (2022). Evaluation of scorebased approaches for ex post triage in intensive care units during the COVID-19 pandemic: a simulation-based analysis. Notfall & Rettungsmedizin https://europepmc.org/articles/pmc9073506/ bin/10049_2022_1035_moesm1_esm.docx (Accessed June 13, 2022). 3. Bellomo, R., Ackerman, M., Bailey, M., Beale, R., et al. (2012). A controlled trial of electronic automated advisory vital signs monitoring in general hospital wards. Critical Care Medicine 40(8): 2349–2361.

4. Berthelsen, P. G., Cronqvist, M. (2003). The first intensive care unit in the world: copenhagen 1953. Acta Anaesthesiologica Scandinavica 47(10): 1190–1195.

5. Hatib, F., Jian, Z., Buddi, S., Lee, C., Settels, J., Sibert, K., et al. (2018). Machine-learning algorithm to predict hypotension based on highfidelity arterial pressure waveform analysis. Anesthesiology 129(4): 663–674. 6. Heller, A. R., Sören, T. M., Lauterwald, B., Reeps, C., et al. (2020). Detection of deteriorating patients on surgical wards outside the ICU by an automated MEWS-based early warning system with paging functionality. Annals of Surgery 271(1): 100–105.

7. Hempel, C., Turton, E., Hasheminejad, E., Bevilacqua, C., et al. (2020). Impact of simulator-based training on acquisition of transthoracic echocardiography skills in medical students. Annals of Cardiac Anaesthesia 23(3): 293–297. 8. Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., et al. (2019). MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data 6(1): 317.

501

502

Digitalization in Anesthesiology and Intensive Care

9. Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L. H., Feng, M., et al. (2016). MIMIC-III, a freely accessible critical care database. Scientific Data 3(May): 160035.

10. King, C. R., Abraham, J., Kannampallil, T. G., Fritz, B. A., et al. (2019). Protocol for the effectiveness of an anesthesiology control tower system in improving perioperative quality metrics and clinical outcomes: the tectonics randomized, pragmatic trial. F1000Research 8. https://doi.org/10.12688/f1000research.21016.1 (Accessed June 13, 2022). 11. Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., Faisal, A. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine 24(11): 1716–1720.

12. Liem, V. G. B., Hoeks, S. E., van Lier, F., de Graaff, J. (2018). What we can learn from Big Data about factors influencing perioperative outcome. Current Opinion in Anaesthesiology 31(6): 723–731. 13. Ludikhuize, J., Borgert, M., Binnekade, J., Subbe, C., Dongelmans, D., Goossens, A. (2014). Standardized measurement of the modified early warning score results in enhanced implementation of a rapid response system: a quasi-experimental study. Resuscitation 85(5): 676–682.

14. Marx, G., Dusch, M., Czaplik, M., Balzer, F., J., et al. (2019). Telemedicine in the four pillars of anaesthesiology position paper of the German Society of Anaesthesiology and Intensive Care Medicine (DGAI) and German Society of Telemedicine (DG Telemed). Anasthesiologie & Intensivmedizin 60: 191–207. 15. Marx, G., Bickenbach, J., Fritsch, S. J., Kunze, J. B., Maassen, O., Deffge, S., et al. (2021). Algorithmic surveillance of ICU patients with acute respiratory distress syndrome (ASIC): protocol for a multicentre stepped-wedge cluster randomised quality improvement strategy. BMJ Open 11(4): e045589.

16. MIT Critical Data. (2016). Secondary Analysis of Electronic Health Records. Retrieved from https://library.oapen.org/bitstream/ handle/20.500.12657/28012/1001985.pdf?sequence=1 (Accessed June 13, 2022). 17. Murray-Torres, T. M., Wallace, F., Bollini, M., Avidan, M. S., Politi, M. C. (2018). Anesthesiology control tower: feasibility assessment to support translation (ACT-FAST)-a feasibility study protocol. Pilot and Feasibility Studies 4(January): 38. 18. Myles, P. S., Boney, O., Botti, M., Cyna, A. M., Gan, T. J., et al. (2018). Systematic review and consensus definitions for the standardised

References

endpoints in perioperative medicine (StEP) initiative: patient comfort. British Journal of Anaesthesia 120(4): 705–711.

19. Neuhaus, C., Schild, S., Eismann, H., Baus, J., Happel, O., Heller, R. A., Richter, T. et al. (2020). Funktionalität Und Bedienung von eGENA, Der Elektronischen Gedächtnis- Und Entscheidungshilfe Für Notfälle in Der Anästhesiologie. Retrieved from https://opus.bibliothek.uniaugsburg.de/opus4/frontdoor/index/index/docId/80692, (Accessed June 13, 2022).

20. Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., Badawi, O. (2018). The eICU collaborative research database, a freely available multi-center database for critical care research. Scientific Data 5(September): 180178.

21. Römmele, C., Neidel, T., Heins, J., Heider, S., Otten, V., et al. (2020). Bed capacity management in times of the COVID-19 pandemic: a simulation-based prognosis of normal and intensive care beds using the descriptive data of the University Hospital Augsburg. Anaesthesist 69: 717–725. 22. Sauer, C. M., Dam, T. A., Celi, L. A., Faltys, M. (2022). Systematic review and comparison of publicly available ICU data sets—a decision guide for clinicians and data scientists. Critical Care Medicine 50(6): e581. 23. Schwappach, F., Conen, D., Frank, O. (2018). Empfehlung Zur Einführung Und Zum Betreiben Eines Frühwarnsystems Zur Detektion Sich Unbemerkt Verschlechternder Patienten. Stiftung Patientensicherheit Schweiz S1–S32. 24. Stone, C., Pollard, L., et al. (2018). The MIMIC code repository: enabling reproducibility in critical care research. Journal of the American Medical Informatics Association. Retrieved from https://academic. oup.com/jamia/article-abstract/25/1/32/4259424 (Accessed June 13, 2022). 25. Vistisen, S. T., Pollard, T., Enevoldsen, J., et al. (2021). VitalDB: fostering collaboration in anaesthesia research. British Journal of Anaesthesia 127: 184–187.

503

Chapter 25

Digital Palliative Care

Irmtraud Hainsch-Müller,a Christoph Aulmann,a and Eckhard Eichnerb aInterdisciplinary

Center for Palliative Care, University Hospital Augsburg, Augsburg, Germany bAugsburg Hospice and Palliative Care GmbH, Augsburg, Germany [email protected], [email protected]

25.1 Introduction Palliative care offers a holistic treatment approach for patients with distressing symptoms of physical, psychological, social, or spiritual nature. Palliative care cannot heal and does not serve to prevent fatal diseases. However, it aims to maintain quality of life and autonomy until the end [33]. An integral part of palliative care is the cooperation of the different professions and disciplines. This includes the medical, nursing, physiotherapeutic, psychological, pastoral, and artistic fields. At the end of life, patients are often no longer able to make decisions about themselves and can no longer express their will. Relatives and caregivers often reach their emotional and intellectual limits. Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

506

Digital Palliative Care

The task and goal of palliative care is the identification of problems as well as early integration into treatment for symptom control and psychosocial stabilization. The healthcare system, like most other industries, has experienced a significant shift from analog to digital processes in recent years. The COVID pandemic further accelerated this development in some areas [9]. Successful communication and teamwork are ongoing challenges influenced by this development.

Figure 25.1 Multi-professional, continuous approach [6].

Palliative care routinely involves intensive personal attention to the patient in the treatment of distressing symptoms of any kind. This is an essential quality feature of palliative care and should not be replaced by digitalization. However, as digital transformation is universal, palliative care providers need to help ensure that

Integration of Digital Palliative Care

the linkage points to palliative care are optimal and thus improve patient care. Patients, as well as relatives from different generations and social backgrounds, should not be overburdened in this phase of life. On the other hand, digitalization is natural for the younger generation of patients. A good digital quality of care that enables greater independence and autonomy is demanded and assumed here. An essential component of palliative care is the multi-professional approach (Fig. 25.1), through which the diverse needs of patients can be adequately taken into consideration from the beginning of the illness to the last phase of life [6]. Currently, a large proportion of interprofessional communication takes place informally via mobile phone, tablet, or PC. Teams have organized themselves via communication platforms such as WhatsApp or other tools that are not formally designed for the professional and data protectioncompliant context.

25.2 Integration of Digital Palliative Care 25.2.1 Early Integration/Advance Care Planning

Since the positive study results of J. Temel, the early integration of palliative care in severe diseases with no prospect of cure has been recommended by national and international oncology societies [27]. In this context, personal health care should include the preparation of a patient decree, health care proxy, and care directive. Personal counseling will certainly continue to be of particular importance in the future. As the resource of counseling staff cannot cope with the increasing demand for counseling, the supplementation of the counseling offered with digital media is urgently needed. In the context of the repeated contact restrictions due to the COVID pandemic, the online offer of “palliative care” has been expanded, for example, the homepage of the German Society for Palliative Medicine (DGP). Advance care planning in a palliative context is anchored in law, in Germany for example by the Act on the Improvement of Hospice and Palliative Care in Germany (Hospiz- und Palliativgesetz – HPG) passed by the Bundestag in 2015 [8]. Many Advance Care Planning

507

508

Digital Palliative Care

instruments such as living wills, powers of attorney, and emergency plans are now available digitally [16].

25.2.2 Symptom Control in the Context of Palliative Care Treatment

Both the inpatient and ambulatory treatment settings of palliative patients have specific special requirements. The treatment teams have high aspirations to provide the patient with highly individualized and high-quality care. Medical societies and organizations have defined quality criteria such as pain reduction or dyspnea relief, which should have occurred within a defined period (e.g., 48 hours) [19]. The actions required from the onset of a symptom to the therapeutic response are a multi-step process. Digital approaches can be used for patient monitoring. Some digital approaches are:

∑ ∑ ∑ ∑ ∑ ∑ ∑

Electronic medical health records (EMR) Applications (APPs) Software agents (BOTs) Computerized decision support systems (CDSS) Wearables (WEAR) Virtual reality (VR) Smart homes

The German Federal Institute for Drugs and Medical Devices (BfArM) has specified so-called reference functions that are used to classify an app as a medical device:

∑ Decision support or independent decision-making (e.g., for therapeutic measures) ∑ Calculation (e.g., medication dosages) ∑ Monitoring a patient and collecting data (e.g., recording pain intensity)

With the help of the “Pain App,” for example, it is possible for pain patients to document their pain occurrence regarding intensity, duration, resting and exertion pain [7, 23]. This information can be viewed in a web portal and can be used for further care. Currently,

Integration of Digital Palliative Care

however, there are still concerns about the data security of such apps [14]. People with advanced tumor diseases often suffer from various symptoms, the diversity, and intensity of which are frequently underestimated by treatment providers [1, 15, 28]. Therefore, patients should record their symptoms themselves and not be assessed exclusively by care providers. The use of robots and assistance systems to support everyday life is still in its infancy and is the subject of controversial debate. It is possible to include them in interactive conversations and educational programs. For example, they could be used as part of discussions on palliative care issues in educational programs and public relations work. It is now also possible to use robots to simulate human intimacy and physical affection in the final phase before death (end-of-lifecare-machine) [5]. Certainly, these technical developments do not represent a patent solution to staff shortages in the care of the terminally ill, but it is nevertheless essential to address them.

25.2.2.1 PROMs, the basis of patient-centered medicine

PROMs (patient-reported outcome measures) are the gold standard in patient-centered medicine and are increasingly being recorded digitally. There are many approaches to capture or control such e-PROMs (electronically captured PROMs) with apps on smartphones or computers. Self-monitoring of oncological patients, for example, has shown that not only their quality of life improved but also that their lifespan was extended [3]. “Wearables” are small portable computers commonly known as pedometers, or for monitoring sleep. Especially in times of the COVID pandemic, when it is not always possible to physically visit patients, such systems are a promising option. As a next step, e-PROMs and wearables, i.e., subjective symptoms and objective parameters (steps, heart rate, or heart rate variability), can then be combined. In the future, it will then be possible to combine all collected data in a cloud and analyze it with machine learning. The use of data collected by smart homes about the patients living in them is also conceivable [20, 30].

509

510

Digital Palliative Care

Table 25.1 Examples of factors related to digitization that may improve or complicate the patient-professional relationship in palliative care

Example of factors related to digitisation Increasing use of telemedicine Increasing use of smartphone apps, sensors and wearable devices for monitoring patients’ condition Automated speech, language and image processing

Increasing use of robotics to provide care

Increasing use of digital devices for routine communication in palliative care, such as arranging home visits

Potential direct and immediate impacts to the patient-professional relationship in palliative care Potential direct access to specialist palliative care providers even for those living in rural and remote areas, although possible problems in access to psychological support. Potential problems for privacy and confidentiality of data. Impact on relationship that focuses on the data generated rather than holistic concerns of the patient.

Patient reports and images may be screened to alert professionals of signs indicative of psychological distress which may mean more rapid access to care, or may be perceived as depersonalising. Focus may be on ‘easy to see’ physical aspects such as wound management, with less attention on psychosocial and spiritual aspects of palliative care.

Potential that routine aspects of care may in the future be delivered by robots in the home which may support dying patients to remain there, even those living alone. However, concerns about how isolated and lonely patients may be withtout professional caregivers. May directly disadvantage older patients with low levels of health and digital literacy, poor communication skills, diminished decisional capacity, or those without access to internet connectivity or appropriately enabled devices.

Integration of Digital Palliative Care

Example of factors related to digitisation Increasing use of internet-based health and palliative care information

Potential direct and immediate impacts to the patient-professional relationship in palliative care May empower patients and family carers to better navigate health systems and readily obtain disease-related and service information, but there are no quality controls on the information sources. Professionals may feel disempowered or reluctant to critically evaluate patient demands.

Apps and computer-based interventions are suitable for psychosocial support and therapy, as many people today communicate more often via phone and computer than through direct conversation. Especially in cases of mental illness with social withdrawal, such an approach can be helpful. A large metaanalysis of female patients showed that depressive symptoms can be significantly reduced by such approaches [10]. Pastoral care can also contact patients digitally (via Skype, Zoom, etc.). There are also online prayer groups and discussion groups [22]. This has led to a surge in the development of teaching and research on spiritual care. However, there is a risk that more attention will be drawn to digital processes and collected data than to actual patient care. Factors that influence the relationship between patients and professional caregivers are listed in Table 25.1 [21].

25.2.3 Patient Care/Management/Limits and Opportunities

Modern technology can be used to identify patients with palliative needs. One idea in this context is to ask the “surprise question” (“Would you be surprised if the patient died in the next six months?”) to the electronic medical record or computer system. This can be done by a simple query of negative prognostic factors. Carefully balancing the opportunities and risks and involving all patients at all stages of care is a challenge. The challenge is that invasions of privacy can arise through surveillance, or automatisms and algorithms. The WHO has published a recommendation on this [13].

511

512

Digital Palliative Care

Table 25.2 Pros and cons of telehealth in palliative care (from [21]) Pros

Cons

Patients report ease of discussing sensitive or personal problems to the clinician

Inability to engage in nonverbal social behaviors

Reduced no shows and cancellations

Technology difficulties and lack of connectivity can lead to patient and clinician frustration

Improved patient satisfaction

Streamlined reimbursements No wait time for patients or caregivers

Patients and caregivers avoid traveling to clinical setting

Ability increase access to specialty services for underserved populations especially in rural or low- and middle-income areas Environmental benefits from reduced travel and vehicle emissions

Improved productivity at work by reducing healthcare appointment travel time Ability to gather previously inaccessible information about the patient when viewing them in their home setting Increased provider and clinic efficiency with decreased wait times while the patient is triaged for visit

Difficulties gathering accurate information without a physical exam

Cultural and language barriers between patient/caregiver and clinician Reviewing/updating different institutions electronic health records

Privacy concerns (if the patient lives in a communal situation) Patient or caregiver not answering their phones (telephone-based telehealth)

Clinician stress in trying to attain and maintain rapport via remote connections, including the absence of human touch for both patients and clinicians Could create a “digital divide” and add to healthcare disparities for patients who do not have available devices and connectivity for telehealth visits

Provider visual and physical fatigue if multiple video visits sessions in one setting

Integration of Digital Palliative Care

It is important to pay attention to safety, and of course, it should also be investigated whether the intervention has a positive effect. Particular attention should be paid to how the intervention is accepted by the patient and a feasibility analysis should be carried out beforehand. Likewise, the knowledge, attitude, and behavior of the participants are to be analyzed. Lastly, attention should also be paid to the costs of such an intervention. New systems should not be bought because they are new or just to spend money on a new system, which unfortunately happens quite often with digital renewal. The digital workplace in the field of palliative care is currently the subject of a scientific project, funded by the Bavarian Research Institute for Digital Transformation (bidt) [18]. The technical, cultural, and personal barriers to telehealth are summarized in Table 25.2 [31].

25.2.4 Cross-Sectoral Care

In the German health care system, cross-sectoral care is understood to be the networking of the inpatient hospital sector with the institutions providing outpatient care, such as nursing services, general practitioners, and specialists or – in the case of palliative medical and nursing care – also specialized ambulatory palliative care (SAPV). The main challenges of such cross-sectoral care using digital communication and data exchange are data protection and data security in addition to the technical infrastructure. Thus, it must not be overlooked that patient-related data is always highly sensitive data, which requires appropriate security technology in terms of structure, processes, data storage, encryption, etc. In view of the increase in hacking attacks on institutions of the German health system with, for example, encryption Trojans, continuing ransom demands, and blackmail attempts in recent years, individual institutions have paid more attention to their security and protection against hacker attacks. These security requirements massively complicate the cross-sectoral exchange of data and the development of simple, practice-oriented solutions and make them considerably more expensive.

513

514

Digital Palliative Care

In view of the vulnerable personal data on the one hand and the threat potential on the other, it must also be stated that the development of proprietary solution approaches should at best be used for research projects with a time limit and may not be operated, or only in compliance with the corresponding security standards and data protection. Instead, palliative care should be connected to the nationwide projects of the electronic case file and the electronic health card within the telematics infrastructure (TI). This serves to improve the efficiency, quality, and transparency of care and is a promising approach for cross-sectoral care [29]. Likewise, the patient data management systems (PDMS) used by the large system houses should generally provide appropriately secured interfaces, which enable the logged and approved access of the ambulatory assigned providers to the data of the respective jointly cared-for patient. Especially in view of the regularly limited lifespan of patients with their increasing dependency on the loss of mobility, corresponding diagnostic data are of importance. Also encouraging is the multitude of telemedicine applications that help to reduce the need for patients to see a doctor [25]. For palliative patients, this could represent a significant increase in the quality of life and the quality of care, if qualified counseling, diagnostics, and therapy are provided by on-site professionals at the point of care – often the patient’s home. As a result, hospital admissions are avoided.

25.2.5 Digital Bereavement Counseling/Settlement of the Digital Estate

In the last decades, before the turn of the millennium and at the beginning of this century, bereavement work has undergone considerable change with transformations in social structures. With a predominantly rural population, bereavement work took place and still takes place mainly within the family and close social environment. The migration to the cities resulted in changed social contacts. Next to intensive family contacts and contacts that have existed over generations, social contacts independent of the place of residence are becoming increasingly important with the changes of residence and the development of digital media. Bereavement work

Research in the Field of Digital Palliative Care

no longer takes place only in the private setting. Whereas in the past it was mainly telephone offers [26], there are now various digital offers that are currently still in a very dynamic state of development [4]. Another aspect is the holding of mourning rituals, which are now also (complementarily) held in a digital environment. Due to social contacts in digital media, there is now also a post-mortem need to regulate digital documents and traces. Legal regulations have been created in Germany for this purpose [32]. According to a ruling (12.7.2018-IIIZR 183/17) of the Bundesgerichtshof (BGH), the digital estate is to be treated like the inheritance of objects. Different providers (Google, Facebook) offer their own regulations (e.g., account inactivity manager or estate contact) for dealing with the digital estate. Companies also offer themselves for the administration of the digital estate [11].

25.3 Research in the Field of Digital Palliative Care

Considerable research activity has developed in the field of palliative care in recent decades. Numerous departments of palliative medicine have been established at German universities. In this young field, innovative approaches are being worked on intensively. A traditional study design will be of little use in the future, as it often operates with questionnaires and one-off measurements at fixed points in time. Today, the data stream in social media is already being continuously analyzed, which can yield new developments and research approaches [24]. One of these is the inclusion of artificial intelligence in the determination of prognostic factors [2, 17]. General challenges and research fields are:



∑ Providing knowledge and information in a personalized, context-specific, and timely manner ∑ Enabling communication and networking between stakeholders ∑ Participants: Professionals (doctors, nurses, therapists), semi-professionals (volunteers), informal (relatives), patients ∑ Organizations involved: Clinic (palliative care unit, ZNA), Ambulance service (RTW, emergency doctor), nursing home, SAPV, ambulatory care service, hospice

515

516

Digital Palliative Care



∑ The control of information ∑ Top-down: Doctor’s letters, guidelines, etc. ∑ Bottom-up: Patient/proxy in the interest of the patient ∑ Maintaining patient trust (privacy, familiarity/intimacy, reliability), trust in people ∑ Trust in structures (and thus role concepts) – which are relevant for patients and professionals.

25.4 Education

The COVID pandemic and the resulting contact restrictions have led to changes in all areas of education, training, and advancing education. Academic teaching has also been adapted to the situation. Creating videos, and live events with digital broadcasts is now a widely used option. The various teaching platforms, e.g., “Moodle,” serve as a supplement to the students’ knowledge transfer. The possibilities of digital group work in dedicated spaces, as well as visualizations and other interactive elements, are now being used so successfully that they have become an integral part of education. Regarding teaching in palliative care, digitalization will increasingly provide helpful and complementary support systems that offer easily accessible knowledge transfer in the context of studies and further training. Regarding research, many questions remain exciting. For example, the handling of the large amounts of data collected, their utilization, and the associated consequences of “big data” need to be considered.

25.5 Prospects

The central question of the future will be how technological progress will influence the relationship between therapist and patient. Respect for the patient’s wishes and goals, empathic listening, and the preservation of dignity are the core concerns of palliative care. The focus of the treatment should always be on the whole person with all their various needs. How digital technologies can be used in the future to support and improve the relationship between patients and care providers is a challenge for future research.

References

The sequence of action between the use of the electronic health record, the ability of patients to collect and share their own health data, and the remote monitoring of symptoms need to be defined. By linking these areas and understanding the difficulties involved, patient care and the patient-caregiver relationship can be improved. A conscious, critical, and wise approach to multifaceted challenges can promote successful implementation. The risk of practitioners’ attention is more on the screen than on the patient has been known for some time. Doctors spend about 50% of their time in front of the screen and less than 10% with the patient [12]. The amount of information about the patient will increase, but not the knowledge about them as human beings. The time to relate to him will always be limited. Corresponding recommendations as “best practices” for dealing with the electronic health record in terms of management, analysis, and understanding of the large amounts of data must be developed. The key to palliative care is working at the relationship level. Working out what data is important for the patient and their condition and putting it into practice can improve care. However, which data and procedures undermine the patient-treatment relationship is a task for future research. The question is, how do we preserve the existential and emotional concerns of our patients?

References

1. Amann, M., Blum, D. (2021). Digitale palliative care. Praxis 110(15): 851–854, https://doi.org/10.1024/1661-8157/a003784. 2. Avati, A., Jung, K., Harman, S., et al. (2018). Improving palliative care with deep learning. BMC Med Inform Decis Mak. 18(Suppl 4): 122.

3. Basch, E., Deal, A. M., Dueck, A. C., et al. (2017). Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 318(2): 197–198.

4. Beaunoyer, E., Hiracheta Torres, L., Maessen, L., Guitton, M. J., al. (2020). Grieving in the digital era: mapping online support for grief and bereavement. Patient Educ Couns. 103(12): 2515–2524. DOI: 10.1016/j.pec.2020.06.013.

517

518

Digital Palliative Care

5. Chen, D. (2017). Artificial intimacy: end of life care machine. plus insight. https//plusinsight.de2017/11/ars-elecronica-2017-artificialintimicy-end-of-life-care-machine.van-dan-chen/ (Accessed March 30, 2022).

6. Constantine, A., Condliffe, R., Clift, P., et al. (2021). Palliative care in pulmonary hypertension associated with congenital heart disease: systematic review and expert opinion. ESC Heart Fail 8: 1901–1914. 7. Ewers, A. (2017). Pain App ermöglicht mobile Schmerztherapie. Die Schwester Der Pfleger 5(1): 42–42.

8. Federal Law Gazette, Bundesgesetzblatt Jahrgang 2015 Teil I Nr. 48, issued at Bonn on December 7, 2015. https://www.bgbl.de/xaver/bgbl/ start.xav?start=%2F%2F*%5B%40attr_id%3D%27bgbl115048. pdf%27%5D#__bgbl__%2F%2F*%5B%40attr_ id%3D%27bgbl115048.pdf%27%5D__1648539665846. 9. Federal Ministry of Health. Living Will (Accessed March 30, 2022), https://www.bundesgesundheitsministerium.de/ patientenverfuegung.html. 10. Finucane, A. M., O’Donnel, H., Lugton, J., et al. (2021). Digital health interventions in palliative care: a systematic meta-review. Digit Med. 4: 64, https://doi.org/10.1038/s41746-021-00430-7.

11. Firth, J., Torous, J., Nicholas, J., et al. (2017). The efficacy of Smartphone based mental health interventions for depressive symptoms: a metaanalysis of randomized controlled trials. World Psychiatry 16(3): 287– 298. 12. Germany’s Federal Government (2018). Regulating the Digital Estate. https://www.bundesregierung.de/breg-de/aktuelles/digitalennachlass-rechtzeitig-regeln-8420050.

13. Holland, D. E., Vanderboom, C. E., Ingram, C. J., et al. (2014). The feasibility of using technology to enhance the transition of palliative care for rural patients. Comput Inform Nurs. 32(6): 257–266. 14. Jandoo, T. (2020). WHO guidance for digital health: what it means for researchers. Digit Health 6: 2055207619898984. DOI: 10.1177/2055207619898984. 15. Krüger-Brand, H. E. (2016). Apps in sicherheitsrisiken. Dtsch Ärztebl. 45: 14–15.

der

medizin:

viele

16. Laugsand, E. A., Sprangers, M.A., Bjordal, K., et al. (2010). Health care providers underestimate symptom intensities of cancer patients: a multicenter European study. Health Qual Life Outcomes 8: 104.

References

17. Manz, C. R., Chen, J., Liu, M., Chivers, C. et al. (2020). Validation of a machine learning algorithm to predict 180-day mortality for outpatients with Cancer. JAMA Oncol. 6(11): 1723–1730. 18. PALLADiUM: Palliative Care as a Digital Working World, https:// www.uni-augsburg.de/de/fakultaet/philsoz/fakultat/soziologiesozialkunde/forschung/l/palladium/ (Accessed March 30, 2022). 19. Palliative Care – Guidelines Program, https://www.leitlinienprogrammonkologie.de/leitlinien/palliativmedizin/ (Accessed March 30, 2022).

20. Pavic, M., Klaas, V., Theile, G., et al. (2020). Mobile health technologies for continuous monitoring of cancer patients in palliative care aiming to predict health status deterioration: a feasibility study. J Palliat Med. 23(5): 678–685.

21. Payne, S., Tanner, M., Hughes, S. (2020). Digitization and the patient– professional relationship in palliative care. Palliat. Med. 34(4): 441– 443. 22. Peng-Keller, S., Neuhold, D. (2020). Charting Spiritual Care: The Emerging Role of Chaplaincy Records in Global Health Care. Springer, Berlin.

23. Schnell, M. W., Schulz-Quach, C. (Hrsg.) 2019. Basiswissen Palliativmedizin, https://doi.org/10.1007/978-3-662-59285-4_20.

24. Selman, L. E., Chamberlain, C., Sowden, R. et al. (2021). Sadness, despair and anger when a patient dies alone from COVID-19: a thematic content analysis of Twitter data from bereaved family members and friends. Palliat Med. 35(7): 1267–1276. 25. Tanne, Telemedicine Collaboration Networks. https://www.tannetelemed.de/.

26. Taubert, M., Norris, J. (2015). OA57 The digitalisation of dying, loss and grief on social media channels. BMJ Support Palliat Care Suppl 1: A18. DOI: 10.1136/bmjspcare-2015-000906.57.

27. Temel, J. S., Greer, J. A., Muzikansky, A., et al. (2010). Early palliative care for patients with metastatic non–small-cell lung cancer. N Engl J Med. 363(8): 733–742. DOI: 10.1056/NEJMoa1000678.

28. Teunissen, S. C., Wesker, W., Kruitwagen, C., et al. (2007). Symptom prevalence in patients with incurable cancer: a systematic review. J Pain Symptom Manage. 34(1): 94–104.

29. The Electronic Health Card, Federal Ministry of Health, https://www. bundesgesundheitsministerium.de/themen/krankenversicherung/ egk.html (Accessed January 31, 2022).

519

520

Digital Palliative Care

30. Tyler, J., Choi, S. W., Tewari, M., et al. (2020). Real-time, personalized medicine through wearable sensors and dynamic predictive modeling: a new paradigm for clinical medicine. Curr Opin Syst Biol. 20: 17–25.

31. Watts, K. A., Malone, E., et al. (2021). Can you hear me now? Improving palliative care access through telehealth. Res Nurs Health 44: 226–237. 32. Wende, H. (2019). Digitaler Nachlass: wie wir Regelungslücken vermeiden. Schmerzmedizin 35: 55–56.

präventiv

33. WHO Definition of Palliative Care (2002). https://www. dgpalliativmedizin.de/images/stories/WHO_Definition_2002_ Palliative_Care_englisch-deutsch.pdf (Accessed 30 March, 2022).

Chapter 26

Digital Medicine in Pulmonary Medicine

Olaf Schmidt

Pneumology Group Practice, Koblenz, Germany [email protected]

An attempt to present the status quo through descriptions of the currently recommended apps and online portals and by means of case studies.

26.1 Introduction

In pulmonology, considerable progress and further developments in the field of digital medicine and telemedicine have taken place in recent years. In the 4th edition of the German NVL Asthma, a chapter on telemedicine was added for the first time in September 2020: “With this chapter, the guideline group points out that telemedical measures could also become more important in the care of patients with asthma in the future. Nevertheless, the evidence currently identified is not sufficient to make a recommendation on the use of Digital Medicine: Bringing Digital Solutions to Medical Practice Edited by Ralf Huss Copyright © 2023 Jenny Stanford Publishing Pte. Ltd. ISBN 978-981-4968-73-7 (Hardcover), 978-1-003-38607-0 (eBook) www.jennystanford.com

522

Digital Medicine in Pulmonary Medicine

telemedicine procedures” [1, 8, 15, 17]. However, after a little more than a year, the “Deutsche Atemwegsliga e.V.” (German Respiratory League) has identified a total of 10 applications from different indication areas of pneumology as recommended telemedical applications [7, 14]. In the following, I will first try to give an overview of the health apps currently recommended by the “Deutsche Atemwegsliga e.V.” in individual sections. Then I will describe an example of an app and how it works, with the help of which various studies have already been carried out. Using three example studies, I will try to illustrate the development of this application and discuss the results of the individual studies. Finally, I would like to discuss which characteristics I think a telemedical application must fulfill in order to represent an improvement in care for both patients and doctors in the field of pneumology.

26.2 Overview of the Applications Mentioned on the Homepage of the Deutsche Atemwegsliga e.V.

The applications are listed and described in the order in which they are presented on the homepage of the “Deutsche Atemwegsliga e.V.” [14] and in accordance with the information provided by the manufacturers. This does not represent a valuation of the application itself. The name and a short description of the functions of the application are always mentioned; for a more detailed description, I refer to the applications themselves.

26.2.1 Kaia COPD App

“...The app is aimed at patients with COPD, it is intended to accompany users during rehabilitation. The app contains modules from the areas of strength and mobilization exercises, relaxation, and knowledge. The training program is suggested based on the health information provided by the user [5]. There are different exercises to choose from in each area. In the progress category, training results such as a change in breathlessness, physical well-being, and physical activity can be viewed. Push messages are designed to motivate activity...”

Overview of the Applications

26.2.2 Atemwege Gemeinsam Gehen App/Breath Walk Together App “...is an exercise app by AstraZeneca for people with asthma. The app aims to motivate people to be more active and support them in becoming more active in their daily lives. The app contains a training program lasting several weeks with two to three strength and endurance training units per week. The design of the endurance training is determined individually. The maximum heart rate is determined on the basis of a medical stress examination. The individual training heart rate range is read off a table (menu item endurance training). The intensity of the training is based on a Borg scale. For strength training, exercise videos with Heike Drechsler (a German long jump world champion and Olympic champion) are stored. You can choose between easy and difficult exercises. After 12 completed units, a further training program is activated. Warm-up and stretching exercises complete the offer...”

26.2.3 OMROM Asthma Diary App

“...is an app designed to support asthma management in children. It provides reminders to use the prescribed medication. The use of the emergency spray can be documented in the app. The diary function enables an overview of the course of symptoms. The main function is the coupling of the app with the OMRON WheezeScan. The pairing is done via Bluetooth. The WheezeScan is a device that can detect wheezing in children (4 months to 7 years). This app is designed to help parents detect an impending asthma attack at an early stage...”

26.2.4 Vivatmo App

“...is a digital asthma diary. Measured peak flow values, medication, symptoms (cough, sputum, shortness of breath, asthma attack, other symptoms), and notes can be entered manually. After the release of the location, the pollen load is automatically communicated. The essential function of the application is the documentation of FeNO readings, which can be measured with the separately purchased device “Vivatmo me” and transferred via Bluetooth to the app and a calendar overview...”

523

524

Digital Medicine in Pulmonary Medicine

26.2.5 breaszyTrack – dein Asthma-Helfer App/ breaszyTrack – Your Asthma Helper App “...The app provides general information on asthma based on national and international guidelines or recommendations. The focus is on medication therapy and emergency management. Results of the peak flow measurement are entered manually. From these, borderline areas are calculated and visualized according to the asthma traffic light scheme. An asthma value is calculated from a symptom questionnaire. This value is commented, e.g., “in the target range” or “need for improvement.” An emergency plan is stored in the app but is not linked to the entry of very poor peak flow values. The app has a diary function and reminds you to use the medication entered...”

26.2.6 www.copd-aktuell.de is an Online Portal

“...The portal contains extensive information on drug and non-drug treatment of COPD, including exacerbations. Tips are given on how to deal with the disease and on self-help options...”

26.2.7 “Kata – Deine Inhalationshilfe für die Anwendung Eines Dosieraerosols” App/“Kata – Your Inhalations Aid for the Use of a Metered Dose Inhaler” App

“...Respiratory diseases are mainly treated with inhaled medication. The correct use of the prescribed inhalation system is important for the success of the therapy. There are numerous inhalation systems that differ considerably in their application. The Kata app checks the inhalation with a metered dose inhaler. The app requires access to the microphone and camera. The app supports users in their daily therapy with inhaled medication. The most important steps exhalation, inhalation, breath holding, and exhalation are explained. The app gives feedback to users and suggestions for improvement...”

Overview of the Applications

26.2.8 “myAir” App “... The appis designed to help patients monitor their sleep apnea therapy. The therapy adherence should be improved. A further aim is to help patients to help themselves...”

26.2.9 “Nichtraucher Helden” App/“Non-Smoking Heroes” App

“...The app supports smokers to implement the smoking cessation... ” The appis listed as a digital health application (DiGa) and can be prescribed at the expense of the statutory health insurance in Germany.

26.2.10 “SaniQ Asthma!” App

“...is a digital diary that aims to help patients manage their condition. The app can be used for self-monitoring or telemedicine monitoring by treating physicians. When using the telemedicine mode, the doctor must be registered with SaniQ. The app is intended for patients with mild to moderate asthma. It should be noted that the use of SaniQ does not replace a visit to the doctor. The minimum age for app use is 16 years, according to the manufacturer. Medical readings from compatible devices can be sent to the app via Bluetooth. Manual input is possible. The data can be summarized and exported via pdf. The app is a class I medical device.”

26.2.11 “Therakey” is a COPD Online Portal

“... Extensive information is given on the prevention and treatment of COPD. In addition to treatment with medication, long-term oxygen therapy, and non-invasive ventilation, other non-drug treatment options such as exercise, nutrition, education, and respiratory physiotherapy are presented in particular. An integrated fitness program to participate in (Fitmacher COPD) offers a structured, staggered training program over several weeks. Classification is based on an everyday fitness test. It is emphasized that the treatment decision lies with the attending physician. Advice on how to react in an emergency is given and the possibility is offered to create an emergency passport together with the treating persons... ”

525

526

Digital Medicine in Pulmonary Medicine

26.3 Description of the Functioning of the SaniQ App With the SaniQ app (Qurasoft GmbH, Koblenz), patients suffering from chronic lung or heart diseases as well as infectious diseases (e.g., in the context of the COVID-19 pandemic) record relevant vital signs in a digital health diary, to which the attending physician has to access through an encrypted connection. The measured values are clearly processed and doctors receive a valid basis for a more reliable diagnosis and therapy. This allows them to react to changes in the course of the disease at an early stage without the patient having to visit the practice in person. The medication plan can also be transmitted digitally to the patient. With the help of Bluetooth-enabled measuring devices, the patients record various measured values (e.g., FEV1, PEF, SpO2, temperature, blood pressure, and pulse), which are transmitted to the patient’s smartphone (see Fig. 26.1).

Figure 26.1 The following measured values can currently be recorded tele medically.

In principle, many different measured values can be recorded by this application, and disease-relevant information on the current pollen count, the weather situation, and the air quality at the place of stay can be determined and recorded by the app.

Description of the Functioning of the SaniQ App

Patients can document their symptoms, medication intake, and other health information and fill out and save patient questionnaires, such as the ACT (see Fig. 26.2). A findings manager enables the clear filing of doctor’s documents or X-ray images. The app also contains all information about the personal medication stock and informs the user when preparation is running low and should be reordered. The user-friendliness has increased, among other things, by the fact that medicines to be taken can be stored in the app by scanning the PZN code on the packaging with the help of the smartphone camera and saved in the German “Bundeseinheitlicher Medikationsplan” [20, 21].

Figure 26.2 Example of an ACT questionnaire progression over several months, the questionnaires can be answered by the patient in the app.

This information is then transmitted almost in real time to the practice of the attending specialist (see Fig. 26.3). The doctor and his team analyze the data in their own software (SaniQ desk) and analyze it. An AI algorithm that recognizes and supports them and filters preset abnormality patterns to reduce the work intensity in the practice. The medical system SaniQ Desk supports doctors in particular in identifying critical courses of disease. Through intelligent progress monitoring, the system automatically points out patients who show insufficient adherence, a deterioration in

527

528

Digital Medicine in Pulmonary Medicine

Figure 26.3 Data flow and interaction in app-supported healthcare. (1) Data measurement by the patient; (2) Data transfer via smartphone; (3) Preparing and saving into the database; (4) Providing data via web browser; (5) Analysis by therapist/clinicians; and (6) Feedback to the patient via smartphone.

Figure 26.4 Example of the WEB view of the browser on the doctor’s screen with a curve view of the FEV1 value progression over several months.

What Studies Have Already Been Conducted with This Application?

measured values, or an increase in complaints. Given the abundance of treatments, this proved to be a significant increase in efficiency and quality. In case of abnormalities, the doctor and the medical staff can communicate with the patient via the appsystem in a secure messenger function or directly via video chat (see Fig. 26.4). The app and the desktop application are now available in seven languages.

26.4 What Studies Have Already Been Conducted with This Application?

I have been following and supervising the development and use in a real medical setting of the SaniQ app for many years as one of the medical scientific advisers and contact persons. After the first very small proof of concept study on the technology itself with 12 participants in 2016, various other studies have been carried out over the last few years, especially real-world evidence studies. This is in asthma and COPD patients and then in the context of the COVID-19 pandemic as part of the German egePAN Unimed project of the Federal Government, as part of the National Research Network of University Medicine on COVID-19 [2, 24, 25].

26.4.1 Rhineland-Palatinate Breathes Through: Telemedicine for Healthy Lungs

First of all, I would like to present the model project “RhinelandPalatinate breathes through – Telemedicine for healthy lungs,” which was carried out monocentrically in my study center and under my leadership [25]. In this project, the treatment through digital support services as well as telemonitoring by the treating specialist practice was investigated. The feasibility and acceptance of the application were evaluated. Furthermore, effects on the quality of life and the course of the disease were determined. “Rhineland-Palatinate breathes through – Telemedicine for healthy lungs” was a model project to optimize the care of asthmatics with the support of the state government of Rhineland-Palatinate. All participants in the project used the “SaniQ Asthma” health app for people with lung disease for a period of three months and kept their digital health diary in it. For this purpose, they recorded

529

530

Digital Medicine in Pulmonary Medicine

measured values using Bluetooth-enabled measuring devices to describe the lung function as already described above. The two-arm, randomized, monocentric study was conducted in Rhineland-Palatinate in 2018. 120 asthmatics participated in the study over the duration of three months each. 80 participants (telemedicine group) received telemedicine support in addition to conventional treatment, whereas the 40 participants in the control group received conventional treatment only. The objective was to assess the feasibility and acceptance of telemedicine in asthma treatment, measured by adherence. Adherence of at least 50% was to be achieved in the telemedicine group. Adherence per participant was defined as the ratio of existing peak flow measurements to planned peak flow measurements. The average adherence in the telemedicine group was 80.14% (±19.39). The goal of the project was to achieve adherence of more than 50%. This was achieved and validated with a one-sample t-test (p =