Smart Healthcare Engineering Management and Risk Analytics (AI for Risks) 9811925593, 9789811925597

Citation preview

AI for Risks

Shuai Ding Desheng Wu Luyue Zhao Xueyan Li

Smart Healthcare Engineering Management and Risk Analytics

AI for Risks Series Editors Desheng Wu, School of Economics and Management, University of Chinese Academy of Sciences, Beijing, China Jim Lambert, University of Virginia, Charlottesville, VA, USA David L. Olson, Department of Management, University of Nebraska–Lincoln, Lincoln, NE, USA

Risks are widespread in human society, such as in the financial field (various investment risks), in technical fields (risks brought by emerging technologies), in social fields (political risks), and in all aspects of our lives (health risks, natural environment risks, etc.) At the same time, the inherent links between various risks generate systemic risks. The major challenge today is not to deal with new types of risks, but to focus more on those risks that are difficult to distinguish effectively and timely, or risks that have emerged in a different way from the past. This series of books aims to strengthen discussions on frontier hot topics across disciplines, that is, using AI technologies to solve risks problems that exist in the environment, healthcare, technology, and financial fields. The scope of it focuses on the implementation of artificial intelligence technology in dealing with risks, such as risk prediction, assessment, and mitigation. It includes monographs, edited volumes, textbooks and proceedings etc. on the application of artificial intelligence in risk management in the fields of social media, healthcare, the public sector, financial technology, and regulatory technology.

Shuai Ding · Desheng Wu · Luyue Zhao · Xueyan Li

Smart Healthcare Engineering Management and Risk Analytics

Shuai Ding School of Management Hefei University of Technology Hefei, Anhui, China

Desheng Wu School of Economics and Management University of Chinese Academy of Sciences Beijing, China

Luyue Zhao School of Management Hefei University of Technology Hefei, Anhui, China

Xueyan Li School of Management Hefei University of Technology Hefei, Anhui, China

ISSN 2731-6327 ISSN 2731-6335 (electronic) AI for Risks ISBN 978-981-19-2559-7 ISBN 978-981-19-2560-3 (eBook) https://doi.org/10.1007/978-981-19-2560-3 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Preface

The dawn of a new era of healthcare services could already be observed at the horizon. Several different types of new healthcare models have been explored and documented by the industry and academia, such as “Intelligent Healthcare”, “Internet Healthcare”, “eHealth”, “mHealth”, “Telemedicine”, “Remote Healthcare”, “Internet of Healthcare Things”, “Digital Healthcare”, and “Healthcare 4.0.” These new forms of healthcare, characterized by the utilization of various next-generation information technologies, could be abbreviated by “Smart Healthcare.” The definition of Smart Healthcare is to achieve personalized, intelligencified, and interconnected healthcare services through the utilization of next-generation information technologies. Through the interconnection of data, high-speed and timely sharing of information, healthcare knowledge generation, and knowledge-driven services, the efficiency of the healthcare system could be drastically improved in all aspects, especially quality, efficiency, and capacity. With the extraordinary speed of innovation, many systems with the essence of Smart Healthcare have already been put into application in hospitals, community clinics, and personal spaces. For example, remote/intelligent hospital guidance service, provided by web-enabled services and robots, has been implemented in many large healthcare institutions. Health monitoring applications on smartphones and wearable devices are widespread. Diagnostic supporting systems utilizing machine learning and deep learning are gaining popularity. Through the implementation of smart-connected healthcare services, these systems could demonstrate the characteristics of autonomous perception, reasoning, decision-making, and controlling. The general trends of new healthcare systems point to an organic combination of system engineering and management methodologies and next-generation information technologies. While heavily heterogeneous in structures, the development, application, and popularization of such systems all share similar engineering principles and difficulties. Smart Healthcare systems mostly focus at specific healthcare scenarios, optimize the cost-efficiency of healthcare operations, and are the fusion of medical science, information technologies, and management science. The scientific problems they face

v

vi

Preface

are also similar: the governance and utilization of cross-domain data, the coordination between different service processes, and the analytics and control of risks. The functionalities of such systems could all be further enhanced through connectivitydriven cooperation with other healthcare systems. These homogeneous engineering principles and scientific problems are what consist of smart healthcare engineering management, and the illustration and analytics of such are the core and main purpose of this book. Through the process of Smart Healthcare Engineering Management, one of the most important aspects is the existence of risks. While risks also exist in any other system, the risks in Smart Healthcare lean more towards aviation and industry safety than business and finance. The fact that healthcare is directly responsible for lifesaving made it more prone to life-losing risks, which could not be remedied after the life-losing event happens. The qualification, quantification, alleviation, and prevention of risks, or “risk analytics” in Smart Healthcare, are thus completely different than many traditional concepts such as risk management and risk analysis. Smart Healthcare Engineering Management and risk analytics need to accompany each other through its lifecycle, thus the name and the structure of this book. This book will be divided into three parts: trends, theories, and models. We first examine the current developing trends of Smart Healthcare from the perspective of both general industry developments and research trends in specific fields. We then present our theories in Smart Healthcare Engineering Management, focusing on three different aspects: data utilization and governance, information exchange and fusion, and knowledge inference and service. Specific Smart Healthcare system designs and theoretical models of distinct scenarios, together with their risk analytics, are followed. In particular, we have chosen six scenarios corresponding with the cycle of care: non-contact health monitoring, online physician recommendation, intelligent cancer screening, ICU risk analytics, surgical quality control, and intelligent hospital operation management. Together, this book seeks to provide the philosophy behind Smart Healthcare Engineering Management and demonstrative examples of incidence capable innovations of Smart Healthcare technologies and systems. While we hope the language of this book is straightforward, we intend our readers to be scholars, researchers, high-level graduate students, and practitioners from related industries. Hefei, China Beijing, China Hefei, China Hefei, China

Shuai Ding Desheng Wu Luyue Zhao Xueyan Li

Acknowledgments

The writing and editing of this book could not be accomplished without the help of our colleagues, friends, and fellow scholars. Specifically, we would like to first thank Professor Shanlin Yang for his introduction to the System of System Engineering (SoSE) methodologies, the brand-new concept and perspective of “Internet of Healthcare Systems (HIS)”, and the future trends of smart healthcare. We thank Prof. Xiaojian Li, Prof. Xiaodong Gu, Prof. Bo Ouyang, Dr. Hao Wang, and Dr. Cheng song for their wise counsel and visions. We are also thankful to our colleagues at the Hefei University of Technology, particularly Dr. Jinxin Pan, Yan Qiu, Ling Li, Zijie Yue, Jinxin Yang, Jiaxin Wang, Hongmin Zhang, Mengru Jia, Wengying Yu, Rui Tan, Caiyun Zhang, Yu Yang, Yiyang Su, Yuxuan Yang, and Yifan Wang for their participation in the experimentation, editing, and proofreading phase of the book. This book was partially supported by the Ministry of Science and Technology of China (Grant No. 2020AAA0108400, 2020AAA0108402), in part by the National Natural Science Foundation of China (Grant No. 91846107 and 71825007), in part by the 2021 Anhui Industrial Internet Platform Project (Minimally Invasive Medical Industrial Internet Cooperative Servicing Platform), in part by the Chinese Academy of Sciences Frontier Scientific Research Key Project (Grant No. QYZDB-SSW-SYS021), and in part by the International Partnership Program of Chinese Academy of Sciences (Grant No. 211211 KYSB20180042).

vii

Contents

1

2

Basics of Smart Healthcare Engineering Management and Risk Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Smart Healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Importance of Smart Healthcare . . . . . . . . . . . . . . . . . . . . 1.1.2 Definition of Smart Healthcare . . . . . . . . . . . . . . . . . . . . . . 1.1.3 Related Concepts of Smart Healthcare . . . . . . . . . . . . . . . 1.2 Full-Cycle Healthcare Management Revolution . . . . . . . . . . . . . . . 1.2.1 Disease Prevention and Health Risk Management . . . . . 1.2.2 Clinical Assistant Diagnosis and Treatment Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Rehabilitations and Follow-Up Visits Management . . . . 1.3 Smart Healthcare Engineering Management . . . . . . . . . . . . . . . . . . 1.4 Smart Healthcare Engineering Management and Risk Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Theories and Models in Smart Healthcare . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Frontier of Smart Healthcare Engineering Management . . . . . . . . . . 2.1 Non-contact Physical and Mental Health Measurement . . . . . . . . 2.1.1 Physical Health and Vital Monitoring . . . . . . . . . . . . . . . . 2.1.2 Mental Disorder Screening . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Online Physician Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 OHC Physician Answer Quality Analysis . . . . . . . . . . . . 2.2.2 Personalized OHC Physician Recommendation . . . . . . . 2.3 Cancer Screening and Diagnosis Decision Supporting . . . . . . . . . 2.3.1 Intelligent Endoscopic UGI Cancer Screening . . . . . . . . 2.3.2 Medical Image Report Generation for Cancer Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Multi-Dimension ICU Risk Analytics . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 ICU Admission and Discharge Management . . . . . . . . . . 2.4.2 ICU Readmission Prediction and Risk Analytics . . . . . .

1 1 2 3 5 8 8 9 10 11 13 15 18 21 21 23 24 28 30 31 32 34 35 36 38 40

ix

x

Contents

2.4.3 ICU Mortality Prediction and Risk Analytics . . . . . . . . . Minimally Invasive Surgery Quality Control . . . . . . . . . . . . . . . . . 2.5.1 MIS Surgical Tool Detection and Segmentation . . . . . . . 2.5.2 MIS Stage Detection and Duration Prediction . . . . . . . . . 2.6 Frontier Trends in Smart Healthcare Engineering Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

41 42 43 44

Data Utilization and Governance in Smart Healthcare . . . . . . . . . . . . 3.1 Smart Healthcare Data Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Healthcare Data Utilization Regulations . . . . . . . . . . . . . . 3.1.2 Healthcare Data Utilization Process . . . . . . . . . . . . . . . . . 3.2 Smart Healthcare Data Governance Principles . . . . . . . . . . . . . . . . 3.2.1 Healthcare Data Quality Control . . . . . . . . . . . . . . . . . . . . 3.2.2 Safety, Confidentiality, and Transparency of Healthcare Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Smart Healthcare Data Governance . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

57 57 57 58 61 63

Information Exchange and Fusion in Smart Healthcare . . . . . . . . . . . 4.1 Information Exchange in Smart Healthcare . . . . . . . . . . . . . . . . . . . 4.1.1 Healthcare Information Exchange . . . . . . . . . . . . . . . . . . . 4.1.2 Healthcare Information Exchange Supervision . . . . . . . . 4.2 Smart Healthcare Information Fusion . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Healthcare Information Fusion Scenarios and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Multi-Modal Healthcare Information Fusion . . . . . . . . . . 4.3 Information Fusion Framework for COVID-19 Treatment . . . . . . 4.4 Risk Analysis and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

69 69 69 71 72

2.5

3

4

5

Knowledge Inference and Recommendation in Smart Healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Knowledge Inference and Graph Construction in Smart Healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Healthcare Knowledge Inference . . . . . . . . . . . . . . . . . . . . 5.1.2 Healthcare Knowledge Graph Construction . . . . . . . . . . . 5.2 Healthcare Knowledge Recommendation . . . . . . . . . . . . . . . . . . . . 5.3 Healthcare Knowledge-Based Personalized Service . . . . . . . . . . . 5.3.1 Personalized Healthcare Service Mechanism . . . . . . . . . . 5.3.2 Personalized Healthcare Service Design . . . . . . . . . . . . . . 5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

45 47

63 65 67 67

72 74 76 78 80 81 81 82 83 85 88 88 89 90 91

Contents

6

7

8

Non-contact Physical and Mental Health Monitoring . . . . . . . . . . . . . 6.1 Non-contact Vital Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.1 Non-contact BP Monitoring and Hypertension Risk Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.2 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Non-contact Mental Health Monitoring . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Non-Contact Depression Detection and Risk Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Comparative Experiment with the Baseline Model . . . . . . . . . . . . 6.4 Visual Facial Feature Comparative Test . . . . . . . . . . . . . . . . . . . . . . 6.5 Risk Analytics and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xi

93 93 93 100 104 104 113 116 119 119 122

OHC Physician Personalized Recommendation . . . . . . . . . . . . . . . . . . 7.1 OHC Physician Recommendation Framework . . . . . . . . . . . . . . . . 7.1.1 Physician Recommendation Based on Similar Patients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.2 Enhanced Recommendation Based on Similar Physicians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.3 Answer Quality Empowered Physician Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Risk Analysis and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

125 126

Data-Driven Cancer Screening and Risk Analytics . . . . . . . . . . . . . . . 8.1 Data-Driven Cancer Screening and Diagnosis . . . . . . . . . . . . . . . . 8.2 Cancer Screening Framework Fused with Attention Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Intelligent UGI Cancer Screening . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Gastroscopy Report Text Vectorized Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.2 Gastroscopy Report Feature Attention Extraction . . . . . . 8.3.3 Gastroscopy Report Attention Merging and Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.4 Gastroscopy Report Classification and Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.3 Benchmark Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Risk Analytics and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

141 142

127 128 130 131 131 131 138 139

143 145 145 146 148 149 150 150 151 152 152 155 159

xii

9

Contents

ICU Mortality Prediction and Risk Analytics . . . . . . . . . . . . . . . . . . . . 9.1 ICU Risk Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Feature Engineering in ICU Mortality Prediction . . . . . . . . . . . . . . 9.3 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Risk Analytics and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

161 162 163 166 166 168 168 171 173

10 Minimally Invasive Surgery Quality Control . . . . . . . . . . . . . . . . . . . . . 10.1 Surgery Quality Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Surgery Quality Control Framework and Methods . . . . . . . . . . . . . 10.2.1 Quality Control Framework . . . . . . . . . . . . . . . . . . . . . . . . 10.2.2 Quality Control Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Risk Analytics and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

175 175 177 177 178 181 181 183 184 186 188

11 Intelligent Hospital Operation Management and Risk Control . . . . . 11.1 Introduction to Intelligent Hospital . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Intelligent Hospital Operation Management . . . . . . . . . . . . . . . . . . 11.2.1 The Evolution of Intelligent Hospital Operational Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2.2 Hospital Human Resource Management . . . . . . . . . . . . . . 11.2.3 Intelligent Medical Consumables and Equipment Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Patient-Centered Hospital Service Optimization . . . . . . . . . . . . . . 11.3.1 The Evolution of Intelligent Hospital Services . . . . . . . . 11.3.2 Outpatient and Emergency Services Optimization . . . . . 11.3.3 In-Hospital Services Optimization . . . . . . . . . . . . . . . . . . . 11.4 Quality Perception of Clinical Diagnosis and Treatment . . . . . . . . 11.4.1 Smart Healthcare Systems in Hospital . . . . . . . . . . . . . . . 11.4.2 Diagnosis and Treatment Quality Perception . . . . . . . . . . 11.4.3 Diagnosis and Treatment Quality Optimization . . . . . . . . 11.5 Risk Control and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.1 Risks for Intelligent Hospital Management . . . . . . . . . . . 11.5.2 Intelligent Hospital Risk Control . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

189 189 190 191 192 193 194 194 195 197 199 199 200 201 202 203 204 205

Contents

12 Recapitulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 Smart Healthcare Engineering Management . . . . . . . . . . . . . . . . . . 12.2 Future Research Directions of Smart Healthcare Engineering Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.1 Healthcare Data Full-Cycle Security Protection and Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.2 Healthcare Systems Cross-Domain Collaborations . . . . . 12.2.3 Robot-Assisted Healthcare Operation Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xiii

207 207 210 210 210 211 211

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

Chapter 1

Basics of Smart Healthcare Engineering Management and Risk Analytics

1.1 Smart Healthcare The deep integration of a new generation of information technologies with the healthcare industry has entered a new era, accelerating the reformation of healthcare management through the processes of digitization, internetization, and intelligencification. In such a process, varied concepts surrounding the evolution of the healthcare industry were born, such as Remote Healthcare, Internet of Medical Things (IoMT), Internet Healthcare, Integrated Health Services, Digital Healthcare, eHealth, mHealth, etc. These concepts, categorized collectively as “Smart Healthcare”, are already contributing to the re-distribution of high-quality medical resources to the broader audiences. Smart Healthcare aims to integrate and analyze long-term medical and health-related data collected both inside and outside hospitals, such as medical images (radiology, endoscopic, and pathology images), test reports (medical records, routine medical test reports), structured documents (medical inspections documents, third-party evaluation documents, and environmental data), and unstructured data (social media data, health app data, etc.) With the integration of both in-hospital and out-hospital data, smart healthcare is inspiring the integration of medical treatments and population health management. Traditionally, healthcare decision-making is based on linear management and is driven by hospital diagnosis and treatment. On the contrary, by progressively playing a significant role in fields such as chronic illness management, cancer screening, eldercare services, and daily health management, smart healthcare is creating a full-cycle healthcare process. Smart Healthcare Engineering Management (SHEM) is a new concept derived from the mechanisms that allow next-generation information technologies to take root in smart healthcare systems. To fulfill the goal of providing convenient, low waiting time, and low-cost healthcare services, the development of smart healthcare corresponds to the Internet’s evolution regularly, as it is now transitioning from a technical subject to an interdisciplinary social and resource-based subject. The emergence of smart healthcare systems and applications is accelerating the collaboration and systemization of healthcare resources in various scenarios, triggering significant © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_1

1

2

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

revolutions in the healthcare industry’s development paradigms and service systems. Simultaneously, researchers seek to utilize decision-making, optimization service, and cost-cutting management within the framework of smart healthcare to address the problem of limited high-quality medical resources and high medical expenses. In general, smart healthcare engineering management needs to encompass the whole medical and health system process, beginning with disease prevention and selecting medical services before admission, continuing with diagnostic and treatment services after entry, and concluding with prognostic nursing after discharge and rehabilitation. As a crucial component of smart healthcare or any healthcare-related topic, risk is one of the most important aspects that must not be neglected. Many patients and physicians share concerns about healthcare quality, data privacy, and ethics. For example, even though researchers have shown that smart healthcare technology can usually prevent misdiagnosis better than most doctors, the vast majority of patients do not identify the diagnosis results given by artificial intelligence due to a variety of reasons. As a result, identifying, analyzing, and resolving risks in the development and use of smart healthcare will be an essential management issue for the future development of the smart healthcare industry. In the following part of this book, we first illustrate the definition, importance, and developing history of the concept of “smart healthcare”. Changes surrounding the healthcare industry, including the revolution of full-cycle healthcare, and the trends for specific smart healthcare systems are given in this chapter and Chap. 2. The theories of big data governance, information utilization, and knowledge service are discussed separately in Chaps. 3, 4 and 5. Smart healthcare models from specific scenarios are explored in Chaps. 6–11, including health monitoring, online physician recommendation, cancer screening, and diagnosis support, ICU mortality and risk analytics, surgery quality control, and intelligent hospital operation management.

1.1.1 Importance of Smart Healthcare From a global viewpoint, next-generation information technologies are ushering in a new era of healthcare management with digitalization, networking, and intelligence. Since the World Health Organization (WHO) released the first digital health intervention guidelines on April 17th, 2019, governments worldwide have developed health development strategies according to their respective national contexts, which intend to encourage the utilization of next-generation information technologies to improve health service systems. For example, NSF launched the “Smart Health and Biomedical Research in the Era of Artificial Intelligence and Advanced Data Science” (SCH) in 2021 to support and maintain the development of smart healthcare in the United States. In addition, NIH released the “Data Science Strategic Plan” in 2018, which explored the revolutionary changes brought about by technologies, such as machine learning, deep learning, and virtual reality in biomedical research. Meanwhile, the European Union’s Horizon Program launched the “Big Data for Medical Analytics”

1.1 Smart Healthcare

3

research project focused on human health and chronic illness management. Furthermore, the German Research Consortium (DFG) coordinated the establishment of the “NFDI4Health” major information infrastructure project in 2020 to provide longterm funding for the construction of the German national health data infrastructure NFDI and scientific research based on personal health data. Last but not least, Chinese health authorities and medical institutions at all levels have actively promoted smart healthcare management. Academia has similarly conducted considerable research into the integration and innovation of next-generation information technologies and healthcare. For example, Science released a special issue on “Global Health” in September 2014, outlining the world’s most significant achievements in vaccine development, new infection response tactics, rapid diagnostics, and mental and neonatal health promotion. Nature Medicine released a special edition on “Medicine in the Digital Age” in January 2019, which investigated the role of big data and artificial intelligence in novel infections. MIS Quarterly released a special edition on the influence of information technology on the management of chronic disease processes in March 2020 (Bardhan et al. 2017). In June 2021, Information Systems Research released a special issue titled “Unleashing the Power of Information Technology for Strategic Disaster Management” to help worldwide researchers utilize information technology to enhance disaster management (Abbasi et al. 2021). Furthermore, prominent management science publications such as Management Science, Operations Research, Journal of Management Science, and Management World have published many representative academic results in the areas of hospital operation and management, regional data sharing, and online medical services based on next-generation information technology (Savva et al. 2019; Ouyang et al. 2020). In all, smart healthcare is becoming one of the most prominent research subjects in the academia and healthcare industry. However, the theoretical research on smart healthcare management still needs to theoretically explore the architecture, processes, risk management, and specific scenarios of smart healthcare management.

1.1.2 Definition of Smart Healthcare It could be argued that the name “Smart healthcare” originated from IBM’s concept of “Smart Planet” in 2009. However, there is no uniform and precise explanation of the connotation of smart healthcare. Röcker et al. (2014) believe that smart healthcare is an information ecosystem based on the Internet of Medical Things (IoMT) that delivers data analysis and decision support services to a diverse set of users using next-generation information technologies such as artificial intelligence and cloud computing. Suzuki et al. (2013) put forward that the main application scenarios of smart healthcare are pre-diagnosis and preventative health care services. The authors argued that smart healthcare should involve employing small wearable devices to gather user-health and vital data and assess user living environment and health state. Pramanik et al. (2017) pointed out that smart healthcare

4

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

is an emerging and promising research topic “at the intersection of medical informatics, public health, and commerce”. Catarinucci et al. (2011) believed that smart healthcare is based on a healthcare data center that includes electronic medical records, automation, formalization, and intelligence as performance indicators. In all, it is consensus that Smart Healthcare incorporates a comprehensive application of Internet of Things (IoT), Frequency Radio Technology with embedded wireless sensors, cloud computing, and other new information technologies to create an efficient information support system. It also combines technology such as wearable devices, IoT, and 5G mobile network to access and process information dynamically, to link individuals, materials, and healthcare institutions together. It can promote interaction among all parties involved in the healthcare industry, ensuring that each party receive the services they require, assisting parties in making informed decisions, and facilitating the sensible allocation of resources. The definition of “Smart Healthcare” is explored further by Yin et al. in (2018), who defined smart healthcare as “healthcare systems… when they have a decisionmaking ability.” Yin et al. argued that such ability is the outcome of data analytics and “information distillation.” The authors further defined the major tasks of smart healthcare as the following: daily prevention, daily diagnosis, clinical diagnosis, clinical treatment, and daily treatment. They further stressed that smart healthcare enables healthcare to break through the barrier of in-hospital treatment and move on to daily care. In all, the definition presented by Yin et al. (2018) is one of the latest and more sophisticated versions of the smart healthcare framework. However, the literature does not conform to a single “gold standard” definition, even though there are high similarities between each design of smart healthcare framework, together with higher number of published articles each year (Fig. 1.1). This book defines smart healthcare similar to the definition presented by Yin et al., albeit different in some core ideals. Smart healthcare is medical and health-related

Fig. 1.1 Number of articles with the title “Smart Healthcare” in Google Scholar

1.1 Smart Healthcare

5

healthcare systems augmented with next-generation information technologies, such as Wearable Medical & Health Devices (WMHD), Internet of Things (IoT), and Highspeed Low-latency Mobile Network. While Yin et al. considered the decision-making ability of healthcare as the core characteristics of smart healthcare, we consider the decision-making supporting ability as a result of the connections of data and information due to the involvement of next-generation information technologies. Furthermore, we separate the tasks of smart healthcare into the following: Disease Prevention, Clinical Diagnosis, Hospital Treatment, Surgical Control, Disease Rehabilitation, Long-term Chronic Condition Care, and Hospital Operation Management. Similar to other definitions of smart healthcare, our definition of smart healthcare stress the importance of big data utilization. However, we further utilize the DIKW pyramid and consider the process of Data-Information-Knowledge as the most important. In all, our definition of smart healthcare compasses a more extended territory of the healthcare industry. Smart Healthcare Engineering Management focuses on “how to plan, engineer, organize, and manage” complex smart healthcare systems utilizing next-generation information technologies in order to resolve the existing problems of the healthcare industry and achieve the vision of a “smarter, safer, ‘systems serve people’ healthcare”. Synergizing human resources, information, and assets, Smart Healthcare Systems are healthcare service systems utilizing next-generation information technologies to achieve intelligent services such as data collection and fusion, knowledge generation, graphing and personalized service, and risk analysis and control. These systems are implemented in the framework of Smart Healthcare both in-hospital and in-community, encompassing the whole healthcare cycle such as disease prevention, diagnosis, and assessments, hospital management, ICU monitoring, surgical control, rehabilitation management, etc.

1.1.3 Related Concepts of Smart Healthcare Next-generation information technologies are introducing a technological revolution, accelerating healthcare management toward digitization, networking, and intelligence. Several significant concepts, including Telemedicine, the Internet of Medical Things, Internet healthcare, Integrated Health Services, and Digital Health have emerged due to this process. All of these new and diverse concepts have played an essential role in addressing the insufficient and uneven distribution of medical resources. Furthermore, with the continued penetration of next-generation information technologies throughout the entire life cycle of an individual’s health management process, the ubiquitous resources of medical and nursing personnel, materials, space, and systems are becoming more integrated and collaborative. As a result, it is giving rise to new academic concepts of smart healthcare management in line with the characteristics of the new information era to guide scientific research and practical exploration in related fields.

6

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

It is then critical to define the following terms concerning health and technology: telemedicine, Internet of medical things, Internet healthcare, integrated health services, digital health, and smart healthcare. Table 1.1 compares the connections and differences between smart healthcare and current related concepts. The World Health Organization defines telemedicine as information and communication technology for medical care practices such as diagnosis, treatment, consultation, health education, and medical information distribution (World Health Organization 2010). It allows patients to get consultations and treatment from field specialists via remote audio and video exchanges between doctors and patients. Kim et al. (2020) conclude that telemedicine provides remote information interaction services between physicians and patients. Medical IoT improves in-hospital and out-ofhospital healthcare management, health literacy, and patient’s physical health by connecting and using medical equipment, wearable devices, and various biosensors (Rani et al. 2017). Gatouillat et al. (2018) state that Medical IoT has enhanced health care quality by successfully integrating with Artificial Intelligence (AI). IoMT increases the real-time monitoring of health indicators and the level of mastery of real-time physical health situations by physicians and patients. Internet healthcare originated in the United States, along with the creation of several Internet healthcare platforms, including Teladoc, Good Doctor, and PatientsLikeMe during recent years. Bhargava and Mishra (2014) confirmed that Internet Healthcare could help reduce regional differences in medical conditions and promote graded diagnosis and treatment policies. The above concepts with technology as the core driver have enriched healthcare management’s theoretical and methodological systems and encouraged the development and innovative practice of healthcare management. In response to the fragmentation of the existing healthcare service system and the lack of integration of healthcare service providers at all levels, the World Health Organization released the Global Strategy for People-Centered Integrated Health Services in 2015. The content calls for a shift from the basic model of health services at the level of service, management, and funding provided to achieve safe, highquality, affordable healthcare, which focuses on disease prevention, diagnosis, and treatment (World Health Organization 2015). Integrated health management delivers services based on individual needs and preferences via technology-driven resource integration, such as health promotion, disease prevention, clinic treatment, and coordinating lifelong coherent services at all levels. Globally, digital health is growing as a new concept-driven by digital technologies to promote health literacy and associated applied practices in healthcare management, which has been extended to broader scientific concepts and technologies. A subset of the definition of digital health is given in the World Health Assembly resolution of May 2018, which includes eHealth (eHealth), mHealth (mHealth), and telemedicine. From the perspective of technological development, eHealth and mHealth are becoming important components of digital health (Labrique et al. 2020). Among them, eHealth uses information and communication technologies to support health and health-related domains (Oh et al. 2005). The World Health Organization defines mHealth as medical and public health practices supported by mobile devices (Rowland et al. 2020).

Technology driven

Single complex system

Diagnosis; treatment; rehabilitation

Technology driven

Human Synergy

Single complex system

Prevention; care; diagnosis

Teleconsultation; remote guidance; remote education; remote training

Fundamental Idea

Resources synergy

System features

Main function

Real-life Application

Smart ward, smart pharmacy; smart equipment; Smart consumable management

Synergy of objects

Internet of Medical Things

Concept

Telemedicine

Features

Teleconsultation; online follow-up; online health support

Prevention; healthcare; diagnosis

Single complex system

Human synergy; information Synergy

Technology driven

Internet healthcare

Table 1.1 Comparative analysis of smart healthcare and related concepts

Collaborative services for health promotion; disease prevention; Treatment; Rehabilitation

Healthcare; diagnosis; treatment

Multiple integrated complex systems

Human synergy; information synergy

Technology driven; resource integration

Integrated health services

Multiple integrated complex systems

Human synergy; synergy of objects; information synergy

Technology driven; Data support; resource integration

Smart healthcare

eHealth, mHealth, telemedicine

The whole process of health care services

Prevention; healthcare; Prevention; healthcare; diagnosis; treatment; diagnosis; treatment; rehabilitation rehabilitation

Multiple integrated complex systems

Human synergy; information synergy

Technology driven; data support

Digital health

1.1 Smart Healthcare 7

8

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

In all, it is clear that all the above concepts aim to bring an evolution to the current healthcare system through the utilization of next-generation information technologies, artificial intelligence, machine learning, big data, etc. All concepts have the similar objective of increasing the ability and efficiency of healthcare institutions through the integration between next-generation information technologies and traditional hospital functions and infrastructures. While similar in functions, each concept focus on specific healthcare functions or organizations. In this book, the concept “Smart Healthcare” includes the functions of every single one of these comparable concepts, as shown in Table 1.1.

1.2 Full-Cycle Healthcare Management Revolution The development of smart healthcare theories and next-generation information technologies has triggered a revolution in the healthcare industry. Specifically, the traditional focus on in-hospital procedures, i.e. diagnosis and treatments are expanding toward out-hospital and daily health management, which encompasses the full-cycle of human healthcare management processes: disease prevention, health management, transmittable disease control, disease rehabilitation, elderly care, and even insurance services. As a result, many countries across the world are encouraging the development of health management. For example, the Singapore government has formed the Health Promotion Board, which promotes citizens’ adoption of healthy living practices by providing evidence-based medical information and disease prevention programs at homes, workplaces, and schools. All Singaporeans are encouraged to regulate diet, exercise frequently, and get preventative screenings, which all assist in minimizing the probability of disease development and the requirement for medical treatment. From the perspective of process management, smart healthcare process management focuses on the evolution of healthcare technology to improve human health at all levels via the application of smart technologies. It includes themes such as disease prevention and health risk management, assisting diagnosis and treatment, intelligent hospital management, and rehabilitation are changing from disease treatment to health management, supporting the transition to smart healthcare management.

1.2.1 Disease Prevention and Health Risk Management The most significant change brought by smart healthcare is the transformation of traditional disease diagnosis and treatment into a full-cycle healthcare management process, including preventive health risk management. Healthcare management is critical to disease prevention, as it places a greater emphasis on patient self-management, argued Tian et al. (2019). By increasing the health literacy and the self-management ability of individuals, a downgrade in hospitalization cost could

1.2 Full-Cycle Healthcare Management Revolution

9

be expected, that is, emergency treatment could transform into regular visits, and regular visit to hospital could be resolved by patients at home. This process is crucial to alleviate the pressure of the current healthcare system, which is already in overload in many nations which do not have the required infrastructure to support the diversified and high-quantity healthcare need of the population (Willard-Grace et al. 2013). Today, the new health risk management emphasizes real-time patient selfmonitoring, fast feedback of health data, and early intervention of certain behaviors. Wearable devices such as Apple Watch, Huawei sport trackers, and everyday carry devices such as smartphone contributes to health risk management without obstruct normal user behaviors. This transforms the traditional in-hospital, high-accuracy, but situational screening of population health into a low-accuracy but consistent monitoring of user health. It minimizes the risks connected with the condition while also providing potential data for healthcare practitioners to monitor the disease’s prognosis (Andreu-Perez et al. 2015). Although the trend of health monitoring is growing throughout the world, many of the applications developed are still independent and in trail phase. Various options exist for users to choose from when monitoring sleep quality, body weight, diet intake, blood sugar, etc. However, data generated from these “apps” are distinct in formats, and extremely low in information density. Currently, these data are rarely considered in hospital, but are mostly used for self-awareness purpose, which varies greatly in effects. To make use of these data, various industrial standards need to be established, such as data storage formats, encryption standards, privacy policies, etc. Data with low information density need to be analyzed prior to the utilization in hospital, while the authenticity of such data need to be verified.

1.2.2 Clinical Assistant Diagnosis and Treatment Management Smart healthcare provides medical data assistance for clinical diagnosis and treatment by expanding the capability of Electronic Health Record (EHR) at all medical institutions. It is capable of offering a variety of smart health care services with varying characteristics. Applying artificial intelligence, surgical robots, and virtual reality would make disease detection and treatment more efficient. The data collecting, modeling, and analysis processes are expected to be real-time in the smart healthcare system. The structured, semi-structured, and unstructured data collected in the system are modeled and analyzed efficiently and reliably. It uses all types of data mining technology, big data analysis technology, and knowledge discovery theory. The decisionmaking and prediction results are applied to specific situations to obtain better action plans or solutions. Clinical text mining, predictive modeling, patient similarity analysis, genome analysis, and other approaches are examples of these techniques. It has accomplished some results by using artificial intelligence to construct clinical decision support systems, such as hepatitis, lung, and skin cancer detection systems.

10

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

The knowledge and insights accumulated by big data can make disease diagnosis and treatment more intelligent. Currently, researchers are using AI to develop clinical decision support systems such as the detection of diabetic retinopathy, gastrointestinal illness, and skin cancer (Esteva et al. 2017; Mohapatra et al. 2021; Gulshan et al. 2016). AI diagnoses outperform human doctors in terms of precision. In addition, AI-based systems are sometimes more accurate than experienced doctors, particularly in pathology and imaging. For example, IBM Watson provides decision support through in-depth analysis of all clinical and literature data. By using the clinical decision support system, physicians can give expert recommendations based on intelligent algorithms, which could make diagnosis more accurate, reduce iatrogenic risks, and enable patients to receive timely and appropriate treatment. Artificial intelligence innovation has further expanded medical collaboration, which has further extended the scope of medical cooperation originally based on hospital boundaries and formed a cross-institutional and cross-regional collaboration as a whole. In addition, the integration of blockchain and other technologies has greatly increased the credibility, security, and traceability of medical information transmission on the medical network.

1.2.3 Rehabilitations and Follow-Up Visits Management The advancement of next-generation information technologies is critical for improving the effectiveness of patient follow-up, as it could optimize the process of rehabilitation, follow-up, and reexamination for patients. It relies on developed online service platforms to efficiently conduct patients’ online rehabilitation, health assessment, monitoring, which could potentially alleviate readmission pressure and strengthen the entire hospitalization process management cycle. Therefore, it is important to establish a method of rehabilitation follow-up that incorporates both online and offline components and serves to redirect patients throughout the healthcare system appropriately. Consequently, it is necessary to make extensive use of dynamic programming and machine learning-based prediction algorithms to develop a recommendation framework for personalized rehabilitation and follow-up time based on a smart healthcare system and establish a model for re-entry and death risk assessment of reexamination for patients. Wearable devices and non-contact monitoring devices, as such, are capable of collecting data, such as movement and acceleration data, vital signs data (such as body temperature, pulse rate, respiration rate, blood pressure, etc.), social media text and video data. The data could be utilized to assure the completion of rehabilitation schedules and suggest a healthier alternative of behaviors. Suggestions of followup visits could also be scheduled through such a process, albeit detailed algorithms need to be developed. Similarly, by taking these data into consideration, a health risk prediction model using machine learning and deep learning methods could be developed, and patient automatic readmission and mortality risk assessments could be achieved outside hospitals.

1.3 Smart Healthcare Engineering Management

11

1.3 Smart Healthcare Engineering Management Smart healthcare helps the transition from traditional medical treatment to patientcentered healthcare processes, generating an innovative idea of from “people adapt system” to “system serves people”. However, existing medical systems and data are typically located and stored in hospitals or medical organizations separately. Between the passive barriers of distance and protocol differences and active barriers of data protection policies of each institution, it is clear that data sharing and fusion analysis cannot be accomplished among these systems as of today. As a result, patients are unable to access comprehensive electronic health records. On one hand, patients seek medical treatment at different medical institutions, repeated examinations and tests are frequently required, which represent a significant waste of medical resources. On the other hand, physicians are unable to obtain patients’ treatment and medication history, resulting in iatrogenic risks and possible life losses. We propose a smart and interconnected healthcare system framework to address these issues. The framework enables healthcare organizations to deploy big data platforms and technologies, delivers ubiquitous healthcare solutions, and expands smart services to minimize mistakes and healthcare costs. The conceptual foundation of the smart linked healthcare system that enables smart healthcare management is visualized in Fig. 1.2. The basic concept of smart healthcare management is the “1 + X + N” healthcare service framework, as demonstrated in the architecture of the smart healthcare system. The foundation is a single service resource center that integrates people, material, and information resources. Human resources represented in this layer are physicians, nurses, patients, and general practitioners. Material resources include equipment, pharmaceuticals, and consumables. A huge amount of information could be obtained from various sources, including hospitals, insurance firms, medical examination centers, communities, medical commissions, pharmaceutical companies, etc. The core idea behind the “X-dimension” layer is resource virtualization and resource collaborative scheduling. It is typically constituted of life-cycle data service, adaptive cross-regional linkage, and hyperspace positive perception. For example, the IoT collects and transmits healthcare data distributed across various platforms in the process of life-course data service and provides intelligent analysis and decisionmaking services through multi-modal data, such as birth date, admission data, physical examination data, discharge data, diagnosis, and treatment data, and medical history data. Wearable devices, mobile terminals, routers, switches, servers, wireless networks, broadband, and other hardware facilities are commonly used in adaptive cross-regional linking. Families, hospitals, medical institutions, regulatory authorities, and other organizations are interconnected via the IoT to form a comprehensive, intelligent, interconnected network. Medical equipment such as inspection equipment, monitoring equipment, treatment equipment, wearable equipment, data processing equipment,

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

Fig. 1.2 Architecture of smart healthcare management

12

1.4 Smart Healthcare Engineering Management and Risk Analytics

13

display equipment, medical supplies, and materials are all part of the network. Multisource heterogeneous data sets are kept in current departmental, regional, and industrial information systems, referred to as positive hyperspace perception. They could be classified as organized, semi-structured, or unstructured. These data are derived from residents’ electronic health records within hospitals and external sources such as the government, insurance agencies, and Internet platforms. Many different types of health data could be utilized in this framework, such as web and social media data, vital and video monitoring data, transaction and insurance data, biometric data, and user-generated data are examples of these health data types (e.g., doctor prescriptions, e-mails, paper documents) (IHTT 2013). Finally, in the services and application layer, various services, such as humanmachine collaborative diagnosis and treatment, medical and health collaboration, epidemic prevention and control, and intelligent hospital management, are deployed on different application platforms via the interoperable interface in the service layer and the intelligent engine in the cloud. Furthermore, the infrastructure and service content in the application procedures are updated regularly in response to dynamic changes and input from residents’ requirements. Overall, our proposed system design enables effective collaboration among healthcare stakeholders, and it can decrease healthcare expenditures, improve process management, and achieve a better quality of service.

1.4 Smart Healthcare Engineering Management and Risk Analytics Risk analytics represent one of the most important procedures in Smart Healthcare Engineering Management (SHEM). Originated from business and finance, the definition of systematic risks focus on any potential events or a chain of circumstances that lead to instability or decrease of confidence of the implemented system. Other definitions exist, but almost all focus on three main themes: public confidence, system stability, and actual harm caused to the public economy. Comparing to the financial industry, the risks in healthcare industry focuses differently. The stability of the whole healthcare system does not depend on public confidence, and are much more reliable than the economic system. However, the stability of individual smart healthcare system largely depends on whether risks could be resolved and mitigated. The actual harm potential of healthcare systematic risks is mostly in a micro scale than a macro scale in the financial industry: it is most likely that only individual lives would be harmed, but also most likely in very serious ways, even mortality. However, these micro-scale harms would be translated into a macro-level public untrustworthy of the healthcare system through the severity of incidents. In all, the stability of the healthcare system as a whole is not as unstable as the financial industry, but the focus of healthcare system is still the same: public/physician confidence/ trust, stability of individual system, and harm reduction.

14

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

It is within this context that risk analytics of the healthcare industry becomes important. To define Smart Healthcare Risk Analytics, we first need to define what Smart Healthcare Risk Analytics is NOT. First of all, risk analytics is not risk analysis. Risk analysis focus on the qualitative depiction of potential risk factors, while risk analytics focus on quantitatively depict these factors while providing potential solutions. In the case of healthcare, the potential risks in each system need to be quantified instead of qualified, since any unresolved and unquantified risks induce fear in both institution managements and the general public. Secondly, risk analytics is not risk management. Risk management focus on determining whether any risk is “worth taking” within an organization to achieve greater economic gain. In smart healthcare engineering management, there is no systematic risk that is worth taking. Risk Analytics originated from the financial industry, as such we could extract aspects from the financial industry implementation of risk analytics to the healthcare industry. For example, the risk analytics in finance focus on four “L”s: Liquidity, Leverage, Losses, and Linkages. We could adapt these four concepts into the Smart Healthcare Risk Analytics, such as: Liquidity refers to the amount of margin allowed in any risk event. For example, patients who is chronically ill but are only finding a physician represent a much larger liquidity comparing to an ICU patient. The higher the Liquidity, the lower the risk. Loss refers to the amount of direct damage done by the potential outcome of the risk comparing to doing nothing at all. Of course, if we consider the cost of opportunity, we could also compare it with “doing the optimal thing”, but we don’t since it is what Leverage considers. The higher the loss, the higher the risk. Leverage refers to the possible seriousness of the outcome in healthcare. The higher the leverage, the more serious any losses could be in any situation. Remember, in this case we are looking for the “absolute maximum loss = leverage”. For example, the leverage is higher when a surgical tool is left in patient body than when a patient chose an suboptimal physician. The higher the leverage, the higher the risk. Linkage refers to the spillover effect cost by the corresponding loss. In this case we have to make plan for both “average linkage” and “maximum linkage”, similar with financial industry. The higher the linkage, the higher the risk. In the case of Wei Zexi, for example, the Probabilities of Loss is “high”, due to the varies misinformation online that anyone could fall for. The Loss itself is “low” for the physician selection process, since there are many options for patients to choose from and the choice is in patient’s hand. The Leverage is “low”, considering that the condition of the patient could not be much worsened by the choice of a suboptimal physician. The Liquidity is “low” as well, since the patient is in cancerous condition. The Linkage is “high” for both “average linkage” and “maximum linkage”, since the choice of this physician could very much link to an unproven experimental treatment. In this case, illiquidity, probabilities of loss, and a high network effect is the greatest risk in Wei’s case. Let’s suppose the case in which a physician misdiagnosed the condition of an early stage cancer patient. The loss is “low” since misdiagnose = nothing at all. The leverage is “high” since the opportunity cost of misdiagnose is high, especially for early stage patients. Linkage is “medium” for “average linkage”, and “high” for “maximum linkage”. Liquidity is also “high” since the patient is not yet in any critical

1.5 Theories and Models in Smart Healthcare

15

condition. In this case, the high leverage gives a high likelihood of default, and a high network effect, are the most important risks. Let’s look at another case. A young, almost always healthy patient is affected by influenza but not in critical condition. The physician ordered the patient to take the wrong pill, although with no other but placebo effect. The loss is “low” since placebo = doing nothing at all. The Leverage is “medium”, since influenza still kills at the right circumstances. The Liquidity is “high” due to not being in any critical condition and the patient is young. The Linkage is “low” since there are no sign of any linkage rather than letting the patient to recover a few days later than the optimal solution. In all this case has a low risk. In reality, these “high”, “low”, and “medium” all need to be further quantified. The average number of years to live and healthy living years lost need to be calculated separated or averaged for each case. Linkage need to be visualized and probability of linkage need to be estimated. For any smart healthcare systems, all possible scenarios need to be considered, analyzed, and its risk specified. The quantification of risk Rx could be represented with the following formula: Rx = f (Liq, Lev, Lin, Los) ∗ Px ,

(1.1)

where Px represent the probability of the case x happens, f represent the risk calculation function which take the four factors into consideration. Issues will inevitable rise according to Murphy’s law, and the eventual Los is a single case does not depend on the value of Px . Thus, no matter the value of Px , the preventing, alerting, assessing, alleviating, and resolving mechanisms for each case x need to be clearly laid out before any smart healthcare system innovations are put into place. Intensive simulations and stress tests need to be executed to understand the potential unexpected behaviors of smart healthcare systems. In all, Smart Healthcare Engineering Management and Risk Analytics is the process of identify, assess, quantify, and alleviate risks among the newly engineered smart healthcare systems prior to its application in real-life scenarios.

1.5 Theories and Models in Smart Healthcare Smart healthcare management is a closed-loop medical service that encompasses the entire spectrum of consultation, triage, diagnosis, treatment, prognosis, and prevention in the healthcare system. Therefore, it is important to provide scenario-oriented and individualized smart process management services for various user topics such as patients, physicians, and hospitals throughout the closed-loop processes to increase medical system efficiencies. Wearable devices, for example, capture personal physiological signs and other pertinent data for real-time transmission and analysis. Thus, on the one hand, it enables patients to comprehend their physical problems in realtime; on the other hand, it allows medical personnel to undertake remote monitoring

16

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

and early warning of patients’ physical situations. Furthermore, medical resources and services for hospital topics can be integrated to enhance resource scheduling and management inside the hospital. Smart healthcare outlines the many scenarios in the new era and thoroughly addresses major scientific issues related to diagnosis, treatment, operational management, and emergency management. This book introduces three comprehensive scenarios, including data governance, smart applications, and knowledge services. Figure 1.3 shows the whole structure for different scenarios in this book. Firstly, smart healthcare provides medical data governance by expanding the capability at all levels of medical institutions. The conventional clinical research model needs clinicians to collect case data actively. External variables such as departments, hospitals, and locations are frequently restricted to form a large-scale and evenly distributed case sample set. Healthcare data is a resource that offers effective support for residents’ health services, health administration, clinic diagnostic and treatment services, and medical, scientific research. It possesses intrinsic characteristics and

Fig. 1.3 The framework of this book

1.5 Theories and Models in Smart Healthcare

17

qualities such as enormous dynamics, multi-source heterogeneity, and quality disparities. The medical network environment fosters close medical associations for each specialty. A large-scale, high-quality case sample database can be formed through medical collaboration and data sharing, combined with doctors’ clinical experience and intelligent data analysis technology to discover generalized clinical patterns more easily. In this book specifically, Chap. 3 demonstrated how data governance efficiently based on logic and uniform data model can achieve the service function of healthcare big data. Chapter 4 details the information exchange and multi-modal fusion processes of healthcare big data. Secondly, smart healthcare provides knowledge-based services. With organized, complete, and interconnected data and information, organizations need to convert these information into knowledges that physicians and patients could utilize in reallife scenarios. Effective utilization in multi-source, multi-modal and massive healthcare knowledge could be achieved through knowledge services, explored based on the existing knowledge creation theory and knowledge map construction method. In this book, Chap. 5 illustrated that knowledge-based service combining individuality and diversification, the most important component in this session is the dynamic construction of multi-mode healthcare knowledge map, knowledge recommendation, and knowledge personalized services. Thirdly, smart healthcare applications have different technological scenarios. This book utilized Chaps. 6–11 to demonstrate the models of smart healthcare systems utilizing next-generation information technologies. These chapters are ordered via their respective position within the cycle of care. Specifically, Chap. 6 demonstrates the application-able for non-contact physical and mental health monitoring. Chapter 7 focuses on physician personalized recommendation of Online Health Communities (OHC), utilizing physician & patient info, and physician answering qualities. Chapter 8 illustrates a data-driven cancer screening process through deep learning and attention mechanisms, which lead to a highly interpretable and accurate automatic diagnosis process. Chapter 9 covers ICU mortality prediction and risk analytics. A three-dimensional perception technique for minimally invasive surgery and clinic quality control are presented in Chap. 10, and a forecasting method for surgical video remaining time is also proposed. Chapter 11 proposes an intelligent hospital operation management system framework, including hospital human resource management, patient-centered hospital service optimization, and clinical service quality perception. This book intend to give the readers a general understanding of the concept “Smart Healthcare Engineering Management and Risk Analytics”, its principles, and realworld examples. Smart Healthcare represent the future of the healthcare industry, as the vision of fully personalized, full-cycle, convenient, and low-cost healthcare services could be achieved through the utilization of next-generation information technologies. The challenges of the development of smart healthcare, specifically the design and engineering of homogeneous systems, and the alleviation of risks, are the core content in this book.

18

1 Basics of Smart Healthcare Engineering Management and Risk Analytics

References Abbasi A, Dillon-Merrill R, Rao H R, Sheng O et al (2021) Call for papers—special issue of information systems research—unleashing the power of information technology for strategic management of disasters. Inf Syst Res Andreu-Perez J, Leff DR, Ip HMD, Yang GZ (2015) From wearable sensors to smart implantstoward pervasive and personalized healthcare. IEEE Trans Biomed Eng 62(12):2750–2762 Bardhan I, Chen H, Karahanna E, Chen H (2017) Call for papers MISQ special issue on the role of information systems and analytics in chronic disease prevention and management. MIS Q 41(1):1–3 Bhargava HK, Mishra AN (2014) Electronic medical records and physician productivity: evidence from panel data analysis. Manage Sci 60(10):2543–2562 Catarinucci, Esposito A, Tarricone L, Zappatore M, Colella R (2011) Smart data collection and management in heterogeneous ubiquitous healthcare biomedical engineering. In: Laskovski AN (ed) Biomedical engineering: trends in electronics, communications and software. InTech, pp 685–710. ISBN: 978-953-307-475-7 Esteva A, Kuprel B, Novoa RA, Ko J et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7638):115–118 Gatouillat A, Badr Y, Massot B, Sejdi´c E et al (2018) Internet of Medical Things: a review of recent contributions dealing with cyber-physical systems in medicine. IEEE Internet Things J 5(5):3810–3822 Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A et al (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA—J Am Med Assoc 316(22):2402–2410 IHTT (2013) Transforming health care through big data strategies for leveraging big data in the health care industry. http://ihealthtran.com/wordpress/2013/03/iht%C2%B2-releases-big-dataresearch-reportdownload-today/ Kim S-H, Tong J, Peden C (2020) Admission control biases in hospital unit capacity management: how occupancy information hurdles and decision noise impact utilization. Manage Sci 66(11):5151–5170 Labrique A, Agarwal S, Tamrat T, Garrett M (2020) WHO Digital Health Guidelines: a milestone for global health. NPJ Digi Med 3(1):1–3 Mohapatra S, Nayak J, Mishra M, Pati GK et al (2021) Wavelet transform and deep convolutional neural network-based smart healthcare system for gastrointestinal disease detection. Interdisciplinary Sciences: Computational Life Sciences Oh H, Rizo C, Enkin M, Jadad A (2005) What is eHealth (3): A systematic review of published definitions. J Med Internet Res 7(1):e110 Ouyang H, Argon NT, Ziya S (2020) Allocation of intensive care unit beds in periods of high demand. Oper Res 68(2):591–608 Pramanik MI, Lau RYK, Demirkan H, Azad MAK (2017) Smart health: big data-enabled health paradigm within smart cities. Expert Syst Appl 87:370–383 Rani S, Ahmed SH, Talwar R, Malhotra J et al (2017) IoMT: a reliable cross layer protocol for Internet of multimedia things. IEEE Internet Things J 4(3):832–839 Röcker C, Ziefle M, Holzinger A (2014) From computer innovation to human integration: current trends and challenges for pervasive health technologies. In: Holzinger A, Ziefle M, Röcker C (eds) Pervasive health. Human–computer interaction series. Springer, London Rowland SP, Fitzgerald JE, Holme T, Powell J et al (2020) What is the clinical value of mHealth for patients? NPJ Digit Med 3(1):1–6 Savva N, Tezcan T, Yildiz Ö (2019) Can yardstick competition reduce waiting times? Manage Sci 65(7):3196–3215 Suzuki T, Tanaka H, Minami S, Yamada H, Miyata T (2013) Wearable wireless vital monitoring technology for smart health care. In: International symposium on medical information and communication technology, pp 1–4

References

19

Tian S, Yang W, Le GJM, Wang P, Huang W, Ye Z (2019) Smart healthcare: making medical care more intelligent. J Glob Health 3(3):62–65 Willard-Grace R, DeVore D, Chen EH, Hessler D, Bodenheimer T, Thom DH (2013) The effectiveness of medical assistant health coaching for low-income patients with uncontrolled diabetes, hypertension, and hyperlipidemia: protocol for a randomized controlled trial and baseline characteristics of the study population. BMC Fam Pract 14:27 World Health Organization (2010) Telemedicine: opportunities and developments in member states. Report on the second global survey on eHealth. World Health Organization World Health Organization (2015) WHO global strategy on people-centred and integrated health services: interim report. World Health Organization Yin H, Akmandor AO, Mosenia A, Jha NK (2018) Smart healthcare: foundations and trends in electronic design automation 12(4): 401–466. https://doi.org/10.1561/1000000054

Chapter 2

Frontier of Smart Healthcare Engineering Management

Through the advancement of a new generation of information and communication technologies, such as 5G, IoMT, machine learning, etc., the scientific community has already extensively explored the possibilities of utilizing such technologies in varied healthcare processes. From the perspective of process management, it could be argued that every single process of the cycle of health is evolving with such trends, from health monitoring and online health consultation to in-hospital diagnosis and surgery, eventually follow-up examinations and rehabilitations. For example, the process of health monitoring and assessment could be enhanced with wearable or non-contact devices to achieve 24/7 monitoring. Likewise, online consultations could be augmented with virtual reality or meta-verse concepts to enhance the consulting experience. Similarly, remote minimally invasive surgery and diagnosis supporting systems with next-generation information technologies could further alleviate current inaccuracies and iatrogenic risks within the current healthcare system. However, most of such explorations remain non-applicable for various reasons, including the lack of interpretability, low accuracy, and lack of risk provisions. However, the trends among such applications are clear. This chapter aims to explore the state of smart healthcare engineering management applications in the current healthcare industry and academia. Corresponding to the chapters following, this chapter will be separated into five portions: mental and physical health screening, online healthcare community and consultation, diagnosis supporting, in-hospital monitoring and care, and surgical quality control with nextgeneration information technologies.

2.1 Non-contact Physical and Mental Health Measurement The boundary of healthcare is expanding through the popularization of nextgeneration information technologies. The traditional “cycle of care” revolves around a passive cycle of diagnosis and treatment, as patients’ self-conscious becomes the © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_2

21

22

2 Frontier of Smart Healthcare Engineering Management

primary way of discovering the need for healthcare. The first step towards fullcycle healthcare, i.e., the cycle of prevention-diagnosis-treatment-rehabilitation, is utilizing next-generation information technologies on 24/7 health monitoring and risk alert. The industry has already attempted health monitoring through developing novel products and improving existing infrastructures. For instance, built-in gyroscopes in cellphones are utilized to monitor step amount, step length, etc., therefore used to monitor the amount of energy usage and activities (Tetin et al. 2021). Microphones are employed to detect sleep quality-related events, such as snoring, sleep talking, body movements, etc., in order to monitor and improve the sleeping quality of users (Hao et al. 2013). Novel devices such as the Apple watch are known to have saved lives through regular heart rate monitoring tasks. Concepts such as smart home health monitoring utilizing the Internet of Things are also being explored in recent years (Philip et al. 2021). The potential benefits and implications of regular health screening and monitoring are tremendous. The cost-effectiveness of a portion of health screening/preventive services could be high in terms of quality-adjusted life year for individuals, such as colorectal cancer screening for the population where age is greater than 50, cervical cancer screening for all women, hypertension screening for all adults, etc. (Yong et al. 2010). Prevention of chronic diseases could further reduce the cost of disease on the economy due to presenteeism and loss in workdays (DeVol et al. 2007). National health screening programs such as the “Screen For Life (SFL) national health screening programme” from Singapore are supporting the idea of regular health checkups to prevent cancer, while workplace enforced health screening is almost a cultural phenomenon. In all, health monitoring and screening is one of the most important parts of smart healthcare and is actively being explored by the industry. While many forms of health screening could be very cost-effective, some remain below the cost-effectiveness threshold. While screening for diabetes and colorectal cancer are widespread due to their enormous benefit to the economy and low cost, screening for mental health problems, for example, is usually extremely costly and inaccurate. On the other hand, regular monitoring of vitals might not be costly for individuals due to the low cost of readily available products. However, such monitoring usually sees little tangible returns and is hard to enforce due to privacy concerns and uncomfortableness for the regular wearing of monitoring equipment. With the support of next-generation information technologies, however, new forms of lowcost, high-accuracy, non-intrusive, even non-contact health/vital monitoring methods could significantly improve the cost-effectiveness of the related health monitoring requirements. In the following section, we introduce the new low-cost, non-contact health monitoring methods by dividing the focus into two different categories of screening: physical and mental health screening.

2.1 Non-contact Physical and Mental Health Measurement

23

2.1.1 Physical Health and Vital Monitoring Physical health screening has been the focal point of government agencies, corporations, and private individuals. Health screening could be divided into different categories, such as cancer screening, transmissible disease screening, and daily vital recording and screening. Cancer screening, such as prostate cancer, breast cancer, liver cancer, colorectal cancer, and UGI cancer screening, has already been widely adopted throughout society, which will be discussed specifically in Sect. 2.3 and Chap. 8. Transmissible disease screening, such as tuberculosis, hepatitis B, STD, and COVID-19 screening is a major topic since the outbreak of the Coronavirus Disease in early 2020, which has an abundant body of literature. Vital recording and screening, such as sleep quality, body temperature, blood pressure, blood sugar, etc., has also taken up the pace throughout the decade due to the spread of lightweight and easily handled screening instruments and smartphone-based screening algorithms and software. Daily vital screening, specifically, contributes greatly to the monitoring of chronic diseases. For example, diabetes monitoring requires regular blood sugar screening, obesity monitoring requires regular bodyweight screening, and hypertension monitoring requires regular blood pressure monitoring, obviously. Chronic disease monitoring is thus one of the most important objectives of physical health and vital screening, and its effectiveness and difficulty differ greatly for different diseases. Diabetes monitoring, for example, is often easy and accurate with proper devices. However, current blood sugar monitoring devices are intrusive and could cause pain and even infections if misused by individuals. Non-intrusive methods are being developed, albeit relying mainly on user inputs, which are intrinsically unreliable and inaccurate (Gu 2017). Similarly, although hypertension and blood pressure monitoring is non-intrusive, current contact-and-pressure-based devices are often cumbersome, could still cause significant uncomfortableness and pain, and are often inaccurate if users do not properly utilize the devices. The COVID-19 crisis has created the popularity of non-contact body temperature detection devices in public spaces, which are usually non-contact and pain-free. However, these methods are often unreliable due to not applying temperature detection devices in places where the temperature does not differ due to environmental situations. In all, vitals and physical health screening is highly complicated and differs greatly for each different circumstance. Within the different physical and vital screening categories, hypertension and blood pressure monitoring represent one of the most important, while benefiting from the development of next-generation information technologies the most. Hypertension is highly prevalent in the global community: according to Chow et al. (2013), hypertension is present in about 40% of the adult/middle-age population, impacting the lower educated population more than the highly educated. Within the population with hypertension, only 46.5% are aware of their condition. A majority of the hypertension-aware population is receiving treatment, while only about 32.5% of those have conditions controlled (Chow et al. 2013). Only a small portion of those

24

2 Frontier of Smart Healthcare Engineering Management

with hypertension symptoms receive effective treatment, which is a sign of need-forimprovement. For a disease costing $193 billion in healthcare cost and $123 billion in lost productivity in the 2011 United States, it is important that early symptoms and continued monitoring of blood pressure is made available to every corner of the society (Jacob et al. 2017). Traditional blood pressure monitoring methods include catheterization, auscultation, oscillometry, and volume clamping. Catheterization refers to the practice of inserting a catheter into the artery or vein to directly measure the blood flow pressure, which is the gold standard with minimum deviation but invasive (Mukkamala et al. 2015). Auscultation and oscillometric measurement of blood pressure is the most widespread method of blood pressure measuring in hospitals, which requires pressure on the limbs of individuals and listening or mechanically monitoring different pulses to register the blood pressure of individuals, which does not allow the continuous monitoring of blood pressure (Lewis 2019). The volume clamping measurement method does allow for the continuous monitoring of blood pressure but the equipment is cumbersome and is unrealistic in normal settings. To achieve a wider spread of blood pressure daily monitoring, a smaller device, preferably non-contact-based, usable in both social and hospital settings, and achieving the continuous monitoring of blood pressure is needed. Recently, the field of remote Photoplethysmography (rPPG, or IPPG) has been developing rapidly and has shown progress in the remote measuring of heart rate (Sun and Thakor 2016; Yu et al. 2019). rPPG technology utilizes cameras to record the small color changes in the skin to measure pluses, which created a non-contact PPG detection solution. Blood pressure could also be estimated through the extracted rPPG signals through wave feature analysis (Zhou et al. 2019). The wave analysis method is high in interpretability, albeit that the deeper features could not be extracted and environment noises could greatly affect the quality of extracted signals. On the other hand, deep learning methods grant almost no interpretability, while possessing great anti-noise ability and learning ability.

2.1.2 Mental Disorder Screening The importance of mental health screening has been rising sharply throughout the last few decades. According to the WHO (2017), about 322 million people are suffering from depression currently, with the number increased by 18.4% between 2005 and 2015. Globally, depression contributed to 7.5% of all Years Lived with Disability (YLD) and 788,000 death in 2015 (WHO 2017). The COVID-19 pandemic further exacerbated the mental health problem, as the depression symptom rate spiked from 8.5 to 27.8% in the United States, according to Ettman et al. (2020). As a common mental problem, depression is heavily influential, with high prevalence and danger. The symptoms of depression are highly varied, manifesting as a continuous light depression or an almost random, intermittent but heavy depression. Symptoms of depression could appear in both physical and mental aspects. Mental symptoms

2.1 Non-contact Physical and Mental Health Measurement

25

could include long-term relative low morale, sadness, and fatigue. Patients could express the world as black-and-white, lose interest in many accepted activities, avoid social contacts, etc. In the physical aspect, depression patients could lose control of the body and avoid physical needs, leading to sleep deprivation, organ function loss, autonomic dysfunction, etc. In the final stages of depression, suicide attempts could be performed, resulting in a 20-fold risk of suicide among depression patients compared to the general public (Rottenberg 2017; Dibeklio˘glu et al. 2018). Even with the continuous expansion of depression-affected individuals, depression is still a socially marginalized disease, where depressed individuals could be deliberately suppressed, misunderstood, and omitted from society. Clinical diagnosis and treatment of depression are negatively affected by three reasons. Firstly, the diagnosis of depression lacks biological and quantitative reasoning. The diagnosis of depression is usually carried out through questionnaires and psychological interviews, with the final decision based on physician experiences. While expert systems have been developed to take in risk factors and some empirical evidence, these systems are still largely based on depression scales, which are prone to lying and inaccuracies (Maurer et al. 2018). Moreover, the cost of depression diagnosis and treatments is usually high, with demand overwhelming physician supply. Psychologists within developing countries could amount to only 1 in 100,000, while antidepressant medication remains highly-priced in developed countries and loss in productivity cost even higher (Lépine and Briley 2011; Thomas and Morris 2003). Finally, individuals often lack scientific management for their mental problems, while institutions often lack the power of data acquisition and intervention due to data privacy requirements. From the above aspects, it is arguable that depression and mental health problems are similar to cancer for being an early screening problem: it is only through common mental health screening that the most serious consequences of depression, such as loss in productivity, high treatment cost, and mortality. The development of next-generation information technologies created new trends in the diagnosis and risk analytics of depression. For example, newer machine learning methods and AI technologies have greatly contributed to the empirical analysis of depression risk (Pampouchidou et al. 2019). Due to the correlation between long-term mood conditions and psychological health and diagnosis, affective computing has been actively utilized in the screening and supportive diagnosis of depression and anxiety disorders. Affective computing usually extracts data from individual facial expressions, voice and sound, and body movements and constructs a human-machine interface that could eventually detect and respond to the psychological status of individuals (Poria et al. 2017). The addition of relatively empirical ways of depression diagnosis has contributed to the trend of utilizing affective computing in depression diagnosis. For example, the American Psychiatric Association (APA) published “Diagnostic and Statistical Manual of Mental Disorders” (DSM), which detailed several visual/ sound based depression indicators, including uncontrollable movement, slowed body movement, decrease in social activities, etc. (Caligiuri and Ellwanger 2000; Sobin and Sackeim 1997). Compared to traditional affective computing, the field of Automatic Depression Detection (ADD) is more

26

2 Frontier of Smart Healthcare Engineering Management

restrictive in the requirement of social ethics, and the effective detection of quantitative, objective, and accurate depression factors while ensuring the privacy of individual data is becoming one of the most important questions within the field of ADD. Through the mental and physical movement features regularly recorded by visual equipment, machine learning, and deep learning methods could be applied to analyze and extract important features of depression disorder. The question of how to merge data from different sources and modals becomes the critical question of depression detection. In all, it is clear that non-contact, affective computing-based depression and mental health screening are actively being explored applying the next-generation information technologies. In the following section, we shall introduce the varied techniques, and models researchers have been utilizing to achieve the purpose of depression screening. We focus on the research that emphasizes visual cues provided by facial expressions and head movements and divide the literature into two parts: hand-crafted and deep learning features. A.

Hand-crafted Feature-based Non-contact Depression Detection

As the name itself suggests, hand-crafted features are crafted based on the visual attention area of human beings according to statistics or real-world experiences. As such, hand-crafted features are usually much more interpretable. In the field of depression detection, researchers extract hand-crafted features which are highly correlated with the risk of depression from depression patients’ facial images, then model facial images through different classifiers (Pampouchidou et al. 2016). These interpretable hand-crafted features could be utilized to automatically detect depression risks and provide empirical evidence for physicians in quantitative analytics of depression. Hand-crafted features could be further divided into two types: Facial Shape and Textual Features and Facial Action Units (FAU) Features. For facial shape and textual features, Cohn et al. (2009) utilized the Active Appearance Model (AAM) to quantify the position and shape of eyes, eyebrows, nose, mouth, and face outline. With these data, the authors extracted the speed and acceleration data of critical face features, mouth area, etc., and achieved a 79% accuracy in classifying depression patients. With the development of depression risk analytics, the Audio/Visual Emotion Challenge and Workshop (AVEC) introduced depression detection contests. The challenge baseline for AVEC-2013 employed a combination of head location, motion, pose, and appearance features (LPQ), then extracted vectors for each video over short-term Fourier transform (STFT) to detect depression (Valstar et al. 2013). This corresponds to the trend in which videos, especially the AVEC-2013 dataset, instead of images are utilized to extract handcrafted features, where time-series cues could be included in feature extractions. For example, Cummins et al. (2013) utilized Space-Time Interest Points (STIP) and Pyramid of Histogram of Gradients (PHOG) to utilize both video frames and audio signals to fulfill the task of multi-modal depression detection. Meng et al. (2013) extract Motion History Histogram (MHH) from videos and corresponding audio data to recognize the dynamic motion and actions from depression patients, which the authors then utilized on the correlation with depression scale. Based on

2.1 Non-contact Physical and Mental Health Measurement

27

the above results, Wen et al. (2015) extracted 3D facial region information from the AVEC-2013 dataset, then derived dynamic features LPQ from Three Orthogonal Planes (TOP). Al-gawwam and Benaissa (2018) utilized eye blink features extracted from videos to detect depression and achieved 88% accuracy on Adaptive Boost (AdaBoost) classifiers. On the other hand, Facial Action Units (FAU) separate facial expressions into the movement of individual components of our face. Action Units (AU) are numerically labeled based on these individual movements, such as AU6 for cheek raiser, AU17 for chin raiser, AU24 for lip pressor, which is then consolidated into the Facial Action Coding System, or FACS (Ekman and Friesen 2019). Similar to how psychologists could detect depression risks through facial expressions, the extracted FAU could also be applied to automatic depression recognition. For example, Girard et al. (2014) noticed a significant decrease in AU12 and AU15 and an increase in AU14, which contradict the traditional classification of AU15 as sadness. Hamm et al. (2011) developed an automated FACS encode system, which claimed to “objectively describe subtle and ambiguous facial expressions.” The authors then utilized the system to analyze eight typical depression patients and extracted the AU frequency of each case to prove the system’s clinical applicability. Williamson et al. (2014) utilized Computer Expression Recognition Toolbox (CERT) to extract FAUs from the AVEC-2014 contest dataset, which was then utilized for Extreme Learning Machine (ELM) based depression recognition alongside data from audio. In all, hand-crafted feature-based depression detection has been drastically improved throughout the last decade due to the increase in noticeability and viability of video-audio-based depression datasets. However, potential obstacles still exist for applying such detection methods in clinical settings. The design and extraction of hand-crafted features require expert knowledge from psychology, which extends the required procedures and time for utilizing such features. These features also faced the problem of low transferability, which means that a change in data source and settings could easily disrupt the detection accuracy, making it inapplicable for complicated real-life scenarios. Finally, the extraction of hand-crafted features often requires a great amount of computational power and time, which could pose trouble in the face of the amount of data in the current diagnostic scenarios. B.

Deep Learning-based Non-contact Depression Detection

In recent years, deep learning has been utilized in varied computer vision tasks and has achieved high performance. Video-based health screening, for example, could be achieved through the detection of great deviation of facial features. The detection of depression has been explored similarly, which significantly increased the overall accuracy. For example, Jan et al. (2018) proposed a depression monitoring system based on a fusion of hand-crafted features and deep CNN extracted features, then utilized Feature Dynamic History Histogram (FDHH) to capture the feature movements on the time domain, which achieved good results on depression-related public datasets. Zhou et al. (2020) proposed DepressNet, in which visual data is learned through a deep regression model based on deep CNN with a Global Average Pooling (GAP) layer. The authors further proposed a face region feature joint learning model

28

2 Frontier of Smart Healthcare Engineering Management

named multi-region DepressNet (MR-DepressNet), highlighting the most important depression prediction factors. Yang et al. (2017) utilized audio, video, and text data and proposed a multi-modal fusion depression detection network based on Deep CNN and Deep Neural Network (DNN). Dibeklio˘glu et al. (2018) extracted head movements and facial landmarks from volunteering depression diagnostic patients, then introduced Stacked Denoising Autoencoders (SDAE) to reduce the redundancy among data and analyzed the non-linear correlation between the extracted features and depression scale. De Melo et al. (2019) proposed using the C3D network, a variant of 3D CNN, to capture the human face’s global and local time-spatial features to detect the severity of depression in videos. In all, among various deep learning depression screening methods, Convolution Neural Network (CNN) has been a major player, which is not surprising due to its popularity within computer vision. The state-of-the-art deep learning-based methods also achieved relatively high accuracy. However, as with most deep learning methods, few of these methods provide the interpretability offered by hand-crafted features. Furthermore, unlike the hand-crafted feature counterpart that could diagnose based on feature statistics, deep learning-based depression screening requires the whole segment of patient video, raising privacy and data security concerns. The low transferability problem from the hand-crafted feature also persists here, as any changes in the environment lighting, background, etc., could still negatively affect the accuracy of the test result. Thus, high interpretability, high transferability, accurate, and privacy-preserving depression screening method has been the most important pursuit of this field.

2.2 Online Physician Selection In the previous section, we explored next-generation information technologies’ application in health screening, the entrance towards the emergent parts of the cycle of care. While previously the step of health screening is usually performed by physicians, the development of internet healthcare, online healthcare information, and online healthcare communities (OHC) usually results in the absence of physicians during this crucial step. For instance, about 72% of online users search for healthcarerelated information online in 2017 (Nawaz et al. 2018). While it is true that such new phenomenon could create relief for the physicians, it also creates a fundamental, structural fracture in the healthcare system: instead of the family doctors recommending/referring patients to radiologists and pediatricians, the system now creates the potential for the patient to search for his own resource after a relatively accurate diagnose has been already produced. While increasing consumer choices and market competition, obvious risks also come out of such potential. Patients might not understand the dynamics behind medical referrals, which could lead to inefficiencies in patient actions. For example, patients might want to skip steps within the medical referral chain and directly seek specialist assistance without concrete diagnostic information, resulting in back-referral to primary care. Online ads could

2.2 Online Physician Selection

29

result in patients choosing low-quality medical resources, bringing potential harm to such individuals. The incident of Zexi Wei would be the most salient example, in which advertisements on Baidu search engines contributed to the eventual decision of the affected individuals (Li 2016). Supporting and guiding patients through the process of finding medical resources, then, becomes important in the new age of healthcare. Online healthcare communities and Internet healthcare platforms contributed to the medical resource-seeking processes. Hospital organized, offline resource-based “Internet Hospital”, for example, provides services such as online consultation, remote diagnosis, medication consultation, and health management. This kind of online healthcare resource is essentially an extension to the offline healthcare model, through which patients could obtain healthcare service without the need to go to a real hospital. Independently organized Online Healthcare Communities (OHC), on the other hand, composes of patients and physicians volunteering and contributing to the discussion of healthcare-related topics. These OHCs either organize on existing social media platforms, such as subreddit like r/healthcare, r/diabetes, Facebook groups, and Twitter hashtags or create their own platforms and websites. While many social media-based OHCs are open source and volunteering-based, independent platforms tend to form models that allow participating physicians and the platform to gain economic returns (Guo et al. 2017). These three basic models of OHCs: offline resource-based OHC, online social media-based OHC, and independent for-profit OHC, all have similar while different business models and risks towards patients. Offline resource-based OHC is mostly risk-free, due to it being regulated by actual offline hospitals and mostly serves as online advertising for the hospital. Social media-based OHC is composed of free-form discussions, which are the least credible but also highly unlikely to be regulated due to its social media and anonymous nature. Independent platform OHCs stand between these two. When utilizing the service of this kind of OHC, patients expect high-quality medical advice as if they are visiting offline hospitals due to the fact that patients usually pay online physicians for consultations. However, physicians are, in most cases, not bound to provide high-quality answers towards patients in these settings, due to many reasons such as lack of regulations, online-offline work balance, etc. (Wu 2018). This could lead to patients receiving low-quality medical advice and treating them as if they are high-quality, which could potentially bring harm to these individuals. The existence of such paradox, together with the lack of regulation of online platforms and their motive of economic gain, motivates the design of systems to perceive high-quality physicians and to recommend the most suitable physician to patients. In this section, we shall focus on utilizing next-generation information technologies to improve the physician recommendation process on independent OHC platforms. We specifically focus on two critical parts of the question: physician answer quality perception and online physician selection and recommendation.

30

2 Frontier of Smart Healthcare Engineering Management

2.2.1 OHC Physician Answer Quality Analysis The first, more obvious question researchers turn to is ensuring the high quality of healthcare-related answers, whether it comes from a physician or average citizen. In recent years, the quality of health-related content on online sites has become an important topic. It usually has two basic research ideas: one is the classification task, and the other is the sorting problem. Although the technical routes adopted by the two are different, their ultimate goals are the same. Surdeanu et al. (2011) used linguistic motivation features to improve search and achieved a ranking of the quality of answers to non-factual questions by combining a wide range of feature types such as linguistic features, named entity recognition, syntactic parsing, semantic role labels, etc., and show that combining linguistic features can significantly improve accuracy. Ma et al. (2015) proposed a tri-role topic model (TRTM) to model the three roles of users (i.e., as questioner, answerer, and voter, respectively) and the activities of each role, including writing questions, choosing questions to answer, contribute and vote for answers. The results show that TRTM can effectively help users retrieve a more ideal answer ranking, especially for new or less popular questions. Xie et al. (2019) noticed a phenomenon in the research of answering quality, namely “attention divergence”, and designed a new attention mechanism to create a classification method for answers. Zhao et al. (2018) considered the relationship between user activities in the question-answering ranking task and created three new topic-aware features based on the network formed by user profile information and users’ question answering and commenting activities. The three newly created topic-aware features are combined with features such as texture, user, and comment, and a pairwise L2R method SVMRank is used to rank the answers. In general, the textual quality of user-generated content is measured and deployed in research in the field of information systems. The above-mentioned related studies found that the studies on traditional answer quality evaluation mainly rely on the mining of non-text features to evaluate the quality of the answer. In contrast, relatively few studies have analyzed and predicted answer quality based on text features. The wide application based on non-text features shows that the question answering community contains a large number of social features that can be used for mining and exploration, and it also means the difficulty of short text interpretation. With the development of deep learning, many studies believe that deep semantic information extracted from text is also an important factor in evaluating the quality of answers. However, due to the serious problem of feature sparseness in short texts for online consultation, in-depth semantic mining has not been carried out. In addition, the current research is mainly based on non-text features or single text features to model and evaluate the quality of short texts. Few attempts to combine text features containing deep semantics and related non-text features to evaluate the quality of answers.

2.2 Online Physician Selection

31

2.2.2 Personalized OHC Physician Recommendation With the rapid development of next-generation information technologies, people gradually realize the importance of mining effective information from big data, which also makes recommendation systems widely used (Mongia et al. 2020). Recommendation systems create a binary relationship between the user and the item and use the user’s existing historical selection behavior or similarity relationship, so as to dig deeper into the user’s potential interest objects and personalized needs to make personalized recommendations to the user (Chen et al. 2020). Currently, recommendation systems have been distributed to many major businesses, including online shopping, music, social media, advertisements, etc. Personalized recommendation technology has important application value in recommendation efficiency, mining user needs, and improving user satisfaction, and it is also the most efficient way to resolve the problem of “information overload” (Wang et al. 2020). Specifically for the healthcare industry, many OHCs have already established their own recommendation systems, albeit the recommendation algorithms are primitive and the current research is focusing on improving such algorithms. For example, Ju and Zhang (2021) proposed an online pre-diagnosis physician recommendation model that fuses ontology features and disease texts and recommends departments and physicians to patients based on patients’ symptoms, diagnosis, geographic location, doctor’s specialty, department, and other information. Stratigi et al. (2018) introduced a group recommendation model incorporating the concept of fairness to calculate the similarity between users based on the semantic distance between users’ health issues, which aims to provide users with highly relevant and valuable advice through caregivers. After that, the authors further introduced a multi-dimensional recommendation model using collaborative filtering, which took into accounts such as education level, health literacy, and patients’ psychological and emotional states to calculate the semantic similarity between users. The concept of fairness is introduced, and an aggregation method of cumulative preference scores is proposed to provide highly relevant and fair recommendations for patient groups (Stratigi et al. 2020). Li et al. (2020) constructed a physician recommendation model based on combined conditions, including three models of similar patients, medical field, and doctor performance, which can effectively recommend high-quality physicians to patients. Ye et al. (2019) proposed a four-dimensional IT framework based on signal theory. The model uses expertise, online reviews, profile descriptions, and service quality as signals to differentiate high-quality physicians, and derives physician rankings and recommendations to patients. The current state-of-the-art has already explored various ways to recommend physicians, however, much of the research remains limited. Most of the current research uses methods such as content-based recommendation, collaborative filtering recommendation, and ontology creation. The research methods are relatively simple, which will lead to one-sided recommendation results. At the same time, when a patient recommends a physician, the evaluation of the doctor’s choice rarely takes into account the important factors such as the quality of the physicians’ response.

32

2 Frontier of Smart Healthcare Engineering Management

2.3 Cancer Screening and Diagnosis Decision Supporting Within the cycle of care, clinical diagnosis is the first part of the cycle that requires a physician to be present. For individuals, diagnosis consists of most of their visits to doctors, suggesting the importance of such a process. Each patient visiting clinics needs to go through clinical diagnosis, whether through patient-physician conversations or specific diagnostic procedures. As such, demand for physician participation in these procedures is high, even though such a process is already the majority of the daily routines of many physicians. A study in 2008 shows that the time physicians could spend on an individual patient average at 15 min in Belgium, 9.4 min in the UK, 7.8 in Spain, and 7.6 min in Germany (Das et al. 2008). The number becomes abysmal in developing countries, as each physician only has an average of 3.8 min to talk to patients in Dehli, India, and asks only 3.2 questions on average (Das et al. 2008). While increasing the number of physicians through better education and decreasing the number of patients through healthy living initiatives could be long-term solutions, the ever-evolving next-generation information technologies provide avenues for clinical diagnosis supporting tools to increase the efficiencies of physicians. For example, Electronic Health Records (EHR) could enable physicians to skip asking for personal information and medical histories, making patient-physician conversations much more efficient. This section, however, focuses on disease diagnosis, specifically cancer diagnosis, and the utilization of clinical procedure data throughout the process. Diagnosis of cancer represents one of the crucial turning points of any individual while being equally important to clinical disease diagnosis. Take upper gastrointestinal (UGI) cancer as an example: UGI cancer is one of the most common cancer among society, including esophageal cancer, gastric (stomach) cancer, small intestine cancer, pancreas cancer, liver cancer, and gall bladder cancer. According to the “International Agency for Research on Cancer” (IARC), UGI or GI cancer “accounts for 1 in 4 cancer cases and 1in 3 cancer deaths globally”. Similar to many other types of cancers, it is important for patients to detect UGI cancer early to maximize their possibility of recovery. According to Allemani et al. (2018), the UGI cancer 5-year survival rate is as high as 60–70% in East Asian countries such as Korea and Japan, while less than 30% in others. The author attributes this to societal population-based endoscopic screening programs for the early detection of gastric and esophageal cancers. While the obvious solution to the lack of early UGI cancer screening is to design and implement societal-wide endoscopic screening, many other factors could influence the outcome and percentage of early cancer discovery. First of all, endoscopic cancer screening is mostly unavailable in rural areas of developing or developed countries due to resource limits (Hamashima and Goto 2017). Secondly, UGI cancer symptoms at early stages are either too weak to be reported or usually associated with dyspepsia (Axon et al. 1995). This makes the early diagnosis of UGI cancers trickier, and, presumably, physician expertise in the field would also influence whether the diagnosis could be made correctly. Thus, to expand the current endoscopic cancer screening programs without overloading the current system and

2.3 Cancer Screening and Diagnosis Decision Supporting

33

to increase the abilities of physicians at all levels to make the correct diagnosis of UGI cancers are two of the most important problems of UGI cancer early screening. It is through this background that early UGI cancer screening enhanced with next-generation information technologies begin to show its importance. By implementing computer/AI-assisted UGI screening and diagnosis support, physicians at all levels could be provided with a more quantified and accurate cancer diagnosis decision supporting system, which alleviates the heavy load on the healthcare system and potentially could increase the accuracy in early cancer diagnosis. Trends of the industry confirm this theory through the development of electric/digitalendoscopic technologies such as Magnifying Chromoendoscopy (MCE), Narrow Band Imaging (NBI) endoscopy, Painless Electronic gastroscopy, and capsule endoscope. Furthermore, with the next-generation information technologies such as big data analytics and machine learning, data-driven intelligent endoscopic diagnosis system is becoming the main vehicle of UGI cancer detection and diagnosis (Mori et al. 2019). Deep learning techniques have also been utilized in UGI cancer detection, which significantly improves the accuracy of such systems. For instance, Hirasawa et al. (2018) applied a system based on convolutional neural networks (CNN) on the automatic detection of gastric cancer, which could be utilized in application to reduce the workload of physicians. Luo et al. (2019) developed GRAIDS, the Gastrointestinal Artificial Intelligence Diagnostic System, which achieved a diagnostic sensitivity of 0.942, comparable to expert endoscopists (0.945) and far superior to competent and trainee endoscopists (0.858, 0.722). Sun et al. (2020) designed a gastric ulcer classification framework based on deep learning neural network combined with densely-connected architecture and non-local attention mechanism, which achieved comparable results to expert gastroenterologists (F1 of 0.9470 compared to 0.9525). Research shows that an automatic UGI cancer diagnostic system could achieve similar results to human experts and many superior results compared to inexperienced physicians. In theory, automatic systems could be put into application in areas lacking the human resources of endoscopic screening and result in superior efficiency and accuracy as a whole. Although the current automatic diagnosis of UGI cancer has already reached expert accuracy, this does not mean that such systems could be directly put into clinical trials without risks or negative consequences, as there still exist major limitations due to the features of deep learning. In clinical applications, physicians depend on the endoscopic video and images to generate endoscopic reports, which are then interpreted into results. In the process, the main indicators and factors of diagnosis are quantitative diagnosis data and qualitative endoscopic reports based on natural languages. However, deep learning techniques take endoscopic video and images directly and, in most cases, output results directly. This leads to physicians’ difficulties understanding the result and conveying the result accurately to the patients. In order to increase the interpretability of such results, reports mimicking the style of the current style of reports need to be generated, or a new style of interpretable result needs to be developed. Generation of endoscope reports is not a simple task, as there is no gold standard of endoscopic reports. The style and quality of endoscopic reports depend on the report writer, who might have different experience levels and

34

2 Frontier of Smart Healthcare Engineering Management

styles. This part of the chapter would then mainly focus on two topics: endoscopic diagnosis supporting methods and automatic endoscopic report generating methods.

2.3.1 Intelligent Endoscopic UGI Cancer Screening Through the development of big data analysis and AI, applications of clinical diagnosis supporting systems have become common, such as treatment recommendations, abnormality alert, disease risk detection, and health management (Esteva et al. 2017; Zolbanin et al. 2015; Wang et al. 2018). Within the field of UGI cancer early detection, scholars have already achieved great results by utilizing endoscopic images and EHR, combining with statistical modeling, machine learning, and deep learning (Veitch et al. 2015; Rong et al. 2020). EHR contains a rich amount of information that could be mined to extract hidden knowledge and be utilized in decision support and diagnosis support. For example, Mahmoodi et al. (2016) developed an association rule discovering method based on ontology, effectively alleviating the problem of redundant and nonsensical rules generated. Lee and Kim (2017) utilized the electronic health record of clinically ill patients, analyzed the correlation between clinicopathological parameters and lymph node metastasis (LNM) in early gastric cancers, and demonstrated that increasing LNM correlates with the development of tumors. Similarly, Li et al. (2020) retrospectively analyzed patient EHR and postoperative data and investigated the potential influencing factors of late-stage gastric cancer. In all, EHR is currently one of the most important patient datasets that could be utilized to understand the development and factors of diseases such as UGI cancer, therefore helping develop a more effective and personalized treatment plan. The mining of EHR data needs to be assisted with clinical diagnostic data. In the scenario of UGI cancer, clinical diagnostic data mainly consists of endoscopic images, which are much harder to analyze accurately. However, through the rapid growth of computation power, deep learning is achieving the previously impossible and is gaining popularity. For example, Liang et al. (2019) constructed a reiterative learning framework on weakly annotated biomedical images, which represent the first time deep learning application on the segmentation of gastric cancer images. Liu et al. (2020) research focused on Magnification endoscopy with narrow-band imaging (M-NBI), a widely used gastric lesion type diagnostic system. The authors introduced a transfer learning framework that used a pre-trained CNN network to classify gastric M-NBI images and achieved better results than “traditional handcraft features and trained CNNs.” The advancement of medical image analysis technologies also accelerated the development of digital pathology platforms, which in turn accelerated the development and applications of remote pathology and remote tumor diagnosis. Although the current state-of-the-art clinical diagnosis supporting methods are abundant, these are mostly focused on gastric cancer, while research focusing on UGI or GI cancer as a whole is lacking. Moreover, the reliance on endoscopic reports during diagnosis, recording, and explanation is still in place. At the same

2.3 Cancer Screening and Diagnosis Decision Supporting

35

time, the current trend in research only focuses on how to more accurately classify endoscopic images without having reports and interpretability in mind. Although plausible in theory, such systems would result in much hardship in reality, as physicians lack any methods of interpreting and explaining the results to patients, resulting in trust degradation between physicians and patients. Therefore, making artificial intelligence-generated diagnosis results interpretable and explainable becomes the second important factor in clinical diagnosis supporting systems.

2.3.2 Medical Image Report Generation for Cancer Diagnosis Medical Diagnosis images and the reports generated with such images consist of the greater majority of the tools physicians utilize to provide patients with diagnosis and treatment. Automated medical image analysis and report generation are becoming important within the departments which heavily utilize medical images such as endoscopy, radiology, and ultrasound. Automatic medical image report generation, specifically, could provide interpretability to automated image analysis results, eliminate the need for manual report write-ups and alleviate physician workload dramatically, increasing the overall efficiency of medical workers. Through the utilization of deep learnings within clinical diagnosis support, researchers are starting to make an effort to resolve the issue of automatic report generation and are achieving some results. Deep learning-based automatic endoscopic report generation is usually through the Encoder-Decoder Framework, within which the encoder is usually utilized to extract endoscopic features, normally combined with “Convolution Neural Network” (CNN). At the same time, the decoder typically consists of a “Recurrent Neural Network” (RNN) and could generate descriptive texts about the encoded endoscopic images. For example, Shin et al. (2016) applied the encoderdecoder framework in chest X-ray and report data interpretation, representing the first time a CNN/RNN pairing approach is implemented in medical image report generation. More advanced approaches such as attention mechanisms could further enhance the encoder-decoder system and could be employed to highlight important messages within the generated report. Zhang et al. (2017) proposed “Medical image Diagnosis Network” (MDNet), which integrated the CNN/RNN approach with an improved attention mechanism, generating an interpretable diagnosis report with highlighted feature descriptions. Jing et al. (2018) proposed a co-attention mechanism-based deep learning model to create a multi-task learning framework, then applied Long Shortterm Memory (LSTM) network to generate long and consistent texts. Li et al. (2018) proposed a Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent), which introduced template database and reinforcement learning into the field of report generation and could be applied to generate more structured and varied medical

36

2 Frontier of Smart Healthcare Engineering Management

image reports. Huang et al. (2019) proposed a multi-attention mechanism hierarchical model, which folded in patients’ background information to improve chest X-ray report generations. Compared to endoscopic image reading and interpretation, the medical image report generation is much younger. Although some research is conducted for a consistent, interpretable medical image report generation process, these research are scattered throughout the medical image industry. Furthermore, the interpretability and accuracy of medical image reports depend on image analysis accuracy. Without the real-world implementation of the prerequisites, it could be challenging for medical report generation algorithms to be fully utilized.

2.4 Multi-Dimension ICU Risk Analytics Hospitalized patients resemble one of the most risk-prone parts of the population in the healthcare system while also one of the most expensive parts of the cycle of care. The resources of such sections are almost always in high demand, as hospitalization hold accountable for a wide range of conditions, including childbirth, septicemia, heart failure, pneumonia and respiratory failure, and osteoarthritis. Even for the high price, the demand for care has been continuously higher than the supply, further increasing the cost of hospitalization in most countries, for instance, the 35 million hospital stays in the United States in 2016 cost an average of 11,700 USD per stay (Freeman et al. 2018). Such demands create shortages and stress the healthcare system. In particular, ICU has been one of the most strained while also the highest risk department of all hospital sectors. ICU admission has been rising sharply over the last two decades in the United States, from 2.79 million to 4.14 million in six years, an increase of 48.8% over the period of a few years (Mullins et al. 2013). With an average cost of 20–35% of total hospital costs while only supplying 5–10% of hospital beds, ICU occupies one of the most critical and costly positions in every hospital (Armony et al. 2018). This is further fueled by frequent death in the ICU department. Even though the ICU mortality rate decreased by 35% within the last few decades, the overall adult ICU mortality rate still averages from 10 to 29% (SCCM 2021). In Germany, more than one-third of hospital mortalities occur in the ICU/IMC unit (Ay et al. 2020). Such is not to claim either ICU is not cost-efficient, or it fails to protect patient life due to any mistake of itself, but to demonstrate the importance of ICU within the hospital system and show the medical risks within the ICU department and patients. Such importance, which often correlated with the mortalities ICU, has caused serious social turmoil. A Polish study claims that conflicts between ICU staff and family members are the number three cause of conflicts within hospitals, closely behind nursing team conflicts and nurse-physician conflicts (Wujtewicz et al. 2015). In other words, ICU might represent the number one cause of conflict between medical teams and patients. Such conflicts, according to Wujtewicz et al., could lead to a “lack of coherence, lower efficiency and quality of work, and irrational behavior” of the ICU

2.4 Multi-Dimension ICU Risk Analytics

37

team. The majority of studies cite inadequate communication and poor quality of information provision as the leading cause of ICU conflicts. Under such circumstances, the idea of utilizing next-generation information technologies and smart healthcare to alleviate communication and information problems within the ICU department is born. By intensively utilizing medical data, machine learning, and deep learning techniques, researchers aim to provide ICU staff with sufficient information and decrease clinical risks surrounding the ICU departments. Smart Healthcare could easily be utilized in this scenario, as EHR, medical images, and vital monitoring equipment could serve as decision support tools in various ICU tasks, including situation analytics, diagnosis, and treatments. As data collected from ICU increased dramatically due to the increase in data capturing ability through the past decades, data-oriented ICU clinical decision support is becoming more of a viable choice compared to the past. Medical records stated above could assist staff in understanding the risk elements surrounding each patient and its importance ranking, which in turn could transform ICU mortality risk de-escalation from passive processing to active prevention. Thus, the question regarding utilizing medical data and information to effectively manage ICU clinical risks has become one of the most challenging and pressing issues in healthcare engineering management. Particularly, to resolve the ICU resource demand stress problem, many state-of-the-art modeling methods have already been utilized on resource management and decision making, which could alleviate risks regarding delayed treatments. For example, the queuing theory has been tested by Meares and Jones (2020) to resolve ICU staff scheduling problems, and ICU bed needs problems during the COVID-19 crisis. Similar questions have been explored by Shehadeh and Padman (2021), utilizing stochastic elective surgery scheduling in a constrained ICU capacity scenario. However, complex situations and even random events during clinical applications could easily affect these modeling. For example, sudden changes in patient conditions, accidents during ICU procedures, and even conflicts between ICU staff and family members could contribute to such models’ ineffectiveness. Such uncertainties and demand strains during ICU operations are usually “resolved” by hospital management through varied ways of compromise, such as rejecting new ICU admissions or making early ICU discharges. As such, extensive research has also been conducted on effective and risk-minimizing ICU discharging methods to ensure discharged patients’ medical quality and health (Ofoma et al. 2018; van Sluisveld et al. 2017; de Vos et al. 2021). However, due to the complexity of medical conditions within the ICU department, it has been difficult to determine discharge-capable patients with high accuracy. Furthermore, re-admission to ICU and even mortality after discharge is not uncommon after ICU discharge, as the heterogeneous nature of human environments is almost always unpredictable. As such, for cases of compromise, it is wise to conduct extensive tests on such patients to verify their health status to minimize re-admission or mortality risk. In addition, Post Intensive Care Tracking (PICT) protocols have also been explored by researchers such as Flaws et al. (2021) to minimize mortality rate up to 6 months post-discharge.

38

2 Frontier of Smart Healthcare Engineering Management

As mentioned above, among all ICU-related clinical risks, such as delayed treatment risk, prolonged-ICU stay risks, and iatrogenic risks, the most explored and attention receiving risks are re-admission risks and mortality risks. ICU care quality is often correlated with ICU mortality and re-admission rate, as the ability to keep patients from mortality and readmitting to ICU signals the overall caring ability of the ICU and its staff. Furthermore, patients readmitted to the ICU after discharge have a much higher mortality risk, from 16 to 35%, depending on the time period after ICU discharge (Kramer et al. 2012). Not only do such observations link high readmission risks with mortality risks, but they also suggest that re-admission causes unnecessary cost and suffering to patients themselves, the hospital, and society as a whole. Measuring mortality and re-admission risks within the ICU, then, becomes one of the most important objectives of the field of ICU risk prediction, as it is the most effective in reducing patient suffering and hospital costs. The following section explains the previous efforts of researchers in controlling and mitigating ICU risks, especially ICU re-admission and mortality risks.

2.4.1 ICU Admission and Discharge Management Similar to hospital as a whole, the overall objective of ICU is to provide appropriate treatment to patients at the appropriate time with an appropriate price tag, within such the “appropriate treatment” part being the most important towards the health of patients, otherwise prognosis of patients could be worsened while healthcare cost could sharply rise. However, such a task was proven never to be mundane, as varied risks exist in such a process and could lead to severe consequences, especially in the ICU. First, it requires extensive and accurate clinical diagnoses of patients, which is a complicated diagnosis task. Furthermore, even with an accurate diagnosis, such patients must be assigned to an almost randomized environment and managed through medical staff, which is also a challenging operation management problem. Therefore, this section will serve as a summary of how the “diagnosis task” is explored in the scenario of ICU. Specifically, the diagnosis will be separated into three sections: ICU admission and discharge procedure, ICU re-admission prediction and risk analytics, and ICU mortality risk analytics. This section, as a consequence, will tackle the first part of the ICU diagnosis process: the ICU admission and discharge protocol and procedures. As mentioned above, ICU is plagued with low capacity, heavy load, and uncertainties. When faced with these uncertainties and loads, ICU management needs to provide a queue to ICU admission to secure patient safety and care quality. An obvious solution would be to put patients in more severe conditions on top of the queue. However, due to patient conditions’ high uncertainty and heterogeneity, such a queue is often plagued with inaccuracy. Although some admission and discharge classification systems have been utilized, such as the Simplified Acute Physiology Score (SAPS) system (Le Gall et al. 1993; Kramer et al. 2012), there exists no gold standard in such classification systems, and physician experiences are still one of the

2.4 Multi-Dimension ICU Risk Analytics

39

most relied methods when determining discharge and admission, which are prone to human biases. Recent research has explored various methods to optimize the ICU admission decision-making processes. For example, Kim et al. (2015) utilized the Laboratory-based Acute Physiology Score (LAPS) and Comorbidity Point Score (CPS) to measure the acute and chronic risks of patients as a standard for admission, which could relieve ICU shortage. However, such a method disregarded the balance between patient outcome and hospital economic incentives and failed to maximize ICU economic gains, which could prove unpopular if applied in real-life scenarios. To solve this, Yang et al. (2015) constructed a Markov Decision Programming (MDP) model to examine admission control policies when patients arrive in batches. When the authors apply their methods in real-life scenarios, the result shows that “ICU can achieve large equity gains with no efficiency losses” (Yang et al. 2015). With the high demand for ICU services, hospital management will inevitably face ICU flow problems. Currently, hospital management compromises with such problems by discharging current ICU patients into a lower standard of care, either to a Step-Down Unit (SDU) or standard care, to make room for patients with a higher priority. However, although widely practiced, such mechanisms are often adhoc, with high uncertainty and mortality risks when scanning for potential discharge targets. To alleviate such risks, Chan et al. (2012) introduced an ICU privileged discharge protocol based on re-admission rate and mortality rate prediction, which provided insights into early discharge loss analytics. In addition, Shi et al. (2015) developed a stochastic network model for inpatient operations, including a discharge distribution protocol, which decreased the waiting time of a high patient flow period. Similarly, Li et al. (2018) utilized the MDP model to visualize ICU patient dynamics of arrival and discharges. Other than admission and discharge protocols, algorithms ensuring patients receive adequate and appropriate care in a randomized, strained environment are also popular among recent research. For example, Song et al. (2016) utilized the genetic algorithm and Discrete-Event Simulation (DES) to find an optimal patient flow distribution, improving overall hospital system performances. Guo et al. (2017) utilized a feasibility detection procedure to minimize emergency department staff costs in an environment where the expected quality care received is set to have a minimum requirement. These mathematical modeling could alleviate ICU resource distribution and management fairly well, albeit the need and uncertainty of patient ICU staying length is not considered substantially. Such are the focuses of Rouzbahman et al. (2017), who utilized cluster-boosted regression to estimate the patient length of stay in ICU and mortality risk. Bartel et al. (2014) compared random forest and logistical regression methods on their ability to predict the patient length of ICU stay in order to assist ICU management.

40

2 Frontier of Smart Healthcare Engineering Management

2.4.2 ICU Readmission Prediction and Risk Analytics The re-admission rate of ICU has been noticeable as an important index and indicator of healthcare quality. With the mainstreaming of next-generation information technologies, extensive research has been conducted on the ICU re-admission problem with the perspective of data analytics. The majority of research could be divided into two main categories: risk factor selection and re-admission risk prediction. Concerning the question of ICU re-admission risk factors, many attempts have already been established before the end of the last century. For instance, Rosenberg and Watts (2000) summarized ICU re-admission factors of prior studies, which includes symptomatic factors such as PaO2 < 70 mmHg, urine output < 30 mL/h, heart rate > 110 beats/min, clinical factors such as Nosocomial Pneumonia, Hypoxemia, Neurologic disease, intrinsic factors such as age, and even environmental factors such as premature discharge and “intermediate care unit with less than two open beds.” It could be observed that factors contributing to ICU re-admission are extremely complex and hard to correlate to re-admission risks quantitatively. By utilizing nextgeneration information technologies such as machine learning, scholars have begun to examine the quantitative correlation between re-admission factors and risks. For example, Viegas et al. (2017) proposed a feature engineering and fuzzy ensemble modeling combined ICU re-admission prediction framework, which achieved an AUC of 0.77 ± 0.02 in predicting early re-admissions. Waledziak et al. (2019) adopted univariate and multivariate logistic regression models to identify laparoscopic appendectomy patients’ re-admission and adverse event risk factors. Lin et al. (2017) approached the problem of re-admission risk factor analytics using Bayesian Multitask Learning (BMTL) through the analytics of EHR data. On the other hand, ICU re-admission risk analytics has seen a broad application of machine learning and deep learning methods. Lee et al. (2019) introduced a machine learning-based re-admission risk prediction model to the total joint replacement readmission scenario, which confirmed the potential usefulness of machine learning algorithms in ICU re-admission risk analytics. Zheng et al. (2015) designed a Particle Swarm Optimization Support Vector Machine (PSO-SVM) to achieve re-admission prediction, which produced higher accuracy and precision than neural network-based models. Fialho et al. (2012) utilized fuzzy modeling to predict daily re-admission rates of ICU patients. Finally, Zebin and Chaussalet (2019) analyzed the ability of the Long Short-Term Memory (LSTM) network on the task of re-admission risk prediction. Compared with traditional neural network models and logistical regressions, LSTM could achieve superior results. To summarize, it is undoubtedly that the problem of ICU re-admission could be relieved by using machine learning and new deep learning methods. The majority of research on such possibilities focuses on recognizing certain risk factors or readmission risk prediction. However, current research and explorations cannot correlate risk factors with quantitative re-admission risks. Furthermore, risk factors each entangle with each other, which could have a combined effect on re-admission risk, while the current state-of-art could not take into account such combined effects.

2.4 Multi-Dimension ICU Risk Analytics

41

2.4.3 ICU Mortality Prediction and Risk Analytics As the name “Intensive Care Unit” suggests, ICU usually holds patients with the highest probability of mortality, except for those currently undergoing emergency surgery. As such, ICU inevitably has a high mortality rate, thus necessitating mortality rate prediction methods. These tools need to provide physicians and ICU staff with decision support to avoid unnecessary and untimely treatment and serve as an indicator of ICU medical resource optimization. Currently, there exist several scoring and risk prediction systems for clinical and ICU usages, including Acute Physiology and Chronic Health Evaluation (APACHE), Simplified Acute Physiology Score (SAPS), Sepsis Related Organ Failure Assessment (SOFA), and Multiple Organ Dysfunction Score (MODS) (Le Gall et al. 1993; Zimmerman et al. 2006; Vincent et al. 1996; Marshall et al. 1995). These traditional scoring systems rate patient conditions through varied quantifiable data, including clinical diagnosis results, vital data, etc. Moving towards the information age, newer ICU mortality prediction models such as Medical Information Mart for Intensive Care (MIMIC) have been developed using logistical regression models, which are compared with the SAPS scoring system to be a better counterpart (Demirjian et al. 2011). Compared to experienceonly discharging decisions, these tools gave physicians quantitative tools to analyze patient situations and mortality risks. However, prediction tools based on logistical regression are low in accuracy and precision, making them unsuitable for clinical usage. Through the explosive growth of ICU data and the development of machine learning techniques, researchers are increasingly inclined to utilize machine learning models to construct ICU mortality prediction tools because machine learning could better illustrate the non-linear relationships between factors and risks. For example, Venugopalan et al. (2019) combined traditional linear models with neural networks, SVM, and decision trees to classify the mortality results of patients. Lin et al. (2019) constructed a random forest-based model based on EHR from four different hospitals to predict ICU renal failure-induced mortality and claimed to have better results than logistical regression methods. Other researchers also pointed out that decision trees and SVM could gain better results than traditional regression models (Yuan et al. 2020; Barchitta et al. 2021). However, some scholars argue that mixed/integrated methods could improve the variety between prediction models, which could improve the prediction ability of such models. For example, van Wyk et al. (2019) constructed a multi-layer machine learning method to analyze high-frequency, continuous data to increase the prediction ability of ICU mortality. Awad et al. (2017) proposed an ensemble learning random forest, decision trees, naive Bayes, and others to create a combined method called EMPICU framework and tested on the MIMIC-II database. Although the current data-oriented ICU patient mortality risk prediction research is projected to help the effort of real-life scenarios, there still exist problems such as low accuracy and precision, leading to trouble detecting high-risk patients. One noticeable question in the current state-of-art is that the great majority of articles treated false-positive the same as false-negative, while in a real-life scenario, this

42

2 Frontier of Smart Healthcare Engineering Management

could not be true. In reality, a false-positive (low-risk patients flagged as in dire situation) has a much lower impact than a false-negative (high-risk patients being discharged too early). As such, even though the overall accuracy of current state-ofthe-art models is high, the probability of false-negative is still too high to be applied to real-life applications.

2.5 Minimally Invasive Surgery Quality Control Although surgery is an activity that individuals usually want to avoid, it is unavoidable that surgery needs to be completed, at least in some cases. While the previous procedures represent the bulk of the patient cycle of care, surgical procedures are the most intensive part. Extremely risk-prone and requires highly skilled professionals, surgeries are usually done with extreme caution or only under dire situations. With the evolution of medical technologies, Minimally Invasive Surgery (MIS) has gained popularity in the 2000s, together with MIS fellowship, which provides additional training to physicians in need of advanced MIS techniques (Shockcor et al. 2021). Compared with traditional general surgery, MIS has the advantages of smaller intrusion and shares a smaller portion of the risks of traditional surgery other than hypothermia caused by cold CO2 gas pumped into the patient’s stomach during surgery. Recently, MIS has seen a broader combination with the robotic industry to form a trend of robot-assisted surgery applications in major hospitals. Robotic surgical systems such as the Da Vinci surgery robot have already proved useful through their application in major hospitals since 2000. Although the trend clearly favors robotic surgery, it is also clear that fully autonomous robotic surgery will not come any day soon. Therefore, the current research focus is still on Robotic/Endoscopic Minimally Invasive Surgery. While superior to traditional general surgery in many factors, MIS is not without its own risks. Without direct visual observation provided by general surgery, MIS forced physicians to observe surgical situations through endoscope cameras, thus negatively impacting their hand-eye coordination ability. The introductions of MIS also present new information and a higher learning curve than traditional surgeries, leading to a higher complication rate for physicians undergoing the learning process, albeit the problem seems to be alleviated through careful selection of easier patients (Sclafani and Kim 2014). Through the COVID-19 crisis, the operation of the surgical department has been greatly altered. Researchers observed a reduction in the number of emergency surgeries correlating to stay-in-home orders in Italy (Patriti et al. 2020a). However, reports have also shown delayed treatment of surgery-needed situations, leading to an increase in surgical operation challenges, further complicated by the lack of protective equipment and personnel (Patriti et al. 2020b). The uncertainties in physician qualities and learning stages and complex patient conditions contributed to the difficulties in surgical quality control and surgical department operation controls.

2.5 Minimally Invasive Surgery Quality Control

43

In addition, the duration of surgery is often hard to predict, causing scheduling problems in the already heavy-loaded surgical department (Combes et al. 2008). The optimization of surgery department resources thus depends on three important factors: an optimized patient-physician pairing process, a well-developed peri-surgery decision supporting system, and an accurate surgical period prediction and scheduling system. Since the patient-physician pairing process has already been explored in Sect. 2.2 of this book, the following sections will focus on the latter two factors and their state-of-the-art research and applications.

2.5.1 MIS Surgical Tool Detection and Segmentation Minimally Invasive Surgery support starts with the analytics of surgical procedures and status. With the development of next-generation information technologies and robotic surgery equipment, tracking and analyzing surgical processes and status has become much more realistic. One of the readily available data sources would be surgery video provided by endoscope cameras, through which the analysis of MIS surgery tools becomes possible. Tracking and segmenting surgical tools through surgical video could be beneficial through the added ability to review physician ability and real-time surgical decision support combined with surgical tool detections (Funke et al. 2019; Allan et al. 2018). Previously researchers added RFID tags, color labels, and other variants of tracking equipment to accomplish the task of surgical tool tracking (Bardram et al. 2011; Ma et al. 2019; Holden et al. 2014). However, such methods inevitably disrupt the surgical tools in some ways that lead to surgical tool length change and difficulty of disinfection, while achieving only relatively low accuracy in surgical tool labeling. Through the past decade, new surgical tool detection methods based on neural networks and deep learning without disrupting surgical tools have emerged. For example, Choi et al. (2017) designed a surgical tool detection and tracking method based on the real-time object detection system “You Only Look Once” (YOLO). The method transformed the surgical tool detection problem to an easier surgical tool edge position detection problem, which increased the performance of the detection model both in its accuracy and operation cost. Sarikaya et al. (2017) utilized Faster R-CNN target detection, regional proposal network (RPN), and multimodal two-stream convolutional network, which achieved the combination of image and temporal motion cues and a 0.1 s per frame detection rate. Finally, Colleoni et al. (2019) proposed a 3D FCNN architecture in extracting Spatio-temporal features of surgical instruments in 2D videos and successfully proved the usefulness of such features in detecting surgical instruments. In the field of surgical instrument segmentation, Garcia-Peraza-Herrera et al. (2017) designed two deep learning architectures for non-rigid surgical instrument segmentations. Laina et al. (2017) proposed a localization and segmentation combined surgical tool segmentation method, which reformulated the problem into a much easier heat-map regression question. Du et al. (2018) proposed a sensor-free

44

2 Frontier of Smart Healthcare Engineering Management

surgical instrument position estimation method, which first detects the estimated position of surgical instrument joints and association between joint pairs. Joint pairs are then connected through maximum bipartite matching to gain the final estimation of surgical instrument poses. Qin et al. (2020) improved overall segmentation performance through the Multi-angle Feature Aggregation (MAFA) method, which utilized the visual cues gained through active image rotation. Zhao et al. (2020) proposed a dual motion-based method to learn the movement of surgical instruments between video frames, which enables additional data pairs and increases the performance of surgical tool segmentation. By examining state-of-the-art methods, it could be observed that through the implementation of machine learning and deep learning techniques, traditional surgical labeling practices are abolished, turning into pure video and image analytics. This eliminates the additional cost and risks surrounding surgical instruments while inevitably decreasing the accuracy of detection and tracking procedures. Furthermore, surgical procedures and surgical instrument movements are complicated and uncertain, while endoscope camera data could prove to be unreliable in many factors. These contribute to the fact that rarely any research could effectively combat light reflections and movement blur during normal operations. Other issues such as obstruction of view generated by smoke, human organs, or even occasional surgical induced hemorrhage also lead to reduced performance of such methods.

2.5.2 MIS Stage Detection and Duration Prediction As stated above, operational management of the surgical resource in hospitals depends on the accurate estimation and predictions of surgery duration, which could help yield a better operation room (OR) scheduling process. While it is possible to estimate surgery length through pre-surgery diagnosis and planning, the process is currently still too complicated for artificial intelligence and machine learning to understand. As such, the state-of-the-art research typically ignores the pre-operation step, leaving such into the hands of physician experiences, and instead focus on in-operation surgical stage detection and duration prediction. Intraoperative surgical video could provide insights into surgeries, such as the current operation activity, stage, and potential risks such as unintentional hemorrhage. Detection and alert systems could be built on top of such, assisting physician decision-making and decreasing operation risks. Furthermore, the automatic detection of surgery stages and remaining duration could assist surgical room resource optimization, increasing the efficiency of operating rooms. Unlike the previous question of surgical instrument detection, surgical procedure and stage detection require the analysis and modeling of surgical video of a much longer timespan to achieve high accuracy. For example, Twinanda et al. (2017) proposed a CNN-based surgical phase recognition method relying on surgical video only. The proposed architecture EndoNet carries out surgical phase recognition and surgical instrument detection tasks in “a multi-task manner.” However, the visual

2.6 Frontier Trends in Smart Healthcare Engineering Management

45

cues and time-sequence information are not combined efficiently, resulting in lower accuracy. As such, Jin et al. (2018) developed a CNN and RNN combined SV-RCNet, which utilize visual and temporal cues from surgical videos simultaneously, which sufficiently increased the ability for such model to detect surgical stage change and abnormalities. Kannan et al. (2020) proposed Future-State Predicting Long ShortTerm Memory (FSP-LSTM) and achieved the prediction of future hidden video frames with only the first 10 min of the surgery video. Researchers usually utilize stochastic programming or robust optimization in surgery remaining duration prediction tasks. Rather than a longer period of video from a single surgery, the stochastic programming method relies on analyzing a great amount of historical data. For example, Strum et al. (2000) extracted 40,076 surgical cases, examined their goodness-of-fit of normal distribution and log-normal distribution, and concluded that surgical operation remaining duration prediction should assume log-normal distribution instead of normal distribution. Lamiri et al. (2009) assumed the normal distribution of surgical demand and proposed the stochastic planning of operation room resources. They further utilized Monte Carlo simulation to simulate patient demand in different scenarios and investigated the convergence properties. These methods rely heavily on historical data while ignoring unpredictable real-life situations and risks. Aksamentov et al. (2017) became the first to automatically utilize real-time endoscope cameras and other surgical room camera data to predict the remaining surgical duration. Funke et al. (2018) proposed a selfsupervised surgery workflow analysis method, which makes the utilization of unlabeled surgical video possible. On the other hand, Twinanda et al. (2019) proposed a Remaining Surgery Duration automatic prediction method (RSDNet), which utilizes video/ image information of previous segments of surgical video to predict remaining surgical time without any labeling. Similarly, Rivoir et al. (2019) proposed an unsupervised temporal video segmentation method in which remaining surgery duration (RSD) predictions are assigned as an auxiliary learning task. Unfortunately, current state-of-the-art research of RSD prediction of surgical stage analysis is restricted to one segment of the surgery and does not span across the whole surgery while achieving low robustness.

2.6 Frontier Trends in Smart Healthcare Engineering Management Recent innovations of information technologies and healthcare technologies have led to breakthroughs and new trends in smart healthcare. In the previous sections, we picked a few of the most salient examples within the frontier of smart healthcare and introduced their state-of-the-art implementations and research. As we could observe, smart healthcare expands the existing cycle of care from the traditional diagnosistreatment to a much broader “monitoring-diagnosis-treatment-prognosis-recovery” process. This enables the possibility for healthcare institutions and governments

46

2 Frontier of Smart Healthcare Engineering Management

to reduce the frequency of diseases through health monitoring, early diagnosis, and communicable disease monitoring, which could significantly alleviate the pressure on the existing healthcare structure. Furthermore, faster and more accessible healthcare processes such as non-contact physical and mental health monitoring and online physician consultations could lead to a decrease in cost and increase in efficiency as well. An increase in patient satisfaction and healthcare outcome could also be anticipated through processes such as ICU risk analytics and MIS quality control, leading to lower mortality rates and accident rates. In all, the utilization of nextgeneration information technologies in the framework of smart healthcare could lead to drastic overall performance increment of the existing healthcare structure. That’s said, even though great outcomes could be observed through the implementation of such innovations, such new research is still only within the scale of “innovations”. Many of these innovations are barred from actual implementations within the existing healthcare systems. The reasons are multi-folded: the first and the most obvious one is that these technologies are only at their respective infant period. Unlike other fields such as IT or E-Commerce, since healthcare is directly related to the well-being of human lives, the implementation of new healthcare technologies is mostly not a market-based process, but a research-development-trial process, which deaccelerates innovations significantly. However, even before such technologies could be trailed within the current framework, another reason further blocks many of the innovations from reaching that step: the anticipated/ perceived risks of next-generation information technologies. As observed previously, a greater majority of the innovations of smart healthcare rely on the implementation of machine learning and deep learning techniques. These techniques, although some boasted “better than human performances”, are not inherently trustworthy and understandable. Similar to the current trend within the self-driving vehicle industry, although arguably, selfdriving has already achieved better performance compared to an average human being, trust and risk issues are plaguing the implementation of the technology. Therefore, while specific elements are discussed in great detail in the state-of-theart research, rarely do any of the research could delve into real-life implementations, or aim to understand the trends of smart healthcare as a whole. In reality, each of the smart healthcare system components needs to be designed with heavy scrutinization, integrated with the current healthcare system, and then managed with the consideration of continuous update and risk control measurements. Instead of simply competing for the highest accuracy, in diagnosis, for example, real-life scenarios that could disrupt the diagnostic accuracy need to be understood and counter-measures need to be realized. Rather than focusing on increasing the intangible “interpretability”, risk factors that could introduce unhealthy human-machine conflicts need to be listed and easy-to-understand procedures and manuals need to be introduced for these circumstances. All these above considerations lead to the need to systematically find the most viable principle and systematic problems in the current healthcare system and the implementation of the next-generation smart healthcare systems, i.e., discussing smart healthcare in the framework of system engineering. From the perspective of system engineering, we conclude that the current revolution on smart healthcare could

References

47

be summarized as essentially the following concepts: data governance and utilization, i.e. how to better utilize the accumulated/hidden data which is well beyond the ability for any individual to comprehend; information exchange and fusion, i.e. how to quantificationally exchange the already existing information and merge such information to support decision making; knowledge inference and recommendation, i.e. how to generate interpretable medical knowledge through the utilization of information technologies and recommend such knowledge to physicians and individuals in the right circumstances. In the following chapters, we strive to address the need for a design and engineering theories framework for smart healthcare systems and introduce the system design framework for smart healthcare systems in Chaps. 3–5.

References Aksamentov I, Twinanda AP, Mutter D, Marescaux J, Padoy N (2017) Deep neural networks predict remaining surgery duration from cholecystectomy videos. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S (eds) Medical image computing and computerassisted intervention—MICCAI 2017. Springer International Publishing, Cham (Lecture Notes in Computer Science), pp 586–593 Al-gawwam S, Benaissa M (2018) Depression detection from eye blink features. In: 2018 IEEE international symposium on signal processing and information technology (ISSPIT), pp 388–392 Allan M, Ourselin S, Hawkes DJ, Kelly JD, Stoyanov D (2018) 3-D pose estimation of articulated instruments in robotic minimally invasive surgery. IEEE Trans Med Imaging 37(5):1204–1213 Allemani C, Matsuda T, Di Carlo V, Harewood R, Matz M, Nikši´c M et al (2018) Global surveillance of trends in cancer survival 2000–14 (CONCORD-3): analysis of individual records for 37 513 025 patients diagnosed with one of 18 cancers from 322 population-based registries in 71 countries. The Lancet 391(10125):1023–1075 Armony M, Chan CW, Zhu B (2018) Critical care capacity management: understanding the role of a step down unit. Prod Oper Manag 27(5):859–883 Awad A, Bader-El-Den M, McNicholas J, Briggs J (2017) Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. Int J Med Inform 108:185–195 Axon ATR, Bell GD, Jones RH, Quine MA, McCloy RF (1995) Guidelines on appropriate indications for upper gastrointestinal endoscopy. BMJ 310(6983), 853–856. https://doi.org/10.1136/ bmj.310.6983.853 Ay E, Weigand MA, Röhrig R, Gruss M (2020) Dying in the intensive care unit (ICU): a retrospective descriptive analysis of deaths in the ICU in a communal tertiary hospital in Germany. Anesthesiol Res Pract 1(2020):e2356019 Barchitta M, Maugeri A, Favara G, Riela PM, Gallo G, Mura I et al (2021) Early prediction of sevenday mortality in intensive care unit using a machine learning model: results from the SPIN-UTI project. J Clin Med 10(5):992 Bardram JE, Doryab A, Jensen RM, Lange PM, Nielsen KLG (2011) Petersen ST. Phase recognition during surgical procedures using embedded and body-worn sensors. In: 2011 IEEE international conference on pervasive computing and communications (PerCom), pp 45–53 Bartel AP, Chan CW, Kim S-H (Hailey) (2014) Should hospitals keep their patients longer? The role of inpatient care in reducing post-discharge mortality [Internet]. National Bureau of Economic Research (cited 2021 Dec 22) (Working Paper Series). Report No.: 20499. Available from: https:// www.nber.org/papers/w20499 Caligiuri MP, Ellwanger J (2000) Motor and cognitive aspects of motor retardation in depression. J Affect Disord 57(1):83–93

48

2 Frontier of Smart Healthcare Engineering Management

Chan CW, Farias VF, Bambos N, Escobar GJ (2012) Optimizing intensive care unit discharge decisions with patient readmissions. Oper Res 60(6):1323–1341 Chen L, Zhang L, Cao S, Wu Z, Cao J (2020) Personalized itinerary recommendation: deep and collaborative learning with textual information. Expert Syst Appl 144:113070 Choi B, Jo K, Choi S, Choi J (2017) Surgical-tools detection based on Convolutional Neural Network in laparoscopic robot-assisted surgery. In: 2017 39th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp 1756–1759 Chow CK, Teo KK, Rangarajan S, Islam S, Gupta R, Avezum A et al (2013) Prevalence, awareness, treatment, and control of hypertension in rural and urban communities in high-, middle-, and low-income countries. JAMA 310(9):959–968 Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT et al (2009) Detecting depression from facial actions and vocal prosody. In: 2009 3rd International conference on affective computing and intelligent interaction and workshops, pp 1–7 Colleoni E, Moccia S, Du X, De Momi E, Stoyanov D (2019) Deep learning based robotic tool detection and articulation estimation with spatio-temporal layers. IEEE Robot Automat Lett 4(3):2714–2721 Combes C, Meskens N, Rivat C, Vandamme J-P (2008) Using a KDD process to forecast the duration of surgery. Int J Prod Econ 112(1):279–293 Cummins N, Joshi J, Dhall A, Sethu V, Goecke R, Epps J (2013) Diagnosis of depression by behavioural signals: a multimodal approach. In: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge [Internet]. Association for Computing Machinery, New York (cited 9 Jan 2022) (AVEC ’13), pp 11–20. Available from: https://doi.org/10.1145/2512530. 2512535 Das J, Hammer J, Leonard K (2008) The quality of medical advice in low-income countries. J Econ Perspect 22(2):93–114 de Melo WC, Granger E, Hadid A (2019) Combining global and local convolutional 3D networks for detecting depression from facial expressions. In: 2019 14th IEEE international conference on automatic face gesture recognition (FG 2019), pp 1–8 Demirjian S, Chertow GM, Zhang JH, O’Connor TZ, Vitale J, Paganini EP et al (2011) Model to predict mortality in critically Ill adults with acute kidney injury. CJASN 6(9):2114–2120 DeVol R, Bedroussian A, Charuworn A, Chatterjee A, Kim IK, Kim S et al (2007) An unhealthy America: the economic burden of chronic disease (cited 7 Jan 2022). Available from: https:// drum.lib.umd.edu/handle/1903/25632 de Vos J, Visser LA, de Beer AA, Fornasa M, Thoral PJ, Elbers PWG et al (2021) The potential costeffectiveness of a machine learning tool that can prevent untimely intensive care unit discharge. Value in Health [Internet]. 2021 Oct (cited 22 Dec 2021). Available from: https://www.sciencedi rect.com/science/article/pii/S1098301521017423 Dibeklio˘glu H, Hammal Z, Cohn JF (2018) Dynamic multimodal measurement of depression severity using deep autoencoding. IEEE J Biomed Health Inform 22(2):525–536. https://doi. org/10.1109/JBHI.2017.2676878 Du X, Kurmann T, Chang P-L, Allan M, Ourselin S, Sznitman R et al (2018) Articulated multiinstrument 2-D pose estimation using fully convolutional networks. IEEE Trans Med Imaging 37(5):1276–1287 Ekman P, Friesen WV (2019) Facial action coding system [Internet]. American Psychological Association (cited 10 Jan 2022). Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/t27 734-000 Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639): 115–118. https://doi. org/10.1038/nature21056 Ettman CK, Abdalla SM, Cohen GH, Sampson L, Vivier PM, Galea S (2020) Prevalence of depression symptoms in US adults before and during the COVID-19 pandemic. JAMA Netw Open 3(9):e2019686

References

49

Fialho AS, Cismondi F, Vieira SM, Reti SR, Sousa JMC, Finkelstein SN (2012) Data mining using clinical physiology at discharge to predict ICU readmissions. Expert Syst Appl 39(18):13158– 13165 Flaws DF, Barnett A, Fraser J, Latu J, Ramanan M, Tabah A et al (2021) A protocol for tracking outcomes post intensive care. Nursing in Critical Care [Internet] (cited 22 Dec 2021). Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/nicc.12613 Freeman WJ, Weiss AJ, Heslin KC (2018) Overview of U.S. Hospital Stays in 2016: Variation by Geographic Region. HCUP Statistical Brief #246. December 2018. Agency for Healthcare Research and Quality, Rockville, MD. www.hcup-us.ahrq.gov/reports/statbriefs/sb246-Geogra phic-Variation-Hospital-Stays.pdf. Accessed 27 Nov 2019 Funke I, Jenke A, Mees ST, Weitz J, Speidel S, Bodenstedt S (2018) Temporal coherence-based self-supervised learning for laparoscopic workflow analysis. In OR 2.0 context-aware operating theaters, computer assisted robotic endoscopy, clinical image-based procedures, and skin image analysis, edited by Stoyanov D, Taylor Z, Sarikaya D, McLeod J, González Ballester MA, Codella NCF, Martel AL, et al., 85–93. Lecture notes in computer science. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-01201-4_11 Funke I, Mees ST, Weitz J, Speidel S (2019) Video-based surgical skill assessment using 3D convolutional neural networks. Int J CARS. 14(7):1217–1225 García-Peraza-Herrera LC, Li W, Fidon L, Gruijthuijsen C, Devreker A, Attilakos G et al (2017) ToolNet: holistically-nested real-time segmentation of robotic surgical tools. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 5717–5722 Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, Rosenwald DP (2014) Nonverbal social withdrawal in depression: evidence from manual and automatic analyses. Image vis Comput 32(10):641–647 Gu W (2017) Non-intrusive blood glucose monitor by multi-task deep learning: PhD forum abstract. In: Proceedings of the 16th ACM/IEEE international conference on information processing in sensor networks [Internet]. Association for Computing Machinery, New York (cited 8 Feb 2022) (IPSN’17), pp 249–50. Available from https://doi.org/10.1145/3055031.3055049 Guo H, Gao S, Tsui K-L, Niu T (2017) Simulation optimization for medical staff configuration at emergency department in Hong Kong. IEEE Trans Autom Sci Eng 14(4):1655–1665 Hamashima C, Goto R (2017) Potential capacity of endoscopic screening for gastric cancer in Japan. Cancer Sci 108(1):101–107 Hamm J, Kohler CG, Gur RC, Verma R (2011) Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. J Neurosci Methods 200(2):237–256 Hao T, Xing G, Zhou G (2013) iSleep: unobtrusive sleep quality monitoring using smartphones. In: Proceedings of the 11th ACM conference on embedded networked sensor systems [Internet]. Association for Computing Machinery, New York (cited 6 Jan 2022) (SenSys’13), pp 1–14. Available from https://doi.org/10.1145/2517351.2517359 Hirasawa T, Aoyama K, Tanimoto T, Ishihara S, Shichijo S, Ozawa T et al (2018) Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer 21(4):653–660 Holden MS, Ungi T, Sargent D, McGraw RC, Chen ECS, Ganapathy S et al (2014) Feasibility of real-time workflow segmentation for tracked needle interventions. IEEE Trans Biomed Eng 61(6):1720–1728 Huang X, Yan F, Xu W, Li M (2019) Multi-attention and incorporating background information model for chest X-ray image report generation. IEEE Access 7:154808–154817 Jacob V, Chattopadhyay SK, Proia KK, Hopkins DP, Reynolds J, Thota AB et al (2017) Economics of self-measured blood pressure monitoring: a community guide systematic review. Am J Prev Med 53(3):e105–e113 Jan A, Meng H, Gaus YFBA, Zhang F (2018) Artificial intelligent system for automatic depression level analysis through visual and vocal expressions. IEEE Trans Cognit Dev Syst 10(3):668–680

50

2 Frontier of Smart Healthcare Engineering Management

Jin Y, Dou Q, Chen H, Yu L, Qin J, Fu C-W et al (2018) SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans Med Imaging 37(5):1114– 1126 Jing B, Xie P, Xing E (2018) On the automatic generation of medical imaging reports. In: Proceedings of the 56th annual meeting of the association for computational linguistics (vol 1: Long Papers) [Internet]. Association for Computational Linguistics, Melbourne (cited 2 Jan 2022), pp 2577– 2586. Available from: https://aclanthology.org/P18-1240 Ju C, Zhang S (2021) Doctor recommendation model for pre-diagnosis online in China: integrating ontology characteristics and disease text mining. In: 2021 IEEE 6th international conference on big data analytics (ICBDA), pp 38–43 Kannan S, Yengera G, Mutter D, Marescaux J, Padoy N (2020) Future-state predicting LSTM for early surgery type recognition. IEEE Trans Med Imaging 39(3):556–566 Kim S-H, Chan CW, Olivares M, Escobar G (2015) ICU admission control: an empirical study of capacity allocation and its implication for patient outcomes. Manage Sci 61(1):19–38 Kramer AA, Higgins TL, Zimmerman JE (2012) Intensive care unit readmissions in U.S. hospitals: Patient characteristics, risk factors, and outcomes. Crit Care Med 40(1):3–10 Laina I, Rieke N, Rupprecht C, Vizcaíno JP, Eslami A, Tombari F et al (2017) Concurrent segmentation and localization for tracking of surgical instruments. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S (eds) Medical image computing and computer-assisted intervention—MICCAI 2017. Springer International Publishing, Cham (Lecture Notes in Computer Science), pp 664–672 Lamiri M, Grimaud F, Xie X (2009) Optimization methods for a stochastic surgery planning problem. Int J Prod Econ 120(2):400–410 Le Gall J-R, Lemeshow S, Saulnier F (1993) A new simplified acute physiology score (SAPS II) based on a European/North American Multicenter Study. JAMA 270(24):2957–2963 Lee HW, Kim K (2017) Acquisition of histologic diversity contributes to not only invasiveness but also lymph node metastasis in early gastric cancer. Pathol-Res Pract 213(9):1023–1028 Lee HK, Jin R, Feng Y, Bain PA, Goffinet J, Baker C et al (2019) An analytical framework for TJR readmission prediction and cost-effective intervention. IEEE J Biomed Health Inform 23(4):1760–1772 Lépine J-P, Briley M (2011) The increasing burden of depression. Neuropsychiatr Dis Treat 7(Suppl 1):3–7 Lewis PS (2019) Oscillometric measurement of blood pressure: a simplified explanation. A technical note on behalf of the British and Irish Hypertension Society. J Hum Hypertens 33(5):349–351 Li X, Liu D, Geng N, Xie X (2019) Optimal ICU admission control with premature discharge. IEEE Trans Autom Sci Eng 16(1):148–164 Li Y, Xiong H, Li X (2020) Recommending doctors online based on combined conditions. Data Anal Knowl Discov 4(8):130–142 Li F, Chen Z, Tan B, Liu Y, Zhao Q, Fan L et al (2020) Influential factors and prognostic analysis of blood vessel invasion in advanced gastric cancer. Pathol-Res Pract 216(3):152727 Li CY, Liang X, Hu Z, Xing EP (2018) Hybrid retrieval-generation reinforced agent for medical image report generation. In: Proceedings of the 32nd international conference on neural information processing systems. Curran Associates Inc., Red Hook (NIPS’18), pp 1537–1547 Li AH (2016) Whom to trust when sick? The Wei Zexi Incident, the Chinese Internet and the Healthcare System in China. China Perspectives 2016(2016/4):79–83 Liang Q, Nan Y, Coppola G, Zou K, Sun W, Zhang D et al (2019) Weakly supervised biomedical image segmentation by reiterative learning. IEEE J Biomed Health Inform 23(3):1205–1214 Lin Y-K, Chen H, Brown RA, Li S-H, Yang H-J (2017) Healthcare predictive analytics for risk profiling in chronic care: a Bayesian multitask learning approach. MIS Q 41(2):473–495 Lin K, Hu Y, Kong G (2019) Predicting in-hospital mortality of patients with acute kidney injury in the ICU using random forest model. Int J Med Inform 125:55–61

References

51

Liu X, Wang C, Bai J, Liao G (2020) Fine-tuning pre-trained convolutional neural networks for gastric precancerous disease classification on magnification narrow-band imaging images. Neurocomputing 392:253–267 Luo H, Xu G, Li C, He L, Luo L, Wang Z et al (2019) Real-time artificial intelligence for detection of upper gastrointestinal cancer by endoscopy: a multicentre, case-control, diagnostic study. Lancet Oncol 20(12):1645–1654 Ma X, Song C, Chiu PW, Li Z (2019) Autonomous flexible endoscope for minimally invasive surgery with enhanced safety. IEEE Robot Automat Lett 4(3):2607–2613 Ma Z, Sun A, Yuan Q, Cong G (2015) A tri-role topic model for domain-specific question answering. In: Twenty-Ninth AAAI conference on artificial intelligence [Internet] (cited 14 Feb 2022). Available from: https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9601 Mahmoodi SA, Mirzaie K, Mahmoudi SM (2016) A new algorithm to extract hidden rules of gastric cancer data based on ontology. Springerplus 5(1):312 Marshall JC, Cook DJ, Christou NV, Bernard GR, Sprung CL, Sibbald WJ (1995) Multiple organ dysfunction score: a reliable descriptor of a complex clinical outcome. Crit Care Med 23(10):1638–1652 Maurer DM, Raymond TJ, Davis BN (2018) Depression: screening and diagnosis. AFP 98(8):508– 515 Meares HD, Jones MP (2020) When a system breaks: queueing theory model of intensive care bed needs during the COVID-19 pandemic. Med J Aust [Internet]. 2020 May 15 (cited 21 Dec 2021) 212(10). Available from: https://www.mja.com.au/journal/2020/212/10/when-system-bre aks-queueing-theory-model-intensive-care-bed-needs-during-covid Meng H, Huang D, Wang H, Yang H, AI-Shuraifi M, Wang Y (2013) Depression recognition based on dynamic facial and vocal expression features using partial least square regression. In: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge [Internet]. Association for Computing Machinery, New York (cited 9 Jan 2022) (AVEC ’13), pp 21–30. Available from: https://doi.org/10.1145/2512530.2512532 Mongia A, Jhamb N, Chouzenoux E, Majumdar A (2020) Deep latent factor model for collaborative filtering. Signal Process 169:107366 Mori Y, Kudo S, Mohmed HEN, Misawa M, Ogata N, Itoh H et al (2019) Artificial intelligence and upper gastrointestinal endoscopy: current status and future perspective. Dig Endosc 31(4):378– 388 Mukkamala R, Hahn J-O, Inan OT, Mestha LK, Kim C-S, Töreyin H et al (2015) Toward ubiquitous blood pressure monitoring via pulse transit time: theory and practice. IEEE Trans Biomed Eng 62(8):1879–1901 Mullins PM, Goyal M, Pines JM (2013) National growth in intensive care unit admissions from emergency departments in the United States from 2002 to 2009. Acad Emerg Med 20(5):479–486 Nawaz MS, Mustafa RU, Lali MIU (2018) Role of online data from search engine and social media in healthcare informatics [Internet]. Applying big data analytics in bioinformatics and medicine. IGI Global (cited 26 Jan 2022), pp 272–293. Available from: https://www.igi-global.com/cha pter/role-of-online-data-from-search-engine-and-social-media-in-healthcare-informatics/www. igi-global.com/chapter/role-of-online-data-from-search-engine-and-social-media-in-healthcareinformatics/182951 Ofoma UR, Dong Y, Gajic O, Pickering BW (2018) A qualitative exploration of the discharge process and factors predisposing to readmissions to the intensive care unit. BMC Health Serv Res 18(1):6 Pampouchidou A, Simos PG, Marias K, Meriaudeau F, Yang F, Pediaditis M et al (2019) Automatic assessment of depression based on visual cues: a systematic review. IEEE Trans Affect Comput 10(4):445–470 Pampouchidou A, Marias K, Tsiknakis M, Simos P, Yang F, Lemaître G et al (2016) Video-based depression detection using local Curvelet binary patterns in pairwise orthogonal planes. In: 2016 38th Annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 3835–3838

52

2 Frontier of Smart Healthcare Engineering Management

Patriti A, Eugeni E, Guerra F (2020a) What happened to surgical emergencies in the era of COVID19 outbreak? Considerations of surgeons working in an Italian COVID-19 red zone. Updates Surg. 72(2):309–310 Patriti A, Baiocchi GL, Catena F, Marini P, Catarci M (2020b) FACS on behalf of the Associazione Chirurghi Ospedalieri Italiani (ACOI). Emergency general surgery in Italy during the COVID-19 outbreak: first survey from the real life. World J Emerg Surg 15(1):36 Philip NY, Rodrigues JJPC, Wang H, Fong SJ, Chen J (2021) Internet of Things for in-home health monitoring systems: current advances, challenges and future directions. IEEE J Sel Areas Commun 39(2):300–310 Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inf Fusion 37:98–125 Qin F, Lin S, Li Y, Bly RA, Moe KS, Hannaford B (2020) Towards better surgical instrument segmentation in endoscopic vision: multi-angle feature aggregation and contour supervision. IEEE Robot Automat Lett 5(4):6639–6646 Rong G, Mendez A, Bou Assi E, Zhao B, Sawan M (2020) Artificial intelligence in healthcare: review and prediction case studies. Engineering 6(3):291–301 Rosenberg AL, Watts C (2000) Patients readmitted to ICUs. Chest 118(2):492–502 Rottenberg J (2017) Emotions in depression: what do we really know? Annu Rev Clin Psychol 13(1):241–263 Rouzbahman M, Jovicic A, Chignell M (2017) Can cluster-boosted regression improve prediction of death and length of stay in the ICU? IEEE J Biomed Health Inform 21(3):851–858 Sarikaya D, Corso JJ, Guru KA (2017) Detection and localization of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection. IEEE Trans Med Imaging 36(7):1542–1549 SCCM, Critical Care Statistics [Internet]. Society of Critical Care Medicine (SCCM) (cited 21 Dec 2021). Available from: https://sccm.org/Communications/Critical-Care-Statistics Sclafani JA, Kim CW (2014) Complications associated with the initial learning curve of minimally invasive spine surgery: a systematic review. Clin Orthop Relat Res 472(6):1711–1717 Shehadeh KS, Padman R (2021) A distributionally robust optimization approach for stochastic elective surgery scheduling with limited intensive care unit capacity. Eur J Oper Res 290(3):901– 913 Shi P, Chou MC, Dai JG, Ding D, Sim J (2015) Models and insights for hospital inpatient operations: time-dependent ED boarding time. Manage Sci 22:150422112841002 Shin H-C, Roberts K, Lu L, Demner-Fushman D, Yao J, Summers RM (2016) Learning to read chest X-rays: recurrent neural cascade model for automated image annotation. In 2016 (cited 2 Jan 2022), pp 2497–506. Available from: https://openaccess.thecvf.com/content_cvpr_2016/ html/Shin_Learning_to_Read_CVPR_2016_paper.html Shockcor N, Hayssen H, Kligman MD, Kubicki NS, Kavic SM (2021) Ten year trends in minimally invasive surgery fellowship. JSLS 25(2):e2020.00080 Sobin C, Sackeim HA (1997) Psychomotor symptoms of depression. Am J Psychiatry 154(1):4–17 Song J, Qiu Y, Liu Z (2016) Integrating optimal simulation budget allocation and genetic algorithm to find the approximate Pareto patient flow distribution. IEEE Trans Autom Sci Eng 13(1):149–159 Stratigi M, Kondylakis H, Stefanidis K (2020) Multidimensional group recommendations in the health domain. Algorithms 13(3):54 Stratigi M, Kondylakis H, Stefanidis K (2018) FairGRecs: fair group recommendations by exploiting personal health information. In: Hartmann S, Ma H, Hameurlain A, Pernul G, Wagner RR (eds) Database and expert systems applications. Springer International Publishing, Cham (Lecture Notes in Computer Science), pp 147–155 Strum DP, May JH, Vargas LG (2000) Modeling the uncertainty of surgical procedure times: comparison of log-normal and normal models. Anesthesiology 92(4):1160–1167 Sun Y, Thakor N (2016) Photoplethysmography revisited: from contact to noncontact, from point to imaging. IEEE Trans Biomed Eng 63(3):463–477

References

53

Sun M, Liang K, Zhang W, Chang Q, Zhou X (2020) Non-local attention and densely-connected convolutional neural networks for malignancy suspiciousness classification of gastric ulcer. IEEE Access 8:15812–15822 Surdeanu M, Ciaramita M, Zaragoza H (2011) Learning to rank answers to non-factoid questions from web collections. Comput Linguist 37(2):351–383 Tetin I, Antonenko E, Epishev V (2021) Employee health monitoring system using smart insoles. Natl Assoc Sci 1(67):26–30 Thomas CM, Morris S (2003) Cost of depression among adults in England in 2000. Br J Psychiatry 183(6):514–519 Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2017) EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97 Twinanda AP, Yengera G, Mutter D, Marescaux J, Padoy N (2019) RSDNet: learning to predict remaining surgery duration from laparoscopic videos without manual annotations. IEEE Trans Med Imaging 38(4):1069–1078 Valstar M, Schuller B, Smith K, Eyben F, Jiang B, Bilakhia S et al (2013) AVEC 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge [Internet]. Association for Computing Machinery, New York (cited 9 Jan 2022) (AVEC’13), pp 3–10. Available from: https://doi.org/ 10.1145/2512530.2512533 van Sluisveld N, Bakhshi-Raiez F, de Keizer N, Holman R, Wester G, Wollersheim H et al (2017) Variation in rates of ICU readmissions and post-ICU in-hospital mortality and their association with ICU discharge practices. BMC Health Serv Res 17(1):281 van Wyk F, Khojandi A, Kamaleswaran R (2019) Improving prediction performance using hierarchical analysis of real-time data: a sepsis case study. IEEE J Biomed Health Inform 23(3):978–986 Veitch AM, Uedo N, Yao K, East JE (2015) Optimizing early upper gastrointestinal cancer detection at endoscopy. Nat Rev Gastroenterol Hepatol 12(11):660–667 Venugopalan J, Chanani N, Maher K, Wang MD (2019) Novel data imputation for multiple types of missing data in intensive care units. IEEE J Biomed Health Inform 23(3):1243–1250 Viegas R, Salgado CM, Curto S, Carvalho JP, Vieira SM, Finkelstein SN (2017) Daily prediction of ICU readmissions using feature engineering and ensemble fuzzy modeling. Expert Syst Appl 79:244–253 Vincent J-L, Moreno R, Takala J, Willatts S, De Mendonça A, Bruining H et al (1996) The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Springer Wal˛edziak M, Lasek A, Wysocki M, Su M, Bobowicz M, My´sliwiec P et al (2019) Risk factors for serious morbidity, prolonged length of stay and hospital readmission after laparoscopic appendectomy—results from Pol-LA (Polish Laparoscopic Appendectomy) multicenter large cohort study. Sci Rep 9(1):14793 Wang F, Zhang S, Henderson LM (2018) Adaptive decision-making of breast cancer mammography screening: a heuristic-based regression model. Omega 76:70–84 Wang Y, Guo C, Chu Y, Hwang J-N, Feng C (2020) A cross-domain hierarchical recurrent model for personalized session-based recommendations. Neurocomputing 380:271–284 Wen L, Li X, Guo G, Zhu Y (2015) Automated depression diagnosis based on facial dynamic analysis and sparse coding. IEEE Trans Inf Forensics Secur 10(7):1432–1441 WHO (2017) Depression and other common mental disorders: global health estimates. Depression and other common mental disorders: global health estimates (Internet) (cited 9 Jan 2022). Available from: https://apps.who.int/iris/handle/10665/254610 Williamson JR, Quatieri TF, Helfer BS, Ciccarelli G, Mehta DD (2014) Vocal and facial biomarkers of depression based on motor incoordination and timing. In: Proceedings of the 4th international workshop on audio/visual emotion challenge (Internet). Association for Computing Machinery, New York (cited 9 Jan 2022) (AVEC ’14), pp 65–72. Available from: https://doi.org/10.1145/266 1806.2661809

54

2 Frontier of Smart Healthcare Engineering Management

Wu B (2018) Patient continued use of online health care communities: web mining of patient-doctor communication. J Med Internet Res 20(4):e9127 Wujtewicz M, Wujtewicz MA, Owczuk R (2015) Conflicts in the intensive care unit. Anaesthesiol Intensive Therapy 47(4):360–362 Xie Y, Liu S, Yao T, Peng Y, Lu Z (2019) Focusing attention network for answer ranking. In: The World Wide Web conference [Internet]. Association for Computing Machinery, New York (cited 13 Feb 2022) (WWW ’19), pp 3384–3390. Available from: https://doi.org/10.1145/3308558.331 3518 Yang M, Fry MJ, Scurlock C (2015) The ICU will see you now: efficient–equitable admission control policies for a surgical ICU with batch arrivals. IIE Trans 47(6):586–599 Yang L, Jiang D, Xia X, Pei E, Oveneke MC, Sahli H (2017) Multimodal measurement of depression using deep learning models. In: Proceedings of the 7th annual workshop on audio/visual emotion challenge [Internet]. Association for Computing Machinery, New York (cited 9 Jan 2022) (AVEC ’17), pp 53–59. Available from: https://doi.org/10.1145/3133944.313 3948 Ye Y, Zhao Y, Shang J, Zhang L (2019) A hybrid IT framework for identifying high-quality physicians using big data analytics. Int J Inf Manage 47(August): 65–75. https://doi.org/10.1016/j.iji nfomgt.2019.01.005 Yong PL, Saunders RS, Olsen L (2010) Missed prevention opportunities [Internet]. The healthcare imperative: lowering costs and improving outcomes: workshop series summary. National Academies Press (US) (cited 7 Jan 2022). Available from: https://www.ncbi.nlm.nih.gov/books/ NBK53914/ Yu Z, Peng W, Li X, Hong X, Zhao G (2019) Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement. In 2019 (cited 9 Feb 2022), pp 151–60. Available from: https://openaccess.thecvf.com/content_I CCV_2019/html/Yu_Remote_Heart_Rate_Measurement_From_Highly_Compressed_Facial_ Videos_An_ICCV_2019_paper.html Yuan K-C, Tsai L-W, Lee K-H, Cheng Y-W, Hsu S-C, Lo Y-S et al (2020) The development an artificial intelligence algorithm for early sepsis diagnosis in the intensive care unit. Int J Med Inform 141:104176 Zebin T, Chaussalet TJ (2019) Design and implementation of a deep recurrent model for prediction of readmission in urgent care using electronic health records. In: 2019 IEEE conference on computational intelligence in bioinformatics and computational biology (CIBCB), pp 1–5 Zhang Z, Xie Y, Xing F, McGough M, Yang L (2017) MDNet: a semantically and visually interpretable medical image diagnosis network. In 2017 (cited 2 Jan 2022), pp 6428–6436. Available from: https://openaccess.thecvf.com/content_cvpr_2017/html/Zhang_MDNet_A_Semanti cally_CVPR_2017_paper.html Zhao C, Xu L, Huang H (2018) Exploiting user activities for answer ranking in Q&A forums. In: Romdhani I, Shu L, Takahiro H, Zhou Z, Gordon T, Zeng D (eds) Collaborative computing: networking, applications and worksharing. Springer International Publishing, Cham (Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering), pp 693–703 Zhao Z, Jin Y, Gao X, Dou Q, Heng P-A (2020) Learning motion flows for semi-supervised instrument segmentation from robotic surgical video. In: Martel AL, Abolmaesumi P, Stoyanov D, Mateus D, Zuluaga MA, Zhou SK et al (eds) Medical image computing and computer assisted intervention—MICCAI 2020. Springer International Publishing, Cham (Lecture Notes in Computer Science), pp 679–689 Zheng B, Zhang J, Yoon SW, Lam SS, Khasawneh M, Poranki S (2015) Predictive modeling of hospital readmissions using metaheuristics and data mining. Expert Syst Appl 42(20), 7110–7120. https://doi.org/10.1016/j.eswa.2015.04.066 Zhou Y, Ni H, Zhang Q, Wu Q (2019) The noninvasive blood pressure measurement based on facial images processing. IEEE Sens J 19(22):10624–10634

References

55

Zhou X, Jin K, Shang Y, Guo G (2020) Visually interpretable representation learning for depression recognition from facial images. IEEE Trans Affect Comput 11(3):542–552 Zimmerman JE, Kramer AA, McNair DS, Malila FM (2006) Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 34(5):1297–1310 Zolbanin HM, Delen D, Hassan ZA (2015) Predicting overall survivability in comorbidity of cancers: a data mining approach. Decis Support Syst 74:150–161

Chapter 3

Data Utilization and Governance in Smart Healthcare

Healthcare data are the essential elements of healthcare engineering, serving as the foundation to make critical clinical decisions (Yang et al., 2022). Healthcare data can be categorized into different types, including electronic health records, pharmacy prescriptions, health surveys, insurance records, genomics-driven trial data, etc. However, the multi-source healthcare data across diverse healthcare systems results in major obstacles to data utilization. As a result, data governance is critical for effectively organizing and managing the information assets that support healthcare institutions. This chapter addresses healthcare data utilization and governance mechanisms in this regard. We first present a compliant utilization process from data storage to disclosure and emphasize four crucial principles across the entire data governance process. Then data governance framework from a time, space and service perspective was introduced, and some risks need to avoid in the conclusion part, the chapter structure as follows (Fig. 3.1).

3.1 Smart Healthcare Data Utilization 3.1.1 Healthcare Data Utilization Regulations Lack of data standardization is a significant problem affecting the data usage for patient care and the quality of healthcare operations. Standardization of healthcare data enables cross-domain data management professionals to work more efficiently and consistently. Healthcare data standardization regulations are a series of regulations to standardize the formulation and implementation of big healthcare data, which plays a vital role in illegally accelerating data flow and rationality. It comprehensively provides data utilization of the entire cycle of data standards to realize data integrity, timing, and consistency. This section summarized healthcare-related data standards, models, algorithms, and technical specifications (see Table 3.1). © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_3

57

58

3 Data Utilization and Governance in Smart Healthcare

Fig. 3.1 The structure of this chapter

3.1.2 Healthcare Data Utilization Process Massive amounts of healthcare-related data are being generated due to the rapid development of next-generation information technologies, which efficiently help healthcare big data management and provide significant value to all healthcarerelated stakeholders. In this books, Healthcare Data utilization refers to data collection, storage, processing, usage, provision, interaction, and disclosure, which realize big healthcare data from disorderly dispersion to orderly coordination. Healthcare data utilization needs transparency, traceability, immutability, privacy, and security. As a result, it is critical to encompass data extraction and utilization security. The following subsections demonstrated the data utilization process. Healthcare data governance systems face challenges regarding flexible access and different data standardization among organizations. A substantial number of present healthcare systems have different sources. As a result, it is crucial to multisource coding and traceable data quality control in the healthcare compliant utilization process, which improves the quality of information and optimizes healthcare decision-making. Figure 3.2 presents the data aggregation coding and traceable information quality control. Coding errors, missing data, and incorrect information need to clean before data utilization. It consists of a health data store, a security manager, and a temporary dataset saved vertically. The original multi-source data is transferred to the temporary dataset, and the most critical procedure is the temp model merging horizontally; this allows for dividing data into different categories, following that MDM can be transmitted into the standardized model.

3.1 Smart Healthcare Data Utilization

59

Structuring and systematically managing healthcare big data and information assets improves the data quality and optimizes healthcare decision-making efficiently and effectively through a compliant utilization process. The medical records are Table 3.1 Laws, norms, and models related to healthcare data governance Type

Organization

Time

Name

Content

Laws and regulations

Standing Committee of the National People’s Congress

2021

People’s Republic of China Data Security Method

Data security and development, data security system, data security protection obligations, government data security and openness, legal responsibility

Laws and regulations

Standing Committee of the National People’s Congress

2021

People’s Republic of China Personal Information Protection Act

Personal information processing rules, individual rights in personal information processing activities, obligations of personal information processors

Laws and regulations

Standing Committee of the National People’s Congress

2019

People’s Republic of China Basic Health care and Health Promotion Law

Basic medical and health services, medical and health institutions, health personnel, drug supply guarantee, health promotion, supervision, and management

Standard specification

National Health Commission

2018

Safety and Service Management Measures, National Health and Medical Big Data Standards

Medical data standard management, medical data security management, medical data service management (continued)

60

3 Data Utilization and Governance in Smart Healthcare

Table 3.1 (continued) Type

Organization

Time

Name

Content

Standard specification

International Organization for Standardization

2016

Medical Information Security Management Standard ISO/IEC27002:2016

Medical data standard management, medical data security management, medical data service management

Standard specification

International Organization for Standardization

2017

ISO 12052:2017, International Define information Standard for Medical Images objects, data and Related Information structures and semantics, message exchange, point-to-point communication support, media storage, communication support, grayscale images

Standard specification

International Organization for Standardization

2009

Health Information Exchange Standards ISO/HL7 27931-2009

Information exchange, software organization, document and record architecture, medical logic

Standard specification

International Organization for Standardization

2019

Health Information Exchange Standards FHIR Fourth Edition

Basic layer, implementation support, security and privacy, consistency, terminology, interactive technology, medical administration, clinical diagnosis and treatment, clinical reasoning

Standard specification

National Technical Committee on Information Technology of Standardization Administration

2018

Data Management Capability Maturity Assessment Model GB/T 36073-2018

Data strategy, data governance, data architecture, data application, data security, data quality, data standards, data life cycle (continued)

3.2 Smart Healthcare Data Governance Principles

61

Table 3.1 (continued) Type

Organization

Time

Name

Content

Standard specification

National Technical Committee on Information Technology of Standardization Administration

2018

Information Technology Services—Governance—Part 5: Data Governance Specification GB/T 349655-2018

Data governance framework, top-level design, data governance environment, data governance domain, data governance process

Standard specification

Internet Society of China

2021

Data Security Governance Capability Evaluation T/ISC-0011-2021

Assessment level, data security strategy, data life cycle security, basic security

Governance model

DAMA International Data Management Association

2017

Data Management Capability Maturity Model EDM-DCAM

Data management strategy, data management process and capital, data architecture, technology architecture, data quality management, data governance, data operations

fragmented across different hospitals and healthcare-related providers. Therefore, all patient data needs to be integrated automatically to get a continuous and accurate medical history. This can be accomplished by ensuring that all patient healthcare data is maintained up-to-date, traceable, and tamper-proof. Based on the value chain, the data development and utilization process develop a collaborative data repository from multiple sources, including data collection, storage, processing, use, provision, interaction, and disclosure (See Fig. 3.3).

3.2 Smart Healthcare Data Governance Principles Healthcare data utilization and governance outline four important principles, such as quality, privacy, transparency, and security, which can result in remarkable improvements to current healthcare data utilization and governance. The following sections concentrate on the smart healthcare data quality control and safety-related principles.

62

3 Data Utilization and Governance in Smart Healthcare

Fig. 3.2 The method for data aggregation coding and traceable information

Fig. 3.3 Data development and utilization process

3.2 Smart Healthcare Data Governance Principles

63

Fig. 3.4 Workflow on the healthcare quality control

3.2.1 Healthcare Data Quality Control High data quality in healthcare ensures and supports high-quality treatment, favorable patient outcomes, and strategic healthcare decision-making. Data quality control improves the quality of information and optimizes healthcare decision-making; thus, it is critical to implement a complete data quality standardization. This section is to understand quality control for remarkable improvements to data utilization and governance in digital health systems. Multi-source healthcare data are systematically gathered, filtered, and extracted to avoid erroneous outcomes and acquire higher quality data. The procedure for ensuring the quality of healthcare is shown in Fig. 3.4. The primary object of data quality control is to guarantee dataset homogeneity when data is transmitted to ensure consistency of big data. The consistency of data quality control comprises three subdomains: (1) Contextual consistency refers to the extent to which healthcare datasets are utilized within the same field. Relevance, credibility, readability, correctness, and secrecy are critical components of this form of consistency (Caballero et al. 2014). (2) Temporal consistency expresses the concept that data must be comprehended in a consistent time series, resulting in comparable data from a different time slot should be incomparable. (3) Operational consistency incorporates technological operational effects on the creation and utilization of data, which is critical for verifying the reliability of healthcare data. The primary connectivity aspects are availability, mobility, accuracy, completeness, and traceability.

3.2.2 Safety, Confidentiality, and Transparency of Healthcare Data Handling big healthcare data is challenging since the data are usually disseminated across different healthcare institutions. The majority of present healthcare systems are centralized, making them susceptible to data loss due to the growth in cyberattacks, human errors, and natural disasters. Present healthcare systems lack transparency,

64

3 Data Utilization and Governance in Smart Healthcare

Fig. 3.5 Framework on the Healthcare Blockchain. Adapted from Chen et al. (2018)

reliable traceability, immutability, auditability, privacy, and security. By utilizing a blockchain-based data system for smart healthcare, most of the above concerns could be alleviated. All healthcare data saved on the blockchain system follows a defined method, ensuring that healthcare practitioners have a thorough conception of patients’ healthcare histories, which healthcare-related organizations can readily access and use. In addition, all data saved on the blockchain is secure, secret, and transparent (Fig. 3.5). As seen in Fig. 3.4, blockchain is a critical technology for safety, ensuring big healthcare data integrity from several roles. Patients’ healthcare data was stored and maintained in the blockchain. Patients’ data can be authorized to hospitals as they visit; de-identifying or anonymizing data has been a standard approach to ensure participant confidentiality and reduce the risk of disclosures. Simultaneously, service providers include physicians, technicians, pharmacists, data analysts, and healthcare policymakers responsible for providing important healthcare services such as clinical decisions, diagnosis and treatment, authorized prescriptions, and healthcare recommendations. Transparency is one of the most important goals of healthcare data governance, and it undoubtedly improves the efficiency of healthcare services. Transparency of health data assists in ensuring a fully auditable process. Enforcing a higher level of transparency does not imply a break in patient privacy. Simultaneously, blockchain enables transparency via encryptions and control mechanisms, offering authorized control and empowering privacy over healthcare data. Public blockchain involving

3.3 Smart Healthcare Data Governance

65

healthcare-related transactions is traceable. Blockchain technological transparency enables healthcare facilities to have comprehensive knowledge about components.

3.3 Smart Healthcare Data Governance From the perspective of resources, development, and risk, healthcare big data encompasses three levels: (1) it is a healthcare service that increases the quality of diagnosis and treatment that improves healthcare operational efficiency (Yang and Zhou 2015); (2) it has the characteristics of multi-source heterogeneity, correlation complexity, potential value, and quality difference (Aminpour et al. 2020); (3) it is possible to induce iatrogenic risks and privacy leaks (Kohli and Tan 2016). Healthcare data is a valuable resource contributing significantly to healthcare services, health management, and clinic diagnostic and treatment services. Healthcare data must be ensured to support efficient management and decision-making accountability. Data governance is a term used in smart healthcare to describe a collection of organizational management behaviors to increase data value and facilitate scientific decision-making. It is the process of planning, supervising, and controlling the administration of healthcare-related data resources, focusing on who is authorized to make these choices and under what rules and regulations. Through data governance and deep mining, data can transfer into high-value information, which helps patients, healthcare institutions, and society. Figure 3.6 demonstrates healthcare data governance architecture from a macro-level perspective. Big healthcare data integrates and collaborates with ubiquitous healthcare resources such as people, material, and information in the data and information layer. Massive data are generated by physicians, general practitioners, patients, nurses, and technicians from various sources, including physiological data collected by biomedical sensors, diagnosis, and treatment, genomic, medical insurance, social media, etc. And the composition of healthcare data is relatively complex; it can be divided into different categories, such as public healthcare data, clinic data, patients’ data, behavior data, etc. In the development and utilization layer, big healthcare data can be a resource for healthcare decision-making through the collection, storage, processing, use, provision, interaction, and disclosure. It provides identifying, defining, and classifying data in subject domains so that users conveniently identify the positioning, efficient retrieval, and data exchange of information resources. The key principles within the smart healthcare hybrid cloud service platform are data quality, transparency, privacy, and safety. These four principles are throughout all aspects and the full-cycle of healthcare data governance. The hybrid cloud platform copies with the data fragmentation, which can be extracted from each system to form important data. Then, information can be distributed to each system to ensure accuracy and integrity. Specifically, on the one hand, community hospitals typically have a comparable database for storing patient information, including uploading encrypted medical documents and assisting patients in querying the block summary. Before delivering the encrypted summaries to the superior hospital, the community hospital

66

3 Data Utilization and Governance in Smart Healthcare

Fig. 3.6 Smart healthcare data governance architecture

must sign them with its private key. Additionally, authorized community hospitals may serve as consensus orders in the system, enhancing fault-tolerant capabilities. On the other hand, the hospital departments could perform the same functions as community hospitals due to sharing data with cloud access or interoperable local access. The core conception in the data service layer is to collaborate data resources from different dimensions and make full use of them: time, space, and services. First is the residential life-cycle data service; it can support decision-making services through multi-modal data generated by the citizens’ life-course. Second is cross-regional and hyperspace healthcare big data from different healthcare-related spaces; these data can serve various institutions for an intelligent decision. The last dimension is big healthcare data through exchange and fusion to serve patients in pension, prevention, diagnosis, treatment, rehabilitation services. To conclude, healthcare institutions are generating data at a tremendous speed, and it has intrinsic characteristics and attributes such as high dynamicity, multi-source heterogeneity, and quality variations, resulting in enormous quantity and diversity of healthcare big data are typically uninformative. The data governance framework in this scenario is to transition compliance with hospital internal rules and external requirements from manual audits to automated, real-time inspections and changedriven processes that analyze risks quickly.

References

67

3.4 Conclusion Healthcare big data provides great advantages to healthcare institutions when used properly. Therefore, it is beneficial to generate and utilize healthcare data under legislation and regulation in the healthcare industry. To safeguard individual data, regulations come in a variety of ways. For instance, the European Union released the General Data Protection Regulation in 2020, while the Chinese government published the Privacy Protection Law in 2021. Data protection legislation will strengthen with growing public and private concerns about data privacy and security. Sharing health data properly needs to also take into consideration all the healthcare-related stakeholders, including device users, patients, researchers, and companies. The majority of health-related data generated by the IoT devices is controlled by service providers or device manufacturers or is scattered across numerous health care systems, with no clear definition of roles and responsibilities. These databases are beneficial for healthcare applications, research, and commercial endeavors. However, data sovereignty needs to be established since segmented data cannot be transferred outside their restricted settings. Therefore, it is critical to have an effective data sovereignty and security protection system that safeguards healthcare institutions’ whole data operation process to preserve data sovereignty. Therefore, cross-border data sharing necessitates risk assessment and compliance protection in the context of digitization to protect patient privacy and security. In all, data governance is a systematic endeavor related to organizational management practices to enhance the value of data and facilitate scientific decision-making. Data utilization in smart healthcare needs to adhere rigorously to data security laws and regulations. Nonetheless, it creates various governance strategies and technology standards specifically customized to the smart healthcare environment. In addition, regulating healthcare data and establishing data ownership guidelines are critical for avoiding risks.

References Aminpour P, Gray SA, Jetter AJ, Introne JE, Arlinghaus R (2020) Wisdom of stakeholder crowds in complex social-ecological systems. Nat Sustain 3(3):191–199 Caballero I, Serrano M, Piattinni M (2014) A data quality in use model for big data. In: ER 2014: advances in conceptual modelling. Springer, Cham, pp 65–74 Chen Y, Ding S, Xu Z, Zheng H, Yang S (2018) Blockchain-based medical records secure storage and medical service framework. J Med Syst 43(1):5. https://doi.org/10.1007/s10916-018-1121-4 Kohli R, Tan SL (2016) Electronic health records: how can is researchers contribute to transforming healthcare. MIS Q 40(3):553–574 Yang SL, Ding S, Gu DX et al (2022) Healthcare big data driven knowledge discovery and knowledge service approach. Manage World 38(01):219–229 (in Chinese). https://doi.org/10.19744/j.cnki. 11-1235/f.2022.0014 Yang SL, Zhou KL (2015) Management issues in big data: the resource-based view of big data. J Manage Sci China 5(05):1–8 (in Chinese)

Chapter 4

Information Exchange and Fusion in Smart Healthcare

Traditional healthcare information systems lack standardized interface design, making it difficult for intra-hospital, inter-hospital, and regional healthcare data to circulate efficiently, erecting obstacles to regional cooperation, remote diagnosis, and treatment. Healthcare institutions demand uniform standards across different departments to assure data integrity. Data within such an institutional framework is inevitably complex and vast, making data exchange and fusion a significant concern in smart healthcare. Therefore, it is critical to connect various data channels among hospitals and healthcare institutions to achieve high-quality aggregation and cross-border healthcare information fusion (Fig. 4.1). This chapter demonstrates the healthcare information exchange and system interoperability process, in which the Master Patient Index (MPI) is the unique identifier, subsequently, multi-dimensional, multi-time and space proactive supervision, and traceability principles for exchange supervision were present. We then propose a new common framework for multi-source information fusion, and a case study of COVID-19 Treatment information and the “Four One” paradigm. Finally, we finish the chapter by discussing barriers to expertise, operation, resource allocation, and risks to social ethics, security, and privacy.

4.1 Information Exchange in Smart Healthcare 4.1.1 Healthcare Information Exchange Patients are unaware of the multiple locations where their healthcare data is stored. As such, information exchange with other healthcare institutions is critical. However, most healthcare institutions have difficulties in information exchange because of the diversity, volume, and distribution of the ingested information. Given that the interoperability of hospital information is often low, information mobility across various healthcare institutions is severely restricted. This might result from technological or © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_4

69

70

4 Information Exchange and Fusion in Smart Healthcare

Fig. 4.1 The structure of this chapter

organizational impediments, which leave physicians without critical information. In addition, because most standard data mining algorithms are incapable of dealing with a great variety of healthcare information, most important healthcare information will not be properly interchanged across systems. Healthcare organizations with several locations need to translate healthcare data into a standard format and recognized terminology to facilitate data interchange. Healthcare Information Exchange (HIE) is a powerful component in smart healthcare system engineering, which refers to data resources from different healthcare institutions that can be exchanged to improve healthcare big data efficiency. Furthermore, according to the healthcare management requirements, the collected data should be reorganized, supplemented, and composited with various data description information, or metadata. Therefore, it is important to set up system interoperability and preretrieval method across different healthcare institutions to ensure data transparency, standardization, and traceability. Master Patient Index (MPI) is the core element in the system interoperability process. Data can be identified the mapping relationship between different IDs of the same patient in different systems based on unique identifiers such as ID number, SSN, or other related identifiers. When an authorized user requests healthcare-related data, the database will provide the user with the necessary information. We segment tasks into different segments and assigned each node just one duty. In this manner, we can boost the system’s efficiency and adjust the number of various nodes as required, which is critical for the system’s scalability. Users are provided in various departments across hospitals to upload and download data. In this process, endorsers are responsible for achieving consensus. Committers are accountable for adding

4.1 Information Exchange in Smart Healthcare

71

data to the ledger by agreement and ensuring the ledger’s consistency by regularly broadcasting the ledger’s hash value to the whole network. Except these, security classification mechanisms need to be established to fully guarantee information privacy and security during transmission, storage, and invocation. Healthcare organizations implement data governance standards to ensure internal privacy and externally enacted legislation compliance. While most healthcare organizations have specific policies defining how and when privileged users may access healthcare systems and data, they lack an effective means of enforcing, monitoring, controlling, and auditing privileged usage patterns. Some researchers emphasized methods for HIE security in previous research. For instance, three distinct healthcare information exchange providers in Texas to develop a model to examine HIE network providers’ involvement and sustainability (Demirezen et al. 2016), the driving influence of labor input on the HIE usage behavior of isomorphism providers using social network theory and system isomorphism theory (Yaraghi et al. 2015). In addition, a framework for data privacy and protection in a network-physical-social system environment based on location-sensitive hash technology and a disruption mechanism to offer differential privacy protection (Qi et al. 2020).

4.1.2 Healthcare Information Exchange Supervision Supervision of the HIE across various healthcare institutions is critical for data transmission. Information exchange supervision refers to activities such as identification, measurement, monitoring, and early warning of various data quality problems caused by each stage of the data exchange, from planning, acquisition, storage, sharing, maintenance, application, to extinction. Multi-dimensional supervision means healthcare-related institutions at all levels have different needs for information, management, supervision, and maintenance. Multi-time and space proactive characteristic refers to HIE supervision in historical data preservation, real-time dynamic monitoring, and trend analysis. The important element is traceability in healthcare information exchange supervision, such as healthcare data usage time, location, and situation. Suppose there are problems with data quality; it can be quickly traced to sources. Many countries worldwide establish rules and legislation for healthcare information exchange. For example, the Health Insurance Portability and Accountability Act (HIPAA) in the United States, the medical data regulatory system governs the breadth of data consumption and oversight of medical information by specifying general requirements and implementation standards. The Data Protection Act 2018 (DPA2018) of the United Kingdom creates an information commissioner’s office inside the federal government’s regulatory structure to increase monitoring, evaluation, and security assessments of data usage. The European Union specifically acknowledged genetic information usage in the 1997 European Convention on Human Rights and Biomedicine, which prohibits the exploitation of biological and medical breakthroughs and establishes restrictions on bioethics and the right to

72

4 Information Exchange and Fusion in Smart Healthcare

private life. Additionally, it prohibits judgments based on genetic features and regulates predictive genetic testing for therapeutic reasons. The GDPR of 2016 governs the majority of personal data processing as well. To conclude, it is critical to investigate HIE promotion mechanisms that prioritize healthcare data privacy, transparency, and security to supervise the full-cycle healthcare data service process with manageable risks. With the concept of privacy, transparency, and security of data interaction throughout data management that utilizes blockchain architecture and encryption for healthcare protection technologies. The goal is to achieve data compliance in the health information exchange process between healthcare institutions and strengthen the oversight of the entire data sharing process.

4.2 Smart Healthcare Information Fusion Healthcare information systems are being built with the health and well-being of approximately 7 billion people in mind. Recently in the COVID-19 pandemic, the amount of healthcare information produced by healthcare institutions has drastically increased, requiring intelligent connectivity to improve the cooperation between institutions. Therefore, it is critical to conduct a thoughtful analysis of multimodal healthcare data and build trans-regional and trans-institutional collaborative preventive and control mechanisms. Information fusion refers to the process of efficiently combining, integrating, and correlating massive volumes of heterogeneous multi-source healthcare data and information to get high-quality and useful information. For example, integrating nextgeneration information technologies into the healthcare field has resulted in smart hospitals cooperating on cross-domain medical resources, offering an opportunity to address the integration and governance of multi-modal healthcare data.

4.2.1 Healthcare Information Fusion Scenarios and Methods Patient data comprises electrocardiograms, photographs, endoscope videos, etc., in clinical diagnosis and treatment. These unstructured and structured healthcare databases offer an untapped reservoir of data. Next-generation information technologies have quickly established a role in disease diagnostic decision-making. Collaborative healthcare operations within diverse hospital group management organizations improve efficiency by combining several sources and modes of data. There are different scenarios for multi-source and multi-modal information fusion, for instance, public health, hospital operations, clinical diagnosis and treatment, and healthcare supervision. Therefore, it needs to combine various requirements to break through standard differences, quality differences, and timeliness differences.

4.2 Smart Healthcare Information Fusion

73

Information fusion in residents’ health scenarios, for example, covers the data service requirements of the whole process of residents’ healthcare management. For example, a deep learning-based multi-source fusion and forecasting framework that combines weather, air quality, and medical booking time series to provide residential healthcare forecasts was put forward (Piccialli et al. 2021). For information fusion in different medical institutions, a human-organization-technology adaptive data governance framework was established, including system quality, information quality, service treatment, information use, user satisfaction, organizational structure and organizational environment (Yusof et al. 2008). There are four types of information fusion in clinical diagnosis and treatment: medical images, text-based information, multi-modal health data, and physiological signal detection technology. For instance, a multi-modal medical image fusion method was applied based on structural block decomposition and fuzzy logic technology (Yang et al. 2019). To be specific, this method was used to extract salient features to construct incomplete fusion graphs and flexible psychological fusion graphs. After the filter processes the initial fusion map, the weighted average method is used to obtain the final fusion map. Furthermore, to meet the information fusion needs of health supervision scenarios, a supervised deep multi-modal fusion framework to identify egocentric human behaviors in multi-modal data automatically was set up (Bernal et al. 2017), and the complex relationship between the physical environment and human behavior based on multi-source information fusion were made possible by ambient intelligence in hospital space (Haque et al. 2020). Each medical record contains a combination of text, photos, and graphs. Because each information is unstructured and unencrypted, processing these massive files is challenging. The merging of digital healthcare data presents potential concerns for patients and residents. Healthcare records are growing exponentially and must be managed efficiently. The following methods for information fusion can mitigate the risk. The first is integrating multi-modal healthcare information via cross-indexing and mapping. To address the problem of cross-domain data heterogeneity caused by differences in patient identity across departments, this section describes a crossindexing technique for associating high-dimensional heterogeneous medical and health data in the absence of critical information, as well as an innovative medical data model compatible with multiple data modalities and specifications such as text, image, physiological signal, and medical cases. In light of the difficulties associated with synchronous storage of multi-modal information and remote interaction during remote diagnosis and treatment, the crossplatform mixed-frequency synchronous interaction technology for multi-modal information is innovated, and the intelligent assisted consultation method is invented, which enables the synchronous transmission and storage of multi-modal data and thus improves the quality of intelligent remote consultation. Cloud computing emerges as a potential option in this case due to its scalability, flexibility, and robustness, as well as its geographical independence. The second aspect is establishing a healthcare big data platform for intelligent hospitals that is cross-border governed and capable of integrating multiple services.

74

4 Information Exchange and Fusion in Smart Healthcare

This section demonstrates a protocol for cross-border healthcare big data fusion, which implements cross-border governance and multi-service integration in response to data governance challenges such as multimodality, heterogeneity, and deficiency in regional medical collaboration.

4.2.2 Multi-Modal Healthcare Information Fusion Although healthcare information could materialize through different standards, forms, resolutions, etc., the general forms of such information are, although constantly updating, predictable. Moreover, many of these forms also share the same origin. For example, Computed Tomography (CT) results in a series of tomographic images that could be reconstructed into a 3D recreation of the body. On the other hand, endoscopic examination results in endoscopic video, which is also a series of images except for time series, not space series. This section provides a multi-modal healthcare information fusion general framework for data analysis and deep learning. Healthcare information exists in a variety of forms. Single data points, such as patient age, gender, immediate blood pressure, body temperature, are easy to understand and are usually directly utilized as features. Datapoint time series, such as heart rate history, O2 level time series records, blood pressure monitoring series, are still interpretable but less obvious to average human beings. Images are one of the most studied healthcare information forms, such as X-ray images and ultrasound images. Image-series, such as MRI scan images, CT scan images, are of great complexity and usually handled by experts. Audio files could also prove useful in healthcare, such as consultation records with depressed patients, heartbeat sound audio. Video data, such as endoscopic, surgical, and consultation videos, are of much higher complexity and lower information density. To fuse all these forms of information through the means of computation would be a daunting task. However, such fusions are made available through the exponential increase in computing power and the new techniques in deep learning and machine learning. The first two things that need to be confirmed to create a framework are the input and output. Since the input is already clear, we need to define the output of information fusion. Healthcare information analysis is typically introduced to develop decision-supporting or risk control mechanisms. In most cases, the questions are simplified as classification problems or regression problems. That is, computers are either supposed to give a yes/no result (binary classification), a category result (nominal classification), or a number that is close to the ground truth (regression). To achieve this result, various classifiers are utilized in the final step of deep learning, such as Softmax and naive Bayes. The most important part of a multi-modal information fusion model lies in the middle of the whole process. To “fuse” information with different modalities is much more complicated than simply concatenating one information after another. Eventually, extracted features need to be presented to the classifier algorithm, while the exact ways to extract features from different information modals are always

4.2 Smart Healthcare Information Fusion

75

a great conflict in research. In most cases, multi-modal features are constructed via one of the two ways. The first one is to process each modal through different networks separately, then combine the multi-modal features only at the end, called “shadow multiple modalities.” Another way is to process the modals together through deep hidden layers of the same network, called “deep multiple modalities” (Gao et al. 2020). In this case, we shall only consider the latter version of multi-modal information fusion. Within the “deep multiple modalities,” many neural network algorithms also exist, each giving different performances in different problems. Each modality goes through a deep model to result in features and highly abstracted representations from each modal in each case. These representations then are concatenated, resulting in the concatenated multi-modal features. These features, together with the already highly abstracted data features, are then fed into another deep network to ensure the features from different modals could interact with each other. Due to the nature of some of the modals, some of the neural networks allow features from different modals to be fused before the representation. For example, Convolutional Neural Network (CNN) is mainly designed for image feature extractions. However, through clever manipulation of the input data, specifically by converting audio into spectrograms and text series into 1-dimension vectors, information from different modals could be combined together before the steps of finding high-level abstracted features. Other mechanisms such as the attention mechanism could be inserted after the features from each modal are analyzed and before the final deep neural network. We could visualize multi-modal healthcare information fusion deep learning framework with the following Fig. 4.2:

Fig. 4.2 Healthcare information multimodal fusion deep learning framework

76

4 Information Exchange and Fusion in Smart Healthcare

4.3 Information Fusion Framework for COVID-19 Treatment In the healthcare scenarios of COVID-19, pandemic prevention and control need to collect raw data from different data sources and extract information efficiently. Consequently, a full understanding of the patient’s prior medical history is mandatory before prescribing any medication for proper treatment. Generally, a medical practitioner needs a patient’s past medical history to provide superior and quality treatment. In addition, the patient’s medical history enables clinicians to assess various factors, including previous medication use, drug allergy information, and prior treatment records, which may result in the development of more appropriate treatment plans. Unfortunately, most healthcare management systems rely on manual data input, processing, and storage. As a result, many problems exist in integrating information, such as inconsistent data standards, system heterogeneity, and rapid spread. Therefore, we need to propose a framework to solve those problems, called Information Fusion Framework for COVID-19 Treatment, to guarantee continuous patient-centered prevention and control. The “Four One” paradigm: One ID, One Data, One Algorithm, and One Service, can be defined as the use of information and communication technology to enable standardized access and traceability features in the Pandemic Clinical Big Data Center, which conducts COVID PCR testing and fever patient treatments, and designated locations across various departments, including medical institutions, disease control, residential hospital, traffic centers, and public security control center. The system also provides daily data reporting, epidemic mapping, analysis, epidemiological research, and disease-specific research, as shown in Fig. 4.3. One ID. Multiple dimensions of correlation could be used to standardize diverse types of patient data supplied by hospitals, disease control, reporting, and political data bureaus and to get patient information. Real patient information data was discovered based on the unique identifiers of patients, which is called the Master Patient Index (MPI). After the identity information is recorded, the personnel information may be merged and matched. Then each patient can be assigned a unique ID number to guarantee the information is unique. The platform connects all healthcare systems to maintain patient data integrity and intelligently matches the data from each system by defining matching standards such as HL7, V2, V3, and IHE PIX/PDQ (see Fig. 4.4). One Data. Patient EHR could originate from various medical facilities and include physician information, patient information, medical image data, pandemic-related data, etc. Such data and information must conform to national, provincial, and industry standards. In addition, a mapping tool should be designed to support a wide variety of source database formats and value set ranges, and the rate of automated mapping match should be high. Finally, standard data output must be considered, including a consistent coding process, identification document, consistent medical record category, and administration methods (See Figs. 4.4 and 4.5).

4.3 Information Fusion Framework for COVID-19 Treatment

Fig. 4.3 Information fusion framework for COVID-19 treatment

Fig. 4.4 “One ID” framework

77

78

4 Information Exchange and Fusion in Smart Healthcare

Fig. 4.5 “One Data” framework

One Algorithm. The algorithm engine used to handle multi-modal healthcare data need to be specified. Healthcare-related information and data could include text, voice, image, and video in the diagnostic and treatment process. To be exact, corpus construction, semantic analysis, semantic fusion, and semantic mining are the essential components of the text. Voice processing techniques include feature extraction, semantic alignment, noise reduction and frequency enhancement, and semantic understanding. Classification of images, object detection, object localization, and object segmentation are all examples of image extraction. Finally, video processing methods include slicing, prediction, real-time tracking, and quality control. For instance, Rib Suppression Algorithm Based on U-Net in Chest Radiographs (Jiao et al. 2019), a deep-learning pipeline for the chest X-ray diagnosis and discrimination of viral, non-viral, and COVID-19 pneumonia (Wang et al. 2021), deep learning models of the Long-Short-Term Memory(LSTM) model (Chimmula and Zhang 2020), GRU and Bi-LSTM (Shahid et al. 2020). One Service. The precise concept of “high-quality service” varies according to the application. One of information fusion’s goals is to create accurate knowledge inferences, which is critical for the success of IoT systems. Information fusion enhances network performance in IoT contexts based on data transfer. Additionally, the information is reliable and consistent, which is critical for using large healthcare data to assist prevention and management. The functions of data sharing, epidemic mining, and closed-loop management must be emphasized via data interchange and multi-source information fusion. The following Fig. 4.6 presents details in One Service.

4.4 Risk Analysis and Conclusion The exchange of information between institutions is a double-edged sword of healthcare efficiency and patient safety. While artificial intelligence technologies are starting to be employed in clinical diagnostics, the following several barriers and risks must be solved or avoided.

4.4 Risk Analysis and Conclusion

79

Fig. 4.6 “One Service” framework

Expertise barrier. Technical expertise is the most critical component of effective innovation. As a result, a lack of technical competence may result in an organization’s inability to accept technology developments. For example, Healthcare data analysis efforts will be futile until healthcare institutions examine all data acquired in a warehouse. Furthermore, smaller healthcare institutions, particularly without technical knowledge, cannot effectively employ and adapt a medical big data system, resulting in system performance deterioration. Therefore, it is thought that the skill of medical personnel is an essential element in the effective adoption of healthcare big data systems. Operation barrier. Operation barrier impedes innovation in the “specialization trap,” which refers to overspecialization and reluctance to change. Given the massive amounts of data generated daily, ranging from terabytes to exabytes, standard relational databases are limited in their ability to support big healthcare data storage. As a result, big data is gaining prominence in data development and administration. Resource constraint. The time and expense of using big medical data are the primary reasons for the failure to construct a big data warehouse in the healthcare sector. In addition, a lack of data standards results in the rising cost of developing healthcare big data systems. As a result, healthcare institutions have the challenge of allocating sufficient financial and human resources to turn raw data into valuable knowledge.

80

4 Information Exchange and Fusion in Smart Healthcare

A great portion of research in smart healthcare is devoted to innovating healthcare services via new technology. However, it is worth noting that such advances often introduce possible weaknesses into the existing system. Therefore, certain risks should be considered. The first is social and ethnic risks. AI has transformed virtually every part of medicine while also bringing techniques that pose ethical questions about privacy, autonomy, liberty, and fairness. The second category is risks to privacy and security. Concerns about privacy and security come naturally due to the usage of such large amounts of data since sensitive personal information may be disclosed to the public and abused.

References Bernal EA, Yang X, Li Q et al (2017) Deep temporal multimodal fusion for medical procedure monitoring using wearable sensors. IEEE Trans Multimedia 20(1):107–118 Chimmula VKR, Zhang L (2020) Time series forecasting of COVID-19 transmission in Canada using LSTM networks. Chaos, Solitons, Fractals. 135:109864–109864 Demirezen EM, Kumar S, Sen A (2016) Sustainability of healthcare information exchanges: a game-theoretic approach. Inf Syst Res 27:240–258 Gao J, Li P, Chen Z, Zhang J (2020) A survey on deep learning for multimodal data fusion. Neural Comput 32(5):829–864 Haque A, Milstein A, Fei-Fei L (2020) Illuminating the dark spaces of healthcare with ambient intelligence. Nature 585(7824):193–202 Jiao QL, Zhu M, Wang BQ, Liu CL (2019) Rib suppression algorithm based on U-net in chest radiographs. Comput Syst Appl 28(10):164–169 (in Chinese). http://www.c-s-a.org.cn/1003-3254/ 7100.html Piccialli F, Giampaolo F, Prezioso E et al (2021) Artificial intelligence and healthcare: forecasting of medical bookings through multi-source time-series fusion. Inf Fusion 74:1–16 Qi L, Hu C, Zhang X et al (2020) Privacy-aware data fusion and prediction with spatial-temporal context for smart city industrial environment. IEEE Trans Industr Inf 17:4159–4167 Shahid F, Zameer A, Muneeb M (2020) Predictions for COVID-19 with deep learning models of LSTM, GRU and Bi-LSTM. Chaos Solitons Fractals 140:110212. https://doi.org/10.1016/j. chaos.2020.110212 Wang G, Liu X, Shen J, Wang C et al. (2021) A deep-learning pipeline for the diagnosis and discrimination of viral, non-viral and COVID-19 pneumonia from chest X-ray images. Nat Biomed Eng 5(6):509–521. https://doi.org/10.1038/s41551-021-00704-1. Epub 2021 Apr 15. Erratum in: Nat Biomed Eng 5(8):943 Yang Y, Wu J, Huang S et al (2019) Multimodal medical image fusion based on fuzzy discrimination with structural patch decomposition. IEEE J Biomed Health Inform 23(4):1647–1660 Yaraghi N, Du AY, Sharman R et al (2015) Health information exchange as a multisided platform: adoption, usage, and practice involvement in service co-production. Inf Syst Res 26:1–18 Yusof MM, Kuljis J, Papazafeiropoulou A et al (2008) An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inf 77(6):386–398

Chapter 5

Knowledge Inference and Recommendation in Smart Healthcare

Healthcare data has exploded globally as a result of the continual growth of healthcare informatization, digitalization, and intelligence. Hospitals, community health service centers, health-related institutions such as medical examination institutes, and medical IT firms have accumulated a large amount of healthcare data (Bhattacharyya et al. 2020). Data become information for decision-making purposes only when processed, generating knowledge, and providing instruction to users. Knowledge is defined as systematic rules formed by information inference and verification, which is essential for making optimal choices in healthcare. A well-organized approach for smart healthcare knowledge-based service is crucial for improving physician decision-making capability and achieving highquality intelligent diagnosis and treatment, further driving healthcare innovation (Yang et al., 2022). This chapter introduces multi-modal healthcare knowledge inference and graphs from the perspective of information resources. Then, it proposes a healthcare knowledge recommendation and personalized service on the basis of the doctors’ preferences and knowledge validity, the chapter structure is represented by Fig. 5.1.

5.1 Knowledge Inference and Graph Construction in Smart Healthcare For healthcare decision-making, a general medical knowledge map need to be constructed using medical books, professional documents, medical guidelines, and academic monographs as the foundation, and a dynamic knowledge view based on clinical diagnosis and treatment path is created (Malik et al. 2019). Knowledge updating is accomplished via a self-learning mechanism, which has developed a variety of medical entities. Therefore, it is necessary to efficiently understand knowledge inference and restructure graph construction precisely to boost knowledge generation and guarantee knowledge updating in smart healthcare. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_5

81

82

5 Knowledge Inference and Recommendation in Smart …

Fig. 5.1 The structure of this chapter

5.1.1 Healthcare Knowledge Inference In the healthcare field, there are three different kinds of knowledge. The first is general medical knowledge acquired through healthcare-related textbooks, diagnosis and treatment guidelines, and clinical dictionaries. The second is medical and healthcare case knowledge, which contains expert knowledge. The last is medical and health inference knowledge, acquired through various intelligent algorithm mining (Ben-Assuli and Padman 2020). Therefore, it is critical to produce and exploit big healthcare data to accelerate knowledge service innovation and increase resource usage efficiency. The next section describes the technique for managing case-based health knowledge (CBHKM). Case-based Inference (CBI) is a knowledge process similar to the actual process of human decision-making (Gu et al. 2019). When confronted with new challenges, physicians often draw on prior experience dealing with comparable circumstances and then formulate a solution to the present problem by suitably adapting and amending the prior experience. CBI’s knowledge inference process comprises four critical steps, named the 4Rs: Retrieve, Reuse, Revise, and Retain. Healthcare case inference has four distinguishing characteristics; the first is the case pool collecting professional knowledge and containing a wealth of knowledge. The second is that CBI is reused by acquiring historical knowledge, significantly improving problemsolving efficiency. The third is it can recommend a relatively complete initial solution with strong interpretability. The last is CBI can recommend a relatively complete initial solution with strong interpretability (Gu et al. 2017).

5.1 Knowledge Inference and Graph Construction in Smart Healthcare

83

The majority of the CBHKM is devoted to Swarm Intelligence Self-Organizing (SISO), which refers to subjects with knowledge in different institutions achieving indepth connectedness, enabling group knowledge development. Given the knowledgebased service requirements for complex decision-making tasks such as disease prevention, assistant diagnosis, clinical scientific research, and health promotion, it is critical to combine doctors’ clinical experience, medical knowledge, and artificial intelligence technology. In smart healthcare, multiple knowledge disciplines associated with healthcare management cooperate on the diagnostic and treatment service process. This collaboration is motivated by swarm intelligence incentives, consensus building, or knowledge inference. By analyzing the knowledge requirements for smart healthcare management scenarios such as healthcare collaboration, disease prevention and control, and intelligent hospital management, we could design an evaluation system for swarm intelligence agent contribution based on swarm intelligence participation, knowledge absorption capacity, and knowledge storage in smart healthcare. It is necessary to make rational use of domain experts’ knowledge and experience to stimulate swarm intelligence in order to create a more efficient healthcare knowledge service. For specialized healthcare knowledge creation, swarm intelligence tasks require a high time constraint, unified representation, and rigorous organization, such as multi-center clinical diagnosis and treatment tasks. Hence, a multi-modal decisionmaking evidence construction method needs to be considered, utilizing a dynamic Bayesian network model and focusing on causality. In addition, an expert and group decision can be engaged to expedite the resolution process using an evidence-based decision support approach and the reliability distribution function when decision outcomes are in disagreement. Suppose there is an unclear evidence situation, it is a good way to develop a knowledge inference framework on the basis of an analytic synthesis algorithm to facilitate professional group decision fusion and knowledge development in smart healthcare (see Fig. 5.2).

5.1.2 Healthcare Knowledge Graph Construction Public health knowledge bases application is relatively mature currently. For example, the world-renowned public health knowledge base MedlinePlus, WebMD, the rare disease knowledge base Orphanet, and the diabetes health education knowledge base Brainfood, all provide practical support and assistance for the public health consultation and decision-making. However, most service websites are controlled by businesses and lack expert categorization. As a result of the rapid development of a new generation of information technology, it is necessary to build a comprehensive knowledge base in smart health based on this experience. The goal of healthcare big data analysis is to mine massive volumes of data for valuable medical information. However, various issues remained due to the complexity of knowledge sources, including unequal knowledge quality and repeated

84

5 Knowledge Inference and Recommendation in Smart …

Fig. 5.2 Swarm intelligence self-organization mechanism of smart healthcare

knowledge from diverse data sources after developing the SISO method. In addition, the intricacy of the data utilized to develop knowledge often results in difficulty with the generated knowledge’s interpretability. Therefore, to integrate data, information, techniques, and experience in the healthcare industry, it is necessary to disambiguate, validate, and update heterogeneous data, followed by effective knowledge graph technology (KGT). With the development of deep learning technologies, the KGT is rapidly becoming the primary driving force behind the development of artificial intelligence. Incorporating a knowledge graph into a medical information system makes it feasible to express, organize, manage, and use diverse medical big data more effectively, bringing the system’s intelligence closer to human cognitive thinking. Although medical KGT is still in its infancy, current KGT in the medical industry faces several challenges, including poor efficiency limited extensibility. In reality, services based on common medical knowledge or single case facts struggle to meet smart healthcare’s diversified and customized demands. Completing the dynamic building and updating of KGT provides a strong foundation for smart healthcare management decision-making by increasing dependability, iterative updating, and traceability inference capability. Consequently, it is vital to use case knowledge

5.2 Healthcare Knowledge Recommendation

85

and SISO to extend group intelligence consistently, to be precise, the dynamic construction of multi-modal healthcare knowledge graphs takes case construction, heterogeneous information integration, and knowledge graph construction into account. The foundation of creating a multi-modal healthcare knowledge graph is to gather machine-readable multi-modal medical knowledge bases. Developing the multi-modal healthcare knowledge graph is reduced to three modules: knowledge representation, medical knowledge integration, and medical knowledge reasoning. Medical knowledge representation takes a huge quantity of organized, semistructured, or unstructured medical data. It extracts the knowledge graph’s parts, such as entity, relationship, and attribute, and stores them acceptably and efficiently in the knowledge base. Medical knowledge integration, based on knowledge representation, integrates and processes the contents of a multi-modal medical knowledge base. Finally, medical knowledge reasoning enhances the knowledge base’s logic and expression capability and updates or supplements existing knowledge for the multi-modal healthcare knowledge graph (see Fig. 5.3).

5.2 Healthcare Knowledge Recommendation Comprehensive utility and diversity of knowledge recommendation are two important elements. Comprehensive utility refers to reliable knowledge service recommendation that requires accurate knowledge validity data. After gathering the existing knowledge, the non-negative matrix factorization algorithms are utilized to assess the knowledge validity first. The non-negative matrix factorization method can reduce the error between the prediction and the real value, ensuring the determined knowledge validity data is constantly approaching the real value. In real life, various variables influence physicians’ adoption of medical knowledge, including the correlation between knowledge and disease, the subject’s degree of acceptability, and the breadth of knowledge. As a result, a knowledge recommendation model is built that incorporates multi-agent preferences and diversity, and the comprehensive utilization of knowledge is examined when selecting knowledge, taking into account the diverse needs of various agents for knowledge qualities. Regarding the diversity of knowledge recommendation, its function Div is created to guide the system in recommending medical information to physicians, considering the variety of knowledge content and sources in the recommendation list. The specific calculation method of Div is as follows: ⎧ √ ⎪ Div(L) = Div(Lk) · Div(Ls) ⎪ ⎨  Sim i,m s.t.Div(Lk) = 1 − K (K2−1) (5.1) i,m∈L&i=m ⎪ ⎪ ⎩ Div(Ls) = T (L) T

86

5 Knowledge Inference and Recommendation in Smart …

Fig. 5.3 Construction of multi-modal knowledge graph

Div(Lk) and Div(Ls) represent the diversity of knowledge content and the diversity of knowledge sources, respectively. Div(Lk) is calculated by converting in-table diversity (ILS), which represents the average similarity of any two items in the recommendation list, to ensure that a larger diversity; Div(Ls) is calculated using the coverage ratio, T (L) and T represent the number of sources in the knowledge recommendation list and the whole recommendation system, respectively, so the larger the Div(Ls), the greater the diversity of knowledge sources in the recommendation list. Using the arithmetic mean method to fuse the two types of diversity, the recommendation list becomes more varied in terms of knowledge and sources as Div(L) rises (Yang et al. 2022). Three different modes of knowledge suggestion delivery might be referred to. The first path involves matching coarse-grained demand semantic information to the public health knowledge base and directly providing users with smart health knowledge via a dialogue retrieval service based on the ontology knowledge base;

5.2 Healthcare Knowledge Recommendation

87

the second path involves matching customized demand semantic information to the public health knowledge base. The personalized matching is performed on the extracted knowledge units. Finally, various dimensions and aspects of the knowledge units are integrated to provide users with content-oriented, in-depth knowledge services; the third path investigates how to provide third-party resource owners and service providers with content-oriented, in-depth knowledge services. The healthcare knowledge recommendation mechanism (HKRM) is designed to keep track of the residential physical health and to aid physicians in making correct diagnoses and prescription selections. To address the imbalance between supply and demand for medical resources, the suggestion of medical knowledge urges physicians to seek information about diagnosis and treatment services to enhance their diagnostic and treatment skills. However, doctors lack the necessary information to acquire trustworthy knowledge, and present systems do not consider physician preferences when meeting the personalized demands of knowledge services. On this basis, HKRM is built using user needs and health portraiture; the following content is based on the structure of “User demand image construction- Demand and resource matching- Knowledge recommendation mechanism.” User demand image construction is the first step. The foundation of offering knowledge recommendations is ontology-based and real-time modeling, as well as discovering user demands. The user portrait has gained widespread attention, referring to the process of mining the user’s demographic characteristics, social network relationships, and behavioral patterns, then summarizing and abstracting them into a labeled form. User requirements portrait is critical for articulating user demands and investigating individualized user knowledge advice. The user’s fundamental data and indicators (such as heart rate, weight, blood pressure, illness history and so forth) may be examined through a personal health big data management and analysis platform; as a result, a user’s health picture can be produced. Simultaneously, based on user interaction data (such as browsing content, browsing behavior, and background knowledge), the user’s natural qualities, interest attributes, social attributes, and ability attributes can be adjusted to create a user’s demand profile. Matching demand with resources is an important step. The user demand semantic network is constructed to provide knowledge suggestions for user demands, and it is important to accomplish semantic matching between user needs and knowledge recommendations. One of the most effective solutions is to describe the user’s information using machine-understandable structured knowledge, which gives knowledge needs and suggestions, notably by constructing a semantic network of user requirements. The purpose of establishing a demand semantic network is to learn from the Web of Data construction approach, incorporate linked data technology, and develop a demand semantic network comparable to a data network. After obtaining user demand information, the linked data technology is used to standardize and semantically describe the internal associations of various demand nodes to create a demand network, enabling the interconnection and integration of demand data. A knowledge recommendation mechanism is formed in the end. After constructing the demand semantic network, it is vital to investigate how to match health knowledge resources to deliver customizable suggestions. When examining the routes

88

5 Knowledge Inference and Recommendation in Smart …

for knowledge recommendation, the following issues must be considered: breakdown of complex needs, matching mechanism and algorithm, and customized recommendation scheme design.

5.3 Healthcare Knowledge-Based Personalized Service Traditional knowledge services do not adequately account for doctors’ dynamic, personalized needs and the evolutionary characteristics of full-cycle health services of medical activities. Therefore, it is hard to determine doctors’ knowledge service needs under the new era’s smart medical and health model. In addition, incomplete knowledge data degrades the quality of knowledge services, making it difficult for service subjects to screen knowledge quickly and effectively. As a result, the traditional knowledge-based service is unable to respond actively. Thus, it is necessary to model doctors’ needs and behavioral characteristics based on diagnosis and treatment experience, developing active knowledge services for intelligent healthcare management. This section explains the mechanics and engineering behind the healthcare knowledge service.

5.3.1 Personalized Healthcare Service Mechanism The healthcare big data knowledge service mechanism needs the relative independence of each component and their mutual connection and interaction. This section discusses the critical mechanism that needs to be considered in healthcare knowledge services. The mechanism for knowledge sharing. Patients who share a chronic ailment connect in a dedicated chronic disease community to exchange information about their doctor’s diagnosis, recovery impact, and medicine. There is information exchange among doctors, such as the CBI discussed above. When confronted with the same condition, doctors might benefit from other physicians’ diagnoses to recommend personalized services. The specific implementation can be accomplished by establishing data interface standards, establishing a mechanism for data interconnection among different departments, establishing a mechanism for rewarding and punishing data interconnection, and incorporating data sharing. The mechanism for service security. While intelligence provides efficiency, it also introduces various security issues, including design flaws, data privacy concerns, and network security. This requires establishing a security guarantee mechanism to ensure the system’s normal functioning, regular infrastructure inspection and maintenance, data backup, anomaly detection and interception, network security protection, and risk management. The mechanism for embedding services. With the assistance of the existing intelligent terminal, the health big data knowledge service further expands its ubiquitous

5.3 Healthcare Knowledge-Based Personalized Service

89

service capabilities, and the embedding will be strengthened appropriately. Furthermore, health big data knowledge service emphasizes the development of a continuous and effective interaction mechanism with users, forming and strengthening trust between users and knowledge services through interaction, progressing from cognitive to structural to relational embedding. The mechanism for feedback. User evaluation is a critical measure for assessing knowledge services. After using the service, the customer needs instantly submit feedback through email or telephone follow-up, even can make a customized recommendation about the service. In addition, the system may identify and progressively improve the existing system through user feedback and service records.

5.3.2 Personalized Healthcare Service Design Smart healthcare promotes the growth of healthcare knowledge via swarm intelligence and the production of intelligent knowledge graphs. It allows ongoing knowledge acquisition for various healthcare management situations, providing important support for the growth of smart healthcare systems and knowledge-based service supply. It is required to enhance the capability of the intelligent healthcare system to correlate features, complaints, illnesses, treatments, drugs, impact information, and user health status and behavior information. As a result, we can fully use our vast interdisciplinary knowledge base and deliver high-quality knowledge-based services to fulfill healthcare-related issues’ diversified and customized demands. By overcoming the complex challenge of independent and dynamic optimization of healthcare knowledge-based service, we develop a full-cycle service model, including knowledge creation, inference, application, and feedback, as illustrated in Fig. 5.4. The personalized and diverse information service focuses on knowledge preference, screening, suggestion, and interactive feedback. In smart healthcare, it is important to accurately model the knowledge requirements of residents, medical

Fig. 5.4 Healthcare knowledge-based service design

90

5 Knowledge Inference and Recommendation in Smart …

staff, managers, scientific researchers. Therefore, we could develop a model of participants’ healthcare knowledge demands and preferences to provide a dataset describing the causation, function, and associated processes inside the healthcare knowledge ecosystem. Precisely, constructing a multi-granularity dynamic knowledge recommendation model, formulating an optimization strategy for the knowledge recommendation service based on subjective user feedback and knowledgebased service utility, and establishing the information feedback mechanisms are essential for personalized knowledge-based recommendation in the smart healthcare environment. By utilizing the current knowledge and demand models with a collaborative filtering algorithm, we can create a hidden Markov model to examine patients’ healthcare requirements on the basis of demographic information, knowledge acquisition, and adopted behaviors. Additionally, multiple attribute matching examines the demand for health information at various levels and the multi-modal relationships within healthcare knowledge development. Then, develop a mechanism for multi-granularity knowledge suggestion to address the individualized, diverse, and developing knowledge demands in a smart healthcare setting. Finally, a knowledgebased service optimization approach throughout the active service mode’s life cycle needs to be considered using time series analysis.

5.4 Conclusion The use of knowledge is primarily limited to static medical knowledge such as medical codes, clinical guidelines, and pharmacopeias. In contrast, dynamic clinical knowledge is difficult to utilize effectively, and the value of high-quality medical resources is not fully realized. Implementing a knowledge service system powered by big data in healthcare has resulted in a considerable improvement in healthcare services. Integrating the healthcare knowledge services powered by big data promotes data governance, integration, aggregation, knowledge management, sharing, service, and value presentation. It also enables the comprehensive integration and collaborative services of horizontal and vertical medical and health resources. Effectively developing, utilizing, and transforming the intrinsic value of healthcare knowledge and sharing knowledge reasonably via intelligent networks, it can be dispersed throughout various medical institutions and communities, greatly facilitating the sinking of high-quality medical knowledge. The main content for this chapter is on knowledge generation and the provision of knowledge-based services in smart healthcare management. First, we discussed the self-organization method of swarm intelligence based on smart healthcare. Knowledge creation must be accomplished via swarm intelligence incentives, consensus building, inference, and swarm intelligence knowledge production. The dynamic building of a multi-modal healthcare knowledge graph is then presented. Second, we describe the technique and algorithm for recommending healthcare information. Finally, we developed the customized and diversified knowledge service model,

References

91

which includes an awareness of knowledge preference, information screening, knowledge suggestion, and interactive feedback to provide a full-cycle knowledge service in smart healthcare.

References Ben-Assuli O, Padman R (2020) Trajectories of repeated readmissions of chronic disease patients: risk stratification, profiling, and prediction. MIS Q 44(1):201–226 Bhattacharyya S, Banerjee S, Bose I et al (2020) Temporal effects of repeated recognition and lack of recognition on online community contributions. J Manag Inf Syst 37(2):536–562 Gu D, Liang C, Zhao H (2017) A case-based reasoning system based on weighted heterogeneous value distance metric for breast cancer diagnosis. Artif Intell Med 77:31–47 Gu D, Deng S, Zheng Q, Liang C, Wu J (2019) Impacts of case-based health knowledge system in hospital management: the mediating role of group effectiveness. Inf Manage 56(8):103162 Malik KM, Krishnamurthy M, Alobaidi M, Hussain M, Malik G (2019) Automated domain-specific healthcare knowledge graph curation framework: subarachnoid hemorrhage as phenotype. Expert Syst Appl 145:113120 Yang SL, Ding S, Gu, Xiao D et al (2022) Healthcare big data driven knowledge discovery and knowledge service approach (In Chinese). Manage World 38(01):219–229. https://doi.org/10. 19744/j.cnki.11-1235/f.2022.0014

Chapter 6

Non-contact Physical and Mental Health Monitoring

In smart healthcare, health screening and monitoring represent “Step 0” of the cycle of care. In many senses, health screening and monitoring are very similar to diagnostic tests. As Blumberg (1957) summarized, screening and diagnosis tests reside on the different end of the same spectrum, and they both give a probability of disease, not a certainty. Traditional health screening tests involve methods such as blood tests, blood pressure tests, chest x-ray, etc., which are already at a much lower cost than other diagnosis tests, albeit also lower accuracy. The next-generation information technologies enable even lower-cost options such as non-contact health screening, and even mental health screening. Moreover, the non-contact nature of such health screening also opens up the possibility for continuous monitoring and decentralized “health monitoring”, in which the condition and timing of health screening and monitoring are determined by individuals. In this chapter, we will explore physical health screening and mental health monitoring specifically, through a novel rPPGbased hypertension risk monitoring method and a facial-feature-based depression risk monitoring framework. The framework of this chapter is shown in Fig. 6.1.

6.1 Non-contact Vital Monitoring 6.1.1 Non-contact BP Monitoring and Hypertension Risk Analytics Blood pressure (BP) represents one of the most important vital signs for in-hospital monitoring, together with pulse rate, respiratory rate, etc. Not only does BP infers the immediate condition of the human body, long-term BP irregularities, specifically hypertension, could lead to many undesired diseases and symptoms in vital organs including the heart, blood vessels, brain, kidney, etc. Unlike other diseases and vital

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_6

93

94

6 Non-contact Physical and Mental Health Monitoring

Fig. 6.1 The structure of this chapter and its position in the cycle of care

signs screening procedures, such as X-rays or complete blood counts (CBC), immediate BP measurements are relatively easy to obtain. Thus, hypertension and BP screening are some of the most economically viable health screening/monitoring programs in the current healthcare system. However, BP measurements require specialized equipment and experienced personnel to achieve relative accuracy, which is only good for hospital and clinical conditions. This led to the problem that communal BP continuous monitoring could be plagued with inaccuracy, not only mentioning that the traditional BP measurement techniques could not achieve daily and long-term BP monitoring. An accurate, continuous, environment-adaptable BP monitoring technique is then required to fully realize the potential of BP monitoring. As stated in Chap. 2, rPPG technology and deep learning could be utilized to perform non-contact continuous blood pressure measuring and hypertension risk analytics. rPPG and PPG are influenced by heart activities, vascular wall functions, and surrounding vain/artery conditions, meaning that these signals are heavily complex. Although previous research has shown significant potential in utilizing rPPG and PPG to predict hypertension risk, there are still three main vulnerabilities surrounding the extraction and feature extraction of PPG signals. First of all, the method based on a single sensor signal lacks rigorous medical theoretical support, and the method of multi-sensor signals has the problems of data matching and calibration, and cannot guarantee the simultaneity of the two signals. Neither of these two contact sensor methods can be applied to special scenarios such as infectious disease, scalding, and infants, and cannot achieve continuous detection. Moreover, current methods based on PPG morphology mainly rely on the medical theoretical basis that PTT is highly correlated with blood pressure. However, the calculation of PTT from PPG signals extracted in a non-contact manner is easily affected by local motion artifacts and difficult to ensure accuracy. Finally, most studies are based on small numbers of healthy participants, which lacks blood pressure data in the abnormal range, suggesting low support for studies of hypertension risk analysis.

6.1 Non-contact Vital Monitoring

95

In order to solve the shortcomings of the traditional contact sensor methods, this study collected face video and palm video and extracted three-way PPG signals from the video to improve the robustness of the PTT algorithm and avoid the impact of local motion on the feature set. It realizes convenient and robust hypertension risk analysis and could deal with special medical scenarios such as infectious patients and burn patients while ensuring the accurate matching of various signals. In order to improve the lack of attention paid to hypertension-related features in existing PPG morphological feature research, this study supplemented the existing feature extraction work for multi-channel PPG signals on the basis of existing research. To support the research work on hypertension risk analysis, we collected a non-contact blood pressure dataset MMVs including 129 volunteers in the laboratory and the Second Affiliated Hospital of Anhui Medical University between October 2020 and January 2021. A total of 813 real samples were collected, of which 180 were from the Endocrinology Department of the Second Anhui Medical Hospital. The dataset consists of one-minute visible-light videos of the face and palm and ground-truth blood pressure. A total of 1069 sample data were used after data enhancement for blood pressure samples in the abnormal range, including 831 samples in the training set (373 samples in the abnormal blood pressure) and 258 samples in the test set (92 samples in the abnormal blood pressure). The age distribution of subjects was 16–83, and the systolic and diastolic blood pressure distributions ranged from [85, 183] and [57, 101], respectively. A.

Non-contact Hypertension Risk Analytics Framework

In this section, PPG signals were obtained from face and palm videos, and the feature extraction of multi-channel signals was carried out on the basis of existing research according to the PPG morphology, and the basic physiological parameters including age, gender, height, weight, and BMI were used as the input of the classification model. The XGBoost binary model was used to detect the risk of hypertension, and the importance of each feature was analyzed to draw relevant conclusions. This study provides an interpretable and supervised objective assessment method for societal blood pressure management and hypertension risk management in special medical scenarios, and proves the validity of the characteristics, enabling real-time hypertension risk analysis. The structure of the method is listed in Fig. 6.2: The first step is to extract PPG from the region of interest (ROI) by using professional software to locate and crop the region of interest in the collected face and hand videos. The regions of interest selected in this study include the forehead, nose, and palm regions. The green channel of the RGB color channels is selected for each video frame and the pixel mean is calculated. Taking 300 consecutive frames in a video, we could obtain a PPG initial signal with a length of 300. The second step is the PPG signal processing: the pixels of the image will tend to have an upward trend or downward trend due to the influence of the environment. De-trending the initial PPG signal can reduce the influence of the offset caused by the environmental interference when acquiring the signal on the later feature extraction, which allows the analysis to focus on the fluctuations in the signal trend itself. The

96

6 Non-contact Physical and Mental Health Monitoring

Fig. 6.2 Framework for non-contact hypertension risk analysis based on multiplex rPPG signal

Fig. 6.3 Example of PPG waveform characteristics and the calculation of PIR, PTT

de-trended signal is filtered to remove noise outside the PPG band and the final PPG waveform is acquired. The third step, feature extraction: extract the peaks and troughs from the three-way PPG signals, and, according to the state-of-the-art theory of blood pressure, use time sliding windows to extract the average interval between the peaks of PPG extracted

6.1 Non-contact Vital Monitoring

97

in the forehead region and the peaks of PPG in the nose region within a cardiac cycle (PTT1), and the average interval between the peaks of PPG extracted from the nose region and the peaks of the palm region (PTT2). Extract the peak-to-trough intensity ratios (PIR1, PIR2, and PIR3) of each region of interest in a cardiac cycle. The final step is hypertension risk analytics. We categorize blood pressure into two categories: normal and abnormal. Together with the collected vitals and basic descriptions, including age, gender, height, weight, etc., these are put into a binary classification model to predict if there exists a hypertension risk. B.

Multiplex rPPG Signal-based Hypertension Risk Analytics

In this chapter, the method of hypertension risk analysis based on face and hand video is elaborated, including PPG signal extraction and signal preprocessing based on face and hand video, PPG waveform morphological feature extraction, and XGBoostbased dichotomous model for hypertension risk analytics. Since the accuracy of face and palm positioning has a great influence on the acquisition of PPG signals, it is necessary to collect face images at the same position and the same size. Using the SiameseFC tracking algorithm, the tracking of the region of interest of the face and hand is realized (Bertinetto et al. 2016). Compared with red/infrared light, the green light has a greater absorption rate for oxyhemoglobin and deoxyhemoglobin and thus has a better signal-to-noise ratio (SNR; Chen et al. 2019). Several studies compared the performance of infrared and green light PPGs and found that green light achieves higher pulse rate detection accuracy than infrared light (Maeda et al. 2008). Therefore, this study selects the green channel among the three RGB channels as the signal extraction source. We calculate the pixel mean of the green (G) channel of the forehead area, nose area, and palm area respectively, and select 300 consecutive frames in a video as the signal-feature-length, thus obtaining the initial PPG signal. The ROI data matrix could be represented with I RG B = {I R ; IG ; I B }, thus the G channel pixel average calculation could be represented with the following formula: m n VG =

1

1 IG

m×n

,

(6.1)

where IG represent the green channel pixel value, m and n represent the length and height of the ROI, respectively. Due to the influence of the environment, the generated PPG signal has an upward or downward trend, which needs to be removed by a de-trending method, so as to obtain a waveform that only contains the fluctuation of the signal itself. Butterworth filters are known for their maximum flatness of passband amplitude (Mello et al. 2007). Therefore we apply a Butterworth filter to remove noise in the frequency band outside [0.8, 2] Hz (representing the heart rate range of 48–120) from the PPG signal. Butterworth filter can be expressed by the following formula:

98

6 Non-contact Physical and Mental Health Monitoring

|H (w)|2 =

1+

1  2n ,

(6.2)

w wc

where H (w) represents the amplitude, w represent the frequency, and wc represent the corner frequency. In order to facilitate the subsequent feature extraction work, it is necessary to avoid the appearance of negative values. This study used the method of normalizing the whole series and adding a minimum negative value. Studies such as Kachuee et al. (2017) have shown that the amount of time it took for pulses to transmit (Pulse transit time, PTT) is negatively corresponding to blood pressure significantly. On the other hand, Ding et al. (2016) argued that since PTT only contains information about high frequency (HF, 0.2–0.35 Hz) blood pressure oscillations, not the low frequency (LF, 0.1–0.15 Hz), a new indicator “photoplethysmogram intensity ratio” (PIR) need to be introduced which correlate with LF blood pressure oscillations significantly, which could increase the accuracy of blood pressure estimations. From the basis of the above research, this study extracted peaks, valleys, PTT, and PIR respectively based on the three-way PPG signals in the forehead, nose, and palm regions, with a total of 11 waveform features. The calculation method of the peaks and valleys is to take the peak (valley) in each cardiac cycle and calculate its mean, the formula is as follows:   n1 L p n2 L v , Vi = , (i ∈ {1, 2, 3}), (6.3) Pi = n1 n2 where i represent the index of PPG signals, n 1 and n 2 represent the number of peaks and valleys respectively, while L p , L v represent the sequence of peaks and valleys. Since there are still some abnormal peaks and valleys in the filtered PPG signal, we adopt an anomaly detection method to find and remove abnormal peaks and valleys. We first calculate the standard deviation and mean of the peak (valley) values, and then calculate the deviation between each peak (valley) value and the mean. If the absolute value of a deviation is greater than twice the standard deviation, the corresponding peaks (valleys) will be discarded as noise when calculating the mean peaks and valleys. The three PTT signals are calculated by applying the time sliding window method to take the absolute value of the time label corresponding to the peak of a cardiac cycle of the forehead region of interest signal minus the absolute value of the time label corresponding to the peak of the other region signal, which could be represented as the following:  P T T1 =

n 1 |Te

n1

− Tb |

 , P T T2 =

n 1 |Tb

n1

− Ts |

 , P T T3 =

n 1 |Te

n1

− Ts |

(6.4)

where Te ,Tb , Ts represents the peak signal timestamp sequences of forehead, nose, and palm, respectively.

6.1 Non-contact Vital Monitoring

99

As defined by Ding et al. (2016), PIR is defined as the ratio of peak intensity to valley intensity in the same cardiac cycle, which is calculated as the following: P I Ri =

Pi , i ∈ {1, 2, 3}, Vi

(6.5)

where i represent the index of PPG signals, Pi and Vi represent the peak and valley value of each PPG signal, respectively. For the classification of hypertension risks, this study used XGBoost to build a binary classification model of hypertension risk. This is due to the strong interpretability and generalization of XGBoost. Furthermore, the regularization boosting technique used can effectively prevent over-fitting and has a high degree of flexibility in operation, while further enhanced with the ability to customize the objective function, evaluation index, regularization item weight, etc. The waveform features extracted from the three-way ROI video are combined with basic human body features such as gender, age, weight, and height as input X to examine the hypertension risk of individuals. Each iteration of XGBoost adds a new tree f k+1 to fit the residual of the previous hypertension risk classification result. The iterative formula of the decision tree is as follows: (k+1)



yi



= yi

(k)

+ μf k+1 (xi ),

(6.6)

where μ represents the learning rate of the decision tree. The higher the learning rate, the faster the model converges while risking not getting to the global optimum (k) result, and vice versa. μ is chosen to be 0.3 in this study. yi represents the prediction function of the k’s iteration. xi represents the ith input sample. The objective function of the decision tree could then be represented as the following: 

Obj =

n 



l(yi , yi ) +

i=1

K 

( f k )

(6.7)

k=1

  l yi , yi is the loss function used by the model. Since we determined the hypertension risk analysis problem studied as a binary classification problem, binary logistic regression is used as the loss function. ( f k ) represents the complexity of the kth tree, which adds a penalty for complex trees in order to keep the decision tree as simple as possible. For each iteration, a function is chosen to minimize the Obj value, thus the final hypertension risk estimation function could be represented by: 



yi =

K  k=1

f k (xi )

(6.8)

100

6 Non-contact Physical and Mental Health Monitoring

6.1.2 Case Study Due to the lack of public datasets for non-contact hypertension risk analysis, we collected data samples of 129 volunteers in our laboratory and hospitals from October 2020 to January 2021 to construct multi-physiological multimodal blood pressure videos (MMVs). The blood pressure part of the dataset consists of 1-min visible light videos of faces and palms and true labels of blood pressure, a total of 813 real samples, of which 180 real samples were collected in the Department of Endocrinology, Second Affiliated Hospital of Medical University between December 17, 2020, and December 24, 2020. Visible light videos were captured by an Intel RealSense D435i depth camera with a resolution of 1920 × 1080 pixels and a frame rate of 25 frames per second. The data were collected under uniform ambient light indoors, with the illumination in the ward being 92–125 lx and the illumination in the laboratory being 59–95 lx. An example of a data sample is shown in Fig. 6.4. During the video acquisition process, we used the Omron electronic blood pressure monitor to record the real blood pressure value as the ground truth. In order to verify the performance of this method in the analysis of hypertension risk, we expanded the sample size of hypertension data by flipping operation, and there were a total of 1069 samples after expansion. Eventually, 831 of the samples are utilized as the training set, 258 used as the testing set, within which there exists 258 and 92 hypertension datapoint, respectively. The experiment is performed on a Linux machine with Intel Xeon® CPU E5-2620 v4 @ 2.10 GHz CPU. We utilized traditional performance measuring metrics such as Precision, Recall, F1-Score, Accuracy, and ROC curve, in order to measure the performances of the

Fig. 6.4 Several samples of MMVs dataset

6.1 Non-contact Vital Monitoring

101

proposed method and baseline models. The definitions of these well-known terms are available in Sect. 8.4.2, and need not be repeated. For baseline models, we chose a few established models for binary classification problems, such as Logistic Regression (LR), Random Forest (RF), Supporting Vector Machine (SVM), and XGBoost. The definition of LR and SVM are available in Sect. 2.2, thus we shall only provide descriptions for RF and XGBoost in this section. Random forest uses the idea of ensemble learning to integrate multiple decision trees and there is no relationship between each decision tree. For classification problems, the final output category is determined by the mode of each decision tree output. Assume that there are M samples in the training set, the generation principle is to extract N samples from them by random sampling with replacement, and each time different samples are taken out to form a new training set. Assume that each sample data has K features, randomly select k features from them, and use the best segmentation attributes as nodes to build a decision tree. The previous steps are then repeated to establish multiple independent trees without pruning, and then vote according to the classification results of these trees to determine the final predicted category of the sample. Its advantages are: (1) It is not sensitive to outliers and missing values; (2) It can process a large number of high-dimensional features without dimensionality reduction operations; (3) Its tree structure supports the explicit analysis of the importance of each feature. The disadvantage of random forest is that it has a weak processing ability for data with high noise. XGBoost is a special case of Boosting algorithm. The idea of Boosting algorithm is to integrate many weak classifiers to form a strong classifier. XGBoost completes the construction of a tree by continuously adding trees to perform feature splitting. Each time a new tree is added, it is learning a new function to fit the residual of the last predicted result. After training, k trees will be obtained. When the category of a new sample is to be predicted, according to the characteristics of the sample, each tree will be assigned a corresponding leaf node, each leaf node corresponds to a category, and finally, the mode of the category corresponding to each tree is counted as the sample category. The advantages of XGBoost are as follows: (1) various antioverfitting strategies are applied. (2) Although there is a serial relationship between trees, the nodes at the same level support parallelization. (3) Added processing of sparse data. From the above metrics and benchmark models, the performance of the proposed method is examined. For Precision, Recall, Accuracy, F1-Score, the result is shown in Table 6.1. For the ROC graph, the result is shown in Fig. 6.5. Table 6.1 Comparison of the effects of benchmark models for hypertension risk detection, with XGBoost having the best performance, shown in bold Precision

Recall

Accuracy

F1-Score

ROC_AUC

LR

0.7142

0.6910

0.7364

0.6980

0.74

RF

0.8032

0.7096

0.7791

0.7243

0.79

SVM

0.6866

0.6608

0.7132

0.6671

0.67

XGBoost

0.8501

0.7990

0.8411

0.8149

0.82

102

6 Non-contact Physical and Mental Health Monitoring

RF

LR

SVM

XGBoost

Fig. 6.5 Comparison of ROC curves of benchmark models for hypertension risk analysis

In terms of precision, XGBoost reached 85.01%, and random forest reached 80.32%, which is better than logistic regression and support vector machine, indicating a low misdiagnosis (false positive) rate. In terms of recall rate, XGBoost outperforms other benchmark models by 79.90%, that is, the missed diagnosis (false negative) rate is the lowest, while the recall rate of SVM is the worst, only 66.08%, and its missed diagnosis rate is the highest. In terms of accuracy, XGBoost is much higher than the other benchmark models, while SVM has the lowest accuracy. In the F1-Score indicator, XGBoost still has an excellent performance, reaching 81.49%, while the F1-Score of other benchmark models is between 66 and 73%. The AUC values of logistic regression, random forest, support vector machine, and XGBoost, that is, the area under the ROC curve, are 0.74, 0.79, 0.67, and 0.82, respectively. In this study, feature importance analysis was also carried out, indicating the number of times the analyzed features were used to divide data attributes in all decision trees. In this study, seventeen different features are collected, i.e. the peaks in the forehead region (f_peak), the peaks in the nose region (n_peak), the peaks in the palm region (h_peak), the valleys in the forehead region (f_valley), the valleys in the nose region (n_valley), the valleys in the palm region (h_valley), forehead area PIR ( f_PIR), nose area PIR (n_PIR), palm area PIR (h_PIR), forehead to nose pulse wave transit time (PTT1), nose to palm pulse wave transit time (PTT2), forehead

6.1 Non-contact Vital Monitoring

103

Table 6.2 Feature information Feature name

Index

Meaning

f_peak

f0

Peaks in the forehead region

n_peak

f1

Peaks in the nose region

h_peak

f2

Peaks in the hand palm region

f_valley

f3

Valleys in the forehead region

n_valley

f4

Valleys in the nose region

h_valley

f5

Valleys in the hand palm region

f_PIR

f6

Forehead area PIR

n_PIR

f7

Nose area PIR

h_PIR

f8

Palm area PIR

PTT1

f9

Forehead to nose PTT

PTT2

f10

Nose to palm PTT

PTT3

f11

Forehead to palm PTT

Height

f12

Height

Weight

f13

Weight

Age

f14

Age

Gender

f15

Gender

BMI

f16

Body mass index

to palm pulse wave transit time (PTT3), gender, age, height, weight, and BMI. The importance of each feature to the model detection effect is analyzed one by one based on XGBoost, which is shown in Table 6.2 and Fig. 6.6.

Fig. 6.6 Feature importance analysis chart

104

6 Non-contact Physical and Mental Health Monitoring

As shown in Fig. 6.6, Variables such as age, BMI, PTT, and palm PPG peak were all important features. Among them, age and BMI play the most important role in the process of dividing attributes of the decision tree, and PTT features also have an important contribution to the model effect, 109 and 106 respectively. The obtained results show that the three-way PPG waveform features extracted in this study all have important contributions to the model effect, and the PTT feature has the highest importance to the model, indicating the effectiveness of the multiple PTT features extracted in this study. The meta-features of individual basic attributes also have a great contribution to the model effect, mainly including age and BMI. These two features reflect blood viscosity, blood vessel wall elasticity, and other blood pressure influencing factors to a certain extent, verifying the previous research theories.

6.2 Non-contact Mental Health Monitoring Based on the research status and medical knowledge of depression, we utilize empirical and quantifiable head visual features for patients and doctors’ depression risk assessment process and propose head feature extraction network, head feature channel attention mechanism, and multi-category depression risk classification mechanisms. The framework of the proposed method is shown in Fig. 6.7: Specifically, the framework consists of three major steps. First, a one-dimensional convolutional residual network is selected to perform the convolution operation on the face, and head features after feature dimension reduction, and the gradient descent method of the ResNet is used to optimize the mapping to obtain stronger linear separability. Then the four types of features are mapped to the same feature space through a fully connected layer. After the feature splicing operation, the four types of features are merged into the overall features, attention mechanism is used to perform fine-grained feature selection on the overall features after splicing, and the importance of the four types of head features is evaluated by adding weights to the features of each dimension, in order to increase the network’s attention to the features which could improve the performances of the model. Finally, the classification of depression risk is performed, which separates patients into different levels of depression instead of a yes/no classification.

6.2.1 Non-Contact Depression Detection and Risk Analytics This section introduces in detail the feature extraction and dimensionality reduction, attention mechanism, and feature fusion involved in implementing the depression risk assessment method integrating head pose and multi-dimensional facial feature modeling. Specifically, the multi-dimensional head feature extraction network will be introduced in section A. The attention mechanism of the fusion of four types of

6.2 Non-contact Mental Health Monitoring

105

Fig. 6.7 Framework of depression risk analysis based on head features

features is arranged in section B, and section C will explain the post-fusion depression risk classification standard and model multi-classification mechanism. A.

Multi-Dimensional Facial Feature Extraction Network

Previous studies have shown that visual behavioral features are thought to contain richer depression-related information. In addition, the clinical symptoms of many patients are highly correlated with the severity of depression, including loss of facial expression, abnormal posture, and abnormal head movements. However, visual behavioral features are more complex than audio data or texts, so capturing behavioral information related to depression in the visual channel has been challenging. When communicating with patients with depression, psychologists need to pay attention to the patient’s emotional state through the observation of facial area and from the behavioral information such as posture features and eye sightline provided by the patient unconsciously. Therefore, the proposed method adds head posture and eye gaze features to assess depression risk based on facial features. As the name suggests, the head posture can describe the moving direction and speed of the human head. According to the head posture, people can infer and understand the intention and motivation of others in communication, thus is widely used in the fields of individual attention detection and behavior recognition (Murphy-Chutorian and Trivedi 2009). The psychological and behavioral science of mental illness shows that head

106

6 Non-contact Physical and Mental Health Monitoring

posture can provide useful information about mental, emotional, personality, and cognitive processes. For example, in a study investigating depression patients’ participation in clinical conversations, researchers quantitatively recorded and systematically analyzed the head posture of depressed patients in a hospital environment. During the communication process, the number of nods and head movement speed of the depressed patients were significantly lower than the control gr oup, reflecting the lower degree of participation of the depressed patients (Dibeklio˘glu et al. 2018). Similarly, there is an intrinsic relationship between eye-sight and head posture, and eye-sight more specifically expresses the direction of the human eye and the position of the focus. Eye-sight is considered an emotional expression in the process of individual communication and plays an important role in interpersonal dependence, social reinforcement, subordinate needs, and emotion. Studies have shown that the number of eye contact is directly proportional to the number of emotions it shows. For example, Bodenschatz et al. (2019) find that when children with severe depression meet unfamiliar adults, the direct eye contact between the two is reduced, the eyebrows Movement is reduced, and blinking frequency is increased compared to normal children. Thus, it is arguable that the inclusion of eye-sight information and feature could reveal individuals’ mental state and contribute to the analysis of depression risks. The head posture angle and eye-sight coordinates are obtained through the CLNF model. For the head posture, the two-dimensional coordinates of the facial landmarks are obtained first. Then, the human head is modeled as an intangible rigid object, and the three-dimensional space coordinates in radians and Euler angles are established with the center of the head as the origin. Using the optical center of the orthographic projection camera and the focal length parallel to the X-axis and Y-axis of the image, the two-dimensional coordinates of the facial landmarks are projected to the three-dimensional space centered on the head through the Perspective-N-Point (PNP) matrix (Wang et al. 2018). Finally, the rotation matrix and translation matrix are solved using the coordinate system transformation matrix, which results in the 3° of freedom of head rotation (pitch, roll, and yaw). The head rotation position could then be calculated through the matrix R = (Rx , R y , Rz ) (Ariz et al. 2019). For the calculation of eye-sight, the two-dimensional facial feature coordinates detected by the CLNF model are utilized to calculate the position of eyes and pupils. The coordinate system is established with the pupil as the origin. Then, the optic axis information from the camera origin image to the pupil center is calculated. The intersection between the optic axis and the eyeball is then obtained, such that coordinate calibration could be done when head postures change. Finally, the gaze vector for each eye is calculated separately from the vector from the center of the 3D eyeball to the pupil position, which is shown in Fig. 6.8. B.

One-Dimensional Residual Convolutional Neural Network

With the development of automatic depression detection, the related research on using neural networks to extract and identify depression features has gradually increased in recent years. One-Dimensional Convolutional Neural Network (1D CNN) has been widely studied for its effectiveness and speed in handling complex tasks and has

6.2 Non-contact Mental Health Monitoring

107

Fig. 6.8 Head rotation coordinate (a) and Ocular coordinate system (b), from Dubey and Tomar (2021)

been widely used in automatic music recognition, damage detection, and electrocardiography (Choi et al. 2017). 1D-CNN is usually a hierarchical model that extracts high-level features from raw data and learns more robust features compared to traditional fully connected neural networks. However, in the face of high-dimensional time-series signals, the neural network needs to optimize a large number of connection weights in order to extract richer information, which greatly reduces the network efficiency. The framework of one dimension residual convolutional block (1D-Res-CNN) is shown in Fig. 6.9. 1D-Res-CNN block has two branches. When the input data is x, the first branch learns the feature f (x) by fitting each weight value after passing through the stacked weight layer structure.x in the second branch is directly mapped by skip connections. The residual structure finally adds the two branches and passes through the ReLU nonlinear activation function to obtain the final output Y . Assume the input is the head feature of length: x = {x1 , x2 , . . . , xl }. For the first weighted layer W1 , slide a 1D convolution kernel of size 1 × k with stride s to the

Fig. 6.9 One dimensional residual convolutional block

108

6 Non-contact Physical and Mental Health Monitoring

input feature signal x in order to get output feature Y1 . For example, the i th feature of Y1 , y1 is: yi = W T xi:i+k−1 + b1 ,

(6.9)

where w is the convolution kernel vector, b1 is the bias of the first weighted layer, xi:i+k−1 is the input feature x start from the i th node with a length of k. Through the ReLU non-linear activation function f the non-linear output yi could be represented with the following:   yi = f W T xi:i+k−1 + b1 , where ReLU non-linear activation function is: z, z > 0 f (z) = 0, z ≤ 0

(6.10)

(6.11)

The final output feature vector Y1 could be represented with:

 Y1 = y1 , y2 , . . . , ym ,

(6.12)

where m is the length of Y  . When s = 1, m = l; s ≥ 2, m = l/s. After the first weighted layer and the activation function f , the output feature signal Y1 is utilized as the input for the second weighted layer W2 and the above operations are repeated to get the final output feature signal Y2 , which is added into identity mapping h(x). Through another ReLU activation function, the final output of the 1D-Res-CNN block Y is the following:   Y = f (Y2 + h(x)) = f W2 Y1 + b2 + h(x)

(6.13)

We let R(x) represent the residual function learned from the stacked weighted layers. The representation of the whole residual block is then the following:

C.

R(x) = W2 f (W1 x + b1 ) + b2

(6.14)

L(x) = f (h(x) + R(x))

(6.15)

Multi-Dimensional Feature Processing Network

Based on the characteristics of data, in order to avoid problems such as excessive parameters and reduced feature learning performance, we proposed to utilize a onedimensional convolution residual module to process and extract depression-related head features, which could more effectively obtain head features without increasing

6.2 Non-contact Mental Health Monitoring

109

Fig. 6.10 Head feature processing network

the training complexity. Two feature extraction methods are designed due to the differences in head features. The facial Action Units (AU) processing network is shown in Fig. 6.10a, while facial landmarks, eye gazes, and the head posture feature processing network are shown in 6.10b. The layer detail of the network is as the following: 1.

2.

3.

4.

Input Layer: The input to 6.10a are the pre-processed Facial AU features, while the input to 6.10b are the post-PCA processed head landmarks, postures, and eye gazing features. Convolution Layer: For 6.10a, the facial AU features are constructed through a time-series vector matrix with fixed dimension number d, convolved from the column dimension of the facial motion unit using a one-dimensional convolution with a kernel size of d × 3. On the other hand, facial landmarks, eye gazing, and head postures are more complicated and abstract. Thus, two convolution layers are used to process these features, shown in 6.10b. Flatten Layer: The sum of the feature vector matrix passing through the convolutional layers and the feature matrix connected using shortcuts are flattened into a 1-dimension vector here. FC Layer: The flattened vectors are the input of the FC layer, with a neuron number of 128.

Dropout mechanisms are also utilized within the 1D-Res-Convolution block. The dropout eliminates p percentages within the convolution kernel, reducing overfitting while making the model much more resistant to noises. For example, the utilization of dropout could be represented with the following: yi = r · W T xi:i+k−1 + b1

(6.16)

110

6 Non-contact Physical and Mental Health Monitoring

yi = f (yi ),

(6.17)

where r is the Bernoulli distribution based on percentage p, thus the residual function could be represented by: R(x) = W2 f (r · W1 x + b1 ) + b2 D.

(6.18)

Attention Mechanism fusion Multi-dimensional Head Features

In recent years, a large number of studies used the mask method to form the attention mechanism in the model. By training the model to autonomously learn and assign feature weights in the feature mapping process, a new weight vector is generated, and important information in the feature space is marked to strengthen the important information and features. However, the four types of head behavior visual features are unevenly distributed in the feature space after fusion, and there is much high-dimensional redundant information on different feature channels. Therefore, making such a network autonomously learn important information in different types of features becomes a key issue. We introduce the attention network as a mechanism that fuses the weight distribution of multi-dimensional head features to fuse four types of features, reduces training parameters, and enhances the information of key types of features. The overall process can be divided into head feature splicing, fusion feature flattening, self-learning weight excitation, and feature scale transformation. The process is shown in Fig. 6.11: 1.

Head Feature Splicing: In order to learn the importance of the head features of each dimension, a simple splicing operation is performed on the four types of head features mapped by the one-dimensional residual convolution (1D-ResCNN) feature processing network and a fully connected layer, represented as the following: X = (X A , X L , X H , X G ),

(6.19)

where X A , X L , X H , X G represent AU features, facial landmarks, Head posture, and Eye gazes. The final overall head feature X is used as the input of the attention mechanism.

Fig. 6.11 Attention mechanism with multi-dimensional head features fusion

6.2 Non-contact Mental Health Monitoring

2.

111

Head Feature Flattening: After the traditional one-dimensional convolution process is performed on the input head features, the feature map f of the head features is obtained. We performed an operation called feature extrusion Fs (·) on the feature map, which is to put feature map through a one-dimensional convolution kernel to obtain a vector of size 1 × d. All information on feature map f will be flattened into a single corresponding value, which could highlight each feature’s information and make the weight learning process of each dimension feature more accurate. z = Fs ( f d ) =

3.

L 1 f d ( j) L j=1

Self-learning Weight Excitation: After the Fs (·) operation, the self-learning weight excitation operation Fex (·, W ) is undertook to train and learn the weight distribution between each feature vector. Specifically, the non-linear excitation function is used to calculate the weight of each dimension feature to continuously adjust the network weight W when the network is back-propagated for training to learn the changing patterns of the importance of each dimension feature in each training process. s = Fex (z, W ) = δ(W z) δ=

4.

(6.20)

1 , 1 + e−x

(6.21) (6.22)

where δ is the sigmoid function. As a common step function with a value range of (0,1), the sigmoid function can solve the extreme distribution of weights caused by other excitation functions due to its smooth transition between linearity and nonlinearity and could avoid causing excessive feature weights. The product of the 1 × d vector after the sigmoid function is directly utilized as probability weight s. Feature Scale Transformation: Based on ResNet, we utilized skip connections to optimize network structure and reduce the size of model parameters. This allows the learning of global information through a convolution kernel with the shallower layers of the network to increase the correlation between different features and improve the quality of underlying features. The excitation weight s is used to scale the multi-dimensional feature map f input by the skip connection, and the features of each channel domain between different head features are weakened and enhanced at different scales to obtain head feature map f˜d with unique weight channel attention distribution. f˜d = Fscale ( f d , sd ) = sd · f d

(6.23)

112

6 Non-contact Physical and Mental Health Monitoring

After using attention mechanism to extract the one-dimensional head feature vector and adjust the convolution kernel size, optimizer type, and hyperparameter selection in the structure, using different hyper-parameters and optimizers could provide a more efficient way to train the attention mechanism in the network, and quickly obtains the weight ratio of the signals in each channel, which is conducive to improve the accuracy of the model and reduce the computation load. E.

Multi-category Depression Risk Analytics Standard

In clinical practice, there is no consensus on the evaluation criteria for depression classification in the world, and different depression classifications correspond to different treatment plans. However, there do exist a few commonly used depression classification models. For example, European Staging Model (ESM) separate depression condition into five levels based on the difference in treatment periods. Although this indicates the time it could take for patients to recover, it lacks a concrete classification method and is low in operability (Ruhé et al. 2012). The Antidepressant Treatment History Form (ATHF) added other influence factors, such as antidepressant prescription category, amount, and the recovery situation, into the scale and categorized depression patients from Level 0 to Level 5 (Sackeim 2001). The Massachusetts General Hospital Staging Model (MGH-S model) further included the influencing factors and intervention plans. The level categorization system is also converted into a numerical system, pinpointing each patient more accurately on the scale (Ruhé et al. 2012). While the classification of already diagnosed depression patients could rely on medication and intervention, the un-diagnosed mass requires a different method of classification. A great majority of the pre-diagnosis analysis of depression risk depends on questionnaires. For example, the eight-item Patient Health Questionnaire depression scale (PHQ-8) shown in Table 6.3 is utilized as a diagnostic algorithm in some studies instead of psychiatric interviews in some studies (Kroenke et al. 2009). PHQ-8 has been confirmed to be an effective scale in large-scale clinical research, as a higher PHQ-8 score usually correlates with a heavier degree of depression. Although PHQ-8 could be effectively utilized for fast diagnosis of depression due to its questionnaire nature, psychiatrist knowledge is still needed for a more detailed diagnosis. Based on the classification standard of PHQ-8, a fully connected layer is used in front of the classifier, which spatially integrates the hidden layer features and Table 6.3 PHQ-8 questionnaire scoring and classification

Depression level

Score

Depression risk

0

0–4

Not depressed

1

5–9

Low

2

10–14

Intermediate

3

15–19

Intermediate-High

4

20–25

Severe

6.2 Non-contact Mental Health Monitoring

113

further extracts useful information from the feature map. Multiple fully connected layers can realize a large number of nonlinear transformations, which help to obtain implicit expressions of the input data. The hidden features extracted from the feature maps of the original data are then utilized as the input of the classification layer. Finally, a Softmax classifier is added for depression risk analysis, and a multi-class model is trained using the cross-entropy loss function as the target function of the model. The loss function is designed as shown in Formula (6.24).

Loss = E(P(z), Q(z)) = −

k 

  P j (z)log Q j (z) ,

(6.24)

j=1

where P(z) is the ground truth distribution, Q(z) is the model predicted distribution, K = 0, 1, 2, 3, 4 represent the level of depression risk.

6.2.2 Case Study This section utilizes comparative experiments to examine the classification performance of the proposed Attentional Head Depression Detection (Att-HDD) method. The following parts of the section will introduce the experimental data and environment, the evaluation index of the experiment, the benchmark model for horizontal comparison, and finally, the results and charts of the comparison experiment, and summarize the results. A.

Data

We used the Distress Analysis Interview Corpus of human and computer interview— Wizard-of-Oz interview (DAIC-WOZ) public dataset collected in 2014 (Gratch et al. 2014). After preprocessing the collected features, the overall data contains the complete data files of 183 respondents, and the average duration of each data sample file is about 20 min. In order to protect the information of the subjects and meet the requirements of ethics and morality, the video interview video files of the dataset have been converted by relevant software before being released. All the data do not contain real face information, and the identity of the personnel cannot be identified. The experimental data in this section adds head posture and eye-sight data. The changes of the two types of data over time are shown in Fig. 6.12 as an example. In order to effectively measure the generalization ability of the neural network model, in the process of training the model, we randomly divided the interviewees into a training set and a test set in a ratio of 8:2. We divide the dataset into four types: healthy, mild, moderate, moderately severe, and severe, marked with 0, 1, 2, 3, and 4, respectively. All results are based on the hardware CPU of Intel Xeon ® CPU E5-2620 v4 @ 2.10 GHz.

114

6 Non-contact Physical and Mental Health Monitoring

Fig. 6.12 Head posture feature change chart (a) and gaze change chart (b)

B.

Metrics

To test the performance of the proposed method, traditional classification metrics such as Accuracy, Precision, Recall, and F1-score are utilized as performance tracking metrics. Although the calculation of these metrics is more complicated than binary classification problems, these metrics are still viable. Furthermore, Receiver Operating Characteristic (ROC) Curves and Area Under Curve (AUC) are also utilized. The specifics of multi-class classification metrics are further explored in Sect. 8.4.2. C.

Benchmark Models

The benchmark models selected for this chapter are Logistic Regression (LR), Support Vector Machines (SVM), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU). This section serves as a brief introduction to these benchmark models, within which some are also utilized in Chap. 8. Logistic Regression (LR) is a classic statistic model widely implemented in classification problems, such as automatic disease diagnosis and health risk factor detections. On top of linear regression, LR maps the results to the (0, 1) interval through the Sigmoid function and divides the estimated probability into corresponding categories through thresholds. Logistic regression models usually use the maximum likelihood method to estimate the model parameters, use the gradient descent method to find the minimum deviation, obtain the classification results, and use the prediction results of linear regression to approximate the logarithmic probability of the true markers. The formulas are as follows: In

y = wT x + b, 1−y

(6.25)

where y is the possibility of sample x to be positive, 1 − y as the possibility of negative (in binary classification problems). w are the weight of the LR model, and b represent the bias.

6.2 Non-contact Mental Health Monitoring

115

Support Vector Machines (SVM) is a supervised machine learning method based on the principle of statistics, which is defined as a linear classifier on the feature space. Because of its advantages in dealing with nonlinear and high-dimensional sample problems, it is often used in data and text classification, pattern recognition, or time series forecasting problems. SVM maps the training samples to a higher-dimensional feature space to find a hyperplane to classify the samples and solves the maximum margin hyperplane for the learning samples, that is, the maximum classification interval from the data point to the hyperplane. These points closest to the hyperplane are called “support vector,” thus the name of the method. The entire solution finding process can eventually be transformed into a convex quadratic programming problem, which can also be solved using stochastic gradient descent when sufficient  samples  are available. For a certain training data sample  D = {(x1 , y1 ), x2 , y2 , . . . , xm , ym }, yi {−1, +1}, the basic formula for SVM could be represented as the following:   1 min ||w||2 s.t.yi wT x i + b ≥ 1, i = 1, 2 . . . m, w,b 2

(6.26)

where w = (w1 ; w2 ; · · · ; wd ) determines the hyperplane vectors and b as the displacement. Long Short-Term Memory (LSTM) is a variant of Recurrent Neural Network (RNN). In addition to using the state h to save information like RNN, it also adds the cell state controlled by the forget gate, the input gate, and the output gate to adjust the long-term memory preservation state C (Graves 2012). Due to the structure of LSTM, it could retain the ability to learn information from long time series while avoiding the vanishing and exploding gradients problems in RNN. The forget gate f t controls the remaining information of the previous cell state Ct−1 in the current cell state Ct through operations such as sigmoid layer and isomorphic matrix. The input gate It controls the current network input xt influence on the current state Ct to avoid unnecessary information coming into the memory of the network. The output gate Ot controls the influence of Ct on the current output ot . The network finally utilizes backpropagate to calculate the weight matrix of every gate module W and bias b. The above explanation of LSTM could be represented as the formulas below: It = σ (Wi · xt + Wi · h t−1 + Wi × Ct−1 + bi )

(6.27)

  f t = σ W f · xt + W f · h t−1 + W f × Ct−1 + b f

(6.28)

Ct = f t × Ct−1 + It × tanh(Wc · xt + Wc · h t−1 + bc )

(6.29)

Ot = σ (Wo · xt + Wo · h t−1 + Wo × Ct−1 + bo )

(6.30)

h t = Ot × tanh(Ct ),

(6.31)

116

6 Non-contact Physical and Mental Health Monitoring

where σ represent the sigmoid function, which limits the output to (0, 1), while tanh limits the output to (−1, 1), × represent Hadamard product. Gated Recurrent Unit (GRU) was introduced to increase the performance of longterm memory problems and are very similar to LSTM (Chung et al. 2014). Although both are variants of RNN, it is shown that GRU is easier to train than LSTM. This is due to the update gate and reset gate structure of GRU. The reset gate rt determines the merge method of the previous state h t−1 and the current input information xt . The update gate Z t selects and update the information between the current state h t and h t−1 . The output information from the update gate and all memory information becomes the output yt . The update gate in GRU could achieve the effect of both the input gate and forget gate in LSTM, reducing the parameter numbers of networks while achieving similar performance. The formulas related to GRU are the following:

D.

Z t = σ (Wz xt + C Z h t−1 ) + b Z

(6.32)

rt = σ (Wr xt + Cr h t−1 ) + br

(6.33)

h t = tanh(Wt xt + C(rt × h t−1 )) + bh

(6.34)

h t = (1 − Z t ) × h t−1 + Z t × h t

(6.35)

Results

To test the performance of the proposed Att-HDD method, the conducted experiments will have their result separated into two parts. First, the superiority of the method needs to be verified by analyzing the comparative experimental results of Att-HDD compared to the baseline model and the improvement level of the classification results by the attention mechanism used in feature fusion. Second, the performance of different visual feature facial motor units, facial landmark, head pose, and eyesight in the multi-classification of depression risk was evaluated, proving the reliability of selecting multi-dimensional features compared to single-dimensional features for depression risk assessment.

6.3 Comparative Experiment with the Baseline Model To compare the performance of the method proposed in this chapter, the baseline models referenced above (LR, SVM, LSTM, GRU) are utilized to conduct a comparative experiment. Using the cross-entropy loss function and Adam optimizer for

6.3 Comparative Experiment with the Baseline Model

117

Table 6.4 Comparative result of LR, SVM, LSTM, GRU, HDD, and Att-HDD, the proposed Att-HDD performed best in all categories, as shown in bold texts Model

Accuracy

Precision

Recall

F1-score

LR

0.6667

0.6467

0.7108

0.6472

SVM

0.6667

0.7674

0.6667

0.6694

LSTM

0.7879

0.7333

0.8158

0.7455

GRU

0.7273

0.7576

0.7908

0.7590

HDD

0.7576

0.8258

0.7576

0.7760

Att-HDD

0.7879

0.8333

0.8158

0.8108

training, the accuracy, precision, recall, and F1-score corresponding to various baseline models are obtained, as shown in Table 6.4. In addition, the ROC curve and AUC data are listed in Fig. 6.12. By comparing the index matrices of each model, analyzing the various indicators of each model in predicting depression risk, and the comprehensive indicators of the predictive ability of each model, the following conclusions are obtained. LR and SVM perform similarly in terms of accuracy, which are considerably lower than neural network-based models. LSTM and Att-HDD are similar in the performance of accuracy. In terms of precision, Att-HDD and HDD are significantly better than the baseline models; however, we could notice that adding the attention mechanism only improved the precision by 0.75%. For the overall F1-score, Att-HDD outperforms all baseline models by at least 6.82%. In the experiment comparing the effect of the attention mechanism, it can be seen that the Att-HDD model using attention in the process of feature extraction and fusion performs the best. This shows that the weights of various head features obtained through the attention mechanism can effectively enhance the features which are conducive to improving the classification effect, which results in obtained head features with better expressive ability (Fig. 6.13). The multi-classification experiment conducts a more in-depth analysis of depression risk detection. The ROC curve and AUC are calculated through each class of depression, with classes 0 to 4 representing the depression level not depressed, low, intermediate, intermediate-high, and severe. Through the macro-average AUC data, it could be noticed that none of the baseline methods could exceed the performance of Att-HDD (0.90). More specifically, although LSTM and GRU have higher or equal AUC for classes 0 (not depressed) and 4 (severe depression), Att-HDD performs significantly better in classes 1, 2, and 3. This suggests that Att-HDD could more accurately determine the more delicate line between severe and mild depression. In all, Att-HDD outperforms all baseline models in both tests.

118

6 Non-contact Physical and Mental Health Monitoring

Fig. 6.13 ROC curve and AUC data for baseline model comparison

6.5 Risk Analytics and Conclusion

119

Table 6.5 Comparison of experimental results between individual modality. We could see that the combination of all four modals show better performance, shown in bold Model

Accuracy

Precision

Recall

F1-score

FAUs

0.7879

0.7874

0.7879

0.7780

Landmarks

0.7576

0.7321

0.7057

0.7676

Head pose

0.7676

0.7657

0.8033

0.7576

Gazes

0.7576

0.7620

0.8076

0.7576

All

0.7879

0.8333

0.8158

0.8108

6.4 Visual Facial Feature Comparative Test As mentioned above, Att-HDD utilizes four kinds of visual facial features as inputs: Facial Action Units (FAU), Facial Landmarks, Head Poses, and Eye Gazes. To understand the influence of each input dimension on the analysis of depression risk, we input individual modality into the Att-HDD model separately, which is shown in Table 6.5. It could be observed that all four modals output satisfactory results, although none single modal could exceed the performance of the combination of all four modals, suggesting the complementary nature between the modals. Accuracy, precision, recall, and F1-score, Att-HDD improves by 2.65%, 8.83%, 1.02%, and 5.63% over the best single feature. From the characteristic ROC curve of each dimension, it can be seen that the effect of Att-HDD on the risk assessment of depression at all levels is above average (Fig. 6.14). The multi-classification result of each modality has some differences. For example, FAU performs lower when classifying severe depression but has a significantly higher performance when classifying those without depression. FAU may be correlated with emotional state and facial expression, while the loss of facial expression and emotion state abnormality is more salient for the determination of the existence of depression rather than the severity of depression. Similarly, head pose and facial landmark perform significantly worse than eye gaze for class 2 (intermediate) depression detection. It is possible that such features do not change significantly until severe depressions are present. It is arguable that a combination of these features with appropriate weight could achieve the best overall result.

6.5 Risk Analytics and Conclusion In this chapter, we introduced non-contact diagnosis technologies in smart healthcare and provided physical health screening and mental health screening examples respectively. For physical health screening, we selected hypertension as an example to demonstrate the ability of next-generation information technologies in such a field. Based on the existing basic medical theory, we explored the acquisition of three-way PPG signals from facial video and palm video, and extracted waveform

120

6 Non-contact Physical and Mental Health Monitoring

Fig. 6.14 ROC curve and AUC comparison between each modality

6.5 Risk Analytics and Conclusion

121

morphological features combined with human meta-features as the input of the XGBoost ensemble learning model. The proposed non-contact hypertension risk analysis model based on multiplex rPPG has good performance in all aspects. Based on the XGBoost model, the importance of each feature is analyzed, and it is concluded that the three-way PPG morphological features have important contributions to the experimental results, and the human meta-features and PTT are the main contributions, which verifies the previous research conclusions. For mental health screening, we present the Att-HDD methods fused with head feature modeling to generate more refined judgments of depression risk. Combined with clinical grading standards for depression, on the basis of using the facial motor unit and facial feature point data, the head posture and eye-sight of patients with depression were included, and multi-dimensional head features were combined to analyze depression risk. One of the fundamental principles of smart healthcare is to alleviate existing medical care pressure through the extension of medical care to health care. Through early identification of physical and mental problems, effective measures could be taken without burdening the more high-priority resources such as emergency care and ICUs. Physical and mental health management thus represents one of the unique features of smart healthcare, unavailable from previous healthcare structures. While intuitively health management and screening could lead to a decrease in both individual healthcare costs and human suffering, risks still could rise from such monitoring and interventions. Although health screening could help individuals to understand and manage their health conditions, misdiagnosis, especially false positives, could lead to nocebo effects, through which actual ailment could manifest. Moreover, health screening could detect temporary fluctuations, which could be interpreted as chronic diseases. For example, blood pressure increases drastically after high cardiac output events, such as sports events. Blood sugar increases and decreases depending highly on the time after the meal. Depression risk diagnosis could be influenced by short-term facial expression changes and also short-term terrible events, such as loss of close relatives. In other words, normal, short-term fluctuation could easily be considered as long-term ailments in some cases. Assuming for a non-specific condition, α percentage of the population is affected by the condition. In cases that all of the population is screened, with PF P as the probability of false-positive, PF N as the probability of false negative. The overall risk added by such a health screening project would be:  R+ = μ(1 − α)PF P + βα PF N ,

(6.36)

where μ represents the nocebo coefficient, which implies the social risk and healthcare system cost due to the false-positive diagnosis. Normally, a false-negative diagnosis does not add to the risk factors, due to that the false-negative diagnosis equals no diagnosis being done (β = 0). In some cases such as transmissive diseases, a false negative could cause some reckless behaviors and allow individuals to transmit diseases onto others. For example, COVID-19 screening false negatives lead to access

122

6 Non-contact Physical and Mental Health Monitoring

allowance through negative test results, in which case the β risk coefficient is positive. To decrease PF P and PF N in real-life implementations, not only do algorithms need to be as reliable as possible, regulations are needed to specify the conditions in which the results are scientifically reliable, and strict rules need to be followed to alleviate fluctuations of such screenings. Existing health screening techniques, such as blood tests, chest X-rays, and ultrasound, all have similar requirements and rules. However, health monitoring presents a new challenge towards the industry, due to that it is impossible to have a regulated environment for non-contact-based health screening, due to most of these screenings being done outside hospital conditions. The reliability and condition adapting ability of health screening technologies would thus become one of the most important research directions in the future.

References Ariz M, Villanueva A, Cabeza R (2019) Robust and accurate 2D-tracking-based 3D positioning method: application to head pose estimation. Comput vis Image Underst 180:13–22 Bertinetto L, Valmadre J, Henriques JF, Vedaldi A, Torr PHS (2016) Fully-convolutional siamese networks for object tracking. In: Hua G, Jégou H (eds) Computer vision—ECCV 2016 workshops. Springer International Publishing, Cham, pp 850–65. (Lecture Notes in Computer Science) Blumberg MS (1957) Evaluating health screening procedures. Oper Res 5(3):351–360 Bodenschatz CM, Skopinceva M, Ruß T, Suslow T (2019) Attentional bias and childhood maltreatment in clinical depression—An eye-tracking study. J Psychiatr Res 112:83–88 Chen X, Cheng J, Song R, Liu Y, Ward R, Wang ZJ (2019) Video-based heart rate measurement: recent advances and future prospects. IEEE Trans Instrum Meas 68(10):3600–3615 Choi K, Fazekas G, Sandler M, Cho K (2017) Convolutional recurrent neural networks for music classification. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 2392–2396 Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:14123555 [cs] [Internet]. 2014 Dec 11 [cited 2022 Jan 13]. Available from: http://arxiv.org/abs/1412.3555 Dibeklio˘glu H, Hammal Z, Cohn JF (2018) Dynamic multimodal measurement of depression severity using deep autoencoding. IEEE J Biomed Health Inform 22(2):525–536 Ding X-R, Zhang Y-T, Liu J, Dai W-X, Tsang HK (2016) Continuous cuffless blood pressure estimation using pulse transit time and photoplethysmogram intensity ratio. IEEE Trans Biomed Eng 63(5):964–972 Dubey D, Tomar GS (2021) Image alignment in pose variations of human faces by using corner detection method and its application for PIFR system. Wireless Pers Commun [Internet]. 2021 [cited 2022 Feb 27]; Available from: https://doi.org/10.1007/s11277-021-09330-1 Gratch J, Artstein R, Lucas G, Stratou G, Scherer S, Nazarian A et al. (2014) The distress analysis interview corpus of human and computer interviews. In: Proceedings of the Ninth international conference on language resources and evaluation (LREC’14) [Internet]. Reykjavik, European Language Resources Association (ELRA), Iceland [cited 2022 Jan 13], pp 3123–3128. Available from: http://www.lrec-conf.org/proceedings/lrec2014/pdf/508_Paper.pdf Graves A (2012) Long short-term memory. In: Graves A (eds) Supervised sequence labelling with recurrent neural networks [Internet]. Springer, Berlin, [cited 2022 Jan 13], pp 37–45. (Studies in Computational Intelligence). Available from: https://doi.org/10.1007/978-3-642-24797-2_4 Kachuee M, Kiani MM, Mohammadzade H, Shabany M (2017) Cuffless blood pressure estimation algorithms for continuous health-care monitoring. IEEE Trans Biomed Eng 64(4):859–869

References

123

Kroenke K, Strine TW, Spitzer RL, Williams JBW, Berry JT, Mokdad AH (2009) The PHQ-8 as a measure of current depression in the general population. J Affect Disord 114(1–3):163–173 Maeda Y, Sekine M, Tamura T, Moriya A, Suzuki T, Kameyama K. Comparison of reflected green light and infrared photoplethysmography. In: 2008 30th annual international conference of the IEEE engineering in medicine and biology society. pp 2270–2272 Mello RGT, Oliveira LF, Nadal J (2007) Digital Butterworth filter for subtracting noise from low magnitude surface electromyogram. Comput Methods Programs Biomed 87(1):28–35 Murphy-Chutorian E, Trivedi MM (2009) Head pose estimation in computer vision: a survey. IEEE Trans Pattern Anal Mach Intell 31(4):607–626 Ruhé HG, van Rooijen G, Spijker J, Peeters FPML, Schene AH (2012) Staging methods for treatment resistant depression. A Syst Rev J Affect Disord 137(1):35–45 Sackeim HA (2001) The definition and meaning of treatment-resistant depression. J Clin Psychiatry 62:10–17 Wang P, Xu G, Cheng Y, Yu Q (2018) A simple, robust and fast method for the perspective-n-point problem. Pattern Recogn Lett 108:31–37

Chapter 7

OHC Physician Personalized Recommendation

Online Healthcare Community (OHC) often represents the starting point of individuals’ awareness of the cycle of care, where potential patients seek healthcare advice and physician/hospital recommendations. As stated above, OHC could appear in at least three major forms: social media-based OHC, independent organization-based OHC, offline hospital-based OHC (Guo et al. 2017). Each offers healthcare advice with a different degree of trustworthiness and helpfulness, connecting patients with offline medical resources. One of the biggest differences between smart healthcare and the traditional healthcare industry is that the start line and a great portion of the healthcare process are done online, leading to great convenience and efficiency for both the patients and healthcare agencies. Compared with the face-to-face consultation in the traditional medical model, when patients seek a doctor’s help on an online platform, he/she can usually learn about the doctor’s historical consultation experience and online evaluation through the information displayed by the doctor or the platform, and indirectly understand the doctor’s online consultation history, and then make a choice through the perceived quality of his/her service. However, due to the lack of relevant medical expertise of patients, when faced with massive amounts of information in OHC, patients perceive greater uncertainty, which makes them often in a passive state with weak information. At the same time, with the increase in the number of doctors on the online consultation platform, even if the platform’s expert registration mechanism can guarantee the qualifications of doctors, there is still great uncertainty in the quality of doctors’ answers. Therefore, providing patients with useful resources is critical to realizing the vision of participatory medicine. When recommending a suitable doctor for a patient, the key indicator of the doctor’s response quality needs to be considered, so as to improve the quality of online consultation services and patient satisfaction. In view of the above shortcomings, this chapter mainly aims to solve two problems: one is to consider the “passive” problem of patients during online consultation; the other is to consider the control of the quality of doctors’ answers. A hybrid doctor

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_7

125

126

7 OHC Physician Personalized Recommendation

Fig. 7.1 The structure of this chapter and its position in the cycle of care

recommendation model with three sub-models is designed and constructed, that is, doctor recommendations based on similar patients, extended doctor recommendations based on similar doctors, and doctor recommendations fused with answer quality. Each of the three different sub-models provides reasonable solutions so as to quickly and accurately recommend high-quality doctors to patients. When looking for a suitable doctor for a patient, this method firstly looks for a doctor who matches their health needs from the perspective of the patient, then expands the range of suitable doctors from the perspective of the doctor, and finally considers the key indicator of the quality of the doctor’s answer to control service quality. To a certain extent, this method can solve the problem of patient consultation quickly and with high quality, improve the patient’s medical experience, the quality of medical treatment, and the quality of medical services in general. Figure 7.1 shows the structure of this chapter.

7.1 OHC Physician Recommendation Framework This section mainly discusses the design of the proposed physician recommendation model, including three recommendation strategies: recommendation based on similar patients, enhanced recommendation based on similar physicians, and mixed recommendation with integrated answer quality evaluation, as shown in Fig. 7.2. In short, when recommending physicians based on similar patients, the contentbased recommendation idea is adopted, the word2vec model is used to train the patient consultation text, and its vector representation is obtained. On this basis, the cosine similarity between patients is calculated, and the top K is sorted in descending order to obtain a physician recommendation set based on similar patients A. For physician recommendation based on physician similarities, the recommendation idea based on collaborative filtering is adopted, and data is accumulated according to the physicians’ expertise and experience to calculate the field overlap between set A

7.1 OHC Physician Recommendation Framework

127

Fig. 7.2 Physician recommendation model framework

and the remaining physicians, so as to expand and form doctor recommendations to obtain a physician set B. For the mixed physician recommendation with answer quality evaluation, the idea of mixed recommendation is adopted, the answer quality factor of doctor set B is calculated according to the doctor’s historical answer quality evaluation results, and the doctor scores and answer quality factors of the above two recommendations are linearly weighted. The final scores of doctors in set B are obtained, which are sorted into descending order and within which the top K physicians are selected, determine the final set of doctor recommendations, and return them to the target patient.

7.1.1 Physician Recommendation Based on Similar Patients According to the theory of conformity and trust, when patients choose doctors, they are more inclined to choose doctors recommended by similar patients, that is, patients with similar diseases who share the experience or evaluate patients are often more trusted and accepted by potential patients. Before physician recommendation and consultation, patients need to fill out their own condition either through a form or through natural language. The more similar the condition of the patients, the more similar the description of the condition, and the greater the possibility of mutual reference between the two patients. Therefore, the similarity of patient consultation texts could be utilized to calculate and find similar patients and their choices of physicians. Due to the severe feature sparsity and high dimensionality of patient consultation texts, we adopted the Word2Vec model to construct a text feature vector representation model based on word vectors. Word2Vec is essentially an algorithm based on word embedding, which learns semantic information in an unsupervised manner, that is, through an embedding space, the more semantically similar words are closer in the vector space. The model makes full use of the context information of words for training, converts words into fixed low-dimensional real vectors, and solves the

128

7 OHC Physician Personalized Recommendation

problem of high dimensionality. It has been widely used in NLP fields such as text clustering and synonym search, which has achieved good semantic representation effects. The training corpus first went through jieba segmentation, stop word removal, and part-of-speech filtering. Then use the word2vec model to train the word vector model, and calculate the word vector of the keywords of the test consultation text in the trained word2vec model, and the accumulated results are synthesized into sentence vectors. Each text sequence of a patient consultation text could be represented with f i = {w1 , w2 , . . . , wk }, within which k is the number of keywords. The word vector for each keyword Wi could be represented with Vi = {Vi1 , Vi2 , . . . , Vim }, within which m is the dimensionality of the word vector. After the word vector of each keyword is accumulated, the feature vector of each consultation text is obtained: di =



Vi

(7.1)

After which cosinesimilarity  is used to calculate the similarity between patient consultation texts sim di , d j , which is the similarity between patient i and j. The similarity score is then used to calculate the score of similarity for physicians:   Scor e p = sim(i, j) = sim di , d j =

di ∗ d j   di  ∗ d j 

(7.2)

The score is then ranked to find the most similar patients of the current patient, with which a set of physician A could be calculated.

7.1.2 Enhanced Recommendation Based on Similar Physicians When choosing a doctor, patients often not only consider similar patients, the type of disease and diagnosis and treatment experience that physicians have expertise at also affect the choice of patients. Physician recommendation based on similar patients mainly determines the initial candidate according to the similarity of the patient’s condition. However, this recommendation strategy can only find doctors who have been consulted by patients with the same condition as the target patient, and the probability of finding similar patients is correspondingly reduced, resulting in an unsatisfactory result. If implemented in a real-life scenario, it could lead to the situation in which only a few physicians with a high amount of platform history get recommended, while others remain unnoticed and under-utilized. Considering the similar professional knowledge and clinical experience among physicians, there is a certain similarity in diagnosis and treatment ability. Therefore, we introduce a second doctor recommendation strategy based on physician similarities, which main purpose is to discover physicians with similar knowledge

7.1 OHC Physician Recommendation Framework

129

structures. Specifically, we start from the physician’s professional knowledge and empirical knowledge and calculate the similarity of the professional knowledge and the empirical knowledge of the physicians in the recommendation set A, respectively. We further sum up the two to obtain the physician’s knowledge similarity to determine the extended doctor recommendation set. This could help the cooperation and competition among physicians with similar backgrounds, and is also beneficial for users to find more suitable options. Within the same department, all physicians should have the ability to diagnose and treat diseases belonging to the department, but there are differences in the diagnosis and treatment capabilities. Theoretically, there is a many-to-many relationship between physicians and diseases. A physician can be proficient at multiple diseases, and each disease can also be treatable with multiple physicians. Therefore, it can be considered that the knowledge structure between each physician is at least to some degree, different. At present, physicians on most OHCs will publish the descriptions of the diseases they are good at diagnosis and treatment on their homepages, and their expertise will explain in detail the types of diseases and the scope of diagnosis and treatment they are good at. The more similar the descriptions, the more professional similarities the two doctors have. This could be utilized to calculate the professional proficiency similarities between physicians. First, pre-processing steps such as jieba segmentation and stop word removal are done, and the processed physician description is vectorized through one-hot encoding. Cosine similarity is utilized to calculate the similarities Sim g between the subject physicians with the pre-defined recommendation set A. Furthermore, the “post-diagnosis evaluation” section on the doctor’s personal homepage counts the cases of diseases that have been consulted and summarizes the doctor’s diagnosis and treatment experience comprehensively. Therefore, the name of the disease in the clinical experience accumulated by the doctor is represented as the doctor’s experience feature. The similarities Sim e between two physicians i and j could be calculated with Jaccard method, which could be shown as the following formula:      Si ∩ S j    Si ∩ S j   =     Sim e (i, j) = J Si , S j =  (7.3)  Si ∪ S j  |Si | +  S j  −  Si ∩ S j    J Si , S j represent the Jaccard similarity coefficient, which is a representation of the percentage of intersection elements within the combined union elements of the two physicians. The similarity of doctors’ professional knowledge and experience knowledge could then be combined to express the similarity between doctors by calculating the similarity of doctors’ knowledge, so as to determine the extended doctor recommendation set. In order to facilitate the calculation of the final physician recommendation based on similarities, and to ensure the fairness of each index during the integration process, the similarity of physicians’ expertise knowledge and the similarity of experience knowledge is normalized using Min–Max, as shown below:

130

7 OHC Physician Personalized Recommendation

Sim i_n =

Sim i − min{Sim i } i = g, e max{Sim i } − min{Sim i }

(7.4)

The final physician similarity Sim d could then be calculated by simply adding the two different dimensions of similarities. Based on recommendation A calculated using patient similarity, the final score for each physician could then be represented as the following: Scor ed =

m 1  Sim d (i, j) i = 1, 2, . . . , k m i=1

j = 1, 2, . . . , n,

(7.5)

where m is the number of physicians within the recommendation set A, and n is the number of physicians of the rest on the platform. By setting a threshold value for Scor ed , we could retrieve an extended physician recommendation set, which could be combined with recommendation set A to retrieve the final set B. For the ease of further calculation, the Scor e p from set A and Scor ed from set B are both represented with Scor ex and treated the same.

7.1.3 Answer Quality Empowered Physician Recommendation When a patient chooses a doctor on the online consultation platform, he/she will first browse the basic information of the doctor according to his/her own health status, and after evaluating the various options of the doctor, form his/her own judgment on the overall level of the doctor, and decide whether to choose to purchase the doctor’s online service. In the current research on patients’ medical choice behavior, researchers use electronic word-of-mouth information such as system-generated word-of-mouth, patient-generated word-of-mouth, patient voting, and gift-giving to judge the service quality of doctors. Few studies have used the quality of doctors’ responses as the basis for evaluating the quality of doctors’ services. Contrary to the current state-of-the-art research, when patients receive online medical services, they often pay more attention to whether the doctor’s answer can meet their own health needs. Therefore, it is very important to consider the quality of the doctor’s answer when recommending a doctor to a patient. Our previous research involved a physician answer quality automatic analysis method, which could be utilized to obtain the historical answer quality of physicians in recommendation set B. Assume that a physician with an index number of t has x related historical answer, within which, m of them are determined to be high quality, thus (x − m) of low quality, then the answer quality score of the physician would be: Scor e Q =

m x

(7.6)

7.2 Case Study

131

The other scores from the previous steps are then retrieved to be weighed against each other, resulting in the final recommendation score of a physician: Scor e = αScor ex + β Scor e Q ,

(7.7)

where α and β are two weight factors between (0,1), and α + β = 1. The final score is sorted with value descending, which results in the final recommendation ranking set.

7.2 Case Study 7.2.1 Data We retrieved a total of 15,000 doctor-patient question-and-answer texts and their corresponding doctor homepage information from November 4, 2019, to January 2, 2021, from the “diabetes” disease module of the Chinese OHC platform Haodaifu through web crawling. We removed the cases where the physician’s reply or the patient question is less than 15 characters. After further removing 7 unreachable hyperlinks, there remain 13,412 data points to train the word2vec model. The final selectable physicians were further shrunk down by eliminating physicians with less than 10 previous answers, which ends up being 8974 patient-physician interactions from 232 physicians. We randomly selected 30 existing patient records as the target experiment subjects, with the remaining 8944 patients as the “similar patient” base. Some of the data is shown in Table 7.1 and Fig. 7.3. In the discussion of the recommendation strategy model in this chapter, in order to visually display the recommendation process without loss of generality, we select patient No. 903 as the target patient for empirical demonstration. The target number of the physician at that time was 30, and the consultation text information could be translated as: “Insulin resistance, fasting blood sugar is 5.7, half an hour after a meal 9.9, an hour 9.5, 9.0 after two hours. Insulin release, fasting 51.80, half an hour after a meal 292.05, an hour 300, 297.68 after two hours. I also have severe fatty liver, alanine aminotransferase 126, please take a look if you have time.” All experiments in this chapter are done on a computer with Intel(R) Core(TM) i7-9750HCPU @ 2.60 GHz, with software such as Pycharm and Anaconda.

7.2.2 Results According to the symptom or disease information input by the user, the word2vec model is used to vectorize the consultation text of the target patient and other patients, calculate the similarity between the target patient and other patients, and calculate the

132

7 OHC Physician Personalized Recommendation

Table 7.1 Examples of physician and his/her patient answer history Index Name

Accumulated experience history

0

杨建梅 Diabetes (313), Hyperthyroidism (88), Hypothyroidism (41), Thyroid neoplasm (26), Thyroid disease (21), Thyroiditis (15), Adrenal Disorders (13), Hypertension (12), Thyroid Carcinoma (7), Endocrine disease (5), Primary Aldosteronism (3), Adrenal Incidentaloma (2), Pheochromocytoma (2), Cushing Syndrome (1), Hypoparathyroidism (1), 3-Hyper (1), Diabetic Foot (1), Hyperparathyrodism (1), Gout (1)

1

郭夏

2

刘安雷 Common Cold (5), Coughing (4), Gastritis (4), Hypertension (3), Pneumonia (2), Arrhythmia (2), Fever (2), Pulmonary Nodule (1), Stomach Problem (1), Upper respiratory tract infection (1), Hepatitis C (1), Tonsillitis (1), Spondylosis (1), Stomachache (1), Stomach Cancer (1), Interstitial Lung Disease (1), Lung Disease (1), Diarrhea (1), Renal Failure (1), Pleural Effusion (1), Lung Abscess (1), Anemia (1), Fatty Liver (1), Lung Cancer (1)

3

苏娜

4

崔利军 Diabetes (6), Hyperthyroidism (2), Gout (1), Thyroid neoplasm (1)

5

王志宏 Diabetes (60), Hyperthyroidism (57), Thyroid neoplasm (34), Hypothyroidism (21), Thyroid disease (10), Gout (9), Thyroiditis (7), Endocrine disease (7), Adrenal Disorders (5), Obesity (3), Precocious Puberty (3), Water and Electrolyte Disturbances and Acid–base Balance (2), Hyperinsulinemia (2), Diabetes induced Kidney Disease (1), Low Blood Sugar (1), Congenital Adrenal Hyperplasia (1), Hyperprolactinemia (1), Osteoporosis (1), Hyperlipidemia (1), Hypertension (1), Insomnia (1)

6

刘向阳 Diabetes (7), Obesity (5), Malnutrition (1), Low Blood Sugar (1)

7

唐子惠 Diabetes (313), Hyperthyroidism (88), Hypothyroidism (41), Thyroid neoplasm (26), Thyroid disease (21), Thyroiditis (15), Adrenal Disorders (13), Hypertension (12), Thyroid Carcinoma (7), Endocrine disease (5), Primary Aldosteronism (3), Adrenal Incidentaloma (2), Pheochromocytoma (2), Cushing Syndrome (1), Hypoparathyroidism (1), 3-Hyper (1), Diabetic foot (1), Hyperparathyrodism (1), Gout (1)

…

…

231

鲜玉军 Cerebral Infarction (3), Hypertension (1), Common Cold (1)

Diabetes (5), Thyroid disease (2), Hyperthyroidism (2), Thyroiditis (2), Hypothyroidism (2), Thyroid neoplasm (1)

Coronary Heart Disease (10), Arrhythmia (3), 3-Hyper (1), Atrial Fibrillation (1)

…

probability value of the doctor being recommended, so as to determine the candidate doctor recommendation set A. Considering the small scale of the training corpus, in order to achieve better results, the vector_size is set to 100 when training the word2vec model, and the obtained consultation text vector is shown in Table 7.2. In order to ensure that the most suitable high-quality doctors can be obtained, the threshold is set at μ = 0.7 to ensure that the corresponding number, but not too many matching doctors can be returned under the condition of high similarity. Since these similar patients all went to a physician for consultation, we calculated the mean of the similarity of similar patients answered by each physician, i.e. the recommending physician’s score based on patient similarities Scor e p .

7.2 Case Study

133

Fig. 7.3 Example of physician–patient interactions on Haodaifu platform

Table 7.2 Patient consultation text vector Patient

Vector 1

2

3

4

5

…

0

−0.03154

−0.07151

0.00808

−0.01487

−0.06481

…

100 0.00017

1

0.01734

0.00126

0.00383

−0.01336

0.00485

…

−0.00264

2

0.81000

0.26000

0.33000

0.79000

0.32000

…

0.27000

3

0.02228

−0.02715

0.00200

0.05200

0.01175

…

−0.05245

4

−0.01302

0.03017

−0.01096

0.01118

−0.01178

…

−0.02090

5

−0.01172

0.03775

−0.02337

0.02498

0.00986

…

0.00493

…

…

…

…

…

…

…

…

8973

−0.01791

0.00772

−0.01696

0.06554

0.05658

…

−0.00477

As shown in Table 7.3, according to the descending order of Scor e p , the similar patient set A of patient No. 903 is {15, 167, 193, 189, 52, 30, 9, 123, 184, 85, 117, 198, 148, 16}. For this recommendation strategy, it is set that the top K doctors in the candidate doctor set A are highly similar doctors and are considered to be successfully returned. When K = 10, the recommended doctor set is {15, 167, 193, 189, 52, 30, 9, 123, 184, 85}. In this case, the actual physician No. 30 the patient consulted is within the recommendation set A. For physician similarities, we calculated the similarity of doctor’s expert knowledge and experience knowledge respectively, which is shown in Tables 7.4 and 7.5. The overall physician similarities are calculated through the sum of these

134

7 OHC Physician Personalized Recommendation

Table 7.3 Similar patient and consultation information for Patient No. 903 Doctor Patient Patient question

Similarity Scorep

rank

15

8430

Two weeks after taking the drug, the blood sugar 0.822882 0.822882 1 was normal, 5.5 on an empty stomach and 9.9 two hours after a meal. diabetes

167

4891

Fasting blood sugar in the morning was 7.7, 0.800353 0.800353 2 postprandial blood sugar was 10.5 in 1 h, and 9.7 in 2 h. diabetes

193

5268

Fasting blood glucose 9–10 in the past year, 2-h postprandial blood glucose 9–11, diabetes

0.751398 0.746398 3

4416

No family history, blood sugar has been high 14 recently, and blood sugar was occasionally measured before, but it did not exceed 8. Now blood sugar is 14, blood pressure is 140, 80

0.741398

189

4916

On May 6, the fasting blood sugar was 8.93, and 0.739751 0.739751 4 after drinking glucose for two hours, it was 20.07. On May 13, the fasting blood sugar was 8.04, and two hours after the meal, it was 11.35. diabetes

52

1716

1 h after a meal 6.5 2 h after a meal 7.0 3 h after a 0.764263 0.736659 5 meal 7.5 Blood sugar is often the highest fasting blood sugar 5–5.5 in three hours. Blood sugar does not return to fasting level in four or five hours

6645

At present, ten units of insulin glargine injection, 0.709056 one tablet of sitagliptin phosphate (100 mg) before meals, two tablets of metformin (0.5 g of Gehuazhi) each time with three meals, fasting blood sugar 4–5, two hours after meal blood sugar 11.5–13.2 Fasting blood sugar 4–5, two-hour postprandial blood sugar 11.5–13.2 Two-hour glucose 15.5. diabetes

30

6915

At present, the fasting blood sugar in the morning 0.736338 0.736338 6 of taking the medicine is controlled below 6, but the blood sugar is still high two hours after the meal, which has been around 17. Diabetes, fasting blood sugar 17.9

…

…

…

16

4151

In March, fasting blood sugar was 4.9 and 7.1 in 0.702198 0.702198 14 2 h after meals. Last year, the uric acid test was 490. Three days ago, I bought a blood glucose meter for the elderly and accidentally tested it for myself…

…

…

…

7.2 Case Study

135

Table 7.4 Expertise similarities for recommendation set A for No. 903 Extend Index 15

167

193

189

52

30

9

123

184

85

117

198

148

16

60

0.81 0.26 0.33 0.79 0.32 0.61 0.30 0.60 0.17 0.23 0.26 0.17 0.27 0.79

99

0.79 0.32 0.40 0.58 0.23 0.50 0.29 0.44 0.21 0.22 0.25 0.21 0.33 0.58

143

0.81 0.26 0.33 0.79 0.32 0.61 0.30 0.60 0.17 0.23 0.26 0.17 0.27 0.79

173

0.80 0.29 0.36 0.68 0.28 0.59 0.26 0.52 0.19 0.20 0.23 0.19 0.30 0.68

Table 7.5 Experience similarities for recommendation set A for No. 903 Extend Index 15

167

193

189

52

30

9

123

184

85

117

198

148

16

60

0.37 0.28 0.34 0.58 0.38 0.30 0.26 0.20 0.26 0.19 0.38 0.12 0.16 0.48

99

0.52 0.27 0.32 0.68 0.31 0.38 0.25 0.39 0.28 0.18 0.37 0.12 0.15 0.53

143

0.29 0.22 0.36 0.58 0.40 0.44 0.20 0.46 0.18 0.17 0.36 0.17 0.14 0.55

173

0.54 0.32 0.31 0.53 0.42 0.52 0.29 0.25 0.29 0.24 0.4

0.18 0.19 0.51

two knowledge similarities, and the threshold is set at 0.7 again for the extended recommendation physician set B. After calculation, it is found that based on the doctor recommendation set A, only four doctors numbered 60, 99, 143, and 173 have recommendation scores greater than 0.7 for No.903, as shown in Table 7.6. At this time, the doctor recommendation set B of the target patient is {15, 167, 193, 189, 52, 30, 9, 123, 184, 85, 117, 198, 148, 16, 60, 99, 143, 173}. According to descending order, when K = 10, its recommended doctor set is {15, 167, 193, 189, 173, 52, 30, 9, 143, 123}. Finally, taking the extended doctor recommendation set B as input, the quality of the doctor’s historical answers are evaluated, and the doctor’s corresponding answer quality factor is obtained as the doctor’s recommendation score in terms of answer quality, which is shown in Table 7.7. The doctor-recommended set B ranked by quality is {143, 198, 60, 52, 30, 9, 184, 15, 123, 85, 193, 167, 117, 99, 189, 16, 148, 173}. When K = 10, its recommended doctor set is {143, 198, 60, 52, 30, 9, 184, 15, 123, 85}. In order to judge the feasibility and rationality of the hybrid recommendation model proposed in this chapter, and to further measure the quality of the recommendation results, we further conducted model tests on 30 patients. The purpose is to judge whether the doctors who successfully returned in the doctor recommendation set of the 30 target patients include the doctor who visited the target patient at that time. The feasibility and effectiveness of the recommendation model proposed in this chapter can be better judged if the doctor who visited the target patient at that time is included. Figure 7.4 shows the overall accuracy regarding the different values of α.

1.18

1.30

1.10

1.33

99

143

173

15

Index

60

Extend

0.59

0.46

0.57

0.52

167

0.66

0.68

0.70

0.66

193

1.21

1.36

1.25

1.36

189

0.68

0.71

0.52

0.68

52

1.10

1.04

0.87

0.90

30

0.53

0.48

0.52

0.54

9

Table 7.6 Overall physician similarities for recommendation set A for No. 903

0.76

1.05

0.81

0.79

123

0.46

0.32

0.47

0.40

184

0.42

0.38

0.38

0.40

85

0.61

0.60

0.60

0.62

117

0.35

0.31

0.31

0.26

198

0.47

0.39

0.47

0.41

148

1.19

1.33

1.10

1.26

16

0.74

0.73

0.71

0.71

Scored

136 7 OHC Physician Personalized Recommendation

7.2 Case Study Table 7.7 Overall recommendation score of physician answer quality for No. 903

137 Physician no

ScoreQ

Rank

143

1.0000

1

198

1.0000

2

60

0.9697

3

52

0.9677

4

30

0.9645

5

9

0.9643

6

184

0.9412

7

15

0.9205

8

123

0.9200

9

85

0.8750

10

193

0.8649

11

167

0.8333

12

117

0.8235

13

99

0.8000

14

189

0.7895

15

16

0.6406

16

148

0.5385

17

173

0.4737

18

Fig. 7.4 Variation of accuracy and returned physician number with different values of α

It can be noticed from the figure that the doctor’s recommendation accuracy changes continuously with the change of α, and shows a trend of first increasing and then decreasing. When α = 0.6, the doctor’s recommendation accuracy rate reached the highest 86.67%, that is, 26 of the 30 patients successfully returned to the doctors

138

7 OHC Physician Personalized Recommendation

who had been seen by the patients at that time. The mixed recommendation strategy achieved good recommendation results. Therefore, it is considered that when α = 0.6 and β = 0.4, the mixed recommendation effect of personalized doctors is the best. At the same time, it can also be concluded that for the recommendation of health consulting services, the first two recommendation strategies are relatively more important, and the quality of the answer is also indispensable. To sum up, the final recommendation result of this chapter not only considers whether the recommended doctor has diagnosed and treated patients with similar diseases and the knowledge structure of the doctor but also considers the response quality score that measures the quality of the doctor’s service. The final recommendation results more comprehensively reflect the ability of physicians. Therefore, it can be considered that the hybrid recommendation model of the combination of doctor recommendation strategies proposed in this chapter is ideal, with a high accuracy rate, and can help patients recommend high-quality doctor resources.

7.3 Risk Analysis and Conclusion In order to quickly discover high-quality and suitable physician resources, we proposed a hybrid physician recommendation model, which includes physician recommendations based on similar patients, extended physician recommendations based on expertise and experience similarities, and physician recommendations fused with answer quality evaluation. The above recommendation strategy uses the data from the “Haodaifu Online” platform to carry out experiments, which verifies the operability of the model and the validity of the recommendation results. This method can greatly promote the rational distribution and use of medical resources, relieve the pressure of medical treatment to a certain extent, and improve the quality of medical treatment, which has important practical value and social significance. While it is usually interpreted as a minor problem, physician incompetence is still sometimes widely reported throughout the news and is, apparently, “not an isolated case” (Keegan et al. 2021). Furthermore, even if physician incompetence would only cause minimal harm, pseudoscience and health frauds still largely exist within all developed countries. For developing countries, physician errors contribute to at least one-third of all adverse effects within hospitals (Wilson et al. 2012). While procedures and training to ensure physician competency is required, recommendation systems could become an alternative solution to help patients find competent physicians. Furthermore, the navigation provided by such recommendation systems could alleviate the pressure and loss of efficiency due to the traditional physician referral system. However, by pushing patients towards the more competent and experienced physicians without increasing the number and quality of physician service as a whole, we suffer the risk of further overloading these physicians who are already in short supply. Recommendation systems could also, by design, discriminate against inexperienced physicians, even though their current competencies do not signify the future potential. Discrimination could also occur if the answering quality of physicians is

References

139

not on par with other physicians due to time constraints or even a lesser grasp of the knowledge used if in a multi-lingual setting, even if the actual competency of the physician could not be determined simply with online answer quality. This could be extremely problematic, especially in the cases where the actual treatments do not happen online but offline, such as surgery, physical examinations, etc. In reality, recommendation systems are often tainted with even more issues due to economic incentives. Low-quality physician answers and experiences could be hidden to improve platform reputation. Recommendation algorithms could be distorted to favor physicians who have economically contributed to the platforms. Unlike all other scenarios presented in this book, the subject of this chapter is independently operated OHC, which is far more sophisticated than hospital and communitycontrolled smart healthcare components. Thus, it is unlikely that the risk within the system could be fully rationalized and quantified. However, it is undeniable that further research should focus on how the incompetent physician risk could be further avoided through a better recommendation system.

References Guo S, Guo X, Fang Y, Vogel D (2017) How Doctors gain social and economic returns in online health-care communities: a professional capital perspective. J Manag Inf Syst 34(2):487–519 Keegan W, Tessier W, Story J (2021) Where does it begin and how to stop it: opportunities to prevent “bad” physicians. Mo Med 118(3):206–210 Wilson RM, Michel P, Olsen S, Gibberd RW, Vincent C, El-Assady R et al (2012) Patient safety in developing countries: retrospective estimation of scale and nature of harm to patients in hospital. BMJ 13(344):e832

Chapter 8

Data-Driven Cancer Screening and Risk Analytics

Disease diagnosis represents the central phase of the cycle of care within any healthcare system and one of the most important phases for physician involvement. Traditional diagnosis, prior to the invention of modern diagnosis techniques, involves a qualitative approach of symptom observation and patient-physician conversation. Physicians then refer to previous similar cases, either from personal experiences or medical textbooks, to specify treatment processes. With the invention of more scientific diagnosis processes and examination methods, quantitative diagnoses, such as urine sugar tests, full blood count, ultrasonography, and endoscopic examination, begin to be put into application. The current healthcare system relies heavily on these quantitative methods, which has gained the reputation of high accuracy but also lack of humanitarian care. With the increasing load on physicians and the presence of online misinformation, the bond between patients and physicians through conversation becomes more and more unlikely, which creates distrust between patients and physicians. Next-generation information technologies could be utilized to extend the ability of physicians during the process of diagnosis, but could also provide extra challenges and induce new risks. Machine learning and deep learning-based diagnosis decisionsupporting tools could in themselves perform better than the majority of physicians. While providing accuracy that no physician could achieve, it also fails on tasks no physician would fail on, raising security and responsibility concerns. Physicians could become confused by the decision-making processes of such systems, while others might become too reliant on them, leading to human–machine conflicts and an overall degradation in physician competency. It is thus crucial for smart healthcare to develop interpretable, risk-controlled, and accurate machine diagnosis systems which let physicians remain the central force of diagnosis while allowing patients to understand the decision-making processes of both the algorithm and the physician. The purpose of this chapter is to provide an example of such a diagnosis supporting system, an attention-mechanism combined UGI cancer support diagnosis method. Currently, UGI cancer support diagnosis focuses on the accuracy of a two-way classification problem: whether there is cancer or not. However, UGI cancer symptoms © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_8

141

142

8 Data-Driven Cancer Screening and Risk Analytics

Fig. 8.1 The structure of this chapter and its position in the cycle of care

are much more complicated than a yes/no question and could be induced by visible symptoms but are not considered to be cancerous yet. Furthermore, the diagnosis supporting systems lacks interpretability overall, which could be alleviated through the utilization of attention mechanisms. The structure of this chapter is shown in Fig. 8.1.

8.1 Data-Driven Cancer Screening and Diagnosis A leading cause of medical cost and mortality, cancer has been at the center of medical research throughout the last few decades. While the obvious focal point is the cure of cancer, early diagnosis has proven to be the best way to avoid incurring treatment costs and death of patients (Wardle et al. 2015). While contributing to more than a million cases of cancer each year, early diagnosis of UGI cancer has a 5-year survival rate of up to 80%, which suggests the importance of frequent and common diagnosis programs such as colonoscopy procedures (Menon and Trudgill 2014). In clinical practice, early diagnosis of UGI cancer is usually made through Esophagogastroduodenoscopy procedure, or Gastroscopy is short. While it is often assumed that early cancer diagnosis is fool-proof and accurate, diagnosis of cancer with gastroscopy procedures requires experience and expertise in the diagnosis procedures. Misdiagnosis is rampant at about 11.3%, while expert and novice physicians could have a diagnosis accuracy difference of more than 20% (Menon and Trudgill 2014; Luo et al. 2019). The situations are exacerbated by shortages of medical resources, as the population of developing countries often are provided with no early cancer screening or poor-quality cancer screening. The need for fast, cheap, and accurate cancer screening boosted the development of computer-based cancer diagnosis support and automatic cancer diagnosis systems.

8.2 Cancer Screening Framework Fused with Attention Mechanism

143

There exists a body of literature specifically regarding Upper Gastric-Intestinal (UGI) cancer automatic diagnosis, early/late-stage categorization, and treatment plan recommendations (Chidambaram et al. 2021). However, many insufficiencies still exist within the current state-of-the-art systems. Specifically, gastroscopy reports are often difficult to process due to their unstructured nature and are often ignored. The current state-of-the-art UGI cancer diagnosis support systems could not automatically chunk and recognize the meanings of gastroscopy reports, leading to a disconnect between human recognized features and UGI cancer, limiting the accuracy of cancer diagnoses. Moreover, current literature mostly focuses on “cancer” itself, while cancer-related and early cancer signaling features are often ignored. For example, benign tumors and mucosal lesions are often categorized as “not cancer,” which is technically true but still signals high possibilities of developing into cancer. Finally, UGI cancer diagnosis supporting systems currently have low interpretability. The utilization of deep learning in cancer diagnosis, while substantially increasing computer diagnosis accuracy, is often difficult to understand. Such processes are often described as “black boxes,” as physicians find it difficult to understand the diagnosis process, while patients are more likely to disbelieve such a diagnosis. This chapter proposes an attention-mechanism combined UGI cancer support diagnosis method based on the insufficiencies stated above. The method merged clinical medical data and physician clinical experiences and mimicked physician clinical diagnosis processes to support UGI cancer diagnosis. We first pre-process text-based gastroscopy reports, including report feature wording database and report text lexical separation. Then, GRU-attention and CNN-attention are utilized to extract key feature information from pre-processed gastroscopy reports (Zhang et al. 2019; Yin et al. 2016). The two attention level features are also merged to create more accurate diagnosis results. By introducing attention-mechanism, keywords in gastroscopy reports that support the diagnosis of UGI cancer are highlighted, which undergoes attention mechanism-based visualization, to show the correlation level between gastroscopy report texts and cancer diagnosis. This method is similar to how physicians diagnose cancer clinically while highlighting important information and features for diagnosing UGI cancer, which provides higher interpretability for both patients and physicians. Furthermore, the method could also categorize patients into three categories: healthy, cancer patients, and pre-cancer/ possible patients. Such is to increase the accuracy of early cancer and pre-cancer symptoms diagnosis to patients and physicians to manage cancer-like symptoms as soon as possible.

8.2 Cancer Screening Framework Fused with Attention Mechanism In clinical practice, physicians mainly rely on the gastroscopy text report based on esophagogastroduodenoscopy examination video/image data to make UGI cancer

144

8 Data-Driven Cancer Screening and Risk Analytics

Fig. 8.2 The framework of UGI cancer diagnosis support system with attention-mechanism merged

diagnosis decisions. Key features and information are extracted through the comprehensive analysis of these data and information, combined with years of expertise and clinical experience. The framework of upper gastrointestinal cancer diagnosis support proposed in this chapter includes four layers: text-vector transformation layer, feature attention extraction layer, attention-merging weighting layer, and text-based categorization layer, shown in Fig. 8.2. The four different layers could be described as the following: 1.

2.

3.

Text-vector Transformation Layer: The gastroscopy report and diagnosis result data are combined to create a sample dataset. Then, a custom dictionary is constructed based on the knowledge and experiences of gastroscopy physicians. Gastroscopy reports are then separated into tokens through Natural Language Processing (NLP) techniques. Finally, these tokens are converted into vectors through word embedding techniques. Feature Attention Extraction Layer: due to the apparent superiority of Gate Recurrent Unit (GRU) and Convolutional Neural Network (CNN) in the field of text feature extraction, we utilized GRU and CNN to extract semantic features within the processed gastroscopy report text sequences. In order to further extract and highlight key features of gastroscopy reports, we introduced the GRU-attention and CNN-attention proposed by Zhang et al. (2019) and Yin et al. (2016) separately to capture the overall and regional semantic features. Attention Fusion/Weighting Layer: We first create a fusion of the GRU-attention and CNN-attention, as the merged attention of feature level. The merged feature

8.3 Intelligent UGI Cancer Screening

4.

145

level attention is then weighted with the CNN hidden layer vector, outputted as the final text semantics vector. Text-based Categorization Layer: The text semantics are processed and normalized through the Softmax function, and a 3-way categorization of healthy, possible cancer, and cancer is produced.

Other than the above procedures, to improve our method’s interpretability, we introduced a visualization technique, which could provide the relation between input data and output results. For the input gastroscopy report, attention weight is visualized to highlight the most important decision factors (words, phrases, short sentences) of the final result to achieve relatively high interpretability.

8.3 Intelligent UGI Cancer Screening This section serves as an in-depth introduction to the elements of UGI cancer intelligent diagnosis support mechanisms. The first section mainly introduces the Chinese token separation and word embedding techniques. GRU and CNN attention mechanisms will be introduced in the second section. The third section discusses the weighting and merging of the two attention mechanisms, and the final section will be about the visualization and categorization of results.

8.3.1 Gastroscopy Report Text Vectorized Representation One of the main reasons the gastroscopy report is hard for computers to interpret is that it is unstructured, unlike image and clinical examination data. This unstructured nature is further complicated by the fact that our research subject is Chinese, which is much harder to interpret than other prominent languages. The first step to extracting features from such reports is to structurize and vectorize the entire report. We utilized the Reverse Maximum Matching (RMM) algorithm to construct tokens out of the original Chinese gastroscopy reports. Then, we used word embedding to convert tokens into vectors. A.

Tokenization of Gastroscopy Report

The current state-of-the-art Chinese tokenization methods are the following: rule/dictionary-based tokenization, mutual information-based statistical tokenization, and mixed-method tokenization. The maximum Matching (MM) algorithm belongs to the rule-based tokenization method. The method uses lexicons/wordlist as evidence, works through the document forward to find the longest word that matches records in the lexicon list, then repeats the process until the whole document is processed. The Reverse Maximum Matching (RMM) algorithm is similar to MM. Only when the current string does not match any string in the dictionary

146

8 Data-Driven Cancer Screening and Risk Analytics

does RMM omit the first character instead of the last character in the MM algorithm. Due to the complexity of the Chinese language, in practice, RMM is usually more accurate than the MM method (Zhang et al. 2006). As such, we would utilize the RMM algorithm in our model. From a pre-constructed customized gastroscopy report from previous work, we removed stop words and tokenized the gastroscopy report to create a dataset of word sequences. B.

Word Embedding

The word sequence dataset, while much more useable data, is still unstructured data and could not be interpreted by machines. To create machine-adaptable data out of natural languages, texts need to be converted into numbers, in most cases vectors, to be implemented in machine learning methods. Such is the fundamental problem of text mining and information retrieval, called Text Representation. In this case, we utilize Word Embedding to convert text tokens into vectors, which is one solution among numerous text representation solutions. The process of word embedding is to project a high-dimension space where the dimension number is equal to the total token number into a much lower-dimension count continuous vector space, with each word as a vector in the real number field. In such a low dimension count vector space, the distance between tokens with similar meanings and semantics would be closer to each other. In our research, the numerical representation of tokens and words is taken from the index position of the self-defined dictionary, which through a word embedding layer could be constructed into vectors in a low-dimension vector space, as shown in Formula (8.1): V = H · W,

(8.1)

where V ∈ R d represent token vectors in the low dimension space, W ∈ R |V |∗d represent word embedding matrix and H ∈ R |V | represents high dimension token vectors.

8.3.2 Gastroscopy Report Feature Attention Extraction Gastroscopy reports record the position, location, expression, and other information related to UGI lesions. The information is among the most important factors of how physicians make diagnoses. When writing gastroscopy reports, gastroscopy doctors would follow a certain number of rules and policies, and some indicators of cancer would be clearly laid out in some sections, which makes “local information” important in computerized analysis. On the other hand, it might also be important to acquire the full picture of the whole report to understand the “overall information.” As such, we utilize both local feature-focused CNN-attention and overall relation attention GRU-attention to respectively extract local and overall semantic information, which could improve the total amount of semantic information we extract.

8.3 Intelligent UGI Cancer Screening

A.

147

Feature Attention Extraction based on CNN-Attention

Convolutional Neural Network (CNN) is often utilized in the field of computer vision in order to process images that are constructed with grid information. For the sequential text data of the gastroscopy report, each token goes through the word embedding layer and is transformed into a d dimension vector. Each report contains s tokens so that a matrix of A ∈ R s×d could be constructed, which could be treated as an image to be processed using CNN. Due to the fact that gastroscopy report is sequential, tokens contain correlations within a certain range of “context,” thus, one dimension convolution is often utilized to process text-based features. With the convolution kernel w ∈ R d×s and a token sequence length s, the one-dimension convolution operation could be represented with Formula (8.2): oi = w T · At:t+s−1 ,

(8.2)

where At:t+s−1 represent the t to t + s − 1 line of matrix A. After a bias b is added, a non-linear activation function f is implemented to get convolution feature attention Ct . Common activation function includes sigmoid, tanh, relu, etc. The convolution feature attention function then could be represented as the following: Ct = f (oi + b)

(8.3)

Each convolution kernel w could obtain a Ct from each token sequence. We utilize multiple convolution kernels to operate token vectors from gastroscopy reports and take the average of the multiple kernels to obtain a relatively stable feature attention value C in order to suppress some of the noises within our data. The calculation would be represented as: c=

n 1 Ci , n i=1

(8.4)

where n is the number of convolution kernels, Ci represents the i th attention value output. After the averaging of attention value output from each convolution kernel, a stable attention value c ∈ R s could be obtained. Each Ci represents the degree of importance of the i th token within the gastroscopy report token sequence, as such, the weight of such token. B.

Feature extraction based on GRU-Attention

Gated Recurrent Unit (GRU) is originated from Long Short-Term Memory (LSTM) networks in order to solve some of the existing problems within the LSTM framework. GRU represents a smaller while superior counterpart of LSTM. The forward propagation formula of the GRU network is shown below:    rt = σ Wr · h t−1 , xt

(8.5)

148

8 Data-Driven Cancer Screening and Risk Analytics

   z t = σ Wz · h t−1 , xt

(8.6)

   ht = tanh Wh˜ · rt ∗ h t−1 , xt

(8.7)

h t = (1 − z t ) ∗ h t−1 + z t ∗ ht ,

(8.8)

where rt represent a reset gate, which is utilized to control the previously hidden state unit h t−1 influence on current input xt . If the hidden unit h t−1 has no influence on the current input xt , then rt is reset to 0. z t represent an update gate, which is utilized to control the amount of information being carried from the previous timestamp, which could judge the overall semantic influence of current input xt on the overall input. σ represent a sigmoid function, [] represent the connection between two vectors. We applied the GRU model to process gastroscopy report text, which results in the hidden state vector of token sequences on GRU. Then, the hidden state vector h t is displayed for each time t, then attention mechanisms are applied to the model, in order to calculate the attention weight value of token sequences for each time step, which could be represented as the following: bt = tanh(w ∗ h t + bw )

(8.9)

exp(bt ∗ u) gt =  , t exp(bt ∗ u)

(8.10)

where w and u are parameter matrixes, bw represent the biases vector, and gt is the attention weight value after normalization.

8.3.3 Gastroscopy Report Attention Merging and Weighting After the procedures of CNN-attention and GRU-attention, the overall and regional attention weight distribution of gastroscopy report token sequences could be obtained. We could then use the CNN coded feature attention vector ci as the vector representation of token sequences of each time step t. Then, the two attention weight ct from CNN and gt from GRU are average to merge the two kinds of different attentions, which are combined with the CNN coded feature vector to create a weighted sum, to obtain the final semantic representation vector v of input token sequences. The calculations are represented with Formula (8.11): v=

 (ct + gt ) ∗ ci 2 t

(8.11)

8.3 Intelligent UGI Cancer Screening

149

8.3.4 Gastroscopy Report Classification and Visualization A.

Cancer Screening Reporting Using Text Semantic Classification

The final semantic representation vector v from Sect. 8.3.3 is utilized to classify semantics and reports. Our classification includes healthy, cancer, and suspected cancer, which would make it a multi-classification problem. In practice, the multiclassification problem is often solved via a softmax classifier. Softmax classifier gives each category a probability value to represent the possibility that an input falls within such category. The calculation of such is represented in formula (8.12) e zi , So f tmax(z i ) =  N zn n e

(8.12)

where z i represents the output vector of the i th node, N represents the number of nodes output, which means the number of categories with the classification problem, in this case, 3. With the utilization of softmax, the semantic representation vectors could be transformed into a probability distribution value, which could then be used to find the most likely category that one piece of gastroscopy report would fall into. B.

Visualization of Gastroscopy Report Text

In order to improve the interpretability of UGI cancer diagnosis decision support systems, we implemented attention mechanisms in the gastroscopy report semantic feature extractions to represent the influences of each token, phrase, and paragraph on the overall feature representation. Such is to visualize the most important words and phrases on the diagnosis to increase the acceptability and interpretability of the final result. Due to the fact that the gastroscopy reports, a.k.a token sequences, all have different lengths, we first need to find the longest token sequence length max(s). For those below the maximum length, s number of empty vectors are added to the end of the token sequence in order to secure the same length of different gastroscopy reports. The merged GRU/CNN attention is then used to calculate the attention distribution of the input token sequence. Next, we apply the average attention weight β of the token sequences already filled with empty vectors, then apply visualization color based on the attention of multiples of β. The average attention weight β is calculated from Formula (8.13): β=

1  (gt + ct ) , s s 2

(8.13)

where gt represent the GRU attention value of gastroscopy report text token sequence at the t time step, ct represent the CNN attention value of the token sequence at the t position.

150

8 Data-Driven Cancer Screening and Risk Analytics

8.4 Case Study This section would put the proposed attention-mechanism UGI cancer diagnosis support decision method into the test. Examinations are executed, and the constructed model is compared with benchmark models. First, we need to introduce the dataset we used and the distribution of experimental data. Then the metrics and benchmark models of our experiment are introduced. Next, the model is utilized on real-life data to examine our model’s UGI cancer diagnosis capability. Finally, our model’s interpretability is examined by actually visualizing some of the text semantics of our dataset.

8.4.1 Data The data in this section are provided by a certain large-scale hospital located in Anhui, China. The data are gastroscopy reports, including the examined organ names, lesion positions, lesion structure description, diagnosis result, etc. The dataset consists of reports from 6 positions: esophagus, cardia, stomach fundus, angular incisures, antrum pylorus, and duodenum. We first categorized the dataset into different positions. Then the dataset went through pre-processing procedures such as duplication removal. The final data count is 8399, within which healthy 5849, suspect cancer 541, and cancer 2009. The distribution of our dataset is shown in Fig. 8.3: The dataset is separated into three categories to conduct comparative experiments with a factor of 6:2:2: training set 5039, testing set 1680, and verification set 1680. The distribution of each kind of data is shown in Table 8.1.

Fig. 8.3 Overview of experimental data distribution

8.4 Case Study

151

Table 8.1 Experimental data distribution Training set

Healthy

Suspect cancer

Cancer

Total

3509

324

1206

5039

Testing set

1161

97

422

1680

Verification set

1179

120

381

1680

8.4.2 Metrics In order to analyze and examine the accuracy of the proposed method, we chose a few standard performance evaluation indexes in classification problems, such as Accuracy, Precision, Recall, Receiver Operating Characteristics (ROC) area under the curve (AUC), F1-score, etc. To adapt these 2-way classification indexes into multi-category classification capable, it is common to separate the multi-category classification problems into several 2-way classification problems. The calculation of these metrics is as the following: accuracyi =

T Pi + T Ni T Pi + F Pi + F Ni + T Ni

(8.14)

T Pi T Pi + F Pi

(8.15)

pr ecision i = r ecalli = F1 − scor ei =

T Pi T Pi + F Ni

2 · pr ecision i · r ecalli pr ecison i + r ecalli

T P Ri = F P Ri =

T Pi T Pi +F Ni F Pi F Pi +T Ni

,

(8.16) (8.17)

(8.18)

where i represents the 2-way classification where the ith category is considered positive while the other categories are considered negative. T P represents “True Positive,” F P represents “False Positive,” vice versa. Other than the above representations, the ROC curve could be drawn on a plane where F P R is the X-axis, T P R is the Y-axis. AUC is simply the proportion of area under the ROC curve. In multi-classification problems, macro-averaging is usually utilized to test the overall performance of models. Macro-averaging gives each category the same weight and averages all performance metrics to get an overall rating. Unfortunately, due to the imbalances within our dataset, it is not ideal to give every category the same weight. Thus, we introduce the micro-averaging method, which establishes a global confusion matrix for each sample in the dataset regardless of its category, and

152

8 Data-Driven Cancer Screening and Risk Analytics

then calculates the corresponding index values to reduce the impact of imbalance in sample numbers of different categories.

8.4.3 Benchmark Models In order to prove the effectiveness of the model proposed in this chapter, we select state-of-the-art methods and models that have performed well in the text classification field in recent years as the benchmark model for comparison experiments. The models selected are as follows: (1) TextRNN (Lai et al. 2015): In text classification tasks, RNN is better at capturing longer sequential information, and its structure is flexible and can be changed arbitrarily, such as replacing LSTM units with GRU units. The text classification performance of TextRNN is also high, but its interpretability is poor; (2) TextCNN (Zhang and Wallace 2015): The key point of CNN is that the use of convolution kernels can capture the local correlation characteristics of the input data. Therefore, CNN is often used to extract key information in sentences in text classification tasks. TextCNN mainly uses a one-dimensional convolutional layer and a time-series maximum pooling layer, and treats text data as a one-dimensional image to obtain text features; (3) Gated Recurrent Unit (GRU) + Attention (Zhang et al. 2019): As a variant of LSTM network, it is lightweight and with higher training performances. The GRU network integrated with the attention mechanism can obtain the source input data sensitive to the output result, realize the key information of the input data, and improve the extraction efficiency of text features; (4) Convolutional Neural Network (CNN) + Attention (Yin et al. 2016): In recent years, there have been many breakthroughs in the research of text classification based on neural networks. When traditional CNN processes text classification, it learns text sentences through every single channel, and the information between different channels is not connected with each other. After the attention mechanism is introduced, the attention weight of different channel information can be calculated to link the information between different channels and display the degree of influence of different inputs on the output results, thus improving the model’s interpretability.

8.4.4 Results In order to prove the performance of the UGI cancer intelligent diagnosis supporting method proposed in this chapter on the diagnosis of upper gastrointestinal diseases, we conducted the following experiment: The upper gastrointestinal disease classification method based on GRU-Attention and CNN-Attention is compared with the four models of TextCNN, TextRNN, GRU_ATT, and CNN_ATT selected in 8.4.3 to verify the effectiveness of the method proposed. First, three types of data of healthy samples, suspected samples, and cancer samples on different models are compared to study the accuracy, recall, F1-score, and AUC values of each category

8.4 Case Study

153

of data on different models; then, the accuracy of different models and the evaluation index values and AUC values under the micro-averaging are studied, and the overall classification effects of different models are analyzed and verified. A.

Single Category Comparative Experiment

Table 8.2 lists the upper gastrointestinal disease multi-classification method proposed in this chapter and the selected four models of TextCNN, TextRNN, GRU_ATT, and CNN_ATT. Table 8.3 lists the classification comparison test results of the method proposed in this chapter and the comparison model on the early disease/suspect cancer sample data, and Table 8.4 lists the classification comparison test results of the method proposed and the four comparison models on the cancer sample data. Moreover, the ROC curve of each category under the model proposedr and the ROC curve of the overall data category under the micro-average calculation method is shown in Fig. 8.3a. The categories and overall ROC curves of the comparison model with attention mechanism are shown in Fig. 8.3b, and the categories and overall ROC curves of the remaining comparison models are shown in Fig. 8.3c. In the comparative test of healthy samples, in addition to the AUC index, the method proposed in this chapter is superior to the four text classification models of TextCNN, TextRNN, GRU_ATT, and CNN_ATT in various experimental indicators, as model performance can be increased by up to 4%. In the comparison experiment of suspected samples, the accuracy, F1-score value, and AUC value of the research method are all better than the comparison model. In the Recall index value, the TextRNN model is better than the model in this study. In the comparative test of cancer samples, this method is significantly better than the benchmark model in various Table 8.2 Comparative experiment of healthy samples, with the best performing model in bold Model

Precision

Recall

F1-score

AUC

TextCNN

0.9500

0.9700

0.9600

0.9703

TextRNN

0.9600

0.9500

0.9500

0.8366

GRU_ATT

0.9400

0.9400

0.9400

0.9531

CNN_ATT

0.9300

0.9400

0.9400

0.9527

Ours

0.9600

0.9800

0.9700

0.9694

Table 8.3 Comparative experiment of suspected samples, with the best performing model in bold Model

Precision

Recall

F1-score

AUC

TextCNN

0.6800

0.5300

0.5900

0.9174

TextRNN

0.5500

0.7500

0.6300

0.8317

GRU_ATT

0.5100

0.6200

0.5600

0.8757

CNN_ATT

0.5000

0.5100

0.5100

0.8690

Ours

0.8700

0.7200

0.7900

0.9264

154

8 Data-Driven Cancer Screening and Risk Analytics

Table 8.4 Comparative experiment of cancer samples, with the best performing model in bold Model

Precision

Recall

F1-score

AUC

TextCNN

0.8900

0.9200

0.9100

0.9838

TextRNN

0.9600

0.9000

0.9300

0.9787

GRU_ATT

0.9400

0.8700

0.9000

0.9722

CNN_ATT

0.9200

0.8900

0.9000

0.9750

Ours

0.9800

0.9300

0.9500

0.9858

experimental indicators, and the performance of various experimental indicators can be improved by 1–6%. B.

Overall Category Comparative Experiment

Given the imbalanced characteristics of the experimental data categories, we adopt the micro-averaging calculation method to establish a global confusion matrix for each sample in the data set regardless of the category to calculate the corresponding index value. Table 8.5 lists the comparison test results of the multi-classification method of upper gastrointestinal diseases proposed in this chapter and the selected four models of TextCNN, TextRNN, GRU_ATT, and CNN_ATT on the dataset overall. The experimental results show that in terms of accuracy, recall, F1-score, and AUC value, the method proposed is better than the comparison model by 0.41–5%. In addition, the proposed method is better than the comparison model by 2–5% in terms of accuracy. The accuracy, precision, recall, and F1-score are shown in Fig. 8.4. From the above comparative experimental results, it can be seen that the intelligent assisted diagnosis method for UGI cancer we proposed is the best overall, and it is better than the benchmark model in terms of accuracy, precision, recall, F1 value, and AUC value. At the same time, in the face of the imbalance of experimental data categories, the experimental model has the same performance capabilities for categories with sparse data and is better than the benchmark model. In addition, compared with various mature text classification networks, this method has advantages in avoiding over-fitting and improving model performance for the experimental data. Therefore, the performance of this method is better than the overall performance of CNN-ATT, GRU-ATT, and other models (Fig. 8.5). Table 8.5 Experiment results for all models, our model, shown in bold, outperforms the benchmark models Model

Accuracy

Precision

Recall

F1-score

AUC

TextCNN

0.9237

0.9155

0.9300

0.9321

0.9775

TextRNN

0.9305

0.9263

0.9344

0.9200

0.9178

GRU_ATT

0.8988

0.9103

0.9004

0.9154

0.9667

CNN_ATT

0.9000

0.8952

0.9102

0.8923

0.9661

Ours

0.9525

0.9488

0.9500

0.9475

0.9816

8.5 Risk Analytics and Conclusion

155

Fig. 8.4 The ROC curve for each model. a CNN-GRU ROC curve. b Attention merged CNN / GRU ROC curve. c RNN/ CNN Model ROC curve

8.5 Risk Analytics and Conclusion In this chapter, we proposed a UGI cancer diagnosis supporting model fused with attention mechanism. The method is proposed to alleviate the common problems in computer disease diagnosis, such as lack of interpretability, difficulties in processing unstructured report documents, etc. The proposed method is constructed with a textvectorization processing module, feature attention extraction module, dual-attention fused weight module, and text semantic classification techniques. We consider the most salient contributions and healthcare engineering management insights would be the following: (1) although medical diagnosis texts are unstructured, extremely varied, and are sometimes inaccurate, the medical reports could be transformed into structured data and be analyzed further through deep learning algorithms, such as demonstrated in this chapter; (2) attention mechanisms

156

8 Data-Driven Cancer Screening and Risk Analytics Accuracy

Precision 0.9525

0.96 0.94

0.9237

0.9305

0.96

0.9488

0.94

0.92

0.92 0.8988

0.9263 0.9155

0.9103

0.9

0.9

0.9

0.88

0.88

0.86

0.8952

0.86 TextCNN

TextRNN GRU_ATT CNN_ATT

Ours

TextCNN

TextRNN GRU_ATT CNN_ATT

Recall 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87

F1-score 0.95

0.93

Ours

0.9344

0.96 0.94

0.9475 0.9321 0.92

0.92

0.9102 0.9004

0.9154 0.8923

0.9 0.88 0.86

TextCNN

TextRNN GRU_ATT CNN_ATT

Ours

TextCNN

TextRNN GRU_ATT CNN_ATT

Ours

Fig. 8.5 Overall experimental results

provide significant improvements to both diagnosis performances and the interpretability of the model, as we demonstrate in this chapter that the proposed model could perform better than the benchmark models in most cases; (3) by either generate or interpret examination report, supporting diagnosis systems could achieve a higher interpretability degree, which could be further enhanced through the utilization of attention mechanisms and highlighting the most important decision factors in the report. While the risk of getting cancer has been one of the major concerns for individuals, cancer diagnosis support in smart healthcare does not decrease the risk of people getting cancer. The risk this study tries to alleviate instead is the probability of misdiagnosis. While it is usually regarded that added support of computer-based diagnosis is beneficial, in reality, the effect is always two-folded. Let us assume that for a non-specific type of cancer, the current diagnosis could accurately diagnose α percent of the cases. While it could be seen as simple as a single number, statistics have shown that difference in physician experiences leads to significantly different accuracy, as stated above. The average accuracy of novice, experienced, and expert (veteran) physicians could then be described as α N , α E , αV , respectively. Assuming the state-of-the-art diagnostic support systems could achieve a combined accuracy of αC , the theoretical benefit of such diagnostic supporting systems would be as the following: min = max(αC , αx )

(8.19)

max = min(αC + αx , 1)x ∈ {N , E, V },

(8.20)

8.5 Risk Analytics and Conclusion

157

where min refer to the cases where for all the patients the computer diagnosed correctly, a human physician could also diagnose correctly. This means that if αC < α N then theoretically the minimum benefit of a computer-based diagnostic supporting system would be 0. To put it in layman’s language, it means that if a diagnostic supporting system is more unreliable than its worst human counterpart, then the system is more or less useless. max , however, refers to cases where the diagnostic supporting system’s accuracy all lies in the cases where physicians made the wrong diagnosis. While obviously unrealistic, it does mark the ceiling of the performance of a human–machine collaboration system. In the state-of-the-art cancer diagnosis systems, it is already not rare to notice that αC could sometimes have a greater value than αV , so that min > 0 is confirmed and max is as close to 1 as possible. However, these cancer diagnosis supports often find their way into real-life scenarios extremely difficult. It is common for physicians to raise concerns about the low interpretability of such methods, while patients feel untrustworthiness for the supporting system no matter the accuracy of their diagnosis. From the perspective of risk analytics, it is important to quantify these perceived risk components and examine the risk added by these diagnosis systems. To further examine the risks within the automatic diagnosis systems, we need to further break down the diagnosis results and accuracies. Since each of the diagnoses by human and by computer could result in four outcomes: T P, T N , F P, F N , we could then construct a 2D matrix for all the possible outcomes of human–machine collaborated diagnosis (Table 8.6). The implementation of a machine diagnosis supporting system leads to a highly complex human–machine system. From a layman’s perspective, it could be argued that the system is structurally worse than human-only diagnosis, as the new system brings 6 potential risky outcomes out of 8 total possible outcomes, rather than 2 risky outcomes out of 4 in the previous systems. A more complex system could further be observed if the experiences of physicians are considered. It would be intuitive Table 8.6 All possible results of a two-way classification of a human–machine collaborative system Human

True positive

True negative

False positive

False negative

True positive

True reinforcement

N/A

N/A

Human–machine conflict (true machine)

True negative

N/A

True reinforcement

Human–machine conflict (true machine)

N/A

False positive

N/A

Human–machine conflict (true human)

False reinforcement

N/A

False negative

Human–machine conflict (true human)

N/A

N/A

False reinforcement

Machine

158

8 Data-Driven Cancer Screening and Risk Analytics

for less experienced physicians to have less confidence in their diagnosis, and thus the outcomes where human–machine conflict happens are reversed from veteran physicians. Specifically, in the case of veteran physicians, the risk of misdiagnosis is higher for cases where the machine is right. On the other hand, the risk of misdiagnosis for novice physicians is higher for cases when the machine is wrong. Assuming the cases where these cancer diagnosis outcomes happen could be represented as: α H T P , α H T N , α H F P , α H F N , α M T P , α M T N , α M F P , α M F N , the risk of misdiagnosis due to the human–machine conflict would be represented as:  R+ = σ (α H T P ∩ α M F N + α H T N ∩ α M F P ) + (1 − σ )(α H F P ∩ α M T N + α H F N ∩ α M T P )

(8.21)

Within which σ represent the percentage possibility where physicians change their diagnosis based on machine results. While supporting diagnosis systems are aiming to reduce the risk of physician misdiagnosis by offering an alternative of higher accuracy machine diagnosis, in reality, it could only reduce the risk by correcting human errors when false diagnoses are given by the human, thus creating human–machine conflicts, shown in the following formula:  R− = σ (α H F P ∩ α M T N + α H F N ∩ α M T P )

(8.22)

Thus:  R− +  R+ + (1 − σ )(α H T P ∩ α M F N + α H T N ∩ α M F P ) = α H M ,

(8.23)

where α H M represent all cases where human–machine conflict happens. While theoretically, the system could benefit the human physicians when  R+ <  R− , neither σ nor α M T N + α M T P (machine accuracy) are the deciding factors. It is only when α M T N + α M T P = 1 and σ = 1 that  R+ = 0 and the maximum benefit could be achieved, i.e. the physicians fully trust the machine and the machine is never wrong, which is obviously unrealistic. In reality, the dynamics of human–machine collaboration are much more complex and highly volatile. Furthermore, as could be observed, physicians are put under  R+ and  R− in all situations, no matter if their diagnosis is right or wrong. This could lead to stress and a lack of trust in the system and physicians themselves. However, we could observe that no matter the system behavior, if σ is high, then  R− would also be higher. This is the reason and the objective of this chapter and the overall objective of research trying to increase the interpretability of machine diagnosis supporting systems.

References

159

References Chidambaram S, Sounderajah V, Maynard N, Markar SR (2021) Diagnostic performance of artificial intelligence-centred systems in the diagnosis and postoperative surveillance of upper gastrointestinal malignancies using computed tomography imaging: a systematic review and meta-analysis of diagnostic accuracy. Ann Surg Oncol [Internet]. Nov 11 [cited 2022 Jan 4]. Lai S, Xu L, Liu K, Zhao J (2015) Recurrent convolutional neural networks for text classification. In: Proceedings of the 29th AAAI conference on artificial intelligence. AAAI Press, Texas, pp 2267–73. (AAAI’15) Luo H, Xu G, Li C, He L, Luo L, Wang Z et al (2019) Real-time artificial intelligence for detection of upper gastrointestinal cancer by endoscopy: a multicentre, case-control, diagnostic study. Lancet Oncol 20(12):1645–1654 Menon S, Trudgill N (2014) How commonly is upper gastrointestinal cancer missed at endoscopy? A meta-analysis. Endosc Int Open 2(2):E46–50 Wardle J, Robb K, Vernon S, Waller J (2015) Screening for prevention and early diagnosis of cancer. Am Psychol 70(2):119–133 Yin W, Schütze H, Xiang B, Zhou B (2016) ABCNN: attention-based convolutional neural network for modeling sentence pairs. Trans Assoc Comput Linguist 4:259–272 Zhang Y, Wallace B (2015) A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. 2015 Oct 13 [cited 2022 Jan 6]; Available from: https://arxiv.org/abs/1510.03820v4 Zhang L, Li Y, Meng J (2006) Design of Chinese word segmentation system based on improved Chinese converse dictionary and reverse maximum matching algorithm. In: Feng L, Wang G, Zeng C, Huang R (eds) Web information systems—WISE 2006 workshops. Springer, Heidelberg, pp 171–81. (Lecture Notes in Computer Science). Zhang B, Xiong D, Su J (2019) A GRU-gated attention model for neural machine translation. arXiv:170408430 [cs] [Internet]. 2019 Nov 13 [cited 2022 Jan 4]. Available from: http://arxiv. org/abs/1704.08430

Chapter 9

ICU Mortality Prediction and Risk Analytics

Hospitalization represents the phase that only patients in critical situations would experience. Many different hospitalization units exist in hospitals, including neonatal units, infant health units, pediatric units, step down units, oncology units, pre- and post-surgical units, medical units, rehabilitation wards, long-term care wards, and all the Intensive Care Unit (ICU) variant of all the above units. These hospital units represent the most costly and risky sector of the healthcare industry. According to the Centers for Medicare & Medicaid Services, hospital care cost about 1.19 trillion USD in 2018, represent roughly 1/3 of the total national health expenditures, and are projected to increase to 2.08 billion by 2028. Furthermore, due to the severe or developing conditions of patients in hospitalization, it represents the part of the cycle of care that results in the highest mortality rate, with readmission rate of 7% and death rate between 10% and 29% (Armony et al., 2018, Wang et al., 2019). A great amount of attention is put into the safety procedures of these sectors of healthcare, thus the innovation in such units must focus on reliability and interpretability. The problems of hospitalization units boil down to two central problems: inadequate number of hospital beds and inadequate procedure of monitoring and discharge. The capacity of each hospital is limited and the shortage of hospital beds is rampant throughout the world. Obviously, if the required treatment is not delivered due to a societal overall lack of medical resources, it leads to an increase in hospital bed cost and inevitable human life losses. To alleviate this problem, hospital beds need to rotate as fast as possible when the overall capacity of healthcare systems could not be raised quickly enough. To increase the rotation of patients, recovering patients need to be identified and discharged early to give space for other patients in more serious conditions. This represents a need for highly accurate patient situation monitoring and assessment procedures. Currently, this procedure is usually done through physician experiences and quantification scales. Inaccuracies in these procedures could lead to readmission cases or even life loss after discharge, which could become scandalous for hospitals. While normal hospital sections face a much lower chance of life loss, the ICU represents the place where inaccuracies would materialize in the worst form. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 S. Ding et al., Smart Healthcare Engineering Management and Risk Analytics, AI for Risks, https://doi.org/10.1007/978-981-19-2560-3_9

161

162

9 ICU Mortality Prediction and Risk Analytics

Fig. 9.1 The structure of this chapter and its position in the cycle of care

In this chapter, we present a novel ICU mortality prediction and risk analytics method based on machine learning and feature engineering. The purpose of such a method is to reduce the readmission and mortality rate of ICU units and increase the rotation rate and accuracy of patients, which leads to an increase in ICU performances. The structure of this chapter is shown in Fig. 9.1.

9.1 ICU Risk Management As a microcosm of hospital specialization, Intensive Care Units (ICU) treats patients in various critical conditions and is the department with the highest incidence rate of clinical risk events. According to data from the Society of Critical Care Medicine (SCCM), since 2009, ICUs in various countries have been operating at full capacity. 36% of patients are at risk of delayed treatment, and abnormal length of hospital stay due to shortage of medical resources. The infection rate of patients in ICU is as high as 24.41, 50% of patients have various complications during treatment, the average readmission rate in ICU is as high as 7%, and the mortality rate is between 10 and 29%. Therefore, ICU patients are more likely to have clinical risks such as readmission, death, and hospital infection compared with general wards. These clinical risks bring serious harm to the health of patients and the reputation of the hospitals. The management of these risks is necessary for scheduling medical resources and helping medical staff make reasonable decisions to improve the quality and efficiency of treatment. However, due to the complicated conditions of patients in the ICU, monitoring equipment, and complicated treatment measures, it is difficult to accurately assess the risk of patients only relying on experience-based judgment. In the context of smart healthcare, the rapid penetration of big data and artificial intelligence provides opportunities for developing data-driven ICU risk management.

9.2 Feature Engineering in ICU Mortality Prediction

163

Among them, mortality risk is the most urgent for patients. However, due to the complex conditions of patients in the ICU, diverse assessing equipment, and complicated treatment measures, patients’ mortality risk could not be accurately assessed by relying only on the experiences and judgments of medical staff. With the exponential growth of ICU data in recent years, many scholars advocate using machine learning methods to build ICU mortality prediction models. Venugopalan et al. (2019) combined support vector machines with generalized linear models to predict whether ICU patients would decease. Lin et al. (2019) used electronic case data from four hospitals to build a random forest model to predict the mortality rate of ICU patients with renal failure, and the results are better than the logistic regression algorithm. In addition, the use of mixing methods could enhance the diversity and accuracy of different prediction models. Wyk et al. (2019) developed a multi-level analysis using multiple base models to improve the performance of predicting the mortality risk of ICU patients. Awad et al. (2017) proposed an integrated random forest model, which compared with three models based on single algorithms such as naive Bayes and decision trees to achieve the highest accuracy. Although the existing ICU mortality prediction research based on medical and health data has brought great help to clinical decision-making, there is still a problem of low mortality prediction performance due to the inaccurate identification of risks. The main reason for this problem is that existing studies did not consider the impact of risk factors on mortality risk and the interactions between risk factors when selecting ICU mortality risk factors. In order to solve this problem, this research proposes a mortality risk prediction method that introduces the characteristic discrimination and independence to measure the impact of risk factors on the death risk and the impact of different risk factors and select the most relevant risk factors, and further build a mortality prediction model based on the random forest to improve the performance of mortality risk prediction.

9.2 Feature Engineering in ICU Mortality Prediction In clinical practice, doctors usually need to assess mortality risk in patients by observing relevant risk factors. Accurate selection of ICU death risk factors could lead to a more effective prediction and assessment of mortality risk of patients. We constructed a mortality risk factor selection method that considered feature discrimination and independence based on the maximum information coefficient and information gain rate. Furthermore, based on the random forest algorithm, we designed a mortality risk prediction method that considered feature discrimination and independence to improve the predictive performance of patient death risk. From the perspective of informatics, this chapter uses the information gain rate to measure the identification of features for mortality. In information gain, whether the selected factors are helpful in predicting the mortality and what extent can be measured by the amount of information contained in the feature. In actual ICU conditions, the values of some indicators of patients are always similar. For example,

164

9 ICU Mortality Prediction and Risk Analytics

the maximum body temperature of patients will not exceed 40°C, with 38°C and 39°C account for the majority. In contrast, others laboratory indicators have more values. In order to avoid the use of information gain to calculate the feature identification will be biased towards factors with more values, this chapter used the information gain rate to define the identification. Features with higher identification are more related to mortality. For the mortality prediction model, U is the data set, and C is the category label. C has two different values: C = 0, which means that patient will not die; C= 1, which means that patient will die. P(Ci ) represents the proportion of the number of cases with the category label Ci . The uncertainty of the prediction model category label can be defined as Formula (9.1). It means the uncertainty of the prediction model when no medical data is provided. H (C) = −

2 

P(Ci ) log2 P(Ci )

(9.1)

i=1

For any feature   X , there are d different values. x j represents the j-th value of feature X. P Xj represents the proportion of the number of cases whose feature X is X j . P Ci |x j represents the proportion of the number of cases whose feature X is X j among the cases labeled Ci . When the feature X is used as input, the degree of uncertainty in the model will be reduced accordingly. The degree of uncertainty reduction can be calculated using Formula (9.2). H ( C|X ) = −

d 2        P xj P Ci |x j log2 Ci |x j j=1

(9.2)

i=1

The amount of information contained in feature X is used to measure the degree of credibility that it contributes to the prediction model. The specific calculation is shown in Formula (9.3). I (C; X ) = H (C) − H ( C|X )

(9.3)

The more information the feature X brings, the more important the feature X is. However, assuming that there are a large number of values of a certain feature, such as ID (assuming that ID is used as one of the risk factors in mortality prediction). Because the ID of each patient is different, it can be seen from the formula (9.3) that ID has a strong correlation with the mortality prediction. It is obviously incorrect. As shown in Formula (9.4), this chapter introduces information gain rate to measure the identification of features quantitatively. Among them, H (X ) represents the uncertainty when dividing the data set U with feature X . It can be calculated by Formula (9.5). G(X ) =

I (C; X ) H (X )

(9.4)

9.2 Feature Engineering in ICU Mortality Prediction

165

2 d       H (X ) = P Ci j log2 P Ci j

(9.5)

j=1 i=1

In Formula (9.5), Ci j represents  the category label of the i-th type when the feature X takes the j-th value, and P Ci j indicates the proportion of this case. Then the identification of X can be further refined as:    2   2  d j=1 P x j i=1 P C i |x j log2 C i |x j − i=1 P(C i ) log2 P(C i ) G(X ) =     d 2 j=1 i=1 P C i j log2 P C i j (9.6) The identification can measure the correlation between the features and mortality, but the correlation between the features cannot be determined. There are many studies that have shown that Pearson’s coefficient can well find the linear correlation between two features. But it is not suitable for measuring the correlation between the indicators of ICU patients. The maximum information coefficient (MIC) can be used to measure the nonlinear correlation between two random variables. This chapter uses the MIC to measure the independence between risk factors. For the given two feature X and Y , xi and yi respectively represent the values of feature X and Y of the j-th patient in the data set. Furthermore, n is the number of cases in the data set. According to the definition of the MIC, this chapter constructs a coordinate system with features X and Y as the horizontal and vertical axes. We let the X -axis direction be divided into |X | parts and the Y -axis direction into |Y | parts. Then the correlation between X and Y is: M(X, Y ) =

max

|X ||Y |