Advanced Computational Intelligence in Healthcare-7 (Studies in Computational Intelligence, 891) [1st ed. 2020] 3662611120, 9783662611128

Citation preview

Studies in Computational Intelligence 891

Ilias Maglogiannis Sheryl Brahnam Lakhmi C. Jain   Editors

Advanced Computational Intelligence in Healthcare-7 Biomedical Informatics

Studies in Computational Intelligence Volume 891

Series Editor Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland

The series “Studies in Computational Intelligence” (SCI) publishes new developments and advances in the various areas of computational intelligence—quickly and with a high quality. The intent is to cover the theory, applications, and design methods of computational intelligence, as embedded in the fields of engineering, computer science, physics and life sciences, as well as the methodologies behind them. The series contains monographs, lecture notes and edited volumes in computational intelligence spanning the areas of neural networks, connectionist systems, genetic algorithms, evolutionary computation, artificial intelligence, cellular automata, self-organizing systems, soft computing, fuzzy systems, and hybrid intelligent systems. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution, which enable both wide and rapid dissemination of research output. The books of this series are submitted to indexing to Web of Science, EI-Compendex, DBLP, SCOPUS, Google Scholar and Springerlink.

More information about this series at http://www.springer.com/series/7092

Ilias Maglogiannis Sheryl Brahnam Lakhmi C. Jain •

•

Editors

Advanced Computational Intelligence in Healthcare-7 Biomedical Informatics

123

Editors Ilias Maglogiannis Department of Digital Systems University of Piraeus Piraeus, Greece

Sheryl Brahnam Computer Information Systems Missouri State University Springfield, MO, USA

Lakhmi C. Jain School of Electrical and Information Engineering University of South Australia Adelaide, Australia

ISSN 1860-949X ISSN 1860-9503 (electronic) Studies in Computational Intelligence ISBN 978-3-662-61112-8 ISBN 978-3-662-61114-2 (eBook) https://doi.org/10.1007/978-3-662-61114-2 © Springer-Verlag GmbH Germany, part of Springer Nature 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer-Verlag GmbH, DE part of Springer Nature. The registered company address is: Heidelberger Platz 3, 14197 Berlin, Germany

Preface

During the last decades, there has been a significant increase in the level of interest regarding the use of computational intelligent systems in the fields of biomedicine and healthcare provision. As the production and collection of digital health-related data continue to expand (the field known as electronic healthcare) and new technologies enable the rapid processing of large data volumes (the field known as big data), healthcare organizations are asking medical personnel and nurses to interact increasingly with computer systems to perform their duties. Furthermore, health data computational analysis is used for developing expert systems and machine learning algorithms to serve as diagnostic tools or to monitor therapeutic procedures or even to create new therapies. At the organization level, health data analysis is used to drive the strategic decisions of managers and stakeholders, while assisting policy makers in public health policies. In this context, the fields of computational analysis and artificial intelligence in medicine attract a lot of researchers working in the AI domain. This book is a part of a series devoted to this emerging field of computational intelligence (CI) in health care, attempting to provide surveys and practical examples of artificial intelligent applications in the areas of human–machine interface (HMI) and affective computing, machine learning, big health data and visualization analytics, computer vision and medical image analysis. The volume also addresses new and emerging topics in CI for electronic healthcare such as the utilization of social media (SM) and the introduction of new intelligent paradigms in the critical areas of security and privacy. A summary of the contributions per chapter follows. Chapter “Human–Machine Interfaces for Motor Rehabilitation” seeks to describe the principles of modern non-invasive human-machine interface (HMI) systems and to present current trends regarding the methods used to capture physiological and non-physiological motor-related data in order to control external devices within a rehabilitation framework. Furthermore, in regard to classification and parameter complexity, computational intelligence tools, machine learning approaches and simulation testing are presented. The relevant applications are discussed within a taxonomy based on the nature of the motor-related source data, v

vi

Preface

while methodological aspects and future challenges concerning the design of HMI systems for rehabilitation purposes are also included. Awareness of the emotion in human–computer interaction is a challenging task when building human-centred computing systems. Emotion is a complex state of mind that is affected by external events, physiological changes and, generally, human relationships. Researchers suggest various methods of measuring human emotions through the analysis of physiological signals, facial expressions, voice, etc. Chapter “Passive Emotion Recognition Using Smartphone Sensing Data” presents a system for recognizing the emotions of a smartphone user through the collection and analysis of data generated from different types of sensors on the device. Data collection is carried out by an application installed on the participants’ smartphone provided that the smartphone remains in their pocket throughout the experiments. The collected data is processed and utilized to train different classifiers (decision trees, naïve Bayes and k-nearest neighbours). The described system applied to a smartphone demonstrates the feasibility of an emotion recognition approach through a user-friendly scenario for users’ activity recognition. Chapter “Intelligent and Immersive Visual Analytics of Health Data” deals with the massive amounts of health data have been created together with the advent of computer technologies and next-generation sequencing technologies. Analytical techniques can significantly aid in the processing, integration and interpretation of the complex data. Visual analytics field has been rapidly evolving together with the advancement in automated analysis methods such as data mining, machine learning, and statistics, visualization and immersive technologies. Although automated analysis processes greatly support the decision-making, conservative domains such as medicine, banking and insurance need trusts on machine learning models. Explainable artificial intelligence could open the black boxes of the machine learning models to improve the trusts for decision-makers. Immersive technologies allow the users to engage naturally with the blended reality in where they can look at the information in different angles in addition to traditional screens. This chapter reviews and discusses the intelligent visualization, artificial intelligence and immersive technologies in the health domain. This chapter illustrates also the ideas with various case studies in genomic data visual analytics. The chapter “Interactive Process Indicators for Obesity Modelling Using Process Mining” belongs in the same area of big data analytics. World Health Organization defines overweight and obesity as abnormal or excessive fat accumulation that represents a risk to health. Obesity and overweight are associated with increased risk of comorbidities and social problems that negative impact on the quality of life. Due to the complexity of the problems, it is necessary to classify obesity based on a set of factors rather than a simple increase in body weight. The objectives of this work were, to examine BMI and data available from comorbidities associated to obesity, from a dynamic perspective thanks to the use of process mining tools, in order to obtain patterns of patients’ behaviours. On the other hand, the goal and contextualized interactive Key Process Indicators (iKPIs) in the field of obesity and related conditions to support health professionals to interact with the process. Modelling iKPIs has enhanced views, which will help the professionals to better

Preface

vii

perception of these processes. Professionals will monitor patient’s progress iteratively and will interact with the system to fine-tune interventions and treatments. The developed strategy can support both the characterization of general process-based KPI and the analysis of individual and personalized aspects of the processes going from general to individual. This method was applied to real data extracted from a tertiary hospital in Spain. Chapter “Recent Machine Learning Approaches for Single-Cell RNA-seq Data Analysis” involves the processing of DNA sequencing, which has become an extremely popular assay with researchers claiming that in the distant future, the DNA sequencing impact will be equal to the microscope impact. Single-cell RNA-seq (scRNA-seq) is an emerging DNA sequencing technology with promising capabilities, but with major computational challenges due to the large-scaled generated data. Given the fact that sequencing costs are constantly decreasing, the volume and complexity of the data generated by these technologies will be constantly increasing. Towards this direction, major computational challenges are posed at the cell level, in particular, when focusing on the ultra-high dimensionality aspect of the scRNA-seq data. The main challenges are related to three pillars of machine learning (ML) analysis, classification, clustering and visualization methods. Although there has been remarkable progress in ML methods for single-cell RNA-seq data analysis, numerous questions are still unresolved. This review records the state-of-the-art classification, clustering and visualization methods tailored for single-cell transcriptomics data. Chapter “A Review on State-of-the-Art Computer-Based Approaches for the Early Recognition of Malignant Melanoma” reviews the state-of-the-art techniques used in the development of computational intelligent systems for the detection of skin cancer and especially melanoma, which continues to be a rare form of skin cancer but causes the majority of skin cancer-related deaths. The most common technique for detection of melanoma is dermoscopy (or dermatoscopy or epiluminescence microscopy ELM), which performs the examination through an optical system (magnifying glass) with a light source (polarized light), allowing an in-depth visualization of features used for the diagnosis. Over the past decades, the efforts have been made to create computer-based systems able to analyse such dermoscopy images, assisting the early detection of skin cancer, while also allowing repeatability of results. One major issue of image dermoscopy is the inability to detect early melanoma or cases that lack optical features. To deal with that issue researchers have focused lately on molecular techniques. Aim of this chapter is to present the state-of-the-art concerning the detection methods of malignant melanoma and describe the contributions made in this area of research. Chapter “Cardiovascular Disease Stratification Based on Ultrasound Images of the Carotid Artery” deals with the identification of cardiovascular disease (CVD) through ultrasound scans of the arteries and, more specific, the common carotid artery (CCA). Measurement of the intima–media thickness (IMT) of the CCA is an established indicator of CVD. Several reports have indicated differences of the IMT of CCA and related then with various risk factors as well as their

viii

Preface

association with the risk of stroke. This chapter presents the work presented in this direction. The presented CVD stratification system is based on image normalization, speckle reduction filtering and active contour segmentation, for segmenting the CCA, performing IMT measurements and providing the differences between the left and right sides. The results are based on a group of 1104 longitudinal-section ultrasound images acquired from 568 men and 536 women out of which 125 had cardiovascular symptoms (CVD). The main findings can be summarized as follows: (1) there was no significant difference between the CCA left side IMT and the right side IMT; (2) there were statistically significant differences for the IMT measurements between the normal group and the CVD group for both the left and the right sides; and (3) there was an increasing linear relationship of the left and right IMT measurements with age for the normal group. Chapter “Forecasting and Prevention Mechanisms Using Social Media in Health Care” involves the utilization of social media (SM), which is establishing a new era of tools with multi-usage capabilities. Governments, businesses, organizations, as well as individuals, are engaging in, implementing their promotions, sharing opinions and propagating decisions on SM. We need filters, validators and a way of weighting expressed opinions in order to regulate this continuous data stream. This chapter presents trends and attempts by the research community regarding: (a) the influence of SM on attitudes towards a specific domain, related to public health and safety (e.g. diseases, vaccines, mental health), (b) frameworks and tools for monitoring their evolution and (c) techniques for suggesting useful interventions for nudging public sentiment towards best practices. Based on the state of the art, authors in this chapter discuss and assess whether SM can be used as means of prejudice or esteem regarding online opinions on health care. We group the state of the art in the following categories: virus-illness outbreaks, antivaccination, mental health, social trends and food and environment. Furthermore, we give more weight to virus-illness outbreaks and the antivaccination issues/trends in order to examine disease outbreak prevention methodologies and vaccination/antivaccination incentives, while discussing their performance. The goal is to consolidate the state of the art and give well-supported directions for future work. To sum up, this chapter discusses the aforementioned concepts and related biases, elaborating on forecasting and prevention attempts using SM data. Finally, chapter “Security and Privacy Issues for Intelligent Cloud-Based Health Systems”, which concludes this the volume, deals with intelligent security and privacy in the healthcare IT sector. New technological advances such as cloud computing provide benefits and have changed the way we store, access and exchange information. Especially, in the healthcare IT sector, cloud-based systems offer great potential, from many perspectives, including improved medical diagnosis, accurate and faster prediction and cost-effective management treatment. In an attempt to assist cloud providers and healthcare organizations to secure their cloud-based environment and to adopt the appropriate measures for data protection, this chapter presents an overview of the security and privacy requirements of cloud-based healthcare systems. Specifically, this chapter starts with the presentation of the reported threats in cloud-based health systems, continues with the

Preface

ix

identified objectives and assets and concludes with measures for the mitigation of the identified threats. Due to the fact that migration into cloud-based healthcare systems, in most cases, implies that data subjects lose control of their data, and many scientists have raised their worries about this. It is therefore needed to reconsider security, privacy and trust requirements, in the context of cloud computing. This chapter makes concrete recommendations for improving the protection level of cloud-based health organizations, cloud providers, hospitals and patients. The included in this book chapters, while of course not comprehensive in addressing all the possible aspects of the aforementioned areas, are indicative of the explosive nature of interdisciplinary research going on in this area. All three editors, we are indebted to the authors who have contributed chapters on their respective fields of expertise and worked hard in order of deadlines to be met and for the overall book to be meaningful and coherent. Piraeus, Greece Springfield, USA Adelaide, Australia

Ilias Maglogiannis Sheryl Brahnam Lakhmi C. Jain

Contents

Human–Machine Interfaces for Motor Rehabilitation . . . . . . . . . . . . . . Ioannis Kakkos, Stavros-Theofanis Miloulis, Kostakis Gkiatis, Georgios N. Dimitrakopoulos and George K. Matsopoulos

1

Passive Emotion Recognition Using Smartphone Sensing Data . . . . . . . I. P. Kalogirou, A. Kallipolitis and Ilias Maglogiannis

17

Intelligent and Immersive Visual Analytics of Health Data . . . . . . . . . . Zhonglin Qu, Chng Wei Lau, Daniel R. Catchpoole, Simeon Simoff and Quang Vinh Nguyen

29

Interactive Process Indicators for Obesity Modelling Using Process Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zoe Valero-Ramon, Carlos Fernandez-Llatas, Antonio Martinez-Millana and Vicente Traver Recent Machine Learning Approaches for Single-Cell RNA-seq Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aristidis G. Vrahatis, Sotiris K. Tasoulis, Ilias Maglogiannis and Vassilis P. Plagianakos A Review on State-of-the-Art Computer-Based Approaches for the Early Recognition of Malignant Melanoma . . . . . . . . . . . . . . . . Georgia Kontogianni and Ilias Maglogiannis

45

65

81

Cardiovascular Disease Stratification Based on Ultrasound Images of the Carotid Artery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Efthyvoulos Kyriacou and Christos Loizou

xi

xii

Contents

Forecasting and Prevention Mechanisms Using Social Media in Health Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Paraskevas Koukaras, Dimitrios Rousidis and Christos Tjortjis Security and Privacy Issues for Intelligent Cloud-Based Health Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Dimitra Georgiou and Costas Lambrinoudakis

Human–Machine Interfaces for Motor Rehabilitation Ioannis Kakkos, Stavros-Theofanis Miloulis, Kostakis Gkiatis, Georgios N. Dimitrakopoulos and George K. Matsopoulos

Abstract Neurological disorders affect a large part of the population, causing cognitive and motor impairments. To that end, non-pharmacological interventions targeting support and restoration of the disrupted functions have been a major issue in modern society. New technologies enable effective communication between the affected individual and an external system, establishing the concept of a human– machine interface (HMI). This chapter seeks to describe the principles of modern noninvasive HMI systems and to present current trends regarding the methods used to capture physiological and non-physiological motor-related data in order to control external devices within a rehabilitation framework. Furthermore, in regard to classification and parameter complexity, computational intelligence tools, machine learning approaches and simulation testing are presented. The relevant applications are discussed within a taxonomy based on the nature of the motor-related source data, while methodological aspects and future challenges concerning the design of HMI systems for rehabilitation purposes are also included. Keywords Motor impairments · Rehabilitation · Human–machine interfaces · Motor functions modeling · Computational intelligence

1 Background According to the World Health Organization (WHO), every year 15 million people suffer from stroke, almost 5 million of whom suffer from severe residual impairment. Furthermore, 250,000–500,000 people suffer a spinal cord injury annually, usually causing partial disability or quadriplegia [1]. Typically, such neurological disorders I. Kakkos (B) · S.-T. Miloulis · K. Gkiatis · G. N. Dimitrakopoulos · G. K. Matsopoulos School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece e-mail: [email protected] G. N. Dimitrakopoulos Department of Medicine, University of Patras, Patras, Greece © Springer-Verlag GmbH Germany, part of Springer Nature 2020 I. Maglogiannis et al. (eds.), Advanced Computational Intelligence in Healthcare-7, Studies in Computational Intelligence 891, https://doi.org/10.1007/978-3-662-61114-2_1

1

2

I. Kakkos et al.

have the ability to affect motor functions by disrupting any stage of the neuromuscular motor pathway which mainly consists of the brain, the spinal cord with its connected nerves and the muscle groups (which function as signal receptors and movement actuators). Motion obstruction can occur as a result of a single-stage disturbance, while the rest may stand unharmed. Indicatively, in the case of a spinal cord injury, both brain and muscle groups are intact; however, the information channel between the two is compromised [2]. In a different manner, a stroke may tamper with certain brain functions, thus impairing or altering their potency, without damaging the information flow to the muscle groups [3]. In both cases (or any disorder with similar symptoms), motor rehabilitation systems usually work under the assumption that muscular potential remains mostly unaffected, while a flawed “control signal” is delivered to the muscle groups either due to the fact that the brain does not generate the appropriate signal or due to transmission problems. A major cause of such conditions is an imbalance between excitatory and inhibitory inputs of the nerve signal to motor neurons (spasticity), driving an increment in the muscle activity [4]. Moreover, central nervous system motion disorders include loss of muscle function, decreased movement control, lower limb reflexes and muscle atrophy [5]. People suffering from such motor impairments have to endure the social and economic costs of their reduced motor independence, relying on caregivers or healthcare professionals for assistance. To overcome the neurophysiological deficits and utilize residual brain and muscular functions, immediate rehabilitation processes take place involving various multidisciplinary teams (doctors, physiotherapists, social workers, etc.) [6]. In order to maximize patients’ benefits and meet their demands with regards to the activities of daily living, motor rehabilitation methods focus on (re)training neural pathways, restoring range of motion and keeping musculature active [7]. However, the rehabilitation phase is time-consuming, resulting in social exclusion and high support costs for people with reduced mobility [8]. In addition, the impatience of the people involved in fast recovery can lead to disappointment and frustration, thus patients often become skeptical regarding the outcomes of rehabilitation processes and their commitment diminishes [9]. On top of that, the motor recovery procedures are often incomplete and lack personalization, while the constantly increasing costs are restrictive for people with medium and low economic ability [10, 11]. All of the above necessitate the implementation of interventions to support the affected motor functions, as well as applications for gradual rehabilitation procedures, designed for the enhancement of the patients’ independence and the restoration of their daily lives to normal. To that end, available hardware modules and artificial intelligence algorithms can provide reliable setups and reduce rehabilitation costs by employing simultaneous motor function assistance and rehabilitation capabilities.

Human–Machine Interfaces for Motor Rehabilitation

3

2 Human–Machine Motor Rehabilitation Interfaces A human–machine interface (HMI) acts as a dynamic connection between a person and a machine, system or device, allowing bidirectional interactions. In general, HMIs involve a wide range of hardware and software setups enabling the translation of real-time user inputs into system commands [12]. Since HMIs provide flexible customization in combination with user-friendliness, they are frequently utilized in the rehabilitation of human motion by digitizing specific kinetic characteristics and communicating them in real time to a controller that in turn supports or enhances motor functions [12]. To decipher each motion, motor characteristics are constantly quantified, including joint angles, step length as well as physiological features derived by sensor measurements of muscle and brain activity [13, 14]. As such, noninvasive electroencephalographic (EEG) and electromyographic (EMG) recordings have been consistently employed in modeling the motor brain mechanisms in parallel with muscular contractions of various muscle groups [14]. In this fashion, to achieve the desired modeling level, the EEG and EMG data recordings are taken under appropriate experimental protocols, carefully designed to capture the task-related signals of both healthy individuals and patients. Furthermore, experimental designs frequently entail real and imaginary motion tasks, employing visual and auditory stimuli to exclude motion-irrelevant stimulus-induced activity [15]. The above data can be used either individually within a single-modality HMI (physiological/non-physiological) or incorporated into a hybrid framework in order to provide motor rehabilitation capabilities. This taxonomy depends on the source data extracted for interpreting the human intention and/or action. As such, the primary step in designing an HMI is the extraction of motion information, where the main goal is to identify and amplify the desired movement in a way that feels natural and fluid to the impaired individual. To achieve this, the extracted data describing real-time motion parameters are fed into classification algorithms used to provide a control signal for a receiver device, therefore interpreting a corresponding human intention or action [12]. The next step after motor function analysis and modeling entails the incorporation of these results into a real-time adaptable framework. HMIs seek to interpret the user’s intent based on movement parameters and subsequently generate an appropriate output, which corresponds to a specific rehabilitation device command. However, in order to assess the output efficacy in accordance with the desired movement, it is necessary to capture indicative response parameters to provide the controller with sufficient closed-loop feedback. Accordingly, the feedback signal provides information regarding the muscle contraction and/or the movement result from a kinetic scope, thus allowing real-time control adjustments [16, 17]. In addition, the continuous inflow of feedback data enables adaptive training of the pattern recognition algorithms, steadily improving the classification performance and causing the HMI response to converge to the individual’s everyday needs [16]. In the context of motor rehabilitation protocols, the systems driven by the HMI control signal mostly include orthoses, robotic exoskeletons and stimulators [18–20].

4

I. Kakkos et al.

Typically, although the distinction between the two terms is not always clear, orthotics refers to a wide variety of motor support devices/accessories, while exoskeletons refer to a more precise class of mechanical devices [21]. As to robotic exoskeletons, they have been classified depending on the nature of their contribution to the movement, which for example can be either resistive (for strengthening purposes), assistive (augmenting the user’s movement) or passive (movement performed solely by the exoskeleton) [20, 22]. The stimulator devices most commonly used for triggering muscular activity perform functional electrical stimulation (FES). This technique is based on applying electrical pulses to muscle groups, resulting in their stimulation and subsequent contraction [23]. To that extent, the FES output serves as a substitute for the “normal” control signal, in order to cause muscle contraction and therefore assist a patient in performing a certain motion. As a result, FES treatment can be used for both motion support and patient training, aiming for restoration through neurorehabilitation [24]. It is important to note that the individual should maintain some fundamental muscular strength to serve as a baseline for the application of FES. For instance, gait-support FES systems require that the individual is able to walk even for a few meters with external support [19]. In Fig. 1, the basic components of an HMI

Fig. 1 Rehabilitation HMI design

Human–Machine Interfaces for Motor Rehabilitation

5

system are shown, consisting of physiological and non-physiological data recordings by various sensors, the processing via a computational model and the feedback loop, where the control output is driving a stimulator or an exoskeleton. In comparing normal and FES-induced muscle contractions, it should be noted that artificially elicited activations lack a smooth onset that should correspond to uniform recruitment of motor units [25]. Instead, they possess an “all-or-nothing” aspect and depend heavily on variables such as the electrode position and characteristics as well as the stimulation parameters (waveform, frequency, pulse width, etc.) [25, 26]. Due to that fact, the parameters for every FES application must be chosen with the aim of achieving the optimal response (i.e., muscle activation) while applying a low amount of current, in order to minimize possible side effects. In this context, various FES devices and corresponding rehabilitation schemes have been proposed [26].

3 Computational Intelligence Tools As described above, the control of the receiver devices within a human–machine interface requires the efficient translation of the physiological or non-physiological measurements into system commands. The latter represents distinct operation states, which are equivalent to different classes based on the output of pattern recognition algorithms used to maximize the variance among the extracted features [12, 27]. These machine learning frameworks, built by employing mathematical models, are utilized to find common structures within the input values, assigning them to predefined labels either through training and testing or by clustering data points. Feedback incorporation can further enhance algorithmic output by increasing training paradigms and therefore adjust to dynamic changes. As such, k-nearest neighbor (kNN), support vector machines (SVM) with different kernel functions and linear discriminant analysis (LDA) classifiers have illustrated the potential of real-time classification in plenty of motor-related paradigms [28, 29]. Soft labeling techniques that incorporate fuzzy class inclusion have also been consistently adapted in the effective discrimination between motor functions [30]. Furthermore, artificial neural networks (ANNs) and convolutional neural networks (CNNs) that can take into account nonlinear functions and process large amounts of complex data are equally popular in HMI system implementations [27, 31]. Moreover, ensemble learning allows for more flexible computing by combining multiple classifiers or regression methods in comparison with the separate employment of the learning algorithms [32]. Since there is no golden standard regarding the design of an ideal HMI system, the abundance of available pattern recognition methods presents a complication in determining an approach with global acceptance. In this view, the selection of the optimal classifiers for the HMI implementation is based on accuracy results as well as computational cost and speed [33]. On top of that, the overall performance of a pattern recognition algorithm heavily depends on the input features. The successful choice of meaningful motor-related features that quantify the motion system processes necessitates the comprehensive analysis of the precise interactions between

6

I. Kakkos et al.

the user and the hardware, entailing modeling on both biological and kinematic levels. In pursuit of effective HMI implementation, further insight into the multimodal neuromuscular mechanisms has to be obtained by defining global modeling metrics concerning motor processes. Moreover, motion models are required to be evaluated under akin realistic conditions, regulated under the premise of absolute patient safety, which is a major priority in determining the optimal system parameters [34]. Accordingly, effective modeling often results in the development of simulation protocols to ensure the successful integration of the system components, avoid unnecessary hardware modifications, reduce potential cost and attain the intended functionalities. Biological-level modeling includes the study of the coupling between measuring devices and skin by taking into account parameters such as the electrical impedance of skin, as well as the study of electric current distribution among various layers (skin, fat, muscle and bone) [35]. On the other hand, kinematic-level modeling refers to the multifactorial analysis of the motor compound covering muscle/joint representation and locomotion modeling [36]. For instance, during walking, the interval between two consecutive heel strikes or stride lengths comprises a gait cycle. Simple as it might seem, gait analysis encompasses an ample number of parameters, prompting the development of various evaluation techniques. Gait parameters often include velocity, step length, foot height, gait autonomy (maximum walking duration and distance travelled), joint angles, fall events, ground reaction forces, etc. [37]. Hand gestures and finger movements require an even greater number of specifications due to the much more complex nature of upper limb maneuverability as well as hand–eye coordination versatility parameters [13]. The mathematical analysis of muscles and joints is usually conducted using differential equation models and simulation software that allows the representation of musculoskeletal structures and their behavior under various motion patterns [38]. Muscle groups to be considered must designate agonist–antagonist pairs, while proposed models account for non-optimal muscle behavior (e.g., fatigue or spasms), since recent studies indicate that such deviations from optimal operation create vastly different behavior [36]. In this regard, motor system modeling raises a very high level of complexity with an equally high computational cost in relation to the available input features for the HMI receiver control. Given that the extracted feature space extends over a very large number of multi-dimensional vectors, the incorporation of all available components compromises the efficient online motion detection [39]. Furthermore, due to the noise and inter-individual anatomical variations, only a small number of these feature vectors present high discriminative power. To strike a balance between computational complexity and prediction accuracy, recent studies try to obtain the optimal subset of discriminative features (dimensionality reduction) by employing feature selection methods [40, 41]. The application of such algorithms offers model simplification, shorter training times and over-training bias elimination, while contributing to the hardware scale minimization [40, 42]. Testing the feature selection algorithms is crucial in order to verify not only algorithmic accuracy but also performance within real-time and realistic environmental constraints [43]. Hence, parameter incorporation should be carried out progressively with increasing complexity from simple

Human–Machine Interfaces for Motor Rehabilitation

7

Fig. 2 Optimal model design for HMI implementation

movement initiation and termination to more complicated movements [44]. A general workflow of an HMI system implementation is presented in Fig. 2. Specifically, the input data are converted to feature vectors, and then pattern recognition and a feature selection processes take place in order to maximize accuracy and embed the final computational model to the HMI system. Within this scope, the incorporation of simulation variables offers feature estimation validity, while reducing timeand resource-consuming modifications that fall under the premise of extending HMI designs to more complex motion patterns.

4 Non-physiological Modalities The primary step in designing an HMI is the extraction of motion information, which can be achieved by utilizing physiological and non-physiological data. Nonphysiological modeling encompasses sensors that analyze the mechanical aspects of the human motor functions, namely parameters such as force data, joint angles (e.g., hip, knee, ankle or elbow) and limb position. In particular, force data can be acquired using piezoelectric sensors that are placed under the foot [31] (as a foot switch) in order to analyze gait parameters such as ground forces, center of mass, step length and speed [37]. These sensors can also be applied to external mechanisms such as orthotic

8

I. Kakkos et al.

devices [17], robotic prostheses/exoskeletons [31] or customized gloves in order to extract data such as grasping strength [45]. Joint angle data are typically measured using inertial sensors like gyroscopes and accelerometers [31], while additional limb position data can be acquired via marker-based or markerless motion capture systems [46], which are mainly used in training or rehabilitation protocols within a controlled environment. Other means of extracting movement data include textile sensors that have a more subtle effect to user appearance compared to the conventional electronics of the broad “wearable” category [47]. An indicative example is the use of fabric bio-impedance sensors, where the change in muscle and blood vessel volume due to current flow provides information on joint movement [48]. Moreover, the available sensor data can be collected and analyzed via mobile apps and cloud platforms in order to enhance rehabilitation monitoring and training management [49]. Generally, non-physiological rehabilitation systems employ robotic exoskeletons using inertial or force sensor data for providing movement support during rehabilitation training. As far as the upper limbs are concerned, the data recorded can correspond to elbow, forearm or wrist angles [50], while for lower limb rehabilitation knee/hip joint angles and ground forces are measured using gyroscopes and pressure sensors [51]. Similar platforms have been implemented for remote rehabilitation monitoring after surgery, allowing injury prevention and thus minimization rehabilitation setbacks [52]. Additional systems include wearable accessories such as robotic gloves using pressure sensors to measure finger flexion/extension and resistance training exoskeletons, where input parameters are acquired via rotating forearm movements [53, 54]. FES rehabilitation has also been applied within non-physiological systems, mainly based on foot switches and joint angles, while the use of common repetitive motion patterns (e.g., cycling) has additionally been employed for rehabilitation training [55]. Other FES protocols are not based on real-time measurements, but rather on stimulation adjustments by physiotherapists, who control the FES device according to the patients’ movements [34].

5 Physiological Modalities The study of the neuromuscular motion mechanisms requires the consideration of the multiple interactions engaging the brain and muscular responses under everyday motor patterns. In this regard, many recent studies have attempted to detect the prevalent measures that quantify the relationship between the distinct EMG patterns and the corresponding brain oscillatory activity during various motor processes (planning, preparation and execution) [56]. Since the use of HMI for neurorehabilitation purposes mainly aims to promote the recruitment of selected brain areas in regard to the muscular activation, the prevalent corresponding measure is corticomuscular coherence, which quantifies the relationship between the spectrums of brain and muscle signals [57]. To that end, neurological functional structures, neuronal interaction and neurophysiological motor characteristics have been thoroughly studied and

Human–Machine Interfaces for Motor Rehabilitation

9

analyzed during muscle contractions demonstrating distinct band power alterations reflecting contraction intensity [58]. In addition to corticomuscular spectral densities of specific frequency bands, event-related activity has demonstrated effective analysis on early movement brain activation [59]. Specifically, movement prediction has been achieved by analyzing brain signals within a time frame of approximately 0.5–2.0 s before motion execution [59, 60]. During that interval, the relevant cortical processes cause the manifestation of a low-frequency (0–5 Hz) negative shift in the EEG signal known as movementrelated cortical potential (MRCP) that represents movement planning and preparation. MRCP has been associated with motor imagery tasks, thus has proven useful for HMI studies and rehabilitation for patients with the inability to perform a movement [59]. Other movement-related potentials include the Bereitschaftspotential (BP) [61], which occurs around 1.0–1.5 s prior to movement onset having maximum amplitude over the primary motor cortex and the contingent negative variation (CNV), occurring within the same time frame as the BP and representing the interval between a preparation and an execution command stimulus [62]. Alternatively, movement intention phenomena can be detected through the analysis of event-related desynchronization in the alpha (8–12 Hz) and central beta (16–24 Hz) bands [60]. These manifestations usually occur during the interval prior to movement and offer a robust indicator of the time window when signal pattern alterations can be detected, usually about 2 s before voluntary real or imaginary movement [59]. Regarding the latter, motor imagery is a mental process extensively utilized in the approximation of motor neurological systems, sharing cortical activation areas with real motion execution while not being influenced by the individual’s ability for motion performance [15]. Due to the extensive research of brain signal fluctuations in regard to motion, incorporation of EEG in motor rehabilitation HMI systems has been widely adopted [33, 63]. These systems provide direct real-time interaction between the human cognitive processes and an external application, allowing direct brain–machine communication. In this direction, exoskeletons have been applied for both lower and upper limb rehabilitation, while system training has been enhanced via the use of motor imagery [64–66]. Results display both the patient’s ability to adequately control the exoskeleton as well as the feasibility of applying robotic movement support in rehabilitation protocols. Furthermore, the emergence of EEG-driven FES treatment in recent years has shown promising results in studies involving foot drop treatment and hand movement support in stroke patients [67, 68]. Other applications include EEG-controlled FES training for treating children with spastic cerebral palsy [69]. In this regard, the evidence demonstrates that the combination of FES and EEG-based technologies is more beneficial in facilitating brain activation as opposed to sole FES employment [70]. Notably, the recovery effects elicited through FES are assumed to remain active after the patient participation to the rehabilitation program [71]. Similar to the EEG-based systems, EMG-controlled exoskeletons have also been utilized for rehabilitation purposes. Surface EMG recordings serve as a means of quantifying either larger muscle group data such as grasping force and knee extension or more precise patterns such as individual finger movement, which are not easily accessible via brain activity measurements [27, 72, 73]. In that scope, the muscular

10

I. Kakkos et al.

activity can also modulate an electrical stimulation device in applications such as wrist movement rehabilitation training and foot drop treatment in motion impaired patients by causing muscle contraction to facilitate movement and regain muscle activation [74, 75]. The EMG measurements have also been used in both exoskeletons and FES systems for training results assessment, studying the target response of an unaffected limb, estimating the appropriate movement parameters, or even driving the impaired limb via contralateral control [76, 77]. At the same time, incorporation of signals of different modalities into a hybrid HMI has given rise to a number of intriguing possibilities, since it allowed the implementation of muscle-assisted therapy and extended the study of corticomuscular interactions in the control of multi-degree-of-freedom devices. Hybrid rehabilitation HMI systems combine either physiological with non-physiological measurements or different physiological modalities, namely EEG and EMG recordings into a multipleinput system with the intent of identifying multi-dimensional information and therefore providing movement support [18, 57]. As such, these systems combine EMG data with angle/force data [78, 79] or EEG and EMG physiological measurements with generalized motion analysis sensors [80, 81], providing integrated applications for movement prediction/detection, muscle stimulation or robotic exoskeleton control. In this context, the multimodal physiological real-time measurements have shown the ability to augment the control loop between brain and muscle groups, hence providing a valuable addition to rehabilitation protocols [82]. These protocols do not only facilitate motor restoration procedures, but also additional objectives such as the suppression of spasms or tremors, by defining motion thresholds that can distinguish between voluntary and non-voluntary movements [83].

6 Future Challenges Despite the large number of HMI studies, several conditions need to be met so that low-cost real-world rehabilitation HMI applications can be realized. This is to be expected, since most rehabilitation systems involve research studies and not commercial applications, thus employing large-scale devices in a controlled environment, lacking the portability required for day-to-day motion support [20, 63]. In addition, the detection and identification of EMG signals experience major issues, since pattern recognition deteriorates due to EMG signal changes as a consequence of limb position, electrode shifts, muscle fatigue and inter-subject variability [84, 85]. Another aspect to be taken into consideration is the functional expansion and reorganization of the brain concerning compromised cortical functions [86]. Convergent evidence suggests that HMI protocols as well as exercise and physical training enhance rehabilitation processes, as sensory and motor representations undergo constant plasticity. Depending on the motor impairment condition, the choice of different interventions in regard to motor task strategy training could determine the course of the prevalent neuroplasticity processes [87]. Research has also shown that the efficacy of these mechanisms depends heavily on the time frame during which

Human–Machine Interfaces for Motor Rehabilitation

11

the rehabilitation interventions are applied [8]. In this light, a generalized mechanism governing the control of a rehabilitation device does not offer the personalized design required. To address this issue, brain connectivity and network analysis methods on the multi-channel EEG recordings have illustrated the potential of successfully displaying intra-brain interactions [88, 89]. However, to the best of our knowledge, barely any research has been conducted regarding the application of functional connectivity during the performance of motor tasks. Furthermore, HMIs for rehabilitation purposes entail high degrees of customization due to signal variability, on top of the variations introduced by the different neurological disorders. Consequently, extensive user training should be performed in order for the system to adapt to every individual patient case [11]. The requisite training protocols could aid the users in getting accustomed to the devices and making the most of the functionalities during their everyday life. Finally, given the broad use of corticomuscular analysis, the resulting kinetic models and the developed tools should be further validated through open access data [90], linking the results with the findings of international publications. This validation ensures the theoretical confirmation of the identified biomarkers, providing systematic mapping and specialization among muscle–brain activities.

References 1. Health topics, (n.d.). https://www.who.int/health-topics 2. Restorative Neurology of Spinal Cord Injury. Oxford University Press, Oxford, New York (2011) 3. Hatem, S.M., Saussez, G., Della Faille, M., Prist, V., Zhang, X., Dispa, D., Bleyenheuft, Y.: Rehabilitation of motor function after stroke: a multiple systematic review focused on techniques to stimulate upper extremity recovery. Front. Hum. Neurosci. 10, 442 (2016). https:// doi.org/10.3389/fnhum.2016.00442 4. Thibaut, A., Chatelle, C., Ziegler, E., Bruno, M.-A., Laureys, S., Gosseries, O.: Spasticity after stroke: physiology, assessment and treatment. Brain Inj. 27, 1093–1105 (2013). https://doi.org/ 10.3109/02699052.2013.804202 5. Emos, M.C., Agarwal, S.: Neuroanatomy, upper motor neuron lesion. StatPearls, StatPearls Publishing, Treasure Island (FL) (2019). http://www.ncbi.nlm.nih.gov/books/NBK537305/ 6. Dworzynski, K., Ritchie, G., Fenu, E., MacDermott, K., Playford, E.D.: Rehabilitation after stroke: summary of NICE guidance. BMJ 346, f3615 (2013). https://doi.org/10.1136/bmj.f3615 7. Nas, K., Yazmalar, L., Sah, ¸ V., Aydın, A., Öne¸s, K.: Rehabilitation of spinal cord injuries. World J. Orthop. 6, 8–16 (2015). https://doi.org/10.5312/wjo.v6.i1.8 8. Cathy, S., Suzanne, A., Winston, B.: Rehabilitation is initiated early after stroke, but most motor rehabilitation trials are not. Stroke 44, 2039–2045 (2013). https://doi.org/10.1161/ STROKEAHA.113.000968 9. Kortte, K.B., Falk, L.D., Castillo, R.C., Johnson-Greene, D., Wegener, S.T.: The Hopkins rehabilitation engagement rating scale: development and psychometric properties. Arch. Phys. Med. Rehabil. 88, 877–884 (2007). https://doi.org/10.1016/j.apmr.2007.03.030 10. Feigin, V.L., Forouzanfar, M.H., Krishnamurthi, R., Mensah, G.A., Connor, M., Bennett, D.A., Moran, A.E., Sacco, R.L., Anderson, L., Truelsen, T., O’Donnell, M., Venketasubramanian, N., Barker-Collo, S., Lawes, C.M.M., Wang, W., Shinohara, Y., Witt, E., Ezzati, M., Naghavi,

12

11.

12.

13.

14.

15.

16.

17.

18. 19.

20.

21. 22. 23.

24.

25.

26.

27.

I. Kakkos et al. M., Murray, C.: Global burden of diseases, injuries, and risk factors study 2010 (GBD 2010) and the GBD stroke experts group, global and regional burden of stroke during 1990–2010: findings from the global burden of disease study 2010. Lancet Lond. Engl. 383, 245–254 (2014) Carbone, V., van der Krogt, M.M., Koopman, H.F.J.M., Verdonschot, N.: Sensitivity of subjectspecific models to hill muscle–tendon model parameters in simulations of gait. J. Biomech. 49, 1953–1960 (2016). https://doi.org/10.1016/j.jbiomech.2016.04.008 Andreoni, G., Parini, S., Maggi, L., Piccini, L., Panfili, G., Torricelli, A.: Human machine interface for healthcare and rehabilitation. In: Vaidya, S., Jain, L.C., Yoshida, H. (eds.) Advanced Computational Intelligence Paradigms in Healthcare-2, pp. 131–150. Springer, Berlin, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72375-2_7 Rudisch, J., Butler, J., Izadi, H., Zielinski, I.M., Aarts, P., Birtles, D., Green, D.: Kinematic parameters of hand movement during a disparate bimanual movement task in children with unilateral Cerebral Palsy. Hum. Mov. Sci. 46, 239–250 (2016). https://doi.org/10.1016/j.humov. 2016.01.010 Reis, P., Hebenstreit, F., Gabsteiger, F., von Tscharner, V., Lochmann, M.: Methodological aspects of EEG and body dynamics measurements during motion. Front. Hum. Neurosci. 8 (2014). https://doi.org/10.3389/fnhum.2014.00156 Munzert, J., Lorey, B., Zentgraf, K.: Cognitive motor processes: the role of motor imagery in the study of motor representations. Brain Res. Rev. 60, 306–326 (2009). https://doi.org/10. 1016/j.brainresrev.2008.12.024 Yin, Y.H., Fan, Y.J., Xu, L.D.: EMG and EPP-integrated human-machine interface between the paralyzed and rehabilitation exoskeleton. IEEE Trans. Inf. Technol. Biomed. 16, 542–549 (2012). https://doi.org/10.1109/TITB.2011.2178034 Villa-Parra, A.C., Delisle-Rodriguez, D., Souza Lima, J, Frizera-Neto, A., Bastos, T.: Knee impedance modulation to control an active orthosis using insole sensors. Sensors 17 (2017). https://doi.org/10.3390/s17122751 Pons, J.L.: Rehabilitation exoskeletal robotics. IEEE Eng. Med. Biol. Mag. 29, 57–63 (2010). https://doi.org/10.1109/MEMB.2010.936548 Bhatia, D., Bansal, G., Tewari, R.P., Shukla, K.K.: State of art: functional electrical stimulation (FES), Int. J. Biomed. Eng. Technol. 5, 77–99 (2011). https://doi.org/10.1504/ijbet.2011. 038474 Dzahir, M.A.M., Yamamoto, S.: Recent trends in lower-limb robotic rehabilitation orthosis: control scheme and strategy for pneumatic muscle actuated gait trainers. Robotics 3, 120–148 (2014). https://doi.org/10.3390/robotics3020120 Herr, H.: Exoskeletons and orthoses: classification, design challenges and future directions. J. Neuro Eng. Rehabil. 6, 21 (2009). https://doi.org/10.1186/1743-0003-6-21 Yue, Z., Zhang, X., Wang, J.: Hand rehabilitation robotics on poststroke motor recovery. Behav. Neurol. (2017). https://doi.org/10.1155/2017/3908135 Ghosh, P., Mazumder, A., Banerjee, A., Tibarewala, D.N.: Electromyography-based functional electrical stimulation (FES) in rehabilitation. Biomed. Image Anal. Min. Tech. Improv. Health Outcomes 337–355 (2016). https://doi.org/10.4018/978-1-4666-8811-7.ch016 Soekadar, S.R., Birbaumer, N., Slutzky, M.W., Cohen, L.G.: Brain-machine interfaces in neurorehabilitation of stroke. Neurobiol. Dis. 83, 172–179 (2015). https://doi.org/10.1016/j.nbd. 2014.11.025 Martin, R., Sadowsky, C., Obst, K., Meyer, B., McDonald, J.: Functional electrical stimulation in spinal cord injury: from theory to practice. Top. Spinal Cord Inj. Rehabil. 18, 28–33 (2012). https://doi.org/10.1310/sci1801-28 Takeda, K., Tanino, G., Miyasaka, H.: Review of devices used in neuromuscular electrical stimulation for stroke rehabilitation. Med. Devices Auckl. NZ. 10, 207–213 (2017). https:// doi.org/10.2147/MDER.S123464 Atzori, M., Cognolato, M., Müller, H.: Deep learning with convolutional neural networks applied to electromyography data: a resource for the classification of movements for prosthetic hands. Front. Neurorobotics 10 (2016). https://doi.org/10.3389/fnbot.2016.00009

Human–Machine Interfaces for Motor Rehabilitation

13

28. Zhang, Y., Wang, Y., Zhou, G., Jin, J., Wang, B., Wang, X., Cichocki, A.: Multi-kernel extreme learning machine for EEG classification in brain-computer interfaces. Expert Syst. Appl. 96, 302–310 (2018). https://doi.org/10.1016/j.eswa.2017.12.015 29. Latif, M.Y., Naeem, L., Hafeez, T., Raheel, A., Saeed, S.M.U., Awais, M., Alnowami, M., Anwar, S.M.: Brain computer interface based robotic arm control. In: 2017 International Smart Cities Conference ISC2, pp. 1–5 (2017). https://doi.org/10.1109/isc2.2017.8090870 30. Hsu, W.-C., Lin, L.-F., Chou, C.-W., Hsiao, Y.-T., Liu, Y.-H.: EEG classification of imaginary lower limb stepping movements based on fuzzy support vector machine with kernel-induced membership function. Int. J. Fuzzy Syst. 19, 566–579 (2017). https://doi.org/10.1007/s40815016-0259-9 31. Jung, J.-Y., Heo, W., Yang, H., Park, H.: A neural network-based gait phase classification method using sensors equipped on lower limb exoskeleton robots. Sensors 15, 27738–27759 (2015). https://doi.org/10.3390/s151127738 32. Ahangi, A., Karamnejad, M., Mohammadi, N., Ebrahimpour, R., Bagheri, N.: Multiple classifier system for EEG signal classification with application to brain–computer interfaces. Neural Comput. Appl. 23, 1319–1327 (2013). https://doi.org/10.1007/s00521-012-1074-3 33. Wolpaw, J., Wolpaw, E.W. (eds.): Brain-Computer Interfaces: Principles and Practice, 1st edn. Oxford University Press, Oxford, New York (2012) 34. Kapadia, N., Masani, K., Catharine Craven, B., Giangregorio, L.M., Hitzig, S.L., Richards, K., Popovic, M.R.: A randomized trial of functional electrical stimulation for walking in incomplete spinal cord injury: effects on walking competency. J. Spinal Cord Med. 37, 511–524 (2014). https://doi.org/10.1179/2045772314y.0000000263 35. Davies, L., Chappell, P., Melvin, T.: Modelling the effect of hydration on skin conductivity. Skin Res. Technol. Off. J. Int. Soc. Bioeng. Skin ISBS Int. Soc. Digit. Imaging Skin ISDIS Int. Soc. Skin Imaging ISSI 23, 363–368 (2017). https://doi.org/10.1111/srt.12344 36. Lynch, C.L., Graham, G.M., Popovic, M.R.: A generic model of real-world non-ideal behaviour of FES-induced muscle contractions: simulation tool. J. Neural Eng. 8, 046034 (2011). https:// doi.org/10.1088/1741-2560/8/4/046034 37. Muro-de-la-Herran, A., Garcia-Zapirain, B., Mendez-Zorrilla, A.: Gait analysis methods: an overview of wearable and non-wearable systems. Highlighting Clin. Appl. Sens. 14, 3362–3394 (2014). https://doi.org/10.3390/s140203362 38. Delp, S.L., Anderson, F.C., Arnold, A.S., Loan, P., Habib, A., John, C.T., Guendelman, E., Thelen, D.G.: OpenSim: open-source software to create and analyze dynamic simulations of movement. IEEE Trans. Biomed. Eng. 54, 1940–1950 (2007). https://doi.org/10.1109/TBME. 2007.901024 39. Verleysen, M., François, D.: The curse of dimensionality in data mining and time series prediction. In: Cabestany, J., Prieto, A., Sandoval, F. (eds.) Computational Intelligence and Bioinspired Systems, pp. 758–770. Springer, Berlin Heidelberg (2005) 40. Chandrashekar, G., Sahin, F.: A survey on feature selection methods. Comput. Electr. Eng. 40, 16–28 (2014). https://doi.org/10.1016/j.compeleceng.2013.11.024 41. Dimitrakopoulos, G.N., Kakkos, I., Thakor, N.V., Bezerianos, A., Sun, Y.: A mental fatigue index based on regression using multiband EEG features with application in simulated driving. In: Conference Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3220–3223 (2017). https://doi.org/10.1109/embc. 2017.8037542 42. Dimitrakopoulos, G.N., Kakkos, I., Dai, Z., Lim, J., deSouza, J.J., Bezerianos, A., Sun, Y.: Task-independent mental workload classification based upon common multiband EEG cortical connectivity. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 1940–1949 (2017). https://doi.org/ 10.1109/tnsre.2017.2701002 43. Dotov, D.G., Bardy, B.G., Dalla Bella, S.: The role of environmental constraints in walking: effects of steering and sharp turns on gait dynamics. Sci. Rep. 6 (2016). https://doi.org/10. 1038/srep28374 44. Klein, J., Spencer, S.J., Reinkensmeyer, D.J.: Breaking it down is better: haptic decomposition of complex movements aids in robot-assisted motor learning. IEEE Trans. Neural Syst. Rehabil. Eng. 20, 268–275 (2012). https://doi.org/10.1109/TNSRE.2012.2195202

14

I. Kakkos et al.

45. Carbonaro, N., Mura, G.D., Lorussi, F., Paradiso, R., Rossi, D.D., Tognetti, A.: Exploiting wearable goniometer technology for motion sensing gloves. IEEE J. Biomed. Health Inform. 18, 1788–1795 (2014). https://doi.org/10.1109/JBHI.2014.2324293 46. Vito, L.D., Postolache, O., Rapuano, S.: Measurements and sensors for motion tracking in motor rehabilitation. IEEE Instrum. Meas. Mag. 17, 30–38 (2014). https://doi.org/10.1109/ MIM.2014.6825386 47. Fleury, A., Sugar, M., Chau, T.: E-textiles in clinical rehabilitation: a scoping review. Electronics 4, 173–203 (2015). https://doi.org/10.3390/electronics4010173 48. Lee, B.W., Lee, C., Kim, J., Lee, M.: Optimum conductive fabric sensor sites for evaluating the status of knee joint movements using bio-impedance. Biomed. Eng. Online 10, 48 (2011). https://doi.org/10.1186/1475-925X-10-48 49. Postolache, O.: Physical rehabilitation assessment based on smart training equipment and mobile APPs. In: 2015 E-Health Bioengineering Conference (EHB), pp. 1–6 (2015). https:// doi.org/10.1109/ehb.2015.7391530 50. Song, Z., Guo, S.: Design process of exoskeleton rehabilitation device and implementation of bilateral upper limb motor movement. J. Med. Biol. Eng. 32, 323–330 (2011). https://doi.org/ 10.5405/jmbe.987 51. Zhang, X., Xiang, Z., Lin, Q., Zhou, Q.: The design and development of a lower limbs rehabilitation exoskeleton suit. In: 2013 ICME International Conference on Complex Medical Engineering, pp. 307–312 (2013). https://doi.org/10.1109/iccme.2013.6548260 52. Rajanna, V., Vo, P., Barth, J., Mjelde, M., Grey, T., Oduola, C., Hammond, T.: KinoHaptics: an automated, wearable, haptic assisted, physiotherapeutic system for post-surgery rehabilitation and self-care. J. Med. Syst. 40, 60 (2015). https://doi.org/10.1007/s10916-015-0391-3 53. Polygerinos, P., Wang, Z., Galloway, K.C., Wood, R.J., Walsh, C.J.: Soft robotic glove for combined assistance and at-home rehabilitation. Robot. Auton. Syst. 73, 135–143 (2015). https://doi.org/10.1016/j.robot.2014.08.014 54. Song, Z., Guo, S., Pang, M., Zhang, S., Xiao, N., Gao, B., Shi, L.: Implementation of resistance training using an upper-limb exoskeleton rehabilitation device for elbow joint. J. Med. Biol. Eng. 34, 188–196 (2014). https://doi.org/10.5405/jmbe.1337 55. Miura, N., Watanabe, T., Sugimoto, S., Seki, K., Kanai, H.: Fuzzy FES controller using cycle-tocycle control for repetitive movement training in motor rehabilitation. Experimental tests with wireless system. J. Med. Eng. Technol. 35, 314–321 (2011). https://doi.org/10.3109/03091902. 2011.591480 56. Kim, B., Kim, L., Kim, Y.-H., Yoo, S.K.: Cross-association analysis of EEG and EMG signals according to movement intention state. Cogn. Syst. Res. 44, 1–9 (2017). https://doi.org/10. 1016/j.cogsys.2017.02.001 57. Enders, H., Nigg, B.M.: Measuring human locomotor control using EMG and EEG: current knowledge, limitations and future considerations. Eur. J. Sport Sci. 16, 416–426 (2016). https:// doi.org/10.1080/17461391.2015.1068869 58. Chai, J., Chen, G., Thangavel, P., Dimitrakopoulos, G.N., Kakkos, I., Sun, Y., Dai, Z., Yu, H., Thakor, N., Bezerianos, A., Li, J.: Identification of gait-related brain activity using electroencephalographic signals. In: 2017 8th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 548–551 (2017). https://doi.org/10.1109/ner.2017.8008410 59. Shakeel, A., Navid, M.S., Anwar, M.N., Mazhar, S., Jochumsen, M., Niazi, I.K.: A review of techniques for detection of movement intention using movement-related cortical potentials. Comput. Math. Methods Med. (2015). https://doi.org/10.1155/2015/346217 60. Ahmadian, P., Cagnoni, S., Ascari, L.: How capable is non-invasive EEG data of predicting the next movement? A mini review. Front. Hum. Neurosci. 7, 124 (2013). https://doi.org/10. 3389/fnhum.2013.00124 61. Schultze-Kraft, M., Birman, D., Rusconi, M., Allefeld, C., Görgen, K., Dähne, S., Blankertz, B., Haynes, J.-D.: The point of no return in vetoing self-initiated movements. Proc. Natl. Acad. Sci. U. S. A. 113, 1080–1085 (2016). https://doi.org/10.1073/pnas.1513569112 62. Kato, Y.X., Yonemura, T., Samejima, K., Maeda, T., Ando, H.: Development of a BCI master switch based on single-trial detection of contingent negative variation related potentials.

Human–Machine Interfaces for Motor Rehabilitation

63.

64.

65.

66.

67.

68.

69.

70.

71.

72.

73.

74.

75.

15

In: Conference Proceedings of Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 4629–4632 (2011). https://doi.org/10.1109/iembs.2011. 6091146 Remsik, A., Young, B., Vermilyea, R., Kiekhoefer, L., Abrams, J., Evander Elmore, S., Schultz, P., Nair, V., Edwards, D., Williams, J., Prabhakaran, V.: A review of the progression and future implications of brain-computer interface therapies for restoration of distal upper extremity motor function after stroke. Expert Rev. Med. Devices. 13, 445–454 (2016). https://doi.org/10. 1080/17434440.2016.1174572 Ang, K.K., Chua, K.S.G., Phua, K.S., Wang, C., Chin, Z.Y., Kuah, C.W.K., Low, W., Guan, C.: A randomized controlled trial of EEG-based motor imagery brain-computer interface robotic rehabilitation for stroke. Clin. EEG Neurosci. 46, 310–320 (2015). https://doi.org/10.1177/ 1550059414522229 López-Larraz, E., Trincado-Alonso, F., Rajasekaran, V., Pérez-Nombela, S., Del-Ama, A.J., Aranda, J., Minguez, J., Gil-Agudo, A., Montesano, L.: Control of an ambulatory exoskeleton with a brain-machine interface for spinal cord injury gait rehabilitation. Front. Neurosci. 10, 359 (2016). https://doi.org/10.3389/fnins.2016.00359 Bhagat, N.A., Venkatakrishnan, A., Abibullaev, B., Artz, E.J., Yozbatiran, N., Blank, A.A., French, J., Karmonik, C., Grossman, R.G., O’Malley, M.K., Francisco, G.E., Contreras-Vidal, J.L.: Design and optimization of an EEG-based brain machine interface (BMI) to an upperlimb exoskeleton for stroke survivors. Front. Neurosci. 10, 122 (2016). https://doi.org/10.3389/ fnins.2016.00122 Jure, F.A., Carrere, L.C., Gentiletti, G.G., Tabernig, C.B.: BCI-FES system for neurorehabilitation of stroke patients. J. Phys: Conf. Ser. 705, 012058 (2016). https://doi.org/10. 1088/1742-6596/705/1/012058 Do, A.H., Wang, P.T., King, C.E., Abiri, A., Nenadic, Z.: Brain-computer interface controlled functional electrical stimulation system for ankle movement. J. Neuro. Eng. Rehabil. 8, 49 (2011). https://doi.org/10.1186/1743-0003-8-49 Kim, T.-W., Lee, B.-H.: Clinical usefulness of brain-computer interface-controlled functional electrical stimulation for improving brain activity in children with spastic cerebral palsy: a pilot randomized controlled trial. J. Phys. Ther. Sci. 28, 2491–2494 (2016). https://doi.org/10.1589/ jpts.28.2491 Chung, E., Kim, J.-H., Park, D.-S., Lee, B.-H.: Effects of brain-computer interface-based functional electrical stimulation on brain activation in stroke patients: a pilot randomized controlled trial. J. Phys. Ther. Sci. 27, 559–562 (2015). https://doi.org/10.1589/jpts.27.559 Biasiucci, A., Leeb, R., Iturrate, I., Perdikis, S., Al-Khodairy, A., Corbet, T., Schnider, A., Schmidlin, T., Zhang, H., Bassolino, M., Viceic, D., Vuadens, P., Guggisberg, A.G., Millán, J.R.: Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. Nat. Commun. 9, 2421 (2018). https://doi.org/10.1038/s41467-018-04673-z Aach, M., Cruciger, O., Sczesny-Kaiser, M., Höffken, O., Meindl, R.C., Tegenthoff, M., Schwenkreis, P., Sankai, Y., Schildhauer, T.A.: Voluntary driven exoskeleton as a new tool for rehabilitation in chronic spinal cord injury: a pilot study, Spine. J. Off. J. North Am. Spine Soc. 14, 2847–2853 (2014). https://doi.org/10.1016/j.spinee.2014.03.042 Ockenfeld, C., Tong, R.K.Y., Susanto, E.A., Ho, S.-K., Hu, X.: Fine finger motor skill training with exoskeleton robotic hand in chronic stroke: stroke rehabilitation. IEEE Int. Conf. Rehabil. Robot. Proc. 2013, 6650392 (2013). https://doi.org/10.1109/ICORR.2013.6650392 Chen, M., Wu, B., Lou, X., Zhao, T., Li, J., Xu, Z., Hu, X., Zheng, X.: A self-adaptive footdrop corrector using functional electrical stimulation (FES) modulated by tibialis anterior electromyography (EMG) dataset. Med. Eng. Phys. 35, 195–204 (2013). https://doi.org/10. 1016/j.medengphy.2012.04.016 Shindo, K., Fujiwara, T., Hara, J., Oba, H., Hotta, F., Tsuji, T., Hase, K., Liu, M.: Effectiveness of hybrid assistive neuromuscular dynamic stimulation therapy in patients with subacute stroke: a randomized controlled pilot trial. Neurorehabil. Neural Repair 25, 830–837 (2011). https:// doi.org/10.1177/1545968311408917

16

I. Kakkos et al.

76. Knutson, J.S., Harley, M.Y., Hisel, T.Z., Hogan, S.D., Maloney, M.M., Chae, J.: Contralaterally controlled functional electrical stimulation for upper extremity hemiplegia: an early-phase randomized clinical trial in subacute stroke patients. Neurorehabil. Neural Repair 26, 239–246 (2012). https://doi.org/10.1177/1545968311419301 77. Loconsole, C., Leonardis, D., Barsotti, M., Solazzi, M., Frisoli, A., Bergamasco, M., Troncossi, M., Foumashi, M.M., Mazzotti, C., Castelli, V.P.: An emg-based robotic hand exoskeleton for bilateral training of grasp. In: 2013 World Haptics Conference (WHC), pp. 537–542 (2013). https://doi.org/10.1109/whc.2013.6548465 78. Popovi´c Maneski, L., Topalovi´c, I., Joviˇci´c, N., Dedijer, S., Konstantinovi´c, L., Popovi´c, D.B.: Stimulation map for control of functional grasp based on multi-channel EMG recordings. Med. Eng. Phys. 38, 1251–1259 (2016). https://doi.org/10.1016/j.medengphy.2016.06.004 79. Liu, L., Chen, X., Lu, Z., Cao, S., Wu, D., Zhang, X.: Development of an EMG-ACC-based upper limb rehabilitation training system. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 244–253 (2017). https://doi.org/10.1109/TNSRE.2016.2560906 80. Rincon, A.L., Yamasaki, H., Shimoda, S.: Design of a video game for rehabilitation using motion capture, EMG analysis and virtual reality. In: 2016 International Conference on Electronics, Communications and Computers CONIELECOMP, pp. 198–204 (2016). https://doi. org/10.1109/conielecomp.2016.7438575 81. Sarasola-Sanz, A., Irastorza-Landa, N., Lopez-Larraz, E., Bibian, C., Helmhold, F., Broetz, D., Birbaumer, N., Ramos-Murguialday, A.: A hybrid brain-machine interface based on EEG and EMG activity for the motor rehabilitation of stroke patients. IEEE Int. Conf. Rehabil. Robot. Proc. 2017, 895–900 (2017). https://doi.org/10.1109/ICORR.2017.8009362 82. Rayegani, S.M., Raeissadat, S.A., Sedighipour, L., Rezazadeh, I.M., Bahrami, M.H., Eliaspour, D., Khosrawi, S.: Effect of neurofeedback and Electromyographic-biofeedback therapy on improving hand function in stroke patients. Top. Stroke Rehabil. 21, 137–151 (2014). https:// doi.org/10.1310/tsr2102-137 83. Grimaldi, G., Manto, M., Jdaoudi, Y.: Quality parameters for a multimodal EEG/EMG/kinematic brain-computer interface (BCI) aiming to suppress neurological tremor in upper limbs. F1000Research 2, 282 (2013). https://doi.org/10.12688/f1000research. 2-282.v2 84. Khushaba, R.N., Takruri, M., Miro, J.V., Kodagoda, S.: Towards limb position invariant myoelectric pattern recognition using time-dependent spectral features. Neural Netw. 55, 42–58 (2014). https://doi.org/10.1016/j.neunet.2014.03.010 85. Kyranou, I., Vijayakumar, S., Erden, M.S.: Causes of performance degradation in non-invasive electromyographic pattern recognition in upper limb prostheses. Front. Neurorobotics. 12 (2018). https://doi.org/10.3389/fnbot.2018.00058 86. Dimyan, M.A., Cohen, L.G.: Neuroplasticity in the context of motor rehabilitation after stroke. Nat. Rev. Neurol. 7, 76–85 (2011). https://doi.org/10.1038/nrneurol.2010.200 87. Levin, M.F., Kleim, J.A., Wolf, S.L.: What do motor “recovery” and “compensation” mean in patients following stroke? Neurorehabil. Neural Repair 23, 313–319 (2009). https://doi.org/ 10.1177/1545968308328727 88. Kakkos, I., Dimitrakopoulos, G.N., Gao, L., Zhang, Y., Qi, P., Matsopoulos, G.K., Thakor, N., Bezerianos, A., Sun, Y.: Mental workload drives different reorganizations of functional cortical connectivity between 2D and 3D simulated flight experiments. IEEE Trans. Neural Syst. Rehabil. Eng. Publ. IEEE Eng. Med. Biol. Soc. 27, 1704–1713 (2019). https://doi.org/ 10.1109/tnsre.2019.2930082 89. Dimitrakopoulos, G.N., Kakkos, I., Dai, Z., Wang, H., Sgarbas, K., Thakor, N., Bezerianos, A., Sun, Y.: Functional connectivity analysis of mental fatigue reveals different network topological alterations between driving and vigilance tasks. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 740–749 (2018). https://doi.org/10.1109/tnsre.2018.2791936 90. EEG motor movement/imagery dataset, (n.d.). https://www.physionet.org/pn4/eegmmidb/

Passive Emotion Recognition Using Smartphone Sensing Data I. P. Kalogirou, A. Kallipolitis and Ilias Maglogiannis

Abstract Awareness of the emotion in human–computer interaction is a challenging task when building human-centered computing systems. Emotion is a complex state of mind that is affected by external events, physiological changes and, generally, human relationships. Researchers suggest various methods of measuring human emotions through the analysis of physiological signals, facial expressions, voice, etc. This chapter presents a system for recognizing the emotions of a smartphone user through the collection and analysis of data generated from different types of sensors on the device. Data collection is carried out by an application installed on the participants’ smartphone provided that the smartphone remains in their pocket throughout the experiments. The collected data are processed and utilized to train different classifiers (decision trees, naïve Bayes and k-nearest neighbors). Emotions are classified in the following six categories: happiness, neutral, sadness, disgust, fear, surprise. Initial results show that the system classifies user’s emotions with 82.83% accuracy. The proposed system applied to a smartphone demonstrates the feasibility of an emotion recognition approach through a user-friendly scenario for users’ activity recognition. Keywords Affective computing · Machine learning · Emotion recognition · Smartphone sensors

1 Introduction The improved processing power of smartphones along with the integration of sophisticated sensors has ignited researchers’ interest toward developing systems that detect the user’s emotional state. Smartphones are not equipped with sensors that can detect emotion, but the scientific progress in the field of artificial intelligence reveals new perspectives for the introduction of emotions into the world of mobile technology. I. P. Kalogirou · A. Kallipolitis · I. Maglogiannis (B) Department of Digital Systems, University of Piraeus, Grigoriou Lampraki 126 Piraeus, 18532 Piraeus, Greece e-mail: [email protected] © Springer-Verlag GmbH Germany, part of Springer Nature 2020 I. Maglogiannis et al. (eds.), Advanced Computational Intelligence in Healthcare-7, Studies in Computational Intelligence 891, https://doi.org/10.1007/978-3-662-61114-2_2

17

18

I. P. Kalogirou et al.

The dedicated field of computer science that deals with emotion recognition is called affective computing. Affective computing refers to recognizing human emotions based on several types of data collected by the user whose emotional state is assessed [1]. The main goal is the development of systems that recognize, interpret, process and simulate human emotions. A representative example of affective computing technologies is the detection of a user’s emotional state (via sensors, microphone or camera) and the appropriate response by performing specific functions, such as recommending specific content that fit the user’s emotion. In the case of smartphones, the device must interpret the user’s emotional state and adapt its behavior to this state. Predicting user’s emotions from smartphone can be useful for other systems that interact with humans as well. Such systems, for example, are recommendation systems that recommend song, videos or advertisements matched with user’s emotional state. A smartphone gathers information about the user’s emotion from different sources. Facial expressions, posture, gestures, speech and typing rate reflect changes in the user’s emotional state and can be detected and interpreted. For example, a smartphone’s camera may capture user’s images which are, in turn, processed and inserted into predictive models to extract meaningful information. In our approach, we focus on the scenario of a smartphone application to collect data for the classification of user emotional status. This approach ensures that users will not be burdened with the utilization of additional devices. The system is divided into five modules. The first module is represented by the smartphone which is a data aggregator. It collects data from internal and external sensors. These sensors gather data like degree of movement of user and location. Google Fit is an application which helps to achieve this purpose. Weather conditions received from software that is installed in device and face images collected from smartphone’s camera are additional sources of information. The second module of the system is the Microsoft Emotion API (Microsoft Cognitive Services available online at https:// azure.microsoft.com/en-us/services/cognitive-services/), which analyzes the images and detects the prevalent emotion. The third module of the system is the data fusion module, where data from Google Fit and Emotion API are mixed to form a fused dataset. The fourth module is the emotion recognition module where the collected data are categorized into six different emotions by means of a classifier. The final module is the emotion diary module through which the classification results are stored in the smartphone for future use. Figure 1 depicts the general workflow of the system. The remainder of this chapter is structured in five more sections, as follows: Sect. 2 presents the related work, while Sect. 3 describes the proposed emotion analysis system methodology. Section 4 reports the experiments conducted and the corresponding results. Finally, Sect. 5 summarizes this work and Sect. 6 presents interesting aspects of future work.

Passive Emotion Recognition Using Smartphone Sensing Data

19

Fig. 1 General workflow of the proposed system

2 Background Information and Related Work 2.1 Emotions In order to define emotions, three closely related terms are clarified: core affect, emotion and mood. The term core affect is defined as a “neurophysiological state consciously accessible as a simple primitive non-reflective feeling most evident in mood and emotion but always available to consciousness” [2]. The term emotion is defined by Russell and Feldman Barrett [3] as a “complex set of interrelated sub-events concerned with a specific object.” Emotions are intense experiences for someone or something [4]. In addition, mood is an experience that tends to be less intense than emotion due to the absence of a certain stimulus from the environment [5]. According to Frijda [6], mood is “the appropriate designation for affective states that are about nothing specific or about everything—about the world in general.” For example, emotions are reactions to a person (e.g., we feel pleasure when we see a friend) or to an event (e.g., to feel angry with a rude customer). We show our emotions when we are “happy about something, angry with someone or feel afraid of something.” Moods, by contrast, are not usually directed at a person or event. However, emotions can turn into moods when we lose focus from the event or object that caused the emotion. Also, unlike moods, emotions have the tendency of revealing themselves with facial expressions (anger, disgust). The categorization of emotions in distinct categories has been studied by many researchers. Ekman [7] defines anger, disgust, enjoyment, fear, sadness and surprise

20

I. P. Kalogirou et al.

Fig. 2 Emotions by Ekman [7]

anger

disgust

enjoyment

fear

sadness

surprise

Fig. 3 Connection between emotions by Lövheim. Source https://en.wikiversity. org/wiki/Motivation_and_ emotion/Book/2015/L% C3%B6vheim_cube_of_ emotion

as the basic emotions (Fig. 2). Another list of emotions was defined by Tomkins [8], consisting of eight emotions: interest, joy, surprise, distress, fear, shame, anger and disgust. The Tomkins’ list is the basis of a more recent survey on the connections between emotions by Lövheim (Fig. 3) [9]. A model of feel-based dimensions was proposed by Russell [10]. It is called “A Circumplex model of affect” and seeks to model the different kinds of emotions in a two-dimensional graph (Fig. 4). These two dimensions have shown to carry the majority of information related to the perceived difference between the different emotions.

2.2 Related Work The topic of affective computing has occupied several researchers in the last years. Many developed systems analyze human physiological signals, typing patterns or

Passive Emotion Recognition Using Smartphone Sensing Data

21

Fig. 4 “A Circumplex model of affect” by Russell [10]. Source https://affectivecomputingblog. wordpress.com/category/using-emotional-data-to-improve-learning-in-pervasive-learningenvironment/

Fig. 5 Proposed workflow for the classification of emotions

facial expressions. K. H. Kim et al. developed a novel recognition system based on the processing of physiological signals and showed that emotion recognition is feasible with a 78,43% and 61,76% accuracy for three and four emotions, respectively [11]. The system consisted of characteristic waveform detection, feature extraction

22

I. P. Kalogirou et al.

and emotion classification by means of a support vector machine. Epp et al. developed an emotion recognition system by analyzing the rhythm of users’ typing patterns on a standard keyboard [12]. Various keystrokes of participants were gathered to build a C4.5 decision tree classifier for 15 emotional states. They modeled six emotional states for confidence, hesitance, nervousness, relaxation, sadness and tiredness with accuracy ranging from 77.4 to 87.8%. In addition, their results show promise for anger and excitement, with an accuracy of 84%. J. N. Bailenson et al. presented automated, real-time models built with machine learning algorithms which use videotapes of subjects’ faces in conjunction with physiological measurements to predict rated emotion [13]. A support vector machine classifier with a linear kernel and a LogitBoost based on weak classifiers methodology was applied in an approach which was unique because of the unusually rich dataset. The level of detail in both input and output provided the opportunity for a number of important advances in their learning algorithms. R. Likamwa et al. presented a first of its kind smartphone software system, MoodScope, which infers the mood of its user based on how the smartphone is used [14]. Linear regression was performed for each dimension of mood without making any changes to the data. Leave-One-Out-Cross-Validation (LOOCV) was utilized for the evaluation of the algorithm. The accuracy reached 73%. To improve the accuracy of regression, sequential forward selection (SFS) was utilized. A personal model was, consequently, created which performed a regression for each of the user characteristics. By the combination of advantages of the last two models into a hybrid one (high precision and no training), the accuracy reached 75%. A. Kallipolitis et al. and A. Menychtas et al. presented different approaches to perform emotion analysis of patients during video teleconferences with medical personnel. Both approaches achieved classification of emotional status based on features extracted from facial images. In [15], handcrafted robust features were extracted based on the bag of visual words technique achieving 87,5% accuracy, whereas in [16] deep learning features were utilized for the same purpose. Significant to mention in both cases is the fact that emotion analysis takes place in real time.

3 Methodology In order to determine the current emotional state of the users, a machine learning approach has been adopted, consisting of the following steps: • • • •

Data collection Data analysis Data fusion Classification.

The first step which is data collection includes the gathering of various types of information through the native sensors of the smartphone. The types of information are depicted in Table 1 and mainly refer to activity, location and audiovisual content produced by the user. The second step involves the preprocessing and analysis of

Passive Emotion Recognition Using Smartphone Sensing Data

23

Table 1 Features Activity

The calculation of the value derives from the combination of three factors. These factors are recorded for every individual by the Google Fit application. The first factor is the distance that an individual covers in one day. The second factor is the number of steps walked. The third factor is the type of activity which is declared by the participant

Location

The location data of the participants are provided by Google Maps. A specific value is assigned to the location where the participant spends most of his time between 10 a.m. and 9 p.m. Figure 6 shows user’s location

BMI

Weight and height recording are recorded through the Google Fit application

Calories intake

The number of calories that participants consume

Calories burned

The burned calories are calculated using the Google Fit application

Water

The amount of water that participants consume

Weather

Weather conditions as defined by the Google weather

Emotion

The result of Microsoft Emotion API process

the collected data. Activity labels extracted from related sensors are categorized into three types (low, medium and high) according to certain thresholds defined through experimentation. Location labels (home, out) are assigned according to data collected by the Google Maps application (Fig. 6). Other data such as calories burned, weather, water intake and body mass index (BMI) are registered as is, whereas user’s photograph, audio or video is analyzed by an emotion detection application and categorized into six different emotions. The above-mentioned information is fused in one training set, and a predictive model based on a decision tree classifier digests the smartphone’s preprocessed and analyzed data. The training set retrieves the ground truth regarding the emotion label from the Microsoft Emotion API. Once the predictive model is created, unlabeled instances from the system can be classified through the model without the use of the Microsoft API. Figure 5 shows the general workflow for classifying emotions through the proposed system.

3.1 Microsoft Emotion API It is a Windows Presentation Foundation (WPF) application implemented in C # code [17–19]. The Microsoft Emotion API takes an image as input and returns results from a set of emotions for each face of the image, as well as a bounding frame for the faces of the image. The algorithm analyzes and picks out the emotions that are displayed in the face. The emotions detected by the API using the facial attributes are happiness, neutral, sadness, disgust, fear and surprise. We choose the best rated emotion of each image and save it in a database. Then, we use it as the value of feature “emotion” (see Table 1). Figure 7 presents the user interface of the Microsoft application.

24

I. P. Kalogirou et al.

Fig. 6 User’s location via Google Maps

3.2 Google Fit Google Fit [20] is an open ecosystem that allows developers to upload fitness data to a central repository where users can access their data from different devices and applications in one location. The data are exported by Google Fit in .txt files. The files consist of location, weight, height and calories information as presented in Table 1 along with the remaining collected data.

Passive Emotion Recognition Using Smartphone Sensing Data

25

Fig. 7 Microsoft Emotion application

4 Experiments 4.1 Data Collection The number of people who participated in the data collection was nine in total, aged from 25 to 35 years old. Provision of personal data took place after explanation of the collection’s process and participants’ consent. The process lasted seven days and a total of 63 entries were collected. Basic guidelines for the participants included installation of the Google Fit application and basic adjustments concerning their daily activities. For example, if participants followed a specific type of exercise like cycling, running, they were prompt to register the information in the designated field of the application. Data derive from five sources. The first source is the Google Fit application, from which data such as the participant’s activity, weight, position and calories burned during the day are extracted. The second source is an in-house application where participants recorded the amount of water consumption and intake calories. The third source is the Google weather application for weather data. The fourth source is the Google Maps application for the location data. Essential to the provision of ground truth by the Microsoft Emotion API is the facial image captured by the camera

26

I. P. Kalogirou et al.

Fig. 8 Decision tree for emotion classification

of the participant’s smartphone at the end of the day (fifth source). Moreover, the RapidMiner machine learning tool was employed for the analysis and classification of data [21].

4.2 Experimental Results For the purpose of evaluating the performance of the proposed system, certain metrics such as classification accuracy and the error rate are utilized. The accuracy is calculated by the percentage of correct predictions in relation to the total number of examples. Another useful metric when evaluating classification schemes is the Cohen’s Kappa coefficient that compares the observed accuracy to the expected accuracy [22]. The results of three different classifiers used to identify the user’s emotional state are presented. The accuracy of decision trees compared to the other two classifiers is the highest and reaches 82.38%, the classification error is 17.62%, and the Kappa coefficient was 0.737. Moreover, it is classes of sadness, disgust and surprise which are better identified than the other three. Figure 8 illustrates the decision tree for our data. The corresponding results are shown in Table 2.

5 Conclusion In this work, we proposed an emotion recognition approach for affective computing on mobile devices. The methodology is based on various real-world data and information captured by smartphones equipped with diverse built-in sensors. Moreover, a built-in application was utilized to gather users’ personal photographs and send them to the Microsoft Emotion Analysis API. Emotions are categorized into six emotional

Passive Emotion Recognition Using Smartphone Sensing Data Table 2 Results

27

Decision trees

Naïve Bayes

k-NN

Accuracy (%)

82.38

79.05

61.90

Classification error (%)

17.62

20.95

38.10

Kappa coefficient

0.737

0.687

0.442

Absolute error

0.226

0.290

0.381

Root mean square error

0.357

0.417

0.579

Correlation

0.758

0.858

0.331

states, happiness, neutral, sadness, disgust, fear, surprise. Predictive models based on decision trees, naïve Bayes and k-NN classifiers were built, and the best classification accuracy was achieved by decision trees and it equals 82.38%. It should be noted, however, that the dataset size is limited. There are also additional factors that affect the classification results. Firstly, the participants selected for data collection had similar daily activities and lacked heterogeneity. This resulted in most data being similar and the classification accuracy being high. Secondly, the number of participants was small. Sharing of personal data and daily activities information is not easy for the participants, even though all data were anonymized. Despite the improvements to be made and the limitations of the dataset, the results show that a person’s emotional state is reflected in the movements recorded by the smartphone when walking or exercising. Furthermore, the proposed methodology does not require user intervention and does not require additional equipment nor elaborated interaction. Therefore, the emotional sensing method based on a daily routine recording application can lead to the creation of a more sophisticated system for non-invasive mobile emotion recognition applications.

6 Future Work Dealing with small amount of data, as mentioned, causes lack of generalization for the predictive model, which, in turn, leads to poor classification performance data for individuals outside the group of participants. Therefore, future work should be directed to the provision of a sufficient number of participants through techniques that enhance privacy. By observing the data, further explanation of the data collection process to the participants is required. However, this process should be performed to such an extent that does not eliminate the idea of data collection from a natural environment, thus resulting in lower quality data. We also intend to examine as future work fuzzy classifiers, since the emotional state is characterized by a vector where all basic emotions contribute and not just the dominant one.

28

I. P. Kalogirou et al.

References 1. Picard, R.W.: Affective Computing. The MIT press, Cambridge (1997) 2. Russell, J.A., Barrett, F.L.: Core affect. In: Sander, D., Scherer, K.R. (eds.) The Oxford Companion to Emotion and the Affective Sciences, p. 104. Oxford University Press, New York (2009) 3. Russell, J. A, Barrett, L. F. (1999). Core affect, prototypical emotional episodes, and other things called Emotion: Dissecting the elephant. J. Pers. Soc. Psychol. 76, 805 4. Frijda, N.H.: Moods, emotion episodes and emotions. In: Lewis, M., Haviland, J.M. (eds.) Handbook of Emotions. Guilford Press, Heidelberg (1993) 5. Weiss, H. M., Cropanzano, R.: Affective events theory: A theoretical discussion of the structure, causes and consequences of affective experiences at work, vol. 18. In: Staw, B. M., Cummings, L. L. (eds.) Research in Organizational Behavior, CT, JAI Press, Greenwich (1996) 6. Frijda, N.H.: Mood. In: Sander, D., Scherer, K.R. (eds.) The Oxford Companion to Emotion and the Affective Sciences, pp. 258–259. Oxford University Press, New York (2009) 7. Ekman, P., Davidson, R.J.: The Nature of Emotion: Fundamental Questions. Oxford University Press, USA (1994) 8. Tomkins, S. S.: Affect theory. In: Scherer, K. R., Ekman, P. (eds.) Approaches to Emotion (1984) 9. Lövheim, H.: A new three-dimensional model for emotions and monoamine neurotransmitters. Med. Hypotheses 78(2), 341–348 (2011) 10. Posner, J., Russell, A.J., Peterson, B.: The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(3), 715–734 (2005) 11. Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Med. Biol. Eng. Compu. 42(3), 419–427 (2004) 12. Epp, C., Lippold, M., Mandryk, R.: Identifying emotional states using keystroke dynamics. Proceedings Conference on Human Factors in Computing Systems, Canada, Vancouver, pp. 715–724 (2011) 13. Bailenson, J.N., Pontikakis, E., Mauss, I., Gross, J.J., Jabon, M., Hutcherson, C., Nass, C., Oliver, J.: Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int. J. Hum Comput Stud. 66(5), 303–317 (2008) 14. Likamwa, R., Liu, Y., Lane, N. D., Zhong, L.: MoodScope: Building a mood sensor from smartphone usage patterns. MobiSys 2013—Proceedings of the 11th Annual International Conference on Mobile Systems, Applications and Services (2013) 15. Kallipolitis, A., Galliakis, M., Menychtas, A., Maglogiannis, I.: Emotion analysis in hospital bedside infotainment platforms using speeded up robust features, vol. 559. In: MacIntyre, J., Maglogiannis, I., Iliadis, L., Pimenidis, E. (eds.) Artificial Intelligence Applications and Innovations. AIAI 2019. IFIP Advances in Information and Communication Technology, Springer, Cham (2019) 16. Menychtas, A., Galliakis, M., Tsanakas, P., Maglogiannis, I.: Real time Integration of Emotion Analysis into Healthcare Platforms, EMBC (2019) 17. https://docs.microsoft.com/en-us/azure/cognitive-services/emotion/home 18. https://github.com/Microsoft/Cognitive-Emotion-Windows 19. https://blogs.microsoft.com/ai/microsofts-project-oxford-helps-developers-build-moreintelligent-apps/ 20. https://developers.google.com/fit/overview 21. Hofmann, M., Kilnkenberg, R.: RapidMiner: Data Mining and Business Application Use Cases. Taylor & Francis, CRS Press (2013) 22. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960)

Intelligent and Immersive Visual Analytics of Health Data Zhonglin Qu, Chng Wei Lau, Daniel R. Catchpoole, Simeon Simoff and Quang Vinh Nguyen

Abstract Massive amounts of health data have been created together with the advent of computer technologies and next generation sequencing technologies. Analytical techniques can significantly aid in the processing, integration and interpretation of the complex data. Visual analytics field has been rapidly evolving together with the advancement in automated analysis methods such as data mining, machine learning and statistics, visualization, and immersive technologies. Although automated analysis processes greatly support the decision making, conservative domains such as medicine, banking, and insurance need trusts on machine learning models. Explainable artificial intelligence could open the black boxes of the machine learning models to improve the trusts for decision makers. Immersive technologies allow the users to engage naturally with the blended reality in where they can look the information in different angles in addition to traditional screens. This chapter reviews and discusses the intelligent visualization, artificial intelligence and immersive technologies in health domain. We also illustrate the ideas with various case studies in genomic data visual analytics.

Z. Qu · C. W. Lau School of Computer, Data and Mathematical Sciences, Western Sydney University, Sydney, Australia D. R. Catchpoole Tumour Bank, Children’s Cancer Research Unit, Kids Research, Children’s Hospital at Westmead, Westmead, Australia Discipline of Paediatrics and Child Health, Faculty of Medicine, University of Sydney, Sydney, Australia Faculty of Information Technology, University of Technology Sydney, Sydney, Australia S. Simoff · Q. V. Nguyen (B) MARCS Institute for Brain, Behaviour and Development, School of Computer, Data and Mathematical Sciences, Western Sydney University, Sydney, Australia e-mail: [email protected] © Springer-Verlag GmbH Germany, part of Springer Nature 2020 I. Maglogiannis et al. (eds.), Advanced Computational Intelligence in Healthcare-7, Studies in Computational Intelligence 891, https://doi.org/10.1007/978-3-662-61114-2_3

29

30

Z. Qu et al.

1 Introduction Visualization, a graphical representation of abstract data, is an important part of cognitive systems which can provide the highest bandwidth channel from the computer to the human [9]. The visualization can be functioned as a cognitive tool providing ability to comprehend vast amounts of data, including allowing the perception of emergent properties that are not anticipated, enabling problems with the data to become immediately apparent, facilitating understanding of both large-scale and small-scale features of the data, and facilitating hypothesis formation [20, 22, 64]. Recently, artificial intelligence (AI) and immersive technologies have been used in visualizations and visual analytics process in several fields, which are becoming a new trend in the genomic visualization evolution [48]. AI has already been part of our everyday life and has been considered as one of the keys to our civilization’s brightest future [35]. AI could boost the next generation of visualization, namely intelligent visualization. The objective of intelligent visualization can be explained as “give everybody the right information at the right time and in the right way” [38]. Intelligent visualization assists a human user to handle tedious or repetitive tasks by learning from previous sessions and input data. Intelligent visualization may also combine machine learning algorithms to make highlevel, goal-oriented decisions, which makes data visualization technology directly accessible to a wide range of application scientists [15, 65]. Intelligent health data visualization can be used to find the relationship between genomic data and diseases and aid in the process of targeted and personalized therapy [41]. Immersive visualizations using virtual reality (VR) and augmented reality (AR) technologies have gained popularity in the medical community. It is mainly used for diseases’ diagnostics, volume rendering and modeling, patient care, and treatment. Immersive visualization improves the diagnostics and volume rendering in the medical field [26]. The adaptation of the immersive technologies in the medical field could improve the accuracy of the diagnosis, reduce the error cost, and provide better patient care management. VR and AR techniques immerse users into a digitally created space and simulate movement in a three -dimensional space to increase the bandwidth of data available to our brains [28, 42, 57]. The immersive tools could allow the users to interact with the data in a way that is more natural to human cognition and movement. This includes reaching out to manipulate virtual objects constructed from the data with natural user interaction ways, moving around the objects to view them from a clearer perspective, and highlighting the items of interest with a point of the fingers. Medical data visualizations are powerful methods for interactively exploring medical data, especially when combined with statistical or machine learning methods. Medical visual analysis tools can be used for more in-depth and thorough data analysis that produce more profound insights, as well as richer understandings that enable researchers to ask bolder questions [46]. New approaches are trying to combine existing visualization tools with new technological opportunities, especially in AI and immersive technologies, to maximize human knowledge and intuition [16, 19, 46].

Intelligent and Immersive Visual Analytics of Health Data

31

Fig. 1 Genomic data visualization methods and environments [51]

Figure 1 shows recent and popular visual analytics methods for genomic data. AI has been applied to assist the analysis processes, and VR/AR has also been used as one of the presentation environments. Two-dimensional (2D) and threedimensional (3D) scatter plots, networks, heatmaps, and coordinates are the four traditional visualization methods for genomic data analysis which is still essential in the current visual analytics tools. Clustering, an AI technique that involves the grouping of data points to classify each data point into a specific group, could also be used in the analysis to enhance the classification of these methods. AI and data visualization can work together for better data analytics. AI algorithms support data visualization by automatically identifying patterns and making predictions. Meanwhile, data visualization methods can interpret and support AI by framing predictive modeling problems as well as evaluating or presenting the outcomes. Some interactive visualization projects have been extended to new environments such as VR, AR, large and high-resolution displays as well as mobile devices. A number of genomic and cancer visualization tools have also supported these new environments to enhance human’s perception in such natural human pattern recognition environments [34].

2 Visualization for Health Data Health data visualization is a rapidly evolving area. The field has been progressed significantly due to the improvement of hardware acceleration, standardized exchangeable file formats, dimensionality reduction, visual feature selection, multivariate data analysis, interoperability, 3D rendering and visualization of complex data at different

32

Z. Qu et al.

resolutions, especially the area of image processing combined with AI-based pattern recognition [48]. Intelligent visualization still takes time to gain more acceptance by diverse audiences. It is a challenge in improving the outcomes and trust in the algorithmic and automatic methods with interactive visualizations. Intelligent and interactive visualization of complex genomic data is an effective way to bring the insight of information and to discover the relationships, non-trivial structures, and irregularities or regularities that may pertain to the disease course of the patients. Statistics and basic visualizations without effective interaction and capabilities to handle the visual data mining process are often insufficient for exploration and decision making. Intelligent visualization can focus on scatterplot of patient population such as using TabuVis [43] and patient-to-patient comparisons through the biological data and then display the multi-dimensional data in cooperation with the automated analysis [44]. In more focus, intelligent genomic visualization can support experts in the process of hypothesis generation concerning the roles of genes in diseases and find the complex interdependencies between genes by bringing gene expressions in the context with pathways [29]. Figure 2 shows popular visual analytics and visualization tools of health data that are classified by years. Most genomic data visualization tools between the years 2000–2015 use traditional visualization methods such as scatterplots, heatmaps, genomic coordinates, networks, and clustering. Since 2016, more advanced analytic techniques such as machine learning algorithms for predictions and personalized medicine have been applied together with the visualizations. Some visualization tools are deployed in emerging environments such as mobile and immersive devices. In addition, the integration between the tools is also an important factor to enhance their capability to meet the new demands. For example, Epiviz [11] can obtain annotation data from the UCSC [18], Gitools [49] can get a heatmap visualization from IGV [54], and RNASeqBrowser [1] is compatible with UCSC as shown in the figure with purple arrows. Health analytical tools can be used to visualize data from the contextual population level to a more personalized approach and from a reactive method to proactive methods focus on prevention, wellness, and most importantly—the individual [5, 25]. Intelligent visualization combined with AI algorithms for genomic data is still a big challenge, yet it is becoming a new direction in the medical data visualization evolution. The detail description and discussion on the tools are presented in [51].

3 Visual Artificial Intelligence for Health Data In recent years, AI has been used in data visualizations, including multivariate genomic data due to the research advancement in the field and computational power [45, 53]. Machine learning (ML) method, as one branch of AI, is a way of solving problems without explicitly codifying the solution and a way of building systems that improve themselves over time. Their goals are to build predictive or descriptive models from characteristic features within training datasets and then use those

Intelligent and Immersive Visual Analytics of Health Data

33

Fig. 2 Tools timeline and integration through the years. The blue arrows present for the timeline, and the purple arrows show the integration among the tools [51]

34

Z. Qu et al.

features to draw conclusions from other similar datasets. For example, in cancer detection, diagnosis and management, ML helps identify significant factors in highdimensional datasets of genomic, proteomic, chemical or clinical data. It can also be used to understand the predicate of underlying diseases [21]. AI and ML have been applied in genomics for analyzing genome sequencing, gene editing, clinical workflow, and direct-to-consumer genomics [55]. Future applications of ML in genomic field are diverse, and they potentially contribute to the development of patients’ diagnoses or treatments or population-specific pharmaceutical drugs to look at the role of genetics in the context of how an individual responds to drugs [55]. For example, machine learning models were used to determine a stable dose of Tacrolimus in renal transplant patients in 2017 [61]. Tacrolimus is commonly administered to patients following a solid organ transplantation to prevent “acute rejection” of the new organ [61]. Health data visualization tools could assist in improving our understanding of the cancer study and lead to new methods of diagnosing and treating the diseases. Personalized genomic cancer medicine uses the latest genome sequencing to look at the genetics of cancer rather than treating it based on location to allow us to understand inherited cancer risks and find more effective treatments for people with cancer [60]. Health data visualization is entering a new era with the advancement in AI and emerging environments in immersive spaces and mobile devices. New technologies and cognitive frameworks are opening new horizons enabling more accurate and contextual data visualizations. Although having extraordinary predictive abilities, the ML models and their algorithms are hard to be understood and maybe even harder to be trusted, especially in conservative fields such as the medical industry [47]. Visualizing machine learning models and predictive results in a meaningful way can interpret the complex algorithms and outcomes that help clinicians, researchers, and domain experts understand and trust the predictive results. We next present one of our recent works on data visualization with a structured rhabdomyosarcoma dataset. In this study, we visualize explainable machine learning model, real-time predictive results, and the 3D scatterplot of the patient cohort in a similarity space.

3.1 Case Study—Rhabdomyosarcoma The following case study used a structured set of 101 patients’ rhabdomyosarcoma (RMS) dataset from the Westmead Children Hospital [52]. RMS is the most common soft tissue childhood sarcoma with an incidence rate of 17 new cases per year in Australia [63]. The two major histological subtypes of RMS are alveolar (ARMS) and embryonal (ERMS). ERMS patients have a more favorable prognosis. This difference in prognosis has led researchers to use molecular markers with the aim of developing more accurate classifiers of RMS subtypes. The Unity3D game engine platform was used to develop the prototype, and C# programming language was used to develop machine learning algorithms. We used SQLite database to manage the training data.

Intelligent and Immersive Visual Analytics of Health Data

35

As we only had a small number of patient cohort and they were certain to be accurate, we used all of them to train our decision tree model and then showed all of them in a 3D scatter plot to illustrate a scenario. For example, when we need to select a better treatment method for a new patient, we can choose a different treatment method for the new patient and predict the patient’s result alive or deceased. When a new patient’s genomic data comes, we want to predict the patient’s status as either alive or deceased based on three attributes: Histology, Sex, and Age-Status. We visualized the trained machine learning model in (i) a hierarchical plot or tree on the left panel, (ii) a 3D scatter plot for all RMS patients at the center, and (iii) real-time prediction results on the top right in another tree structure. The users can interact between the tree model and scatter plot, input new patient data on the bottom fields, and get the prediction results in another tree plot on the top right (see Fig. 3). In the 3D scatter plot, the red color items show ARMS patients, while the blue color ones are ERMS patients. The patients are located in the similarity space based on their genetic similarities which are defined by a feature selection process on the gene expression microarray dataset to identify the markers (e.g. probesets/genes in this

Fig. 3 Visualization illustrates the machine learning model [52], where A a machine learning model tree. B A 3D scatter plot for 101 RMS patients. C A new patient data input fields. We choose three attributes which are Histology, Sex, and Age-Status as “ARMS,” “Male,” “Favorable,” respectively, and D the real-time prediction result

36

Z. Qu et al.

study). The similarity space and markers potentially contribute to the differentiation between ERMS and ARMS patients [40]. In the visualization, the capsule shapes stand for female patients while the cubes are for male patients. The patients with the yellow halo are deceived while the patients without the halo are alive. This 3D scatter plot is connected with a machine learning model tree on the left panel. When the user chooses a branch on the ML tree model, the related group of patients’ shapes spin. For example, if the users want to highlight the group of patients with attributes: Histology as “ERMS,” Sex as “Male,” and AgeStatus as “Favorable,” they click on the related branch in the tree plot (see Fig. 3, Part A), and a corresponding group of blue cubes is spinning in the 3D scatter plot. If the user chooses a branch in the tree plot with Histology as “ARMS,” Sex as “Female,” and Age-Status as “Unfavorable,” then a group of red capsules will spin in the 3D scatter plot. The user can also choose the father branch ERMS to select or spin all the blue color patients. For the machine learning prediction process and results, we illustrate them on the top-right section (see Fig. 3, Part D). For example, in Fig. 3, Part C, we choose the values “ARMS” for Histology, “Male” for Sex, and “Favorable” for Age-Status as the new patient data attributes. When the button “Decide!” is clicked, the real-time result is shown. In this case, the real-time prediction result is green color indicating “True” which means the patient would survive. If the predicted result is red color indicating “False,” then the patient is deceased. The users may choose to show the patient label in the visualization when required. In this case study, the decision tree shown in Part A is trained by all the patient data. They are all accurate and can all fit into the machine learning model. Each branch node represents a choice between several alternatives, and each leaf node represents a decision. This decision tree fits all the training examples and is fully grown to give 100% accuracy of the data. In most cases, the training data could not all fit into the machine learning model, which ends up the less accurate decision trees. The overfitting data are caused by two major situations which are the presence of noise and lack of representative instances. The decision tree avoided the overfitting by pruning sections of the tree that provide little power to classify instances. Pruning reduces the complexity of the final classifier and improves predictive accuracy by reduction of overfitting. Moreover, the complexity of the model structure of all the patient data will decide the machine learning model tree’s structure and make the tree plot differently. If there are thousands of millions of patients’ data, only partial data should be chosen to be visualized to reduce the overwhelming in the human perception. The scatter plot also has ways to deal with big data by adding specific interactions to show the whole dataset or a subset of the data.

4 Immersive Technologies Virtual reality enables the psychophysical immersive experience in an artificial computer-generated virtual environment [58]. Augmented reality, usually, is built

Intelligent and Immersive Visual Analytics of Health Data

37

upon virtual reality in integrating and overlaying the virtual environment into the user’s real world and allowing the user to interact with the virtual objects in the context of his actual surroundings [10, 56]. Specialized equipment such as a headmounted display (HMD) or cave automatic virtual environment (CAVE) system is required for the use of VR/AR technologies. The sensor and camera on the equipment will help the system to determine and track the user moment and move the point of view accordingly. Shan et al. [56] developed an AR visualization, which runs on the mobile platform to deliver real-time 3D brain tumor volume rendering. It allows the clinicians to visualize and communicate with the patients on their tumors sizes and locations. The visualization uses the facial features of the patient as the tracking point to project the reconstructed brain tumor model onto the same location as the subject’s actual anatomy. Immersive analytics combine the use of immersive technologies, natural user interface and visualize best-known techniques to achieve transparent cognitive experience and to enable visual analytics in the immersive environment [32, 37, 50]. Immersive analytics allow parallel inspection of the massive amounts of data by visualizing the complex data structure in unlimited display space [39]. The immersive technologies are also capable of providing the user with additional sensory cues in creating a vibrant and realistic sensory stimulus that enhances the visual analytics process [50]. The user can fully immerse in the visualization in the immersive environment, allowing more data and structure of the data that can be presented to allow the ubiquitous data around the user visually comprehensible [32]. There have also been numerous empirical studies that show the effectiveness of immersive analytics in helping user engagement and productivity [2, 8, 36, 39]. VR and AR visualizations have also been adopted in the medical field. The immersive environment has natural 3D view for data analysis that are potentially useful for the information presentation and group decisions among clinicians and researchers, which are very popular in the medical industry. VR and AR are mainly used in diseases’ diagnostics, volume rendering and modeling [7, 12, 13, 30, 62], patient care and pain treatment [14, 17, 23, 27, 31, 33]. It has helped to improve the accuracy and efficiency of diseases’ diagnostics and volume rendering [12, 13, 62]. Immersive visualizations are also used extensively in patient care management. For example, it has been found to be effective for acute pain management [23]. Immersive technologies have also been introduced in an inpatient rehabilitation program [14, 24, 27]. The VR/AR techniques were found to be at least as effective as conventional therapy in stroke rehabilitation, Parkinson’s diseases and upper limb rehabilitation in subacute and chronic stroke by systematic studies and case study done by Laver et al. [27], Dockx et al. [14], and Llobera et al. [31], respectively. Immersive visualization has also been used to examine the relationship between gene and disease. Bhavnani et al. [3] developed an immersive visualization, which utilizes an immersive 3D network visualization to inspect the possible hidden association between regulated genes and renal diseases. The finding is positive in terms of the use of immersive technologies, and the 3D viewing method has improved

38

Z. Qu et al.

the chances in making discoveries. We next present a case study on our recently developed immersive visualization of genomic and biomedical data.

4.1 Case Study—Immersive Visualization for Cancer Data Our interactive visualization tool shows the whole group of patients’ data with a 3D scatter plot as well as individual patient’s details, zooms and rotates the plot, compares gene among several patients and interacts with users and shows the comparison visualization between selected patients [42]. We have developed a tool that supports the immersive visual analytics of genomics data that can be run on different platforms including mixed reality devices. This case study shows the genomic SNP profiles of 100 pediatric B cell lymphoblastic leukemia–lymphoma (ALL) patients treated at the Children’s Hospital at Westmead. The data were generated using Affymetrix expression microarrays (U133A, U133A 2.0, and U133 Plus 2.0) and Illumina NS12 SNP microarrays, respectively. Prior to the visualization, we used the attribute importance ranking that was implemented in random forest algorithm [6] to identify the genes of interests. Using these datasets, we built the 3D similarity space using a linear dimensionality reduction method such as principal component analysis where the distances between patients indicated their genetic similarity [42]. The visualization presents a 3D similarity space of the entire cohort of ALL patients’ gene expression data. Patients with a similar genetic profile will be placed closer together in the 3D scatter plot. Figure 4 illustrates a 3D scatter plot from the tool run in a mixed reality device, Microsoft HoloLens. Avatars are used to represent the patients. Due to the immersive nature of the visualization, the user is surrounded by the avatars. In this environment, the clinician may detect structure and relationships between data points with the aid of immersive analytics, which otherwise would be difficult on a conventional display [39]. The unlimited display space in the immersive virtual environment also facilitates and improves the data comparison processes [32]. The use of glyphs, in the form of avatars, could potentially be able to attract the attention of the user and to facilitate the cognitive stimulant compared to other types of visual designs [4]. A glyph utilizes visual channels, such as color, shape, and texture to enable the depiction of the various data attributes. In this case, the color was used to indicate if the patients’ survived while the gender of the avatar is shown as the image of boys and girls as shown in Fig. 4. The visualization tool also allows the user to select a patient-of-interest and make a visual comparison of the similarity in gene expression compared to the patient population. The visual transformation method follows the same concept in our previous work [42], where patients with high similarity will be moved closer together for the selected patients while the dissimilar patients will be pushed away. This feature allows the user to drill into the details of the similarity of gene expression for a particular gene and allows detailed investigation of the gene similarity (see Fig. 5).

Intelligent and Immersive Visual Analytics of Health Data

39

Fig. 4 Prototype of our immersive visualization showing a 3D similarity space of ALL patients on Microsoft HoloLens. Red and green colors are used to indicate deceased and survived patients, respectively. Most deceased patients are found on the top left corner circled in red indicating their genomic similarity in the analysis

Fig. 5 Focusing on the area identified in Fig. 4 (circled in black), the user can zoom and rotate the visualization to find, compare, and analyze the gene expression value for these patients. In this case, the focus is to find the deceased patient, ALL70, and compare with survived patients, ALL122 and ALL3. The heatmap of the genes of interest shows ALL70 has a very distinct gene expression values for the listed gene compared to the survived patients [26]

5 Discussion Visualization can amplify human cognitive abilities to understand complex processes and support decisions. Intelligent visualization research seeks new display, control

40

Z. Qu et al.

panels, features, and workflows to improve the capabilities of users that integrate with one or more artificial methods. Visualization researchers study perceptual and cognitive theories that guide design, as well as develop new tools and quantitative and qualitative evaluation methods to validate their hypotheses and refine their theories. In health data research, genomic research is critical to progress against cancer. Genomic and cancer data visualization tools can assist in improving our understanding of the biology of cancer and lead to better ways of diagnosing and treating the disease. Health data analytics have been evolving rapidly because of the advancement in genomic technologies and research. The evolution requires continuous development of visualization techniques and tools. As this rapid scientific evolution continues, cancer researchers are highly dependent on computers to manage, analyze, and visualize data. As big data evolves, data visualizations can help turn the most convoluted facts and figures into clear stories or message, aiding further research efforts and promoting awareness around important issues. For health data, specifically, intelligent visual analytics help viewers and researchers alike digest medical information, track trends and outbreaks of diseases, and ultimately get closer to much-needed cures. Health data visualizations have been developed with emerging sources of artificial intelligence and new visual environments such as VR/AR, immersive large screens and mobile devices. AI may assist physicians to make better clinical decisions or even replace human judgement in certain functional areas of health care. AI is playing an integral role in the evolution of the field of genomics. The increasing availability of healthcare data and rapid development of big data analytic methods have made possible for successful applications of AI in health care. Powerful AI techniques can unlock clinically relevant information hidden in the massive amount of data, which in turn can assist clinical decision making. AI supports the new approach to patient care, called personalized medicine, that encompasses genetics, behaviors, and environment intending to implement a patient or population-specific treatment method. This is because no treatment method can fit to all patient cohort and vice versa. AI and machine learning have been applied in genomics for analyzing genome sequencing, gene editing, clinical workflow, and direct-to-consumer genomics. Future applications of machine learning in the field of genomics are diverse and may potentially contribute to the development of patient or population-specific pharmaceutical drugs to look at the role of genetics in the context of how an individual responds to drugs [55]. Immersive displays, including VR/AR technologies, have been adopted in healthcare industry for a long time. For instance, medical researchers have created 3D models of patients’ internal organs using VR since the 1990s as summarized in [59]. Recently, with the new and better VR and related technologies, they have been used to plan complex operations, to reduce anxiety in cancer patients and to help patients overcome balance and mobility problems resulting from stroke or head injury. VR/AR environments are expected to bring a revolution in genomic data visualization as it could integrate meta-genomic data in virtual worlds. Approaching the problem from a different angle, mixed reality devices such as Google Glass, Microsoft HoloLens, and Magic Leap offer an augmented reality experience which can facilitate the learning process of the biological systems because it builds on

Intelligent and Immersive Visual Analytics of Health Data

41

exploratory learning. The natural immersive user interface would be an area of focus to allow seamless and effective use of the visualization. The familiarity with the immersive technologies of the millennials will continue to drive the rapid adaptation of the VR/AR applications in the future workplace as they become the dominant group in the workforce. Immersive analytics are capable of displaying complex data structure and relationship in a limitless displace space compared to a conventional monitor display [32]. This environment allows the introduction of additional sensory cues to enrich and provide realistic sensory stimuli that accelerate and enhance the visual analytics process. Immersive analytics inherited the capabilities of visual analytics by putting together immersive technologies, natural user interface, and visualization best-known techniques to achieve a transparent cognitive experience of visual analytics [32, 37, 50]. For example, a comparison of patients’ genomic data in the new space would be intuitive and natural compared to traditional 2D display. The immersive nature would help to provide the nature immersion activities to the user in the center of the visualization that could possibly lead to new discovery. In summary, health data, especially genomic and cancer data visualization tools, are essential to facilitate decision making for the treatment methods or targeted medicine. Artificial intelligence and immersive technologies have been used in recent years to create visualization tools that can explore complex health data. Further efforts are needed to develop new tools to meet the changing needs of the medical field.

References 1. An, J., Lai, J., Wood, D.L., Sajjanhar, A., Wang, C., Tevz, G., Lehman, M.L., Nelson, C.C.: RNASeqBrowser: a genome browser for simultaneous visualization of raw strand specific RNAseq reads and UCSC genome browser custom tracks. BMC Genom. 16, 145 (2015). https://doi.org/10.1186/s12864-015-1346-2 2. Arya, A., Nowlan, N., Sauriol, N.: Data-driven framework for an online 3D immersive environment for educational applications. In: Proceedings of the International Conference on Education and New Learning Technologies, pp. 4726–4736 (2010) 3. Bhavnani, S., Ganesan, A., Hall, T., Maslowski, E., Eichinger, F., Martini, S., Saxman, P., Bellala, G., Kretzler, M.: Discovering hidden relationships between renal diseases and regulated genes through 3D network visualizations. BMC Res. Notes 3(1), 296 (2010). https://doi.org/ 10.1186/1756-0500-3-296 4. Borgo, R., Kehrer, J., Chung, D.H., Maguire, E., Laramee, R.S., Hauser, H., Ward, M., Chen, M.: Glyph-based visualization: foundations, design guidelines, techniques and applications. In: Eurographics (STARs), pp. 39–63 (2013) 5. Boudreaux, E.D., Waring, M.E., Hayes, R.B., Sadasivam, R.S., Mullen, S., Pagoto, S.: Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl. Behav. Med. 4(4), 363–371 (2014). https://doi.org/10.1007/s13142-0140293-9 6. Breiman, L.: Radom forests. Mach. Learn. 45, 5–32 (2001) 7. Calì, C., Baghabra, J., Boges, D.J., Holst, G.R., Kreshuk, A., Hamprecht, F.A., Srinivasan, M., Lehväslaiho, H., Magistretti, P.J.: Three-dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues. J. Comp. Neurol. 524(1), 23–38 (2016). https://doi.org/10.1002/cne.23852

42

Z. Qu et al.

8. Camilleri, V., de Freitas, S., Montebello, M., McDonagh-Smith, P.: A case study inside virtual worlds: use of analytics for immersive spaces (2013). https://doi.org/10.1145/2460296. 2460341 9. Card, S.K., Mackinlay, J.D., Shneiderman, B.: Readings in information visualization: using vision to think. The Morgan Kaufmann series in interactive technologies. Morgan Kaufmann Publishers, an Francisco, California (1999) 10. Chang, Y., Peng Xu, W., Wang, L.: Research on 3D Visualization of Underground Antique Tomb Based on Augmented Reality, vol. 336–338 (2013). https://doi.org/10.4028/www.scientific.net/ AMM.336-338.1434 11. Chelaru, F., Smith, L., Goldstein, N., Bravo, H.C.: Epiviz: interactive visual analytics for functional genomics data. Nat. Methods 11(9), 938–940 (2014). https://doi.org/10.1038/nmeth. 3038 12. Claudia, E., Peter, E., Bernd, E., Katrin, E., Torsten, E.: Interactive 3D visualization of structural changes in the brain of a person with corticobasal syndrome. Front. Neuroinformatics 8 (2014). https://doi.org/10.3389/fninf.2014.00042 13. David, B.D., Clifford, A.W., Gibson, J.D., John, M.B., Max, W.: Augmented reality: advances in diagnostic imaging. Multimodal Technol. Interact. 1(4), 29 (2017). https://doi.org/10.3390/ mti1040029 14. Dockx, K., Bekkers, E.M.J., Van den Bergh, V., Ginis, P., Rochester, L., Hausdorff, J.M., Mirelman, A., Nieuwboer, A.: Virtual reality for rehabilitation in Parkinson’s disease. Cochrane Database Syst. Rev. 12 (2016). https://doi.org/10.1002/14651858.cd010760.pub2 15. Fuchs, R., Waser, J., Groller, M.E.: Visual human + machine learning. IEEE Trans. Vis. Comput. Graph 15(6), 1327–1334 (2009). https://doi.org/10.1109/TVCG.2009.199 16. García-Hernández, R.J., Anthes, C., Wiedemann, M., Kranzlmüller, D.: Perspectives for using virtual reality to extend visual data mining in information visualization. In: 2016 IEEE Aerospace Conference, 5–12, pp. 1–11 (2016). https://doi.org/10.1109/aero.2016.7500608 17. Gold, J.I., Belmont, K.A., Thomas, D.A.: The neurobiology of virtual reality pain attenuation. Cyberpsychology Behav. Impact Internet, Multimed. Virtual Real. Behav. Soc. 10(4), 536 (2007). https://doi.org/10.1089/cpb.2007.9993 18. Goldman, M., Craft, B., Swatloski, T., Cline, M., Morozova, O., Diekhans, M., Haussler, D., Zhu, J.: The UCSC cancer genomics browser: update 2015. Nucleic. Acids Res. 43, D812–817 (2015). https://doi.org/10.1093/nar/gku1073 19. Golestan Hashemi, F.S., Razi Ismail, M., Rafii Yusop, M., Golestan Hashemi, M.S., Nadimi Shahraki, M.H., Rastegari, H., Miah, G., Aslani, F.: Intelligent mining of large-scale bio-data: bioinformatics applications. Biotechnol. Biotechnol. Equip. 32(1), 10–29 (2017). https://doi. org/10.1080/13102818.2017.1364977 20. Green, T.M., Ribarsky, W., Fisher, B.: Visual analytics for complex concepts using a human cognition model. In: 2008 IEEE Symposium on Visual Analytics Science and Technology, vol. 19–24, pp. 91–98 (2008). https://doi.org/10.1109/vast.2008.4677361 21. Joseph, A.C., David, S.W.: Applications of machine learning in cancer prediction and prognosis. Cancer Inform. 2, 59–78 (2006) 22. Keahey, T.A.: Using visualization to understand big data. Adv. Vis. (2013) 23. Keefe, J.F., Huling, A.D., Coggins, J.M., Keefe, F.D., Rosenthal, M.Z., Herr, R.N., Hoffman, G.H.: Virtual reality for persistent pain: a new direction for behavioral pain management. Pain 153(11), 2163–2166 (2012). https://doi.org/10.1016/j.pain.2012.05.030 24. Kiper, P., Szczudlik, A., Agostini, M., Opara, J., Nowobilski, R., Ventura, L., Tonin, P., Turolla, A.: Virtual reality for upper limb rehabilitation in subacute and chronic stroke: a randomized controlled trial. Arch. Phys. Med. Rehabil. 99(5), 834–842.e834 (2018). https://doi.org/10. 1016/j.apmr.2018.01.023 25. Krisa, D., Tailor, S.I.: Data visualization in health care: optimizing the utility of claims data through visual analysis (2014) 26. Lau, C.W., Nguyen, Q.V., Qu, Z., Simoff, S., Catchpoole, D.: Immersive intelligence genomic data visualisation. Paper Presented at the ACM (2019)

Intelligent and Immersive Visual Analytics of Health Data

43

27. Laver, K.E., Lange, B., George, S., Deutsch, J.E., Saposnik, G., Crotty, M.: Virtual reality for stroke rehabilitation. Cochrane Database of Syst. Rev. 11 (2017). https://doi.org/10.1002/ 14651858.cd008349.pub4 28. Leung, M.K.K., Delong, A., Alipanahi, B., Frey, B.J.: Machine learning in genomic medicine: a review of computational problems and data sets. Proc. IEEE 104(1), 176–197 (2016). https:// doi.org/10.1109/jproc.2015.2494198 29. Lex, A., Streit, M., Kruijff, E., Schmalstieg, D.: Caleydo: design and evaluation of a visual analysis framework for gene expression data in its biological context. In: 2010 IEEE Pacific Visualization Symposium (PacificVis), vol. 2–5, pp. 57–64 (2010). https://doi.org/10.1109/ pacificvis.2010.5429609 30. Lin, Q., Xu, Z., Li, B., Baucom, R., Poulose, B., Landman, B.A., Bodenheimer, R.E.: Immersive virtual reality for visualization of abdominal CT. In: Medical Imaging 2013: Image Perception, Observer Performance, and Technology Assessment. International Society for Optics and Photonics, p. 867317 (2013) 31. Llobera, J., González-Franco, M., Perez-Marcos, D., Valls-Solé, J., Slater, M., Sanchez-Vives, M.: Virtual reality for assessment of patients suffering chronic pain: a case study. Exp. Brain Res. 225(1), 105–117 (2013). https://doi.org/10.1007/s00221-012-3352-9 32. Luboschik, M., Berger, P., Staadt, O.: On Spatial Perception Issues in Augmented Reality Based Immersive Analytics (2016). https://doi.org/10.1145/3009939.3009947 33. Maani, C.V., Hoffman, H.G., Morrow, M., Maiers, A., Gaylord, K., McGhee, L.L., Desocio, P.A.: Virtual reality pain control during burn wound debridement of combat-related burn injuries using robot-like arm mounted VR goggles. J. Trauma: Inj. Infect. Crit. Care 71(1 supplement), S125–S130 (2011). https://doi.org/10.1097/TA.0b013e31822192e2 34. Matte-Tailliez, O., Toffano-Nioche, C., Ferey, N., Kepes, F., Gherbi, R.: Immersive visualization for genome exploration and analysis. In: 2006 2nd International Conference on Information and Communication Technologies, vol. 24–28, pp. 3510–3515 (2006). https://doi.org/10.1109/ ictta.2006.1684982 35. Mills, M.: Artificial Intelligence in Law: The State of Play 2016 Thomson Reuters S031401/3– 16 (2016) 36. Moran, A., Gadepally, V., Hubbell, M., Kepner, J.: Improving big data visual analytics with interactive virtual reality (2015). https://doi.org/10.1109/HPEC.2015.7322473 37. Müller, C., Krone, M., Huber, M., Biener, V., Herr, D., Koch, S., Reina, G., Weiskopf, D., Ertl, T.: Interactive molecular graphics for augmented reality using Hololens. J. Integr. Bioinform. 15(2). https://doi.org/10.1515/jib-2018-0005 38. Natalia Andrienko, G.A.: Intelligent visualisation and information presentation for civil crisis management. Trans. GIS 11(6), 11 (2007). https://doi.org/10.1111/j.1467-9671.2007.01078.x 39. Nguyen, H., Marendy, P., Engelke, U.: Collaborative Framework Design for Immersive Analytics (2016). https://doi.org/10.1109/BDVA.2016.7787044 40. Nguyen, Q.V., Alzamora, P., Ho, N., Huang, M.L., Simoff, S., Catchpoole, D.: Unlocking the complexity of genomic data of RMS patients through visual analytics. In: Paper presented at the 2012 International Conference on Computerized Healthcare, pp. 17–18. Hong Kong (2012) 41. Nguyen, Q.V., Gleeson, A., Ho, N., Huang, M.L., Simoff, S., Catchpoole, D.: Visual analytics of clinical and genetic datasets of acute lymphoblastic leukaemia. In: Lu, B.-L., Zhang, L., Kwok, J. (eds.) Neural Information Processing: 18th International Conference, ICONIP 2011, pp. 13– 17. Shanghai, China. Proceedings, Part I. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 113–120 (2011). https://doi.org/10.1007/978-3-642-24955-6_14 42. Nguyen, Q.V., Khalifa, N.H., Alzamora, P., Gleeson, A., Catchpoole, D., Kennedy, P.J., Simoff, S.: Visual analytics of complex genomics data to guide effective treatment decisions. J. Imaging 2(4), 29 (2016). UNSP 2910.3390/jimaging2040029 43. Nguyen, Q.V., Qian, Y., Huang, M.L., Zhang, J.W.: TabuVis: a tool for visual analytics multidimensional datasets. Sci. China-Infr. Sci. 56(5), 1–12 (2013). ARTN 05210510.1007/s11432013-4870-1 44. Nguyen, Q.V., Nelmes, G., Huang, M.L., Simoff, S., Catchpoole, D.: Interactive visualization for patient-to-patient comparison. Genomics Inf. 12(1), 21–34 (2014). https://doi.org/10.5808/ GI.2014.12.1.21

44

Z. Qu et al.

45. Nilsson, N.J.: The Quest for Artifical Intelligence: A History of Ideas and Achievements (2009) 46. Olshannikova, E., Ometov, A., Koucheryavy, Y., Olsson, T.: Visualizing big data with augmented and virtual reality: challenges and research agenda. J. Big Data 2(1) (2015). https:// doi.org/10.1186/s40537-015-0031-2 47. Patrick, H., Wen, P., SriSatish, A.: Ideas on interpreting machine learning. O’Reilly (2017) 48. Pavlopoulos, G.A., Malliarakis, D., Papanikolaou, N., Theodosiou, T., Enright, A.J., Iliopoulos, I.: Visualizing genome and systems biology: technologies, tools, implementation techniques and trends, past, present and future. Gigascience 4, 38 (2015). https://doi.org/10.1186/s13742015-0077-2 49. Perez-Llamas, C., Lopez-Bigas, N.: Gitools: analysis and visualisation of genomic data using interactive heat-maps. PLoS ONE 6(5), e19541 (2011). https://doi.org/10.1371/journal.pone. 0019541 50. Polys, N., Mohammed, A., Iyer, J., Radics, P., Abidi, F., Arsenault, L., Rajamohan, S.: Immersive analytics: crossing the gulfs with high-performance visualization (2016). https://doi.org/ 10.1109/IMMERSIVE.2016.7932376 51. Qu, Z., Lau, C.W., Nguyen, Q.V., Zhou, Y., Catchpoole, D.R.: visual analytics of genomic and cancer data: a systematic review. Cancer Inform. 18, 1176935119835546 (2019) 52. Qu, Z., Zhou, Y., Nguyen, Q.V., Catchpoole, D.R.: Using visualization to illustrate machine learning models for genomic data. Paper Presented at the ACM (2019) 53. Ribeiro, M., Singh, S., Guestrin, C.: Why Should I Trust You? Explaining the Predictions of Any Classifier. arXivorg (2016) 54. Robinson, J.T., Thorvaldsdottir, H., Winckler, W., Guttman, M., Lander, E.S., Getz, G., Mesirov, J.P.: Integrative genomics viewer. Nat. Biotechnol. 29(1), 24–26 (2011). https://doi.org/10. 1038/nbt.1754 55. Sennaar, K.: Machine Learning in Genomics—Current Efforts and Future Applications (2018). https://www.techemergence.com/machine-learning-in-genomics-applications/ 56. Shan, Q., Doyle, T.E., Samavi, R., Al-Rei, M.: Augmented reality based brain tumor 3D visualization. In: 8th International Conference on Emerging Ubiquitous Systems and Pervasive Networks (EUSPN 2017), vol. 113, pp. 400–407 (2017). https://doi.org/10.1016/j.procs.2017. 08.356 57. Shilling, C.: How Augmented Reality will Change Data Visualization (2017). http://blog. i2econsulting.com/how-augmented-reality-will-change-data-visualization/ 58. Simpson, R.M., LaViola, J.J., Laidlaw, D.H., Forsberg, A.S., van Dam, A.: Immersive VR for scientific visualization: a progress report. IEEE Comput. Graphics Appl. 20(6), 26–52 (2000). https://doi.org/10.1109/38.888006 59. Slater, M., Sanchez-Vives, M.V.: Enhancing Our Lives with Immersive Virtual Reality 3(74) (2016). https://doi.org/10.3389/frobt.2016.00074 60. Stevens, E.A., Rodriguez, C.P.: Genomic medicine and targeted therapy for solid tumors 111 (2015). https://doi.org/10.1002/jso.23699 61. Tang, J., Liu, R., Zhang, Y.L., Liu, M.Z., Hu, Y.F., Shao, M.J., Zhu, L.J., Xin, H.W., Feng, G.W., Shang, W.J., Meng, X.G., Zhang, L.R., Ming, Y.Z., Zhang, W.: Application of machinelearning models to predict tacrolimus stable dose in renal transplant recipients. Sci. Rep. 7, 42192 (2017). https://doi.org/10.1038/srep42192 62. Venson, J., Berni, J., Maia, C., Da Silva, A., D’Ornelas, M., Maciel, A.: Medical imaging VR: Can Immersive 3D Aid in Diagnosis? 02–04 (2016). https://doi.org/10.1145/2993369.2996333 63. Wachtel, M., Runge, T., Leuschner, I., Stegmaier, S., Koscielniak, E., Treuner, J., Odermatt, B., Behnke, S., Niggli, F., Schafer, B.: Subtype and prognostic classification of rhabdomyosarcoma by immunohistochemistry. J. Clin. Oncol. 24(5), 816–822 (2006). https://doi.org/10.1200/JCO. 2005.03.4934 64. Ware, C.: Information Visualization Perception for Design (2013) 65. Wei, L., Huang, X., Huang, M.L., Nguyen, Q.V.: Applying graph layout techniques to web information visualization and navigation. In: Paper Presented at the IEEE Int’l Conference on Computer Graphics, Imaging and Vision (CGIV07). Bangkok, Thailand, 13 Aug 2007

Interactive Process Indicators for Obesity Modelling Using Process Mining Zoe Valero-Ramon, Carlos Fernandez-Llatas, Antonio Martinez-Millana and Vicente Traver

Abstract World Health Organisation defines overweight and obesity as abnormal or excessive fat accumulation that represents a risk to health. Obesity and overweight are associated with increased risk of comorbidities and social problems that negatively impact quality of life. Due to the complexity of the problems, it is necessary to classify obesity based on a set of factors rather than a simple increase in body weight. The objectives of this work were to examine BMI and data available from comorbidities associated to obesity, from a dynamic perspective thanks to the use of process mining tools, in order to obtain patterns of patients’ behaviours. On the other hand, to develop a set of human-readable and contextualised interactive process indicators (iPIs) in the field of obesity, related conditions support health professionals to interact with the process. Modelling iPIs as enhanced views will help the professionals to better perceive of these processes. Professionals will monitor the patient’s progress iteratively and will interact with the system to fine-tune interventions and treatments. The developed strategy can support both the characterisation of general process-based PI and the analysis of individual and personalised aspects of the processes going from general to individual. This method was applied to real data extracted from a tertiary hospital in Spain. Keywords Behaviour modelling · Process mining · Obesity · Interactive indicators

Z. Valero-Ramon (B) · C. Fernandez-Llatas · A. Martinez-Millana · V. Traver Universitat Politècnica de València, Camino de Vera S/N, 46022 Valencia, Spain e-mail: [email protected] C. Fernandez-Llatas e-mail: [email protected] A. Martinez-Millana e-mail: [email protected] V. Traver e-mail: [email protected] © Springer-Verlag GmbH Germany, part of Springer Nature 2020 I. Maglogiannis et al. (eds.), Advanced Computational Intelligence in Healthcare-7, Studies in Computational Intelligence 891, https://doi.org/10.1007/978-3-662-61114-2_4

45

46

Z. Valero-Ramon et al.

1 Introduction Obesity is considered a chronic disease which worldwide prevalence has increased over the last two decades, and concretely, since 1980, the prevalence of obesity has doubled to an extent that nearly a third of the world population is now classified as overweight or obese [1, 2]. Indeed, it is recognised as an international epidemic, not only in adults, but also in children, being a major public health concern across the globe irrespective of sexes or geographical location [1]. Based on the latest estimates in European Union countries, overweight affects 30–70% and obesity affects 10– 30% of adults [3]. Despite countries efforts to reduce or even stop this burden [4], the prevalence of obesity across Europe is forecast to rise by 2025 [5], showing more efforts should be done to reverse this situation. Even more, if these forecasts prove accurate, this will further hinder efforts for healthcare cost containment [6]. In Spain, the situation follows the global estimations, according to data from the European Health Survey in 2014, 52.7% of its adult population (over 18 years old) would be over a normal weight, placing 35.7% overweight and 16.9% obese [7]. Overweight and obesity occur when energy intake exceeds energy expenditure over an extended period [8]. The definition for overweight, according to the World Health Organisation (WHO), is having a body mass index (BMI) greater than or equal to 25 kg/m2 and below 30 kg/m2 ; similarly, the definition for obesity is having a BMI greater than or equal to 30 kg/m2 [9]. However, it is not only a problem of excess of weight, obesity and overweight are both associated with increased risk of comorbidities and with psychological and social problems that have negative impact on quality of life [10]. This situation can lead to further morbidity and mortality. Moreover, overweight and obesity are inside the five leading global risks for mortality in the world [11]. In this line, a study provides a comprehensive estimation of the incidence of 18 comorbidities attributable to overweight and obesity using standardised and consistent definitions and methodologies [12]. More concretely, literature directly associates overweight and overweight with cardiometabolic factors, including risk factors (hypertension, hyperlipidaemia and type II diabetes mellitus) and cardiovascular diseases (ischemic heart disease, cerebrovascular disease and peripheral vascular disease), asthma and musculoskeletal disorders (osteoarthritis of the lower limbs and sciatica) [13–15]. Despite the importance of detecting those at risk of being obese or overweight, there is a controversy in the literature about what is the best way to measure obesity. The most commonly used definition of obesity in the social science literature is based on BMI [16]. Its popularity stems in part from its convenience, safety and minimal cost, and its use has been widespread despite the fact BMI has some weaknesses [17]. Indeed, BMI obesity classification has the same limitations as a weight, that is, it ignores body composition [16]. Consequently, it results in substantial misclassification of individuals into weight classifications [16]. Notwithstanding there is some agreement on the fact BMI is not the most accurate way of measuring obesity, there is no consensus on which is the best measure, candidates include waist circumference;

Interactive Process Indicators for Obesity Modelling …

47

waist-to-hip ratio; total body fat; and per cent body fat [18]. In fact, obesity definitions based on previous measures have their own strengths and weaknesses, as they depend on the strengths and weaknesses of the measure on which they are founded. Moreover, the use of different measures of fatness generates different obesity rates that challenge the knowledge about who is obese and why [16]. This could result in a considerable underestimation of the grave consequences of the overweight and obesity epidemic. However, this is not only the fact of having different obesity definitions. The traditional approach considering one of the previous variables for obesity is able to characterise a concrete status of an individual, but these measures could not show behavioural processes by themselves. In the case of BMI, it classifies an individual as obese or not in a particular moment, without considering previous states or future trends. Nevertheless, BMI and comorbidities are not static, they evolve towards different destinations, and they are usually measured and stored during long periods in electronic health records (EHR). Therefore, they are susceptible to be treated and analysed as dynamic variables over time, this will allow considering behaviours, both individual and collective. Likewise, due to the role of obesity and overweight on a set of comorbidities, it is necessary to classify obesity based on other factors, rather than a simple increase in body weight. Additionally, it is necessary a different and complementary approach for obesity, taking into consideration all aspects of individuals as a process, instead of considering single and isolated variables. Going a step forward, BMI long-term trajectories are needed to better understand the problem. On one hand, this may facilitate public understanding of this complex public health issue, and on the other hand, this will help to sensitise health professionals to promote systematic detection of obesity and overweight, giving them new tools and information about the real processes that obese population is following. Using all the information gathered from EHR regarding weight and obesity-related comorbidities could be possible to extract patterns that will allow analysing individual processes as they are dynamic procedures that evolve over time showing behavioural patterns of people. The rationale of this paper is to propose the use of interactive process mining for discovering and obtaining person’s obesity-related processes over time, which will allow measuring behaviour changes in a qualitative and quantitative way. Moreover, we have introduced a formal model, and so health professional could formulate open questions and their corresponding answers for a concrete health problem, called interactive process indicators (iPIs). The model allows them to define what they want to infer from data, and iPIs convert these questions to formal representations in the form of enhanced views for dynamic variables. This work is structured as follows: next section states process mining technology and interactive paradigm used in this paper. In results section, the flows inferred using process mining technology are explained. Finally, a discussion and conclusions part concludes the work.

48

Z. Valero-Ramon et al.

2 Methods The standardisation of care process in medicine, known as clinical pathways, has been approached through knowledge-based temporal abstraction (KBTA or TA). Mainly, TA is a method to achieve a switch from a qualitative time-stamped representation of raw data to a qualitative interval-based description of time series, with the objective of abstracting high-level concepts from time-stamped data. In the literature, there are works approaching clinical pathways with TA in some areas, such as prognosis of the risk for coronary heart disease [19], for defining typical medial abstraction patterns [20] and for the assessment of costs related to diabetes mellitus [21]. These works tried to create an automatic summarising of patient’s current state based on patient’s data through temporal abstraction, and however, the great majority of clinical variables (such as weight or blood pressure) have numerical values, but TA techniques are based on discrete labels, excluding important information from the analysis. Other work performed a dual approach, TA with data mining for BP and temperature [22]. Present study goes a step forward implementing a data-drive approach, not only presenting a temporal abstraction stratification, but also showing the temporal flow of the process. Nevertheless, it is not just about representing time-stamped data in a qualitative way, but also a question of what data represent and how to show it. This is crucial to achieve a real understanding and knowledge about health processes. For this, it is needed a theoretical knowledge about the process itself, and to have an updated and precise information about how the process is really happening. For our concrete problem, it is not the same knowing obesity stage, that seeing an individual evolution over time. In addition, health professionals should be able to inquire what and how they want to know about processes, and usually these questions are complex and not formalise for machine learning systems. Examples of those questions could be, ‘At what point does a person who accumulates fat begin to have other complications?’ and ‘Has a person who is losing or gaining weight the same comorbidities?’ In order to find a method for formalising complex questions, the business sector gives us an interesting approach. In business, the concept of key performance indicators (KPIs) were often used for providing a measurable value that demonstrates how effectively a company is achieving key objectives, in other words a measure of the level of performance of a process. KPIs are usually defined to be SMART (Specific, Measurable, Achievable, Relevant and Time). It means, the measure has a specific purpose for the business, it is measurable to really get a value of the KPI, the defined norms have to be achievable, the improvement of a KPI has to be relevant to the success of the organisation and finally, it must be time-phased, which means the value or outcomes are shown for a predefined and relevant period. This concept could support the formalisation of health professionals’ questions in a formal, flexible and open way. Moreover, it is important to incorporate health professional knowledge, not only at the questions definition stage, but also within the rest of the process. Machine learning systems are not error-free, so human intervention is needed to verify and/or

Interactive Process Indicators for Obesity Modelling …

49

correct results provided [23]. Interactive paradigm defines this concept by integrating the human activity into the process [24]. It assures a close collaboration between an automatic learning algorithm and the human in order to not only provide models that professionals can use for a better understanding of the actual process, but also to correct those models according to human knowledge so new perceptual and cognitive models are provided. Process mining features, enabling experts to add their own knowledge to the learning process and correcting actions in a iterative way, make process mining perfectly work in combination with the interactive paradigm to define and present health questions in the way KPIs do it. Process mining could be a powerful solution supporting healthcare professionals in understanding patients’ pathways regarding obesity [25]. Succinct, process mining is a relatively new paradigm based on syntactical data mining framework thought to help experts in the understanding of complex process, in a comprehensive, objective and exploratory means [26, 27]. Process mining supplies algorithms, tools and methodologies to display what is actually taking place within a process that usually does not correspond with the observed one [28]. Process mining prioritises human understandably over accuracy in learning processes, this is one of the main reasons process mining is being introduced in health care, that can be used to obtain knowledge from health information, in order to comprehend dynamic health care processes. In this way, the application of process mining technologies can be used for supporting health professionals in the management of obesity process understanding, and more important the combination of process mining technologies and clustering algorithms allows individuals stratification based on their dynamic BMI evolution and comorbidities. More concretely, PMApp is a process mining tool specifically designed to create custom process mining dashboards in the healthcare domain. These dashboards are totally customised applications allowing the selection of the most adequate views and tools for each problem [29]. On the other side, PMCode is a .NET Frameworkbased toolkit for creating process mining algorithms, filters and enhancers that allow creating custom applications for each process mining problem [29]. PMApp allows integrating PMCode ocmponents in dashboards in an easy way for composing custom process mining solutions. PMApp Tool is built using Process Choreography Paradigm [30]. This is intended to provide custom dashboards that can be constructed via services composition, so it can be created custom dashboard depending on the questions to solve [31]. The discovery algorithm implemented in PMCode is parallel activity log inference algorithm ( PALIA ) [25]. PALIA has been widely tested in real healthcare scenarios and compared with other process mining discovery algorithms [25]. It has been used for analysing the protocols by diabetes patients in their follow up [32]; also to measure and discover the individualised behaviour of elderly people in risk of dementia [33]; for the characterisation of the emergency flow, allowing to measure organisational changes effects [29] and to discovery the flow of surgery department [25]. With the objective of discovering different population behaviours behind obesity and its related comorbidities, it is needed to use tools for extracting these patterns such as clustering algorithms [32]. PMApp tool implements a quality threshold clustering

50

Z. Valero-Ramon et al.

algorithm using edition distance workflow algorithm [33, 34] for this purpose. Moreover, PMApp allows the selection of threshold between 0.0 and 1.0 representing the per cent distance between two samples. A minimum distance (0.0) means equal flows, while a maximum distance (1.0) produces traces with no common activities. Consequently, we have designed a scenario where obesity-related and timestamped data from a concrete population were considered. This scenario was within the international research project CrowdHEALTH [35]. It is partially funded by the 2020 Programme of the European Commission that intends to integrate high volumes health-related heterogeneous data from multiples sources with the aim of supporting policy-making decisions. Obesity and overweight use case was located in Spain, concretely in the Valencian region. The main objectives of the use case were systematic detection of obesity and overweight, and the use of risk comorbidities due to obesity in detection, in order to sensitise professionals of the National Health System to promote the systematic detection of both, obesity and overweight, within the population. Non-detection problem is foreseen in Valencia region with available data, analysing Health records from Health Department Valencia La Fe (Valencia City) it is obtained that only 2% of the population has an obesity or overweight diagnosis. In contrast, obesity prevalence in the region is 14.5% [36], evidencing the non-detection and non-screening problem regarding obesity in the general population. This problem is even more sever, as early recognition and treatment of obesity can reduce progression of the condition and prevent the development of secondary complications or comorbidities that arise from excess weight, but weight screening is not generalised in general practitioners practise. In this scenario, health professionals and policy-makers need understandable ways to infer proper information and understanding that could answer open questions regarding health processes. In this paper, we show the possibilities of using interactive process mining paradigm for formalising health professionals’ questions regarding obesity, in a proper model. These questions are, Could obesity be re-defined as a dynamic process? How dynamic obesity is related with its comorbidities over time? We have formalised these questions so we can represent person’s related information flow to answer them, and we called it interactive process indicators (iPIs). An iPI is an enhanced graphical representation of a person behaviour taking into account all needed variables regarding a concrete health dimension/question, during a period. This interactive paradigm lets experts acquire knowledge and understand about what is actually happening during a period, instead of visualising isolated data of a single individual that do not provide extra insight from the health process. Thanks to process mining, iPIs are shown in a graphical and easy way, providing readable and contextualised information. Moreover, this technology allows comparing iPIs in order to evaluate the effectiveness of interventions changes. An iPI is a non-static or automatic task, but an iterative process of knowledge discovery that involves trial and failure, and it is often necessary to modify model parameters until result achieves the desired properties, and during this process, knowledge from health professionals is crucial. Based on the definition of iPIs, interactive process mining paradigm will be used to identify, classify and graphical present iPIs for obesity and overweight problem. Process mining uses the events collected from the

Interactive Process Indicators for Obesity Modelling …

51

person’s information for creating an understandable flow that explains the person’s individual process. Moreover, they allow experts to evaluate the effectiveness of the interventions and to enable personalised interventions. Therefore, process mining technology is a perfect candidate to obtain iPIs. Once preliminary iPIs were defined, we were at the best position for applying process mining technology to obtain the desire results. For that, we performed the following steps: (1) to select relevant variables from La Fe data set; (2) to obtain event log; (3) to apply discovery process with PALIA; (4) to perform quality filters in order to obtain better quality flows; (5) to cluster with Quality Threshold algorithm (using Edition Distance Workflow); and (6) to heat maps. PMApp includes the possibility of applying heat maps over the flow. The heat maps apply a colour gradient showing time spent in each of the activities of the flow (nodes), and in the transitions, the colour represents the number of events occurred, it means the number of executions that follow a concrete path.

3 Results In this work, we considered BMI (weight, height) and blood pressure measures during a period from a real data set. The results show how process mining discovery techniques are able to infer obesity generic behaviour of patients in a broad way and to stratify patients in different groups based on their behaviour flow evolution. This supports the idea of considering obesity and related variables, such as blood pressure, as dynamic ones, showing how they evolve over time, and representing considered population behaviours. Data set considered in the study was based on historical information from patients from Health Department Valencia La Fe (Valencia City). This health department is responsible of 300,000 inhabitants regarding primary and secondary health care. Valencia La Fe data set extracted for the work corresponds to the period from 2012 to 2016, from primary care, hospital admissions, emergency, outpatient and morbidity diagnosis, as described in Table 1.

3.1 iPI0: Semantic Definition for Dynamic Obesity For this fist iPI0, we worked with PRIMARY CARE table information, concretely with those patients with weight and height measures, as the objective was to obtain BMI behaviours. Cut-off BMI points considered were those specify by WHO for adults, this is: underweight for BMI 80%, Spec > 70%, Acc = 98.8%

[62]

Basic relief assignment

Comparison of similar methods

2

Acc = 98.8%

[53]

Logistic regression, ANN, SVM, kNN, decision trees

Detection of pigmentation network

3 or 2 groups, nevus versus dysplastic versus melanoma

Logistic regression ~ ANN ~ SVM > kNN, decision trees

[19]

Automated classification

3 group comparison versus clinical diagnosis

3 or 2 groups, dysplastic and melanoma versus nevus/melanoma versus dysplastic and nevus

3 class. Auto. Sens = 73%, Clin. Sens = 72%, dys&mel Sens = 87%, Spec = 92%, dys&nev Sens = 77%, Spec = 84%

[28]

LDA, neural networks

PCA, focus on regions

2 Melanoma versus dysplastic

LDA 4PCs Acc = 96.2%/2PCs Acc = 96% NN 4PCs Acc = 100%/2PCs Acc = 84.6%

[20]

Discriminant analysis

Automatic assignment for border parameters

2

Acc = 80%, Sens = 85.88%, Spec = 74.12

[56]

Neural networks

Automatic assignment for border parameters

2

Acc = 92%

[68]

Size function on SVM

ABCDE rule

2

Sens > 84%, Spec > 83%, →Sens = 100% − Spec = 63.65%

[65]

SVM, LDA

–

2

SVM Acc = 94.1%, LDA Acc = 88%

[55]

Discriminant analysis

10 features

2

Acc = 96%, Sens = 90%, Spec = 98.3%

[8]

Criteria detection

7-point checklist

2

–

[32]

SVM

Texture classification

2

Acc = 70%

[35]

Back-propagation neural networks, SVM

BNN to SVM

2

BNN Acc = 95%, SVM Acc = 85%

[36]

Neural networks

–

2

Acc = 74.5%, Sens = 67.5%, Spec = 80.5% (continued)

94

G. Kontogianni and I. Maglogiannis

Table 1 (continued) Reference

Classification system

Information

Groups for comparison

Resultsa

[69]

Supervised classification of Markov fields

Pattern analysis

2

Acc = 86%

[21]

SVM, Gaussian maximum likelihood, kNN

Class combination

3

Acc = 83.75%, SVM only Acc = 72.45%

[54]

Graph-based

Pigmentation network detection

2

Acc = 94.3%

[3]

Decision tree for reticular pattern

Pigmentation network detection

2

Sens = 86%, Spec = 81.67%

[67]

Ensemble classifiers

SVM fusion

2

Sens = 93.76%, Spec = 93.84%

[66]

Ensemble classifiers

Discriminant analysis, ANN, kNN, SVM, decision trees

2

Acc = 91%, Sens = 97%, Spec = 65%

[71]

Deep learning

Sparse coding, SVM

2, melanoma versus non-melanoma, melanoma versus atypical

Mel versus non Acc = 93.1%, Sens = 94.9%, Spec = 92.8% Mel versus atyp Acc = 73.9%, Sens = 73.8%, Spec = 74.3%

[75]

Automatic scoring

ABCD rule

2

Sens = 91.25%, Spec = 95.83%

[72]

Deep learning

–

2

Acc = 76%

[70]

Fuzzy class of image pixels

Based on decision trees

2

Acc = 88%, Sens = 90.71%, Spec = 83.44%

[45]

SVM

Entropy-based features

2

Acc = 97.5%

a Acc

accuracy, Sens sensitivity, Spec specificity

the feature extraction step, using statistical methods and supervised learning to lower the feature level, to achieve better performance on classification. The number of features that were used in each case varies, as does their nature. It is clear, though, that the success of classification depends mainly on feature selection [19, 26, 35, 45, 55]. The most common classification methods are rule-based, e.g., [13, 16–18, 20, 53, 59]. More advanced techniques such as neural networks and support vector machines are presented in works like [11, 12, 45, 56, 58, 65, 117, 118], while the k-nearest neighborhood classification scheme is applied in [19, 119]. Evidence theory (upper and lower probabilities induced by multi-valued mapping) based on the concept of lower and upper bounds for a set of compatible probability distributions is used in [62] for melanoma detection. The success rates for the methods presented in the literature indicate that the work toward automated classification of lesions and melanoma may

A Review on State-of-the-Art Computer-Based Approaches …

95

provide good results. Detailed descriptions and results regarding the methods used in existing dermoscopy analysis systems are presented in [5, 120]. Molecular techniques have led to the discovery of a number of biomarkers for the detection of the disease. The biological function of these biomarkers is known, proving that linkage to histology and imaging is possible. Still only few efforts have been made to understand the molecular or histological level of association with the clinical view (image), to base the prediction on this association. Research by Zouridakis et al. [33], Claridge et al. 76], Buzug et al. [77] showed that there is a clear connection between histological and image features, suggesting a combination of parameters to be used as features for detection. Suggestions that the phenotypical characteristics of nevi are correlated to genetic variants have been given [96–98]. Some groups have used data fusion to improve predictions concerning melanoma [83, 103, 106, 108, 112, 121]. Research on the latter issue is at least scanty; therefore, further investigation is needed.

References 1. Shain, A.H., Bastian, B.C.: From melanocytes to melanomas. Nat. Rev. Cancer 16, 345–358 (2016). https://doi.org/10.1038/nrc.2016.37 2. Melanoma Research Foundation. https://www.melanoma.org/home-page. Accessed 11 Feb 2019 3. Arroyo, J.L.G., Zapirain, B.G.: Detection of pigment network in dermoscopy images using supervised machine learning and structural analysis. Comput. Biol. Med. 44, 144–157 (2014) 4. Maglogiannis, I., Kosmopoulos, D.I.: Computational vision systems for the detection of malignant melanoma. Oncol. Rep. 15, 1027–1032 (2006) 5. Mishra, N.K., Celebi, M.E.: An overview of melanoma detection in dermoscopy images using image processing and machine learning (2016). arXiv:160107843 [cs, stat] 6. Argenziano, G., Fabbrocini, G., Carli, P., De Giorgi, V., Sammarco, E., Delfino, M.: Epiluminescence microscopy for the diagnosis of doubtful melanocytic skin lesions. Comparison of the ABCD rule of dermatoscopy and a new 7-point checklist based on pattern analysis. Arch. Dermatol. 134, 1563–1570 (1998) 7. Argenziano, G., Soyer, H.P., Chimenti, S., Talamini, R., Corona, R., Sera, F., Binder, M., Cerroni, L., De Rosa, G., Ferrara, G.: Dermoscopy of pigmented skin lesions: results of a consensus meeting via the Internet. J. Am. Acad. Dermatol. 48, 679–693 (2003) 8. Betta, G., Di Leo, G., Fabbrocini, G., Paolillo, A., Scalvenzi, M.: Automated application of the “7-Point Checklist” diagnosis method for skin lesions: estimation of chromatic and shape parameters, pp. 1818–1822. IEEE (2005) 9. Ogorzałek, M., Nowak, L., Surowka, G., Alekseenko, A.: Melanoma in the clinic—diagnosis, management and complications of malignancy. In: Modern Techniques for Computer-Aided Melanoma Diagnosis (2011) 10. Manousaki, A.G., Manios, A.G., Tsompanaki, E.I., Panayiotides, J.G., Tsiftsis, D.D., Kostaki, A.K., Tosca, A.D.: A simple digital image processing system to aid in melanoma diagnosis in an everyday melanocytic skin lesion unit. A preliminary report. Int. J. Dermatol. 45, 402–410 (2006) 11. Umbaugh, S.E., Moss, R.H., Stoecker, W.V.: Applying artificial intelligence to the identification of variegated coloring in skin tumors. IEEE Eng. Med. Biol. Mag. 10, 57–62 (1991)

96

G. Kontogianni and I. Maglogiannis

12. Umbaugh, S.E., Wei, Y.-S., Zuke, M.: Feature extraction in image analysis. A program for facilitating data reduction in medical image classification. IEEE Eng. Med. Biol. Mag. 16, 62–73 (1997) 13. Tomatis, S., Bono, A., Bartoli, C., Tragni, G., Farina, B., Marchesini, R.: Image analysis in the RGB and HS colour planes for a computer-assisted diagnosis of cutaneous pigmented lesions. Tumori 84, 29–32 (1998) 14. Herbin, M., Bon, F.-X., Venot, A., Jeanlouis, F., Dubertret, M., Dubertret, L., Strauch, G.: Assessment of healing kinetics through true color image processing. IEEE Trans. Med. Imaging 12, 39–43 (1993) 15. Lohmann, W., Paul, E.: In situ detection of melanomas by fluorescence measurements. Naturwissenschaften 75, 201–202 (1988) 16. Bono, A., Tomatis, S., Bartoli, C., Cascinelli, N., Clemente, C., Cupeta, C., Marchesini, R.: The invisible colours of melanoma. A telespectrophotometric diagnostic approach on pigmented skin lesions. Eur. J. Cancer 32, 727–729 (1996) 17. Chwirot, B., Chwirot, S., Redzi´nski, J., Michniewicz, Z.: Detection of melanomas by digital imaging of spectrally resolved ultraviolet light-induced autofluorescence of human skin. Eur. J. Cancer 34, 1730–1734 (1998) 18. Sanders, J., Goldstein, B., Leotta, D., Richards, K.: Image processing techniques for quantitative analysis of skin structures. Comput. Methods Programs Biomed. 59, 167–180 (1999) 19. Ganster, H., Pinz, P., Rohrer, R., Wildling, E., Binder, M., Kittler, H.: Automated melanoma recognition. IEEE Trans. Med. Imaging 20, 233–239 (2001) 20. Grana, C., Pellacani, G., Cucchiara, R., Seidenari, S.: A new algorithm for border description of polarized light surface microscopic images of pigmented skin lesions. IEEE Trans. Med. Imaging 22, 959–964 (2003) 21. Rahman, M.M., Bhattacharya, P.: An integrated and interactive decision support system for automated melanoma recognition of dermoscopic images. Comput. Med. Imaging Graph. 34, 479–486 (2010) 22. Pehamberger, H., Steiner, A., Wolff, K.: In vivo epiluminescence microscopy of pigmented skin lesions. I. Pattern analysis of pigmented skin lesions. J. Am. Acad. Dermatol. 17, 571–583 (1987) 23. Stolz, W., Landthaler, M.: Classification, diagnosis and differential diagnosis of malignant melanoma. Der Chirurg; Zeitschrift fur alle Gebiete der operativen Medizen 65, 145–152 (1994) 24. Menzies, S.W., Ingvar, C., Crotty, K.A., McCarthy, W.H.: Frequency and morphologic characteristics of invasive melanomas lacking specific surface microscopic features. Arch. Dermatol. 132, 1178–1182 (1996) 25. Dermoscopy. http://www.dermoscopy.org/. Accessed 11 Feb 2019 26. Maglogiannis, I., Pavlopoulos, S., Koutsouris, D.: An integrated computer supported acquisition, handling, and characterization system for pigmented skin lesions in dermatological images. IEEE Trans. Inf Technol. Biomed. 9, 86–98 (2005) 27. Goodson, A.G., Grossman, D.: Strategies for early melanoma detection: approaches to the patient with nevi. J. Am. Acad. Dermatol. 60, 719–735 (2009) 28. Maglogiannis, I., Caroni, C., Pavlopoulos, S., Karioti, V., Koutsouris, D.: Utilizing artificial intelligence for the characterization of dermatological images, pp. 362–368 (2001) 29. Grana, C., Pellacani, G., Seidenari, S., Cucchiara, R.: Color calibration for a dermatological video camera system, pp. 798–801. IEEE (2004) 30. Gutenev, A., Skladnev, V., Varvel, D.: Acquisition-time image quality control in digital dermatoscopy of skin lesions. Comput. Med. Imaging Graph. 25, 495–499 (2001) 31. Abbas, Q., Garcia, I.F., Celebi, M.E., Ahmad, W.: A feature-preserving hair removal algorithm for dermoscopy images. Skin Res Technol 19, e27–e36 (2013). https://doi.org/10.1111/j.16000846.2011.00603.x 32. Yuan, X., Yang, Z., Zouridakis, G., Mullani, N.: SVM-based texture classification and application to early melanoma detection, pp. 4775–4778. IEEE (2006)

A Review on State-of-the-Art Computer-Based Approaches …

97

33. Zouridakis, G., Doshi, M., Mullani, N.: Early diagnosis of skin cancer based on segmentation and measurement of vascularization and pigmentation in nevoscope images, pp. 1593–1596. IEEE (2004) 34. Celebi, M.E., Iyatomi, H., Schaefer, G., Stoecker, W.V.: Lesion border detection in dermoscopy images. Comput. Med. Imaging Graph. 33, 148–153 (2009) 35. Chiem, A., Al-Jumaily, A., Khushaba, R.N.: A novel hybrid system for skin lesion detection, pp. 567–572 . IEEE (2007) 36. Messadi, M., Bessaid, A., Taleb-Ahmed, A.: Extraction of specific parameters for skin tumour classification. J. Med. Eng. Technol. 33, 288–295 (2009) 37. Motoyama, H., Tanaka, T., Tanaka, M., Oka, H.: Feature of malignant melanoma based on color information, pp. 230–233. IEEE (2004) 38. Abbas, Q., Garcia, I.F., Celebi, M.E., Ahmad, W., Mushtaq, Q.: A perceptually oriented method for contrast enhancement and segmentation of dermoscopy images. Skin Res. Technol. 19, e490–e497 (2013). https://doi.org/10.1111/j.1600-0846.2012.00670.x 39. Lee, T., Ng, V., Gallagher, R., Coldman, A., McLean, D.: DullRazor: a software approach to hair removal from images. Comput. Biol. Med. 27, 533–543 (1997) 40. Maglogiannis, I., Delibasis, K.: Hair removal on dermoscopy images. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2960–2963 (2015) 41. Celebi, M.E., Wen, Q., Hwang, S., Iyatomi, H., Schaefer, G.: Lesion border detection in dermoscopy images using ensembles of thresholding methods. Skin Res. Technol. 19, e252– e258 (2013). https://doi.org/10.1111/j.1600-0846.2012.00636.x 42. Jain, S., jagtap, V., Pise, N.: Computer aided melanoma skin cancer detection using image processing. Procedia Comput. Sci. 48, 735–740 (2015). https://doi.org/10.1016/j.procs.2015. 04.209 43. Taouil, K., Romdhane, N.B., Bouhlel, M.S.: A new automatic approach for edge detection of skin lesion images, pp. 212–220. IEEE (2006) 44. Mete, M., Kockara, S., Aydin, K.: Fast density-based lesion detection in dermoscopy images. Comput. Med. Imaging Graph. 35, 128–136 (2011). https://doi.org/10.1016/j.compmedimag. 2010.07.007 45. Nasir, M., Attique Khan, M., Sharif, M., Lali, I.U., Saba, T., Iqbal, T.: An improved strategy for skin lesion detection and classification using uniform segmentation and feature selection based approach. Microsc. Res. Tech. 81, 528–543 (2018). https://doi.org/10.1002/jemt.23009 46. Zhou, H., Li, X., Schaefer, G., Celebi, M.E., Miller, P.: Mean shift based gradient vector flow for image segmentation. Comput. Vis. Image Underst. 117, 1004–1016 (2013). https://doi. org/10.1016/j.cviu.2012.11.015 47. Kwasnicka, H., Paradowski, M.: Melanocytic lesion images segmentation enforcing by spatial relations based declarative knowledge, pp. 286–291. IEEE (2005) 48. Otsu, N.: A threshold selection method from gray-level histograms. Automatica 11, 23–27 (1975) 49. Arroyo, J.L.G., Zapirain, B.G., Zorrilla, A.M.: Blue-white veil and dark-red patch of pigment pattern recognition in dermoscopic images using machine-learning techniques. In: 2011 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), pp. 196–201 (2011) 50. Kaur, R., Albano, P.P., Cole, J.G., Hagerty, J., LeAnder, R.W., Moss, R.H., Stoecker, W.V.: Real-time supervised detection of pink areas in dermoscopic images of melanoma: importance of color shades, texture and location. Skin Res. Technol. 21, 466–473 (2015). https://doi.org/ 10.1111/srt.12216 51. Ruela, M., Barata, C., Marques, J.S., Rozeira, J.: A system for the detection of melanomas in dermoscopy images using shape and symmetry features. Comp. Meth. Biomech. Biomed. Eng. 5, 127–137 (2017). https://doi.org/10.1080/21681163.2015.1029080 52. Sadeghi, M., Lee, T.K., McLean, D., Lui, H., Atkins, M.S.: Detection and analysis of irregular streaks in dermoscopic images of skin lesions. IEEE Trans. Med. Imaging 32, 849–861 (2013). https://doi.org/10.1109/TMI.2013.2239307

98

G. Kontogianni and I. Maglogiannis

53. Dreiseitl, S., Ohno-Machado, L., Kittler, H., Vinterbo, S., Billhardt, H., Binder, M.: A comparison of machine learning methods for the diagnosis of pigmented skin lesions. J. Biomed. Inform. 34, 28–36 (2001) 54. Sadeghi, M., Razmara, M., Lee, T.K., Atkins, M.S.: A novel method for detection of pigment network in dermoscopic images using graphs. Comput. Med. Imaging Graph. 35, 137–143 (2011) 55. Tanaka, T., Yamada, R., Tanaka, M., Shimizu, K., Oka, H.: A study on the image diagnosis of melanoma, pp. 1597–1600. IEEE (2004) 56. Zhang, Z., Moss, R.H., Stoecker, W.V.: Neural networks skin tumor diagnostic system, pp. 191–192. IEEE (2003) 57. Hansen, G.L., Sparrow, E.M., Kokate, J.Y., Leland, K.J., Iaizzo, P.A.: Wound status evaluation using color image processing. IEEE Trans. Med. Imaging 16, 78–86 (1997) 58. Boldrick, J.C., Layton, C.J., Nguyen, J., Swetter, S.M.: Evaluation of digital dermoscopy in a pigmented lesion clinic: clinician versus computer assessment of malignancy risk. J. Am. Acad. Dermatol. 56, 417–421 (2007) 59. Stanley, R.J., Moss, R.H., Van Stoecker, W., Aggarwal, C.: A fuzzy-based histogram analysis technique for skin lesion discrimination in dermatology clinical images. Comput. Med. Imaging Graph. 27, 387–396 (2003) 60. Maglogiannis, I., Doukas, C.N.: Overview of advanced computer vision systems for skin lesions characterization. IEEE Trans. Inf Technol. Biomed. 13, 721–733 (2009) 61. Blanzieri, E., Eccher, C., Forti, S., Sboner, A.: Exploiting Classifier Combination for Early Melanoma Diagnosis Support, pp. 55–62. Springer, Berlin (2000) 62. Lefevre, E., Colot, O., Vannoorenberghe, P., de Brucq, D.: Knowledge modeling methods in the framework of evidence theory: an experimental comparison for melanoma detection, pp. 2806–2811. IEEE (2000) 63. Denoeux, T.: A k-nearest neighbor classification rule based on Dempster-Shafer theory. IEEE Trans. Syst. Man Cybern. 25, 804–813 (1995) 64. Appriou, A.: Multisensor signal processing in the framework of the theory of evidence. Tiré à part—Office national d’études et de recherches aerospatiales (1999) 65. Maglogiannis, I.G., Zafiropoulos, E.P.: Characterization of digital medical images utilizing support vector machines. BMC Med. Inform. Decis. Mak. 4, 4 (2004) 66. Abedini, M., Chen, Q., Codella, N.C.F., Garnavi, R., Sun, X.: Accurate and scalable system for automatic detection of malignant melanoma. In: Celebi, M.E., Mendonça, T., Marques, J.S.: Dermoscopy Image Analysis, pp. 293–343. CRC Press (2015) 67. Schaefer, G., Krawczyk, B., Celebi, M.E., Iyatomi, H.: An ensemble classification approach for melanoma diagnosis. Memetic Comp. 6, 233–240 (2014). https://doi.org/10.1007/s12293014-0144-8 68. d’Amico, M., Ferri, M., Stanganelli, I.: Qualitative asymmetry measure for melanoma detection, pp. 1155–1158. IEEE (2004) 69. Serrano, C., Acha, B.: Pattern analysis of dermoscopic images based on Markov random fields. Pattern Recogn. 42, 1052–1057 (2009) 70. Garcia-Arroyo, J.L., Garcia-Zapirain, B.: Recognition of pigment network pattern in dermoscopy images based on fuzzy classification of pixels. Comput. Methods Progr. Biomed. 153, 61–69 (2018). https://doi.org/10.1016/j.cmpb.2017.10.005 71. Codella, N., Cai, J., Abedini, M., Garnavi, R., Halpern, A., Smith, J.R.: Deep learning, sparse coding, and SVM for melanoma recognition in dermoscopy images. In: Zhou, L., Wang, L., Wang, Q., Shi, Y. (eds.) Machine Learning in Medical Imaging, pp. 118–126. Springer International Publishing (2015) 72. Codella, N.C.F., Nguyen, Q., Pankanti, S., Gutman, D.A., Helba, B., Halpern, A.C., Smith, J.R.: Deep learning ensembles for melanoma recognition in dermoscopy images. IBM J. Res. Dev. 61, 5:1–5:15 (2017). https://doi.org/10.1147/jrd.2017.2708299 73. Venot, A., Devaux, J., Herbin, M., Lebruchec, J., Dubertret, L., Raulo, Y., Roucayrol, J.: An automated system for the registration and comparison of photographic images in medicine. IEEE Trans. Med. Imaging 7, 298–303 (1988)

A Review on State-of-the-Art Computer-Based Approaches …

99

74. Cideciyan, A.V.: Registration of ocular fundus images: an algorithm using cross-correlation of triple invariant image descriptors. IEEE Eng. Med. Biol. Mag. 14, 52–58 (1995) 75. Kasmi, R., Mokrani, K.: Classification of malignant melanoma and benign skin lesions: implementation of automatic ABCD rule. IET Image Proc. 10, 448–455 (2016). https://doi.org/10. 1049/iet-ipr.2015.0385 76. Claridge, E., Cotton, S., Hall, P., Moncrieff, M.: From colour to tissue histology: physics-based interpretation of images of pigmented skin lesions. Med. Image Anal. 7, 489–502 (2003) 77. Buzug, T.M., Schumann, S., Pfaffmann, L., Reinhold, U., Ruhlmann, J.: Functional infrared imaging for skin-cancer screening, pp 2766–2769. IEEE (2006) 78. Edmondson, P.C., Curley, R.K., Marsden, R.A., Robinson, D., Allaway, S.L., Willson, C.D.: Screening for malignant melanoma using instant photography. J. Med. Screen. 6, 42–46 (1999). https://doi.org/10.1136/jms.6.1.42 79. Feit, N.E., Dusza, S.W., Marghoob, A.A.: Melanomas detected with the aid of total cutaneous photography. Br. J. Dermatol. 150, 706–714 (2004). https://doi.org/10.1111/j.00070963.2004.05892.x 80. Banky, J.P., Kelly, J.W., English, D.R., Yeatman, J.M., Dowling, J.P.: Incidence of new and changed nevi and melanomas detected using baseline images and dermoscopy in patients at high risk for melanoma. Arch. Dermatol. 141, 998–1006 (2005). https://doi.org/10.1001/ archderm.141.8.998 81. Lee, T.K., Atkins, M.S., King, M.A., Lau, S., McLean, D.I.: Counting moles automatically from back images. IEEE Trans. Biomed. Eng. 52, 1966–1969 (2005) 82. Wong, R., Tran, V., Talwalker, S., Benson, N.R.: Analysis of RNA recovery and gene expression in the epidermis using non-invasive tape stripping. J. Dermatol. Sci. 44, 81–92 (2006) 83. Rothberg, B.E.G., Brackenm M.B., Rimm, D.L.: Tissue biomarkers for prognosis in cutaneous melanoma: a systematic review and meta-analysis. J. Natl. Cancer Inst. (2009) 84. Gerami, P., Jewell, S.S., Morrison, L.E., Blondin, B., Schulz, J., Ruffalo, T., Matushek IV, P., Legator, M., Jacobson, K., Dalton, S.R.: Fluorescence in situ hybridization (FISH) as an ancillary diagnostic tool in the diagnosis of melanoma. Am. J. Surg. Pathol. 33, 1146–1156 (2009) 85. Wachsman, W., Zapala, M., Udall, D., Paik, A., Hata, T., Walls, L., Wong, R., Schork, N., Chang, S.: Differentiation of melanoma from dysplastic nevi in suspicious pigmented skin lesions by non-invasive tape stripping. Training 100, 100 (2007) 86. Wachsman, W., Morhenn, V., Palmer, T., Walls, L., Hata, T., Zalla, J., Scheinberg, R., Sofen, H., Mraz, S., Gross, K.: Noninvasive genomic detection of melanoma. Br. J. Dermatol. 164, 797–806 (2011) 87. Chandler, W.M., Rowe, L.R., Florell, S.R., Jahromi, M.S., Schiffman, J.D., South, S.T.: Differentiation of malignant melanoma from benign nevus using a novel genomic microarray with low specimen requirements. Arch. Pathol. Lab. Med. 136, 947–955 (2012) 88. Liu, W., Peng, Y., Tobin, D.J.: A new 12-gene diagnostic biomarker signature of melanoma revealed by integrated microarray analysis. PeerJ 1, e49 (2013) 89. Hoek, K., Rimm, D.L., Williams, K.R., Zhao, H., Ariyan, S., Lin, A., Kluger, H.M., Berger, A.J., Cheng, E., Trombetta, E.S.: Expression profiling reveals novel pathways in the transformation of melanocytes to melanomas. Can. Res. 64, 5270–5282 (2004) 90. Smith, A.P., Hoek, K., Becker, D.: Whole-genome expression profiling of the melanoma progression pathway reveals marked molecular differences between nevi/melanoma in situ and advanced-stage melanomas. Cancer Biol. Ther. 4, 1018–1029 (2005) 91. Riker, A.I., Enkemann, S.A., Fodstad, O., Liu, S., Ren, S., Morris, C., Xi, Y., Howell, P., Metge, B., Samant, R.S.: The gene expression profiles of primary and metastatic melanoma yields a transition point of tumor progression and metastasis. BMC Med. Genomics 1, 13 (2008) 92. Scatolini, M., Grand, M.M., Grosso, E., Venesio, T., Pisacane, A., Balsamo, A., Sirovich, R., Risio, M., Chiorino, G.: Altered molecular pathways in melanocytic lesions. Int. J. Cancer 126, 1869–1881 (2010)

100

G. Kontogianni and I. Maglogiannis

93. Rose, A.E., Poliseno, L., Wang, J., Clark, M., Pearlman, A., Wang, G., Medicherla, R., Christos, P.J., Shapiro, R., Pavlick, A.: Integrative genomics identifies molecular alterations that challenge the linear model of melanoma progression. Can. Res. 71, 2561–2571 (2011) 94. Iakovidis, D.K., Pelekis, N., Kotsifakos, E.E., Kopanakis, I., Karanikas, H., Theodoridis, Y.: A pattern similarity scheme for medical image retrieval. IEEE Trans. Inf Technol. Biomed. 13, 442–450 (2009) 95. Lanckriet, G.R., De Bie, T., Cristianini, N., Jordan, M.I., Noble, W.S.: A statistical framework for genomic data fusion. Bioinformatics 20, 2626–2635 (2004) 96. Cuéllar, F., Puig, S., Kolm, I., Puig-Butille, J., Zaballos, P., Martí-Laborda, R., Badenas, C., Malvehy, J.: Dermoscopic features of melanomas associated with MC1R variants in Spanish CDKN2A mutation carriers. Br. J. Dermatol. 160, 48–53 (2009). https://doi.org/10.1111/j. 1365-2133.2008.08826.x 97. Vallone, M.G., Tell-Marti, G., Potrony, M., Rebollo-Morell, A., Badenas, C., Puig-Butille, J.A., Gimenez-Xavier, P., Carrera, C., Malvehy, J., Puig, S.: Melanocortin 1 receptor (MC1R) polymorphisms’ influence on size and dermoscopic features of nevi. Pigment Cell Melanoma Res. 31, 39–50 (2018). https://doi.org/10.1111/pcmr.12646 98. Zalaudek, I., Argenziano, G., Mordente, I., Moscarella, E., Corona, R., Sera, F., Blum, A., Cabo, H., Di Stefani, A., Hofmann-Wellenhof, R., Johr, R., Langford, D., Malvehy, J., Kolm, I., Sgambato, A., Puig, S., Soyer, H.P., Kerl, H.: Nevus type in dermoscopy is related to skin type in white persons. Arch. Dermatol. 143, 351–356 (2007). https://doi.org/10.1001/ archderm.143.3.351 99. Jaffe, C.C.: Imaging and genomics: is there a synergy? Radiology 264, 329–331 (2012). https://doi.org/10.1148/radiol.12120871 100. Katrib, A., Hsu, W., Bui, A., Xing, Y.: “RADIOTRANSCRIPTOMICS”: a synergy of imaging and transcriptomics in clinical assessment. Quant. Biol. 4, 1–12 (2016). https://doi.org/10. 1007/s40484-016-0061-6 101. Kor, S., Tiwary, U.: Feature level fusion of multimodal medical images in lifting wavelet transform domain. Conf. Proc. IEEE Eng. Med. Biol Soc. 2, 1479–1482 (2004). https://doi. org/10.1109/IEMBS.2004.1403455 102. Patwardhan, S.V., Dai, S., Dhawan, A.P.: Multi-spectral image analysis and classification of melanoma using fuzzy membership based partitions. Comput. Med. Imaging Graph. 29, 287–296 (2005). https://doi.org/10.1016/j.compmedimag.2004.11.001 103. Winnepenninckx, V., Lazar, V., Michiels, S., Dessen, P., Stas, M., Alonso, S.R., Avril, M.F., Ortiz Romero, P.L., Robert, T., Balacescu, O., Eggermont, A.M., Lenoir, G., Sarasin, A., Tursz, T., van den Oord, J.J., Spatz, A.: Gene expression profiling of primary cutaneous melanoma and clinical outcome. J. Natl. Cancer Inst. 98, 472–482 (2006). https://doi.org/10. 1093/jnci/djj103 104. Ye, J., Chen, K., Wu, T., Li, J., Zhao, Z., Patel, R., Bae, M., Janardan, R., Liu, H., Alexander, G.: Heterogeneous data fusion for Alzheimer’s disease study, pp. 1025–1033. ACM (2008) 105. Metsis, V., Huang, H., Andronesi, O.C., Makedon, F., Tzika, A.: Heterogeneous data fusion for brain tumor classification. Oncol. Rep. 28, 1413–1416 (2012). https://doi.org/10.3892/or. 2012.1931 106. Kashani-Sabet, M., Venna, S., Nosrati, M., Rangel, J., Sucker, A., Egberts, F., Baehner, F.L., Simko, J., Leong, S.P., Haqq, C.: A multimarker prognostic assay for primary cutaneous melanoma. Clin. Cancer Res. 15, 6987–6992 (2009) 107. Li, Y., Patra, J.C.: Genome-wide inferring gene–phenotype relationship by walking on the heterogeneous network. Bioinformatics 26, 1219–1224 (2010) 108. Mann, G.J., Pupo, G.M., Campain, A.E., Carter, C.D., Schramm, S.-J., Pianova, S., Gerega, S.K., De Silva, C., Lai, K., Wilmott, J.S.: BRAF mutation, NRAS mutation, and the absence of an immune-related expressed gene profile predict poor outcome in patients with stage III melanoma. J. Investig. Dermatol. 133, 509–517 (2013) 109. Zinn, P.O., Majadan, B., Sathyan, P., Singh, S.K., Majumder, S., Jolesz, F.A., Colen, R.R.: Radiogenomic mapping of edema/cellular invasion MRI-phenotypes in glioblastoma multiforme. PLoS ONE 6, e25451 (2011). https://doi.org/10.1371/journal.pone.0025451

A Review on State-of-the-Art Computer-Based Approaches …

101

110. Jamshidi, N., Diehn, M., Bredel, M., Kuo, M.D.: Illuminating radiogenomic characteristics of glioblastoma multiforme through integration of MR imaging, messenger RNA expression, and DNA copy number variation. Radiology 270, 1–2 (2013). https://doi.org/10.1148/radiol. 13130078 111. Gevaert, O., Xu, J., Hoang, C.D., Leung, A.N., Xu, Y., Quon, A., Rubin, D.L., Napel, S., Plevritis, S.K.: Non-small cell lung cancer: identifying prognostic imaging biomarkers by leveraging public gene expression microarray data—methods and preliminary results. Radiology 264, 387–396 (2012). https://doi.org/10.1148/radiol.12111607 112. Moutselos, K., Maglogiannis, I., Chatziioannou, A.: Integration of high-volume molecular and imaging data for composite biomarker discovery in the study of melanoma. Biomed. Res. Int. 2014, 145243 (2014). https://doi.org/10.1155/2014/145243 113. Lazova, R., Pornputtapong, N., Halaban, R., Bosenberg, M., Bai, Y., Chai, H., Krauthammer, M.: Spitz nevi and Spitzoid melanomas: exome sequencing and comparison with conventional melanocytic nevi and melanomas. Mod. Pathol. 30, 640–649 (2017). https://doi.org/10.1038/ modpathol.2016.237 114. Hoadley, K.A., Yau, C., Wolf, D.M., Cherniack, A.D., Tamborero, D., Ng, S., Leiserson, M.D.M., Niu, B., McLellan, M.D., Uzunangelov, V., Zhang, J., Kandoth, C, Akbani, R., Shen, H., Omberg, L., Chu, A., Margolin, A.A., Van’t Veer. L.J., Lopez-Bigas, N., Laird, P.W., Raphael, B.J., Ding, L., Robertson, A.G., Byers, L.A., Mills, G.B., Weinstein, J.N., Van Waes, C., Chen, Z., Collisson, E.A., Cancer Genome Atlas Research Network, Benz, C.C., Perou, C.M., Stuart, J.M.: Multiplatform analysis of 12 cancer types reveals molecular classification within and across tissues of origin. Cell 158:929–944 (2014). https://doi.org/ 10.1016/j.cell.2014.06.049 115. Kavakiotis, I., Xochelli, A., Agathangelidis, A., Tsoumakas, G., Maglaveras, N., Stamatopoulos, K., Hadzidimitriou, A., Vlahavas, I., Chouvarda, I.: Integrating multiple immunogenetic data sources for feature extraction and mining somatic hypermutation patterns: the case of “towards analysis” in chronic lymphocytic leukaemia. BMC Bioinf. 17, 173 (2016). https:// doi.org/10.1186/s12859-016-1044-3 116. Przybyła-Kasperek, M., Nowak-Brzezi´nska, A., Simi´nski, R.: Decision fusion methods in a dispersed decision system—a comparison on medical data. In: Nguyen, N.T., Papadopoulos, G.A., J˛edrzejowicz, P., Trawi´nski, B., Vossen, G. (eds.) Computational Collective Intelligence, pp. 139–149. Springer International Publishing (2017) 117. Ercal, F., Chawla, A., Stoecker, W.V., Lee, H.-C., Moss, R.H.: Neural network diagnosis of malignant melanoma from color images. IEEE Trans. Biomed. Eng. 41, 837–845 (1994) 118. Rubegni, P., Cevenini, G., Burroni, M., Perotti, R., Dell’Eva, G., Sbano, P., Miracco, C., Luzi, P., Tosi, P., Barbini, P.: Automated diagnosis of pigmented skin lesions. Int. J. Cancer 101, 576–580 (2002) 119. Ballerini, L., Fisher, R.B., Aldridge, B., Rees, J.: A color and texture based hierarchical K-NN approach to the classification of non-melanoma skin lesions. In: Celebi, M.E., Schaefer, G. (eds.) Color Medical Image Analysis, pp. 63–86. Springer, Netherlands, Dordrecht (2013) 120. Korotkov, K., Garcia, R.: Computerized analysis of pigmented skin lesions: a review. Artif. Intell. Med. 56, 69–90 (2012) 121. Valavanis, I., Maglogiannis, I., Chatziioannou, A.: Exploring robust diagnostic signatures for cutaneous melanoma utilizing genetic and imaging data. IEEE J. Biomed. Health Inf. 190–198 (2015)

Cardiovascular Disease Stratification Based on Ultrasound Images of the Carotid Artery Efthyvoulos Kyriacou and Christos Loizou

Abstract Cardiovascular disease (CVD) can be identified through ultrasound scans of the arteries and more specific the common carotid artery (CCA). Measurement of the intima–media thickness (IMT) of the CCA is an established indicator of CVD. Several reports have indicated differences in the IMT of CCA and related then with various risk factors as well as their association with the risk of stroke. Along this direction; this chapter presents methods for the stratification of CVD based on manual and automated IMT measurements for both the left and right common carotid arteries. The results are based on a group of 1104 longitudinal ultrasound images acquired from 568 men and 536 women out of which 125 had cardiovascular symptoms (CVD). The main findings can be summarized as follows: (1) there was no significant difference between the CCA left side IMT and the right side IMT; (2) there were statistical significant differences for the IMT measurements between the normal group and the CVD group for both the left and the right sides; (3) there was an increasing linear relationship of the left and right IMT measurements with age for the normal group. Keywords Cardiovascular disease stratification · Ultrasound image analysis · Carotid arteries · Intima media complex analysis

1 Introduction Cardiovascular disease (CVD) is the largest cause of death worldwide; though over the last two decades, cardiovascular mortality rates have declined in many highincome countries [1]. Atherosclerosis, which builds up on artery walls, is the main E. Kyriacou (B) Department of Computer Science and Engineering, Frederick University, Limassol, Cyprus e-mail: [email protected] C. Loizou Department of Electrical Engineering, Computer Engineering and Informatics, Cyprus University of Technology, Limassol, Cyprus e-mail: [email protected] © Springer-Verlag GmbH Germany, part of Springer Nature 2020 I. Maglogiannis et al. (eds.), Advanced Computational Intelligence in Healthcare-7, Studies in Computational Intelligence 891, https://doi.org/10.1007/978-3-662-61114-2_7

103

104

E. Kyriacou and C. Loizou

(a) Left CCA IMC

(b) Right CCA IMC

Fig. 1 Automated ultrasound imaging IMC segmentation of the CCA: a left side (IMTmean = 0.72 mm, IMTmax = 0.89 mm, IMTmin = 0.53 mm, IMTmedian = 0.64 mm), and b right side (IMTmean = 0.56 mm, IMTmax = 0.61 mm, IMTmin = 0.39 mm, IMTmedian = 0.55 mm), respectively

reason leading to CVD and can result in heart attack and stroke [1, 2]. Identification of CVD can initially be based on artery intima–media thickness (IMT). Non-invasive measurement of the carotid intima–media thickness (IMT) can be easily done using b-mode ultrasound imaging. This is a measurement of the thickness of the innermost two layers of the arterial wall and provides the distance between the lumen–intima and the media–adventitia (see Fig. 1). The IMT can be observed and measured as the double-line pattern on both walls of the longitudinal images of the common carotid artery (CCA) [2], and it is well accepted as a validated surrogate marker for atherosclerosis disease. It is a fact that the increase in the IMT of the CCA is directly associated with an increased risk of myocardial infarction and stroke, especially in elderly adults without any history of CVD [1–3]. Thus, the IMT may be used for the screening of the population as in order to achieve stratification of cardiovascular disease [1–3]. Automated and manual measurement of common carotid IMT using b-mode ultrasound has been presented in several studies [4, 5], while a small number of studies investigated IMT measurements in both the left and right CCA as these are provided in Table 1 [6–13]. IMT was measured in both CCA sides and expressed as the mean value obtained from these measurements [6, 9]. Yet it is not entirely clear whether the IMT is equal in both CCA arteries as several studies have shown a prediction for an increase in IMT in the left CCA. A theoretical, not well investigated explanation that the increased shear stress forces in the left CCA are responsible for this inequality was described in [10, 11]. As it is well accepted that there are differences in shear stresses between the left and the right CCA, it can be supported that there are also differences in measurements as well as other mechanical characteristics between the two arteries. However, most studies presented in Table 1 were performed in adult individuals above the age of 50 years old [6–12], thus excluding the possibility to reveal the differences at an earlier stage of the disease [6]. It was also shown in [14], that the CCA IMT may be possibly used in the prediction of possible infarct side and in the prediction of potential risk of stroke by evaluating the IMT on both sides of the CCA. There have been few studies regarding the effect of carotid IMT sidedness on the various risk factors associated with carotid IMT [9, 11, 15]. Because of the different anatomical origins of the left versus the right CCA, it was speculated that hemodynamics, age, gender, blood lipid level, blood glucose level, and other risk

1998

1999

2002

2003

2007

2011

2014

Willekes [12]

Sun [13]

Rodriquez [7]

Arbel [6]

Luo [9]

Loizou [16]

1999

2007

2011

2014

Willekes [12]

Vicenzini [30]

Lee [10]

Loizou [16]

Manual/automated

Manual

Manual

Manual/Ultramark

Manual

Manual/automated

QIMT

– /M’ATH

Manual B-mode

Manual B-mode

Manual/Ultramark

Manual/automated

Manual

Method

125

149

1655

13

60

976

447

98

102

1781

16

50

1500

N

0.87(0.24)

0.83(0.24)

–

0.74(0.74)

1.23(1.03)

0.74(0.20)

0.49(0.09)

–

0.75(0.11)

0.68(0.12)

0.52(0.46)

0.88(0.25)

0.79(0.21)

–

0.86(0.23)

–

0.97(0.21)

0.71(0.46)

–

0.73(0.20)

–

0.625(0.078)

–

–

0.49(0.45)

0.85(0.19)

0.80(0.18)

0.83(0.21)

–

0.73(0.61)

1.29(1.08)

0.70(0.17)

0.47(0.09)

–

0.71(0.11)

0.66(0.12)

0.55(0.42)

0.86(0.24)

0.78(0.22)

Manual

0.81(0.18)

–

0.95(0.19)

0.73(0.69)

–

0.69(0.19)

–

0.626(0.075)

–

–

0.51(0.64)

0.84(0.20)

–

Auto

Right CCA mean (std.) (mm)

Manual

Auto

Left CCA mean (std.) (mm)

0.02

0.72

–

0.67

0.42

0.56

–

–

0.001

0.0004

0.55

0.43

0.65

Manual

p-value

0.04

–

0.001

0.03

–

0.77

0.002

0.884

–

–

0.83

0.49

–

Auto

CVD Cardiovascular events; N Number of cases investigated; S, NS indicate significantly and non-significantly different (shown underlined) at p ≥ 0.05 and p < 0.05, respectively

1997

Bots [11]

CVD group

1997

Schmidt [15]

Year

Bots [11]

Normal group

Study

Table 1 An overview of manual and automated IMT measurement studies for the left and right CCA in normal and cardiovascular disease groups

Cardiovascular Disease Stratification Based on Ultrasound … 105

106

E. Kyriacou and C. Loizou

factors would have different effects depending on whether the left or right CCA was considered [9]. In [10], the side differences in CCA IMT measurements and their prognostic values, among patients with stable coronary artery disease, were evaluated. The study showed that the left and right CCA may exhibit different prognostic values in the investigated population. Based on this information, we can investigate intima–media thickness measurement differences between the left and right CCA sides based on both manual and automated snake’s-based segmentation measurements. The initial effort and questions addressed are the following: 1. Is there a difference between CCA left-side IMT and right-side IMT for the normal group and the cardiovascular disease (CVD) group? 2. Is there a difference between left and right IMT measurements for the normal group versus the corresponding measurements for the cardiovascular disease group? 3. Is there an increase in left and right IMT measurements with age? 4. Can automated left and right IMT measurements be used to replace the manual measurements? 5. Can classification modelling be used to differentiate between the normal group, and the cardiovascular disease group based on left and right IMT measurements? The findings of studies like these may be helpful for the understanding of potential mechanisms that underlie the development of increase of carotid IMT at a relatively early stage of cardiovascular disease. This chapter is presenting the data published in [16], where at a retrospective study, manual and automated segmented IMT from 1104 subjects were analysed. IMT can be measured through the segmentation of the intima–media complex (IMC), which corresponds to the intima and media layers (see Fig. 1) of the arterial wall. A number of techniques were proposed for the segmentation of the IMC in ultrasound images of the CCA which are discussed in [4, 5]. In three recent studies performed by our group [17–19], we presented a semi-automatic method for IMC segmentation based on snakes. In [19], an automated system based on active contours and active contours without edges were proposed. The measurements were in all the above cases [17–19] made on a normalized rectangular region of interest where speckle removal had been applied [20]. In [18], we presented an extension of the integrated system proposed in [17] where also the intima and media layers of the CCA could be segmented.

Cardiovascular Disease Stratification Based on Ultrasound …

107

2 Materials and Methods 2.1 Recording of Ultrasound Images A total of 1104 B-mode longitudinal section ultrasound images of the CCA display the vascular wall as a regular pattern (see Fig. 1) that correlates with anatomical layers and was recorded. The images were acquired from 568 men and 536 women at a mean ± std age of (59.27 ± 10.79) years, out of which 125 had cardiovascular symptoms (67.83 ± 8.92 years) and the rest 979 had no symptoms (58.18 ± 10.48 years). The ages of the two groups were statistically significantly different (p = 0.0001). The images were furthermore partitioned into three different age groups. In the first group, we included 224 images from patients who were younger than 50 years old (4 with CVD). In the second group, we had 400 patients who were 50–60 (25 with CVD) years old. In the third group, we included 480 patients who were older than 60 years old (96 with CVD). A written informed consent was obtained according to the instructions of the local ethics committee. The ATL HDI-5000 ultrasound scanner (Advanced Technology Laboratories, Seattle, USA) [21] was used with a linear probe (L12-5) covering the frequency range of 12–5 MHz. Assuming a nominal frequency of 7 MHz, sound velocity propagation of 1540 m/s, and 2 cycles per pulse, we thus have a spatial pulse length of 0.44 mm (wavelength × number of cycles (0.22 mm × 2)) with an axial resolution of 0.22 mm ((spatial pulse length)/2)) [21, 22]. Images were captured at a resolution of 576 × 768 pixels with 256 grey levels. We used bicubic spline interpolation to resize all images to a standard pixel density of 20 pixels/mm [28]. Brightness adjustments of ultrasound images (see also Fig. 1) were carried out in this study based on the method introduced in [23]. This improves image compatibility by reducing the variability introduced by different gain settings, different operators, different equipment, and facilitates ultrasound tissue comparability. Algebraic (linear) scaling of the images was manually performed by linearly adjusting the image so that the median grey-level value of the blood was 0–5, and the median grey level of the adventitia (artery wall) was 180–190 [23]. The scale of the grey level of the images is ranged from 0 to 255. Thus, the brightness of all pixels in the image was readjusted according to the linear scale defined by selecting the two reference regions.

2.2 Manual IMT Measurements A cardiovascular expert manually delineated (using the mouse) the IMC [18] on all left and right longitudinal ultrasound images of the CCA after image normalization (see Sect. 2.1) and speckle reduction filtering (see the end of Sect. 2.2). The IMC was measured by selecting 20–40 consecutive points for the intima and the adventitia layers. The manual delineations were performed using a system implemented in

108

E. Kyriacou and C. Loizou

Fig. 2 Manual IMC segmentation of the CCA, using the program created by the group and distributed by LifeQ Medical Ltd.: a Right-side common carotid artery (IMTmean = 0.86 mm, IMTmax = 0.89 mm, IMTmin = 0.82 mm)

MATLAB® from our group and distributed by LifeQ medical (see Fig. 2). The measurements were performed between 1 and 2 cm proximal to the bifurcation of the CCA on the far wall over a distance of 1.5 cm starting at a point 0.5 cm and ending at a point 2.0 cm proximal to the carotid bifurcation. The bifurcation of the CCA was used as a guide, and all measurements were made from that region. The IMT was then calculated as the average of all the measurements. The measuring points and delineations were saved for comparison with the snake’s segmentation method. All sets of manual segmentation measurements were performed by the expert in a blinded manner, both with respect to identifying the subject and delineating the image. The correctness of the work carried out by a single expert was monitored and verified by at least another expert.

2.3 Speckle Reduction Filtering (DsFlsmv) For speckle reduction, the filter DsFlsmv (despeckle filter linear scaling mean variance—DsFlsmv) was first introduced in [24], evaluated for ultrasound images of the carotid in [20], and was applied prior to IMC segmentation. The filters of this

Cardiovascular Disease Stratification Based on Ultrasound …

109

type utilize first-order statistics such as the variance and the mean of a pixel neighbourhood and may be described with a multiplicative noise model [25]. The moving window size for the despeckle filter DsFlsmv was 5 × 5, and the number of iterations applied to each image was two.

2.4 Snakes Segmentation Before running the IMC snake’s segmentation algorithm, an IMC initialization procedure was carried out for positioning the initial snake contour as close as possible to the area of interest [17]. The Williams & Shah snake segmentation method [26] was used to deform the snake and segment IMC borders in each image. The snake contour, v(s), adapts itself by a dynamic process that minimizes an energy function (E snake (v, s)) defined as [26]: E snake (v(s)) = E int (ν(s)) + E image (v(s)) + E external (v(s))  = (αs E cont (v(s)) + βs E curv (v(s)) + γs E image (v(s)) + E external (v(s)))ds.

(1)

s

where E int (v(s)), E image (v(s)), E external (v(s)), E cont (v(s)), E curv (v(s)) are the internal, image, external, continuity, and curvature energies of the snake, and α(s), β(s) and γ (s) the strength, tension, and stiffness parameters, respectively. The method was proposed and evaluated in [17], in 100 ultrasound images of the CCA and more details about the model can be found there. For the Williams & Shah snake, the strength, tension, and stiffness parameters were equal to αs = 0.6, βs = 0.4, and γs = 2 respectively. The extracted final snake contours (see Fig. 1) correspond to the adventitia and intima borders of the IMC. The distance is computed between the two boundaries, at all points along with the arterial wall segment of interest moving perpendicularly between pixel pairs and then averaged to obtain the mean IMT (IMTmean ). Also, the maximum (IMTmax ), minimum (IMTmin ), and median (IMTmedian ) IMT values were calculated.

2.5 Cardiovascular Disease Classification Modelling Cardiovascular disease classification modelling was used to differentiate between the normal and the CVD group based on the left and right IMT measurements and age. The C-SVM network [27] was investigated using the Gaussian radial basis function (RBF) kernel and the linear kernel. Significantly, a better performance was obtained using the RBF kernel tuned based on the methodology proposed in [28]. More specifically, runs were performed on randomly selected groups of 100 normal and 100 CVD subjects above the age of 50. Runs were performed using the left,

110

E. Kyriacou and C. Loizou

right IMT measurements and age. For each set of parameters, the procedure was repeated for 10 times and the average results were calculated. The leave-one-out method was used for validating all the classification models. The values of c and γ were fine-tuned using a grid method on the training set as described in [28]. The performance of the classifier models was measured using the parameters of the receiver operating characteristic (ROC) metrics: true positives (TP), false positives (FP), false negatives (FN), true negatives (TN), sensitivity (SE), specificity (SP), and the area under the curve (AUC) for ROC curves. We also computed the percentage of correct classifications score (%CC) based on the correctly and incorrectly classified ROIs.

2.6 Statistical Analysis The Wilcoxon rank-sum test was used in order to identify if for each set of IMT measurements, a significant difference (S) or not (NS) exists between the extracted IMT measurements, with a confidence level of 95%. The Mann–Whitney rank-sum test was also used in order to identify significant differences between three different age groups. For significant differences, we require p < 0.05. Furthermore, box plots for the two different structures were plotted for the left and right CCA. Bland– Altman plots [13], with 95% confidence intervals, were also used to further evaluate the agreement between the left and right IMT measurements. Also, the Pearson correlation coefficient, ρ, between the left and right automated IMT measurements, as well as the manual and automated IMT measurements, were investigated, which reflects the extent of a linear relationship between two data sets. We furthermore use regression analysis to investigate the relationship between the IMT left and right CCA measurements and age.

3 Results Figure 1a illustrates an example of the automated IMC segmentations of the left side CCA (IMTmean = 0.72 mm, IMTmax = 0.89 mm, IMTmin = 0.53 mm, IMTmedian = 0.64 mm), while Fig. 1b shows the right side CCA (IMTmean = 0.56 mm, IMTmax = 0.61 mm, IMTmin = 0.39 mm, IMTmedian = 0.55 mm), respectively. The image was acquired from an asymptomatic male subject at the age of 52. It is shown, for this example that left and right IMT measurements of the CCA are different. Table 2 presents the manual and automated mean, standard deviation, median, minimum, and maximum measurements for the left and right CCA IMT for the normal (N = 976) and cardiovascular disease (N = 125) subjects. It is shown that the left CCA IMT side has slightly higher measurements than the right side for both the manual and automated measurements for both the normal and CVD subjects. However, there were no significant differences between the left- and the right-side IMT

0.73

IMTauto

0.2

0.2

0.62/0.73/0.84

0.61/0.71/0.81

0.23/0.48/0.63

0.31/0.51/0.64

0.86

IMTauto

0.2

0.2

0.70/0.82/1.06

0.71/0.81/1.01

0.31/0.53/0.71

0.39/0.53/0.71 0.72/0.89/1.09

0.71/0.85/1.05

0.68/0.78/0.90

0.66/0.75/0.89

1 Q1/Q2/Q3

0.81

0.8

0.69

0.7

0.2

0.2

0.2

0.2

Std.

0.69/0.80/0.89

0.71/0.81/0.91

0.60/0.72/0.81

0.60/0.70/0.80

1 Q1/Q2/Q3

0.36/0.53/0.66

0.33/0.52/0.66

0.23/0.48/0.60

0.20/0.47/0.62

1 Q1/Q2/Q3

Minimum

0.67/0.81/0.99

0.66/0.82/0.98

0.63/0.73/0.88

0.59/0.70/0.91

1 Q1/Q2/Q3

Maximum

0.04/0.02/0.02/0.03

0.03/0.04/0.01/0.03

0.46/0.77/0.72/0.85

0.67/0.56/0.34/0.78

p-values

IMT man , IMT auto Manual and automated left and right IMT measurements. 1 Q1, Q2, Q3 quartile values corresponding to P25%, P50%, and P75%, respectively, given for minimum, median, and maximum, respectively. Median values are given in bold. The p-values in the last column refer to Wilcoxon rank-sum test performed on the left versus right sides at p < 0.05 for the mean, median, minimum, and maximum IMT measurements. Underlined values show a significant difference

0.87

IMTman

Cardiovascular disease (CVD) subjects (N = 125)

0.74

IMTman

Normal subjects (N = 976)

1 Q1/Q2/Q3

1 Q1/Q2/Q3

Mean

Right CCA Maximum

Median

Minimum

Median

Mean

Std.

Left CCA

Table 2 Manual and automated mean, standard deviation, median, minimum and maximum values for the left and right CCA ultrasound imaging IMT in normal and cardiovascular disease subjects

Cardiovascular Disease Stratification Based on Ultrasound … 111

112

E. Kyriacou and C. Loizou

measurements for the normal group, but there were significant differences between the two sides for the CVD group. We also found higher IMT mean and median values for the subjects that already developed CVD events when compared to the normal group. Figure 3 illustrates box plots for the left- and right-side CCA IMT manual measurements (IMTLM, IMTRM) and automated measurements (IMTLA, IMTRA) for the normal (see Fig. 3a, b) CVD subjects (see Fig. 3b), respectively. No significant differences were found between the manual and the automated IMT measurements for both the normal and the CVD groups (using the Wilcoxon rank-sum test: (1)

Fig. 3 Box plots for the left and right CCA IMT manual measurements (IMTLM, IMTRM) and automated measurements (IMTLA, IMTRA) for a normal and b CVD groups, respectively. Interquartile range (IQR) values are shown above the box plots. Straight lines connect the nearest observations with 1.5 of the IQR of the lower and upper quartiles. Unfilled circles, inverted filled triangles, unfilled triangles, and filled triangles correspond to outliers with values beyond the ends of the 1.5 × IQR, whereas filled inverted triangles, unfilled circles, and unfilled squares correspond to outliers within 2.5 × IQR and 3.5 × IQR, respectively

Cardiovascular Disease Stratification Based on Ultrasound …

113

Normal group: Left CCA p = 0.72, Right CCA p = 0.67, (2) CVD group: Left CCA p = 0.92, Right CCA p = 0.64). Figure 4a illustrates a Bland–Altman plot between the left and the right automated segmentation measurements of the IMT. The difference of the two measurements between left and right IMT sides was (0.03 + 0.27) mm and (0.03 – 0.20) mm. In Fig. 4b, we show the Bland–Altman plot between the left manual versus automated IMT measurements with a difference between manual and automated measurements of (−0.02 + 0.56) mm and (−0.02 − 0.60) mm. Figure 4c illustrates the Bland– Altman plot between the right automated versus manual IMT measurements with a difference between manual and automated measurements of (−0.01 + 0.53) mm and (−0.01 − 0.55) mm, respectively. The left side of Table 3 presents the results after performing the nonparametric Wilcoxon rank-sum test, at p < 0.05, between the left and the right CCA, for both the manual and automated measurements. The right side of Table 3 shows the comparisons made for the left and right CCA for the normal group versus the CVD group. No significant differences were found for the normal group while significant differences were found for the CVD group between the left and the right CCA IMT measurements (as documented in Table 2). The Mann–Whitney rank-sum test performed between the manual and the automated IMC segmentation measurements for three different age groups (60) that showed significant differences exist for both the left and the right sides when comparing the normal group versus the CVD group for the age groups 50–60 and above 60 years of age. Table 4 displays the average values of the results from the ten runs using SVM classifier. Figure 5a, b, illustrates the results of the mean left automated (IMTLA) versus the mean right automated (IMTRA) IMT measurements using regression analysis for the normal and the CVD group, respectively. In Fig. 5a, we have an intercept of 0.0983 and a slope of 0.904 with a standard error of 0.0144 and 0.0197 of the intercept and the slope, respectively, and an F-ratio of 2849.16. The correlation coefficient was 0.88 (p = 0.11). It is shown that the IMT exhibits a linear relationship between the two sides. Figure 5a also shows that the confidence interval limits for the IMT for the left and right sides are ±0.21 mm. In Fig. 5b, we have an intercept of 0.476 and a slope of 0.481 with a standard error of 0.08985 and 0.1085 of the intercept and a slope, respectively, and an F-ratio of 2333.76. The correlation coefficient was 0.73 (p = 0.07). Furthermore, it is shown that in this study the values of the IMT in a normal carotid artery may vary between 0.20 and 0.91 mm, while the values for the CVD group may vary between 0.21 and 1.05 mm (see also Table 2), depending on age, and this is also consistent with other studies [4–14, 29]. Figure 6a presents regression plots showing a linear increase of the left automated CCA IMT with age for the normal group with a correlation coefficient ρ = 0.1 (p = 0.001), whereas Fig. 6b illustrates a linear increase of the left automated CCA IMT with age for the CVD group with a correlation coefficient ρ = 0.16 (p = 0.075) (Table 4).

114 Fig. 4 Regression lines (Bland–Altman plots) of a automated IMT left and right measurements. b IMT left automated versus manual and c IMT right automated versus manual. Normal subjects are indicated with circles while CVD subjects are indicated with crosses

E. Kyriacou and C. Loizou

NS(0.7)/S(0.02)/S(0.03) ρ = 0.49/ρ = 0.64/ρ = 0.78

S(0.02)/S(0.01)/S(0.02) ρ = 0.31/ρ = 0.53/ρ = 0.45

NS(0.55)/S(0.04)/S(0.001) ρ = 0.66/ρ = 0.42/ρ = 0.59

S(0.02) ρ = 0.83 (p = 0.01)

NS(0.23)/S(0.001)/S(0.001) ρ = 0.61/ρ = 0.33/ρ = 0.29

NS(0.39)/S(0.031)/S(0.051) ρ = 0.49/ρ = 0.65/ρ = 0.91

S(0.023) ρ = 0.84 (p = 0.001)

NS(0.43)/S(0.001)/S(0.001) ρ = 0.45/ρ = 0.77/ρ = 0.82

S(0.045) ρ = 0.33 (p = 0.001)

Normal versus CVD

Right CCA

p-values are given in parentheses. S, NS indicate significant and non-significant difference at p ≤ 0.05. Underlined values show significant difference

S(0.02) ρ = 0.47 (p = 0.001)

NS(0.7)/S(0.03)/NS(0.08) ρ = 0.45/ρ = 0.56/ρ = 0.78

S (0.0001)/S(0.001)/S(0.001) ρ = 0.37/ρ = 0.72/ρ = 0.67

NS(0.77) ρ = 0.84 (p = 0.0014)

S(0.04) ρ = 0.37 (p = 0.0001)

NS(0.56) ρ = 0.59 (p = 0.001)

Manual

Automated

Normal versus CVD

Normal S(0.004) ρ = 0.11 (p = 0.006)

Left CCA CVD

Left versus Right CCA

Table 3 Median IMT left and right, manual, and automated segmentation comparisons based on the Wilcoxon rank-sum test at p < 0.05 and the correlation coefficient, ρ, for normal (n = 976) and cardiovascular groups (CVD) (n = 125) also comparisons for three different age groups, 60 (n = 480), are given

Cardiovascular Disease Stratification Based on Ultrasound … 115

116

E. Kyriacou and C. Loizou

Fig. 5 Regression lines for the automated IMTLA versus IMTRA in (mm) with confidence interval limits of ±0.21 and confidence intervals for the correlation coefficient (dashed) for a the normal Group, and b the CVD group of all subjects investigated in this study

Fig. 6 Regression plots showing in a the linear increase of the left automated CCA IMT with age for the normal group with a correlation coefficient ρ = 0.12 (p = 0.001) and b the linear increase of the left automated CCA IMT with age for the CVD group with a correlation coefficient ρ = 0.16 (p = 0.075) (ρ given in the box corresponds to the correlation coefficient and p corresponds to the significance level) Table 4 Normal versus cardiovascular disease classification models based on the IMT manual (M) and automated (A) measurements using SVM with RBF kernels IMT Left

M/A Right

x

Classification models results CC %

Sensitivity%

Specificity%

AUC

M

59 ± 4.6

75 ± 8.5

43 ± 9.0

0.575 ± 0.0412 0.617 ± 0.0408

A

58 ± 4.2

80 ± 8.8

37 ± 4.9

x

M

58 ± 3.6

73 ± 9.9

44 ± 9.1

0.562 ± 0.0301

x

A

56 ± 3.1

54 ± 16.5

58 ± 18.5

0.593 ± 0.0402

x

x

M

59 ± 5.0

74 ± 6.3

44 ± 6.8

0.587 ± 0.0411

x

x

A

64 ± 3.5

77 ± 9.2

51 ± 5.2

0.655 ± 0.0306

x

Cardiovascular Disease Stratification Based on Ultrasound …

117

4 Discussion The results presented through these studies [16] showed that: (1) There is a difference between the CCA left-side IMT and the right-side IMT. The left-side IMT is slightly higher than the right-side IMT (see Tables 2 and 3). However, there was no significant difference between the left-side and rightside IMT measurements for the normal group. Similar findings to these were also reported in [11] by Bots et al., where they have investigated 1500 cases, in [15] by Schmidt et al., where they have investigated 50 cases, and in [12] in 16 cases (as documented also in Table 1). Moreover, the IMT measurements of this study are very close to the studies in [7, 12] as provided in Table 1. There were only two studies [7, 8] that showed significant differences between the left and the right sides for the normal group. Significant differences between the left and the right sides were shown for the CVD group for this study as well as in [12, 30] and in contrast to the findings of the study in [10] that showed no significant differences between the two sides (see Table 1). (2) Significant differences were found between the normal group and the CVD group for both the left- and the right-side IMT measurements (see Table 3). In addition, there were significant differences between the normal group and the CVD group for the age groups 50–60, and >60 years old. These findings are also in agreement with [9]. Moreover, as expected, subjects from the CVD group exhibit higher IMT measurements when compared to the normal group. This finding is also in agreement with the study of [10]. (3) An increasing linear relationship was found of the left and right IMT measurements with age for the normal group as also found in other studies [9, 16, 30, 31] (see Fig. 5a). A low linear relationship was also estimated for the CVD group (see Fig. 5b). Furthermore, there is a linear relationship between the left- and the right-side IMT measurements (see Fig. 4) for both the normal and the CVD groups and for both the manual and the automated segmentation measurements. These findings are in agreement with other studies [31–35]. It should be furthermore noted that the low correlation coefficients found in Figs. 4b and 5 are due to the scattering of the IMT measurements especially for the CVD group. (4) No statistical significant differences were found between the automated and the manual IMT measurements for both CCA sides, and therefore, automated measurements may be used to replace the manual measurements. No significant differences between the automated and the manual IMT measurements were also reported in [6, 10–12, 15]. The mean difference between the manual and the automated mean IMT measurements found in this study was 0.01 mm, which is in the same order as also found in other studies performed by our group [17, 18, 36, 37] and also documented in [4]. (5) Classification disease modelling should be used with caution to differentiate between the normal group and the cardiovascular disease group based on either the left or the right or both IMT measurements. The highest correct classification

118

E. Kyriacou and C. Loizou

score was achieved by 64 ± 3.5% using the automated left and right IMT measurements. The classification methods need further investigation in order to improve the overall correct classification performance.

References 1. Mendis, S., Puska, P., Norrving, B.: Global Atlas on Cardiovascular Disease Prevention and Control. WHO (2011) 2. Touboul, P., Hennerici, M.G., Meairs, S., Adams, H., et al.: Mannheim carotid intima-media thickness and plaque consensus (2004–2006–2011). Cereb. Dis. 34, 290–296 (2012) 3. van der Meer, I.M., Bots, M.L., Hofman, A., et al.: Predictive value of noninvasive measures of atherosclerosis for incident myocardial infarction: the Rotterdam study. Circulation 109(9), 1089–1094 (2004) 4. Loizou, C.P.: A review on ultrasound common carotid artery image and video segmentation techniques. Med. Biol. Eng. Comput. 52(12), 1073–1093 (2014) 5. Molinari, F., Zeng, G., Suri, J.S.: A state of the art review on intima-media thickness measurement and wall segmentation techniques for carotid ultrasound. Comput. Methods Program. Biomed. 100(3), 201–221 (2010) 6. Arbel, Y., Maharshak, N., Gal-Oz, A., Sapira, I., et al.: Lack of difference in the intimal medial thickness between the left and the right carotid arteries in the young. Acta Neurol. Scand. 115(6), 409–412 (2007) 7. Rodriguez Hernandez, S.A., Kroon, A.A., van Boxtel, M.P., et al.: Is there a side predilection for cerebrovascular disease? Hypertension 42(1), 56–60 (2003) 8. Sun, Y., Lin, C.-H., Lu, C.-J., Yip, P.-K., Chen, R.-C.: Carotid atherosclerosis, intima media thickness and risk factors—an analysis of 1781 asymptomatic subjects in Taiwan. Atherosclerosis 164(1), 89–94 (2002) 9. Luo, X., Yang, Y., Cao, T., Li, Z.: Differences in left and right carotid artery intima media thickens and the associated risk factors. Clin. Radiol. 66(5), 393–398 (2011) 10. Lee, S.W.-L., Hai, J.J.S.H., Kong, S.-L., Lam, Y.-M., et al.: Side differences of carotid intimamedia thickness in predicting cardiovascular events among patients with coronary artery disease. Angiology 62(3), 231–236 (2011) 11. Bots, M.L., de Jong, P.T.V.M., Hofman, A., Grobbee, D.E.: Left, right, near or far wall common carotid intima-media thickness measurements: associations with cardiovascular disease and lower extremity arterial atherosclerosis. J. Clin. Epidemiol. 50(7), 801–807 (1997) 12. Willekes, C., Hoeks, A.P.G., Bots, M.L., Brands, P.J., et al.: Evaluation of off-line automated intima-media thickness detection of the common carotid artery based on M-line signal processing. Ultrasound Med. Biol. 25(1), 57–64 (1999) 13. Bland, J.M., Altman, D.G.: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1 327(8476), 307–310 (1986) 14. Onbas, O., Kantarci, M., Okur, A., Bayraktutan, U., Edis, A., Ceviz, N.: Carotid intima-media thickness: is it correlated with stroke side? Acta Neurol. Scand. 111(3), 169–171 (2005) 15. Schmidt, C., Wendelhag, I.: How can the variability in ultrasound measurement of intima-media thickness be reduced? Studies of interobserver variability in carotid and femoral arteries. Clin. Physiol. 19(1), 45–55 (1999) 16. Loizou, C. Nicolaides, A., Kyriacou, E., Georgiou, N., Griffin, M., Paqttichis, C.: A comparison of ultrasound intima media thickness of the left and right common carotid artery. IEEE J. Trans. Eng. Health Med. 3, https://doi.org/10.1109/jtehm.2015.2450735, 2015 17. Loizou, C.P., Pattichis, C.S., Pantziaris, M., Tyllis, T., Nicolaides, A.: Snakes based segmentation of the common carotid artery intima media. Med. Biol. Eng. Comput. 45(1), 35–49 (2007)

Cardiovascular Disease Stratification Based on Ultrasound …

119

18. Loizou, C.P., Pattichis, C.S., Nicolaides, A.N., Pantziaris, M.: Manual and automated media and intima thickness measurements of the common carotid artery. IEEE Trans. Ultras. Ferroel. Freq. Contr. 56(5), 983–994 (2009) 19. Petroudi, S., Loizou, C., Pantziaris, M., Pattichis, C.: Segmentation of the common carotid intima-media complex in ultrasound images using active contours. IEEE Trans. Biomed. Eng. 59(11), 3060–3069 (2012) 20. Loizou, C.P., Pattichis, C.S., Christodoulou, C.I., Istepanian, R.S.H., Pantziaris, M., Nicolaides, A.: Comparative evaluation of despeckle filtering in ultrasound imaging of the carotid artery. IEEE Trans. Ultras. Ferroel. Freq. Contr. 52(10), 1653–1669 (2005) 21. A Philips Medical System Company: Comparison of image clarity, SonoCT real-time compound imaging versus conventional 2D ultrasound imaging. ATL Ultrasound, Report (2001) 22. Nicolaides, A., Sabetai, M., Kakkos, S.K., Dhanjil, S., Tegos, T., Stevens, J.M., et al.: The asymptomatic carotid stenosis and risk of stroke (ACSRS) study. Aims and results of quality control. Int. Angiol. 22(3), 263–272 (2003) 23. Elatrozy, T., Nicolaides, A., Tegos, T., Zarka, A.Z., Griffin, M., Sabetai, M.: The effect of Bmode ultrasonic image standardization of the echodensity of symptomatic and asymptomatic carotid bifurcation plaque. Int. Angiol. 17(3), 179–186 (1998) 24. Lee, J.-S.: Refined filtering of image noise using local statistics. Comput. Graph. Image Process. 15(4), 380–389 (1981) 25. Loizou, C.P., Pattichis, C.S.: Despeckle filtering for ultrasound imaging and video, vol. I, Algorithms and software, synthesis lectures on algorithms and software in engineering, Ed. Morgan & Claypool Publishers, San Rafael, CA, USA (2015) 26. Williams, D.J., Shah, M.: A fast algorithm for active contours and curvature estimation. Int. J. Graph. Vis. Imag. Proc. Imag. Unders. 55(1), 14–26 (1992) 27. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Tech. 2(3) (2011). Software available at: http://www.csie.ntu.edu.tw/~cjlin/libsvm 28. Kyriacou, E.C., Petroudi, S., Pattichis, C.S., Pattichis, M.S., Griffin, M., Kakkos, S., Nicolaides, A.: Prediction of high-risk asymptomatic carotid plaques based on ultrasonic image features. IEEE Trans. Inf. Tech. Biomed. 16(5), 966–973 (2012) 29. Maxwell, B.G., Maxwell, J.G., Brinker, C.C.: Left-side preference in carotid endarterectomies. Am. Surg. 66(8), 793–796 (2000) 30. Vicenzini, E., Ricciardi, M.C., Puccinelli, F., Altieri, M., et al.: Common carotid artery intimamedia thickness determinants in a population study. J. Ultr. Med. 26(4), 427–432 (2007) 31. Loizou, C.P., Pantziaris, M., Pattichis, M.S., Kyriakou, E., Pattichis, C.S.: Ultrasound image texture analysis of the intima and media layers of the common carotid artery and its correlation with age and gender. Comput. Med. Imag. Graph. 33(4), 317–324 (2009) 32. Rosfors, S., Hallerstam, S., Jensen-Urstad, K., Zetterling, M., Carlström, C.: Relationship between intima–media thickness in the common carotid artery and atherosclerosis in the carotid bifurcation. Stroke 29, 1378–1382 (1998) 33. Balasundaram, J.K., Wahida Banu, R.S.D.: A non-invasive study of alterations of the carotid artery with age using ultrasound images. Med. Biol. Eng. Comput. 44(9), 767–772 (2006) 34. Graf, S., Gariery, J., Massonneau, M., Armentano, R.L., et al.: Experimental and clinical validation of arterial diameter waveform and intimal media thickness obtained from B-mode ultrasound image processing. Ultrasound Med. Biol. 25(9), 1353–1363 (1999) 35. Litwin, M., Wühl, E., Jourdan, C., Trelewicz, J., Niemirska, A., Fahr, K., et al.: Altered morphological properties of large arteries in children with chronic renal failure and after renal transplantation. J. Am. Soc. Nephrol. 16, 1494–1500 (2005) 36. Loizou, C.P., Murray, V., Pattichis, M.S., Pantziaris, M., Pattichis, C.S.: Multiscale amplitude modulation-frequency modulation (AM-FM) texture analysis of ultrasound images of the intima and media layers of the carotid artery. IEEE Trans. Inf. Tech. Biomed. 15(2), 178–188 (2011) 37. Loizou, C.P., Kasparis, T., Lazarou, T., Pattichis, C.S., Pantziaris, M.: Manual and automated intima-media thickness and diameter measurements of the common carotid artery in patients with renal failure disease. Comput. Biol. Med. 53, 220–229 (2014)

Forecasting and Prevention Mechanisms Using Social Media in Health Care Paraskevas Koukaras, Dimitrios Rousidis and Christos Tjortjis

Abstract Social media (SM) is establishing a new era of tools with multi-usage capabilities. Governments, businesses, organizations, as well as individuals are engaging in, implementing their promotions, sharing opinions and propagating decisions on SM. We need filters, validators and a way of weighting expressed opinions in order to regulate this continuous data stream. This chapter presents trends and attempts by the research community regarding: (a) the influence of SM on attitudes towards a specific domain, related to public health and safety (e.g. diseases, vaccines, mental health), (b) frameworks and tools for monitoring their evolution and (c) techniques for suggesting useful interventions for nudging public sentiment towards best practices. Based on the state of the art, we discuss and assess whether SM can be used as means of prejudice or esteem regarding online opinions on health care. We group the state of the art in the following categories: virus–illness outbreaks, anti-vaccination, mental health, social trends and food and environment. Furthermore, we give more weight to virus–illness outbreaks and the anti-vaccination issues/trends in order to examine disease outbreak prevention methodologies and vaccination/anti-vaccination incentives, whilst discussing their performance. The goal is to consolidate the state of the art and give well-supported directions for future work. To sum up, this chapter discusses the aforementioned concepts and related biases, elaborating on forecasting and prevention attempts using SM data. Keywords Forecasting and prevention mechanisms · Social media (SM) · Health care · Vaccinations · Monitoring

P. Koukaras · D. Rousidis · C. Tjortjis (B) The Data Mining and Analytics Research Group, School of Science and Technology, International Hellenic University, 14th Km Thessaloniki, Moudania, 570 01 Thermi, Greece e-mail: [email protected] P. Koukaras e-mail: [email protected] D. Rousidis e-mail: [email protected] © Springer-Verlag GmbH Germany, part of Springer Nature 2020 I. Maglogiannis et al. (eds.), Advanced Computational Intelligence in Healthcare-7, Studies in Computational Intelligence 891, https://doi.org/10.1007/978-3-662-61114-2_8

121

122

P. Koukaras et al.

1 Introduction Social media (SM) are being used by more and more people every day, since they offer various services and features. This new ecosystem of abundant information can be used as the perfect place for establishing new social connections, to propagate information, to share opinions, etc. The rapid growth of social media networks (SMNs) initiated a new era for data analytics [1]. All this information can be utilized in order to monitor, evaluate and hopefully prevent any type of hazardous incidents [2]. Thus, this chapter attempts to assess the effectiveness of forecasting and prevention mechanisms using SM data in various healthcare domains. We refer to the following healthcare categories: virus–illness outbreaks, anti-vaccination, mental health, social trends and food and environment all associated with SM data, yet focusing on the last one (virus–illness outbreaks) since it seems to have greater socio-economic impact, as detailed in Sects. 3.1–3.3. “To our community, Andrew Wakefield is Nelson Mandela and Jesus Christ rolled up into one”. The above statement was made by J. B. Handley back in 2010 [3], co-founder of Generation Rescue, a group that disputes vaccine safety. It is a strong indication of the mentality, the dedication and the obstinacy of the anti-vaccination movement. We focus on the influence of SM on attitudes towards healthcare issues, monitoring their evolution and useful interventions for nudging public sentiment towards best practices. Having researched the state of the art, we assess whether SM can be used as a means of prejudice or esteem regarding healthcare opinions. We survey and examine outbreak prevention methodologies and focus on vaccination incentives. The review of the literature demonstrates that there is an increase in the number of healthcare-related terms in posts at SM after a disease outbreak. However, there is still no evidence that increased SM traffic can lead to an outbreak and there is not a highly effective forecasting model that could predict an outbreak by harnessing SM data [4, 5]. We make a bold note on a court decision in Italy that adjudicated compensation because of the measles, mumps, and rubella (MMR) vaccine, and a Chinese vaccine distribution scandal resulted in the reduction of vaccination in kids and therefore in an increase of virus-related symptoms [6, 7]. Thus, SM can reflect attitudes towards health issues, such as vaccinations. Appropriate interventions can raise awareness and improve expected coverage of immunization campaigns. As for the theoretical part of implementing any prevention or forecasting on health care, we refer to the importance of understanding the positive healthcare attitudes developed within SM. In order to do so, we need to collect and observe the data. User discussions with pro and anti-attitudes can be captured concluding that alternative methods of using authoritative sources are needed when correcting misleading claims. It is evident that those who are against “good” practices demonstrate conspiratorial beliefs, do not trust the government and are “resolute and in-group focused in language” [8].

Forecasting and Prevention Mechanisms Using Social …

123

The chapter is structured as follows: (a) the literature review that surveys forecasting and prevention attempts on virus–illness outbreaks, anti-vaccination, mental health, social trends and food and environment, (b) evaluation of impact on science, economy and society, (c) directions for future work and (d) conclusion noting the key points of this work.

2 Literature Review This section reviews the state of the art that focuses on aspects concerning the impact of SM networks on vaccination decisions [9], virus–illness outbreaks, antivaccination, mental health, social trends and food and environment. We refer to these categories by grouping research attempts on these domains, but we first focus on virus–illness outbreaks which seem to have greater socio-economic impact.

2.1 Virus–Illness Outbreaks Any virus–illness outbreak can cost millions of lives in case it gets out of control. Thus, researches do their best to take advantage of any tool available to prevent such catastrophes. We live in an era of abundant information, and oftentimes power shifts from doctors to patients result in actions like skewing science, shifting hypotheses, censoring dissent and attacking critics. Back in 2013, Z. Jakab, WHO Regional Director for Europe, said that “Considering the human costs of measles, a preventable disease that can lead to long-term health complications and even death, we cannot afford to be complacent” [10]. Two major studies by [11] and [12] concluded that “standard guidance for preventing measles transmission in healthcare settings includes (1) increasing measles awareness amongst providers, especially amongst persons presenting with fever, rash and travel history; (2) ensuring all HCP has evidence of measles immunity at the time of employment and has such data electronically available at the work site; (3) allowing only HCP with evidence of measles immunity to provide care to patients with measles; and (4) instituting a screening plan to identify suspected measles cases for immediate isolation during a measles outbreak”. So, “immunization against misinformation” becomes a necessity, combated with better education, emotional appeals and harnessing SM itself by Web-based decision aids, real-time tracking attitudes or SM campaigns [13]. This statement emphasizes the power of SM as a tool for mitigating virus–illness outbreaks. Next, we will refer to viruses/illnesses and the state of the art concerning them.

124

2.1.1

P. Koukaras et al.

Influenza

Santillana et al. [14] proposed a methodology that can combine search, SM and traditional data sources, like real-time hospital visits, in order to improve influenza surveillance. Their model produced accurate weekly influenza-like illness predictions for up to four weeks ahead of Centers for Disease and Prediction and one week ahead of Google Flu Trends. This study confirmed that combining multiple sources produces better results by using each source independently. Traditional data are costly and have limited coverage and low sensitivity to emerging public health issues. In [15], researchers used SM data to employ natural language classifiers to examine and analyse behavioural patterns regarding influenza in three dimensions: temporality (by week), geography (by region) and demography (by gender). SM can mislead, but also can serve as a corrective to false information.

2.1.2

Fever Outbreaks

Subramani et al. [16] examined whether it is possible to combine text mining and real-time analytics of Twitter data in order to predicting hay fever outbreaks in Australia. The researchers managed to identify the impact of specific weather and climate variables that assist future forecasting that with the aid of hay fever related tweets can assist future forecasting hay fever outbreaks.

2.1.3

Wild Poliovirus

A similar study was conducted by [17]. Triggered by a wild poliovirus type 1 (WPV1) outburst in May 2013 in Israel, they examined the importance of social networking in a national vaccine campaign. They found that SM are increasingly significant for addressing vaccine concerns and they urged those involved in the healthcare sector to be using SM as a means to inform the public.

2.1.4

Measles

In [18], the authors collected data from Twitter for a period of 1 month before and 2 months after a measles outbreak. The authors managed to identify an increase in the number of vaccine-related tweet messages during the outbreak, with positive messages “dramatically increasing”. However, according to the graph that they provided for this 3-month period, it is shown that the negative vaccine tweets are, constantly, at least twice in volume as the positive ones, and therefore there is no observation of an increase to vaccine-related tweets prior to the outbreak. Radzikowski et al. [19] presented a quantitative study of Twitter narrative after a measles outbreak in 2015. They collected around 670 k tweets from across the globe,

Forecasting and Prevention Mechanisms Using Social …

125

referring to vaccination from 1 February 2015 and for a 40-day period. They identified the dominant terms, the communication patterns for retweeting, the narrative structure of the tweets, the age distribution of those involved and the geographical patterns of participation in the vaccination debate in SM. The most important result from this research was that there is a strong connection between the engagements of Twitter users in vaccination debates and non-medical exemption from schoolentry vaccines. More specifically, they provided evidence that “Vermont and Oregon with the highest rates of exemption from mandatory child school-entry vaccines had notably higher rates of engagement in the vaccination discourse on Twitter”. However, these tweets were collected after the measles outbreak, and therefore there is no evidence that the outbreak could be predicted.

2.2 Anti-vaccination After referring to virus and illness outbreaks, we elaborate on issues regarding their mitigation/tackling. Thus, this subsection discusses on state of the art that deals with anti-vaccination threats like vaccine refusal and misinformation. Although there are organizations observing disease outbreaks for EU/EEA countries, there are still fatalities [20]. At the same time, research concerning anti-vaccination movement sites concludes that this type of sites relies heavily on emotional appeal [21]. Another work studies anti-vaccine movements and why vaccine-related controversies arise. For example, some people oppose vaccination to support specific political or cultural agendas. To approach this anti-vaccination ideology, three (3) concepts are proposed to refer to vaccine criticism: the anti-vaccine movement, the marginally anti-vaccine movements and the occasionally vaccine-critical movements [22]. Moran et al. [23] selected 480 anti-vaccine websites (210 of them were news sites or blogs, and only 9 were SM and studied the persuasive tactics used to influence parents who search information about vaccines on the Web). The research provided an insight and a better understanding about the channels and the strategies that people against vaccination use and thus offered additional tools to the proponents of vaccination for confronting the anti-vaccine movement. A major research has been conducted by Tomeny et al. [24] regarding the temporal trends, the geographic distribution and the demographic connection of anti-vaccine beliefs in Twitter for a 6-year period from 2009 up to 2015. According to their results, the volume of anti-vaccine tweets was steady from 2009 to 2014 and there were noticeable spikes caused by vaccine-related news such as measles outbreak in California that commenced on December 2014. The last spike of anti-vaccine tweets however follows the outbreak; therefore, the research did not show a connection between the tweets and the outbreak and thus failed to predict it.

126

2.2.1

P. Koukaras et al.

Vaccine Refusal

Vaccine refusal is another great challenge for efforts being done to eradicate vaccinepreventable diseases where clusters of non-conformers paradoxically develop due to successful control efforts [25]. SM is used as a very effective medium for interaction between proponents of medical products and technologies and the public, but at the same time is a tool for more organized proposition [26]. An illegal Chinese vaccine distribution network in 2016 triggered the study by [27]. The authors found that 78% of the 28,000 respondents received information by the SM exclusively. Despite the fact that the respondents maintained their trust in scientific organizations and professionals by 81.6% and 88%, respectively, this vaccine scandal made them lose their trust and confidence in vaccines and resulted in reluctance by 43.7% to vaccinate their children. A study in Italy [28] tried to explore the connection between MMR vaccination coverage and online search trends and SM activity for a 5-year period from 2010 to 2015. The results showed a significant negative correlation between the MMR vaccination coverage and Internet search query data (p = 0.043), tweets (p = 0.013) and posts (p = 0.004). The trigger event was a court decision in the country that awarded vaccine injury compensation for a case of autism. Therefore, as the authors concluded, it was not possible to demonstrate if the decrease of the percentage of Italians that are been vaccinated that was observed after 2013 is related to the increase of the anti-vaccination tweets in Twitter and posts in Facebook.

2.2.2

Misinformation

Research suggests that vaccine-selective information in general information websites tends to be more persuasive than anti-vaccine websites that contain more radical views [29]. Fear of resurgence of vaccine-preventable diseases is expressed if SM were to act as a “misinformer” for the healthcare system [30]. Evrony and Caplan [31] provide evidence of the misleading and false information disseminated by a specific organization with more than 3000 followers in Facebook. In a study by Faasse et al. [32], the language that was used in Facebook comments, both pro- and anti- vaccination, was analysed. Their results were very intriguing, as they identified that anti-vaccination comments demonstrated a more analytical thinking and they were more scientific-based with more references to medical language. On the other hand, the pro-vaccination comments showed greater stress and anxiety and were more oriented on family-related content and social processes. Twitter data proved to be an optimistic medium to explain differences in coverage that were not explained by socio-economic factors. Vaccine coverage diminished in states where misinformation, conspiracies and safety concerns had greater levels of exposure; thus, negative opinions for vaccines in SM affect vaccine acceptance [33]. In [34], the authors investigated whether SM, mainly Facebook and Twitter, can be used to correct health misinterpretations. They identified the differences between Facebook and Twitter, like the one-side associations, the hashtag conventions, the

Forecasting and Prevention Mechanisms Using Social …

127

popularity, the levels of use, etc. Their results showed that SM can offer effective correction of misinformation and reduction of misperceptions regarding viruses and vaccines. They also proved that including the source to a SM post had a stronger positive effect on the readers. SM contains misinformation for parent vaccination decisions. Officials try to keep high immunization rates to prevent outbreaks, often fighting against trends that reflect tensions between personal liberty and public health. A framework with Web-based interventions to help parents/SM users make evidence-based decisions is proposed [35].

2.3 Mental Health There are many SM users that propagate information regarding their mental stability or any mental issues they might face. The following research attempts deal with this idea of using SM in order to predict, monitor and detect mental health of their users. Gao et al. [36] created a predictive computational model of text specificity using data from Twitter. Their objective was to employ the model for mental health prediction purposes. As stated by the researchers and was validated by the results of the evaluation, they were able to reach a high accuracy on identifying people with depression and they managed to find that people with moderate/severe depression would write less specific SM posts. In [37], McClellan used SM and, most specifically, Twitter, to monitor mental health discussion. They collected around 176 million tweets from 2011 to 2014 with content associated with suicide or depression. They managed to identify expected spikes in tweet volume following a behavioural health event, like “World Suicide Prevention Day”, “Bell Let’s Talk Day”—“a programme designed to break the silence around mental illness and support mental health across Canada” or even a famous person’s suicide, like the one of Robin Williams. At the same time though, they succeeded in detecting unexpected increases in response to unanticipated events. According to the researchers, their model provides “an objective and empirically based measure to identify periods of greater interest for timing the dissemination of credible information related to mental health”. Ricard et al. [38] explored the utility of user-generated content from questionnaires and community-generated content from Instagram for detecting depression. The authors concluded that the combination of data from the two sources can be more informative, as they had significantly statistically better performance for prediction depression than using the data separately, as the two sources complement each other with non-overlapping information.

128

P. Koukaras et al.

2.4 Social Trends SM users share their social trends which oftentimes become viral, yet resulting in dangerous outcomes. Researchers are trying to find ways of detecting these trends and predict/plan mitigation actions. El Tayeby et al. [39] provided a feasibility study on identifying drinking-related contents in Facebook through mining heterogeneous data, such as text, images and videos. The authors examined if it is achievable to detect a dangerous social trend, binge drinking in students. According to their results, the researchers were able to identify drinking-related posts and build accurate prediction models. In [40], Baghel et al. proposed a methodology for detecting another seriously dangerous trend, the Kiki challenge, which is a viral dance video challenge, where people are dancing in the tune of a song in unsafe places like a moving car and roadsides, in traffic, etc. Their method managed to identify potentially dangerous videos and distinguish them from non-dangerous ones, thus safeguarding public safety. The objective of the study by Zhan et al. [41] was to identify SM-active e-cigarette users and any e-cigarette use patterns for existing or potential users. The researchers collected 332,906 e-cigarette-related posts from reddit. According to their findings, user-generated content extracted from online communities was linked to the intentions of using e-cigarettes and positively related to the ratings of associated products like devices and liquids. The following study reinforces the theory that healthcare decisions in general and vaccination decisions specifically are made within a social context [42]. The role of blogs on the shaping of public opinion was examined by Burke-Garcia et al. [43]. The authors studied the influence of “mommy bloggers”, which are defined as “mothers who blog about their children, motherhood, parenting or related topics” on human papillomavirus (HPV) vaccine, a STI that infects mostly young people, aged 15– 24 years old. According to their findings, these bloggers, which are around 3.9 million in the USA alone, are considered as key/trusted sources amongst their followers, who can be thousands for a single blogger. Another study regarding social network influence on parents’ vaccine decisions points out that the percentage of parents’ network members recommending nonconformity was more predictive of parents’ vaccination decisions than any other variable including parents’ own perceptions of vaccination [44]. In [45], researchers studied data from articles that were shared via Twitter, they identified clusters of significant vaccine concepts within both positive and negative semantic networks, and they also provided tables with central concepts of each sentiment network ranked by eigenvector centrality score. They concluded that an augmentation of the scope and variability of current attitudes and beliefs towards vaccines can be achieved with a proper semantic network analysis of vaccine sentiment in online SM.

Forecasting and Prevention Mechanisms Using Social …

129

2.5 Food and Environment Studies have been conducted in the domains of food and environment SM trends. SM users post their dietary options generating opportunities for researchers to track food quality, habits, etc. Respectively, posts regarding quality of air, sea pollution, waste, etc., can offer similar opportunities. De Choudhury et al. [46] mined data from Twitter in order to investigate whether SM platforms can provide empirical quantitative evidence for studying and understanding the dietary choices and nutritional languages of people living in urban areas where it is difficult to buy affordable or good-quality fresh food, the so-called food desert. Their proposed predicting model demonstrated a high accuracy, over 80%, and outperformed baseline methods by 6–14%. Another food-related study was conducted by Sadilek et al. [47], who examined whether it is possible to predict and prevent food-borne illnesses by applying data mining techniques on data collected from Twitter. They gathered around 16 k foodvenue-related tweets within a five-day period, and they created a tool that identifies dangerous and problematic food venues that their poor food quality can lead to foodborne illnesses. Their tool could provide a huge improvement to the identification of this potential health issues, compared to the public health controls. According to the researchers’ predictions, the use of their tool can prevent more than 9000 cases of food-borne illness and more than 550 hospitalizations per year in the area of Las Vegas where the tweets were mined from. In their research, Chen et al. [4] are studying one of the most severe sequences of environmental pollution that leads to dangerous health issues: smog. Using data from one of the biggest and popular Chinese microblogging platforms, weibo.com, and combining them with data retrieved from physical satellite and ground sensors, they proposed an artificial neural network (ANN) smog predictive model. According to their findings, combining SM data and physical data performs better than using those data independently and improves prediction of smog-related disasters. This section showcased research attempts on various categories on health care. We could say that the possibilities for new research are evident. As long as SM users stay active (post, share, propagate), research can be done regarding monitoring, trends, forecasting and prevention attempts.

3 Evaluation of Impact on Science, Economy and Society The literature review section noted important attempts on specific topics on health care in conjunction with SM, but what could actually be the benefit of utilizing such research attempts for science, economy and the community members? The answer to this question is to be discussed and presented on this section. According to numerous studies (detailed in Sects. 3.1–3.3), many countries pay a high toll in terms of both human lives lost and working hours and financially because of diseases and virus outbreaks.

130

P. Koukaras et al.

3.1 Science Impact As stated earlier, further exploitation of research areas regarding SM health care can lead to numerous groundbreaking scientific breakthroughs. High-performing forecasting, prevention frameworks/mechanisms, etc., are such examples. We start by referring to measles, which is a highly contagious disease with a high global burden that has been eliminated in many countries [40]. However, despite substantial disease containment, outbreaks keep on reappearing [36, 37, 41]. One of the reasons for the various outbreaks, even in countries with high GDP [38] and very high health and well-being ranking [48], is the emergence of the anti-vaccination movement [8, 13, 23, 31, 32, 43, 45]. The role of SM is diverse, yet channelled in two main streams: initially for raising awareness on anti-vaccination adverse effects, but at the same time for manipulating and spreading fake news, thus enlarging the anti-vaccination movement. Any implementation of a forecasting and prevention mechanism contributes to monitoring public attitudes and predicting various events using SM analytics, as well as interventions using influencer injection techniques.

3.2 Economic Impact As for the economic burden of a healthcare incident, there are numerous statistics enhancing the importance of taking actions that may include SM methods/models/mechanisms. For example, measles outbreaks resulted in costs of several millions USD [10–12, 49–53]. Each patient treatment costs from several hundreds [50] up to several thousands of dollars [51], without counting indirect costs, such as the loss of working hours [49]. Therefore, raising awareness using SM can aid countries economically. More beneficial results could emerge by providing a predictive mechanism using SM data. In detail, an outbreak in the Netherlands in 2013–14 costs $4.7 million [11]. The results indicate that “5% decline in MMR vaccine coverage in the USA would result in an estimated threefold increase in measles cases for children aged between 2 and 11 years nationally every year, with an additional $2.1 million in public sector costs” [53]. These numbers would be substantially higher if unvaccinated infants, adolescents and adult populations were also considered. Over 2002–2003, the direct costs of measles incurred by the national health service of Italy were e17.6–22 million. This would have paid for vaccinating up to 1.9 million children, which would also have prevented many cases of mumps and rubella. The 5154 hospitalizations during this period cost about e8.8 million [51]. Doctors point out the need of an immediate diagnosis as “instituting immediate airborne isolation and ensuring rapidly retrievable measles immunity records for HCPs are paramount in preventing healthcare-associated spread and in minimizing hospital outbreak response costs” [11].

Forecasting and Prevention Mechanisms Using Social …

131

The WHO urges that not only the direct costs should be considered [10]. There are other indirect costs that also should be considered, like children’s missed school days leading to loss of educational momentum. At the same time, parent should stay at home to care for their children leading to taking a leave of absence. For instance, a US outbreak 15.120 h were lost in furloughs [49]. According to [10], an outbreak in Duisburg, Germany, in 2006 led to the 311 schoolchildren affected missing total of 2854 days of school and the 30 employed adults affected losing a total of 301 days of work [25]. A study of 10 western European countries revealed that a mother misses 8–24 h of work in caring for a child with an uncomplicated case of measles [52].

3.3 Social Impact Another important takeaway lesson is the actual value of alerting society regarding ways of protecting/shielding themselves from diseases/viruses/bad habits, etc. Whilst the priority of the healthcare system is to minimize deaths by raising awareness regarding diseases like measles and polio, and the effectiveness of vaccines, research pointed out the need for prediction or even an alert of a virus outbreak [11]. Monitoring SM data for keywords like measles, viruses, vaccines, etc., could also provide a warning [11]. The need of an immediate public notification system is indicated by Chen et al. [49]. Having stated paradigms from possible socio-economic benefits, as well as reasoning for more scientific breakthroughs on this domain, we will next provide directions for future work.

4 Directions for Future Work This section discusses mitigation attempts and highlights important concepts in order to tackle healthcare issues utilizing SM. One direction is virus–illness outbreak prevention by implementing a framework that monitors SM data and injects positive trends regarding vaccination. Such efforts for publicly available tools that facilitate real-time monitoring of vaccine conversation and sentiments are already being made [54] with lots of room for improvements. Another important topic is communicating vaccine science to the public. Fear and lack of knowledge generate more challenges. Suggestions on how to defend the methods and results of scientific investigation were proposed in [55] which can be used for developing a framework mitigating such issues. Combining data from SM features and the Web can improve the performance of forecasting algorithms [56–58]. Any attempt on forecasting and prevention on health care should comply with the international guidance on developing complex interventions [59] and exhibit enhanced capacity to influence peoples’ immunization/health issue knowledge, attitudes and beliefs, as well as to promptly identify peaks in

132

P. Koukaras et al.

anti-immunization/health issue trends that can contribute to decreased immunization adherence. Essential components are: • Improved forecasting algorithms, such as the ones mentioned in [5, 60, 61], along with various real public-life variables (like a false research study, a court decision, a vaccine-related scandal) to enhance the chances for a virus outburst warning, blunting the various economic and social implications of such outbursts. • An intervention layer in which there are hybrid SM intervention approaches in other domains like [62–64] that capitalize on SM real-time peer-to-peer posts, supported by daily automatic messages sustaining and directing the information content, SM training group formations, etc. • New technologies can speed up information sharing whereas contributing to vaccine hesitancy, refusals and disease outbreaks. A categorization by concerns, vaccine, disease, location, source of report and overall positive or negative sentiment towards vaccines is presented in [65], applying a priority level and using descriptive statistics to disaggregate data by country and vaccine type, and monitor evolution of events in time and location with high vaccine concerns. The essences of time and effectiveness could help immunization programmes to address specific public concerns. Further inquiries could be made regarding these issues. • Metrics that can be used for defining social network graphs in two ways, identifying the most influential users and calculating their impact score for further analysis by public health experts, giving a better insight into concern diversity in different regions, reason for hesitancy and the role of opinion leaders in the broader vaccination debates [66]. • Results from [67] show that algorithmic and social corrections are equally effective in limiting misperceptions and correction occurs for both high and low conspiracy belief individuals. SM campaigns are recommended for use, to correct global health misinformation. All in all, the attempt should be able to detect, monitor, predict and mitigate disease outbursts. After all, even the slightest prediction of a virus outbreak or healthcare incident can prevent deaths, decrease healthcare costs, reduce hours on furlough due to taking care of sick kids, etc.

5 Conclusion We conclude that the literature is inconclusive on which method is most appropriate for effective SM-based interventions to raise awareness towards vaccinations and other healthcare issues. However, the correlation of real-life events and virus outbursts is evident. A general guideline implies that combining multiple sources produces better results than using each source independently, but it is not always possible.

Forecasting and Prevention Mechanisms Using Social …

133

To support this statement, we take vaccination as an example. We observe an increase in the number of vaccine-related terms in posts on SM after a disease outbreak, but is it possible to effectively predict the outbreak? The desirable intervention should be the actual prevention of the outbreak. Small groups are usually formed to force “gentle” propaganda against vaccine non-conformers [9]. New technologies offered by social networks and the Web generate new threats and opportunities, so effective communication strategies should be implemented to tackle vaccine hesitancy [68]. Efforts to increase perceived vaccine safety are being made using Web analytics to identify, track and neutralize antivaccine sentiment to fortify public health communication and education strategies about flu vaccines [69]. SM can be related to vaccination rates. Monitoring Twitter and other SM channels showed that negative opinions spiked when vaccine batches were recalled, whilst positive views dominated after vaccine was shipped easing public worries [70]. Although we reported numerous such research attempts, the final “verdict” remains adjourned: there is still no evidence that increased SM traffic can lead to an outbreak and there is no forecasting model that could predict the outbreak based on SM, although the correlation of real-life situations and virus outbursts is evident through the paradigms reported in this chapter. It is however possible to get better results by focusing on specific types of SM, the ones better suited to run information campaigns, on a par with the evolution of SM types [71].

References 1. Koukaras P., Tjortjis C.: Social media analytics, types and methodology. In: Tsihrintzis G., Virvou M., Sakkopoulos E., Jain L. (eds.) Machine Learning Paradigms. Learning and Analytics in Intelligent Systems, vol. 1. Springer, New Yory (2019) 2. Rousidis D., Koukaras P., Tjortjis C.: Social Media Prediction A Literature Review, to appear at Multimedia Tools and Applications, Springer. (2020). https://doi.org/10.1007/s11042-01908291-9 3. Dominus, S.: The crash and burn of an autism guru. New York Times 2011 April 20. http:// www.nytimes.com/2011/04/24/magazine/mag-24Autism-t.html. Last Accessed 23 Sep 2017 4. Chen, J., Chen, H., Wu, Z., Hu, D., Pan, J.Z.: Forecasting smog-related health hazard based on social media and physical sensor. Inf. Syst. 64, 281–291 (2017) 5. Zhao, L., Chen, F., Lu, C. T., & Ramakrishnan, N. (2015, June). Spatiotemporal event forecasting in social media. In: Proceedings of the 2015 SIAM International Conference on Data Mining, pp. 963–971. Society for Industrial and Applied Mathematics 6. Stewart, A. M. (2009). Vaccines and Autism: The Evidence and the Law 7. Qiu, J., Hu, H., Zhou, S., Liu, Q.: Vaccine scandal and crisis in public confidence in China. Lancet 387(10036), 2382 (2016) 8. Mitra, T., Counts, S., Pennebaker, J.W.: Understanding Anti-Vaccination Attitudes in Social Media. In: ICWSM, pp. 269–278. (2016) 9. Brunson, E.K.: The impact of social networks on parents’ vaccination decisions. Pediatrics (2013). (peds-2012) 10. World Health Organization: Measles Costs 2013. Retrieved from http://www.euro.who.int/en/ media-centre/sections/press-releases/2013/04/measles-costs/ (2017)

134

P. Koukaras et al.

11. Suijkerbuijk, A.W., Woudenberg, T., Hahné, S.J., Lochlainn, L.N., de Melker, H.E., Ruijs, W.L., Lugnér, A.K.: Economic costs of measles outbreak in the Netherlands, 2013–2014. Emerg. Infect. Dis. 21(11), 2067 (2015) 12. Marx, G.E.: Public health economic burden associated with two single measles case investigations—colorado, 2016–2017. MMWR. Morb. Mortal. Wkly. Rep. 66 (2017) 13. Kata, A.: Anti-vaccine activists, Web 2.0, and the postmodern paradigm–An overview of tactics and tropes used online by the anti-vaccination movement. Vaccine. 30(25), 3778–3789 (2012) 14. Santillana, M., Nguyen, A.T., Dredze, M., Paul, M.J., Nsoesie, E.O., Brownstein, J.S.: Combining search, social media, and traditional data sources to improve influenza surveillance. PLoS Comput. Biol. 11(10), e1004513 (2015) 15. Huang, X., Smith, M.C., Paul, M.J., Ryzhkov, D., Quinn, S.C., Broniatowski, D.A., Dredze, M.: Examining patterns of influenza vaccination in social media (2017) 16. Subramani, S., Michalska, S., Wang, H., Whittaker, F., Heyward, B.: Text mining and realtime analytics of twitter data: a case study of australian hay fever prediction. In: International Conference on Health Information Science, pp. 134–145. Springer, Cham (2018) 17. Rubin, L., Landsman, K.: The importance of social networking in a national polio vaccine campaign. Pediatrics. e20154658 (2016) 18. Dredze, M., Broniatowski, D.A., Smith, M., Hilyard, K.M.: Understanding vaccine refusal: why we need social media now. Am. J. Prev. Med. 50(4), 550 (2016) 19. Radzikowski, J., Stefanidis, A., Jacobsen, K.H., Croitoru, A., Crooks, A., Delamater, P.L.: The measles vaccination narrative in Twitter: a quantitative analysis. JMIR Public Health Surveill. 2(1) (2016) 20. European Centre for Disease and Control Prevention: Retrieved from: https://ecdc.europa. eu/en/news-events/epidemiological-update-measles-monitoring-european-outbreaks-15September-2017 (2017) 21. Wolfe, R.M., Sharp, L.K., Lipsky, M.S.: Content and design attributes of antivaccination web sites. JAMA 287(24), 3245–3248 (2002) 22. Ward, J.K.: Rethinking the antivaccine movement concept: a case study of public criticism of the swine flu vaccine’s safety in France. Soc. Sci. Med. 159, 48–57 (2016) 23. Moran, M.B., Lucas, M., Everhart, K., Morgan, A., Prickett, E.: What makes anti-vaccine websites persuasive? A content analysis of techniques used by anti-vaccine websites to engender anti-vaccine sentiment. J. Commun. Healthc. 9(3), 151–163 (2016) 24. Tomeny, T.S., Vargo, C.J., El-Toukhy, S.: Geographic and demographic correlates of autismrelated anti-vaccine beliefs on Twitter, 2009–15. Soc. Sci. Med. (2017) 25. Omer, S.B., Orenstein, W.A., Koplan, J.P.: Go big and go fast—vaccine refusal and disease eradication. N. Engl. J. Med. 368(15), 1374–1376 (2013) 26. Wilson, K., Keelan, J.: Social media and the empowering of opponents of medical technologies: the case of anti-vaccinationism. J. Med. Internet Res. 15(5) (2013) 27. Gu, Z., Badger, P., Su, J., Zhang, E., Li, X., Zhang, L.: A vaccine crisis in the era of social media. Natl. Sci. Rev. (2017) 28. Aquino, F., Donzelli, G., De Franco, E., Privitera, G., Lopalco, P.L., Carducci, A.: The web and public confidence in MMR vaccination in Italy. Vaccine 35(35), 4494–4498 (2017) 29. Ward, J.K., Peretti-Watel, P., Larson, H.J., Raude, J., Verger, P.: Vaccine-criticism on the internet: new insights based on French-speaking websites. Vaccine 33(8), 1063–1070 (2015) 30. Bolton, K., Memory, K., McMillan, C.: Herd Immunity: Does Social Media Affect Adherence to the CDC Childhood Vaccination Schedule? Purs.- J. Undergrad. Res. Univ. Tenn. 6(1), 5 (2015) 31. Evrony, A., Caplan, A.: The overlooked dangers of anti-vaccination groups’ social media presence. Hum. Vaccines Immunother. 1 (2017) 32. Faasse, K., Chatman, C.J., Martin, L.R.: A comparison of language use in pro-and antivaccination comments in response to a high-profile Facebook post. Vaccine 34(47), 5808–5814 (2016) 33. Dunn, A.G., Surian, D., Leask, J., Dey, A., Mandl, K.D., Coiera, E.: Mapping information exposure on social media to explain differences in HPV vaccine coverage in the United States. Vaccine 35(23), 3033–3040 (2017)

Forecasting and Prevention Mechanisms Using Social …

135

34. Vraga, E.K., Bode, L.: I do not believe you: how providing a source corrects health misperceptions across social media platforms. Inf. Commun. Soc. 1–17 (2017) 35. Glanz, J.M., Kraus, C.R., Daley, M.F.: Addressing parental vaccine concerns: engagement, balance, and timing. PLoS Biol. 13(8), e1002227 (2015) 36. World Health Organization: Measles and rubella surveillance data 2017. Retrieved from http:// www.who.int/immunization/monitoring_surveillance/burden/vpd/surveillance_type/active/ measles_monthlydata/en/ (2017) 37. World Health Organization: Measles and rubella surveillance data—data, statistics and graphics 2017. Retrieved from http://www.who.int/immunization/monitoring_surveillance/data/en/ (2017) 38. List of Countries by Projected GDP 2017: Statistics times. Retrieved from http://statisticstimes. com/economy/countries-by-projected-gdp.php (2017) 39. ElTayeby, O., Eaglin, T., Abdullah, M., Burlinson, D., Dou, W., Yao, L.: A feasibility study on identifying drinking-related contents in Facebook through mining heterogeneous data. Health Inform. J. (2018). https://doi.org/10.1177/1460458218798084 40. World Health Organization: Measles 2017. Retrieved from http://www.who.int/mediacentre/ factsheets/fs286/en/ (2017) 41. World Health Organization: Global vaccine action plan—secretariat annual report 2016. Retrieved from http://www.who.int/immunization/global_vaccine_action_plan/gvap_2016_ secretariat_report_measles.pdf (2017) 42. Pullen, L.C.: Vaccination choice influenced by social networks. Retrieved by http://www. medscape.com/viewarticle/782558 (2013) 43. Burke-Garcia, A., Berry, C.N., Kreps, G.L., Wright, K.B.: The power and perspective of mommy bloggers: formative research with social media opinion leaders about HPV vaccination. In: Proceedings of the 50th Hawaii International Conference on System Sciences (2017) 44. Wendy Sue Swanson: Social networks influence parents’ vaccine decisions. Retrieved by: http://www.kevinmd.com/blog/2013/08/social-networks-influence-parents-vaccine-decisions. html (2013) 45. Kang, G.J., Ewing-Nelson, S.R., Mackey, L., Schlitt, J.T., Marathe, A., Abbas, K.M., Swarup, S.: Semantic network analysis of vaccine sentiment in online social media. Vaccine (2017) 46. De Choudhury, M., Sharma, S., Kiciman, E.: Characterizing dietary choices, nutrition, and language in food deserts via social media. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing, pp. 1157–1170. ACM (2016) 47. Sadilek, A., Kautz, H. A., DiPrete, L., Labus, B., Portman, E., Teitel, J., Silenzio, V.: Deploying nEmesis: preventing foodborne illness by data mining social media. In: AAAI, pp. 3982–3990 (2016) 48. Rankings 2017.: THE LEGATUM PROSPERITY INDEX™ 2017. Retrieved from http://www. prosperity.com/rankings (2017) 49. Chen, S.Y., Anderson, S., Kutty, P.K., Lugo, F., McDonald, M., Rota, P.A., Ortega-Sanchez, I.R., Komatsu, K., Armstrong, G.L., Sunenshine, R., Seward, J.F.: Health care–associated measles outbreak in the United States after an importation: challenges and economic impact. J. Infect. Dis. 203(11), 1517–1525 (2011) 50. Wichmann, O., Siedler, A., Sagebiel, D., Hellenbrand, W., Santibanez, S., Mankertz, A., Vogt, G., Treeck, U.V., Krause, G.: Further efforts needed to achieve measles elimination in Germany: results of an outbreak investigation. Bull. World Health Organ. 87(2), 108–115 (2009) 51. Filia, A., Tavilla, A., Bella, A., Magurano, F., Ansaldi, F., Chironna, M., Nicoletti, L., Palù, G., Iannazzo, S., Declich, S., Rota, M.C.: Measles in Italy, July 2009 to September 2010. Euro. Surveill. 16(29), 19925 (2011) 52. Carabin, H., Edmunds, W.J., Kou, U., Van den Hof, S.: The average cost of measles cases and adverse events following vaccination in industrialised countries. BMC Public Health 2(1), 22 (2002) 53. Lo, N.C., Hotez, P.J.: Public health and economic consequences of vaccine hesitancy for measles in the United States. JAMA Pediatr. 171(9), 887–892 (2017)

136

P. Koukaras et al.

54. Bahk, C.Y., Cumming, M., Paushter, L., Madoff, L.C., Thomson, A., Brownstein, J.S.: Publicly available online tool facilitates real-time monitoring of vaccine conversations and sentiments. Health Aff. 35(2), 341–347 (2016) 55. Larru, B., Offit, P.: Communicating vaccine science to the public. J. Infect. 69, S2–S4 (2014) 56. Dong, W., Liao, S., Xu, Y., Feng, X.: Leading effect of social media for financial fraud disclosure: a text mining based analytics (2016) 57. Isotalo, V., Saari, P., Paasivaara, M., Steineker, A., Gloor, P.A.: Predicting 2016 US presidential election polls with online and media variables. In: Designing Networks for Innovation and Improvisation, pp. 45–53. Springer, New York (2016) 58. Elshendy, M., Colladon, A.F., Battistoni, E., Gloor, P.A.: Using four different online media sources to forecast the crude oil price. J. Inf. Sci. https://doi.org/10.1177/0165551517698298 (2017) 59. Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., Petticrew, M.: Developing and evaluating complex interventions: the new Medical Research Council guidance. Int. J. Nurs. Stud. 50(5), 587–592 (2013) 60. Schoen, H., Gayo-Avello, D., Takis Metaxas, P., Mustafaraj, E., Strohmaier, M., Gloor, P.: The power of prediction with social media. Internet Res. 23(5), 528–543 (2013) 61. Phillips, L., Dowling, C., Shaffer, K., Hodas, N., Volkova, S.: Using social media to predict the future: a systematic literature review. arXiv preprint arXiv:1706.06134. (2017) 62. Pechmann, C., Pan, L., Delucchi, K., Lakon, C.M., Prochaska, J.J.: Development of a Twitterbased intervention for smoking cessation that encourages high-quality social media interactions via automessages. J. Med. Internet Res. 17(2) (2015) 63. Bull, S.S., Levine, D.K., Black, S.R., Schmiege, S.J., Santelli, J.: Social media–delivered sexual health intervention: a cluster randomized controlled trial. Am. J. Prev. Med. 43(5), 467–474 (2012) 64. Martinez, O., Wu, E., Shultz, A.Z., Capote, J., Rios, J.L., Sandfort, T., Manusov, J., Ovejero, H., Carballo-Dieguez, A., Baray, S.C., Moya, E., Matos, J.L., DelaCruz, J.J., Remien, R.H., Rhodes, S.D.: Still a hard-to-reach population? Using social media to recruit Latino gay couples for an HIV intervention adaptation study. J. Med. Internet Res. 16(4) (2014) 65. Larson, H.J., Smith, D.M., Paterson, P., Cumming, M., Eckersberger, E., Freifeld, C.C., … Madoff, L.C.: Measuring vaccine confidence: analysis of data obtained by a media surveillance system used to analyse public concerns about vaccines. Lancet Infect. Dis. 13(7), 606–613 (2013) 66. Kostkova, P., Mano, V., Larson, H. J., Schulz, W. S.: Who is spreading rumours about vaccines?: influential user impact modelling in social networks. In: Proceedings of the 2017 International Conference on Digital Health, pp. 48–52. ACM (2017) 67. Bode, L., Vraga, E.K.: See something, say something: Correction of global health misinformation on social media. Health Commun. 1–10 (2017) 68. Stahl, J.P., Cohen, R., Denis, F., Gaudelus, J., Martinot, A., Lery, T., Lepetit, H.: The impact of the web and social networks on vaccination. New challenges and opportunities offered to fight against vaccine hesitancy. Med. Mal. Infect. 46(3), 117–122 (2016) 69. Seeman, N., Rizo, C.: Assessing and responding in real time to online anti-vaccine sentiment during a flu pandemic. Healthc. Q. (Toronto, Ont.). 13, 8–15 (2010) 70. Finnegan, G.: What can social media tell us about vaccination rates? Vaccines today, Retrieved by: https://www.vaccinestoday.eu/stories/what-can-social-media-tell-usabout-vaccination-rates/ (2011) 71. Koukaras, P., Tjortjis, C., Rousidis, D.: Social media types: introducing a data driven taxonomy. Computing. 102(1), 295–340 (2020)

Forecasting and Prevention Mechanisms Using Social …

137

Paraskevas Koukaras holds a BSc in computer science from A.T.E.I of Thessaloniki and a MSc with distinction in ICT systems from the International Hellenic University (IHU), and he is a Ph.D. candidate in social media analytics at the same institution. He has been teaching programming in the private sector, reviewing for various international conferences and acting as Laboratory Assistant in database courses at IHU. He is also Research Associate at the Centre of Research and Technology Hellas (CERTH-ITI). His current research focuses on influence detection, monitoring and intervention mechanisms and forecasting and prediction models in social media. Dimitrios Rousidis holds a BSc in physics from Aristotle University of Thessaloniki, Greece, and an MSc in computation from UMIST, UK, and he is a Ph.D. candidate in social media analytics at the International Hellenic University, Thessaloniki, Greece. He has been Adjunct Professor in two technological institutes of Greece, he has been teaching databases (relational databases and NoSQL databases) at the DEI College which is affiliated with the Northampton University, UK, and he has extensive teaching experience in adult education programmes. His current research focuses on improving forecasting mechanisms and models in social media. Christos Tjortjis holds a DEng (Hons) from Patras, computer engineering and informatics, a BSc (Hons) from Democritus Law School, Greece, an MPhil in computation from UMIST, and a Ph.D. in informatics from the University of Manchester, UK. He is tenured Associate Professor in knowledge discovery and software engineering systems at the School of Science and Technology, International Hellenic University. He is Director for the MSc in data science, the MSc in ICT systems and the EMJMD MSc in smart cities and communities. He was Lecturer at UMIST, computation, and the School of Informatics and Computer Science, at the University of Manchester, and Adjunct Senior Lecturer in engineering informatics and telecoms, W. Macedonia, and in computer science, Ioannina. His research interests are in data mining and software engineering. He worked in 14 R&D projects, leading 3. He published over 60 papers in int’l refereed journals and conferences and was PC Member in over 80 int’l conferences. He leads the Data Mining and Analytics Research Group, comprising 5 Ph.D. and 5 MSc students.

Security and Privacy Issues for Intelligent Cloud-Based Health Systems Dimitra Georgiou and Costas Lambrinoudakis

Abstract Technology is already affecting every aspect of life, and our health is no exception. Artificial intelligence (AI) has become one of the most emerging technologies over the last few years in almost every environment. New technological advances such as cloud computing provide benefits and have changed the way we store, access and exchange information. Especially, in the Healthcare IT sector, cloud-based systems offer great potential, from many perspectives, including improved medical diagnosis, accurate and faster prediction and cost-effective management treatment. In an attempt to assist cloud providers and healthcare organizations to secure their cloudbased environment and to adopt the appropriate measures for data protection, we present an overview of the security and privacy requirements of cloud-based healthcare systems. Specifically, this chapter starts with the presentation of the reported threats in cloud-based health systems, continues with the identified objectives and assets and concludes with measures for the mitigation of the identified threats. Due to the fact, migration into cloud-based healthcare systems, in most cases, implies that data subjects lose control of their data, many scientists have raised their worries about this. It is therefore needed to re-consider security, privacy and trust requirements, in the context of cloud computing. This chapter makes concrete recommendations for improving the protection level of cloud-based health organizations, cloud providers, hospitals and patients. Keywords Cloud computing · Health systems · Security · Threats · Mitigations

1 Introduction In recent years, cloud computing has a racial development in technological environment. But, what is exactly cloud computing? It refers to on-demand, self-service D. Georgiou (B) · C. Lambrinoudakis Systems Security Laboratory, Department of Digital Systems, School of Information and Communication Technologies, University of Piraeus, 150 Odyssea Androutsou Str., 18532 Piraeus, Greece e-mail: [email protected] © Springer-Verlag GmbH Germany, part of Springer Nature 2020 I. Maglogiannis et al. (eds.), Advanced Computational Intelligence in Healthcare-7, Studies in Computational Intelligence 891, https://doi.org/10.1007/978-3-662-61114-2_9

139

140

D. Georgiou and C. Lambrinoudakis

Internet infrastructure that enables the user to access computing resources anytime from anywhere [1]. It is a new paradigm for improving services. The combination of cloud computing with artificial intelligence facilitates the creation of a huge network capable of storing huge amounts of data and at the same time having the capacity to improve on the go. While there is no doubt about the benefits that cloud computing offers to different organizations and specifically to healthcare organizations, its adoption greatly depends on several factors concerning data security and users’ privacy. In the healthcare field, cloud computing offers great benefits regarding the easy access of health information, but the most important one is the possibility that patients and doctors can gain access to their sensitive data from anywhere in the world with the use of Internet. For this reason, governments, healthcare providers and authorities in different countries are encouraging healthcare organizations to adopt cloud computing. However, how can cloud providers and hospitals guarantee the security, privacy and confidentiality of their patients’ sensitive data? The risks include possible harm to patient safety or loss of personal health information and should be addressed. So, preventive measures for the protection of patients’ data are necessary. The threats against cloud systems are divided into three separate categories (1) Infrastructure and host-related threats that affect the infrastructure of the Cloud, (2) service providerrelated threats that may affect the patients who seek a service and (3) generic threats that may affect both the infrastructure and the service provider (Table 1). The last two columns of the table provide information on whether the specific threat can be addressed in a health system, either through some technical solutions or through some organizational/procedural countermeasures (non-technical solution). The decision on whether each threat is covered (●), partially covered (◯) or not covered (–) has been based on the related published work and on the personal judgment and experience of the authors. The rest of chapter is organized as follows: Sect. 2 presents the methodology followed for presenting the research results. Section 3 gives an overview of the threats in the sensitive security and privacy context of health cloud systems. Section 4 presents attack scenarios against hospital cloud servers in order to demonstrate practical examples of implementation and to validate these in hospitals. Section 5 presents an overview of assets and objectives of health cloud systems and a mapping of them with the specific group—categories of threats in the proposed methodology. Section 6 discusses cloud computing as a solution and presents ways to mitigate the identified threats with the scope to support cloud-based health information systems. Section 7 provides the concluding remarks of the current chapter. As described by the National Institutes of Standards and Technology (NIST) [2], cloud computing model consists of five essentials characteristics, three delivery models and four deployment models [3]. The five key characteristics of cloud computing that have been listed by Mell and Grance, namely location-independent resource pooling, on-demand self-service, rapid elasticity, broad network access and measured service, are being presented in Fig. 1 [4, 5]. The three common requirements for securing cloud computing are: confidentiality, integrity and availability.

Security and Privacy Issues for Intelligent Cloud-Based …

141

Table 1 Threat’s assessment Threats Infrastructure and host

Technical

●

Unauthorized physical access

● ◯

Deficient training Password guessing

Service provider

Generic

●: Covered – : Not covered ◯: Partially covered

Non-technical

Natural disasters

●

Unauthorized data access

◯

Security logs compromisation

●

Network breaks

◯

Privilege escalation

◯

Ineffective data deletion

●

Malicious scanning/observation

–

Insecure/obsolete cryptography

◯

Insufficient logging/monitoring

◯

Cloud service failure/termination

◯

Third-party suppliers’ failure

◯

Lock-in

◯

Compliance problem

●

Infrastructure’s modifications

◯

Data processing

◯

Administrative changes

◯

Data interception

◯

Browser security

◯

Injection vulnerabilities

◯

Customer’s negligence/cloud security

◯

Management interface exposure

◯

Loss of governance

◯

Social engineering

◯

DDOs

◯

Encryption/key exposure

◯

Service engine exposure

◯

Malware and Trojan Horses

◯

Malicious insider of cloud provider

–

142

D. Georgiou and C. Lambrinoudakis

Fig. 1 Key characteristics of cloud computing

• Data confidentiality: Enables clients’ data to remain secure and confidential from unauthorized access. This problem occurs when sensitive data is outsourced to the cloud server. In this case, encryption is necessary. • Enable cloud information integrity: Data integrity is one of the critical elements in most information systems. Data integrity refers to the fact that data must be reliable and accurate. Data integrity means data should be kept from unauthorized modification. Data integrity could help in getting lost data or notifying if there is data manipulation. • Ensure availability: Data availability means that information must be available for the authorized users. Data availability is one of the biggest concerns of service providers. Moreover, apart from the regulatory, technical and financial challenges imposed by cloud computing, organizations have to deal with other challenges including: operational, technological, privacy, accountability, data/service reliability and data management. However, the various companies that are offering today their services and tools for cloud-based health systems employ solutions which are mainly focusing on the provision of generic approaches and frameworks that allow organizations to evaluate their current security readiness level and propose some generic guidelines for moving toward to cloud computing. They do not provide specific methods, mitigation techniques and policies to tackle the above challenges. As a result, a lot of securityand privacy-related challenges remain. The main aim of our research is to deliver an innovative security- and privacy-oriented cloud-based system, which will facilitate scoping and processing of health data, providing at the same time good practices and recommendations.

2 Methodology The methodology adopted for our research has been based on the one presented in our paper [6], with some differentiation on the categorization of threats, according to cloud-based health system needs. Six fundamental goals drove the collection and analysis of the survey data, as follows:

Security and Privacy Issues for Intelligent Cloud-Based …

• • • • • •

143

Review of existing studies on cloud computing security issues Identification of threats against cloud computing infrastructures Classification of threats in distinct categories–Gates Elicitation of security requirements Development of good practices and recommendations for new actions Definition of security policy rules.

According to the security policy presented in [6], the security threats against cloud infrastructures have been derived from the cloud security alliance [7], and some new cloud-specific security threats have been categorized in four Gates-categories, namely: process functions and controls, HR, legal and compliance and technology. Here, new threats against cloud, according to the last publications of cloud security alliance [8], ENISA threat landscape [9] and ISACA [10], are being introduced. Based on our previous publications on cloud computing threats [11–13], in this chapter, we present the ones that are closely related to cloud-based health information systems. Subsequently, security measures that can achieve the required protection levels by mitigating fully or partly the assessed risks are identified. The classification of threats for health systems is done in five general categories: (1) Identity and access management, (2) data, (3) regulatory, (4) rational and (5) technology. The work presented in this chapter has been structured as follows: • “Threat identification”: Collection of relevant information and classification of threats in one of the aforementioned five categories. • “Attack scenarios”: Describes five attack scenarios (each one exploiting threats from a different threat category) against cloud-based health servers. • “Objectives identification”: Identification of the main objectives of a cloud-based health system. • “Assets identification”: Identification of the essential assets of a cloud-based health system. Furthermore, each asset is linked with specific threats that may affect it. • “Mitigation of threats”: It focusses on current practices, and moreover, it makes recommendations for future steps for achieving a higher level of security in cloudbased health systems.

3 Threats of Cloud-Based Health Systems Nowadays, health care is focusing on accessing medical records anytime and at anyplace. More and more, healthcare providers are offering health solutions and services that can be consumed or integrated by customers over a cloud. Security and privacy protection of patients’ records in the e-health cloud are of vital importance and involves various requirements. Three types of cloud models are usually employed in cloud-based health services and are generally classified as private cloud, public

144

D. Georgiou and C. Lambrinoudakis

Fig. 2 General categories of threat in cloud-based health systems

cloud and hybrid/federated cloud. Operating a system in a cloud environment hides risks in several dimensions. This section provides a systematic overview of the threats in e-health clouds. In contrast to our previous publications on cloud computing threats, in this chapter, we present the threats that are more related to cloud-based health information systems. The threat model includes the assets, threats, objectives and vulnerabilities of the system. The major threats are classified in one of the following five general categories: (1) Identity and access management, (2) data, (3) regulatory, (4) operational and (5) technology (Fig. 2). Health organizations must understand and mitigate these threats to better leverage their cloud computing initiatives. 1. Identity and access management: The following threats are associated with inappropriate access of cloud computing resources: • • • • • • • • • •

User access provisioning Deprovisioning Records management/records retention Superuser access Staff Vendors Unrestricted access to computers Malicious insider threat Insufficient identity Credentials and access management

Security and Privacy Issues for Intelligent Cloud-Based …

145

• Physical manipulation/damage • Identity medical theft (fraud). 2. Data: The following threats are associated with loss, leakage or unavailability of data: • • • • • • • • • • •

Data segregation and isolation Information security and data privacy requirements Data protection risks Legal risks Subpoena and e-discovery Risk from changes of jurisdiction Data breaches Data loss Cross-border transfer of data Information leakage Lack of privacy.

3. Regulatory: The following threats are associated with non-compliance to various governmental, national/geographic regulations or legal and regulatory requirements: • • • • • • •

Complexity to ensure compliance Lack of industry standards Lack of certifications for cloud Providers Lack of international regulations Lack of visibility into service provider operations Loss of compliance Third-party control.

4. Operational: The following threats are associated with the execution of business activities and services: • • • • • • • • • • • •

Service reliability and uptime Disaster recovery SLA customization and enforcement Control over quality Abuse of high-privilege roles Natural disaster Malicious insider Web application attacks Lack of user control The abuse of cloud services Unintentional disclosure Insufficient knowledge.

5. Technology: The following threats are associated with evolving technologies and lack of standardization:

146

D. Georgiou and C. Lambrinoudakis

• • • • • • • • • • • • • • • • • • • •

Evolving technology Cross-vendor compatibility and integration Customization limitations Technology choice and proprietary lock-in Inadequate disposal of old hardware Isolation failure Online medical devices Shared technology vulnerabilities Unsecured mobile devices System vulnerabilities System failures Social engineering attacks Denial of service Ransomware Botnets Cloud burst security Account hijacking Insecure interfaces and API’s Backup vulnerabilities Prohibition against cross-border transfer.

This threat model presents some of the major threats in the context of cloud-based healthcare systems. In order for any cloud-based health system to be efficient, reliable and trustworthy, strong security policies must be in place to effectively counter these threats of unlawful and unauthorized privilege escalation. Our work proposes a threat identification that will help cloud users, cloud providers and organizations to make informed decisions about risk mitigation within a cloud security policy strategy. To be more self-explained, a scenario-based approach has been chosen. Scenarios can be simply integrated within development methodologies and can be adapted to the methodology’s document and concepts. This is due to the fact that scenarios can be represented in several ways [14]. Five attack scenarios, which are considered particularly relevant to hospitals, are described in this section for specific threats (medical identity theft, data breach, third party, Web application attack, social engineering attack) and threat groups according to our threats classification scheme.

4 Attack Scenarios for Specific Threat Groups (Categories of Threats) Extrapolated from the challenges, there are some prominent threats against cloud computing. In this section, we present five attack scenarios (Tables 2, 3, 4, 5 and 6) illustrating these main concerns. The attack scenarios are specific to the threat category that each of the five selected threat belongs (Fig. 2). Such attacks may relate to different threats according to the proposed methodology.

Security and Privacy Issues for Intelligent Cloud-Based …

147

Table 2 Attack scenario for threat group #1 Attack scenario 1 Threat group: Category#1—Identify and access management Threat: Medical identity theft An identity theft attack is the attack intending to steal the most confidential and sensitive information. In the case of hospitals, highly sensitive data, including names, home addresses, personal data, social security number, ID data or usernames and passwords, genetic data, biometric data, health-related data, can be used by criminals to obtain expensive medical services, devices and prescription medications by pretending to come from trustworthy entity Table 3 Attack scenario for threat group #2 Attack scenario 2 Threat group: Category#2 data Threat: Data breach A data breach refers to a security incident in which information is accessed without authorization. This data resides in the targeted organization’s systems or networks and is proprietary or sensitive in nature. Data breaches may involve personal health information (PHI), personally identifiable information (PII), trade secrets or intellectual property. In the case of hospitals, the cybercriminal makes initial contact using either a network or social attack, gets into hospital’s computer and accesses confidential data. Once the hacker extracts the data, the attack is considered successful. According to a publication of Health Insurance Portability and Accountability Act [15], healthcare data breaches have resulted in the loss, theft, unauthorized accessing, impermissible disclosure or improper disposal of 100,000 or more healthcare records in 2019. This can be done physically by accessing a computer or network to steal local files or by bypassing network security remotely Table 4 Attack scenario for threat group #3 Attack scenario 3 Threat group: Category#3 regulatory Threat: Third party Health organizations that use cloud services have to deal with issues such as loss of control over sensitive data. Using third-party file sharing, that means the data’s security, integrity and authenticity are beyond the control of the health organization, but third-party controls them. As cloud services are designed to save data in real time, at any attack of the cloud providers, sensitive data could be viewed, stolen or changed by an unauthorized person Table 5 Attack scenario for threat group #4 Attack scenario 4 Threat group: Category#4 operational Threat: Web application attack Websites depend on databases to deliver the required information to visitors. Serious weaknesses or vulnerabilities allow hackers to gain direct and public access to databases in order to churn sensitive data—this is known as a Web application attack. In a hospital, some hackers may maliciously inject code within vulnerable Web applications to trick users or employees and redirect them toward phishing sites. This technique is called cross-site scripting

148

D. Georgiou and C. Lambrinoudakis

Table 6 Attack scenario for threat group #5 technology Attack scenario 5 Threat group: Category#5 technology Threat: Social engineering attack Social engineering is the art of manipulating people, so they give up confidential information. The criminals are usually trying to trick the employees giving them their passwords or sensitive information, or access their computer to secretly install malicious software that will give them access to their passwords and then control over your computer. The social engineering attack could be: an e-mail from a friend, an e-mail from another trusted source, baiting scenarios and response to a question you never had. Social engineering attacks could be divided in a human-side of hacking or a computer based, which is carried out with the help of computers (e.g., phishing). A computer-based attack could be when individuals sent phishing e-mails containing links or attachments that appear to be innocent in nature. An employee of the healthcare organization opens an e-mail attachment, which contains a malicious link embedded into the content. It will immediately infect the users’ computers and begin to spread throughout the rest of the health system. When opened, malware took over the computer and accessed the employee’s computer, which had patient data stored on it. A human-side of hacking, attack could be a fake nurse. A social engineer (criminal) purchases and creates a fake ID tag. The criminal walks into hospital and gets into doctors’ offices. He may pretend to text while really taking pictures of patient files without being questioned

5 Cloud-Based Health System’s Objectives and Assets As detailed below, the type and extent of ICT usage significantly affect the objectives as well as the related challenges and opportunities. In order to achieve an adequate level of protection in a cloud-based health system, it is critical that the following objectives, presented in Fig. 3, are met. From the literature review [15–18], it has been confirmed that, with respect to all objectives presented above, health systems will benefit from a cloud implementation, regarding patients’ safety. In addition, this study presents the relevant assets that are considered critical for the proper operation of a health system both for the patients and the relevant stakeholders. Hospitals have a wide range of assets that are essential for their operations and thus need to be protected. Unique identification of information assets of the health systems is the first step to ensure the protection of information. But, in order to make it clear, we should explain what an asset is. An asset is defined as something useful or valuable and what we are trying to protect. So, the task of any cloud provider and healthcare organization is to ensure that existing assets currently should meet the criteria of the system. For these reasons, the management of IT assets requires well-developed processes and clear policies. • • • • • •

Behavioral data Biometric information Availability of information Health data Configuration/device information Remote care system assets

Security and Privacy Issues for Intelligent Cloud-Based …

Fig. 3 Objectives of cloud-based health systems

• • • • • • •

Networked medical devices Identification systems Mobile devices Wireless sensors Interconnected clinical information systems Digitalization of health information Hospital information systems (HIS) – – – – – – – – – –

Laboratory information systems (LIS) Radiology information systems (RIS) Pharmacy information system (PIS) Pathology information system Blood bank system Clinical data Picture archiving and communication systems (PACS) Electronic health record components Patient health record service ePrescription service

• Databases and Storage components • Monitoring and logging of information exchanges

149

150

D. Georgiou and C. Lambrinoudakis

• • • • •

Physical operations Credentials User (SLAs, regulations and end-user data) Human agents User health (safety against malfunction and safety against unauthorized physical access) • User information and privacy. Security is about protecting what is considered to be an asset to individual users and organizations from misuse and malicious acts. Cloud computing security refers to the controls that must be implemented in order to prevent the loss of data, information or resources belonging to a cloud service provider or to its patients. In order to have secured cloud systems, cloud users and cloud service providers should act cooperatively. This chapter aims to contribute to a discussion around how to achieve those objectives. In Table 7 of our chapter, we present the threat exposure of assets.

6 Security and Privacy Solutions in Cloud Health Systems The need to manage health risks and to improve patient’s safety and care is now a priority issue on the international health policy agenda. All the systems used by health care institutions should guarantee the secure storage of patients’ data, data availability and integrity. The emergence of healthcare with cloud computing has introduced a number of specific security and privacy threats and requirements. In order to mitigate the threats that we presented previously, specific solutions are necessary. Our proposed approach on the evaluation of cloud-based health risks is based on five steps, as presented in Fig. 4. In cloud computing, anyone of cloud users, cloud service providers and thirdparty auditors has access to the data [19]. Therefore, to reduce the potential threats, the following strategies are proposed in order to mitigate some major threats in cloud-based health (Table 8).

Security and Privacy Issues for Intelligent Cloud-Based …

151

Table 7 Asset exposure to cyber threats Category (threat group)

Assets affected

Category #1 Identify and access management

Databases and storage components Monitoring and logging of information exchanges physical operations Credentials behavioral data Biometric information Digitalization of health information Health data Clinical data

Category #2 Data

Behavioral data Biometric information Health data Clinical data Interconnected clinical information systems

Category#3 Regulatory

Behavioral data Biometric information Health data Clinical data Interconnected clinical information systems

Category #4 Operational

Physical operations Credentials Hospital information systems (HIS) Laboratory information systems (LIS) Radiology information systems (RIS) Pharmacy information system (PIS) Pathology information system Blood bank system Clinical data Picture archiving and communication systems (PACS) Electronic health record components Patient health record service ePrescription service

Category#5 Technology

Interconnected clinical information systems Remote care systems Mobile client devices Networked medical devices Identification systems Networking equipment Wireless sensors Hospital information systems (HIS); Laboratory information systems (LIS); Radiology information systems (RIS); Pharmacy information system (PIS); Pathology information system; Blood bank system; Clinical data Picture archiving and communication systems (PACS), Electronic health record components; Patient health record service; ePrescription service

152

D. Georgiou and C. Lambrinoudakis

Fig. 4 Proposed approach on evaluation of cloud health risks Table 8 Potential mitigation examples and considerations of threats Identity medical theft threat Category/threat group

Category#1 Identify and access management

Assets

Databases and storage components, behavioral data, biometric information, health data, clinical data

Mitigation examples—considerations

Access control mechanisms A. Specify access control policy • It is proposed to define a user access policy, for the resources of the information system, in accordance to the role-based access control • Any user may only acquire a right through the assignment of a role • Depending on hospital’s policy, it is possible to prohibit the simultaneous assignment of specific roles to the same user. The administrator should be able to describe and control such restrictions. Restrictions can be either dynamic (that is, a role cannot be activated as long as another role is active for the same user), or static (that is, a role cannot be assigned to a user assigned to another role at some point in time in the past) • Users should only use the applications and resources necessary to perform tasks associated with their role • The usage rights granted to each category of roles or users are checked at least once every six (6) months to ensure that no more rights are given than what is necessary B. Distribution of roles to users • The distribution of roles to users should be in accordance with the responsibilities they have and with the actions they are required to perform • Role allocation specifies user access only to the information they need to know or to process in order to fulfill their tasks (need to know) • It should be possible to assign a role for a limited time • Role assignment to a user should be dynamic. It should be possible to define the specific assignment time (start and end time). The role assigned to a user is called “active” at any given time if this time is within the assignment period • After authentication, the user has all the rights that result from all the roles assigned to him and are “active” at the moment. The set of rights of a user may change over the lifetime of his/her use of the information system, depending on the set of roles assigned to the user and which are active • Creating new roles and managing the rights of each role will be done centrally by the administrator • Reports on the distribution of privileges per role and of roles to users should be available C. Remote access • Remote access to the hospital’s server should be strictly monitored and be restricted only to authorized administrators (continued)

Security and Privacy Issues for Intelligent Cloud-Based …

153

Table 8 (continued) Identity medical theft threat Data breach Category/threat group

Category#2 Data

Assets

Behavioral data, biometric information, health data, clinical data, interconnected clinical information systems

Mitigation examples—considerations

Authentication mechanisms A. Length and composition of passwords • The passwords should be memorable but at the same time of sufficient length and composition to be difficult to guess or reveal • User passwords should be longer than nine (9) characters long • The passwords must have at least one (1) alphabetic character and at least one (1) digit or special character • More than two (2) repeating characters (e.g., “aaa”) are forbidden • To deal with dictionary attack, the passwords should not be words that can be found in dictionary, days, months, car traffic numbers, etc. • Passwords do not contain the user’s identifier • The system should automatically check that the rules regarding passwords are met and reject passwords that do not follow the rules • Provide users with instructions on how to select memorable and with the appropriate length and composition passwords • The system should prevent users from using passwords that have been used in the past B. Temporary passwords • Temporary passwords must be created in such a way that it is difficult for unauthorized persons to discover them • When users get a system account, give them temporary passwords • Temporary passwords should follow the general rules regarding the length and composition of the passwords • Users must be forced to modify their temporary password when they connect to the system for the first time • Vendor-defined passwords must be modified immediately after installation C. Password distribution • Temporary passwords must be distributed to users in a way that ensures their confidentiality • If the distribution of the passwords is made electronically, they should be sent in encrypted form • The receipt of the password should be confirmed Two-factor authentication • In this case, everything mentioned previously (authentication mechanisms) still holds with the only difference that there will be a second level of confirmation of the validity of the user’s identity through one of the following alternative ways: • Use of the username/password associated with a pseudorandom number, generated at regular intervals by a pseudorandom number generator. In practice, these are small portable devices. The same sequence of pseudorandom numbers will also produce an appropriate implementation (e.g., radius) in the core infrastructures of hospital to achieve authentication • Use a digital certificate stored in a secure device (e.g., smart card) Data storage protection • Personal data of special categories (e.g., medical history) that have a high degree of criticality and which are stored in the central database should be protected using cryptographic methods (continued)

154

D. Georgiou and C. Lambrinoudakis

Table 8 (continued) Identity medical theft threat • The encryption keys utilized should have a sufficient key length according to the current scientific information • The data protection officer should be kept informed of developments in cryptography and cryptanalysis in order to avoid the use of obsolete or weak cryptographic techniques • Use trustworthy manufacturers’ cryptosystems Personnel issues A. Personnel issues prior to recruitment • Perform a statutory audit of the persons who are to take over the responsibilities associated with critical roles of provider’s information systems. • New personnel should be informed of the criticality of the responsibilities they undertake and be trained appropriately • The description of the position and its responsibilities should include clearly the responsibilities for the protection of personal data • There should be no management work for which only one staff member will have the necessary knowledge • The new staff must sign a confidentiality agreement immediately upon recruitment, stating that they have become aware of the data protection policy and are committed to its application • Where there is cooperation with third parties, a confidentiality agreement must be signed • Confidentiality agreements should be reviewed if the working conditions or the terms of a cooperation are altered B. Personnel issues at the end of the employment • Revoke access rights for any employee who has transferred or changed his/her duties or his/her employment relationship that has been interrupted • In the event of an employee leaving or moving, before leaving, the replacement must be informed of the tasks he/she will be required to perform • An employee who leaves upon termination of the employment relationship must deliver documents, keys, entry cards and any other equipment (e.g., optical means, etc.) • The former employee’s access to the systems must be immediately deactivated Security and personal data protection training and awareness programs A. Training and awareness of technical management personnel • Health organization must provide adequate and appropriate training to staff that to use or manage the IT infrastructure, in the topics of personal data protection and information systems and network security, depending on the role of each employee B. Secure usage directives and policies • Health organizations should develop appropriate security policies and follow applicable privacy and security directives Third-party threat Category/threat group

Category#3 Regulatory

Assets

Behavioral data, biometric information, health data, clinical data, interconnected clinical information systems (continued)

Security and Privacy Issues for Intelligent Cloud-Based …

155

Table 8 (continued) Identity medical theft threat Mitigation examples—considerations

Maintenance services by external partners A. Hardware and software maintenance • Cloud provider’s hardware and software must be adequately maintained • Have a maintenance contract for all important items of hardware and software • Record all hardware malfunctions and all software problems B. Protection during the provision of software and hardware maintenance services • All external partners must comply with the data protection policy of provider’s • The external partner is required to report to the data protection officer of any identified security breach or attempt of data violation and to work with him to investigate the incident or attempt. It is also required to maintain and provide every element that may contribute to the investigation of such events • The maintenance contract should include a term that ensures that information transferred to the external partner or to individuals or subcontractors who perform work on his behalf during or in connection with the execution of the contract remains confidential. This term shall have effect after expiry of the contract • The contract should include the physical and logical access rights to provider’s systems, and data that are granted to the external partners • The employees of external partners must always be accompanied by a member of provider’s staff during their stay at the site • After the end of the contract, all the rights granted to the external partner and all the passwords they knew should be deactivated C. Monitoring software maintenance services • Software changes should be checked before the software enters production mode • Testing should be carried out on a test server of provider and not to the one used for production • Keep a record of the employees of external partners (software developers/maintainers) • Keep a record of software maintenance tasks

Web application attacks Category/threat group

Category#4 Operational

Assets

Physical operations Credentials Hospital information systems (HIS) Laboratory information systems (LIS) Radiology information systems (RIS) Pharmacy information system (PIS) Pathology information system Blood bank system Clinical data Picture archiving and communication systems (PACS) Electronic health record components Patient health record service ePrescription service (continued)

156

D. Georgiou and C. Lambrinoudakis

Table 8 (continued) Identity medical theft threat Mitigation examples—considerations

Software/application integrity check • Systems’ software and primary applications should be tested for cases of unauthorized modifications • All servers should employ mechanisms for detecting non-authorized modifications to both system software and application software • Provide a software tool for checking the integrity of the executable files of the software (e.g., by using checksums or cryptographic checksums) Physical security It does not concern data centers, as they meet the necessary standards of protection against physical disasters and thefts. It concerns exclusively the workplaces of the employees using health organization’s information systems A. Physical access control: • Take care for physical access • Employ 24-hours security personnel that meets the requirements of the relevant law • Have a secured deposit for the keys of the buildings and appoint the responsible roles for the keys’ management • It is desirable for the perimeter of the installation to be protected by fence gates and to be monitored by a closed-circuit television (CCTV) • All visitors should get a visitor’s badge and be recorded in the guestbook B. Fire prevention, detection and response • Install fire detectors and corresponding alarms in the rooms where important equipment is installed (continued)

Security and Privacy Issues for Intelligent Cloud-Based …

157

Table 8 (continued) Identity medical theft threat • Fire detection should cover all installations • The fire detection control panel should be installed in an easily accessible location • The use of devices that may cause a fire should be limited to fully controlled areas • The cabinets used to store backups should be fireproof and heat resistant for at least thirty 30 min • In all areas where important equipment and software is installed, there should be a fire-extinguishing system suitable for computers • There should be a clearly visible notice board providing clear instructions for the use of fire and extinguishing equipment as well as emergency exits • Fire-extinguishing equipment must be checked at least once a year and in accordance with the fire brigade’s instructions C. Flood protection and protection from extreme weather phenomena • The water supply control valves should be easily accessible, and there should be signs of their location • Check for potential vulnerabilities in piping (water, heating, etc.) • Pipes should not pass from areas with important equipment • Lightning protection must be done with an external lightning rod, but also with an overvoltage protection system. The interior grounding system should cover all rooms with important equipment and all the equipment located in these rooms D. Detection and prevention of equipment theft • Every case of theft should be investigated • For getting equipment out of the facility, a written permission should be required • It is desired to put labels on all hardware indicating that the item is an asset of health organization E. Protection from power failures • Important equipment should be supported by an uninterruptible power supply (UPS) sufficient to run for at least 20 min • Activate the alarm function in the event of a power failure Assets inventory A. Keeping an inventory of organization’s assets: • Maintain a list of all information resources (data, hardware, software and documentation) • The accuracy of the assets inventory should be checked at least every six (6) months B. Assets’ management: • Every employee who leaves the company should return all the equipment—software—documentation that she/he was using outside the company • The assets inventory should be updated annually Prevention mechanisms A. Health organization • should enforce the use of anti-malware software for all computational devices in IT infrastructures that use health organizations services in order to prevent malicious code installation and propagation in the infrastructure B. Health organization • should install and customize a Web application firewall in order to prevent risks from zero-day attacks (continued)

158

D. Georgiou and C. Lambrinoudakis

Table 8 (continued) Identity medical theft threat Social engineering Category/threat group

Category#5 Technology

Assets

Interconnected clinical information systems “remote care systems, mobile client devices, networked medical devices” identification systems, networking equipment, wireless sensors, hospital information systems (HIS); laboratory information systems (LIS); radiology information systems (RIS); pharmacy information system (PIS); pathology information system; blood bank system; clinical data, picture archiving and communication systems (PACS), electronic health record components; patient health record service; eprescription service

Mitigation examples—considerations

Audit trails and event logs A. Activities and data recording: It is necessary to record activities and data that facilitate the detection of potential breaches or attempts to breach the security of the information system as well as the detailed examination of each suspected incident B. Maintaining the audit trails: Keep an audit trail file for all systems that support multi-user access. Indicatively: • Record when the administrator account is being used • Record the log-on actions • Record the log-off actions • Record the actions of the administrators • Record the changes to the critical data of each subsystem • Record start up and shutdown of the system • Record system failures and corrective actions • Record failed login attempts • Record changes to access and use rights • Record the I/O devices connections on the system • Record the printing actions • Record the use of access accounts in unusual hours (e.g., non-working hours) • Record attempts to use remote workstations C. Recording data related with suspicious incidents: Record the basic data related with each suspicious incident. Indicatively, the following are listed: • Record the user ID • Record the date and time of the event • Record the type of event • Record the files that were accessed • Record the applications that were used • Record the identity of the workstation • Record the status of the data before and after the changes D. Informing users: Inform users about which of their activities are recorded by the system E. Analyzing audit trails and event logs: • Audit trails and event logs should be used to identify a potential-attempted violation • Check for failed attempts to connect or access data • Check for new users’ accounts with enhanced privileges (continued)

Security and Privacy Issues for Intelligent Cloud-Based …

159

Table 8 (continued) Identity medical theft threat • Identify a successful authentication of a user when this follows multiple authentication failures • Identify deviations from the usual use of system resources (e.g., unusually large number of prints by a user) • Identify deviations from the usual use of user accounts (e.g., use during non-working hours) • Detect remote access attempts • Audit trails and event logs should be analyzed at least once (1) per week F. Protecting audit trails and event logs: • Check access to logs that should be restricted to authorized persons only • No role, on its own, should have the right to modify or delete the audit rails/event logs • It is recommended that you use write-once-read-many (WORM) storage media to store audit trails/event logs files • Copies of the audit trails/event logs must be kept in backups • Check at least once per year that files stored in magnetic or optical media are readable • The trails/logs recording must be constantly running and should not be possible to deactivate it • When the available storage space reaches 75% of its capacity, an alert message must be generated Secure software development A. Secure application development should be enhanced by applying security checkpoints and techniques at early stages of development as well as throughout the software development lifecycle Secure system configuration and maintenance A. Secure system configuration techniques and methodologies should be applied to enhance the security and the robustness of the IT infrastructure against common omissions and infrastructure vulnerabilities as they are recommended by industrial practices B. System and software/framework security updates should be enforced immediately as they are released in order to protect the infrastructure from known exploitable vulnerabilities

160

D. Georgiou and C. Lambrinoudakis

7 Conclusions To realize the full promise of AI as a positive strength in our lives, there are intimidating challenges to overcome, and many of these challenges are associated to systems and infrastructures. As cloud computing is an AI infrastructure, it presents numerous benefits compared to traditional ones. It is common truth that the adoption of cloud computing in the healthcare industry will continue to evolve in the coming years. No doubt that standardized security methods in cloud-based applications will bring benefits like storage, capacity and cost reduction to patients, insurance companies and cloud providers when sharing information across medical organizations providing better results. But, unfortunately cloud-based health systems have their own security-related issues that threaten the organizations to adopt the cloud technology. Although the use of cloud computing has rapidly increased, security is still considered as the main issue in the cloud computing environment, especially when it comes to health data. Cloud providers and health organizations must strengthen their security mechanisms, controls and policies in order to increase the trustworthiness of the health data processing. The supplemented framework in cloud addresses the security threats in the healthcare cloud infrastructure and provides proper countermeasures to mitigate them. In our work, we are based on the understanding of threats, assets and objectives in a general security policy strategy that is able to assess confidentiality, integrity and availability for cloud computing services. The presented analysis helps the cloud providers and health organizations to secure the vulnerabilities of their cloud systems, which will eventually lead to minimize the risk and maximize the trust of their systems by patients and the healthcare industry. This chapter proposes several open security research directions in cloud-based health systems that researchers and policy makers have potential to address these challenges. We hope these questions will encourage new research that can advance AI and make it more efficient, understandable, secure and reliable for healthcare systems.

References 1. Mell, P., Grance, T.: The NIST definition of cloud computing. Nat. Inst. Std. Technol. 53(6), 50 (2009) 2. Mell, P., Grance, T.: The NIST definition of cloud computing (2011) 3. Lup¸se, O.S., Vida, M.M., Tivadar, L.S.: Cloud computing and interoperability in healthcare information systems. In INTELLI 2012: the first international conference on intelligent systems and applications (2012) 4. Hassan, T., James, B.D., Gail-Joon, A.: Security and privacy challenges in cloud computing environments. IEEE J. Mag. 8(6), 32–39 (2010) 5. Takabi, H., Joshi, J., Ahn, G.: Security and privacy challenges in cloud computing environments. IEEE Secur. Priv. Mag. 8(6), 24–31 (2011) 6. Georgiou, D.: Security policies for cloud computing. Ph.D. University of Piraeus, Piraeus, Greece. http://dione.lib.unipi.gr/xmlui/bitstream/handle/unipi/11007/Georgiou_Dimitra.pdf? sequence=1&isAllowed=y (2018)

Security and Privacy Issues for Intelligent Cloud-Based …

161

7. Cloud Security Alliance: The Notorious nine: cloud computing top threats in 2013. http://www. cloudsecurityalliance.org/topthreats/ (2013). Accessed 2013 8. Cloud Security Alliance: Top threats to cloud computing: deep dive 2018. https://downloads. cloudsecurityalliance.org/assets/research/top-threats/top-threats-to-cloud-computing-deepdive.pdf,Accessed (2018), 30 September 2018 9. ENISA: Threat landscape report 2018 15 top cyber threats and trends. https://www.enisa. europa.eu/publications/enisa-threat-landscape-report-2018 (2018). Accessed 25 March 2018 10. ISACA: Security considerations for cloud computing https://www.isaca.org/bookstore/Pages/ Product-Detail.aspx?Product_code=SCC (2012). Accessed 25 November 2018 11. Georgiou, D., Lambrinoudakis, C.: A security policy for cloud providers. In Proceedings of the 9th international conference on internet monitoring and protection, ICIMP, pp. 13–21 (2014) 12. Georgiou, D., Lambrinoudakis, C.: Cloud computing security requirements and a methodology for their auditing. In International Conference on e-Democracy, pp. 51–61. Springer, Cham (2015) 13. Georgiou, D., Lambrinoudakis, C.: Security policy rules and required procedures for two crucial cloud computing threats. Int. J. Electron. Gov. 9(3–4), 385–403 (2017) 14. Ryser, J., Glinz, M.: SCENT—a method employing scenarios to systematically derive test cases for system test, Technical Report 2000.03, Institut für Informatik, University of Zurich (2000) 15. Health Insurance Portability and Accountability Act: Healthcare data breach report. https:// www.hipaajournal.com/january-2019-healthcare-data-breach-report/ (2019). Accessed 23 Jan 2019 16. Alex, M.-H.K.: Opportunities and challenges of cloud computing to improve health care services. J. Med. Internet Res. (2011) 17. Abdul Manan, I.A.: Opportunities and threats of cloud computing in healthcare. Int. J. Comput. Appl. 10(2), 0975–887 (2014) 18. Abbas, A., Khan, M., Ali, M., Khan, S., Yang, L.: A cloud based framework for identification of influential health experts from Twitter. In Proceedings of the 15th International Conference on Scalable Computing and Communications (ScalCom), Beijing, China, pp. 831–838 (2015) 19. Velte, A.T., Velte, T.J., Elsenpeter, R.C., Elsenpeter, R.C.: Cloud computing: a practical approach, p. 44. McGraw-Hill, New York (2010)