Intelligent Systems and Applications: Proceedings of the 2020 Intelligent Systems Conference (IntelliSys) Volume 3 [1st ed.] 9783030551896, 9783030551902

Citation preview

Advances in Intelligent Systems and Computing 1252

Kohei Arai Supriya Kapoor Rahul Bhatia   Editors

Intelligent Systems and Applications Proceedings of the 2020 Intelligent Systems Conference (IntelliSys) Volume 3

Advances in Intelligent Systems and Computing Volume 1252

Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Advisory Editors Nikhil R. Pal, Indian Statistical Institute, Kolkata, India Rafael Bello Perez, Faculty of Mathematics, Physics and Computing, Universidad Central de Las Villas, Santa Clara, Cuba Emilio S. Corchado, University of Salamanca, Salamanca, Spain Hani Hagras, School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK László T. Kóczy, Department of Automation, Széchenyi István University, Gyor, Hungary Vladik Kreinovich, Department of Computer Science, University of Texas at El Paso, El Paso, TX, USA Chin-Teng Lin, Department of Electrical Engineering, National Chiao Tung University, Hsinchu, Taiwan Jie Lu, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW, Australia Patricia Melin, Graduate Program of Computer Science, Tijuana Institute of Technology, Tijuana, Mexico Nadia Nedjah, Department of Electronics Engineering, University of Rio de Janeiro, Rio de Janeiro, Brazil Ngoc Thanh Nguyen , Faculty of Computer Science and Management, Wrocław University of Technology, Wrocław, Poland Jun Wang, Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong

The series “Advances in Intelligent Systems and Computing” contains publications on theory, applications, and design methods of Intelligent Systems and Intelligent Computing. Virtually all disciplines such as engineering, natural sciences, computer and information science, ICT, economics, business, e-commerce, environment, healthcare, life science are covered. The list of topics spans all the areas of modern intelligent systems and computing such as: computational intelligence, soft computing including neural networks, fuzzy systems, evolutionary computing and the fusion of these paradigms, social intelligence, ambient intelligence, computational neuroscience, artificial life, virtual worlds and society, cognitive science and systems, Perception and Vision, DNA and immune based systems, self-organizing and adaptive systems, e-Learning and teaching, human-centered and human-centric computing, recommender systems, intelligent control, robotics and mechatronics including human-machine teaming, knowledge-based paradigms, learning paradigms, machine ethics, intelligent data analysis, knowledge management, intelligent agents, intelligent decision making and support, intelligent network security, trust management, interactive entertainment, Web intelligence and multimedia. The publications within “Advances in Intelligent Systems and Computing” are primarily proceedings of important conferences, symposia and congresses. They cover significant recent developments in the field, both of a foundational and applicable character. An important characteristic feature of the series is the short publication time and world-wide distribution. This permits a rapid and broad dissemination of research results. ** Indexing: The books of this series are submitted to ISI Proceedings, EI-Compendex, DBLP, SCOPUS, Google Scholar and Springerlink **

More information about this series at http://www.springer.com/series/11156

Kohei Arai Supriya Kapoor Rahul Bhatia •

•

Editors

Intelligent Systems and Applications Proceedings of the 2020 Intelligent Systems Conference (IntelliSys) Volume 3

123

Editors Kohei Arai Saga University Saga, Japan

Supriya Kapoor The Science and Information (SAI) Organization Bradford, West Yorkshire, UK

Rahul Bhatia The Science and Information (SAI) Organization Bradford, West Yorkshire, UK

ISSN 2194-5357 ISSN 2194-5365 (electronic) Advances in Intelligent Systems and Computing ISBN 978-3-030-55189-6 ISBN 978-3-030-55190-2 (eBook) https://doi.org/10.1007/978-3-030-55190-2 © Springer Nature Switzerland AG 2021 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Editor’s Preface

This book contains the scientific contributions included in the program of the Intelligent Systems Conference (IntelliSys) 2020, which was held during September 3–4, 2020, as a virtual conference. The Intelligent Systems Conference is a prestigious annual conference on areas of intelligent systems and artificial intelligence and their applications to the real world. This conference not only presented state-of-the-art methods and valuable experience from researchers in the related research areas, but also provided the audience with a vision of further development in the fields. We have gathered a multi-disciplinary group of contributions from both research and practice to discuss the ways how intelligent systems are today architectured, modeled, constructed, tested and applied in various domains. The aim was to further increase the body of knowledge in this specific area by providing a forum to exchange ideas and discuss results. The program committee of IntelliSys 2020 represented 25 countries, and authors submitted 545 papers from 50+ countries. This certainly attests to the widespread, international importance of the theme of the conference. Each paper was reviewed on the basis of originality, novelty and rigorousness. After the reviews, 214 were accepted for presentation, out of which 177 papers are finally being published in the proceedings. The conference would truly not function without the contributions and support received from authors, participants, keynote speakers, program committee members, session chairs, organizing committee members, steering committee members and others in their various roles. Their valuable support, suggestions, dedicated commitment and hard work have made the IntelliSys 2020 successful. We warmly thank and greatly appreciate the contributions, and we kindly invite all to continue to contribute to future IntelliSys conferences.

v

vi

Editor’s Preface

It has been a great honor to serve as the General Chair for the IntelliSys 2020 and to work with the conference team. We believe this event will certainly help further disseminate new ideas and inspire more international collaborations. Kind Regards, Kohei Arai Conference Chair

Contents

Multimodal Affective Computing Based on Weighted Linear Fusion . . . Ke Jin, Yiming Wang, and Cheng Wu

1

Sociophysics Approach of Simulation of Mass Media Effects in Society Using New Opinion Dynamics . . . . . . . . . . . . . . . . . . . . . . . . Akira Ishii and Nozomi Okano

13

ORTIA: An Algorithm to Improve Quality of Experience in HTTP Adaptive Bitrate Streaming Sessions . . . . . . . . . . . . . . . . . . . . Usman Sharif, Adnan N. Qureshi, and Seemal Afza

29

Methods and Means for Analyzing Heat-Loss in Buildings for Increasing Their Energy Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . Veneta Yosifova

45

The Energy Conservation and Consumption in Wireless Sensor Networks Based on Energy Efficiency Clustering Routing Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gaudence Stanslaus Tesha and Muhamed Amanul Intelligent Control of Traffic Flows Under Conditions of Incomplete Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elena Sofronova and Askhat Diveev

55

73

Virtual Dog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . José Luis Pastrana-Brincones

88

Context-Aware Transfer of Task-Based IoT Service Settings . . . . . . . . . Michael Zipperle, Achim Karduck, and In-Young Ko

96

Agent-Based Architectural Models of Supply Chain Management in Digital Ecosystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Alexander Suleykin and Natalya Bakhtadze

vii

viii

Contents

A Deep Insight into Signature Verification Using Deep Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Umair Muneer Butt, Fatima Masood, Zaib unnisa, Shahid Razzaq, Zaheer Dar, Samreen Azhar, Irfan Abbas, and Munib Ahmad Measures to Ensure the Reliability of the Functioning of Information Systems in Respect to State and Critically Important Information Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Askar Boranbayev, Seilkhan Boranbayev, and Askar Nurbekov IoTManager: Concerns-Based SDN Management Framework for IoT Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Radwa Hamed, Mohamed Rizk, and Bassem Mokhtar JomImage: Weight Control with Mobile SnapFudo . . . . . . . . . . . . . . . . 168 Viva Vivilyana, P. S. JosephNg, A. S. Shibghatullah, and H. C. Eaw Smart Assist System for Driver Safety . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Etee Kawna Roy and Shubhalaxmi Kher On the Applicability of 2D Local Binary Patterns for Identifying Electrical Appliances in Non-intrusive Load Monitoring . . . . . . . . . . . . 188 Yassine Himeur, Abdullah Alsalemi, Faycal Bensaali, Abbes Amira, Christos Sardianos, Iraklis Varlamis, and George Dimitrakopoulos Management of Compressed Air to Reduce Energy Consumption Using Intelligent Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 Mohamad Thabet, David Sanders, Malik Haddad, Nils Bausch, Giles Tewkesbury, Victor Becarra, Tom Barker, and Jake Piner Multi-platform Mission Planning Based on Distributed Planning Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Yang Guo and Shao-chi Cheng Development of Artificial Intelligence Based Module to Industrial Network Protection System . . . . . . . . . . . . . . . . . . . . . . . . 229 Filip Holik, Petr Dolezel, Jan Merta, and Dominik Stursa Learning and Cognition in Financial Markets: A Paradigm Shift for Agent-Based Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Johann Lussange, Alexis Belianin, Sacha Bourgeois-Gironde, and Boris Gutkin Agent-Based Simulation for Testing Vehicle-On-Demand Services in Rural Areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 Marius Becherer and Achim Karduck Overcrowding Detection Based on Crowd-Gathering Pattern Model . . . 270 Liu Bai, Chen Wu, and Yiming Wang

Contents

ix

Multi-person Spatial Interaction in a Large Immersive Display Using Smartphones as Touchpads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Gyanendra Sharma and Richard J. Radke Predicting Vehicle Passenger Stress Based on Sensory Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Dario Niermann and Andreas Lüdtke Infinite Mixtures of Gaussian Process Experts with Latent Variables and its Application to Terminal Location Estimation from Multiple-Sensor Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Ryo Hanafusa, Jiro Ebara, and Takeshi Okadome Flying Sensor Network Optimization Using Bee Intelligence for Internet of Things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 Abdu Salam, Qaisar Javaid, Gohar Ali, Fahad Ahmad, Masood Ahmad, and Ishtiaq Wahid A Quantum Model for Decision Support in a Sensor Network . . . . . . . 340 Shahram Payandeh Design and Implementation of a Flexible Platform for Remote Monitoring of Environmental Variables . . . . . . . . . . . . . . . . . . . . . . . . . 353 Francisco de Izaguirre, Maite Gil, Marco Rolón, Nicolás Pérez, and Pablo Monzón Adaptable Embedding Algorithm to Secure Stream Data in the Wireless Sensor Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 Mohammad Amanul Islam Intelligent Monitoring Systemof Environmental Biovariables in Poultry Farms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 Gabriela Chiluisa-Velasco, Johana Lagla-Quinaluisa, David Rivas-Lalaleo, and Marcelo Alvarez-Veintimilla Effect of Analysis Window and Feature Selection on Classification of Hand Movements Using EMG Signal . . . . . . . . . . . . . . . . . . . . . . . . . 400 Asad Ullah, Sarwan Ali, Imdadullah Khan, Muhammad Asad Khan, and Safiullah Faizullah A Convolutional Neural Network Approach for Quantification of Tremor Severity in Neurological Movement Disorders . . . . . . . . . . . . 416 Rajesh Ranjan, Braj Bhushan, Marimuthu Palaniswami, and Alok Verma Brain MR Imaging Segmentation Using Convolutional Auto Encoder Network for PET Attenuation Correction . . . . . . . . . . . . . . . . 430 Imene Mecheter, Abbes Amira, Maysam Abbod, and Habib Zaidi

x

Contents

Colorimetric Analysis of Images Based on Objective Color Data . . . . . . 441 Valery V. Bakutkin, Ilya V. Bakutkin, Yuriy N. Zayko, and Vladimir A. Zelenov A Deep Learning-Based Approach for the Classification of Gait Dynamics in Subjects with a Neurodegenerative Disease . . . . . . 452 Giovanni Paragliola and Antonio Coronato Smartphone-Based Diabetic Retinopathy Severity Classification Using Convolution Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 Sarah Sheikh and Uvais Qidwai Alzheimer Disease Prediction Model Based on Decision Fusion of CNN-BiLSTM Deep Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . 482 Shaker El-Sappagh, Tamer Abuhmed, and Kyung Sup Kwak On Mistakes We Made in Prior Computational Psychiatry Data Driven Approach Projects and How They Jeopardize Translation of Those Findings in Clinical Practice . . . . . . . . . . . . . . . . . . . . . . . . . . 493 Milena Čukić, Dragoljub Pokrajac, and Viktoria Lopez Machine Learning Strategies to Distinguish Oral Cancer from Periodontitis Using Salivary Metabolites . . . . . . . . . . . . . . . . . . . . 511 Eden Romm, Jeremy Li, Valentina L. Kouznetsova, and Igor F. Tsigelny Smart Guide System for Blind People by Means of Stereoscopic Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527 Jesús Jaime Moreno Escobar, Oswaldo Morales Matamoros, Ricardo Tejeida Padilla, Jhonatan Castañón Martínez, and Mario Mendieta López An IoMT System for Healthcare Emergency Scenarios . . . . . . . . . . . . . 545 Tomás Jerónimo, Bruno Silva, and Nuno Pombo Introducing Time-Delays to Analyze Driver Reaction Times When Using a Powered Wheelchair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559 David Sanders, Malik Haddad, Martin Langner, Peter Omoarebun, John Chiverton, Mohamed Hassan, Shikun Zhou, and Boriana Vatchova Intelligent Control and HCI for a Powered Wheelchair Using a Simple Expert System and Ultrasonic Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . 571 David Sanders, Malik Haddad, Peter Omoarebun, Favour Ikwan, John Chiverton, Shikun Zhou, Ian Rogers, and Boriana Vatchova Intelligent System to Analyze Data About Powered Wheelchair Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584 Malik Haddad, David Sanders, Martin Langner, Mohamad Thabet, Peter Omoarebun, Alexander Gegov, Nils Bausch, and Khaled Giasin

Contents

xi

Intelligent Control of the Steering for a Powered Wheelchair Using a Microcomputer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594 Malik Haddad, David Sanders, Martin Langner, Nils Bausch, Mohamad Thabet, Alexander Gegov, Giles Tewkesbury, and Favour Ikwan Intelligent Risk Prediction of Storage Tank Leakage Using an Ishikawa Diagram with Probability and Impact Analysis . . . . . . . . . 604 Favour Ikwan, David Sanders, Malik Haddad, Mohamed Hassan, Peter Omoarebun, Mohamad Thabet, Giles Tewkesbury, and Branislav Vuksanovic Use of the Analytical Hierarchy Process to Determine the Steering Direction for a Powered Wheelchair . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 Malik Haddad, David Sanders, Mohamad Thabet, Alexander Gegov, Favour Ikwan, Peter Omoarebun, Giles Tewkesbury, and Mohamed Hassan Methodology of Displaying Surveillance Area of CCTV Camera on the Map for Immediate Response in Border Defense Military System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631 Hyungheon Kim, Taewoo Kim, and Youngkyun Cha Detecting Control Flow Similarities Using Machine Learning Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638 André Schäfer and Wolfram Amme Key to Artificial Intelligence (AI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647 Bernhard Heiden and Bianca Tonino-Heiden US Traffic Sign Recognition Using CNNs . . . . . . . . . . . . . . . . . . . . . . . . 657 W. Shannon Brown, Kaushik Roy, and Xiaohong Yuan Grasping Unknown Objects Using Convolutional Neural Networks . . . . 662 Pranav Krishna Prasad, Benjamin Staehle, Igor Chernov, and Wolfgang Ertel A Proposed Technology IoT Based Ecosystem for Tackling the Marine Beach Litter Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673 Stavros T. Ponis Machine Learning Algorithms for Preventing IoT Cybersecurity Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679 Steve Chesney, Kaushik Roy, and Sajad Khorsandroo Development of Web-Based Management System and Dataset for Radiology-Common Data Model (R-CDM) and Its Clinical Application in Liver Cirrhosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687 SeungJin Kim, Chang-Won Jeong, Tae-Hoon Kim, ChungSub Lee, Si-Hyeong Noh, Ji Eon Kim, and Kwon-Ha Yoon

xii

Contents

Shared Autonomy in Web-Based Human Robot Interaction . . . . . . . . . 696 Yug Ajmera and Arshad Javed tanh Neurons Are Bayesian Decision Makers . . . . . . . . . . . . . . . . . . . . . 703 Christian Bauckhage, Rafet Sifa, and Dirk Hecker Solving Jigsaw Puzzles Using Variational Autoencoders . . . . . . . . . . . . 708 Mostafa Korashy, Islam A. T. F. Taj-Eddin, Mahmoud Elsaadany, and Shoukry I. Shams Followers of School Shooting Online Communities in Russia: Age, Gender, Anonymity and Regulations . . . . . . . . . . . . . . . . . . . . . . . . . . . 713 Anastasia Peshkovskaya, Yuliya Mundrievskaya, Galina Serbina, Valeria Matsuta, Vyacheslav Goiko, and Artem Feshchenko Discrimination of Chronic Liver Disease in Non-contrast CT Images using CNN-Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 717 Tae-Hoon Kim, Si-Hyeong Noh, Chang-Won Jeong, ChungSub Lee, Ji Eon Kim, SeungJin Kim, and Kwon-Ha Yoon Analysis and Classification of Urinary Stones Using Deep Learning Algorithm: A Clinical Application of Radiology-Common Data Model (R-CDM) Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723 Si-Hyeong Noh, SeungJin Kim, Ji Eon Kim, Chung-Sub Lee, Seng Chan You, Tae-Hoon Kim, Yun Oh Lee, Ilseok Chae, Rae Woong Park, Sung Bin Park, Kwon-Ha Yoon, and Chang-Won Jeong Intelligent Monitoring Using Hazard Identification Technique and Multi-sensor Data Fusion for Crude Distillation Column . . . . . . . . 730 Peter Omoarebun, David Sanders, Favour Ikwan, Mohamed Hassan, Malik Haddad, Mohamad Thabet, Jake Piner, and Amjad Shah Factors Affecting the Organizational Readiness to Design Autonomous Machine Systems: Towards an Evaluation Framework . . . 742 Valtteri Vuorimaa, Eetu Heikkilä, Hannu Karvonen, Kari Koskinen, and Jouko Laitinen RADAR: Fast Approximate Reverse Rank Queries . . . . . . . . . . . . . . . . 748 Sourav Dutta Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 759

Multimodal Affective Computing Based on Weighted Linear Fusion Ke Jin(B) , Yiming Wang, and Cheng Wu School of Urban Rail Transportation, Soochow University, Suzhou, China [email protected]

Abstract. Affective computing is one of the most important research directions in human-computer interaction system which gains increasing popularity. However, the traditional affective computing methods all make decisions based on unimodal signals, which has low accuracy and poor feasibility. In this article, the final classification decision is made from the perspective of multimodal fusion results which combines the decision results of both text emotion network and visual emotion network through weighted linear fusion algorithm. It is obvious that the speaker’s intention can be better understood by observing the speaker’s expression, listening to the speaker’s tone of voice and analyzing the words. Combining auditory, visual, semantic and other modes certainly provides more information than a single mode. Video information often contains a variety of modal characteristics, only one mode is always not enough to describe all aspects of the overall video stream characteristic information. Keywords: Semantic affective analysis · Facial expression analysis · Human-computer interaction · Weighted linear fusion

1 Introduction With the development of the social network people have more and more diversified ways to express their emotions on social platform. Such as through pictures, videos and texts. So how to recognize emotions in multimodal data (paper, voice, image, text, and sensor data) is an opportunity and challenge facing the current field of emotion analysis. Previous affective computing focus on analyzing data got in single mode. But the results got by this way are incomplete. Multimodal data contains more information than unimodal data [1]. And multiple modes can complement with each other which can help AI system understand the user’s emotion better. From the perspective of humancomputer interaction, multimodal affective computing can make machines interact with people in a more natural way. AI system can interpret the user’s emotions based on the expressions and gestures of the people in the images, the tones in the voice, and the natural languages in text. To sum up, the development of multimodal affective computing technology stems from the needs of real life, which makes machines understand emotions more accuracy. © Springer Nature Switzerland AG 2021 K. Arai et al. (Eds.): IntelliSys 2020, AISC 1252, pp. 1–12, 2021. https://doi.org/10.1007/978-3-030-55190-2_1

2

K. Jin et al.

Because the traditional emotion recognition problems are predicted in the standard library and controlled environment, the results of single-task and single-perspective model cannot truly reflect the actual emotional state of the tested personnel. Therefore, we need to learn the joint model based on the data of different modes through multiperspective and multi-task methods, so as to improve the accuracy of emotion state estimation, which will meet the needs of practical engineering. Therefore, to create a new database which integrates with a variety of emotional factors according to the demands of a real practical engineering is necessary. And the algorithm must be highly efficient and easy to implement. The timeless of the system is also important in addition to the way of understanding the emotion. Most of algorithms are proposed based on the standard database (CK+, FER2013, voice database), which has poor real-time performance and is difficult to be applied into a real project. And the database itself has shortcomings, such as incomplete database, small data volume and many other factors [2]. The evaluation index of the algorithm only cares about the accuracy of the standard database but ignores the timeless, feasibility, practicability in the actual engineering application. Although the algorithm can achieve high accuracy on the standard database, the efficiency and accuracy in practical application are still not high. The rest of the paper is organized as below. Section 2 describes the related work in affective computing and summarizes the shortages and advantages in mainstream research direction. Section 3 describes the whole architecture and algorithm. Section 4 describes the experiment environment and shows the simulation results and curves. Section 5 gets the conclusion.

2 Related Work Due to the importance of affective computing, the research of affective has been widely concerned by researchers in facial expressions, speech identification, and natural language process (NLP). A lot of research has been carried out in the fields of classification and fusion of information from different modal, which makes great progress. In 1971, Ekman and Friesen suggested that human emotions can be divided into six kinds: happy, sad, angry, disgust, surprise and fear [3]. Furthermore, FACS [4] was proposed from the perspective of anatomy. Since human being expressed emotions in a variety of ways, the analysis of information based on single mode is not enough and incomplete. In fact human behavior in multimodal affective computing is mutually complementary and indispensable [5]. Currently main signal mode of affective computing is visual, audio, NLP and physiological signal. 2.1 Emotion Analysis Based on Visual Signal Visual is the main way of human beings to perceive outside world. In terms of emotion recognition, facial expression is undoubtedly one of the most important emotion pattern. The algorithm implementation steps of facial expression recognition is divided into face detection and pre-processing, expression feature extraction and expression feature classification [5]. In the field of face detection, traditional and typical methods are based

Multimodal Affective Computing Based on Weighted Linear Fusion

3

on VJ detector relies on Ada boost and Haar characteristic [6], face detector relies on LBP [7] characteristic and DPM [8] detector relies on HOG characteristic [9]. These kinds of face detector perform well in controlled experimental environment but the performance degrades in outdoor scenes in practice. The mainly reason is the features extracted manually are sensitive to illumination, light, occlusion and gesture [8, 10]. In recent years deep learning performs well and gradually becomes a mainstream research direction in the field of facial expression. Facial expression recognition and face detection based on deep learning can be divided into three categories: Cascade CNN method [11–13], proposal based on twostep approach [14, 15], one-step method [16–19]. In the aspect of visual expression feature extraction, traditional method is extracted features manually then categorizes the features by using machine learning methods. For example, Gabor filter in reference [20], LBP features in reference [21], HOG features in reference [22]. However the performance of these manual extracted features is very poor in the actual scene, which is easy to disturbed by the outside environment. Deep learning can extract appropriate features automatically. So features of facial expression are mostly extracted by deep neutral network. But there still exist problems in deep learning method. Deep learning neutral network needs a lot of time to train the parameter, which needs high quality of training data. And the training method and loss-function is also difficult to define [23]. 2.2 Emotion Analysis Based on Text Emotion recognition based on text is an important research direction in NLP (natural language process), which analyzes, processes and extracts subjective texts by using text mining technique. The application of emotion analysis in text is widely increasing. Especially emotion recognition in conversations (ERC) is a challenging task that has recently gained increasing popularity due to its potential applications. According to Poria et al. (2019) [24], ERC presents several challenges such as conversational context modeling, emotion shift of the interlocutors, and others, which make the task more difficult to address. Many scholars have introduced RNN and CNN into the field of text mining to extract emotional words to judge whether the whole sentence is positive or negative. But the pure text emotion ignores the information of the whole context, and it is not very accurate to predict the actual person’s emotion. Simply judging the emotion of words in a sentence isolating from context can’t really reflect a person’s emotion. 2.3 Multimodal Fusion Affective Computing Because of the shortcomings of unimodal emotion analysis, more and more scholars devote themselves to the study in multimodal affective computing. However, due to the great difference of different signals in different feature spaces, it is difficult to reflect the characteristics of different signals at the same time. Two kinds of fusion methods are adopted: pre-classifier emotion feature expression fusion and classifier fraction fusion. Mainly fusion technique can be divided into feature-level fusion, decision-level fusion, hybrid-multimodal fusion, model-level fusion and rule-based level [25]. The Reference [26] proposed a Mod drop multimodal fusion method [25], which is proved to be more accurately and efficiently than the way of feature vector fusion.

4

K. Jin et al.

Reference [27] proposed CCA (Canonical Correlation Analysis) method to carry out the fusion feature according to the correlation of different features. However neither method consider scenario and context information. It is necessary to consider that not only the speaker information but also the scene and context information of the dialogue, and the result is greatly influenced by the content of the previous dialogue. Due to this issue, Ref. [28] use dialog RNN which contains 3 GRU to model the contextual information. But in the model, the global contextual state, the actor state, needs to be predefined. In practical engineering, the global context state is not constant. The people involved in the conversation are not fixed. Dialog information from past scenarios can affect current states. Therefore, the real-time performance of this algorithm is not very good, and it is difficult to achieve. To solve the above problems, this paper designs the networks for text and semantics respectively and trains both the network parameters according to the dialogues and facial expressions in the paragraphs. The results of expression network and semantic network are fused by linear weighting algorithm. And the judgement of the sentiment is made from the fusion result vector. This approach takes into account both contextual information and the speaker’s multiple emotional modes: expression and semantics. The two modes compensate with each other and improve the accuracy of emotion recognition results. In order to reflect the real-time performance of the whole network, based on the original MELD database, we collected the facial expressions of people in the video of corresponding sentences and established the corresponding facial expressions database.

3 Overall Algorithm Architecture 3.1 Semantic Network Architecture We chose text data from MELD (Multimodal Emotion Lines Dataset) database as training and testing data of semantic network, which contain dialogues from the popular TVseries Friends with more than two speakers. There are 10478 sentences and 1038 scenes. Overall, we built the training set, test set and validation set according to the paragraph scenario. We define 24 sentences per scene, with a maximum of 50 words per sentence. Each word corresponds to a coded number. If not enough sentence in a dialog or not enough words in a sentence, zero padding is available. So the dimension of the training tensor is (None, 24, 50). We divided sentiments into 3 categories: negative, neutral and positive. We design a multi-layer CNN to train the tensors. The CNN contains three convolutional layers, three pooling layers, and one fully connected layer. The full connected layer define as softmax. We choose the reLU function as activation function in convolutional layer and tanh function as activate function in full connected layer. The architecture of the CNN is shown as Fig. 1. The training method uses adadelta. And the loss function is categorical cross entropy. Semantic features are extracted by three-layer convolutional network and generalized and pooled by three-layer pooling layer to reduce computation. Semantic features are integrated through embedding. We put 24 sentences in a context into a whole paragraph. We trained the parameter in paragraph. In this way, the network can fully learn the contextual information, so that the network get higher accuracy in emotion recognition.

Multimodal Affective Computing Based on Weighted Linear Fusion Start

Epoch=100

Batch_size=50

embedding_dim

Vocabulary_size

3*embedding_dim activaltion function:reLU (num of convolutional filter is 512) Conv_2D:

4*embedding_dim, Activation function :reLU (num of convolutional filter is 512) Conv_2D:

Conv_2D:5*embedding_dim, activation function:reLU (num of convolutional filter is 512)

MaxPool2D max_pool the dimension of pool filter (sentence_length - 3 + 1, 1)

MaxPool2D max_pool the dimension of pool filter (sentence_length - 4 + 1, 1)

MaxPool2D max_pool the dimension of pool filter (sentence_length - 5 + 1, 1)

Full connect layer:Dense,100 neuron nodes activation function:tanh

Output layer:Dense,The number of neurons is the number of classes needed

Reshape layer

End

Fig. 1. The architecture of semantic CNN

5

6

K. Jin et al.

3.2 Facial Expression Recognition 3.2.1 Facial Database Established We collected the facial expressions images of people in the video of corresponding sentences and established the corresponding facial expressions database. The facial images used in facial expression are all manually extracted by the tool LABELME. Then we located the middle point of two eyes which is used as center of rotation to rotate the face to align the facial features. The face is shown as Fig. 2.

Fig. 2. Facial image after aligned

Each facial image corresponds to the sentence in the semantic network. The serial number before the facial image is the number of the sentences in the MELD. Each facial image is named in the same format. In the middle of the name is the corresponding sentiment classification label. The characters in the video were trained through a multilayer CNN network. The facial expressions also divided into 3 labels: negative sentiment, neutral sentiment and positive sentiment. 3.2.2 Facial Expression CNN Architecture The CNN neutral network contains 3 convolutional layers, 4 max pool layers, 2 full connected layers, 2 dropout layers. The whole architecture is shown as Fig. 3. During the training process, the optimizer is selected as Adam, and the loss function is selected as: sparse categorical cross entropy. The initial learning rate is chosen to be 1e−3. As the training goes on, when the error of the loss function reaches the set value of the threshold, the learning rate is modified to 1e−4 to fine-tune the parameter. 3.2.3 Linear Fusion Weighted Algorithm Linear weighted fusion algorithm is one of the simplest fusion algorithm to apply, which is very convenient for engineering implementation. Each sample included one sentence

Multimodal Affective Computing Based on Weighted Linear Fusion Start

Model=Sequential()

Input tensor width=128 Height=128 channels=1 Convolutional layer::The number of conv filter is 32,the dimension of each filter is (5,5),activation function is reLU

Max pool layer: (2,2)

Convolutional layer::The number of conv filter is 32,the dimension of each filter is (3,3),activation function is reLU

Max pool layer: (2,2)

Convolutional layer The number of conv filter is 64,the dimension of each filter is (3,3),activation function is reLU

Max pool layer: (2,2)

Convolutional layer The number of conv filter is 64,the dimension of each filter is (5,5),activation function is reLU

Max pool layer: (2,2)

Flatten()

Full connected layer number of nodes is 512 activation function is :reLU

dropout:0.5

Full connected layer number of nodes is 512 activation function:reLU

dropout:0.5

Output layer:the number of nodes is the number of classes needed

end

Fig. 3. The architecture of facial expression CNN

7

8

K. Jin et al.

and several facial images corresponding to the dialogue. Both the facial expression network and the semantic network will output the probability corresponding to three emotional categories. By linear weighted fusion of the outputs of the two networks, we can obtain a fused output vector. We make emotional classification based on the classification vector after fusion, the dimension of the fusion vector is 3*1. The index of the largest element is the final classification of the emotion. The algorithm is shown as formula (1). The overall fusion architecture is shown in Fig. 4.  (1) Score(u, i) = βk reck (u, i)

Fig. 4. Overall a