Trends and Applications in Information Systems and Technologies: Volume 2 (Advances in Intelligent Systems and Computing, 1366) 3030726509, 9783030726508
163 22 54MB
English Pages [635]
Preface
Contents
Information and Knowledge Management
GoVegan: Exploring Motives and Opinions from Tweets
1 Introduction
2 Literature Review
3 Methods
3.1 Data Collection
3.2 Clustering
3.3 Interrater Reliability
4 Results
5 Conclusion
References
Five Hundred Most-Cited Papers in the Computer Sciences: Trends, Relationships and Common Factors
1 Introduction
2 Credibility, Article Length, Number of Authors and Field of Study
3 Method
4 Result
5 Conclusion
References
Reasons of Individuals to Trust What They Read on Social Network Sites
1 Introduction
2 Signaling Theory
3 Model and Method
4 Results for the German Sample
5 Results for the English Sample
6 Results of the Regression Analysis
7 Conclusion
References
Perspectives of Companies and Employees from the Great Place To Work (GPTW) Ranking on Remote Work in Portugal: A Methodological Proposal
1 Introduction
2 Background
2.1 Remote Work, Telework, Work at Home or Home-Based Work - Definition
2.2 Remote Work During COVID-19
2.3 Remote Work in Portugal – Wellbeing and Working Conditions
3 Research Design and Methodology
3.1 Research Design
3.2 Sample
3.3 Data Collection and Analysis
4 Conclusion
References
Overlaps Between Business Intelligence and Customer Relationship Management – Is There a Place for E-Commerce?
1 Introduction
2 Method
2.1 Search Strategy
2.2 Screening
2.3 Coding, Data Extraction, and Analysis
3 Results
3.1 Quantitative Analysis
3.2 Content Analysis
3.3 Thematic Analysis
4 Conclusions
References
Perception About of COVID-19 Information in Ecuador
1 Introduction
2 Methodology
3 Results
4 Conclusions
References
Impact of ICT in Rural Areas: Perceptions of Portugal, Spain and Russia
1 Introduction
2 ICT for SMEs in Emerging Economies
3 Case Study
3.1 Regions
3.2 Methodology
3.3 Results
4 Conclusion
References
AMALGAM: A Matching Approach to Fairfy TabuLar Data with KnowledGe GrAph Model
1 Introduction
2 Related Work
3 The AMALGAM Approach
4 Experimental Results
5 Conclusion and Future Works
References
Knowledge in Transition in an Industrial Company
1 Introduction
2 Methodology
3 Findings of the Research
3.1 The Context of the Company Regarding Innovation and Knowledge
3.2 Knowledge Transition Processes
3.3 Knowledge Transition Impacts
4 Conclusions
References
Project Management Practice in SMEs: A Comparative Study of the Portuguese and Danish Economic Context
1 Introduction
2 Background
3 A Comparative Analysis of the Portuguese and Danish Economic Scenario in the Context of SMEs
3.1 Economic Scenario of SMEs
3.2 Priority Policies and Measures for SMEs
3.3 Project Management in SMEs
4 Discussion
5 Conclusions
References
Critical Management Risks in Collaborative University-Industry R&D Programs
1 Introduction
2 Background
3 Research Methodology
3.1 Research Strategy and Methods
3.2 Case Study Background
4 Results
5 Discussion and Conclusions
References
A Project Management Hybrid Model of Software Development in an Academic Environment
1 Introduction
2 Literature Review
2.1 Project Management Concepts
2.2 Project Management in IT Sector
3 Research Methodology
3.1 Choices Made
3.2 Case Study Background
4 Results and Discussion
4.1 Initial Setting and Documents Produced
4.2 The Features of the Hybrid Model Used
4.3 Hybrid Model and Team Performance Evaluation Discussion
5 Conclusions
References
Computerised Sentiment Analysis on Social Networks. Two Case Studies: FIFA World Cup 2018 and Cristiano Ronaldo Joining Juventus
1 Introduction
2 Background
3 Methods
4 Results and Discussion
5 Conclusions
References
Modelling Academic Dropout in Computer Engineering Using Artificial Neural Networks
1 Introduction
2 Educational Data Mining
3 Data and Methodology
3.1 Data Model
3.2 Methodology
4 Implementation and Results
4.1 Training and Refinement of ANNs with All the Independent Variables
4.2 Selection of the Main Explanatory Factors of Dropout
4.3 Evaluation of the Generalization Capacity of the Model Found
4.4 Relative Importance of the Main Explanatory Factors
5 Conclusions and Future Work
References
Evolution of the Data Mining and Machine Learning Techniques Used in Health Care: A Scoping Review
1 Introduction
2 Method
3 Results
4 Discussion
5 Conclusions
References
Learning Analytics Metrics into Online Course’s Critical Success Factors
1 Introduction
2 Background and Related Works
2.1 Improving Online Courses in Literature
2.2 Factors Impacting Quality of Online Course Content
3 Data Analysis and Interpretation
3.1 Data Collection
3.2 Data Preparation
3.3 Data Exploration
3.4 Recommendations for Successful Online Course
4 Conclusion
References
The Influence of COVID-19 in Retail: A Systematic Literature Review
1 Introduction
2 Emerging Concepts
2.1 Epidemiological Summary (COVID-19)
2.2 The Supply Chain Management and Its Relation to COVID-19
3 Methodology
4 Findings
4.1 Quantitative Analysis
4.2 Qualitative Analysis
5 Concluding Remarks
References
Office Evolution from Ancient Age to Digital Age (E-working)
1 Introduction
2 Methodology
3 E-working: Conceptual Framework
4 The Evolution of the Office
5 Discussion and Conclusions
References
Measuring Variability in Acute Myocardial Infarction Coding Using a Statistical Process Control and Probabilistic Temporal Data Quality Control Approaches
1 Introduction
2 Methods
2.1 Data Sources
2.2 Multisource and Temporal Data Variability Assessment
3 Results
4 Discussion
5 Conclusions
References
Business Decision Making Based on Social Media Analysis
1 Introduction
2 Theoretical Framework
3 Growth and Advantages in the Use of Social Media
3.1 Global Analysis
3.2 New Airs on Social Media
3.3 Ages in Social Media
3.4 Analysis in Ecuador
3.5 2companies that Bet on the Use of Social Media
4 Proposal
4.1 Everything is on Social Media
5 Discussion
6 Conclusions
References
The Effects of Abrupt Changing Data in CART Inference Models
1 Introduction
2 Related Work
3 Influential Factors for Adaptive Algorithm Performance
4 Experimental Results
5 Conclusion and Future Work
References
Log Data Preparation for Predicting Critical Errors Occurrences
1 Introduction
2 Dataset Collection and Preliminary Definitions
3 Proposed Methodology
3.1 Summarization Strategy
3.2 Overall Data Preparation Process
4 Experiments
4.1 Dataset Description
4.2 Settings
4.3 Impact of the Error Interval Choice
5 Related Work
6 Conclusion and Future Works
References
Running Workshops to Identify Wastes in a Product Development Sample Shop
1 Introduction
2 Literature Review
2.1 Lean Thinking Principles
2.2 Lean Product Development
2.3 Industry 4.0 in the Sample Shop
3 Research Methodology
4 Running the Workshops
4.1 Workshop Preparation and Results
4.2 Root Causes for the Sample Delivery Impaired
4.3 Lean and Technological Solutions
5 Conclusions
References
Evolutionary Dynamics in Azorean Landscapes: The Land-Use Changes in Forests and Semi-natural Areas in the Archipelago from 1990 to 2018
1 Introduction
2 The Azores Landscape in Brief
3 Land-Uses Changes in Forests and Semi-natural Areas in Azores Archipelago in the Period of 1990–2018
4 Discussion and Conclusions
5 Prospective Research Lines
References
A Comparative Study of Classifier Algorithms for Recommendation of Banking Products
1 Introduction
2 Overview
2.1 MARS Algorithm
2.2 Naïve Bayes Classifier (NBC)
2.3 Support Vector Machine (SVM)
2.4 Neural Net (NNET)
2.5 Classification Metrics
3 Methodology
4 Materials
5 Results
6 Conclusions
7 Future Works
References
New Capabilities of the Geometric Multidimensional Scaling
1 Introduction
2 Overview of Geometric MDS
3 Joint Recalculation of the Coordinates of all Points
4 Dependence of Stress S(a) on a: Search for Minimum
5 Conclusions
References
Hope Amid a Pandemic: Is Psychological Distress Alleviating in South America While Coronavirus Is Still on Surge?
1 Introduction
2 Data Process Analysis
2.1 Selecting the Scope and Social Network
2.2 Find the Relevant Terms to Search
2.3 Build the Query to Collect Twitter Data
2.4 Preprocessing of the Data
2.5 Visualization
3 Results
3.1 Dataset Description
3.2 Evolution of Interest on Twitter Related to Psychological Distress
3.3 Validating the Analysis Using Google Trend
4 Discussion
5 Conclusions
6 Future Research
References
An Entropic Approach to Assess People’s Awareness of the Health Risks Posed by Pesticides in Oenotourism Events
1 Introduction
2 Background
2.1 Thermodynamics and Knowledge Representation and Reasoning
2.2 The Logic Programming Framework
3 Methods
4 A Thermodynamics View of Data Processing
5 Awareness of Health Risks Assessment – A Logic Programming Approach
6 Conclusions and Future Work
References
Knowledge Management Strategies Through Educational Digital Platforms During Periods of Social Confinement
1 Introduction
2 Materials and Methods
3 Analysis and Discussion of the Results
4 Conclusion
5 Future Work
References
Information Foraging on Social Media Using Elephant Herding Optimization
1 Introduction
2 Modeling Information Foraging on Social Media
2.1 Social Networks Representation
2.2 User's Interests
2.3 Information Scent
2.4 Surfing Path -0.5em
3 Using EHO to Forage Information on Social Media
3.1 Elephant Population and Positions
3.2 Building a Surfing Path
3.3 Evaluating the Elephants' Solutions
3.4 Updating Operator
3.5 Separating Operator
4 Experiments
4.1 Dataset Description
4.2 Empirical Parameters Setting
4.3 Foraging Results
4.4 Comparative Study
5 Conclusion and Perspectives
References
Social Vulnerability Segmentation Methodology Based on Key Performance Indicators
1 Introduction
2 Objectives
3 Current Situation
4 Methodology
4.1 Step 1: Data Acquisition and Preprocessing
4.2 Step 2: Generation of KPI
4.3 Step 3: Generation of Vulnerability Axes
4.4 Step 4: Users Vulnerability Segmentation
5 Computational Experience and Results
6 Conclusions and Future Work
References
A Requirements Catalog of Mobile Geographic Information System for Data Collection
1 Introduction
2 Method
2.1 Requirements Specification
2.2 Catalog Development
3 Result
3.1 Identified Requirements
3.2 The Generated Catalog
3.3 Illustration
4 Discussion
5 Conclusion and Future Work
References
Keeping the Beat on: A Case Study of Spotify
1 Introduction
2 Literature Review
2.1 Online Music and Streaming Platforms
2.2 Music Consumption– Traditional vs Digital
2.3 Business Models on Streaming Platforms
2.4 Piracy
3 Spotify
3.1 Karaoke Feature
3.2 Tracking Behavioral Patterns
3.3 Podcasting as a Key Driver for Future Growth
3.4 Evolution of the Platform: Will It Include Video?
3.5 Spotify as a Label: Representing Artists
3.6 Business Models
4 Methodology
5 Results and Discussion
6 Conclusions and Future Research
References
Democratic Talent Management in an Industry 4.0 Environment of Digital Revolution
1 Introduction
2 A Look at the Literature – A Global Disruption Brought on by Technology
3 Understanding the Context
3.1 A Brave New World - Acting in the Unknown and the Unpredictable
3.2 Industry 4.0 – A New Way of Performing
3.3 The Concept of Talent
3.4 Talents and Competencies
4 Methodology
5 Interviews – Field Work
5.1 A Talent for Digital Marketing and Everything Online
5.2 A Talent for People and Establishing Empathy in the Digital World
5.3 A Talent for the Spoken and Written Word
5.4 Leader by Design
5.5 Firms and Employee Motivation
6 Discussion and Conclusions
References
Stimulating Components for Business Development in Latin American SMEs
1 Introduction
2 Materials and Methods
3 Analysis and Discussion of the Results
3.1 Financial Management
3.2 Innovation Processes
3.3 Knowledge Management
3.4 Marketing
4 Conclusion
5 Future Work
References
Handling Industrial Consumer Rights by Using Blockchain
1 Introduction
2 Overview
2.1 Blockchain Basic Concepts
2.2 Blockchain-Based Governance
2.3 Blockchain for Supply Chain
2.4 Blockchain as a Business Model (BBM)
3 Proposed Blockchain Business Model
3.1 Stakeholders and Information Flow
3.2 Mathematical Model
3.3 Proposed Blockchain Business Model (BBM)
3.4 Blockchain Structure Implementation
3.5 Comparing Proposed BBM with Traditional Supply Chain
4 Conclusion and Future Directions
References
How Health Data Are Managed in Mozambique
1 Introduction
2 Theoretical Background
2.1 The Case of Mozambique
2.2 Management of Health Data in Mozambique
3 Methodology
4 Results and Discussion
5 Conclusions
5.1 Practical and Theoretical Implications
5.2 Limitations and Future Research Avenues
References
Multi-perspective Conformance Checking Applied to BPMN-E2
1 Introduction
2 Background
3 Proposal
3.1 Design Goals
3.2 Conversion Phase
3.3 Conformance Checking Phase
4 Proof of Concept
5 Related Work
6 Conclusions and Future Work
References
Event-Driven Ontology Population - from Research to Practice in Critical Infrastructure Systems
1 Introduction
2 Promising Lines of Research
3 Hybrid Ontology Population Approach
4 Conclusion and Future Work
References
Semantic-Based Image Retrieval Using Balanced Clustering Tree
1 Introduction
2 Related Works
3 Semantic-Based Image Retrieval on C-Tree
3.1 Architecture of SBIRCT
3.2 Extracting the Feature Vector
3.3 Structure of C-Tree
3.4 Building the Ontology for Image Datasets
4 Search Algorithms on SBIRCT System
4.1 Content-Based Image Retrieval on C-Tree
4.2 Query Image Classification
4.3 Image Semantic Retrieval and Analysis
5 Experiment
5.1 Application
5.2 Experimental Results
6 Conclusion
References
NoSQL Comparative Performance Study
1 Introduction
2 Related Work
3 Experimental Setup
3.1 Hardware and Software Specifications
3.2 Dataset
3.3 Workloads
4 Experimental Results
4.1 Data Loading
4.2 Cassandra
4.3 MongoDB
4.4 Comparative Analysis: Cassandra Vs. MongoDB
5 Conclusions
References
A Model for Designing SMES' Digital Transformation Roadmap
1 Introduction
2 Industry 4.0 Overview
2.1 Industry 4.0 Most Common Reference Models
2.2 Assessing SMEs Readiness for I4.0
3 (Re)Conceptualisation of Industry 4.0
4 Experiment
5 Conclusion
References
Boosting E-Auditing Process Through E-Files Semantic Enrichment
1 Introduction
2 e-Auditing Process and Standards
2.1 Related Work on Tax E-Auditing
3 Semantic Representation of Tax E-Files
3.1 Exploring E-Files for Issues Findings
4 Conclusions and Future Work
References
Integration of UML Diagrams from the Perspective of Enterprise Architecture
1 Introduction
2 Related Work
3 Integrated View of UML Tools
4 Proposed Model
4.1 UML Diagrams for Enterprise Architecture
4.2 Heuristic Rules to Align the Enterprise Architecture
4.3 Procedure to Check Alignment
5 Conclusions
References
MongoDB, Couchbase, and CouchDB: A Comparison
1 Introduction
2 State of the Art/Related Work(s)
2.1 Scaling Capabilities
3 MongoDB, Couchbase and CouchDB
3.1 Couchbase
3.2 CouchDB
4 Results and Analysis
4.1 YCSB
4.2 Results
5 Conclusions and Future Work
References
Comparing Oracle and PostgreSQL, Performance and Optimization
1 Introduction
2 Related Work
3 Database Management System
4 Experimental Methodology
5 Experimental Setup
5.1 Hardware and Software Platforms
5.2 Database Workload
6 Results and Analysis
6.1 Creating Tables
6.2 Data Loading
6.3 Query Processing Time Without Optimizations
6.4 Query Processing Time with Primary and Foreign Keys
6.5 Query Processing Time with Indexes
7 Conclusions and Future Work
References
Systematic Review of Plant Pest and Disease Identification Strategies and Techniques in Mobile Apps
1 Introduction
2 Methodology
2.1 Scraping
2.2 Cleaning
2.3 Strategy Determination
3 Discussion
3.1 Most Used Strategies
3.2 Strategy Grouping by Organizations
4 Conclusion and Future Work
References
The Acceptance of Content Management Systems in Portuguese Municipalities: A Study in the Intermunicipal Community of Lezíria do Tejo
1 Introduction
2 Discussion
2.1 Information Management in Public Administration Organizations
2.2 Content Management in Public Organizations
3 Methodology
4 Analysis of Results
5 Conclusion
References
GOT: Generalization over Taxonomies, a Software Toolkit for Content Analysis with Taxonomies
1 Introduction
2 Taxonomic Content Analysis
3 Description and Structure of GOT Toolkit
3.1 Asts
3.2 Relevance_analysis
3.3 Taxonomies
4 Conclusion
References
Using Expert Crowdsourcing to Annotate Extreme Weather Events
1 Introduction
2 Background
3 eCSAAP Architecture for Crowd Annotation of Extreme Weather Events
3.1 eCSAAP Runtime
3.2 Extreme Weather Events Visualization and Annotation
4 Discussion
5 Final Remarks
References
Bridging the Gap Between Domain Ontologies for Predictive Maintenance with Machine Learning
1 Introduction
2 Contextualization
2.1 Ontologies for Production and Enterprise Resource Management
2.2 Ontologies for Predictive Maintenance
2.3 Ontologies for Machine Learning and Data Mining
2.4 Applicability to Current Scenario
3 Proposal
3.1 Temporal Representation and Reasoning
4 Conclusions
References
Parametric Study of the Analog Ensembles Algorithm with Clustering Methods for Hindcasting with Multistations
1 Introduction
2 Analog Ensemble Method and Variants
2.1 Analog Ensemble Method
2.2 Similarity Assessment
2.3 Analog Clustering
2.4 Prediction Method
2.5 Error Assessment
3 Computational Experiments
3.1 Meteorological Datasets
3.2 Variation of the Number of Clusters
3.3 Variation of the Number of Analogs
3.4 Variation of the Window Size
3.5 Variation of the Weight Membership
3.6 Results Comparison
3.7 Computational Enhancements
4 Conclusion
References
Business Intelligence Development Process: A Systematic Literature Review
1 Introduction
2 Objectives and Method
2.1 Reviews’ Objective
2.2 Research Method
3 Study Planning
3.1 Research Questions
3.2 Research Strategy
3.3 Publications Selection Process
3.4 Search Scope and Sources Selection
3.5 Inclusion and Exclusion Criteria
4 Study Realization
4.1 Data Collection
4.2 Related Works, Data Overview and Research Questions
5 Conclusion
References
A Meta-modeling Approach to Take into Account Data Domain Characteristics and Relationships in Information Visualizations
1 Introduction
2 Materials and Methods
2.1 Identification of Features
2.2 Meta-modeling
2.3 Automatic Generation
3 Meta-model Modification
3.1 Domain Characterization
3.2 Context Inclusion
4 Meta-model Instantiation Example
5 Discussion
6 Conclusions
References
A Knowledge Management Approach Supporting Model-Based Systems Engineering
1 Introduction
2 Related Work
3 A Knowledge Management Approach Supporting Model-Based Systems Engineering by Q&A Techniques
3.1 Overview
3.2 GOPPRR Approach Supporting MBSE
3.3 Knowledge Graph Modeling Based on GOPPRR Ontology
3.4 Workflow in the Developed Q&A Systems for MBSE Models
4 Cases Study
4.1 Problem Statement
4.2 Q&A Scenario for MBSE Models
4.3 Discussion
5 Conclusion and Future Work
References
A Deep Learning-Based Framework for the Classification of Non-functional Requirements
1 Introduction
2 Related Work
3 Proposed Data Augmentation Approach
4 Experimental Settings
4.1 Corpus for Training
4.2 Feature Learning
4.3 Classification
4.4 Hardware and Software Settings
5 Training and Analysis
5.1 Training and Analysis of Data Augmentation-Less Approach
5.2 Training and Analysis of Data Augmentation Approach
5.3 Training and Analysis of Data Augmentation and Pre-trained Word Embeddings
6 Comparative Analysis of Results
7 Conclusion and Future Work
References
ImageAI: Comparison Study on Different Custom Image Recognition Algorithms
1 Introduction
2 State of Art
3 Implementation
3.1 Dataset
3.2 Parameters
4 Results
5 Conclusion and Future Work
5.1 Future Work
References
Used of Web Scraping on Knowledge Representation Model for Bodies of Knowledge as a Tool to Development Curriculum
1 Introduction
2 Previous Research Works
3 Methodology
3.1 Ontological Methodology
3.2 Agile Methodology
3.3 Web Scrapping
4 Results
5 Conclusions
References
Author Index
Recommend Papers
- Author / Uploaded
- Alvaro Rocha
- Hojjat Adeli
- Gintautas Dzemyda
- Fernando Moreira
- Ana Maria Ramalho Correia
File loading please wait...
Citation preview
Advances in Intelligent Systems and Computing 1366
Álvaro Rocha · Hojjat Adeli · Gintautas Dzemyda · Fernando Moreira · Ana Maria Ramalho Correia Editors
Trends and Applications in Information Systems and Technologies Volume 2
Advances in Intelligent Systems and Computing Volume 1366
Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Advisory Editors Nikhil R. Pal, Indian Statistical Institute, Kolkata, India Rafael Bello Perez, Faculty of Mathematics, Physics and Computing, Universidad Central de Las Villas, Santa Clara, Cuba Emilio S. Corchado, University of Salamanca, Salamanca, Spain Hani Hagras, School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK László T. Kóczy, Department of Automation, Széchenyi István University, Gyor, Hungary Vladik Kreinovich, Department of Computer Science, University of Texas at El Paso, El Paso, TX, USA Chin-Teng Lin, Department of Electrical Engineering, National Chiao Tung University, Hsinchu, Taiwan Jie Lu, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW, Australia Patricia Melin, Graduate Program of Computer Science, Tijuana Institute of Technology, Tijuana, Mexico Nadia Nedjah, Department of Electronics Engineering, University of Rio de Janeiro, Rio de Janeiro, Brazil Ngoc Thanh Nguyen , Faculty of Computer Science and Management, Wrocław University of Technology, Wrocław, Poland Jun Wang, Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong
The series “Advances in Intelligent Systems and Computing” contains publications on theory, applications, and design methods of Intelligent Systems and Intelligent Computing. Virtually all disciplines such as engineering, natural sciences, computer and information science, ICT, economics, business, e-commerce, environment, healthcare, life science are covered. The list of topics spans all the areas of modern intelligent systems and computing such as: computational intelligence, soft computing including neural networks, fuzzy systems, evolutionary computing and the fusion of these paradigms, social intelligence, ambient intelligence, computational neuroscience, artificial life, virtual worlds and society, cognitive science and systems, Perception and Vision, DNA and immune based systems, self-organizing and adaptive systems, e-Learning and teaching, human-centered and human-centric computing, recommender systems, intelligent control, robotics and mechatronics including human-machine teaming, knowledge-based paradigms, learning paradigms, machine ethics, intelligent data analysis, knowledge management, intelligent agents, intelligent decision making and support, intelligent network security, trust management, interactive entertainment, Web intelligence and multimedia. The publications within “Advances in Intelligent Systems and Computing” are primarily proceedings of important conferences, symposia and congresses. They cover significant recent developments in the field, both of a foundational and applicable character. An important characteristic feature of the series is the short publication time and world-wide distribution. This permits a rapid and broad dissemination of research results. Indexed by DBLP, EI Compendex, INSPEC, WTI Frankfurt eG, zbMATH, Japanese Science and Technology Agency (JST), SCImago. All books published in the series are submitted for consideration in Web of Science.
More information about this series at http://www.springer.com/series/11156
Álvaro Rocha Hojjat Adeli Gintautas Dzemyda Fernando Moreira Ana Maria Ramalho Correia •
•
•
•
Editors
Trends and Applications in Information Systems and Technologies Volume 2
123
Editors Álvaro Rocha ISEG University of Lisbon Lisbon, Portugal Gintautas Dzemyda Institute of Data Science and Digital Technologies Vilnius University Vilnius, Lithuania
Hojjat Adeli College of Engineering The Ohio State University Columbus, OH, USA Fernando Moreira DCT Universidade Portucalense Porto, Portugal
Ana Maria Ramalho Correia Department of Information Sciences University of Sheffield Lisbon, Portugal
ISSN 2194-5357 ISSN 2194-5365 (electronic) Advances in Intelligent Systems and Computing ISBN 978-3-030-72650-8 ISBN 978-3-030-72651-5 (eBook) https://doi.org/10.1007/978-3-030-72651-5 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface
This book contains a selection of papers accepted for presentation and discussion at the 2021 World Conference on Information Systems and Technologies (WorldCIST’21). This conference had the scientific support of the University of Azores, Information and Technology Management Association (ITMA), IEEE Systems, Man, and Cybernetics Society (IEEE SMC), Iberian Association for Information Systems and Technologies (AISTI), and Global Institute for IT Management (GIIM). It took place online at Hangra do Heroismo city, Terceira Island, Azores, Portugal, March 30–31 to April 1–2, 2021. The World Conference on Information Systems and Technologies (WorldCIST) is a global forum for researchers and practitioners to present and discuss recent results and innovations, current trends, professional experiences, and challenges of modern information systems and technologies research, technological development, and applications. One of its main aims is to strengthen the drive toward a holistic symbiosis between academy, society, and industry. WorldCIST’21 built on the successes of WorldCIST’13 held at Olhão, Algarve, Portugal; WorldCIST’14 held at Funchal, Madeira, Portugal; WorldCIST’15 held at São Miguel, Azores, Portugal; WorldCIST’16 held at Recife, Pernambuco, Brazil; WorldCIST’17 held at Porto Santo, Madeira, Portugal; WorldCIST’18 held at Naples, Italy; WorldCIST’19 held at La Toja, Spain; and WorldCIST’20, which took place online at Budva, Montenegro. The Program Committee of WorldCIST’21 was composed of a multidisciplinary group of 309 experts and those who are intimately concerned with information systems and technologies. They have had the responsibility for evaluating, in a ‘blind review’ process, the papers received for each of the main themes proposed for the conference: A) information and knowledge management; B) organizational models and information systems; C) software and systems modeling; D) software systems, architectures, applications and tools; E) multimedia systems and applications; F) computer networks, mobility and pervasive systems; G) intelligent and decision support systems; H) big data analytics and Applications; I) human–computer interaction; J) ethics, computers and security; K) health informatics;
v
vi
Preface
L) information technologies in education; M) information technologies in radiocommunications; N) technologies for biomedical applications. The conference also included workshop sessions taking place in parallel with the conference ones. Workshop sessions covered themes such as healthcare information systems interoperability, security and efficiency; user expression and sentiment analysis; gamification application and technologies; code quality and security; amalgamating artificial intelligence and business innovation; innovation and digital transformation for rural development; automatic detection of fake news in social media; open learning and inclusive education through information and communication technology; digital technologies and teaching innovations in COVID-19 times; devops and software engineering; pervasive information systems; advancing eHealth through software engineering fundamentals; blockchain and distributed ledger technology (DLT) in business; innovation and intelligence in educational technology, evolutionary computing for health care; ICT for auditing and accounting; and leveraging customer behavior using advanced data analytics and machine learning techniques. WorldCIST’21 received about 400 contributions from 51 countries around the world. The papers accepted for oral presentation and discussion at the conference are published by Springer (this book) in four volumes and will be submitted for indexing by WoS, EI-Compendex, Scopus, DBLP, and/or Google Scholar, among others. Extended versions of selected best papers will be published in special or regular issues of relevant journals, mainly JCR/SCI/SSCI and Scopus/EI-Compendex indexed journals. We acknowledge all of those that contributed to the staging of WorldCIST’21 (authors, committees, workshop organizers, and sponsors). We deeply appreciate their involvement and support that was crucial for the success of WorldCIST’21. March 2021
Álvaro Rocha Hojjat Adeli Gintautas Dzemyda Fernando Moreira
Contents
Information and Knowledge Management GoVegan: Exploring Motives and Opinions from Tweets . . . . . . . . . . . . Phoey Lee Teh and Wei Li Yap
3
Five Hundred Most-Cited Papers in the Computer Sciences: Trends, Relationships and Common Factors . . . . . . . . . . . . . . . . . . . . . Phoey Lee Teh and Peter Heard
13
Reasons of Individuals to Trust What They Read on Social Network Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tom Sander and Phoey Lee Teh
23
Perspectives of Companies and Employees from the Great Place To Work (GPTW) Ranking on Remote Work in Portugal: A Methodological Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anabela Mesquita, Adriana Oliveira, Luciana Oliveira, Arminda Sequeira, and Paulino Silva
34
Overlaps Between Business Intelligence and Customer Relationship Management – Is There a Place for E-Commerce? . . . . . . . . . . . . . . . . Ionuţ-Daniel Anastasiei and Mircea-Radu Georgescu
41
Perception About of COVID-19 Information in Ecuador . . . . . . . . . . . . Abel Suing, Carlos Ortiz-León, and Juan Carlos Maldonado
56
Impact of ICT in Rural Areas: Perceptions of Portugal, Spain and Russia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . João Paulo Pereira
66
AMALGAM: A Matching Approach to Fairfy TabuLar Data with KnowledGe GrAph Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rabia Azzi and Gayo Diallo
76
vii
viii
Contents
Knowledge in Transition in an Industrial Company . . . . . . . . . . . . . . . Maria José Sousa, Miguel Sousa, and Álvaro Rocha Project Management Practice in SMEs: A Comparative Study of the Portuguese and Danish Economic Context . . . . . . . . . . . . . . . . . . Cristiano Seiji Watanabe, Anabela Pereira Tereso, Aldora Gabriela Gomes Fernandes, Lars Jespersen, and Jens Vestgaard
87
96
Critical Management Risks in Collaborative University-Industry R&D Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Gabriela Fernandes, Joana Domingues, Anabela Tereso, and Eduardo Pinto A Project Management Hybrid Model of Software Development in an Academic Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 Cláudia Dinis, Pedro Ribeiro, and Anabela Tereso Computerised Sentiment Analysis on Social Networks. Two Case Studies: FIFA World Cup 2018 and Cristiano Ronaldo Joining Juventus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Nuno Pombo, Miguel Rodrigues, Zdenka Babic, Magdalena Punceva, and Nuno Garcia Modelling Academic Dropout in Computer Engineering Using Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Diogo M. A. Camelo, João C. C. Santos, Maria P. G. Martins, and Paulo D. F. Gouveia Evolution of the Data Mining and Machine Learning Techniques Used in Health Care: A Scoping Review . . . . . . . . . . . . . . . . . . . . . . . . 151 Carmen Cecilia Sanchez Zuleta, Lillyana María Giraldo Marín, Juan Felipe Vélez Gómez, David Sanguino Cotte, César Augusto Vargas López, and Fabián Alberto Jaimes Barragán Learning Analytics Metrics into Online Course’s Critical Success Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Yosra Mourali, Maroi Agrebi, Ramzi Farhat, Houcine Ezzedine, and Mohamed Jemni The Influence of COVID-19 in Retail: A Systematic Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Marisa Lopes and João Reis Office Evolution from Ancient Age to Digital Age (E-working) . . . . . . . 182 Michal Beno
Contents
ix
Measuring Variability in Acute Myocardial Infarction Coding Using a Statistical Process Control and Probabilistic Temporal Data Quality Control Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 J. Souza, I. Caballero, J. V. Santos, M. F. Lobo, A. Pinto, J. Viana, C. Saez, and A. Freitas Business Decision Making Based on Social Media Analysis . . . . . . . . . . 203 C. Santiago Morales, M. Mario Morales, S. Glenda Toala, B. Alicia Andrade, and U. Giovanny Moncayo The Effects of Abrupt Changing Data in CART Inference Models . . . . 214 Miriam Esteve, Nuria Mollá-Campello, Jesús J. Rodríguez-Sala, and Alejandro Rabasa Log Data Preparation for Predicting Critical Errors Occurrences . . . . . 224 Myriam Lopez, Marie Beurton-Aimar, Gayo Diallo, and Sofian Maabout Running Workshops to Identify Wastes in a Product Development Sample Shop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 Gabriela R. Witeck, Anabela C. Alves, Joana P. de Almeida, Ana J. Santos, Ana L. Lima, and Ricardo J. Machado Evolutionary Dynamics in Azorean Landscapes: The Land-Use Changes in Forests and Semi-natural Areas in the Archipelago from 1990 to 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 Rui Alexandre Castanho, Gualter Couto, José Manuel Naranjo Gómez, Pedro Pimentel, Célia Carvalho, Áurea Sousa, Maria da Graça Batista, and Luís Loures A Comparative Study of Classifier Algorithms for Recommendation of Banking Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 Ivan F. Jaramillo, Ricardo Villarroel-Molina, Bolivar Roberto Pico, and Andrés Redchuk New Capabilities of the Geometric Multidimensional Scaling . . . . . . . . . 264 Gintautas Dzemyda and Martynas Sabaliauskas Hope Amid a Pandemic: Is Psychological Distress Alleviating in South America While Coronavirus Is Still on Surge? . . . . . . . . . . . . 274 Josimar E. Chire-Saire, Khalid Mahmood, Jimy Oblitas-Cruz, and Tanvir Ahmed An Entropic Approach to Assess People’s Awareness of the Health Risks Posed by Pesticides in Oenotourism Events . . . . . . . . . . . . . . . . . 284 Ana Crespo, Rui Lima, M. Rosário Martins, Jorge Ribeiro, José Neves, and Henrique Vicente
x
Contents
Knowledge Management Strategies Through Educational Digital Platforms During Periods of Social Confinement . . . . . . . . . . . . . . . . . . 297 Romel Ramón González-Díaz, Ángel Eduardo Acevedo-Duque, Katia Ninozca Flores-Ledesma, Katiusca Cruz-Ayala, and Santos Lucio Guanilo Gomez Information Foraging on Social Media Using Elephant Herding Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 Yassine Drias, Habiba Drias, Ilyes Khennak, Lydia Bouchlaghem, and Sihem Chermat Social Vulnerability Segmentation Methodology Based on Key Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Nuria Mollá-Campello, Kristina Polotskaya, Esther Sobrino, Teresa Navarro, and Alejandro Rabasa A Requirements Catalog of Mobile Geographic Information System for Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 Badr El Fhel, Lamyae Sardi, and Ali Idri Keeping the Beat on: A Case Study of Spotify . . . . . . . . . . . . . . . . . . . . 337 Inês Gomes, Inês Pereira, Inês Soares, Mariana Antunes, and Manuel Au-Yong-Oliveira Democratic Talent Management in an Industry 4.0 Environment of Digital Revolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Alberto Rendo, Manuel Au-Yong-Oliveira, and Ana Dias Stimulating Components for Business Development in Latin American SMEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 Romel Ramón González-Díaz and Luis Armando Becerra-Perez Handling Industrial Consumer Rights by Using Blockchain . . . . . . . . . 375 M. A. El-dosuky and Gamal H. Eladl How Health Data Are Managed in Mozambique . . . . . . . . . . . . . . . . . . 385 Lotina Burine, Daniel Polónia, and Adriana Gradim Multi-perspective Conformance Checking Applied to BPMN-E2 . . . . . . 394 Rui Calheno, Paulo Carvalho, Solange Rito Lima, Pedro Rangel Henriques, and Mateo Ramos-Merino Event-Driven Ontology Population - from Research to Practice in Critical Infrastructure Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 David Graf, Wieland Schwinger, Werner Retschitzegger, Elisabeth Kapsammer, and Norbert Baumgartner Semantic-Based Image Retrieval Using Balanced Clustering Tree . . . . . 416 Nguyen Thi Uyen Nhi, Thanh The Van, and Thanh Manh Le
Contents
xi
NoSQL Comparative Performance Study . . . . . . . . . . . . . . . . . . . . . . . . 428 Pedro Martins, Paulo Tomé, Cristina Wanzeller, Filipe Sá, and Maryam Abbasi A Model for Designing SMES’ Digital Transformation Roadmap . . . . . 439 Luís Cunha and Cristóvão Sousa Boosting E-Auditing Process Through E-Files Semantic Enrichment . . . 449 Cristóvão Sousa, Mariana Carvalho, and Carla Pereira Integration of UML Diagrams from the Perspective of Enterprise Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459 Luís Cavique, Mariana Cavique, and Armando B. Mendes MongoDB, Couchbase, and CouchDB: A Comparison . . . . . . . . . . . . . . 469 Pedro Martins, Francisco Morgado, Cristina Wanzeller, Filipe Sá, and Maryam Abbasi Comparing Oracle and PostgreSQL, Performance and Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 Pedro Martins, Paulo Tomé, Cristina Wanzeller, Filipe Sá, and Maryam Abbasi Systematic Review of Plant Pest and Disease Identification Strategies and Techniques in Mobile Apps . . . . . . . . . . . . . . . . . . . . . . . 491 Blessing K. Sibanda, Gloria E. Iyawa, and Attlee M. Gamundani The Acceptance of Content Management Systems in Portuguese Municipalities: A Study in the Intermunicipal Community of Lezíria do Tejo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503 Daniela Louraço and Célio Gonçalo Marques GOT: Generalization over Taxonomies, a Software Toolkit for Content Analysis with Taxonomies . . . . . . . . . . . . . . . . . . . . . . . . . . 513 Dmitry Frolov and Boris Mirkin Using Expert Crowdsourcing to Annotate Extreme Weather Events . . . 522 Dennis Paulino, António Correia, João Barroso, Margarida Liberato, and Hugo Paredes Bridging the Gap Between Domain Ontologies for Predictive Maintenance with Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 533 Alda Canito, Juan Corchado, and Goreti Marreiros Parametric Study of the Analog Ensembles Algorithm with Clustering Methods for Hindcasting with Multistations . . . . . . . . . . . . . . . . . . . . . . 544 Leonardo Araújo, Carlos Balsa, C. Veiga Rodrigues, and José Rufino Business Intelligence Development Process: A Systematic Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560 Leonardo Pontes and Adriano Albuquerque
xii
Contents
A Meta-modeling Approach to Take into Account Data Domain Characteristics and Relationships in Information Visualizations . . . . . . 570 Andrea Vázquez-Ingelmo, Alicia García-Holgado, Francisco J. García-Peñalvo, and Roberto Therón A Knowledge Management Approach Supporting Model-Based Systems Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581 Pengfei Yang, Jinzhi Lu, Lei Feng, Shouxuan Wu, Guoxin Wang, and Dimitris Kiritsis A Deep Learning-Based Framework for the Classification of Non-functional Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591 Maliha Sabir, Ebad Banissi, and Mike Child ImageAI: Comparison Study on Different Custom Image Recognition Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602 Manuel Martins, David Mota, Francisco Morgado, Cristina Wanzeller, Pedro Martins, and Maryam Abbasi Used of Web Scraping on Knowledge Representation Model for Bodies of Knowledge as a Tool to Development Curriculum . . . . . . 611 Pablo Alejandro Quezada-Sarmiento, Jon A. Elorriaga, Ana Arruarte, and Luis Alberto Jumbo-Flores Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
Information and Knowledge Management
GoVegan: Exploring Motives and Opinions from Tweets Phoey Lee Teh1(B)
and Wei Li Yap2
1 Department of Computing and Information Systems, Subang Jaya, Malaysia
[email protected] 2 School of Science and Technology, Subang Jaya, Malaysia
Abstract. This report is suggesting the beneficial effect of clustering micro bloggers tweets from 60 hash tags relating to the issue of Veganism. Going Vegan is a wellknown effect on health. We aimed to analyze tweets coming from casual Twitter users and twitter accounts representing the veganism society and industry. We cluster the group of discourse that coming from 60 and more hashtags. These tags include tweets that have tagged with #plantbaseddiet, #vegan food, #vegetarian, etc. We collected n = 50,634 tweets and analyzed n = 25,639 processed tweets. The result shows that sampled tweets, which includes 1) concerns about animal welfare; 2) sustainability (environment) 3) ways to live a healthier lifestyle (Health), and 4) methods and options for Vegan (recipe). Although with 60 + hash tags, this grouping practice allows decision making processes more manageable. This work not only demonstrates the application of a clustering algorithm to collate micro blogs with different hash tags into groups of similar topics but also shown that it is possible to develop a platform for automatically assembling information on the same subject from a range of different micro blogs. The application can significantly assist others, including academic researchers, or businesses, to quickly and effectively find information and knowledge from these sources. This application is possible for society looking for a healthy life. Keywords: Knowledge management · Clustering · Text analysis · Opinion mining · Tweets
1 Introduction Vegan, Pronounced as “Veegan”, was coined in 1944 by a carpenter Donald Watson from a suggestion by early members Mr George A [1]. Henderson and his wife mark the beginning and end of vegetarianism [2]. Strict vegetarian and vegan households around the world are tabulated to acknowledge the importance of veganism diet for global environmental change [2]. There are more than 2.5% of U.S. adults that have adopted a strict vegan lifestyle in the year 2018. Within six months (January to June) in 2019, a total of 250,000 people in the U.K. signed up the Campaign to go Vegan [3]. With the number of people growing, knowledge-sharing hence become critical. Through social media, society can quickly share strategies and ways to support a vegan lifestyle. For instance, Twitter can be a place for connection and information sharing. It allows © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 3–12, 2021. https://doi.org/10.1007/978-3-030-72651-5_1
4
P. L. Teh and W. L. Yap
plenty of information to share and connecting subscribers. Taken these advantages, politicians, agencies, and environmental activist has, in this even, also use this channel to disseminate information online. Studies have revealed that twitter opinion leadership makes a significant contribution to individuals involvement, often, contribute to the area of the political process and decision making happen on social media [4]. In many ways, social media has proven useful on various kinds of matters [5–7], in others words, to gather opinions on what public talk about going Vegan, tweets are the media selected for the purpose. However, Tweets have a limit of 144 characters per post. Multiple hash tags are user generated to indicate the topic or topics of a tweet. For example, about vegan tweets, there is a total of 60 + hash tags has been used [8]. These number of hash tags keeps growing every day. It has created much effort to the user who just wishes to source for one piece of information. This work demonstrates that using clustering can sort data tidily. This work also argues that knowledge from tweets is useful if presented in a more organized manner.
2 Literature Review Social big data are a valuable source for value creation [9, 10]. Research related to Technology–Organization–Environment and Diffusion of Innovations are the most popular theoretical models for big data [9]. That said, the social network (such as Twitter) is currently the primary means of information sources or communication around the world [11, 12]. Clustering the opinion from this big set of data to explore insights about Veganism is a value-creation technique and open to a new opportunity. Value creation is helpful in decision making, such as looking for patterns or trending practice [13]. Let get back to the topic, about Veganism. Is the only value being all about health? Vegan that concern of animal welfare would relate meat with disgust and emotional anguish [14]. Their moral concern leads them to believe that all animal has a right to life and freedom [15]. A business that has a similar arm of code would tag along to influence society as a motive. Some vegan goes on social media with the motive to seek diet opinion. For instance, there are four types of vegan diet [16], 1) Lacto-vegetarian diet, 2) Lacto-Ovo vegetarian diets, 3) Pesco-vegetarian diets and 4) semi-vegetarian diets. The complexity, such as vegetarians consume animal-derived products while vegans strictly refrain from consuming animal-derived products [17]. Or some vegans only consume raw vegetables, nuts, fruits, seeds, sprouted grains and legumes [14]. Further, LactoVegetarian consumes dairy and milk products, but not egg, red meat and fish [14, 18]. Lacto-Ovo Vegetarian consume dairy products and eggs but no fish or meat (e.g., red meat and poultry) [14, 16]. Pesco-vegetarian consumes fish and dairy products, but no red meat or poultry [19–21]. Semi-vegetarian (flexitarian) consume meat occasionally [22]. With all those different types of diets, information seeker that intended to ask for a single piece of knowledge, such as on special diets plan or opinion from tweets could become complicated. Studies have associated the motives to go Vegan for several reasons. For instance, animal welfare [23, 24]; health-conscious [16, 20, 25, 26]; to sustain the planet [27– 31]; to obtain a sense of belonging in social media or gaining social identity [32, 33];
GoVegan: Exploring Motives and Opinions from Tweets
5
or to endorse celebrities/society [14, 34, 35]. Are these opinions discussed on Twitter? The primary objective of this study is to explore the motives and opinions of Veganism in tweets. And the secondary aim is to demonstrate via clustering the opinions, the complexity to get a piece of single information is reduced, despite many hashtags.
3 Methods 3.1 Data Collection A total of 50634 veganism related tweets posted between April 2019 to July 2019 are collected. Sixty-one vegan-related hash tags tweets include #vegan, #veganism, #plantbaseddiet, #plantbased, #govegan, etc. are collected. During the pre-processing, text is cleaned and filtered. Frequency of text was quantified. We listed words such as food (6797), animal (5352), vegetarian (4321), eat (4528) and vegan food (3330) as top 5 after the list of hashtags. We identified the parent node. We placed terms with synonym with parent name, for instance, terms such as’delicious’ as a parent to replace terms such as ‘delicious’, ‘yummyy’, ‘delicious’, ‘yuumy’. These are all done using SAS Enterprise Miner. 3.2 Clustering We used Expectation-Maximization (E.M.) algorithm to group tweets into a disjoint cluster of documents and to provide the descriptive terms for each theme. This algorithm assumes that a mixture model approximates the data distribution by fitting k cluster density functions fh (h = 1, …, k), to a data set with d variables [36]. First, it finds initial parameter estimates. Then, it applies the standard or scaled version of the E.M. algorithm to identify primary clusters and to update parameter estimates. The algorithm stops when two consecutive log-likelihood values differ by a specific amount or when reach five iterations. This step aims to explore the connecting terms, to cluster into a central theme. The reason to select this algorithm is that this approach is different from clustering because clustering allocates each document to a single group while Text Topic node assigns a score for each material and term to each topic. Thresholds is used to identify if the association is strong enough to consider that the term belongs to the subject. Therefore, documents (tweets) may belong to more than one subject or none. 3.3 Interrater Reliability We carried out Human Interrater Reliability test to find the percentage of agreement of association topics clustered. Human Coders is used in making value judgements. Several researchers have performed a similar analysis in their studies. For instance, to evaluate a performance of a system comparing the result with human coders [37]; to assess the irony value in Verbal Irony Procedure (VIP) [38]; to assess the degree of agreement on the hotel reviews from Trip Advisor [39]; and also to rate the sophisticated sarcasm term that was processed by sarcasm detection tools [40]. We have recruited three strict vegans with at least three years of experience to perform read across the text to verify
6
P. L. Teh and W. L. Yap
the reliability of the machine clustered set of discourse. This for a reason to ensure that the coder is conversant or experienced with the phenomenon of Veganism. Based on the 90% of confidence level and margin error of 10, a total number of 265 tweets were sampled from the results to obtain the percentage of agreement between the association of Veganism discussed relevant [41]. Coders with knowledge of Veganism can increase the accuracy of the ratings. We invited three coders named as T, W, and Y with the condition that they have been committed to a strict vegan lifestyle for at least six years and 7 years. Coders T and W stated that the reason to become a vegan spark after the watch of vegan documentaries. Coder Y noted the main reason was a health concern. A a total of 265 tweets are given to each individual to ensure not influenced by one another. They have rated Score = 1 for Yes or score = 0 for No to indicate the agreement of tweet association with the vegan topic, respectively according to human judgement. Cohen’s kappa coefficient (K) is used to measure inter-rater reliability [44]. The percent agreement is summed up and divides by the total number of observations [44]; we repeat this procedure for all the different I.D. of a topic.
4 Results Concept links assist in understanding the association between the terms based on the cooccurrence of terms in the document. The thickness of the line shows the strength of the association between the terms. A thicker link between 2 terms demonstrates a stronger association than a thinner link. Each associated term can be expanded to observe its sub-linked terms to have a better idea of the association between the two significant terms.
Fig. 1. An example of concept link for the term “animal.”
Figure 1 shows an example of the term animal that appears 4989 times. Other terms that are highly related to this term are compassion, “stop animal cruelty”, and cruel, indicating that vegans in twitter think that being vegan and can stop animal cruelty by not consuming or using any animal products. The term such as “compassion” is strongly associated with the term “animal” which indicates that a vegan diet increases compassion
GoVegan: Exploring Motives and Opinions from Tweets
7
for the animal. A total of 150 tweets contain both terms “stop animal cruelty” and “animal”, and 754 tweets includes the term “stop animal cruelty”. SAS enterprise miner allows all the terms to be explored individually. To simplify the results presentation, Table 1 presented the four sets of highly distributed clusters. Table 1. Results from clustering and human coders Cluster-ID
Terms
Percentage
Description
Agreement of human coders (%)
1
Animal, meat, consume, animal welfare, love, dairy, cruelty-free, stop animal cruelty, go cruelty-free, eat plant-based, ditch dairy, milk, product
19.3%
Related to compassion for animals, ditching dairy and adhering plant-based diet can protect animal rights and stop animal cruelty
87%
2
People, time, knowledge, life, today, want, year, live, first, look, world, thing, help, week
22.8%
Related to the lifestyle or duration/period/time of adopting a vegan diet
89%
3
Vegan food, 32.3% vegetarian, delicious, gluten-free, healthy, vegans of instagram, organic, vegetable, what vegans eat, vegan life, vegan food share, healthy food, salad, foodie
Related to what they eat and healthy nutritious discussion
83%
4
Good, food, day, consume, recipe, diet, plant, burger, vegetarian, health, delicious, healthy, free, base, plant-base
Related to vegan 87% recipes and option. The users mentioned that vegan food is deliciously healthy
25.5%
We gave each of the clusters an I.D., we explained in the following paragraph with some sample of exhibits of tweets that we coded to the specific class. Cluster-ID 1, “Animal Welfare” is assigned for this discovery. The discussion is about going Veganism with the opinion that animals have a right to life and freedom [43]. A total of 87% agreement is reached on this association to this cluster.
8
P. L. Teh and W. L. Yap
Sample exhibit of tweets selected from Cluster-ID 1: • 300,000 animals are killed for food every minute! • Do you think that animals don’t feel pain? • Some of the horrible methods that are used to torture animals, so that people can eat them! Please Go Vegan Cluster-ID 2, “Environment” is assigned for this discovery. The discussion mainly about stops consuming animal products. Adhere to the plant-based diet, which can reduce the devastating impact of the dairy industry on the environment. Livestock rearing results in global warming through the methane gas the animals produce [44]. There are 89% of the agreement reached on the association between Veganism by human raters. Sample exhibit of tweetsselected from Cluster-ID 2: • We all love animals. Why do we call some ‘pets’ and others’ dinner? • I made the most yumtastic maple and sriracha roasted chickpeas omg soooooo delicious. Great for snacking, salads or rice bowls. Cluster-ID 3, “Health” is assigned for this discovery. The keywords indicate that there is a discussion about following a vegan diet can improve body wellness, aid weight loss and diabetes. A total of 83% of the agreement is reached on the association with health. Sample exhibit tweets selected from Cluster-ID 3 • The health benefits of a vegan diet are life-changing. And to know the food on your plate has not killed innocent animals is satisfying. Coming this April inspired by the power of the plate and heart. • Eating A Good Is Great For Your Health I’m Just Going All Out Before I Go Fully Vegan Cluster-ID 4, “Recipe” is assigned for this discovery. A total of 87% of the text were agreed with association adhering Veganism with a recipe or other vegan option. Association between Veganism and vegan option is high because vegan options are ubiquitous nowadays [45]. There was very little vegan food to choose during the 1990s [45]. However, things have changed over the past 20 years due to the increase of people who adopt plant-based diets; restaurants have been evolved to satisfy customers’ demands and appetites. Some tweets also mentioned that “Every takeaway place now does some vegan food … every pub has a vegan menu now … every main street has a restaurant where you can be guaranteed to eat vegan food’. In 2018, restaurants and fast-food chains that added vegan options to their menus were Shake Shack, McDonald’s, Ikea, Impossible Burger, A&W and so-on [46]. Sample exhibit of tweets selected from Cluster-ID 4: • Delicious Vegan Recipes That Are Affordable • Had an Impossible Burger today! Soooo good Can’t wait until this comes to Canada.
GoVegan: Exploring Motives and Opinions from Tweets
9
Finally, in this study, there is no I.D. clustered that have any relation to endorsing the celebrities/society as the motive to become vegan. Besides that, assembling this topic not only informing us about the motives of becoming vegan but also exploring the opinions of what people talk about Vegan in tweets, in this case for that specific period in our data collection.
5 Conclusion This article aims to demonstrate the concept of ability to sort information from tweets in a better-organized manner. It can display organized information aggregated from different postings around the world using the relevant hashtag. A further application for it can be various. For instance, tweets concerning animal welfare with photos and videos link can be themed and organized into a more structured and up-to-date education online e-book. Cruelty-free products companies can extend their service to introduce non-animal-based products in the same platform online. A data scientist can refer to this platform to identify research opportunity, such as if nonanimal products may contain harmful ingredients. Brands like The Body Shop, Lush, Aveda, Urban Decay and others may use this platform as the channel to advertise and share their products information. Non-profit organizations such as AHSM (Association for Healthcare Social Media), can come in to promote Veganism for health. World Wildlife Fund (WWF) can seek supports and solutions to protect forests and secure water and to discuss environmental awareness [47] through becoming a vegan. A restaurant such as Impossible Burger, could offer more vegan options and announce in this platform. All with the same goals, to drive the whole nation about Veganism in one platform. United Nation of Global Sustainable Development Goal (About the Sustainable Development Goals United Nations Sustainable Development, n.d.), can use this as an indicator to connect to their activities and grant opportunity. Government to share policy and guideline into this platform. In terms of practical implication, the influencer has many advantages to business and services in promoting their products and services. The implementation of this clustered I.D. can provide products influencers and idea to share opinions using similar I.D., this can shorten the information acquisition steps of opinion seekers. For this case, brands such as The Body Shop, Lush, Aveda, Urban Decay and others may use this information as the channel to communicate their products and services. The limitation of this study is that only one technique of clustering is being used to present the overall opinions. Future research can use others type of clustering technique and provide a new set of representation, another limitation of our study is that the number of tweets is only collected from this specific period of time, if the technique can be continuous using machine language to find any difference of the result from this set of I.D., it would be beneficial. Finally, similarly used of this concept could be replicated on other issues or another movement, such as tweets that are discussing COVID-19, treatments, methods to avoid and adhere, products to use and policy and news. Real-time posting of tweets into one category, and reorganized into an individual perspective using artificial intelligence, provide a proper referencing platform to drive or tackle a particular movement.
10
P. L. Teh and W. L. Yap
References 1. Suddath, C.: A brief history of veganism. Time Magazin (2008) 2. Leahy, E., Lyons, S., Tol, R.S.J.: An Estimate of the Number of Vegetarians in the World. An Estimate of the Number of Vegetarians in the World, vol. 340, pp. 1–44 (2010) 3. Vegan Variety: Plant-based Foods Trend. https://www.preparedfoods.com/articles/119792vegan-variety-plant-based-foods-trend. Accessed 08 May 2018 4. Shane-Simpson, C., Manago, A., Gaggi, N., Gillespie-Lynch, K.: Why do college students prefer Facebook, Twitter, or Instagram? Site affordances, tensions between privacy and selfexpression, and implications for social capital. Comput. Hum. Behav. 86, 276–288 (2018) 5. Kim, J., Brossard, D., Scheufele, D.A., Xenos, M.: Shared information in the age of big data: exploring sentiment expression related to nuclear energy on twitter. J. Mass Commun. Q. 93(2), 430–445 (2016) 6. Sander, T., Teh, P.L., Sloka, B.: Your social network profile reveals you. Int. J. Web Inf. Syst. 13(1), 14–24 (2017) 7. Teh, P.L., Huah, L.P., Si, Y.W.: The intention to share and re-shared among the young adults towards a posting at social networking sites. Adv. Intell. Syst. Comput. 275, 13–21 (2014) 8. Hashtags for #vegans on Instagram, Twitter, Facebook, Tumblr| best-hashtags.com. http:// best-hashtags.com/hashtag/vegans. Accessed 01 April 2020 9. Baig, M.I., Shuib, L., Yadegaridehkordi, E.: Big data adoption: state of the art and research challenges. Inform. Process. Manag. 56(6), 102095 (2019) 10. Del Vecchio, P., Mele, G., Ndou, V., Secundo, G.: Creating value from social big data: implications for smart tourism destinations. Inf. Process. Manage. 54(5), 847–860 (2018) 11. Ficamos, P., Liu, Y.: A topic based approach for sentiment analysis on twitter data. Int. J. Adv. Comput. Sci. Appl. 7(12), 201–205 (2016) 12. Priya, S., Sequeira, R., Chandra, J., Dandapat, S.K.: Where should one get news updates: Twitter or Reddit. Online Soc. Netw. Media 9, 17–29 (2019) 13. Saggi, M.K., Jain, S.A.: Survey towards an integration of big data analytics to big insights for value-creation. Inf. Process. Manage. 54(5), 758–790 (2018) 14. Phua, J., Jin, S.V., Kim, J.: The roles of celebrity endorsers’ and consumers’ vegan identity in marketing communication about Veganism. Journal of Marketing Communications, pp. 1–23 (2019) 15. Compassion for animals, The Vegan Society: Between the Species. https://www.vegansoci ety.com/sites/default/files/CompassionForAnimals.pdf. Accessed 10 April 2019 16. Tonstad, S., Butler, T., Yan, R., Fraser, G.E.: Type of vegetarian diet, body weight, and prevalence of type 2 diabetes. Diab. Care 32(5), 791–796 (2009) 17. Waters, J.: A model of the dynamics of household vegetarian and vegan rates in the U.K. Appetite 127, 364–372 (2018) 18. Praharaj, A.B., Goenka, R.K., Dixit, S., Gupta, M.K., Kar, S.K., Negi, S.: Lacto vegetarian diet and correlation of fasting blood sugar with lipids in population practicing sedentary lifestyle. Ecol. Food Nutr. 56(5), 351–363 (2017) 19. Clarys, P., Deliens, T., Huybrechts, I., Deriemaeker, P., Vanaelst, B., De Keyzer, W., Hebbelinck, M., Mullie, P.: Comparison of nutritional quality of the vegan, vegetarian, semi-vegetarian, pesco-vegetarian and omnivorous diet. Nutrients 6(3), 1318–1332 (2014) 20. Tonstad, S., Nathan, E., Oda, K., Fraser, G.E.: Prevalence of hyperthyroidism according to type of vegetarian diet. Publ. Health Nutr. 18(8), 1482–1487 (2015) 21. Mihrshahi, S., Ding, D., Gale, J., Allman-Farinelli, M., Banks, E., Bauman, A.E.: Vegetarian diet and all-cause mortality: evidence from a large population-based australian cohort - the 45 and up study. Prev. Med. 97, 1–7 (2017)
GoVegan: Exploring Motives and Opinions from Tweets
11
22. Forestell, C.A.: Flexitarian diet and weight control: healthy or risky eating behavior? Front. Nutr. 5, 1–6 (2018) 23. Radnitz, C., Beezhold, B., DiMatteo, J.: Investigation of lifestyle choices of individuals following a vegan diet for health and ethical reasons. Appetite 90, 31–36 (2015) 24. Edmundson, W.A.: Do animals need citizenship? Int. J. Const. Law 13(3), 749–765 (2015) 25. Appleby, P.N., Davey, G.K., Key, T.J.: Hypertension and blood pressure among meat eaters, fish eaters, vegetarians and vegans in EPIC–Oxford. Publ. Health Nutr. 5(05), 645–654 (2003) 26. Tantamango-Bartley, Y., Knutsen, S.F., Knutsen, R., Jacobsen, B.K., Fan, J., Lawrence Beeson, W., Sabate, J., Hadley, D., Jaceldo-Siegl, K., Penniecook, J., Herring, P., Butler, T., Bennett, H., Fraser, G.: Are strict vegetarians protected against prostate cancer? Am. J. Clin. Nutr. 103(1), 153–160 (2016) 27. Henning, B.: Standing in livestock’s “long shadow”: the ethics of eating meat on a small planet. Ethics Environ. 16(2), 63 (2011) 28. Doyle, J.: Celebrity vegans and the life styling of ethical consumption. Environ. Commun. 10(6), 777–790 (2016) 29. Rosi, A., Mena, P., Pellegrini, N., Turroni, S., Neviani, E., Ferrocino, I., Di Cagno, R., Ruini, L., Ciati, R., Angelino, D., Maddock, J., Gobbetti, M., Brighenti, F., Del Rio, D., Scazzina, F.: Environmental impact of omnivorous, ovo-lacto-vegetarian, and vegan diet. Sci. Rep. 7(1), 1–9 (2017) 30. Baroni, L., Filippin, D., Goggi, S.: Helping the planet with healthy eating habits. Open Inform. Sci. 2(1), 156–167 (2018) 31. Chai, B.C., van der Voort, J.R., Grofelnik, K., Eliasdottir, H.G., Klöss, I., Perez-Cueto, F.J.A.: Which diet has the least environmental impact on our planet? A systematic review of vegan, vegetarian and omnivorous diets. Sustain. (Switz.) 11(15), 4110 (2019) 32. Yang, C.-C., Holden, S.M., Carter, M.D.K.: Social media social comparison of ability (but not opinion) predicts lower identity clarity: identity processing style as a mediator. J. Youth Adolesc. 47(10), 2114–2128 (2018) 33. Davis, J.L., Love, T.P., Fares, P.: Collective social identity: synthesizing identity theory and social identity theory using digital data. Soc. Psychol. Q. 82(3), 254–273 (2019) 34. Bratu, S.: The phenomenon of image manipulation in advertising. Econ. Manag. Financ. Markets 5(2), 333–338 (2010) 35. Lundahl, O.: From a moral consumption ethos to an apolitical consumption trend: the role of media and celebrities in structuring the rise of Veganism. University of Vaasa (2017) 36. Getting Started with SAS® Text Miner 12.1. SAS Institute Inc (2008) 37. Lee, K.J., Choi, Y.S., Kim, J.E.: Building an automated English sentence evaluation system for students learning English as a second language. Comput. Speech Lang. 25(2), 246–260 (2011) 38. Burgers, C., van Mulken, M., Schellens, P.J.: Finding irony: an introduction of the verbal irony procedure (VIP). Metaphor Symbol 26(3), 186–205 (2011) 39. Khoo, F.S., Teh, P.L., Ooi, P.B.: Consistency of online consumer’s perceptions of posted comments: an analysis of trip advisor reviews. J. Inform. Commun. Technol. 2(2), 374–393 (2017) 40. Teh, P.L., Ooi, P.B., Chan, N.N., Chuah, Y.K.: A comparative study of the effectiveness of sentiment tools and human coding in sarcasm detection. J. Syst. Inf. Technol. 20(3), 358–374 (2018) 41. Sample Size Calculator. In SurveyMonkey. https://www.surveymonkey.com/mp/samplesizecalculator/ 42. McHugh, M.L.: Interrater reliability: the kappa statistic. Biochemia Med. 22(3), 276–282 (2012) 43. Compassion for animals, The Vegan Society (2018). https://www.vegansociety.com/sites/def ault/files/CompassionForAnimals.pdf. Accessed 08 April 2020
12
P. L. Teh and W. L. Yap
44. Plant-based diet can fight climate change - U.N. BBC News, August. https://www.bbc.com/ news/science-environment-49238749. Accessed 08 April 2020 45. Mawunyo, G.: Vegan options are everywhere nowadays as restaurants evolve to meat demand. ABC Radio Sydney. https://www.abc.net.au/news/2019-05-18/restaurants-takenote-as-morepeople-go-vegan/11119160. Accessed 08 April 2020 46. Sharon, V.: 15 Vegan Options Added to Restaurants and Fast Food Chains in 2018. ABC News. https://www.onegreenplanet.org/vegan-food/15-vegan-options-added-to-restau rantsand-fast-food-chains-in-2018/. Accessed 08 April 2020 47. Ganga, S.D.: Social media and the rise of Vegan. Experiencing Public Relations (2018)
Five Hundred Most-Cited Papers in the Computer Sciences: Trends, Relationships and Common Factors Phoey Lee Teh1(B)
and Peter Heard2
1 Department of Computing and Information Systems, School of Science and Technology,
Sunway University, 47500 Sunway City, Malaysia [email protected] 2 Provost Office, Sunway University, 47500 Sunway City, Malaysia [email protected]
Abstract. This study reveals common factors among highly cited papers in the computer sciences. The 500 most cited papers in the computer sciences published between January 2013 and December 2017 were downloaded from the Web of Science (WoS). Data on the number of citations, number of authors, article length and subject sub-discipline were extracted and analyzed in order to identify trends, relationships and common features. Correlations between common factors were analyzed. The 500 papers were cited a total of 10,926 times: the average number of citations per paper was 21.82 citations. A correlation was found between author credibility (defined in terms of the QS University Ranking of the first named author’s affiliation) and the number of citations. Authors from universities ranked 350 or higher were more cited than those from lower ranked universities. Relationships were also found between journal ranking and both the number of authors and the article length. Higher ranked journals tend to have a greater number of authors, but were of shorter length. The article length was also found to be correlated with the number of authors and the QS Subject Ranking of the first author’s affiliation. The proportion of articles in higher ranked journals (journal quartile), the length of articles and the number of citations per page were all found to correlate to the sub-discipline area (Information Systems; Software Engineering; Artificial Intelligence; Interdisciplinary Applications; and Theory and Methods). Keywords: Data search · Knowledge discovery · Citation · Trends · Common factors
1 Introduction The Institute for Scientific Information (ISI) [1] is a bibliographic database of academic journals with citation indexing and analysis, which allows researchers to find out how many times a given article has been cited and by whom. Although opinion is divided on the merits of the metrics derived from such databases, the proliferation of similar abstract and database services, such as Scopus, Google Scholar and the more focused PubMed, is testament to the growth in their importance. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 13–22, 2021. https://doi.org/10.1007/978-3-030-72651-5_2
14
P. L. Teh and P. Heard
The number of citations received by a given piece of scholarly work is an often-used proxy measure of the quality, importance and impact of the work: higher citation counts are assumed to be indicative of higher quality research, and greater impact. Citation measures, such as the average number of citations per paper, h-index, [2–4], i10-index [5] or g-index [6] may be used as part of academic recruitment, tenure and promotion exercises. At institutional-level citation metrics feed into the major league tables, such as the QS and Time Higher rankings. Citations per faculty (i.e. per member of academic staff) constitutes 20% to the total score in the QS world university rankings [7]; while in the THE world rankings citations per paper contributes 30% [8]. The number of times a piece of scholarly work is cited is thus of great importance to individuals, their academic department and their university. High citation rates are often used as an indicator of quality and impact, and may indicate to other researchers whether or not a particular article is worthy of reading [3] and citing, thus leading to further citations. Thelwall [9] noted that citation counts are used by researchers and research managers to assist in the evaluation of the quality or impact of published research, especially where it is impractical to employ peer judgements or where corroborating data is required. Several other methodologies have been proposed to measure research output. DortaGonzález et. al. [10] for example, proposed three dimensions, namely productivity (number of journal papers); impact (journal citations); and references (bibliographical sources). González-Betancor and Dorta-González [10] proposed an alternative citation impact indicator, based on the percentage of highly cited articles. The potential use of Google Scholar metrics as a feasible and reliable indicator of highly cited documents was examined by Martin-Martin [11], but it was found to be an unreliable method. Chang [12] conducted a study on high impact papers using ISI metrics for the 200 most highly cited journal in the sciences and social science. Results showed that the Sciences and Social Sciences are different in terms of the strength of the relationship of journal performance metrics, although the actual relationships were very similar.
2 Credibility, Article Length, Number of Authors and Field of Study Previous research has shown that citation rates vary with such parameters as author credibility, article length, number of authors and field of study. Author credibility refers to the credentials or other perceived qualities of the author. Perceived author credibility may be used as an indicator of whether or not their research is reliable, of high quality and thus a valuable source of reference. Measures of author credibility include the experience of the author, the ranking of the author’s primary affiliation, and/or the number and prestige of awards received. Plomp [13] found that authors with a greater number of previously published outputs were more likely to receive a greater number of citations for subsequent work. Rodríguez-Navarro [14] noted that Nobel Prize-winning authors enjoy higher citation rates, and Bitetti [15] noted how the more influential a researcher is in a certain field, the greater their citation count. Akre [16] found that research originating from high-income European countries tends to have higher citation rates. In the context of this research, the primary affiliation of the first-named author – both at institutional-level
Five Hundred Most-Cited Papers in the Computer Sciences
15
and at departmental/subject-level – is used as a proxy measure of author credibility. The first named author was used, because, in the computer sciences, the first named author is usually the individual most identified with the work. Previous research [17–20] has demonstrated a relationship between citation count and article length: articles of greater length attracted a high number of citations. The rationale for this observation is that longer papers have more content, which is of potential interest to others. Previous research [21] has also shown that papers with multiple authors are more highly cited than single author works. Gazni [22] showed that single-authored papers received, on average one citation, while multi-authored papers received an average of 2.12 citations per paper. Tahanam [19] also concluded that multiple authorship is a contributory factor to higher citation rates. Oakleaf [23] found that monodisciplinary papers received higher citation rates than multidisciplinary papers, and Akneses and Rip [21] concluded that the majority of citations come from other researchers in the same field. In contrast, [19] showed that research focused on a single field of study had lower citation rates. Other research [10] has shown that citation practices differ across scientific fields, and found no evidence to support the hypothesis that multidisciplinary papers were cited differently to single-discipline outputs. Thus, there remains no consistent view on the impact of multi- versus singledisciplinary work. Much research in the computer sciences is multidisciplinary and we were interested to explore any relationship between the field of study and citation rates. Building on the aforementioned research we postulate that academics with higher levels of peer esteem will, on average, be cited more often than other academics. We thus suggest that publications from researchers affiliated with more highly ranked universities will, on average, be more highly cited: likewise, of authors from highly ranked departments. We also, consider whether authors from higher ranked universities and departments will have proportionately more papers in higher ranked journals. Review articles offer a broader perspective on an area of research, summarising previous work and drawing out more general conclusions. Reviews are therefore a valuable resource to researchers, and we suggest that reviews will, on average, be more highly cited than primary papers. Since reviews collate and summarise multiple studies, we anticipated that reviews will be longer than primary publications. Compiling all the suggestions above, this study seeks to establish if: (1) there is a correlation between article length and citation rates; (2) citation rates correlate with the number of authors; or (3) citation rates vary by sub-discipline.
3 Method The 500 most cited papers in the computer sciences published over the 5 years period January 2013 to December 2017 were downloaded from the Web of Science on 14th October 2018. Parameters extracted directly from the ISI database or the QS rankings organisation included: year of publication; sub-discipline (as defined by Web of Science); number of citations; article length; number of authors; University Ranking (QS) and Subject Ranking (QS). Table 1 shows papers that published earlier generally had more citations. The raw citation data were thus normalised to allow for comparison. Data were normalised by
16
P. L. Teh and P. Heard Table 1. Average citations vs. number of years since publication Number of years since publication Average 5
30.6
4
23.3
3
16.2
2
15.8
1
17
determining the mean citation count for each year and then converting raw counts to fractions of the yearly average. The number of citations received was compared in a pair-wise fashion to see if any common features emerged. Papers were a mix of primary articles and reviews articles. Unless otherwise indicated, results given are for the analysis of both reviews and primary publications.
4 Result The 500 papers were cited a total of 10,926 times over the period: the average was 21.82 citations per paper; median 13; mode 8. The majority of the papers, (74%) received 20 or fewer citations; only 3% had more than 80 citations. Mean citation rates are relatively flat for the first three years post publication, only rising significantly after that. Table 2. Average normalised citation counts compared with QS University World Ranking QS University World Ranking (2019)
Average normalised citation count
QS University World Ranking (2019)
Average normalised citation count
1–50
1.086
451–500
0.615
51–100
1.232
501–550
0.594
101–150
0.982
551–600
0.704
151–200
1.032
601–650
0.693
201–250
1.036
651–700
0.558
251–300
1.412
701–750
1.291
301–350
1.121
751–800
0.842
351–400
0.644
801–1000
0.745
401–450
1.052
Unranked
0.999
Tables 2 and 3 show the relationships between ranking and citations. Data indicate small differences between citation rates for authors from higher ranked universities compared to lower ranked ones. Not all universities and departments (subjects) have a ranking: such universities/subjects are grouped under the heading “unranked”. The group of
Five Hundred Most-Cited Papers in the Computer Sciences
17
Table 3. Average normalised citations compared with QS subject ranking. QS world subject ranking (2019)
Average normalised citation count
QS world subject ranking (2019)
Average normalised citation count
1–50
1.342
301–350
0.637
51–100
0.807
351–400
1.486
101–150
0.731
401–450
0.921
151–200
1.207
451–500
0.720
201–250
1.019
Unranked
0.936
251–300
0.874
unranked institutions includes government and industry laboratories. Data reveal that the average citation rates for publications emanating from universities ranked 350 or higher is greater than and those for lower ranked universities. The difference is not large, but is statistically significant (P = 0.03): papers originating from higher ranked universities receive, on average, about 25% more citations than those from lower ranked universities. Examination of Table 3, reveals no statistically significant relationship between citation rates and subject ranking. Table 4. Percentage of publications by journal quartile Journal quartile
Q1
Q2
Q3
Q4 Q5
Percentage of outputs (%) 57.6 23.9 14.0 3.7 0.7
Table 4 shows the percentage of papers that are published in Q1, Q2, Q3, Q4 and unranked journals (designated Q5), respectively. Overall approximately 58% of the papers are in Q1 journals; 81.5% of papers are in Q1 or Q2 journals. As depicted in Fig. 1, our data indicate no relationship between ranking (university ranking or subject ranking) and the percentage of papers in higher quartile journals. Questions 3 and 4 are thus shown to be proven false. One possible rationale for such a finding might be the efficacy of the peer review process: peer reviews showing no bias towards authors from higher ranked universities or departments. The number of review articles in the sample was relatively small: 35 papers (7%). Nevertheless statistically significant differences in article length and citation rates were observed. The average length of review articles was found to be 1.5 times longer (p < 0.01), whilst the average number of citations was found to be 2.5 times greater (p < 0.01). For review papers, the number of citations also appears to be correlated with the number of authors: citations increase gradually with number of authors, up to 5 authors, then flattens (Slope = 1.06; Rˆ2 = 0.94). Since journal rankings are based on citations [13, 24], it is to be expected that journal rankings and average citations correlate. The mean normalized citation rates are:
18
P. L. Teh and P. Heard
Fig. 1. Proportion of Q1/Q2, and Q3/Q4 papers as a function of QS World University Ranking
Q1 (0.32) > Q2 (0.21): p < 0.001; Q2 > Q3 (0.16): p < 0.05. No significant differences exist in the rates of citation between Q3 and Q4 or unranked journals. Table 5. Journal quartile and the average number of pages from the top 500 papers Journal quartile Number of papers Average number of pages 1
264
15.898
2
132
18.152
3
77
20.896
4
22
23.545
Table 5 reveals that article length is inversely and linearly correlated with journal ranking: higher ranked journal articles are shorter (R2 = 0.998). The Web of Science categorises computer science outputs into five different subdisciplines: Information Systems; Software Engineering; Artificial Intelligence; Interdisciplinary Applications; and Theory and Methods. The highest proportion of papers in Q1/Q2 journals is found for the sub-discipline of Interdisciplinary Application; papers in this sub-discipline are also found to be shorter on average than those of other subdisciplines. In contrast, papers in the sub-discipline areas of Software, and Theory and Methods are much less likely to be in Q1/Q2 journals, and are on average longer: particularly for Theory and Methods papers. Determining whether pressure for space in journals drives shorter papers, or whether Interdisciplinary Applications papers are naturally shorter and at the same time considered closer to the cutting-edge and thence more publishable in Q1 and Q2 journals is an interesting, but open question. The proportion of papers in Q1/Q2, and Q3/Q4 journals with different numbers of authors is shown in Fig. 2. The proportion of Q1 and Q2 journals generally increases with the number of authors: this increase is statistically significant (R2 = 0.80; p < 0.05).
Five Hundred Most-Cited Papers in the Computer Sciences
19
Fig. 2. Proportion of papers in Q1/Q2 and Q3/Q4 journals versus the number of authors
It may simply be that doing high quality work requires input from a greater number of investigators.
Fig. 3. Average article length vs. QS World University Ranking
The overall percentage of single authored papers is 14%, which compares to a global average (across all disciplines) of about 11% [25]. In line with previous findings [26] the proportion of single authored papers varies by sub-discipline: Theory and Methods publications are most likely to single author (17%); Artificial Intelligence and Software papers are least likely to be single author works (6% and 5%, respectively). Data also show that authors from higher ranked universities or departments write marginally longer papers than authors from lower ranked universities and departments (Fig. 3). For example, articles emanating from top 100 ranked departments are on average just over 2.5 pages longer than articles emanating from departments outside of the top 100: average page length = 19.67 and 17.02, respectively (P = 0.03). The relationship between University Ranking and article length is less pronounced, and of marginal statistical significance (P = 0.055). We observed a statistically significant relationship between the sub-discipline area and both the average number of pages and the average (normalised) citation rates (Table 6). In particular, Theory and Methods papers were found to be much longer, because of
20
P. L. Teh and P. Heard
Table 6. Number of articles, average page counts and normalised citation counts by sub-discipline Subject sub-discipline
Number of articles
Average number of pages
Average normalised citation counts per paper
Artificial intelligent
96
16.32
0.57
Information system
113
16.96
1.03
Inter-disciplinary
203
14.49
1.05
Software engineering
42
19.95
0.81
Theory & methods
46
32.63
0.98
substantial space given over to mathematical argument. The mean number of citations per page was 0.71, compared to 0.57 (for AI) to 1.05 (interdisciplinary studies). The reason for the lower impact of AI journal articles is not obvious, but, interestingly, [4] found that there was a greater occurrence of papers being retracted in this sub-discipline over a similar timeframe (2013 to 2017). Whether there is any link between the occurrence of retractions and lower impact remains an open, but interesting question.
5 Conclusion This paper presents an analysis of the 500 most cited papers in the computer sciences over the five-year period 2013 to 2017. Seventy-four percent of papers received 20 or fewer citations, with only 3% receiving more than 80 citations; the average was 21.82 citations per paper. A correlation was found between citation rates and author credibility: authors from universities ranked 350 or higher were more cited than those from lower ranked universities. Relationships were also found between journal ranking and the number of authors, and the article length: higher ranked journals tend to have a greater number of authors, but are shorter in length. The article length was also found to be correlated with the number of authors and the QS Subject Ranking of the first author’s affiliation. The proportion of articles in higher ranked journals, the length of articles and the number of citations per page were all found to depend on the sub-discipline area: the greatest impact, measured in terms of citations per page was found to be in Interdisciplinary Applications, with the lowest in Artificial Intelligence, and Theory and Methods.
References 1. Garfield, E.: Citation analysis as a tool in journal evaluation. Serial Librarian 178(4060), 471–479 (1972) 2. Hirsch, J.E.: hα: an index to quantify an individual’s scientific leadership. Scientometrics 118, 673–686 (2019) 3. Cormode, G., Ma, Q., Muthukrishnan, S., Thompson, B.: Socializing the h-index. J. Informetrics 7(3), 718–721 (2013)
Five Hundred Most-Cited Papers in the Computer Sciences
21
4. Ayaz, S., Masood, N., Islam, M.A.: Predicting scientific impact based on h-index. Scientometrics 114(3), 993–1010 (2018) 5. Noruzi, A.: Impact factor, h-index, i10-index and i20-index of webology. Webology 13(1), 1–4 (2016) 6. De Visscher, A.: What does the g-index really measure? J. Am. Soc. Inf. Sci. Technol. 62(11), 2290–2293 (2011) 7. QS World University Rankings – Methodology|Top Universities. https://www.topuniversit ies.com/qs-world-university-rankings/methodology. Accessed 19 June 2019 8. World University Rankings 2019: methodology | Times Higher Education (THE). https:// www.timeshighereducation.com/world-university-rankings/methodology-world-universityrankings-2019. Accessed 19 June 2019 9. Thelwall, M.: Dimensions: a competitor to Scopus and the web of science? J. Informetrics 12(2), 430–435 (2018) 10. Dorta-González, P., Dorta-González, M.I., Suárez-Vega, R.: An approach to the author citation potential: measures of scientific performance which are invariant across scientific fields. Scientometrics 102(2), 1467–1496 (2014) 11. Martin-Martin, A., Orduna-Malea, E., Harzing, A., Lopez-Cozar, E.D.: Can we use Google Scholar to identify highly-cited documents? J. Informetrics 11(1), 152–163 (2017) 12. Chang, C.L., McAleer, M., Oxley, L.: Coercive journal self citations, impact factor, journal influence and article influence, mathematics and computers in simulation. Int. Assoc. Math. Comput. Simul. (IMACS) 93, 190–197 (2013) 13. Plomp, R.: The highly cited papers of professors as and indicator of a research group’s scientific performance. Scientometrics 29(3), 377–393 (1994) 14. Rodríguez-Navarro, A.: A simple index for the high-citation tail of citation distribution to quantify research performance in countries and institutions. PLoS ONE 6(5), e20510 (2011) 15. Bitetti, M.S.D., Ferreras, J.A.: Publish (in English) or perish: the effect on citation rate of using languages other than English in scientific publications. Ambio 46(1), 121–127 (2017) 16. Akre, O., Barone-Adesi, F., Pattersson, A., Pearce, N., Merletti, F., Richiardi, L.: Differences in citation rates by country of origin for papers published in top-ranked medical journals: do they reflect inequalities in access to publication? J. Epidemiol. Community Health 65(2), 119–123 (2011) 17. Hamrick, T.A., Fricker, R.D., Brown, G.G.: Assessing what distinguishes highly cited from less-cited papers published in interfaces. Interfaces 40(6), 454–464 (2010) 18. Coupé, T.: Peer review versus citations - an analysis of best paper prizes. Res. Policy 42(1), 295–301 (2013) 19. Tahamtan, I., Safipour Afshar, A., Ahamdzadeh, K.: Factors affecting number of citations: a comprehensive review of the literature. Scientometrics 107(3), 1195–1225 (2016). https:// doi.org/10.1007/s11192-016-1889-2 20. Fox, C.W., Paine, C.E.T., Sauterey, B.: Citations increase with manuscript length, author number, and references cited in ecology journals. Ecol. Evol. 6(21), 7717–7726 (2016). https:// doi.org/10.1002/ece3.2505 21. Aksnes, D.W., Rip, A.: Researchers’ perceptions of citations’. Res. Policy 38(6), 895–905 (2009) 22. Gazni, A., Didegah, F.: Investigating different types of research collaboration and citation impact: a case study of Harvard University’s publications. Scientometrics 87(2), 251–265 (2011) 23. Oakleaf, M.: Writing information literacy assessment plans: a guide to best practice. Commun. Inf. Literacy 3(2), 80–90 (2009) 24. Petersen, C.G., Aase, G.R., Heiser, D.R.: Journal ranking analyses of operations management research. Int. J. Oper. Prod. Manag. 31(4), 405–422 (2011)
22
P. L. Teh and P. Heard
25. Baker, S.: Authorship: are the days of the lone research ranger limited? Times Higher Education. https://www.timeshighereducation.com/news/authorship-are-days-lone-researchranger-numbered. Accessed 03 July 2019 26. Al-Hidabi, M.D., The, P.L.: Multiple publications: the main reason for the retraction of papers in computer science. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) Advances in Information and Communication Networks. FICC 2018. Advances in Intelligent Systems and Computing, vol. 886, pp. 551–526. Springer, Cham (2019)
Reasons of Individuals to Trust What They Read on Social Network Sites Tom Sander1(B) and Phoey Lee Teh2 1 Mannheim, Germany 2 Sunway University, Subang Jaya, Malaysia
[email protected]
Abstract. The rationality of believing a piece of information depends on the level of trust of each individual. To make a purchase decision, an individual has to trust the information they have collected from social media content. That said, to influence an individual’s decision, companies have to obtain an individual’s trust successfully. This study aims to investigate what makes an individual trust what they read on social media (SM). Participants of this study are from Germany and United Kingdom (UK). A model is created and tested with regression analysis. That helps to identify the impact experience with social network sites (SNSs) on trust of information in SNSs. The practical outcome is to provide advice to companies whereby to communicate with potential clients to transfer their information successfully and trustworthy. Keywords: Trust · Social media · Content · Signaling theory · Digital networks
1 Introduction Communication is an essential part of our daily life. Individuals communicate to convey knowledge and experience they gained from the information they read. Consider this, to successfully influence consumer’s purchase decision; communication has to be in place beforehand. The objective is to send the most suitable information, with the most significant impact to the right target group. So that individuals able to apply for a role or decide to buy a product [1]. SM offers many opportunities to transfer information economically, with a small budget and low efforts compared with other channels. However, are those information posted appreciated? The difficulty is to ensure people able to trust SM information, which may reduce the value of social networks [2] for companies. The privacy of individuals needs to be protected and user of SNSs open part of their privacy. They resist to use SNSs because their privacy is damaged and their data is misused. Companies violate the privacy in SM [3]. Analyze tools enable companies to explore the private life of SNSs members and to misuse this data for marketing purposes. That influence the acceptability as an information channel. Some people used SNSs as an informal communication channel and companies have had access to the communication and exchanged information. The user resist to use SNS to exchange information. The user reduce the access to information because they leave SNSs [4]. They do not share © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 23–33, 2021. https://doi.org/10.1007/978-3-030-72651-5_3
24
T. Sander and P. L. Teh
anymore critical or private information with other members, this study aims to find out what may affect others to trust the SM content that were privately shared. The experience with content shared in SM platforms can be a reason for an individual to evaluate that the information posted online as unreliable, or rate SM as untrustworthy. The misuse of SM messages from criminals’ group can damage the experience of an individual. Especially in the case when financial information is involved during the process. Journalists have been reporting about negative situations and fake news [5], which advice individuals to be more careful online [6]. Miscommunication of information could lead an individual to find this communication channel less valuable, and hence not to trust the information they have read [7]. This situation can cause a particular impact on companies financially. The impact of SM on our daily life is important. The technical opportunities and behavior of SM user influence the trust in SM. The value of SM is the transfer of information fast and easily between individuals [8]. Individuals can collaborate on various topics and develop new ideas. That support individuals in their future, to develop skills and to improve results of projects e.g. marketing or developing innovative products. That is only possible with trust in the provided information in SNSs [9]. The access to resources and maintenance of relationships is important asset of SNSs. SNSs are communities, electronic networks or organized groups which are connected with ties [10]. In this case, the most popular SNSs are LinkedIn or Facebook [11]. People needs to trust the information and use the information for their decision. Individuals get daily fake news, spam messages and phishing e-mails. This has a heavy influence on individuals in their daily use of SM information and damage the value of this communication tool [12]. Companies are under pressure to identify reasons that influence the trust of individuals in SM content. The government and society can be damaged if people following fake information. In crisis situation, particularly important that people able to trust information to decrease the negative impact on society. A good example is to trust the online information overlying the message that to stay at home is able to stop the spread of Coronavirus during the pandemic. So, what makes an individual trust what they read on SNSs? The authenticity of a piece of information was said to depend on the reputation of the publisher. The reputation status of a publisher is reflected via the number of followers [13]. The more the follower, the higher the number of visibilities on that piece of said information [14]. This scenario allows for broader access to the target group and more extensive involvement from different stakeholders. Experience determines epistemic trust [15]. If once have a positive experience in trusting a piece of information, they will return to the same source. Vice versa, if they have a negative experience, they will avoid to source information from the same channel. For instance, if a product was recommended to be good online, and individual that received the same good experience with the said product will later, be encouraged the returning to the same dealer [16]. The same person that is posting the same positive information leading on a specific product/organization on the same channel, and creating a line of the follower. They are an influencer, who then lead the follower to excellent access to additional and exclusive information [17]. On top of knowing someone virtually, individuals that have also known a person or have previously established relationship physically would also affect the trust [18].
Reasons of Individuals to Trust What They Read
25
The social tie is more durable; therefore, the faith is more substantial. The anonymity is in reality less compared to virtual environments [19]. It is fair to say that marketing commonly presents its products or services in a very positive manner [20]. Hence, information that comes from the social network can be more sensible compared to companies. The authenticity of the information is higher than other sources, and therefore, less doubt on the information presented [21]. The research has been done in UK and Germany. There are differences between countries in the use of SNSs. International companies need global solutions and need to understand their customer in different locations [22]. The research is concentrating on Germany and UK but further countries need to take under consideration to provide a deeper insight and to provide a global answer. This paper is organized as follows. The Signaling theory is presented in next section. Section 3 present the model and analyses the social demographic of the participants. Section 4 and 5 are results obtained from German and UK participants respectively. Section 6 presents the regression analysis for all the items discussed relevant to signaling theory and end with the conclusion.
2 Signaling Theory The signaling theory is a prominent theory to explain the transfer of information. The kind of signal needs to be transferred between individuals and the signal needs to have a value for the participants of the communication, that the individuals take part in the communication [23]. The knowledge which is gained by the information can be important to make a good decision, and cost saving. A typical example is the reason to employ an individual grounded on the collected signals which are later to be transferred in the Curriculum Vitae (CV) in the requirement process or during an interview [24]. SM increase the level of personal information to be observed as a signal to the users. The signal such as body language can be observed through video conference, but not in phone call. Signal is referring to the idea that one credible party trying to convey some information about something to a receiver. The signal is used to make a decision. The most famous example of a scenario to explain this is through the process of purchasing a second hand used car. The sales person has more knowledge about the conditions of the car e.g. former damages of the car from an accident or experience in negotiation about a price for a car. The sales person can choose to hide negative information and to transfer only positive signal to the buyer. That is the advantage of signal received by the sales person against the buyer [25]. This information is named as “economical” benefit. This situation is mentioned in the literature as transaction cost [24]. The value of the information can be explained with the exclusivity of a piece of information that is not freely available from other resources (e.g.: SM). For instance, to obtained the information from purchasing it from a broker, which in this case, can be more expensive comparing to information that can be obtained from SNSs [26]. SNSs are social networks in the internet, on a technical basis. If you are not a member in the social network, you would not be able to expose any signal and send to other individual. People are connecting to SM through posting from each other to
26
T. Sander and P. L. Teh
obtain information. Information is easily convey globally to a large audience on a low level of cost. A typical example from Sobel are about the diamond sellers who exchange diamonds for million of Euros without any guarantee. They take part in the exchange because they are part of a social network with their rules and agreements, that the member of the network would trust each other [27]. Trust is created with experience and based on expectations that were previously fulfilled [18]. This processes are also explained with the social capital theory, that people can increase their social capital with their knowledge [28]. The explanation of the value of the signal in SNSs is influenced by the social capital. As a consequence, this study aims to find out if the investment on SNSs (e.g.: time spend on the SM, the number of contact they have in their network) can affect the trust from other parties.
3 Model and Method The model for this research paper has the dependent variable “trust” and the independent variable “use of SNSs” to explain the influence of the use of SNSs to trust or not to trust information on SNSs. The regression analysis test the impact of use of SNSs on the trust in information in SNSs. The expectation and hypothesis is that the intensive use of SNSs has a positive value on the trust of information in SNSs. That the trust in information in SNSs depends on the use of SNSs. The intensive trust develops skills to evaluate information in SNSs about their trustfulness and experience influence the trust in the information at SNSs [29]. That people with more experience and higher intensive use of SNSs have a higher level of trust in SNSs information [30]. That would explain the extensive use of SNSs as it provides a value to the intensive user.
Contacts in social media Average time spend per day (in minutes) Years of experience using social media
1. 2.
Level of investment in SNSs
3. 4. 5.
The reputation of the person who published the information Your experience with information from this person in the past The number of posts about any given topic from the same person Knowing the person who has written the post personally The veracity of information is higher than other sources
Impact on / Affect Trust
Fig. 1. Impact of investments in social network sites on factors which affect trust on information in SNSs
The independent variables on the left hand side of the model in figure one describe the individual’s investment (e.g.: time, duration and years of experience) in SNSs. Box in the middle describe the 5 dependent variables to the question of “What makes you trust what you read on SNS?”. These dependent variables are signals from each individuals in SNSs. This signals are convey to other SNSs members. These signals have an impact on individual’s trust on information in SNSs. In general, the model explains that the investment (e.g.: contact in SM, average in time spend per day and years of experience using SM) has an impact on the reason
Reasons of Individuals to Trust What They Read
27
which makes individuals to trust what they read on SNSs. The result explains the reason to trust somebody in SNSs affect the trust on information in SNSs. This study has been done with an online survey due to the fact that the research field is online related and participants are expected to be part of the user of SNSs that have access to the internet, which can be cost effective and efficient [31]. The data has been collected between November and December 2018. The questionnaire was designed in both German and English languages. The demographic factors of the participants is presented in Table 1. Up to 56 British participants and up to 144 German individuals take part in the survey. Table 1. Demographic data of the participants of the survey UK n 46–49, German n 133–136 Age in years
UK sample
German sample
Social status
UK sample
German sample
Mean
33.9
34.07
Employed
78.26%
72.93%
Std Dev.
9.31
11.3
Unemployed
2.17%
21.80%
Minimum
18
18
Student
8.70%
0.75%
Maximum
56
69
Self employed 10.87%
4.51%
Median
32
33
Male
51.02%
44.12%
Gender
Female
48.98%
55.88%
The use of SNSs is defined with the duration of membership in years, number of contacts and time spend per day in average per minutes [32]. The details are presented in Table 2. Table 2. Indicators of the use of SNS – number of contacts, duration of membership and time of use in minutes per day, German and UK sample N Contacts in SM - Germany
Mean
Std. Dev. Min Max
137 594.38 1171.34
Average time spend per day (in minutes) - 137 Germany
0
Median
10000 250
45.88
59.29
0
360
30
6.55
4.02
0
15
7
Contacts in SM – UK
49 660.22
635.42
2
Average time spend per day (in minutes) UK
49
59.73
66.78
2
300
30
Years of experience using SM - UK
49
8.71
2.89
2
15
8
Years of experience using SM - Germany
137
3500 500
The data is analyzed with descriptive statistic e.g. mean, standard deviation, median and mode. Further is the data analyzed with an ANOVA and tested with T-test for gender
28
T. Sander and P. L. Teh
differences. The fit of the model is analyzed with regression analysis. The main focus of the paper is on the model calculated with a regression analysis to explain the influence of use of SNSs on the trust in information on SNSs [33].
4 Results for the German Sample The descriptive statistic provides a first overview about the factors and described in detail below. The most positive influence on trust has the item G4. This is an interesting point that the real social network and reality has a large influence on individuals. That influencer and companies need to get in touch on various level with customer to improve the trust level. The lowest influence on the trust in the information has the item G5 with a mode of six and median of five. The expectation to get a more veracity of information, to get the reality from another perspective to increase the reality is not a reason to trust the results more compared with other items. A tendency to a positive influence on the trust is the experience with the information from an individual. That means to publish wrong information or fake information has a negative impact on trust in information on future messages. The result for the item G2 has a median of three and mode of three. The reputation of the person who post the information has a median and mode of four (item G1). The number of posts is not deeply relevant that people from Germany do not trust information in SNSs (item G3). It appears that the number of posts is not relevant for the trust level in a negative or positive way. Table 3. “What makes you trust what you read on SNS?”, German sample, 1 for strongly agree to 6 for strongly disagree ID
Item
N
Mode
Std. Dev.
Median
G1
The reputation of the person who published the information
143
4
1.46
4
G2
Your experience with information from this person in the past
141
3
1.46
3
G3
The number of posts about any given topic from the same person
141
6
1.38
4
G4
Knowing the person who has written the post personally
144
2
1.68
3
G5
The veracity of information is higher than other sources
143
6
1.29
5
It does not exist a statistical significant relevant difference for the explored items between the answers of men and women for the German sample. That has been tested with a t-test.
Reasons of Individuals to Trust What They Read
29
5 Results for the English Sample The response from the participants from the UK have a higher tendency to strongly agree. The item E1, E2 and E4 have median and mode two. On the other side of the scale to strongly disagree with median and mode four are the items E3 and E5. That means there are two groups of items with different impact on the trust level for information. Table 4. “What makes you trust what you read on SNS?”, UK sample, 1 for strongly agree to 6 for strongly disagree ID Item
N
Mode Std. Dev. Median
E1 The reputation of the person who published the information
56 2
1.31
2
E2 Your experience with information from this person in the 56 2 past
1.29
2
E3 The number of posts about any given topic from the same 56 4 person
1.3
4
E4 Knowing the person who has written the post personally
55 2
1.58
2
E5 The veracity of information is higher than other sources
55 4
1.46
4
The descriptive statistic results is strongly supported by the distribution of the answer of the British participants. It does not exist a statistical significant difference for the explored items between the answers of men and women for the English sample. That has been tested with a t-test.
6 Results of the Regression Analysis This section present the results of the regression analysis. The concept is that individuals who use the SNSs intensive use the information and trust the information in a different way compared with individuals who use the social network less intensive. The intensity of use is described with duration of membership in years, number of contacts and how many minutes per day individuals use SNSs. The regression analysis test the impact of the use of SNSs information. That heavy SNS user have a different anticipation of information compared with normal SNSs user. R2 is measurable but on a weak and different level depending on the trust variable G1–G5. There are four items with a significant result (tested with ANOVA) The highest R2 is scored by item G3, it shown that the independent variables explain best the item G3. That the prediction of the trust is possible with the use of SNSs. Follow by G2 and G5 with the result 0.11 for R2 . That the trust in information is predictable with the use of SNSs. The weakest statistically significant result has with the item G1 is 0.09 for R2 . The use of SNSs can explain the level of trust in information from SNSs with different strong impact. The results for the model with the indicators for trust are presented in Table 5.
30
T. Sander and P. L. Teh Table 5. Results of the regression analysis for the German sample
ID
Item
G1
R
R-Square
Corrected R – Square
Standard error of the estimate
The reputation of the 0.29 person who published the information
0.09
0.06
1.43
G2
Your experience with information from this person in the past
0.34
0.11
0.09
0.34
G3
The number of posts about any given topic from the same person
0.42
0.17
0.15
1.27
G4
Knowing the person who has written the post personally
0.25
0.06
0.03
1.64
G5
The veracity of information is higher than other sources
0.33
0.11
0.08
1.24
The item G4 with the strongest tendency to fully agreement is not statistically significant for the regression analysis and has the lowest R2 . That means the impact of the intensity of use of SNS is on a low level. There is some influence but further research needs to find more variables and items to explain the trust in information from SNSs. The corrected r – square is very low, the determination is weak and it is not on a statistical accepted scientific significant level (tested with ANOVA). That means the investment on SNSs does not impact the indicators from E1–E5. The last tables provide the full picture of the results of the investigation but the results cannot be used for any explanation. That mean comments regarding the UK model cannot be done on a serious scientific level with this results.
Reasons of Individuals to Trust What They Read
31
Table 6. Results of the regression analysis for the UK sample ID
Item
E1
R
R-Square
Corrected R – Square
Standard error of the estimate
The reputation of the 0.24 person who published the information
0.06
−0.03
1.08
E2
Your experience with information from this person in the past
0.17
0.03
−0.06
1.11
E3
The number of posts about any given topic from the same person
0.25
0.06
−0.03
1.36
E4
Knowing the person who has written the post personally
0.37
0.14
0.05
1.43
E5
The veracity of information is higher than other sources
0.33
0.11
0.02
1.34
7 Conclusion As a conclusion, the acceptance of information depends on the signal a person holds, which are affected by the environment, situation and person who presents the information. To summarize, there are two items that are highly responsible in influencing the trust of a piece of information. These two items are firstly refer to the frequency of exposure on a same piece of information and secondly, personal experience or knowing the person in real life. Specifically, German show higher needs in having various experiences in order to trust a piece of information. British participants, however, rated less. Reputation is more important factor that influence the trust for UK participants. The results for the regression analysis for the German items is statistic significant but the results for the UK is not on a statistic relevant significant level. There is no significant relevant answer for the UK sample regarding the model. The results of R2 is not for all items for Germany on a statistical relevant level and the R2 is on a low level. The power of the impact of the independent variable on the dependent variables are weak. The regression analysis provides the feedback that the use of SNSs has influence on the trust in SNSs in Germany. The majority of the items can be explained with the intensity of use. The intensity of use is defined with the user’s invested in SNSs. That is important to cluster user of SNSs to provide the best solution e.g. information on a regular basis to create trust, that the individual get experience. The only item without a statistical significant level result is “Knowing the person who has written the post personally”. That means that the use of SNSs influence the reason to believe information on SNSs.
32
T. Sander and P. L. Teh
Further research needs to explore more countries and items to provide more support to the business to decide how to make valuable communication and sending signals to the audience which are used, trusted and accepted.
References 1. Ferreira, A., Antunes, F.: Essential functionalities for commercial internet presence. Int. J. E-bus. Res. 11(1), 56–83 (2015) 2. Li, C., Bernoff, J.: Groundswell, Winning in a World Transformed By Social Technologies. Harvard Business Review Press, Boston (2011) 3. Conde, M.Á., Fonseca, D.: Information society skills: is knowledge accessible for all? Part I. Univ. Access Inf. Soc. 17(2), 223–227 (2018) 4. Fonseca, D., Conde, M.Á.: Information society skills: is knowledge accessible for all? Part II. Univers. Access Inf. Soc. 17(3), 447–451 (2018) 5. Ross, A.S., Rivers, D.J.: Discursive deflection : accusation of ‘fake news’ and the spread of mis- and disinformation in the tweets of president Trump. Soc. Media + Soc. 4, 1–12 (2018) 6. Bakir, V., McStay, A.: Fake news and the economy of emotions: problems, causes, solutions. Digit. J. 0811, 1–22 (2017) 7. Turcotte, J., York, C., Irving, J., Scholl, R.M., Pingree, R.J.: News recommendations from social media opinion leaders: effects on media trust and information seeking. J. Comput. Commun. 20(5), 520–535 (2015) 8. Hemsley, J., Jacobson, J., Gruzd, A., Mai, P.: Social media for social good or evil : an introduction. Soc. Media + Soc. 4, 1–5 (2018) 9. Tsai, C.W.: Applications of social networking for universal access in online learning environments. Univers. Access Inf. Soc. 16(2), 269–272 (2017) 10. Calero Valdez, A., Brell, J., Schaar, A.K., Ziefle, M.: The diversity of why: a meta-analytical study of usage motivation in enterprise social networks. Univers. Access Inf. Soc. 17(3), 549–566 (2018) 11. Ellison, N.B., Steinfield, C., Lampe, C.: The benefits of Facebook ‘friends:’ social capital and college students’ use of online social network sites. J. Comput. Mediat. Commun. 12, 1143–1168 (2007) 12. Simeone, M., Russo, C.: The growing influence of social and digital media. Br. Food J. 119(8), 1766–1780 (2017) 13. Stenger, T.: Social media and online reputation management as practice: first steps towards social CRM? Int. J. Technol. Hum. Interact. 10(4), 49–64 (2014) 14. Maras, M.: The economic costs and consequences of mass communications data retention: is the data retention directive a proportionate measure? Eur. J. Law Econ. 33, 447–472 (2012) 15. Wang, C., Chen, C., Chen, C.: Exploring the different aspects of internet leisure use by college students. Inf. Dev. 31(1), 5–12 (2015) 16. Rezaei, S., Ismail, W.K.W.: Examining online channel selection behaviour among social media shoppers: a PLS analysis. Int. J. Electron. Mark. Retail. 6(1), 28 (2014) 17. Gillin, P.: New media, new influencers and implications for the public relations profession. J. New Commun. Res. II(2), 1–0 (2008) 18. Pruijt, H.: Social capital and the equalizing potential of the internet. Soc. Sci. Comput. Rev. 20(2), 109–115 (2002) 19. Wellman, B., Boase, J., Chen, W.: The networked nature of community online and offline. IT Soc. 1(1), 151–165 (2002) 20. Bernoff, J., Schadler, T.: Empowered. Harvard Business Review Press, Boston (2010)
Reasons of Individuals to Trust What They Read
33
21. Kim, J., Lee, C., Elias, T.: Factors affecting information sharing in social networking sites amongst university students. Online Inf. Rev. 39(3), 290–309 (2015) 22. Sander, T., Sloka, B., Puke, I.: Differences in the exchange of contents of different countries in social network sites. Int. J. E-colaboration 13(2), 17 (2017) 23. Spence, M.: Job market signaling. Q. J. Econ. 87, 355–374 (1973) 24. Fussell, H., Harrison-Rexrode, J., Kennan, W.R., Hazleton, V.: The relationship between social capital, transaction costs, and organizational outcomes: a case study. Corp. Commun. Int. J. 11(2), 148–161 (2006) 25. Emons, W., Sheldon, G.: The market for used cars: new evidence of the lemons phenomenon. Appl. Econ. 41 (2007) 26. Arvidsson, A.: Speaking out: the ethical economy: new forms of value in the information society? Organization 17(5), 637–644 (2010) 27. Sobel, J.: Can we trust social capital? J. Econ. Lit. XL, 139–154 (2002) 28. Behtoui, A.: Beyond social ties: the impact of social capital on labour market outcomes for young Swedish people. J. Sociol. 50(1), 1–14 (2015) 29. Nelmapius, A., Boshoff, C.: A motivational perspective on the user acceptance of social media. South Afr. J. Bus. Manag. 47(4), 1–3 (2016) 30. Sander, T., Teh, P.L.: Is social media changing the awareness of individuals? In: International Conference on New Trends in Social Sciences (2017) 31. Evans, J.R., Mathur, A.: The value of online surveys. Internet Res. 15(2), 195–219 (2005) 32. John, R.B.A., Lucila, O.B.: Personality traits as predictors of Facebook use. Int. J. Psychol. Couns. 8(4), 45–52 (2016) 33. Rosopa, P.J., Kim, B.: Robustness of statistical inferences using linear models with metaanalytic correlation matrices. Hum. Resour. Manag. Rev. 27(1), 216–236 (2017)
Perspectives of Companies and Employees from the Great Place To Work (GPTW) Ranking on Remote Work in Portugal: A Methodological Proposal Anabela Mesquita1 , Adriana Oliveira2 , Luciana Oliveira2(B) Arminda Sequeira2 , and Paulino Silva2
,
1 Polytechnic of Porto and Algoritmi RC, Rua Jaime Lopes Amorim, Matosinhos, Portugal 2 CEOS.PP ISCAP Polytechnic of Porto, Rua Jaime Lopes Amorim, Matosinhos, Portugal
{sarmento,lgo}@iscap.ipp.pt
Abstract. Traditionally, there have been many reasons and factors contributing to the disbelieve in remote work as a large-scale efficient work format. But with the social distancing imposed by the COVID-19 pandemic, remote work (RW) emerged as an adequate solution to continue labor at a large scale, even in the situations where remote work was unforeseeable and, in many cases, happened under inappropriate conditions. Despite some advantages, which are mainly identified by employees, there are other issues, pointed out by organizations, such as required technology infrastructure, HR training, etc. In any case, opting for remote work present some gains and losses for both employers and employees. With the end of the mandatory confinement, organizations reorganized the work format and, in some cases, it consisted in restoring the previous one. At this point, many questions arise being important to evaluate the perspectives of both sides – workers and companies – as well as their perceptions retrieved from the RW during the Spring confinement period, to understand how was the transition period and preview the future in terms of RW becoming part of the work model or even the way the pandemic situation is influencing future changes in businesses models. To achieve that goal, an exploratory survey-based research, and interviews will be carried out. The sample consists of the top 100 private companies of the Great Place To Work (GPTW) Portuguese ranking. Keywords: Remote work · Portugal · Covid-19 · GPTW companies · Employees
1 Introduction The pandemic situation, in March 2020, put more 16 million US knowledge workers and more than 68 thousand Portuguese public service workers in remote work (RW) [1, 2]. This situation was facilitated by the technology as it enabled the communication between workers and companies and between companies and their customers [3–5]. In Portugal, there is the emerging notion that remote work may have come to stay; in © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 34–40, 2021. https://doi.org/10.1007/978-3-030-72651-5_4
Perspectives of Companies and Employees from the Great Place To Work (GPTW)
35
fact, in November 2020, during the second wave, one of the measures advanced by the Government was the compulsory remote work, except for the job positions in which is not viable [6]. A study conducted during April 2020 shows that companies (45%) and workers (55%) want to keep pursuing RW. Among the reasons given are increased productivity and quality of life [7]. Currently, in November 2020, although RW is already decreed and compulsory, many workers and organizations are still adapting to this new reality. As such, this research intends to contribute to deepen workers’ perspectives in relation to RW and add the perspective of the companies. So, the objectives of the study include the identification of the changes introduced by RW during the confinement period, the possibilities that the RW may have opened, both to workers and companies, and the repercussions that it might have in the future organization of work, including the technological infra-structures, the flexibility of work modes, changes in the companies’ business model, etc. Moreover, it is also our aim to compare this perspective with the employees’ point of view. Only a comparison between both perspectives will provide a clear and deep understanding of what happened (and is still happening) in order to identify possible ways to overcome the difficulties felt by both – workers and companies – and emerging opportunities. Thus, in this paper we present the research design and methodology followed to achieve the proposed goals.
2 Background The pandemic caused by the COVID-19 forced a period of confinement and posed several challenges. Many organizations had move employees to RW, requiring them to work from home or other physical spaces and requiring heavy use of ICT. 2.1 Remote Work, Telework, Work at Home or Home-Based Work - Definition Working at distance and working from home are not new phenomena. However, in the last few months its relevance has increased as an answer to maintain jobs active during the pandemic crisis. According to the literature there are different concepts describing the settings in which the work is performed outside of organizations’ facilities. Nowadays, various countries are using slightly different and sometimes overlapping definitions and different terms are being used interchangeably [8], such as “remote work”, “telework”, “work at home” and “home-based work”. According to the International Labour Office [9], these concepts are clearly related and have some degree of overlap. To the ILO (2020), there is no international definition of RW. Nevertheless, it can be described as work carried out entirely or partially outside of the organization. As for telework, ILO [9] considers that it comprises two different components: the work is fully or partly carried out at an alternative location other than the default place of work; the use of personal electronic devices such as a computer, tablet or telephone to perform the work. In the context of this paper, we will use the concept of remote work as it represents an alternative place where the activity can be carried out (whether with technologies or mobile phones, or not). In the pandemic situation, this solution of RW was the option of many companies [10–12] and in this scenario, technology played a very important role, because it increased the opportunity for employees to do their work in an alternative
36
A. Mesquita et al.
workspace and allowed companies to communicate better with different audiences; so, the companies that were more digitally prepared were those that managed to achieve high levels of performance among employees during the pandemic [13]. 2.2 Remote Work During COVID-19 Although many organizations have returned to face-to-face work, in November 2020, with the rise of the second COVID-19 wave, many of them opted for a remote or mixed work regime [15, 16]. The impact of these changes can be seen in the well-being of citizens/workers and also in their working conditions. Well-Being RW allows for a better interconnection and balance between professional, domestic, and leisure activities, increasing productivity, flexibility in organizing personal time, happiness, job satisfaction and reduces the stress of the balance between work and domestic tasks [14]. Among the disadvantages of remote work are lack of socialization, social isolation, some implications for the worker’s psychological and physical health and work invasion of family life [14]. During the first lock down, in a survey conducted by the Eurofund, with more than 90 000 respondents from all over Europe, women reported more often than men difficulties in balancing work and private life [15]. Also, at that time (Spring 2020), a significant portion of respondents reported having had negative feelings in the two weeks preceding the survey, such as loneliness, tension and depression. In July, during the 2nd round of the survey, 4 out of 10 unemployed respondents indicated that they felt left out of society. Nevertheless, optimism about one’s future improved between April and July, but social differences are still evident. In fact, men were more likely to be more optimistic than women in April, and men’s optimism improved more than women’s by July. In the second round, the employees were more optimistic than the self-employed and youngsters [15]. In addition to unemployed people, women showed lower resilience than men; better resilience was found among self-employed respondents as well as in those with tertiary education and aged 50 and over. Working Conditions In a RW regime, the managers play an important role in improving working conditions [16], the organizations need to reevaluate the context of social interaction within the organization and promote the best articulation between the actors [16, 17] and the communication channels should be (re)defined allowing quick communication between employees [18]. The organizations register more satisfied employees at work and there is a reduction in psychological tension, which reveals a better remote worker-environmental adequacy [19]. According to the Eurofund report [15], the experience of working from home during the first lock down appears to have been a positive one for many employees. However, the majority of respondents had to use their own equipment/devices to work and, in some situations, this equipment had to be shared with other family members. Moreover, the respondents had to work in their free time, especially when there were children in the household. They also reported high quantitative demands from the managers and reported feeling isolated. Nevertheless, respondents stated in the July’s
Perspectives of Companies and Employees from the Great Place To Work (GPTW)
37
survey that their preferred format would be a mix presence at the workplace and working remotely. 2.3 Remote Work in Portugal – Wellbeing and Working Conditions During the recent pandemic crisis, many services have sent their employees home, expecting them to, intuitively, know how to work remotely, and manage a completely new context where family and work coexist in the same space and time frame. In a recent research carried out in the Portuguese context [20] during the lock down, with a sample of 305 persons, it was possible to observe that more than half of the respondents (65%) preferred not to be in remote work, 49% of respondents believe that they work more at home than in a regular situation, and most of the participants (68%) do not feel that management controls their working hours. As the main disadvantages of remote work, the respondents indicated: distance from work colleagues (76%); a blend of professional and family life (64%) and lack of support when needed (39%); the lack of an appropriate physical space for work (49%) and the difficulty of access to technology (35%). On the advantages side, respondents identified the following: gain of time (79%), improved schedules management (57%) and flexibility (44%). As difficulty with RW, families, also, mentioned the low interaction/communication with coworkers, the lack of resources such as the internet or a printer, the balance between working with family life/household chores/dedication to children or time/schedule management [21]. In these studies, the focus was on the workers, in their feelings and attitudes, difficulties and survival strategies. In our research proposal, the focus is deepening the discussion on the workers’ perspective in relation to RW while adding the organizations’ perspective.
3 Research Design and Methodology As the literature review showed, in Portugal, as in other countries, the first confinement caused by the Covid-19 virus put many collaborators in RW; this situation is forcing a strong, fast, and disturbing adaptation for many employees and organizations [9, 20, 22]. This project is addressed at the circumstances arising with the second wave of confinement, when Portuguese organizations and workers return to RW [6]. In this context, it is the purpose of this research to deepen the discussion on the workers’ perspective about the remote work, initiated in previous work [20], and also to add the perspective of organizations. Thus, the main objectives that guide the research consist of: • Organizations: to characterize the main challenges (infrastructural, financial, business, market, HR, etc.) arising from the abrupt need to shift operations to remote work; realize if the changes will impact only the way work is performed and for instance contributed to making work models more flexible; understand if those changes will have the possibility to become permanent; understand, due to experience retrieved from this period, if the business model will suffer strategic or minor changes. • Employees: to assess the main challenges faced when transitioning to RW (at professional, personal, financial psychological level); to evaluate the RW situation after
38
A. Mesquita et al.
the emergency period; evaluate the perception about the possibility of the changes to remain in the future and to what extent they are perceived as advantageous. 3.1 Research Design This research fits into a mixed methodological framework - quantitative and qualitative see Fig. 1 – in which we will use large scale surveys aimed at employees and interviews aimed at HR managers.
Fig. 1. Methodological framework
The questionnaire will be disseminated among companies through the HR managers. The interviews will be applied to HR managers, as they have developed, over their professional career, a special sensibility to understand workers points of view and to collect feedback from them to provide leadership with accurate and sustained information. The choice to establish contact through the HR manager is justified by the proximity that such professionals have to the leadership of the companies involved and, on the other hand, the close interaction with the workers that such a professional position requires. 3.2 Sample The ranking of the top 100 companies of the Great Place to Work (GPTW) in Portugal distinguishes a group of companies based on the evaluation of their employees and their internal, external, and social performance profile. These are private companies whose leadership and management are, in the employees’ perception, oriented to the responsible and sustainable development of the organization itself. The publication of the ranking in Portugal was created and managed by EXAME magazine, Everis and AESE, and aims to encourage the dissemination of good practices among Portuguese companies. Furthermore, there are additional benefits in collecting data from this set of companies (100GPTW) because being already referenced, may offer greater potential for transferring good practices nationally and internationally. As for the dimension of the sample we consider that is relevant and representative enough.
Perspectives of Companies and Employees from the Great Place To Work (GPTW)
39
3.3 Data Collection and Analysis The questionnaire addressed to employees will be inserted in Lime Survey and sent, through internal means of organizations, to employees. All employees have the same opportunity to participate and will answer freely and without imposition. The interviews with HR managers (or any other professional in charge of HR management) will take place during the same period of the application of the questionnaires to employees. After data collection, the data analysis stage follows. A content analysis will be performed using MaxQDA software and statistical analysis using SPSS - Statistical Package for the Social Sciences.
4 Conclusion Our knowledge about the challenges companies and employees are facing due to the COVID-19 and the adoption of RW without previous preparation and its impact on society and labor is still in its infancy. As such, this research is expected to deepen the current knowledge of the context of RW from the point of view of employees and organizations. Moreover, we aim to identify the impact of RW on employees in different levels such as professional, personal, psychological, and financial. From the point of view of the organizations, we expect to identify effective work management models, future perspectives and recommendations. These findings will emerge from a set of organizations that are distinguished for having, among others, a culture of care for their workers, thus we expect research to reveal possible paths for harmonizing RW in the current labor panorama. In this paper, we presented the framework of the study as well as the assumptions concerning the empirical practice. Results will contribute to a better knowledge of the situation and allow managers to introduce new and better practices in order to overcome the challenges employees are facing.
References 1. Hanson, R.S.: Report: Remote work in the age of Covid-19 (2020). https://slackhq.com/rep ort-remote-work-during-coronavirus 2. Lopes, M.: Governo quer manter em teletrabalho um quarto dos actuais 68 mil funcionários públicos neste regime (2020) 3. Guerra, A.R.: Serviços de colaboração remota, robôs autónomos, gadgets domésticos e videojogos registam uma grande subida de interesse e utilização (2020). https://www.dinheirovivo. pt/buzz/isolamento-e-teletrabalho-as-tecnologias-em-alta-com-a-crise-covid-19/ 4. WEF: The future of jobs report 2018. World Economic Forum Geneva (2018) 5. Wong, K.: 25 Key Remote Work Statistics for 2020 (2020). https://www.business2commun ity.com/human-resources/25-key-remote-work-statistics-for-2020-02299342 6. Micael, M.: COVID-19: AS NOVAS MEDIDAS DO GOVERNO PARA COMBATE À PANDEMIA (2020). https://tvi24.iol.pt/politica/conselho-de-ministros/covid-19-as-novasmedidas-do-governo-para-combate-a-pandemia 7. Lima, C.R.: Portugueses querem ficar em casa no pós-pandemia (2020) 8. Eurofound and t.I.L. Office: Working anytime, anywhere: The effects on the world of work. Publications Office of the European Union, Luxembourg, and the International Labour Office, Geneva (2017)
40
A. Mesquita et al.
9. ILO: COVID-19: Guidance for labour statistics data collection: defining and measuring remote work, telework, work at home and home-based work (2020). https://www.ilo.org/wcmsp5/ groups/public/---dgreports/---stat/documents/publication/wcms_747075.pdf 10. Belzunegui-Eraso, A., Erro-Garcés, A.: Teleworking in the context of the Covid-19 crisis. Sustainability 12(9), 1–20 (2020) 11. Elavarasan, R.M., Pugazhendhi, R.: Restructured society and environment: a review on potential technological strategies to control the COVID-19 pandemic. Sci. Total Environ. 725, 138858 (2020) 12. Waizenegger, L., et al.: An affordance perspective of team collaboration and enforced working from home during COVID-19. Eur. J. Inf. Syst. 29, 14 (2020) 13. Bartsch, S., et al.: Leadership matters in crisis-induced digital transformation: how to lead service employees effectively during the COVID-19 pandemic. J. Serv. Manag. 32, 71–85 (2020) 14. Baert, S., et al.: The COVID-19 crisis and telework: A research survey on experiences, expectations and hopes (2020) 15. Eurofound: Living, working and COVID-19, COVID-19 series. Publications office of the European Union, Luxembourg (2020) 16. Kristman, V.L., et al.: Supervisor and organizational factors associated with supervisor support of job accommodations for low back injured workers. J. Occup. Rehabil. 27(1), 115–127 (2017) 17. Shaw, W.S., et al.: Opening the workplace after Covid-19: what lessons can be learned from return-to-work research? J. Occup. Rehabil. 30, 299–302 (2020) 18. ILO: An employers’ guide on working from home in response to the outbreak of COVID-19 (2020). https://www.ilo.org/wcmsp5/groups/public/---ed_dialogue/---act_emp/ documents/publication/wcms_745024.pdf 19. Bentley, T., et al.: The role of organisational support in teleworker wellbeing: a socio-technical systems approach. Appl. Ergon. 52, 207–215 (2016) 20. Are we ready for remote work? Preliminary results from Portugal. In: 20ª Conferência da Associação Portuguesa de Sistemas de Informação, Portugal (2020) 21. Tavares, F., et al.: Teleworking in Portuguese communities during the COVID-19 pandemic. J. Enterp. Communities People Places Global Econ. (2020) 22. Ahrendt, D., Cabrita, J., Clerici, E., Hurley, J., Leonˇcikas, T., Mascherini, M., Riso, S., Sándor, E.: Living, working and COVID-19 Facebook Twitter LinkedIn (2020). https://www.eurofo und.europa.eu/sites/default/files/ef_publication/field_ef_document/ef20059en.pdf
Overlaps Between Business Intelligence and Customer Relationship Management – Is There a Place for E-Commerce? Ionu¸t-Daniel Anastasiei and Mircea-Radu Georgescu(B) “Alexandru Ioan-Cuza” University of Ia¸si, 700506 Ia¸si, Romania [email protected], [email protected]
Abstract. This paper aims to determine if there is a place for e-commerce in the literature on the topic of overlaps between Business Intelligence (BI) and Customer Relationship Management (CRM), with implications in other domains such as Social Media, Big Data, and Machine Learning. Academic researchers should also aim to increase applications to e-commerce in their studies whenever there is an overlap between CRM and BI. The practical implications considered the steps for systematic review as acknowledged by the scientific world and the findings were analyzed both quantitatively and qualitatively. With the use of cutting-edge software tools specialized in reviews, the research methodology led to an empiric answer, which can be stated as this article’s contribution to scientific relevance to academics and practitioners. There is indeed a place for e-commerce in the overlaps between BI and CRM, but not in all authors’ speeches. Keywords: Business Intelligence · Customer Relationship Management · E-commerce · Systematic review
1 Introduction This study contributes towards the knowledge agenda of multidisciplinary domains based on Business Intelligence (BI) and Customer Relationship Management (CRM) overlapping, by providing a systematic review of the research addressing the scientific need to set these two domains apart or put them together. There is an ongoing debate on whether BI or CRM is more effective in helping a business succeed. The comparison arises when the tools used by the two share some functions, one of these functions being the use of historical data to identify key trends that companies can leverage to their advantage [1], especially in e-commerce. CRM refers to creating a dialogue with customers in order to develop long-term relationships, by satisfying their expectations and retaining them. Also, this may help acquire new customers based on the historical data collected and interpreted in the past [2]. The main goal of CRM software is to increase the number of customers by refining the quality of the interaction managed by the company, which can ultimately lead to the growth and expansion of organizations [3]. Simultaneously, BI can be interpreted as a set of technologies helping companies use their data to make better decisions [4]. The main © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 41–55, 2021. https://doi.org/10.1007/978-3-030-72651-5_5
42
I.-D. Anastasiei and M.-R. Georgescu
goal of BI software is to understand, transform, and shape the data to gain a competitive advantage on the market, by means of gathering and analyzing a vast amount of external and internal data across the enterprise [5]. At first glance, there are some practical benefits of the overlap between BI and CRM, such as customer data integration in the data warehouses. This leads to a better utilization of data, with cost optimization and availability of the data, which can ultimately contribute to a better understanding and retention of customers [6]. Data mining and OLAP functionalities of BI help organizations to better understand their data in order to enable personalized customer offers, which can improve the relationship with the customer [7, 8]. As shown in the next sections, there can be various data sources for further analysis, such as data from CRM, social media or surveys, but BI tools can help with the processing and extraction of various information in the form of reports or dashboards [9]. Moreover, this feature not only gives a 360° view of a customer’s journey in the company, but also helps with decision-making for employees and managers. This quality can be a crucial factor for a successful integration of CRM and BI [10], but it is necessary to see if this is also true in e-commerce. From the standpoint of practical implications, there are punctual overlaps based on e-commerce companies and their client portfolio or field of activity, but assurance is needed that the current research discourse still includes CRM and BI capabilities, as a whole or separately, in the specialty literature. As our systematic review will show, existing research about the overlaps between BI and CRM extends to vast areas, such as Social Media, predictions, sentiment analysis, and even Big Data. This fact may give rise to a new idea that the connection between these two domains is not direct, but mainly possible through other areas of expertise. In order to present the systematic literature review, this paper comprises four further sections. Section 2 describes the systematic review methodology used in the article. Section 3 presents the results of the quantitative analysis and content analysis of the articles deemed suitable, as well as a thematic analysis. The final section presents the conclusions, limitations, and future research directions.
2 Method The main purpose of a systematic review is to find answers to specific questions, based on a particular methodology that strictly considers a search strategy, with inclusion and exclusion criteria for identifying the studies to be included or excluded [11]. The entire systematic review is a mix of other scientific research methods, explained by many researchers, followed by understanding the importance of literature reviews, the organization and synthesis of ideas involved, and the rigor in detailing references and avoiding plagiarism to increase the quality of the finished output [12]. After the extraction of the final set of articles, all of them are coded and analyzed based on their content. This paper maps 83 articles in total on the topic of BI and CRM overlaps. 2.1 Search Strategy The search strategy was first based on an initial search string (see Table 1) and inclusion/exclusion criteria (see Table 2), which only took into consideration open access
Overlaps Between Business Intelligence and Customer Relationship Management
43
articles written in English, published between 2017 and April 2020, indexed in Clarivate Analytics Web of Science, and having a strong connection with BI and CRM domains. All the cover titles, abstracts, keywords, authors, years, and journal names were extracted using Zotero, a free software that automatically senses research on the web. All the articles analyzed in this paper were published in peer-reviewed journals based on their general trustworthiness in academia [13]. Table 1. Initial search string Topic
Search terms
Customer relationship management Marketing OR digital marketing OR customer relationship management OR crm OR Big Data OR customer interaction OR customer journey OR data integration OR contextual marketing OR customer relationship OR omnichannel delivery OR sales prediction OR predictive model OR predictive marketing OR customer value OR customer experience OR customer intelligence OR customer view OR customer engagement OR customer behaviour OR 360° customer view OR 360° AND Business Intelligence
Business Intelligence OR bi OR data visualisation OR reporting OR etl OR reports OR automation OR analytics OR extract transform load
AND Database & e-commerce
Database OR tables OR customer data OR data model OR data warehouse OR data collection OR data captured OR real time OR e-commerce OR virtual retail
2.2 Screening The initial search produced a total of 35,223 articles, all from Web of Science Core Collection, InCites Journal Citation Reports, Derwent Innovations Index, Clarivate Analytics databases; the study sample was then reduced to a total of 83 articles using search strategy features, as explained in the Prisma diagram (see Fig. 1) [14, 15]. One of the limitations was the inability to access all eligible items. Unfortunately, out of 1,986 potential articles, about 73% were not open access, which can be deemed a major obstacle. Nevertheless, after applying the inclusion and exclusion criteria to all the papers, only 543 potential articles were included in the Zotero library. Therefore, after final exclusions, 83 articles remained for synthesis purposes. 2.3 Coding, Data Extraction, and Analysis In order to extract all the data contained in the papers, all such papers were uploaded into VOSviewer, a software tool for constructing and visualizing bibliometric networks,
44
I.-D. Anastasiei and M.-R. Georgescu Table 2. Inclusion and exclusion criteria
Inclusions
Exclusions
Published between 2017 – April 2020 (12,302)
Published before 2017 (22,921)
Written in English (12,069)
Not in English (233)
Indexed in Web of Science (8,923)
Unpublished articles (3,146)
Linked to IT (3,484)
Not linked to IT (5,439)
Linked to CRM or BI (1,986)
Not linked to CRM or BI (1,498)
Open access (543)
Without open access (1.443)
Final articles (83)
Manual exclusions (460)
Fig. 1. PRISMA diagram
which are bibliographically mapped by identification of keyword, by word frequency, and the connection between them. The mapping technique can be used to construct a map based on a co-occurrence matrix. After the inclusion/exclusion analysis, all articles were exported from Zotero in RIS format, which was used as a data source. This tool can not only create networks between the citations, but also develop clusters, mainly based on abstracts, keywords, and citations. All articles were coded based on the clusters, and the descriptive data analysis was carried out using Excel.
Overlaps Between Business Intelligence and Customer Relationship Management
45
Of all the clusters obtained, the Big Data and analytics clusters really stand out because all CRM articles were split between these two. Also, the coding system was derived from the clusters obtained in VOSviewer.
3 Results 3.1 Quantitative Analysis All the articles included in this paper were analyzed based on multiple features that are recognized as useful for a systematic review. All the articles were analyzed per year, number of journals, countries, author affiliations, and domain. Articles Per Year The analysis included studies from early 2017 to April 2020. There is a significant increase of studies, starting from 16 in 2017 to 34 in 2019 and then 10 in April 2020. Journals This paper contains articles from 52 different journals, the largest number of papers being in the IEEE Access Journal (n = 9), which is an open access scientific journal published by the Institute of Electrical and Electronics Engineers. At the same time, 38 journals had only one article each (see Table 3). Table 3. Number of included articles per journal (n = 83) Rank
Journal
1
IEEE Access
9
2
Journal of Retailing and Consumer Services
6
Journal of Intelligence Studies in Business
6
3 4
5
Articles (n)
Information
3
Journal of Business Research
3
Informatics-Basel
2
Computers in Human Behavior
2
Baltic Journal of Economic Studies
2
Expert Systems with Applications
2
Industrial Marketing Management
2
Journal of Theoretical and Applied Electronic Commerce Research
2
Data
2
Big Data
2
Journal of Retailing
2
Other journals
38
46
I.-D. Anastasiei and M.-R. Georgescu
Countries The first author’s country was considered for the geographical analysis. Most articles (n = 11) are from Great Britain. About 50% of the analyzed articles are from the following countries: United Kingdom, United States of America, Australia, Czech Republic, Turkey, and China (Table 4). Table 4. Number of included articles per country (n = 83) Rank
Country
Articles (n)
%
Rank
Country
Articles(n)
1
Great Britain
11
13.25%
7
Syria
2
2.41%
2
USA
8
9.64%
Not mentioned
2
2.41%
3
Australia
6
7.23%
Indonesia
2
2.41%
4
Czech Republic
5
6.02%
India
2
2.41%
5
Turkey
4
4.82%
Germany
2
2.41%
China
4
4.82%
Canada
2
2.41%
6
Ukraine
3
3.61%
Belgium
2
2.41%
Spain
3
3.61% other
19
22.89%
Italy
3
3.61%
France
3
3.61%
8
%
Authors’ Affiliation As in the case of the geographical analysis, here too we considered the first author’s affiliation to a scientific field. In Table 5, the authors’ affiliation is mainly related to Economics and Business Administration (n = 15), followed by Automation and Computers (n = 12), and Information Technology (n = 9). This is defining for the research considered here and can be interpreted as a multidisciplinary foray into the articles we studied. 3.2 Content Analysis VOSviewer automatically identified five working clusters, the first being the largest (see Fig. 2). These clusters were obtained by counting the frequency of words in all 83 articles, using the “no normalization” analysis method, which is the opposite of z-score (standardization). Normalization usually means resizing values within a range of [0, 1], and standardization means revoking the data to have an average of 0 and a standard deviation of 1, i.e. a unit variation [16]. Clusters have different colors depending on their affiliation and the links between them are nothing but reference links.
Overlaps Between Business Intelligence and Customer Relationship Management
47
Table 5. Number of included articles per authors’ affiliation (n = 83) Rank
Journal
Articles (n)
%
1
Not mentioned
21
25.30%
2
Economy and Business Administration
15
18.07%
3
Automatic Control and Computer Engineering
12
14.46%
4
Computer Science
9
10.84%
5
Informational Systems for Businesses
5
6.02%
6
Business Informatics
3
3.61%
Business Law
3
3.61%
7 8
Data Analytics
3
3.61%
Marketing
2
2.41%
Statistics
2
2.41%
Other
8
1.20%
In terms of Web of Science categories, the sample of articles considered in this research paper featured a diversity of disciplines. Instead of a single CRM cluster, all items like Big Data, analytics, data analytics, sentiment analysis, and customer behavior were treated under the same protocol. The same was taken in consideration for predictive analytics and machine learning, but not for artificial intelligence (Figs. 3, 4 and 5).
Fig. 2. Density visualization
Fig. 4. E-commerce cluster connections
Fig. 3. Analytics cluster connections
Fig. 5. BI cluster connections
48
I.-D. Anastasiei and M.-R. Georgescu
The connections between the clusters can be easily observed. E-commerce is linked with analytics and BI via a direct connection. All three domains overlap only in the case of the link with analytics, which assumes that the use of decision support tools employs Big Data obtained from e-commerce, social media and, customer behavior. There is also a direct connection between BI and all the items in the CRM cluster, which is strongly supported by the inclusion of data warehouse in the inclusion/exclusion phase (see Table 1). 3.3 Thematic Analysis Another method used to identify themes and patters within our research was thematic analysis [17–19]. Thematic analysis is used to identify themes, which can later be synthesized into categories, strictly by the content of the selected papers. All 83 articles have been clustered into thematic categories (see Table 6) identified in Subsect. 3.2. This analysis addressed the question of whether there is a place for e-commerce in the overlap between BI and CRM, which determined if a selected article was eligible or not using an inductive method, which consisted of coding and theme development directed by the content of the articles (Table 7). Table 6. Number of included articles per authors’ affiliation (n = 83) Rank Thematic category
Articles (n) %
1
CRM
25
30.12%
2
Business Intelligence 16
19.28%
3
Social media
15
18.07%
4
Predictions
13
15.66%
5
Big Data
10
12.05%
6
Artificial intelligence 4
4.82%
Most articles were included in the CRM thematic category, which was followed by BI. 44 articles stated that there is a place for e-commerce in their content, which is a considerable amount, 19 did not have any mention of e-commerce, and 20 could not be applied to e-commerce.
Thematic category
CRM
Business intelligence
Social media
Predictions
Big data
Artificial intelligence
Rank
1
2
3
4
5
6
4
10
13
15
16
25
Articles (n)
4 [95–98]
5 89, 91–94]
9 74, 76–80, 82, 84, 85]
5 [36, 41, 44, 45, 49]
8 [20, 21, 23, 24, 26, 30, 32, 33]
13 [52, 59– 67, 70, 100, 102]
Yes
Is there a place for e-commerce?
0
1 [99]
2 [73, 83]
7 [38, 39, 40, 42, 46, 48, 72]
4 [22, 25, 31, 34]
5 [53, 54, 56, 69, 71]
No
Table 7. Is there a place for e-commerce as a thematic category (n = 83)
0
4 [86, 87, 88, 90]
2 [75, 81]
3 [37, 43, 47]
4 [27, 28, 29, 35]
7 [50, 51, 55, 57, 58, 68, 101]
Not applicable
Overlaps Between Business Intelligence and Customer Relationship Management 49
50
I.-D. Anastasiei and M.-R. Georgescu
4 Conclusions To conclude the paper, the authors reflected on the initial reason for undertaking the systematic review based on the assumption that there could be a place for e-commerce in the overlaps between BI and CRM technologies. Although the time frame considered for the search only spanned three years and four months, there was a significant number of articles which treated the subject. Based on the findings of the study, there is indeed a place for e-commerce in most scientific research, but not in as great a number as we initially thought. From the total of 83 articles, only 44 authors considered that e-commerce is relevant for their research and 39 considered that e-commerce is not appliable or did not even mention e-commerce in their speech. Authors still do not treat all CRM or BI topics along with the implications of e-commerce, which does not scientifically prove if it is actually a good or a bad thing for the scientific discourse. Furthermore, the research also looked into other domains, such as Social Media, Artificial Intelligence, and Big Data, because they have implications in terms of CRM or BI, which can be interpreted as a multi-disciplinary diversity. This emerging disciplinary diversity opens an even larger discussion for future studies, which may state the authors’ position unclearly or may have altogether neglected the discussion of e-commerce in their respective discipline areas. The relevance of this article for academics and practitioners can be observed from two perspectives. Firstly, the incursions between two or more domains can have a place between for a single questioned technology, but this statement cannot be applied as a rule. In this case, it is not a rule that two articles in the same domain may or may not have a place for e-commerce in the authors’ discourse. Secondly, with the help of software tools specialized in articles review and a research methodology based on search strategy features, presented in the PRISMA diagram, future studies can be conducted in order to respond to the need of inserting other technologies in the authors’ speech or not. The tendency of the place of e-commerce in the BI and CRM incursions remains at the latitude of IT domain development, as there is no certainty if the tools or domains that we have at our disposal today, will continue to exist in the years to come. But one thing is certain, based on the study conducted, in almost all articles related to the domains of Artificial Intelligence and Machine Learning, there is a place for e-commerce. But this does not necessarily means that if AI takes over the incursions between BI and CRM in the future, there will be a place for e-commerce or not.
References 1. Handzic, M., Ozlen, K., Durmic, N.: Improving customer relationship management through business intelligence. J. Inf. Knowl. Manag. 13(2), 1450015 (2014) 2. Kumar, V.: Customer relationship management. In: Wiley International Encyclopedia of Marketing (2010) 3. Lowell, A., Anton, W.: Banking on CRM. Mortgage Bank. 64(5), 2–16 (2004) 4. Chaudhuri, S., Dayal, U., Narasayya, V.: An overview of business intelligence technology. Commun. ACM 54(8), 88–98 (2011) 5. Williams, S., Williams, N.: The business value of business intelligence. Bus. Intell. J. 8, 30–39 (2003)
Overlaps Between Business Intelligence and Customer Relationship Management
51
6. Khan, A., Ehsan, N., Mirza, E., Sarwar, S.Z.: Integration between customer relationship management (CRM) and data warehousing. Procedia Technol. 1, 239–249 (2012) 7. Mishra, S.: Reviewing the literature. In: Methodological Issues in Management Research: Advances, Challenges, and the Way Ahead, pp. 11–25. Emerald Publishing Limited, Bingley (2019) 8. Chen, I., Popovich, K.: Understanding customer relationship management (CRM). Bus. Process. Manag. J. 9(5), 672–688 (2003) 9. Chen, H., Chiang, R.H., Storey, V.C.: Business intelligence and analytics: from big data to big impact. MIS Q. 36, 1165–1188 (2012) 10. Reid, A., Catterall, M.: Hidden data quality problems in CRM implementation. In: Marketing, Technology and Customer Commitment in the New Economy, pp. 184–189. Springer, Cham (2015) 11. Gough, D., Oliver, S., Thomas, J.: An introduction to Systematic Reviews. SAGE, Los Angeles (2017) 12. Mishra, S., Subudi, R.: Methodological Issues in Management Research: Advances, Challenges and the Way Ahead, p. 11. Emerald Group Publishing (2019) 13. Nicholas, D., Watkinson, A., Jamali, H.R., Herman, E., Tenopir, C., Volentine, R., Allard, S., Levine, K.: Peer review: still king in the digital age. Learned Publishing 28(1), 15–21 (2015) 14. Brunton, J., Thomas, J.: Information management in systematic reviews. In: Gough, D., Oliver, S., Thomas, J. (eds.) An Introduction to Systematic Reviews, pp. 83–106. SAGE, London (2012) 15. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G.: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Clin. Res. Ed BMJ 339, b2535 (2009) 16. Muralidharan, K.: A note on transformation, standardization and normalization. Int. J. Oper. Quant. Manag. 9(1 & 2), 116–122 (2012) 17. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006) 18. Ward, V., House, A., Hamer, S.: Developing a framework for transferring knowledge into action: a thematic analysis of the literature. J. Health Serv. Res. Policy 14(3), 156–164 (2009) 19. Heisig, P., Kannan, S.: Knowledge management: does gender matter? A systematic review of literature. J. Knowl. Manag. 24(6), 1315–1342 (2020) 20. Machado, G., Cunha, I., Pereira, A., Oliveira, L.: DOD-ETL: distributed on-demand ETL for near real-time business intelligence. J. Internet Serv. Appl. 10(1), 21 (2019) 21. Guo, B., Liu, Y., Ouyang, Y., Zheng, V., Zhang, D., Yu, Z.: Harnessing the power of the general public for crowdsourced business intelligence: a survey. IEEE Access 7, 26606– 26630 (2019) 22. Calof, J., Richards, G., Santilli, P.: Integration of business intelligence with corporate strategic management. J. Intell. Stud. Bus. 7(3), 62–73 (2017) 23. Karim, M., Cochez, M., Beyan, O., Ahmed, C., Decker, S.: Mining maximal frequent patterns in transactional databases and dynamic data streams: a spark-based approach. Inf. Sci. 432, 278–300 (2018) 24. Venkatraman, S.: A proposed business intelligent framework for recommender systems. Inform.-Basel 4(4), 40 (2017) 25. Gauzelin, S., Bentz, H.: An examination of the impact of business intelligence systems on organizational decision making and performance: the case of France. J. Intell. Stud. Bus. 7(2), 40–50 (2017) 26. Lamrhari, S., Elghazi, H., El Faker, A.: Business intelligence using the fuzzy-Kano model. J. Intell. Stud. Bus. 9(2), 43–58 (2019) 27. Lopes, A., Coelho, A.: Engaged in integrated reporting? Evidence across multiple organizations. Eur. Bus. Rev. 30(4), 398–426 (2018)
52
I.-D. Anastasiei and M.-R. Georgescu
28. Azeroual, O., Saake, G., Abuosba, M.: ETL best practices for data quality checks in RIS databases. Inform.-Basel 6(1), 10 (2019) 29. Arnott, D., Lizama, F., Song, Y.: Patterns of business intelligence systems use in organizations. Decis. Support Syst. 97, 58–68 (2017) 30. Langlois, A., Chauvel, B.: The impact of supply chain management on business intelligence. J. Intell. Stud. Bus. 7(2), 51–61 (2017) 31. Roberts, R., Laramee, R.: Visualising business data: a survey. Information 9(11), 285 (2018) 32. Muley, A., Gudadhe, M.: Synthesizing high-utility patterns from different data sources. Data 3(3), 32 (2018) 33. Yulianto, M., Girsang, A., Rumagit, R.: Business intelligence for social media interaction in the travel industry in Indonesia. J. Intell. Stud. Bus. 8(2), 72–79 (2018) 34. Brkic, L., Mekterovic, I.: A time-constrained algorithm for integration testing in a data warehouse environment. Inf. Technol. Control 47(1), 43976 (2018) 35. Deb Nath, R., Hose, K., Pedersen, T., Romero, O.: SETL: a programmable semantic extracttransform-load framework for semantic data warehouses. Inf. Syst. 68, 17–43 (2017) 36. Misuraca, M., Scepi, G., Spano, M.: A network-based concept extraction for managing customer requests in a social media care context. Int. J. Inf. Manag. 51, 101956 (2020) 37. Can, U., Alatas, B.: A new direction in social network analysis: ONLINE social network analysis problems and applications. Physica A Stat. Mech. Appl. 535, 122372 (2019) 38. Gómez, M., Lopez, C., Molina, A.: An integrated model of social media brand engagement. Comput. Hum. Behav. 96, 196–206 (2019) 39. Chuang, S.: Co-creating social media agility to build strong customer-firm relationships. Ind. Mark. Manag. 84, 202–211 (2020) 40. Keegan, B., Rowley, J.: Evaluation and decision making in social media marketing. Manag. Decis. 55(1), 15–31 (2017) 41. Trivedi, N., Asamoah, D., Doran, D.: Keep the conversations going: engagement-based customer segmentation on online social service platforms. Inf. Syst. Front. 20(2), 239–257 (2018) 42. Pantano, E., Giglio, S., Dennis, C.: Making sense of consumers’ tweets sentiment outcomes for fast fashion retailers through big data analytics. Int. J. Retail Distrib. Manag. 47(9), 915–927 (2019) 43. Jacobson, J., Gruzd, A., Hernández-García, Á.: Social media marketing: who is watching the watchers? J. Retail. Consum. Serv. 53, 101774 (2020). S0969698918307744 44. Baum, D., Spann, M., Füller, J., Thürridl, C.: The impact of social media campaigns on the success of new product introductions. J. Retail. Consum. Serv. 50, 289–297 (2019) 45. de Oliveira, R.T., Indulska, M., Steen, J., Verreynne, M.: Towards a framework for innovation in retailing through social media. J. Retail. Consum. Serv. 54, 101772 (2020). S0969698918302376 46. Sitta, D., Faulkner, M., Stern, P.: What can the brand manager expect from Facebook? Australas. Mark. J. 26(1), 17–22 (2018) 47. Coskun, M., Ozturan, M.: #europehappinessmap: A framework for multi-lingual sentiment analysis via social media big data (a Twitter case study). Information 9(5), 102 (2018) 48. Liu, X., Shin, H., Burns, A.: Examining the impact of luxury brand’s social media marketing on customer engagement: using big data analytics and natural language processing. J. Bus. Res. 125, 815–826 (2019) 49. Semeradova, T., Weinlich, P.: Computer estimation of customer similarity with Facebook lookalikes: advantages and disadvantages of hyper-targeting. IEEE Access 7, 153365– 153377 (2019) 50. Kornienko, G., Chabanenko, M., Leheza, Y.: Assessment of the economic efficiency of it application at enterprises. Baltic J. Econ. Stud. 4(3), 123–132 (2018)
Overlaps Between Business Intelligence and Customer Relationship Management
53
51. de Ruyter, K., Keeling, D., Cox, D.: Customer-supplier relationships in high technology markets 3.0. Ind. Mark. Manag. 79, 94–101 (2019) 52. Luk, C., Choy, K., Lam, H.: Design of an intelligent customer identification model in e-commerce logistics industry. In: Engineering Application of Artificial Intelligence Conference 2018 (Eaaic 2018), vol. 255, UNSP 04003 (2019) 53. Dalla Pozza, I., Goetz, O., Sahut, J.: Implementation effects in the relationship between CRM and its performance. J. Bus. Res. 89, 391–403 (2018) 54. Halunko, V., Kurkova, K., Oksin, V.: Methods of Assessing the efficiency internet marketing communications. Baltic J. Econ. Stud. 4(4), 76–86 (2018) 55. Oskarsdottir, M., Baesens, B., Vanthienen, J.: Profit-based model selection for customer retention using individual customer lifetime values. Big Data 6(1), 53–65 (2018) 56. Kantorova, K., Bachmann, P.: Social customer relationship management and organizational characteristics. Information 9(12), 306 (2018) 57. Haislip, J., Richardson, V.: The effect of Customer Relationship Management systems on firm performance. Int. J. Account. Inf. Syst. 27, 16–29 (2017) 58. Leonidou, L., Aykol, B., Fotiadis, T., Zeriti, A., Christodoulides, P.: The role of exporters’ emotional intelligence in building foreign customer relationships. J. Int. Mark. 27(4), 58–80 (2019) 59. Zerbino, P., Aloini, D., Dulmin, R., Mininno, V.: Big Data-enabled customer relationship management: a holistic approach. Inf. Process. Manag. 54(5), 818–846 (2018) 60. Gil-Gomez, H., Guerola-Navarro, V., Oltra-Badenes, R., Antonio Lozano-Quilis, J.: Customer relationship management: digital transformation and sustainable business model innovation. Econ. Res.-Ekonomska Istrazivanja 33, 2733–2750 (2019) 61. Suel, E., Polak, J.: Development of joint models for channel, store, and travel mode choice: grocery shopping in London. Transp. Res. Part A-Policy Pract. 99, 147–162 (2017) 62. Finsterwalder, J.: A 360-degree view of actor engagement in service co-creation. J. Retail. Consum. Serv. 40, 276–278 (2018) 63. Fidan, H.: Grey relational classification of consumer’s textual evaluations in e-commerce. J. Theor. Appl. Electron. Commer. Res. 15(1), 48–65 (2020) 64. Choi, D., Chung, C., Young, J.: Sustainable online shopping logistics for customer satisfaction and repeat purchasing behavior: evidence from China. Sustainability 11(20), 5626 (2019) 65. Jasek, P., Vrana, L., Sperkova, L., Smutny, Z., Kobulsky, M.: Comparative analysis of selected probabilistic customer lifetime value models in online shopping. J. Bus. Econ. Manag. 20(3), 398–423 (2019) 66. Kakalejcik, L., Bucko, J., Vejacka, M.: Differences in buyer journey between high- and lowvalue customers of e-commerce business. J. Theor. Appl. Electron. Commer. Res. 14(2), 47–58 (2019) 67. Sturley, C., Newing, A., Heppenstall, A.: Evaluating the potential of agent-based modelling to capture consumer grocery retail store choice behaviours. Int. Rev. Retail Distrib. Consum. Res. 28(1), 27–46 (2018) 68. Balabanis, G., Siamagka, N.: Inconsistencies in the behavioural effects of consumer ethnocentrism the role of brand, product category and country of origin. Int. Mark. Rev. 34(2), 166–182 (2017) 69. Birkin, M.: Spatial data analytics of mobility with consumer data. J. Transp. Geogr. 76, 245–253 (2019) 70. Chen, Q., Zhang, M., Zhao, X.: Analysing customer behaviour in mobile app usage. Ind. Manag. Data Syst. 117(2), 425–438 (2017) 71. Chen, Q., Zhang, M., Zhao, X.: Analysing customer behaviour in mobile app usage. Ind. Manag. Data Syst. 117(2), 425–438 (2017)
54
I.-D. Anastasiei and M.-R. Georgescu
72. Wong, T., Chan, H., Lacka, E.: An ANN-based approach of interpreting user-generated comments from social media. Appl. Soft Comput. 52, 1169–1180 (2017) 73. Eletter, S., Yasmin, T., El Refae, G.: Marketing intelligence in the era of big data. TEM J.-Technol. Educ. Manag. Inform. 8(3), 938–944 (2019) 74. Boone, T., Ganeshan, R., Jain, A., Sanders, N.: Forecasting sales in the supply chain: consumer analytics in the big data era. Int. J. Forecast. 35(1), 170–180 (2019) 75. Martin-Martin, E., Riesco, A., Rodriguez, M.: A language-independent technique for assessing tweet success: an experience report. IEEE Access 6, 62384–62395 (2018) 76. Sigg, S., Lagerspetz, E., Peltonen, E., Nurmi, P., Tarkoma, S.: Exploiting usage to predict instantaneous app popularity: trend filters and retention rates. ACM Trans. Web 13(2), 13 (2019) 77. Pavlyshenko, B.: Machine-learning models for sales time series forecasting. Data 4(1), 15 (2019) 78. Lismont, J., Ram, S., Vanthienen, J., Lemahieu, W., Baesens, B.: Predicting interpurchase time in a retail environment using customer-product networks: an empirical study and evaluation. Expert Syst. Appl. 104, 22–32 (2018) 79. Buettner, R.: Predicting user behavior in electronic markets based on personality-mining in large online social networks. Electron. Mark. 27(3), 247–265 (2017) 80. Jasek, P., Vrana, L., Sperkova, L., Smutny, Z., Kobulsky, M.: Predictive performance of customer lifetime value models in e-commerce and the use of non-financial data. Prague Econ. Pap. 28(6), 648–669 (2019) 81. Anagnostopoulos, C., Savva, F., Triantafillou, P.: Scalable aggregation predictive analytics. Appl. Intell. 48(9), 2546–2567 (2018) 82. Bradlow, E., Gangwar, M., Kopalle, P., Voleti, S.: The role of big data and predictive analytics in retailing. J. Retail. 93(1), 79–95 (2017) 83. Bahrami, M., Bozkaya, B., Balcisoy, S.: Using behavioral analytics to predict customer invoice payment. Big Data 8(1), 25–37 (2020) 84. Silva, A., Moro, S., Rita, P., Cortez, P.: Unveiling the features of successful eBay smartphone sellers. J. Retail. Consum. Serv. 43, 311–324 (2018) 85. Alsheikh, S., Shaalan, K., Meziane, F.: Exploring the effects of consumers’ trust: a predictive model for satisfying buyers’ expectations based on sellers’ behavior in the marketplace. IEEE Access 7, 73357–73372 (2019) 86. Dremel, C., Herterich, M., Wulf, J., vom Brocke, J.: Actualizing big data analytics affordances: a revelatory case study. Inf. Manag. 57(1), 103121 (2020) 87. Ghasemaghaei, M., Calic, G.: Assessing the impact of big data on firm innovation performance: big data is not always better data. J. Bus. Res. 108, 147–162 (2020) 88. Seng, J., Ang, K.: Big feature data analytics: split and combine linear discriminant analysis (SC-LDA) for integration towards decision making analytics. IEEE Access 5, 14056–14065 (2017) 89. Seng, J., Ang, K.: Multimodal emotion and sentiment modeling from unstructured big data: challenges, architecture, & techniques. IEEE Access 7, 90982–90998 (2019) 90. Ul Haq, M., Li, Q., Hassan, S.: Text mining techniques to capture facts for cloud computing adoption and big data processing. IEEE Access 7, 162254–162267 (2019) 91. Çalı, S., Balaman, S: ¸ Improved decisions for marketing, supply and purchasing: mining big data through an integration of sentiment analysis and intuitionistic fuzzy multi criteria assessment. Comput. Ind. Eng. 129, 315–332 (2019) 92. Troisi, O., D’Arco, M., Loia, F., Maione, G.: Big data management: the case of Mulino Bianco’s engagement platform for value co-creation. Int. J. Eng. Bus. Manag. 10, 1111–1128 (2018). 1847979018767770 93. Nahili, W., Rezeg, K., Kazar, O.: A new corpus-based convolutional neural network for big data text analytics. J. Intell. Stud. Bus. 9(2), 59–71 (2019)
Overlaps Between Business Intelligence and Customer Relationship Management
55
94. Liu, W., Cao, G., Yin, J.: Bi-level attention model for sentiment analysis of short texts. IEEE Access 7, 119813–119822 (2019) 95. McLean, G., Wilson, A.: Shopping in the digital world: examining customer engagement through augmented reality mobile applications. Comput. Hum. Behav. 101, 210–224 (2019) 96. Basri, W.: Examining the impact of artificial intelligence (AI)-assisted social media marketing on the performance of small and medium enterprises: toward effective business management in the Saudi Arabian context. Int. J. Comput. Intell. Syst. 13(1), 142–152 (2020) 97. Luo, X., Tong, S., Fang, Z., Qu, Z.: Frontiers: machines vs. humans: the impact of artificial intelligence chatbot disclosure on customer purchases. Mark. Sci. 38(6), 937–947 (2019) 98. Souza, K., Seruffo, M., de Mello Jr, H., Souza, D., Vellasco, M.M.B.R.: User experience evaluation using mouse tracking and artificial intelligence. IEEE Access 7, 96506–96515 (2019) 99. Hair, J., Harrison, D., Risher, J.: Marketing research in the 21st century: opportunities and challenges. Revista Brasileira De Marketing 17(5), 666–681 (2018) 100. Grewal, D., Roggeveen, A., Nordfalt, J.: The future of retailing. J. Retail. 93(1), 43836 (2017) 101. Grace, D., Ross, M., King, C.: Brand fidelity: scale development and validation. J. Retail. Consum. Serv. 52, 101908 (2020). UNSP 102. Darsono, J., Susana, E., Prihantono, E., Kasim, E.: Strategic policies for small and medium businesses in marketing through e-commerce. Entrep. Sustain. Issues 7(2), 1230–1245 (2019)
Perception About of COVID-19 Information in Ecuador Abel Suing(B) , Carlos Ortiz-León, and Juan Carlos Maldonado Grupo de investigación en Comunicación y Cultura Audiovisual, Departamento de Ciencias de la Comunicación, Universidad Técnica Particular de Loja, Loja 11-01-608, Ecuador [email protected]
Abstract. The purpose is to find out the public’s perception of the information transmitted by COVID-19 in Ecuador’s conventional social communication media between March and June 2020, since it is the first country in the region to be affected. The questions: Was there a lack of information?, What information would citizens like to receive? and What strategies should the media implement?” are answered through a qualitative methodological approach of a descriptive and relational type. The research techniques are survey, content analysis, interviews with experts and description of digital publications. The conclusion is that there was little transparency and little research. According to those surveyed, the media should not subordinate deontology to immediacy. The urgency of contrast, consistency, multiple sources and educational approaches is shown. Stories require a humane approach and build transmedia stories. What happened in Ecuador is a reference for other realities, for example in neighboring countries of the Andean área. Keywords: COVID-19 · Media · Perception
1 Introduction COVID-19 is the infectious disease caused by the coronavirus discovered in Wuhan, China, in December 2019 [1]. In March 2020, COVID-19 was designated a pandemic by the World Health Organization [2]. Among the measures decreed by governments to contain the spread of the virus were restrictions on the movement of people, teleworking and distance education, and many more. For the success of the regulations, it is necessary to disseminate the measures and encourage changes in citizen behavior, this occurs partly thanks to the messages and information broadcast through social media, digital telephone services and the Internet. In the pandemic, the media “plays a key role in informing the population in a clear and understandable way, and in promoting behaviors so that people can protect their health and that of their loved ones” [3] (p. 1). Governments, citizens and health authorities recognize the importance of the media for dialogue and learning from other experiences. The positive assessment is general, “they are playing an unprecedented role and with a lot of social responsibility based on what they communicate to the world population” [4] (p. 164), in addition, they exercise © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 56–65, 2021. https://doi.org/10.1007/978-3-030-72651-5_6
Perception About of COVID-19 Information in Ecuador
57
a pedagogical role for “health literacy”, they teach “what to do and how to better face the current pandemic” [5] (p. 74). In addition to this pedagogical recognition, the traditional media have their audience trust, they are “a refuge” [6] (p. 12) for several citizens, so it is understandable that they have had tune upsurges during COVID-19 as shown by the figures of television consumption in Latin America during the quarantine [7], the same situation occurred in other countries. In Spain, printed media, broadcasting media and digital media had a significant increase in news production “to advise the population on the adoption of hygiene practices and the enforcement of social isolation” [8] (p. 8). In the face of the positive impression of citizens on traditional media, signs of concern about the immediacy of information published on social networks are emerging. The rush led to the multiplication of incorrect data or the generation of false news, partly because the guidelines for action came late from governments and international agencies [9]. Among the advantages of the informative production in the traditional media are the contextualization, contrast and verification of the facts directly or through the protagonists, deontological characteristics that were thought to be executed by the citizens through the digital media, but this is not the case. In COVID-19, journalistic values have been tested to “detect erroneous information before it can cause irreparable damage” [10] (p. 3) and to avoid the appearance of “opinions of the most disparate characters produced by Infodemy” [11] (p. 53). There is a counterpoint between the urgency of guidance for citizens and scientific analysis, a kind of elementary and learned knowledge that must be conveyed in the media. In the COVID-19 “citizens consider the search for information and the monitoring of news as key activities” [12] (p. 9), and the “scientific community is facing one of its greatest challenges to solve a global health problem” [13] (p. 2). The purpose of this research is to understand the public’s perception of the information transmitted by the traditional media in Ecuador during the months immediately following the confinement, being the first country to be strongly affected in the region. The city of Guayaquil saw its intensive care units overflowed to attend to patients with COVID-19 [14], to the point that “abandoned bodies were found in the streets [others] were lying in their homes without being removed for several days” [15] (p. 3). Following the health crisis in Ecuador, Peru, and Chile, the World Health Organization “in May 2020 declared Latin America as the new epicenter of the pandemic” [16] (p. 6). COVID-19 showed how helpless the health system was [17]. Corruptionimpregnated bad administrations are unable to rise to the challenge of saving lives. The alternative lies in civil society, organizations and institutions that “with knowledge and in contact with their communities, contribute to containment and mitigation” [18] (p. 1339). The region begins the “construction of a new social contract that strengthens trust in the state” [19] (p. 28). Research on coronavirus infection is linked to the importance of “continuing to use the traditional media for that population that does not use the new digital platforms (…) television and the press have previously been identified as the two media of reference in other health crises” [20] (p. 5), but it should also be avoided that over time less is said about this pandemic [21].
58
A. Suing et al.
The research questions are 1) did the media in Ecuador lack information about COVID-19, 2) what information would citizens like to receive from the media in Ecuador about COVID-19, and 3) what strategies should the media in Ecuador implement to improve information about COVID-19?
2 Methodology The methodological scheme is qualitative, descriptive and relational [22, 23] through a survey, content analysis, two interviews with experts and a quantitative linkage of publications in digital media. The descriptive approach is supported by the production of data through “the very spoken or written words of people” [24] (p. 20). The survey was applied to a non-probabilistic sample for convenience due to the availability of participants, then the responses were reviewed through content analysis in order to allow interpretation of texts [25]. The convenience sample optimizes time and provides information “according to the concrete circumstances surrounding both the researcher and the subjects or groups being researched” [26] (p. 124). Through a form on Google (shorturl.at/diyEG), the participation of citizens in terms of social media in Ecuador was consulted. Eighty-two adults from 34 cities in Ecuador responded between May 21 and May 31, 2020. From these numbers, 12% work in household chores, 44 percent are public or private employees, 24 percent are professionals working independently or in business, and 20 percent are students, retirees, or individuals involved in social services. To reference the publications, the Google advanced search tool was used. The “investigation equation” was constructed with the words: “coronavirus OR COVID-19” and was applied to the news and video web pages of 12 media in the Andean countries (Bolivia, Colombia, Ecuador, and Peru), between March 16, 2020, at the beginning of the confinement measures in Ecuador, and June 16, 2020, the closing date of this investigation. The Alexa tool, a consumption and dissemination ranking, was used for media selection. The population data comes from the statistics of the Economic Commission for Latin America, as of June 2020. The semi-structured interviews conducted by the authors of the study are published in the link space “La Academia Opina”. The profiles of the interviewees are: Dr. Jerónimo Rivera [27], Head of the Visual Communication Area of the Universidad de la Sabana Bogotá and Dr. Denis Renó [28], researcher at Universidade Estadual Paulista.
3 Results For 65 out of 82 respondents there was a lack of information regarding COVID-19. The perception of some people is that COVID-19 was not important at the beginning, before the social isolation measures, then during the first weeks it was limited. They also point out that the media transmitted official or single-source information and that it was rarely questioned, which for them is against the principle of objectivity in journalism. Some participants pointed out inconsistencies in the figures given by health authorities about infections and deaths. The information existed, but it was not real; therefore, it generated negative impacts on national and international audience. There was a lack of background
Perception About of COVID-19 Information in Ecuador
59
information and follow-up, but above all, there was a lack of contrast and verification. The requests for truthfulness and transparency are in all the answers. There was a lack of transparency in the positive cases of COVID-19. The media counteract misinformation and seek better audience behavior based on quality information. It is suggested that investigative journalism be carried out to make visible the absences, errors and strengths of the public health system. Several opinions suggest that the media should change their narrative styles so as not to alarm and be clear in their messages. They should continue to investigate, propose more information on social networks and make use of graphics and data. According to those surveyed, the government hid the morbidity figures or gave contradictory data, influenced the seized media, those of private origin that are administered by the State, and increased uncertainty, which is why they preferred international media such as CNN and NTN 24. Citizens believe that there were shortcomings in Ecuador’s media, particularly in health issues. The absences are from protocols and biosafety campaigns, forms of pandemic contagion, and specialized segments of COVID-19. The pandemic demanded new knowledge and the building of behavior to avoid contagion. Among the needs for information is to know about symptoms, to delve into prevention issues, medical treatment and protocols to avoid contagion in rural areas. There is the impression that the media publish more about big cities and not about medium-sized or small ones. Respondents asked to compare data between national and local media to find out what is happening in the country. Ecuador did not disseminate COVID-19 in detail when there were already consequences in Asia, which participants attribute to the fact that the population has acted erratically. For some, the media must have warned about the pandemic. When asked about what information they would like to receive answers to are grouped into: data and statistics 38%, biosecurity 23%, effects on people and society 15%, humanistic vision 12%, and strategic plans 11%. Through data and statistics, the participants refer to real figures, without distortions, about infected people and the health system, they ask for honesty when transmitting the information otherwise the problem will continue to grow. The figures published in the media during the first two months of confinement did not coincide with the data disseminated through social networks. They also suggest using computer graphics, segmenting the data by parishes or neighborhoods, and addressing the impacts on other sectors of society, such as economy. The issues mentioned in biosafety are related to the origins and consequences for people. Regarding scientific evidence, it is requested to know what it is and how COVID19 is spread, after the contagion, what the protocols and treatments for patients will be, and in case of controlling the spread of the virus how the return to daily life will be. From the beginning of the confinement, cases of corruption in the purchase of medicines were known, so another demanded piece of information is to locate the use of public resources. In the humanistic vision category, respondents say the media should present information as sensitively as possible. They stress the importance of teamwork. Saturation of
60
A. Suing et al.
hospitals and high levels of infection have led to family members not identifying their deceased, which opens a window for the media’s service function. In relation to action plans, the answers are grouped in identifying foreign experiences, listening to the solutions that have been applied in other contexts, and finding strategies for the reactivation of production. Finally, the interest in knowing the advances on vaccines is pointed out. The strategies suggested to improve information about COVID-19 in Ecuador’s media revolve around informational aspects, construction of transmedia stories and greater connectivity. The informative strategies refer to: • Reinforce the stories through dialogues with scientists, physicians and experts to identify perspectives. • Direct and reliable connections with national and international health entities. • To receive and contrast official sources. • More journalistic research and follow good practices in pandemic coverage suggested by international organizations. • Contact with journalists from other countries. • Increased frequency of broadcasting of news spaces, varied content, as well as increased use of videos and interviews. • Promote educational content and raise awareness on citizens. The transmedia story strategy starts with implementing digital platforms, online news sites, blogs or applications. In addition, it aims to increase information and images in media and social networks with the intention of eliciting citizens’ opinions. The purpose is to generate content of massive reach in understandable terms to audiences that will complement the stories, even with posters, slides or graphics. Disseminating information through accessible means is a commitment to equity that is protected by international agreements and is reiterated today. If there is no Internet coverage, information from COVID-19 should be located through television, radio, newspapers, etc. Similarly, coordination between public and private, local and regional media is called for. The perceptions of the respondents are contrasted with the testimonies of experts. For Jerónimo Rivera, television has undergone a transformation. The new television is immaterial, it can be seen through devices (YouTube, Netflix, Amazon, Hulu, etc.). The changes in the audiovisual field had to occur in 10 or 15 years, but they were brought forward by the COVID-19, they happened in an accelerated way. Not everyone was prepared to take on this change. The applications for watching television reach different sectors, these platforms that cannot be defined as cinema or television have positioned themselves strongly in the world, even more now with restricted access cinemas. A complete transformation of the audiovisual world is taking place. Zoom, and other applications, is a new more intimate audiovisual narrative. Society is moving from hyperculture to hypoculture. There is a big change when showing audiovisual content that has been going on for a long time, not only because of the pandemic, but it is also because of the user living realistic moments, such as virtual reality, augmented reality, artificial intelligence. The companies that manage to come out stronger will have a time of prosperity because the world needs new contents.
Perception About of COVID-19 Information in Ecuador
61
For Denis Reno, before COVID-19 many thought that the traditional media would disappear, although according to McLuhan nothing disappears, but changes. Now there is a very strong transformation of the old media and an adaptation to the new format. The traditional media survive thanks to the Web, they are going to be transformed and they are not going to disappear. Even the new media is transformed. There is a new reconfiguration where everything is within the same space. Today every person participates from home. News programs stay all day on the channel’s Web pages. The reconfiguration is the only option now available to the media. We must consider the Web as a place for broadcast and not only for exhibition. Journalism has become in terms of importance, a public utility, because people are looking for more and more information, and there has to be a lot of work behind the cameras and the screen. Journalists have to investigate a lot, they have never worked as hard as they do now, either inside or outside the media, to provide good information. Likewise, citizens have never been as transmediatic as they are now, but with the traditional language of television. Languages are mixed in the same space; this will be a great trend for the next years, so if a hypermedia language is not formed, there will be no dialogue. The traditional audiovisual format presupposes to be in the same space, with the freedom of proximity that already exists. The recordings of people together will be replaced by partial recordings, with different cuts, which is a reinvention of the audiovisual. To conclude the presentation of results, Table 1 is placed in order to locate a parameter that complements the perceptions of the respondents. Per capita coverage of news related to COVID-19, between March and June 2020, from the three media outlets with the Table 1. Publication indicators Country
Media
N°. News N°. Videos Population, millions News and videos per capita
Ecuador
El Comercio
18.900
336
El Universo
18.600
1.640
Bolivia
Ecuavisa
652
174
El Deber
14.700
469
Página Siete
10.800
213
10.800
558
Colombia Minuto 30
Los Tiempos
12.600
454
El Tiempo
16.200
2.080
Perú
El Espectador 19.700
3.350
El Comercio
42.700
30.300
RPP
56.900
La República 45.400
17,6
0,002
11,7
0,003
50,9
0,001
33
0,006
2.790 14.500
Source: Media Web sites, information as of June 16, 2020, and CEPAL
62
A. Suing et al.
largest circulation in the Andean countries shows a low rate in Ecuador in relation to Bolivia and Peru.
4 Conclusions The assertion of the Pan-American Health Organization about the fundamental role of the media in the pandemic is shared by the respondents, who consider that the media oppose quality information to the hoaxes that circulate in social networks, but they refer that there was little transparency through inaccurate data, lack of research and lack of verification function that should be fulfilled by professionals and communication companies. According to those surveyed, the media should not subordinate deontology to immediacy because citizens trust them. In Ecuador, households with Internet access represent less than 50%, according to the latest measurement by the Ecuadorian Institute of Statistics [29], which affects the population’s use of conventional media (open-signal television, newspapers, radio). The traditional media, as referred to by Prof. Denis Renó, survive in and thanks to the Web, is transformed and is not going to disappear. The urgency of contrast, consistency, multiple sources and educational approaches to guide the population is evident. Both the responses to the absences and those referring to proposals reiterate the educational role of the media, which also coincides with numeral 16.10 of the Sustainable Development Goals: “Guarantee public access to information and protect fundamental freedoms” [30], which implies communicating open sources of data and ensuring that journalism is available as an essential service of public interest and builds trust in health communications [31]. Professor Denis Reno agrees with the argument of strengthening values, he mentions that journalists have to do a lot of research to offer good information and that journalism was transformed into terms of public utility because citizens are constantly looking for information. It is pivotal to know the understanding of transmedia narratives, those that according to Scolari [32] correspond to stories in which “the story unfolds through multiple media and communication platforms and in which a part of the consumers assumes an active role in that expansion process”. “Citizens have never been as transmediatic as they are now” [28]. Prof. Rivera points out that the changes in the audiovisual industry came earlier than expected, for example, television is trying to embrace the trends of augmented and virtual reality in the generation of new content, this implies being close to the emerging interests of the audiences, but it does not happen only in entertainment but also in information. Respondents proposed to narrate through multimedia resources to elicit opinions and invite dialogue, thus complementing the stories in an exercise of transmediality. As well as Ecuadorians, citizens of other nations sought news and followed the events of COVID-19 as a key activity in their routines [12], but in the face of inconsistent perceptions and lack of verification, they requested “exact” data and statistics; if they did not obtain them from the national media, they preferred international news networks. Respondents’ answers to the first research question, “Did you lack information on COVID-19 in the Ecuadorian media?” are affirmative, the perception is one of absence. The major informative topics demanded are data and statistics, 38% of the participants in the study, biosecurity 23%, information on the impacts on people and society
Perception About of COVID-19 Information in Ecuador
63
15%, focus and humanistic vision 12%, and strategic plans 11%; this is the answer to the second research question: what information would citizens like to receive from the media in Ecuador about COVID-19? Table 1 attempts to contrast the subjectivity of perceptions with an index of publications in the Andean region. The scale compares the per capita amounts of COVID-19related news and videos published in the three most visited media in each Andean country [33]. The results respond to a “search equation” in Google browser for three months, after March 16, 2020. Under a “ceteris paribus” condition, Ecuadorians receive less information through the websites of their reference media than Peruvians or Bolivians, that is, perceptions coincide with this view of media publications. The third research question related to the strategies that should be implemented by Ecuadorian media to improve COVID-19 information finds an answer in the opinions of those surveyed. These are to improve stories with a human focus, proximity to people and their environments, build transmedia stories and greater connectivity. This descriptive work is a first approach to the study of information in the media in COVID-19, both production and consumption of information should be analyzed with quantitative and mixed approaches. What happened in Ecuador serves as a reference for other realities, and the results of this study projected to the Andean region would allow for a framework to locate correlations with other blocks of nations in the world.
References 1. WHO: Preguntas y respuestas sobre la enfermedad por coronavirus (COVID-19). https:// www.who.int/es/emergencies/diseases/novel-coronavirus-2019/advice-for-public/q-a-cor onaviruses. Accessed 07 July 2020 2. WHO: Alocución de apertura del Director General de la OMS en la rueda de prensa sobre la COVID-19 celebrada el 11 de marzo de 2020. https://www.who.int/es/dg/speeches/detail/ who-director-general-s-opening-remarks-at-the-media-briefing-on-COVID-19---11-march2020. Accessed 07 July 2020 3. OPS: COVID-19. Consejos para informar. Guía para periodistas. OPS, Washington, DC, USA (2020) 4. Hernández, A.: El rol de los medios de comunicación en la pandemia del COVID-19 a nivel mundial. In: Vázquez Atochero, A., Cambero Rivero, S. (ed.) Reflexiones desconfinadas para la era posCOVID-19, pp. 161–178. AnthropiQa 2.0 (2020) 5. Romero, J.: Coronavirus superestrella: el impacto del COVID-19 en la sociedad a través de los medios de comunicación. En Varios. Ensayos desconfinados. Ideas de debate para la post pandemia, pp. 71–84. AnthropiQa 2.0 (2020) 6. Xifra, J.: Comunicación corporativa, relaciones públicas y gestión del riesgo reputacional en tiempos del COVID-19. El profesional de la información 29(2), e290220 (2020). https://doi. org/10.3145/epi.2020.mar.20 7. García, G.: Impacto del COVID-19 en el consumo de TV en América Latina. Revista Ateneo (2020) 8. Lázaro-Rodríguez, P., Herrera-Viedma, E.: Noticias sobre COVID-19 y 2019-nCoV en medios de comunicación de España: El papel de los medios digitales en tiempos de confinamiento. El profesional de la información 29(3), e290302 (2020). https://doi.org/10.3145/epi.2020. may.02
64
A. Suing et al.
9. Bagiotto, V.: Sopa de Letras: elucubraciones en torno a la COVID-19 o cómo hacer caldo discursivo con un virus global. Question/Cuestión 1, e276 (2020). https://doi.org/10.24215/ 16696581e276 10. Pérez-Dasilva, J., Meso-Ayerdi, K., Mendiguren-Galdospín, T.: Fake news y coronavirus: detección de los principales actores y tendencias a través del análisis de las conversaciones en Twitter. El profesional de la información 29(3), e290308 (2020). https://doi.org/10.3145/ epi.2020.may.08 11. Sánchez, J.L.: célula de crisis ante el COVID-19. Más poder local 41, 51–53 (2020) 12. Casero-Ripollés, A.: Impact of COVID-19 on the media system. Communicative and democratic consequences of news consumption during the outbreak. El profesional de la información 29(2), e290223 (2020). https://doi.org/10.3145/epi.2020.mar.23 13. Torres-Salinas, D.: Ritmo de crecimiento diario de la producción científica sobre COVID-19. Análisis en bases de datos y repositorios en acceso abierto. El profesional de la información 29(2), e290215 (2020). https://doi.org/10.3145/epi.2020.mar.15 14. Cué, F.: Castigada por el COVID-19, Guayaquil no tiene espacio “ni para vivos ni para muertos”. France 24, 15 April. https://www.france24.com/es/20200414-covid19-ecuador-gua yaquil-castigada-espacio-vivos-muertos. Accessed 10 July 2020 15. Bajaña, I.: Incidencias del COVID-19 en Ecuador. Question/Cuestión 1, e321 (2020). https:// doi.org/10.24215/16696581e321 16. Vassoler-Froelich, I.: Entrevista con Andrei Cambronero Torres: Análisis de la pandemia de COVID-19 desde Costa Rica. Middle Atlantic Rev. Latin Am. Stud. 4(1), 5–15 (2020) 17. Evans, C., Sabucedo, P., Paz, C.: Supporting practice based evidence in the COVID-19 crisis: three researcher-practitioners’ stories. Couns. Psychol. Q. (2020). https://doi.org/10.1080/ 09515070.2020.1779661 18. Torres, I., Sacoto, F.: Localising an asset-based COVID-19 response in Ecuador. Lancet Global Health 395, 1339 (2020). https://doi.org/10.1016/S0140-6736(20)30868-0 19. Giménez, L., Mosqueira, E.: Latinoamérica frente al COVID-19: Un nuevo contrato social. Foreign Aff. Latinoamérica 20(3), 23–28 (2020) 20. Costa-Sánchez, C., López-García, X.: Comunicación y crisis del coronavirus en España. Primeras lecciones. El profesional de la información 29(3), e290304 (2020). https://doi.org/ 10.3145/epi.2020.may.04 21. Romero, J.: Más allá del coronavirus: las pandemias a través de la historia. In: Vázquez Atochero, A., Cambero Rivero, S. (ed.) Reflexiones desconfinadas para la era posCOVID-19, pp. 17–28. AnthropiQa 2.0. (2020) 22. Hernández, R., Fernández, C., Baptista, M.: Metodología de la Investigación, Quinta edición. McGraw-Hill/Interamericana Editores, S.A., México D.F. (2000) 23. Universia: Tipos de Investigación: Descriptiva, Exploratoria y Explicativa. https://noticias. universia.cr/educacion/noticia/2017/09/04/1155475/tipos-investigacion-descriptiva-explor atoria-explicativa.html. Accessed 15 July 2020 24. Taylor, J., Bodgan, R.: Introducción a los métodos cualitativos de investigación. La búsqueda de significados. Paidos Ibérica S.A., Barcelona (1984) 25. Abela, J.: Las técnicas de Análisis de Contenido: Una revisión actualizada. https://mas tor.cl/blog/wp-content/uploads/2018/02/Andreu.-analisis-de-contenido.-34-pags-pdf.pdf. Accessed 20 July 2020 26. Sandoval, C.: Investigación Cualitativa. Programa de especialización en teoría, métodos y técnicas de investigación social. Arfo Editores, Bogota (2002) 27. Rivera, J.: ¿Cómo queda el cine y la TV después del COVID-19? La Academia Opina. https:// www.facebook.com/medialabutpl3/videos/327416228242058. Accessed 25 July 2020 28. Renó, D.: La reconfiguración de los medios de comunicación. La Academia Opina. https:// www.facebook.com/medialabutpl3/videos/2595045924100914. Accessed 28 July 2020
Perception About of COVID-19 Information in Ecuador
65
29. INEC: Tecnologías de la Información y Comunicación. INEC, Ecuador. https://www.ecu adorencifras.gob.ec/documentos/web-inec/Estadisticas_Sociales/TIC/2018/201812_Princi pales_resultados_TIC_Multiproposito.pdf. Accessed 30 July 2020 30. UNESCO: La reunión de la UNESCO promueve el acceso a la información en el marco del programa de desarrollo sostenible. https://es.unesco.org/news/reunion-unesco-promueve-acc eso-informacion-marco-del-programa-desarrollo-sostenible. Accessed 01 Aug 2020 31. Posetti, J., Bontcheva, K.: DESINFODEMIA Disección de las respuestas a la desinformación sobre el COVID-19. UNESCO (2020) 32. Scolari, C.: Narrativas transmedia: Cuando todos los medios cuentan. Deusto, Barcelona (2013) 33. CAN: Somos Comunidad Andina. https://www.comunidadandina.org/Seccion.aspx?id= 189&tipo=QU&title=somos-comunidad-andina. Accessed 10 Aug 2020
Impact of ICT in Rural Areas: Perceptions of Portugal, Spain and Russia João Paulo Pereira1,2(B) 1 Instituto Politécnico de Bragança, Campus de Santa Apolónia, 5300-302 Bragança, Portugal
[email protected] 2 UNIAG (Applied Management Research Unit), Bragança, Portugal
Abstract. Information and communication technology (ICT) is given to its ability to provide greater access to information and communication to the populations and the quality of service provided than the technological backbone required. The rural economy in most countries is regarded as that which requires intervention in order to foster sustainability and development. There have been many studies of both the value and the use of ICTs in rural small and medium-sized enterprises (SMEs), but there is a lack of more deep and comparative studies. To address this need, this paper presents a study that explore how SMEs in rural areas in Portugal (Alto Trás-os-Montes), Spain (Castilla y León) and Russia (Krasnodar) perceived the importance of ICT and identify the current state. Keywords: ICT · SME · Rural areas
1 Introduction The importance of small and medium-sized enterprises (SMEs) today is indisputable for both developed and developing countries [1]. Providing millions of jobs, the institute of small and medium enterprises is the primary means of sustainable industrial and social diversification of society, thus acting as one of the main drivers of economic development in most countries [2]. However, phenomena such as globalization, the internationalization of national markets, the global economic crisis, financial market volatility, reduced investment, rapidly changing consumer demand are putting increasing pressure on SMEs, encouraging them to find ways to survive and develop in today’s business environment. And one of the ways to improve the competitiveness of enterprises is the use of information and communication technologies. Access to ICT improves business efficiency and the global economy in general. However, today the use of ICT in managing business processes of small and medium-sized enterprises in developing countries is rather moderate [3]. The main purpose of creating and introducing new information technologies is to improve the quality and efficiency of organization management, increase productivity, reduce costs for production and management activities, minimize risks, and so on. Most in-demand are new technologies, that allow to solve the greatest number of similar tasks in the complex. They bring tangible financial returns to the creators. However, © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 66–75, 2021. https://doi.org/10.1007/978-3-030-72651-5_7
Impact of ICT in Rural Areas
67
should be kept in mind that the creation and introduction of new technologies, products and innovations in the market require investment. A typical example in modern small and medium businesses shows that the majority of successful enterprises are ready on average to invest in their information technology infrastructure about 1/20 of their working capital, that is, about 5% of revenue per year [2]. This work analyzes the position of the business environment in SMEs in order to determine their economic status and the degree of adoption of ICT and digital skills, as well as their future prospects. The following regions of the studied countries are considered: Alto Trás-os-Montes (Portugal), Castela e Leão (Spain) and Krasnodar region (Russia). From the demographic point of view, the areas analyzed in this study are characterized by low density, population dispersion, aging and migration of young people. The labor market is characterized by a low level of labor force, a shortage of workers with appropriate qualifications, a low level of entrepreneurship and, as a result, a lack of young people. Thus, ICT is considered necessary to improve the functioning of the company, customer service, as well as to increase sales and enter new markets. The economic structure of the analyzed areas is characterized by a high share of the agricultural sector, and also by some services (in particular, personal and social). Industry and construction have an average weight, and both areas are very concentrated in some areas. Most entrepreneurs (65.4%) started their activities at the first discovery of the opportunity for this. 74.2% admitted that there were some difficulties to start their activities. The greatest difficulties were the lack of the necessary rules and procedures (43.1%) and the search for the necessary funds (21.7%). The surveyed companies also noted that in order to promote the development of SMEs, it would be necessary to establish subsidies, implement measures to increase the population, create budget assistance and reduce the volume of bureaucratic documentation.
2 ICT for SMEs in Emerging Economies The adoption of ICT by SMEs provides the ability to quickly access, evaluate, process and distribute large volumes of data and information about business infrastructure [4]. Therefore, only SMEs that use state-of-the-art technology have the opportunity to enter the international market and remain competitive despite the challenges of globalization, liberalization, etc. Other obvious strengths of ICT are [5]: New business model; Access to new markets; New marketing tools, products and communications services; New ways of working relationships; Improved network performance; Teamwork; Automation of the production process; Cost reduction; Improving communication both within the organization and between organizations; and More access to information and data processing. It is safe to believe that small and medium-sized enterprises are the “engine” of the economy today. For example, in 2017, in the EU, SMEs accounted for 99.8% of all European companies and provided over 90 million people with jobs (over 66% of all jobs) [6]. It is obvious that the life of the SMEs is, on the one hand, a rich field of opportunities, and on the other hand a huge amount of difficulties and “traps”. This situation is typical
68
J. P. Pereira
for countries with developed market economies, and for countries with economies in transition. However, in rural areas the difficulties are bigger. Here are some weaknesses and risks. For example, these are safety risks if proper measures are not followed. Also, do not forget about the reputational risks associated with closer interaction of customers in social networks. Improper use of communication technologies and information can lead to poor performance. The key reason for the limited use of most new technologies by small and medium-sized businesses is the uncertainty of entrepreneurs in obtaining benefits, their commitment to outdated work organization principles, which, combined with limited resources and high risks in implementing software, impedes business development.
3 Case Study 3.1 Regions Alto Trás-os-Montes Region (Portugal) The region has good communication infrastructure and internationalization. Based on a network of qualified scientific and technological equipment, northern Portugal has all the prerequisites for rapid response and development. There is a decrease in population, especially in rural areas of this area. A high aging index is noticed here, that is, a high percentage of the population over 65 years of age. Sparse population density and high dispersion is also marked. There are 1,196,102 companies in Portugal, of which 405,518 are in the north. 12,297 (1.0% existing in the country and 1.6% of the northern part) were located in Alto Tâmega. There are 19,198 companies in Trás-os-Montes (1.6% of the country’s territory and 4.7% of the Northern zone). Alto Tâmega and Trás-os-Montes are characterized by a high share of the agricultural sector, 42.5% and 54.5% respectively (INE, 2020). Castela e Leão (Spain) This study combines the following provinces in Spain: Leão, Zamora, Salamanca, Valladolid, Ávila. Together, the selected provinces cover 58.0% of the autonomous community and 68.5% of the population. The provinces are densely populated, 1,661,153 people lived in 2017. However, in recent years there has been a decrease in population, which affects not only the five selected areas, but also the autonomous community as a whole. Between 2013 and 2017, the selected population decreased in the region by 3.7%. Krasnodar Region (Russia) The population of the region is 5,648,254 people for 2019. The region takes the third place among the subjects of the Russian Federation by the number of inhabitants. The population density is 74.83 people/km2 . The urban population is 54.44%. The level of urbanization is lower than the national average (74.48%) (Rosstat, 2019). By January 1, 2016, 426 municipal formations were formed in the region, including: 7 urban districts, 37 municipal districts, 30 urban settlements, 352 rural settlements.
Impact of ICT in Rural Areas
69
3.2 Methodology To collect the data for the Portuguese and Spanish regions we used the results of the COMPETIC project1 (competic-poctep.com), that made 263 surveys in the Spanish region of Castela e Leão (Leão, Zamora, Salamanca, Valladolid, Ávila) and 170 in the Portuguese regions of Alto Tâmega and Terra de Trás-os-Montes. Field work was conducted from 24 September to 16 November 2018. Sample design: random sample by area, type of company and sector. Sampling error in Portugal: ± 7.52% for global data, 95.5% for a confidence level and estimates of equally likely categories (p = q = 50%). In Spain, the sample size was 263 surveys. Sampling error in Spain: ± 6.04% for global data, 95.5% for a confidence level and estimates of equally likely categories (p = q = 50%). The data from Krasnodar region were collected from the site of the Russian Federal State Statistics Service (Rosstat) and from the statistical compendium of the study «Indicators of the digital economy: 2018». The indicators characterizing research and development in the field of ICT, personnel of the digital economy, the activities of the ICT, content and media sectors, the development of telecommunications were reviewed. Statistical data were used reflecting the demand for digital technologies in the business sector and the social sphere. The data from the scientific and practical manual “Information and communication technologies of the South of Russia” were also reviewed. Materials were taken from the Ministry of Communications and Mass Media, Rosstat, the Ministry of Education and Science of Russia, the Ministry of Culture of Russia, the Bank of Russia, OECD, Eurostat, ITU, the United Nations Department for Economic and Social Development, the World Economic Forum. 3.3 Results In Spain, the main goals of using ICT in each area are improving the functioning and improving the quality of customer service. Entering new markets, as well as reducing costs, are of course most important in trade and tourism (62.2% and 43.7%, respectively), the study of new contacts and opportunities is in the service sector (64.7%). In Portugal, much attention is paid to gaining competitive advantages (46.9%). Also, the introduction of ICT is expected to increase market share and sales (36.3% and 44.2%, respectively). In Russia, the introduction of ICT is expected to progress in related industries, such as big data, quantum technologies, robotics, artificial intelligence and neurotechnology, as well as the industrial Internet and wireless technologies [7]. In the distribution of areas by activity (Table 1) it is clear that the largest process is occupied by the industry, which includes: the extractive industries, industrial processing, electricity, gas, steam and air conditioning, water supply, sanitation, waste management and decontamination [7, 8]. The agricultural sector includes activities such as: agriculture, livestock, forestry and fisheries. Trade includes wholesale and retail trade, car and motorcycle repair, transportation and storage, hospitality, information and communications. Business services are 1 COMPETIC Project - Support to entrepreneurs, self-employed and micro-enterprises in rural
areas to create and develop their businesses taking advantage of ICT opportunities (operation 0381_COMPETIC_2_E).
70
J. P. Pereira
also of great value and include: financial and insurance activities, real estate operations, professional, scientific and technical activities, administrative activities and support services. Other services here include: public administration and defense, compulsory social security, sanitation and social services, arts, entertainment and leisure activities, repair of household products, etc (Fig. 1).
Fig. 1. Distribution of areas by type of activity
General Use of ICT Next table shows companies’ responses on the level of ICT utilization. Conclusions: in Spain, more than 50% of companies have an average or higher than average level, but about 30% are still at a rather low level. For 60% of companies in Portugal is characterized by a low or medium level. In Russia, about 70% of companies are at an average and high level of development. Table 1. Level of ICT use in enterprises by region.
All three countries studied have already appreciated the benefits of using ICT and are therefore ready to invest more and more in this area. More than a third of companies
Impact of ICT in Rural Areas
71
determine the level of ICT implementation as low or very low. However, the share of small companies willing to invest in ICT in the next couple of years was 33% in Spain, 53.5% in Portugal and more than 60% in Russia, and this figure grew by 12% over the last year. Of course, these indicators can and should be higher. Results shows the total national use of ICT in companies with less than 10 employees. Consideration of sectors separately gave the following data: in the agricultural sector, the highest percentage of investment probability (41.3%), it is also high for industry (35.5%) and services (32.2%). However, for the trade sector, in which ICT is used only by 50 percent, the probability of investment is no more than 28.5%. Nevertheless, based on statistics, it should be emphasized that those companies, that indicated that they have a higher level of ICT, use give an even greater prospect of investing over the next two years (Table 2). Table 2. Intensity of using digital technologies in organizations by countries.
The use of ICT business management tools and applications is becoming increasingly popular. The first conclusions regarding such applications – 60% do not use any of them. The most implemented applications in the enterprise are: management software (ERP) (40% in Portugal, 46% in Spain and 17% in Russia), CRM systems (37% in Spain, 24% in Portugal and 12% in Russia). Internet Connection Table 3 presents the types of Internet connections. The most popular is DSL. In Russia, communications, hotel business and manufacturing are leading the way in email. However, ICT is underdeveloped in the transport sector. It is worth noting that more than 90% of all trading in the supply of goods, the performance of work and services in Russia takes place on electronic platforms. Almost all surveyed companies have access to the Internet. The reasons why some companies do not use this technology are associated with the cost of maintenance and connection (26.6%, 30.8% and 30.4% respectively) (Table 4). Security Speaking about computer security, which is an important aspect in business, only 4.6% of companies did not take care of protecting both personal and business data. 9 out of 10 companies have implemented at least one security measure. All surveyed companies
72
J. P. Pereira Table 3. Internet access type.
Table 4. Internet services.
are interested in deploying security measures. Table 3 presents the most popular ways of ensuring security in companies in the regions studied. In Spain, 89.2% have already implemented them to some extent. 80.1% have antivirus, 79.5% protection of personal data, 78.9% have an updated operating system, and 67.7% make backup copies regularly. The least implemented measure is security plans and updates. With regard to various sectors of the economy, in the service sector, 95.0% of companies have implemented some security measures. On the other hand, there is a lower percentage in trade and agriculture (82.1% and 88.0%, respectively).The most common measures in various sectors are antivirus and operating system update (Table 5). Interviewed companies in Portugal also attach great importance to security. 93.5% indicates that information is available only to those who are duly authorized, 62.4% the operating system is up to date, and 61.2%, who always use the original software.
Impact of ICT in Rural Areas
73
Table 5. Security practices. CASTELA E LEÃO
ALTO TRÁS-OS-MONTES
KRASNODAR REGION
Technical means of user authentication
72,3%
93,5%
64.4%
Software preventing unauthorized access to malware
15,36%
26,32%
18,5%
Regularly updated antivirus programs
80,1%
50,6%
87.6%
Firewall and spam filter
64,7%
37,1%
57.5%
Computer or network intrusion detection systems
79,5%
40,6%
33.8%
Backing up data to media
67,7%
55,3%
31.2%
Also, about half of the companies indicate that they back up regularly, their antiviruses are reliable and constantly updated. Compared with Portugal and Spain, Russia faces security problems three times less. However, the security surveyed companies do not neglect. About 4% of companies even use biometric user authentication tools [9]. In the transport sector, the highest percentage of antivirus updates (95.9%), about 90% of the use of electronic signatures is allocated to the service and communication sectors. Strong authentication tools are popular in the mining industry (59.1%). However, the largest percentage of backup in the field of communications (44%). Challenges for ICT Implementation The main “brakes” of ICT use in small and medium-sized companies are the following provisions (Table 6). Consider, for example, Internet technology. Almost all surveyed companies have access to the Internet. The reasons why some companies do not use this technology are associated with the cost of maintenance and connection (26.6% and 18.6%, respectively). The study also identified the following reasons: suppliers and customers do not use the Internet, employees spend a lot of time, and there is uncertainty about security on the Internet. Some of the difficulties are internal and related to corporate culture. Any changes in this case require a budget, and many managers simply are not afraid of return on investment. A huge problem is also the lack of sufficiently qualified staff.
74
J. P. Pereira Table 6. Challenges for ICT implementation. CASTELA E LEÃO
ALTO TRÁS-OS-MONTES
KRASNODAR REGION
Lack of technical connectivity
17%
11,3%
7,8%
Lack of skills for work
15,36%
26,32%
18,5%
Lack of government support
10,08%
7,24%
9,7%
High connection costs
25,6%
30,78%
30,4%
These solutions do not meet the needs of the company
11,6%
7,8%
11,3%
For security and privacy reasons
10,68%
8,96%
12,1%
Lack of awareness
8,96%
7,6%
10,2%
4 Conclusion In the Portuguese and Spanish areas exits an improvement of the ICT policies (clearer in Spain), but remains a big lack in human resources. Most companies and entrepreneurs based in the rural areas of the analyzed territories have the perceptions that their level of use of information and communication technologies is low or very low. SMEs and entrepreneurs in the selected rural areas of Portugal and Spain reflect different difficulties in implementing ICT. While in the Spanish provinces the lack of infrastructure is identified as the main obstacle, in Portugal the maintenance costs are indicated first. In both countries, the lack of human capital with ICT knowledge is the second most cited difficulty. Despite the fact that over the past few years there has been an increase in the Russian ICT market, there are a number of factors hindering the development of this market. First, it is the monopolization of the ICT market in Russia. There are alternative solutions in each area of management, and this creates conditions for the effective use of ICT, by speeding up the search for alternatives, by automating the construction of options for combinations of alternatives in production management, marketing, finance and personnel. The small business success formula begins to work at a tremendous rate, but it is always necessary to evaluate your business as part of a networked economy and, with the help of ICT, quickly assess not only direct but also feedback from the decision and external impacts. ICT growth is accelerated by active infrastructure. The following recommendations were formed during the study: a) Not only take into account, but also effectively manage the risks of ICT; b) Develop new rules, procedures and technologies for the work of the authorities and local self-government on the basis of ICT; c) Stimulate entrepreneurship (services such as: installation and maintenance of ICT, graphic design, financial advice, legal advice for companies, etc.); d) Promote ICT related technologies; e) Conduct incentive programs, establish subsidies and introduce motivational programs; f) Develop
Impact of ICT in Rural Areas
75
opportunities for remote work; g) Economically support specialized training based on technology startups; h) Provide economic assistance and reduce bureaucratic documentation; i) Create more jobs in higher schools offering ICT-related courses in information management; j) Make efforts to raise the awareness of companies and entrepreneurs about ICT opportunities; k) Publish a list of various ICT-related technologies so that each company can choose those that are or may be more useful in its activities. Acknowledgments. Projeto COMPETIC (0381_COMPETIC_2_E) - Co-financiado pelo Fundo Europeu de Desenvolvimento Regional (FEDER) através do Programa Interreg V-A EspanhaPortugal (POCTEP) 2014–2020. UNIAG, R&D unit funded by the FCT – Portuguese Foundation for the Development of Science and Technology, Ministry of Science, Technology and Higher Education. Project no. UIDB/04752/2020.
References 1. Ongori, H.: Information and communication technologies adoption in SMEs. J. Chin. Entrep. 2, 93–104 (2010) 2. Balocco, R., Mogre, R., Toletti, G.: Mobile internet and SMEs: a focus on the adoption. Ind. Manag. Data Syst. 109, 245–261 (2009) 3. Chong, S.: Business process management for SMEs: an exploratory study of implementation factors for the Australian wine industry. J. Inf. Syst. Small Bus. 1, 41–58 (2009) 4. Azam, M.S.: Diffusion of ICT and SME performance. Adv. Bus. Mark. Purch. 23A, 7–290 (2015) 5. Kotelnikov, V.: Information and communication technologies for small and medium-sized enterprises. UN-APCICT (2018) 6. Katua, N.T.: The role of SMEs in employment creation and economic growth in. Int. J. Educ. Res. 2, 20–28 (2014) 7. Abdrakhmanova, G., Vishnevskiy, K., Volkova, G.: Digital Economy Indicators in the Russian Federation. Data Book, National Research University Higher School of Economics. HSE, Moscow (2018) 8. I. V.-A. S.-P. P. (POCTEP). The COMPETIC Project (2014) 9. Abdrakhmanova, G.I., Gokhberg, L.M., Kovaleva, G.G.: Information and Communication Technology by the Population in Russia. National Research University Higher School of Economics (2015)
AMALGAM: A Matching Approach to Fairfy TabuLar Data with KnowledGe GrAph Model Rabia Azzi(B) and Gayo Diallo BPH Center/INSERM U1219, Univ. Bordeaux, 33000 Bordeaux, France {rabia.azzi,gayo.diallo}@u-bordeaux.fr
Abstract. In this paper we present AMALGAM, a matching approach to fairify tabular data with the use of a knowledge graph. The ultimate goal is to provide fast and efficient approach to annotate tabular data with entities from a background knowledge. The approach combines lookup and filtering services combined with text pre-processing techniques. Experiments conducted in the context of the 2020 Semantic Web Challenge on Tabular Data to Knowledge Graph Matching with both Column Type Annotation and Cell Type Annotation tasks showed promising results.
Keywords: Tabular data
1
· Knowledge graph · Entity linking
Introduction
Making web data complying with the FAIR1 principles has become a necessity in order to facilitate their discovery and reuse [1]. The value for the knowledge discovery of implementing FAIR is to increase, data integration, data cleaning, data mining, machine learning and knowledge discovery tasks. Successfully implemented FAIR principles will improve the value of data by making them findable, accessible and resolve semantic ambiguities. Good data management is not a goal in itself, but rather is the key conduit leading to knowledge discovery and acquisition, and to subsequent data and knowledge integration and reuse by the community after the data publication process [2]. Semantic annotation could be considered as a particular knowledge acquisition task [3–5]. The semantic annotation process may rely on formal metadata resources described with an Ontology, even sometimes with multiple ontologies thanks to the use of semantic repositories [6]. Over the last years, tables are one of the most used formats to share results and data. In this field, a set of systems for matching web tables to knowledge bases has been developed [7,8]. They can be categorized in two main tasks: structure and semantic annotation. The structure annotation deals with tasks such as data type prediction and table header 1
FAIR stands for: Findable, Accessible, Interoperable and Reusable.
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 76–86, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_8
AMALGAM
77
annotation [9]. Semantic annotation involves matching table elements into KG [10] e.g., columns to class and cells to entities [11,12]. Recent years have seen an increasing number of works on Semantic Table Interpretation. In this context, SemTab 20202 has emerged as an initiative which aims at benchmarking systems which deals with annotating tabular data with entities from a KG, referred as table annotation [13]. SemTab is organised into three tasks, each one with several evaluation rounds. For the 2020 edition for instance, it involves: (i) assigning a semantic type (e.g., a KG class) to a column (CTA); (ii) matching a cell to a KG entity (CEA); (iii) assigning a KG property to the relationship between two columns (CPA). Our goal is to automatically annotate on the fly tabular data. Thus, our annotation approach is fully automated, as it does not need prior information regarding entities, or metadata standards. It is fast and easy to deploy, as it takes advantage of the existing system like Wikidata and Wikipedia to access entities.
2
Related Work
Various research works have addressed the issue of semantic table annotation. The most popular approaches which deal with the three above mentioned tasks rely on supervised learning setting, where candidate entities are selected by a classification models [14]. Such systems include (i) MTab [15], which combines a voting algorithm and the probability models to solve critical problems of the matching tasks, (ii) DAGOBAH [16] aiming at semantically annotating tables with Wikidata and DBpedia entities; more precisely it performs cell and column annotation and relationship identification, via a pipeline starting from a pre-processing step to enriching an existing knowledge graph using the table information; (iii) ADOG [17] is a system focused on leveraging the structure of a well-connected ontology graph which is extracted from different Knowledge Graphs to annotate structured or semi-structured data. In the latter approach, they combine in novel ways a set of existing technologies and algorithms to automatically annotate structured and semi-structured records. It takes advantage of the native graph structure of ontologies to build a well-connected network on ontologies from different sources; (iv) Another example is described in [18]. Its process is split into a Candidate Generation and a Candidate Selection phases. The former involves looking for relevant entities in knowledge bases, while the latter involves picking the top candidate using various techniques such as heuristics (the ‘TF-IDF’ approach) and machine learning (the Neural Network Ranking model). In [19] the authors present TableMiner, a learning approach for a semantic table interpretation. This is essentially done by improving annotation accuracy by making innovative use of various types of contextual information both inside
2
http://www.cs.ox.ac.uk/isg/challenges/sem-tab/2020/index.html.
78
R. Azzi and G. Diallo
and outside tables as features for inference. Then, it reduces computational overheads by adopting an incremental, bootstrapping approach that starts by creating preliminary and partial annotations of a table using ‘sample’ data in the table, then using the outcome as ‘seed’ to guide interpretation of remaining contents. Following also a machine learning approach, [20] proposes Meimei. It combines a latent probabilistic model with multi-label classifiers. Other alternative approaches address only a single specific task. Thus, in the work of [21], the authors focuses on column type prediction for tables without any metadata. Unlike traditional lexical matching-based methods, they follow a deep prediction model that can fully exploit tables’ contextual semantics, including table locality features learned by a Hybrid Neural Network (HNN), and intercolumn semantics features learned by a knowledge base (KB) lookup and query answering algorithm. It exhibits good performance not only on individual table sets, but also when transferring from one table set to another. In the same vein, a work conducted by [22] propose Taipan, which is able to recover the semantics of tables by identifying subject columns using a combination of structural and semantic features. From Web tables point of view, various works could be mentioned. Thus, in [23] an iterative matching approach is described. It combines both schema and entity matching and is dedicated to matching large set of HTML tables with a cross-domain knowledge base. Similarly, TabEL uses a collective classification technique to disambiguate all mentions in a given table [24]. Instead of using a strict mapping of types and relations into a reference knowledge base, TabEL uses soft constraints in its graphical model to sidestep errors introduced by an incomplete or noisy KB. It outperforms previous work on multiple datasets. Overall, all the above mentioned approaches are based on a learning strategy. However, for the real-time application, there is a need to get the result as fast as possible. Another main limitation of these approaches is their reproducibility. Indeed, key explicit information concerning study parameters (particularly randomization control) and software environment are lacking. The ultimate goal with AMALGAM, which could be categorized as a tabular data to KG matching system, is to provide a fast and efficient approach for tabular data to KG matching task.
3
The AMALGAM Approach
AMALGAM is designed according to the workflow in Fig. 1. There are three main phases which consist in, respectively, pre-processing, context annotation and tabular data to KG matching. The first two steps are identical for both CEA and CTA tasks. Tables Pre-Processing. It is common to have missing values in datasets. Beside, the content of the table can have different types (string, date, float, etc.) The aim of the pre-processing step is to ensure that loading table happens without any error. For instance, a textual encoding where some characters are loaded as noisy sequences or a text field with an unescaped delimiter will cause
AMALGAM
79
Fig. 1. Workflow of AMALGAM.
the considered record to have an extra column, etc. Loading incorrect encoding might strongly affect the lookup performance. To overcome this issue, AMALGAM relies on the Pandas library3 to fix all noisy textual data in the tables being processed.
Fig. 2. Illustration of a table structure.
Annotation Context. We consider a table as a two-dimensional tabular structure (see Fig. 2(A)) which is composed of an ordered set of x rows and y columns. Each intersection between a row and a column determines a cell cij with the value vij where 1 ≤ i ≤ x, 1 ≤ j ≤ y. To identify the attribute label of a column also referred as header detection (CTA task), the idea consists in annotating all the items of the column using entity linking. Then, the attribute label is estimated using a random entity linking. The annotation context is represented by the list of items in the same column (see Fig. 2(B)). For example, the context of the first column in the Fig. 2 is represented by the following items: [A1,B1,...,n]. Following the same logic, we consider that all cells in the same row describe the same context. More specifically, the first cell of the row describes the entity and the following cells the associated properties. For instance, the context of the first row in the Fig. 2 is represented by the following list of items: [A1,A2,A3,A4 ]. Assigning a Semantic Type to a Column (CTA). The CTA task consists in assigning a Wikidata KG entity to a given column. It can be performed by exploiting the process described in Fig. 3. The Wikidata KG allows to look up a Wikidata item by its title of its corresponding page on Wikipedia or other Wikimedia family sites using a dedicated API4 . In our case, the main information needed from the entity is a list of the instances of (P31), subclass of (P279) and part of (P361) statements. To do so, a parser is developed to retrieve this 3 4
https://pandas.pydata.org/. https://www.wikidata.org/w/api.php.
80
R. Azzi and G. Diallo
information from the Wikidata built request. For example, “Grande Prairie” provides the following results: [list of towns in Alberta:Q15219391, village in Alberta:Q6644696, city in Alberta:Q55440238]. To achieve this, our methodology combines wbsearchentities and parse actions provided by the API. It could be observed that in this task, there were many items that have not been annotated. This is because tables contain incorrectly spelled terms. Therefore, before implementing the other tasks, a spell check component is required. As per the literature [25], spell-checker is a crucial language tool of natural language processing (NLP) which is used in applications like information extraction, proofreading, information retrieval, social media and search engines. In our case, we compared several approaches and libraries: Textblob5 , Spark NLP6 , Gurunudi7 , Wikipedia api8 , Pyspellchecker9 , Serpapi10 . A comparison of these approaches could be found in Table 1. Table 1. Comparative of approaches and libraries related to spell-checking. Name
Category
Strengths/Limitations
Textblob
NLP
Spelling correction, Easy-to-use
Spark NLP
NLP
Pre-trained, Text analysis
Gurunudi
NLP
Pre-trained, Text analysis, Easy-to-use
Wikipedia api Search engines Search/suggestion, Easy-to-use, Unlimited access Pyspellchecker Spell checking Simple algorithm, No pre-trained, Easy-to-use Serpapi
Search engines Limited access for free
Fig. 3. Assigning a semantic type to a column (CTA).
5 6 7 8 9 10
https://textblob.readthedocs.io/en/dev/. https://nlp.johnsnowlabs.com/. https://github.com/guruyuga/gurunudi. https://wikipedia.readthedocs.io/en/latest/code.html. https://github.com/barrust/pyspellchecker. https://serpapi.com/spell-check.
AMALGAM
81
Our choice is oriented towards Gurunudi and the Wikidata API with a postprocessing step consisting in validating the output using fuzzywuzzy11 to keep only the results whose ratio is greater than the threshold of 90%. For example, let’s take the expression “St Peter’s Seminarz” after using the Wikidata API we get “St Peter’s seminary” and the ratio of fuzzy string matching is 95%. We are now able to perform the CTA task. In the trivial case, the result of an item lookup is equal a single record. The best matching entity is chosen as a result. In the other cases, where the result is more than one, no annotation is produced for the CTA task. Finally, if there is no result after the lookup, another one is performed using the output of the spell check produced by the item. At the end of these lookups, the matched couple results are then stored in a nested dictionary [item:claims]. The most relevant candidate, counting the number of occurrences, is selected. Algorithm 1: CTA task Input: T able T Output: Annotated T able T foreach col i ∈ T do candidates col ← ∅ foreach el ∈ col do label ← el.value candidates ← wd-lookup (label) if (candidates.size = 1) then candidates col(k, candidates) else if (candidates.size = 0) then new label ← spell-check (label) candidates ← wd-lookup (new label) if (candidates.size = 1) then candidates col(k, candidates) end end end annotate(T .col.i, getM ostCommunClass(candidates col)) end
Matching a Cell to a KG Entity (CEA). The CEA task can be performed by exploiting the process described in Fig. 4. Our approach reuse the process of the CTA task and made necessary adaptations. The first step is to get all the statements for the first item of the list context. The process is the same as CTA, the only difference is where result provides than one record. In this case, we create nested dictionary with all candidates. Then, to disambiguate the candidates entities, we use the concept of the column generated with the CTA task. Next, a lookup is performed by using the other items of the list context in the claims of the first item. If the item is found, it is selected as the target 11
https://github.com/seatgeek/fuzzywuzzy.
82
R. Azzi and G. Diallo
entity; if not, the lookup is performed with the item using the Wikidata API (if the result is empty, no annotation is produced). With this process, it is possible to reduce errors associated with the lookup. Let’s take the value “650” in row 0 of the table Fig. 4 for instance. If we lookup directly in Wikidata, we can get many results. However, if we check first in the statements of the first item of the list, “Grande Prairie”, it is more likely to successfully identify the item.
Fig. 4. Matching a cell to a KG entity (CEA).
Algorithm 2: Algorithm of CEA processing task Input: T able T , T ColsContext Output: Annotated T able T foreach row i ∈ T do F irstEl properties ← ∅ foreach el ∈ row do label ← el.value if (el = 0) then F irstEl properties ← GetP roperties(label, ColsContext) end if (Prop-lookup (label) = ∅) then annotate(T .row.i.el, candidates.value) else candidates ← wd-lookup (label,ColsContext) if (candidates.size = 1) then annotate(T .row.i.el, candidates.value) else if (candidates.size = 0) then new label ← spell-check (label) candidates ← wd-lookup (new label,ColsContext) if (candidates.size = 1) then annotate(T .row.i.el, candidates.value) end end end end end
AMALGAM
4
83
Experimental Results
The evaluation of AMALGAM is done in the context of the SemTab 2020 challenge12 . This challenge is subdivided into 4 successive rounds containing respectively 34294, 12173, 62614 and 22387 CSV tables to annotate. For example, Table 2, lists all Alberta towns with additional information such as the country and the elevation above sea level. The evaluation metrics are respectively the F1 score and the Precision [26]. Table 2. List of Alberta towns, extracted from SemTab Round 1. col0
col4
col5
Grande Prairie City in Alberta
col1
col2
Canada Sexsmith
col3
650
Alberta
Sundre
town in Alberta
Canada Mountain View County
1093 Alberta
Peace River
Town in clberta
Canada Northern Sunrise County 330
Vegreville
Town in Alberta canada
Mundare
635
Alberta Alberta
Tables 3, 4, 5 and 6 report the evaluation of CTA and CEA respectively for round 1, 2, 3 and 4. Thus, it could be observed that AMALGAM handles properly the two tasks, in particular in the CEA task. Regarding the CTA task, these results can be explained with a new revision of Wikidata created in the item revision history and there are possibly spelling errors in the contents of the tables. For instance, “rural district of Lower Saxony” became “district of Lower Saxony” after the 16th April 2020 revision. A possible solution to this issue is to retrieve the history of the different revisions, by parsing Wikidata data history dumps, to use them in the lookup. This is a possible extension to this work. Another observed issue is that spelling errors impacts greatly the lookup efficiency.
12
Table 3. Results of Round 1.
Table 4. Results of Round 2.
TASK F1 Score Precision
TASK F1 Score Precision
CTA
0.724
0.727
CTA
0.926
0.928
CEA
0.913
0.914
CEA
0.921
0.927
Table 5. Results of Round 3.
Table 6. Results of Round 4.
TASK F1 Score Precision
TASK F1 Score Precision
CTA
0.869
0.873
CTA
0.858
0.861
CEA
0.877
0.892
CEA
0.892
0.914
http://www.cs.ox.ac.uk/isg/challenges/sem-tab/2020/index.html.
84
R. Azzi and G. Diallo
From the round 1 experience, we specifically focused on the spell check process of items to improve the results of the CEA and CTA tasks in round 2. Two API services, from Wikipedia and Gurunudi (presented in Sect. 3.) respectively were used for spelling correction. According to the results in Table 4, both F1Score and Precision have been improved. From these rounds, we observed that term with a single word is often ambiguous as it may refer to more than one entity. In Wikidata, there is only one article (one entry) for each concept. However, there can be many equivalent titles for a concept due to the existence of synonyms, etc. These synonymy and ambiguity issues make it difficult to match the correct item. For example, the term “Paris” may refer to various concepts such as “the capital and largest city of France”, “son of Priam, king of Troy”, “county seat of Lamar County, Texas, United States”. This leads us to introduce a disambiguation process during rounds 3 and 4. For these two last rounds, we have updated the annotation algorithm by integrating the concept of the column obtained during the CTA task in the linking phase. We showed that the two tasks can be performed relatively successfully with AMALGAM, achieving higher than 0.86 in precision. However, the automated disambiguation process of items proved to be a more challenging task.
5
Conclusion and Future Works
In this paper, we described AMALGAM, a matching approach to enabling tabular datasets to be FAIR compliant by making them explicit thanks to their annotation using a knowledge graph, in our case Wikidata. Its advantage is that it allows to perform both CTA and CEA tasks in a timely manner. These tasks can be accomplished through the combination of lookup services and a spell check techniques quickly. The results achieved in the context of the SemTab 2020 challenge show that it handles table annotation tasks with a promising performance. Our findings suggest that the matching process is very sensitive to errors in spelling. Thus, as of future work, an improved spell checking techniques will be investigated. Further, to process such errors the contextual based spell-checkers are needed. Often the string is very close in spelling, but context could help reveal which word makes the most sense. Moreover, the approach will be improved through finding a trade-off between effectiveness and efficiency.
References 1. Wilkinson, M., Dumontier, M., Aalbersberg, I., et al.: The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) 2. Wilkinson, M.-D., et all.: The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3(1), 1–9 (2016) 3. Diallo, G., Simonet, M., Simonet, A.: An approach to automatic ontology-based annotation of biomedical texts. In: Ali, M., Dapoigny R. (eds) Advances in Applied Artificial Intelligence. IEA/AIE 2006. Lecture Notes in Computer Science, vol. 4031. Springer, Heidelberg (2006)
AMALGAM
85
4. Dram´e, K., Mougin F., Diallo, G. Large scale biomedical texts classification: a kNN and an ESA-based approaches. J. Biomed. Semantics. 7(40) (2016) 5. Handschuh, S.: Semantic Annotation of Resources in the Semantic Web, pp. 135– 155. Semantic Web Services. Springer, Heidelberg (2007) 6. Diallo, G.: Efficient building of local repository of distributed ontologie. In: 7th IEEE International Conference on SITIS, Dijon, pp. 159-166 (2011) 7. Subramanian, A., Srinivasa, S.: Semantic interpretation and integration of open data tables. In: Geospatial Infrastructure, Applications and Technologies: India Case Studies, pp. 217–233. Springer, Singapore (2018) 8. Taheriyan, M., Knoblock, C.-A., Szekely, P., Ambite, J.-L.: Learning the semantics of structured data sources. Web Semantics: Sci. Serv. Agents World Wide Web 37(38), 152–169 (2016) 9. Zhang, L., Wang, T., Liu, Y., Duan, Q.: A semi-structured information semantic annotation method for Web pages. Neur. Comput. Appl. 32(11), 6491–6501 (2019) 10. Ji, S., Pan, S., Cambria, E., Marttinen, P., Yu, P.S.: A survey on knowledge graphs: Representation, acquisition and applications. CoRRabs/2002.00388(2020) 11. Efthymiou, V., Hassanzadeh, O., Rodriguez-Muro, M., Christophides, V.: Matching Web Tables with Knowledge Base Entities: From Entity Lookups to Entity Embeddings. In: LNCS, pp. 260–277. Springer Int. Publishing (2017) 12. Eslahi, Y., Bhardwaj, A., Rosso, P., Stockinger, K., Cudre-Mauroux, P.: Annotating web tables through knowledge bases: a context-based approach. In: 2020 7th Swiss Conference on Data Science (SDS), pp. 29–34. IEEE (2020) 13. Hassanzadeh, O., Efthymiou, V., Chen, C., Jimenez-Ruiz, E., Srinivas, K.: SemTab2020: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching - 2020 Data Sets, October 2020 14. Hassanzadeh, O., Efthymiou, V., Chen, C., Jimenez-Ruiz, E., Srinivas, K.: SemTab2019: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching - 2019 Data Sets (Version 2019) 15. Nguyen, P., Kertkeidkachorn, N., Ichise, R., Takeda, H.: MTab: matching tabular data to knowledge graph using probability models. In: Proceedings of the SemTab Challenge Co-located with the 18th ISWC Conference (2019) 16. Chabot, Y., Labbe, T., Liu, J., Troncy, R.: DAGOBAH: an end-to-end context-free tabular data semantic annotation system. In: Proceedings of the SemTab Challenge Co-located with the 18th ISWC Conference (2019) 17. Oliveira, D., Aquin, M.: ADOG-annotating data with ontologies and graphs. In: Proceedings of the SemTab Challenge co-located with the 18th ISWC Conference (2019) 18. Thawani, A., Hu, M., Hu, E., Zafar, H., Divvala, N-.T., Singh, A., Qasemi, E., Szekely, P., Pujara, J.: Entity Linking to Knowledge Graphs to Infer Column Types and Properties. In: Proceedings of the SemTab Challenge co-located with ISWC’19 (2019) 19. Zhang, Z.: Effective and efficient Semantic Table Interpretation using TableMiner+.Semantic Web IOS Press 8(6), 921—957 (2017) 20. Takeoka, K., Oyamada, M., Nakadai, S., Okadome, T.: Meimei: an efficient probabilistic approach for semantically annotating tables. In: Proceedings of the AAAI Conference on Artificial Intelligenc, vol. 33, pp. 281–288 (2019) 21. Chen, J., Jimenez-Ruiz, E., Horrocks, I., Sutton, C.: Learning semantic annotations for tabular data. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI-19), pp. 2088–2094 (2019)
86
R. Azzi and G. Diallo
22. Ermilov, I., Ngomo, AC.N.: TAIPAN: automatic property mapping for tabular data. In: Blomqvist, E., Ciancarini, P., Poggi, F., Vitali, F. (eds) Knowledge Engineering and Knowledge Management. EKAW 2016. Lecture Notes in Computer Science, vol. 10024. Springer, Cham (2016) 23. Ritze, D., Lehmberg, O., Bizer, C.: Matching HTML Tables to DBpedia. In: Proceedings of the 5th International Conference on Web Intelligence, Mining and Semantics - WIMS ’15, pp. 1—6. ACM Press (2015) 24. Bhagavatula, C.-S., Noraset, T., Downey, D.: TabEL: entity linking in web tables. In: Proceedings of the The Semantic Web - ISWC 2015, Springer International Publishing, pp. 425–441 (2015) 25. Shashank, S., Shailendra, S.: Systematic review of spell-checkers for highly inflectional languages. Artif. Intell. Rev. 53(6), 4051–4092 (2019) 26. Jim´enez-Ruiz, E., Hassanzadeh, O., Efthymiou, V., Chen, J., Srinivas, K.: SemTab 2019: resources to benchmark tabular data to knowledge graph matching systems. In: Harth, A., et al. (eds) The Semantic Web. ESWC (2020)
Knowledge in Transition in an Industrial Company Maria José Sousa1 , Miguel Sousa2 , and Álvaro Rocha3(B) 1 Instituto Universitário de Lisboa, Lisbon, Portugal
[email protected] 2 University of Essex, Colchester, UK 3 ISEG, Universidade de Lisboa, Lisbon, Portugal [email protected]
Abstract. This research examines knowledge transition in an industrial company. This study presents findings about methods and forms of interaction and knowledge transition between organizational actors in innovation processes. Methodology is qualitative and quantitative, as the data was collected through interviews and questionnaires techniques. This study contributes to the body of knowledge about knowledge transition in innovation processes, and empirically presents the impacts of knowledge transition in several dimensions of the organization activity. Finally, the study provides directions for avenues of future research, and suggests some research questions arising out of these findings that might be explored. Keywords: Knowledge in transition · Industry · Case study · Innovation · Organization
1 Introduction Knowledge can be an enabler or a disabler of innovation [1, 2] because individual knowledge transition and use is a very complex social interaction process [3, 4]. To Davenport and Prusak [7] “knowledge is a fluid mix of framed experience, values, contextual information, and expert insight that provides a framework for evaluating and incorporating new experiences and information”. Other reference authors like Polanyi [8] associate knowledge to action. He says that “knowledge is the ability to act”. Nonaka and Takeuchi [9] explain that knowledge is created by the flow of information associated with the beliefs and commitment of those who possess it. In the view of Nonaka and Takeuchi [9], knowledge is created within the company to make it more successful, to keep it on the market, to increase competitiveness and to keep it ahead of its rivals. Knowledge produced and carried by individuals only reaches its full potential to create economic value when it is embodied in organisational routines, that is, when it has been converted into organisational knowledge. In this context knowledge transition in organizations [6] is currently based on information technology rather than in developing social relationships. However, it is needed a cultural and organisational transformation to promote knowledge transfer among © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 87–95, 2021. https://doi.org/10.1007/978-3-030-72651-5_9
88
M. J. Sousa et al.
employees. Knowledge is needed to reorganize work routines and to be embed into new products and services, leading to sustained competitive advantage of organizations. However, this kind of knowledge is carried in the heads of individuals and the dilemma is how it can be embedded in organisational routines to fully maximize its utility – in this regard knowledge transition mechanisms and tools needs to be developed. Nevertheless, information technologies are part of the essential infrastructure of knowledge transition, but it is not sufficient because knowledge involves thinking, an activity that only human beings are able to do. Extensive literature provides several examples of organisations skilful at knowledge transition and share [19], but most of these case studies do not fully explore why these organisations were successful at this endeavour. To fully understand how to grow this capability, it is probably necessary to understand what factors tend to affect knowledge transition. The literature within the knowledge domain provided the following five factors that might influence that process: 1. Relational channels, frequency, and depth of two-way human-to-human contact [11] 2. Partner similarity, degree of similarity (i.e., interests, background, or education) between individuals [12, 13] 3. Depreciation, loss of knowledge after the share [14, 15] 4. Organisational self-knowledge, what individuals know and use [11] 5. Divergence of interests and congruency of individual and organisational goals [16] However, as Reid [17] argued in his research “the most effective way to disseminate knowledge and best practice is through systematic transfer”. And this can be accomplished in the implementation of knowledge transition routines [18], leading to an organizational culture of knowledge.
2 Methodology The methodological approach was the case study (Yin, 2014), and the data was collected through interviews and a questionnaire application in a multinational company located in Portugal: a) Interviews: the main goal was to collect employees’ opinion including Operators, engineers/Technicians (e.g., software systems, electrical, and project), Managers (e.g., project, marketing, process, and manufacturing), and directors (operations and marketing, production, software development), about the knowledge transition processes and the innovation process that was being implemented in the organisation. There were realized 15 semi-structured interviews, based on interview script, during the 2nd quarter of 2019. b) Questionnaire: administered to the employees of the company, distributed across various functional areas and job positions. The questionnaire was online and selfadministrated and was organized by several dimensions: work organisation, technology, product development, process, external relations, workers’ participation, and knowledge management. The number of responses received were 63, during the 2nd and 3rd week of July 2019.
Knowledge in Transition in an Industrial Company
89
The research design was composed by several actions (Fig. 1):
Fig. 1. Methodology approach
The research examines the perceptions of the organisational actors about the transition of employee’s knowledge to help the organisation in the organisational innovation processes. For this purpose, several interviews were made to different hierarchical organisational actors. The empirical work was organised in six stages. The first stage was a visit to the organisation for the preparation of the interviews and also the questionnaire application procedures. The second stage concerned the realization of the interviews for data collection. The third stage included data analysis based on the collection made through the interviews. The fourth action was the application of the questionnaire, followed by the statistical analysis of the fifth stage; and finally, the sixth stage/action regarding the reflection about the knowledge transfer processes.
3 Findings of the Research 3.1 The Context of the Company Regarding Innovation and Knowledge The company began the whole innovation process by implementing a very structured system with several tools adapted to all organisational dimensions. One of the critical factors of success is the top management involvement in all the processes, and the willingness to create and implement a culture of innovation and change. This culture is being created daily, creating habits and behaviours of participation, communication, and involvement in all aspects – this constant change involves both micro and macro changes.
90
M. J. Sousa et al.
During the interviews sessions, almost all the actors have made suggestion of change, not only involving their workstations, but also the organisation itself. This culture of innovation and participation is deeply integrated in the company organisational life. When we analyse the routines for creating and sharing knowledge, we can find several mechanisms used to facilitate the share: suggestion boxes, openness to make suggestions to the Managers, several types of workshops where employees from different sections participate, and several transversal projects of improvement, quality, and maintenance. Workers use the suggestion boxes as a space where they can uncover new ideas that help improve the organisation. Cross-functional workshops and meetings are a crucial space to share perspectives and to make discussions that provide invaluable knowledge. Organisational actors share their opinions and insights, as well as their own questions, sharing and creating new knowledge. For added impact, outside specialists and even costumers participate in these sessions. Their perspectives can be refreshing and break down the thinking routines of internal workers. Transversal projects or projects related to quality systems also are spaces for workers to share their knowledge and experiences. 3.2 Knowledge Transition Processes The innovation process is a key factor because of the importance of implementing new ways of production and new organisational processes to accomplish higher efficiency. Involving workers in this process requires the use of management tools such as communication and the promotion of workers’ involvement and participation. The company uses several mechanisms to promote knowledge share and develop new ideas. It is important to point out the suggestions system (mainly used to make production improvements), the workshops on innovations and new products, and the knowledge networks (specially the informal ones) (Fig. 2). Looking for another perspective, we can say that the company is a learning space at a technical and organisational level. One of the most effective tools to create and disseminate knowledge is though workshops with people from different sections or people from just only one section. Costumers and external specialist often participate in the workshops and help the discussion and the creation of new knowledge that helps implement new practices, tools, or technology. The workshops in the company can be seen as knowledge creation and sharing processes, like the communities of practice or other processes of linking workers to others with expertise. Relational competences are a key to the capture, use and creation of new knowledge and learning within the company. The participation of all organisational actors in innovation process helps to develop a more consistent knowledge-sharing culture. Employees share ideas and insights naturally and not as something they are forced to do. There is a connection between sharing knowledge and achieving the business goals or solving practical problems. The knowledge transition process among sections and workers is very peculiar, as they implement a new practice, process, or technology in one specific workstation according to the Operator openness to change. When it is working perfectly and new and
Knowledge in Transition in an Industrial Company Knowledge Crical Area
Knowledge Transion Mechanisms Informal networks Workshops Documental tools Prototype projects IT systems Networks
91
Organizaonal Actors (from)
• I&D
• • • • • •
• Quality
• Informal Networks • Workshops • Documental tools • IT tools
• Quality • Producon
• HR
• Informal Networks • Documental tools • IT tools
• Quality • Producon
• Producon & Maintenance
• Communicaon spots • Workshops • Informal Networks • Documental tools • IT tools
• Producon • Quality • Maintenance
• Assembling
• Informal Networks • Documental tools
• Producon • Maintenance • Quality
• Client Service
• Informal Networks • Documental tools • IT tools
• All
• • • •
I&D Quality Producon Maintenance
Fig. 2. Knowledge transition mechanisms
better results are achieved, they share this new knowledge to other workers and transfer it to their workstations, disseminating the new knowledge along the plant. 3.3 Knowledge Transition Impacts Knowledge transition in the company has a huge impact on organizational routines: In the work organisation, the level of responses is also very high, involving all hierarchical levels, and only Project teams and Services’ externalization got very few responses:
92
M. J. Sousa et al.
Total quality management programs 87,5% New work processes
87,5%
Increasing planning processes
87,5%
Self-quality control
75%
Increasing dialogue
75%
Autonomous teams
75%
Network
62,5%
Project teams
12,5%
Services externalization
12,5%
Total Quality Management Programs were implemented with the definition of problem-solving routines and quality standards. Self-Quality Control has increased because of the new management practices and quality standards. New work processes are linked with innovation system principles and all the new and continuous change leading to an increasing dialogue among workers and managers. Autonomous teams refer to team’s autonomy to solve some problems according to the workstation complexity. Increasing planning process through the innovation system instruments with the goal to reduce costs and to increases productivity. Network refers to the informal relationship among workers and Managers to solve all the emerging problems and to find their specific solutions. Increasing dialogue with the creation of the communication corners, the realisation of the workshops and with the visual management procedures. Services’ externalization is only used when the organisation does not have the competencies needed to develop the work, and project teams is a concept which is not very clear in the company. Nevertheless, they work in teams in each section of the plant. In the technology dimension it was mainly the Managers and Middle Managers that answered positively to Acquisition of new information and communication technologies and Acquisition of new production technologies: Acquisition of new information and communication technologies
62,5%
Acquisition of new production technologies
50%
Acquisition of new information and communication technologies in office automation, and acquisition of new production technologies to increase productivity. In Product development seems that some of the Operators do not see any change in the product’s technical characteristics: Technical characteristics 62,5% Design
50%
Packaging
25%
Knowledge in Transition in an Industrial Company
93
Technical characteristics have specifically increased the quality of the projects, as design make them more modern and gave them style, nevertheless packaging does not seem to be relevant in the company activity. In Market dimension ooperators do not seem to be aware of the organization’s Market Share and its exploration of New Markets worldwide: Product and services quality 75% New markets
50%
Market share
50%
Product and services quality have increased with the innovation system, new markets refer to entering the USA market, with the market share increasing since 2000. Regarding Process all participants opinion are aligned and there has been an Increase of production capacity deriving from the organisation’s knowledge share culture, and an increase of Production flexibility: Increase of production capacity 100% Production flexibility
87,5%
Work cost
62,5%
The Increase of production capacity is due to the continuous change in the work and organisation practices; and the production flexibility has increased with the autonomous teams and with the competencies matrix system implemented in the plant; also work coast decreased especially due to the waste reduction and with the new stock management system. Regarding External relations there was a high number of answers that pointed to the Increasing relations with suppliers and Increasing relations with other organisations and community. Operators do not point out the Increasing relations with clients because they do not have a direct contact with them: Increasing relations with suppliers
87,5%
Increasing relations with other organisations 75% Increasing relations with community
75%
Increasing relations with clients
50%
Increasing relations with suppliers got high marks because of the quality standards and because of costs reduction. Increasing relations with clients was attained by making them participate in the innovation process, and by the quality of post-sales support services that helped them solve problems with the equipment’s. Increasing relations with other organisations and the community applies mainly to university developing Innovation Projects (namely the Aveiro University) and to the community’s donations.
94
M. J. Sousa et al.
Almost all participants answered that there was a high level of workers’ participation in the knowledge transition in the organization: Improvement suggestions
100%
Meetings
87,5%
Technical problem solving 75%
Improvement suggestions through the suggestion’s boxes and directly to the Managers. Meetings in the communication spots to discuss the problems and to discuss the new changes. Technical problem-solving routines are increasing and being improved to help solve the problems in lesser time and with less production costs. In respect to Knowledge management Operators and some Technicians do not have the perception about the existence of a Knowledge network or Best practices repositories: Knowledge network
50%
Best practices repositories 50%
Knowledge network refers mainly to informal networks to solve problems; and best practices repositories in databases that can be used for other sections or departments of the company.
4 Conclusions The knowledge transition among employees supports the innovation process of the company studied, and to support that transition the managers create a working environment with different thinking styles and without penalties for failure, encouraging experimentation. They also encourage an open culture, having fewer formal relations, implementing several activities for knowledge sharing. To make the process of knowledge transition effective they promote trust among workers and between workers and Managers, with a culture of participation and involvement since the innovation system implementation. They also create routines, procedures sheets and knowledge databases for problems and solutions related to quality management, and problems and solutions – this facilitates the knowledge transition process. There are also several impacts of the knowledge transition in the organization activities and dimensions, as work organizations, technologies, product development, market, process, external relations, workers’ participation, and knowledge management systems. Finally, it seems to be important to point out that few researchers have examined the transition of knowledge in innovation processes. This is probably since innovation and knowledge are difficult to measure. With this research the goal was to clarify the knowledge transition mechanisms used in innovation processes. Furthermore, future studies are required to determine the importance of different types of knowledge in transitions processes in different organizational activities.
Knowledge in Transition in an Industrial Company
95
References 1. Sousa, M.J., Cascais, T., Rodrigues, J.P.: Action research study on individual knowledge use in organizational innovation processes. In: Rocha, A. (ed.) Advances in Intelligent Systems and Computing, pp. 105–113. Springer, Cham (2015a) 2. Sousa, M.J.: Innovation: the key for creating and sharing knowledge. In: Jamil, G. (ed.) Effective Project Management Through the Integration of Knowledge and Innovation. IGI Global, Hershey (2015b) 3. McAdam, R., McCreedy, S.: The process of knowledge management within organisations: a critical assessment of both theory and practice. Knowl. Process Manag. 6(2), 101–112 (1999) 4. McAdam, R., McCreedy, S.: A critique of knowledge management: using a social constructivist model. New Technol. Work Employ. 15(2), 155–168 (2000) 5. Nonaka, I., et al.: SECI, Ba and leadership: a unified model of dynamic knowledge creation. Long Range Plan. 33, 5–34 (2000) 6. Sousa, M.J., Dal Mas, F., Garcia-Perez, A., Cobianchi, L.: Knowledge in transition in healthcare. Eur. J. Investig. Health Psychol. Educ. 10, 733–748 (2020) 7. Davenport, H.T., Prusak, L.: Working Knowledge: How Organisations Manage What They Know. Harvard Business School Press, Boston (2000) 8. Polanyi, M.: Personal Knowledge. Towards a Post Critical Philosophy. Routledge, London (1958, 1998) 9. Nonaka, I., Takeuchi, H.: The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation. Oxford University Press, New York (1995) 10. Zairi, M., Whymark, J.: The transfer of best practices: how to build a culture of benchmarking and continuous learning. Benchmarking Int. J. 7(1), 62–79 (2000) 11. Rulke, D.L., Zaheer, S., Anderson, M.: Sources of managers’ knowledge of organizational capabilities. Organ. Behav. Hum. Decis. Process. 82(1), 134–149 (2000) 12. Almeida, P., Kogut, B.: Localization of knowledge and the mobility of engineers in regional networks. Manag. Sci. 45(7), 905–917 (1999) 13. Darr, E.D., Kurtzberg, T.R.: An investigation of partner similarity dimensions on knowledge transfer. Organ. Behav. Hum. Decis. Process. 82, 28–44 (2000) 14. Argote, L.: Organizational Learning. Creating, Retaining and Transferring Knowledge. Kluwer Academic Publishers, Norwell (1999) 15. Darr, E., Argote, L., Epple, D.: The acquisition, transfer, and depreciation of knowledge in service organizations: productivity in franchises. Manag. Sci. 41(11), 1750–1762 (1995) 16. Alchian, A., Demsetz, H.: Production, information costs, and economic organization. Am. Econ. Rev. 62, 777–795 (1972) 17. Reid, F.: Creating a knowledge-sharing culture among diverse business units. Employ. Relat. 30, 43–49 (2003) 18. Sousa, M.J.: Knowledge integration in problem solving processes. In: Rocha, Á., Correia, A.M., Wilson, T., Stroetmann, K.A. (eds.) Advances in Intelligent Systems and Computing. Springer, Cham (2013) 19. Yin, R.K.: Case Study Research: Design and Methods. Sage Publications, Thousand Oaks (2014)
Project Management Practice in SMEs: A Comparative Study of the Portuguese and Danish Economic Context Cristiano Seiji Watanabe1(B) , Anabela Pereira Tereso1 , Aldora Gabriela Gomes Fernandes2 , Lars Jespersen3 , and Jens Vestgaard4 1 Engineering School – DPS, Centre ALGORITMI, University of Minho, Guimarães, Portugal
[email protected] 2 Department of Mechanical Engineering, University of Coimbra,
CEMMPRE, Coimbra, Portugal [email protected] 3 IBA International Business Academy, Kolding, Denmark [email protected] 4 Cphbusiness Copenhagen Business Academy, Copenhagen, Denmark [email protected]
Abstract. The project management practice by small and medium-sized enterprises (SMEs) is an urgent demand and a clear concern of European countries for economic reasons in recent years. One major problem in traditional management practices is the focus on efficiency rather than innovation and control rather than empowerment. Several authors mention the existence of a “gap” between the theory and the actual use of project management by SMEs that increasingly require government support. This paper aims to analyze the economic scenario in the context of Portuguese and Danish SMEs that promotes innovation and project management practice through literature review and economic reports. The measures and initiatives adopted to minimize the factors that hinder the realization of businesses highlight the efforts of each country to encourage organizations to adapt to a dynamic and competitive context. The perspective is that the economic scenario presented points favorable aspects for SMEs and serve as parameters for future work and development of management practices appropriate to the size and maturity of SME organizations. Furthermore, the paper strengths the role of SMEs in innovation and the importance of using project management practices. Keywords: Project management practice · SMEs · Innovation
1 Introduction After the financial crisis, which affected Europe between 2010 and 2014, the economic growth returned. The technological development and high market competitiveness lead SMEs to adapt to the traditional project management methods adopted by large corporations. The study’s starting point was based on the use of project management practices existing in Portuguese and Danish SMEs. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 96–105, 2021. https://doi.org/10.1007/978-3-030-72651-5_10
Project Management Practice in SMEs
97
The formulation of the work aims to analyze particularities of project management that support SMEs, underlying structural factors that guide companies in each country to improve the effective use of management and initiatives to promote innovation, knowledge, and training for project managers. Based on the economic data, this study considered the effectiveness of adapted or new practices according to different organizational structures, levels of maturity, project size, and resource limitations. The study is framed to answer the following questions: 1. What is the economic scenario in Portugal and Denmark in the context of SMEs, considering innovation and project management practice? 2. What polices and measures have been implemented by Portugal and Denmark to support SMEs, and how do they impact project management practice? 3. How to promote new project management practices and innovation to meet the needs of SMEs (opportunities for scientific research)? Section 2 presents a brief background and is followed by Sect. 3 that presents, for both countries: the economic scenario of SMEs; policies to support SMEs; and the importance of project management. Finally, Sect. 4 and 5 present the discussion and conclusions.
2 Background There are numerous papers analyzing the performance factors and success of business practices, however, comparative work on the contribution of management practices to SMEs in European countries is scarce [1]. Denmark’s economy is made up of 99.7% of organizations described as SMEs according to European Union definitions [2]. These corporations demand flexible and adaptable organizational forms to promote innovation and a decision making model that contributes to rapid development [2]. Similarly, Portugal has an economy composed mostly of SMEs – 99.9% [3], with limited internal structures to assume more complex project management practices and little effectiveness to follow guidelines developed by the Project Management Institute (PMI) [1]. The ability to meet customer expectations enables a competitive advantage [4]. Therefore, SMEs also need to constantly improve their project management processes and operational performance measures to achieve strategic and financial benefits. The management production approaches seen in large companies are rigid or difficult to apply due to a lack of knowledge and leadership [5]. As well, project management tools and techniques used in large companies may not be adequate for the context of uncertainty and complexity present in SMEs [6, 7]. SMEs play an important role in the economy and innovation with the generation of jobs, wealth, new knowledge, and technologies [8]. Innovations that also depend increasingly on collaborative university-industry Research and Development (R&D) initiatives to be materialized [9].
98
C. S. Watanabe et al.
3 A Comparative Analysis of the Portuguese and Danish Economic Scenario in the Context of SMEs This section presents data, administrative policies and measures from Denmark and Portugal that promote innovation, thus increasing the importance of project management in small and medium-sized enterprises. Project management is a wider and more pluralistic discipline for achieving organizational efficiency, effectiveness and innovation [10]. 3.1 Economic Scenario of SMEs SMEs are the backbone of Portugal’s economy, accounting for 68.3% of the value-added and 77.4% of employment, which means 2.9 people employed on average in each SME. Driven by the manufacturing and retail trade sector, the added value per person employed is e22,900, slightly more than half the EU average [11]. According to OECD [12], the Portuguese manufacturing sector in 2016 was already responsible for 12% in job creation by new companies, which totaled 66,632 companies in the period [3], leveraged by the recovery and economic expansion of 2014–2018 of the automotive industry and the producers of metals [11]. On the other hand, Denmark, in the same period, presented lower participation of the manufacturing sector and a more uniform distribution by sector when compared to the Portuguese graph (see Fig. 1).
Fig. 1. Structure and performance of SMEs. Adapted from OECD [12].
According to the European Commission [11], the positive reflection of the Portuguese economic scenario responsible for the growing number of companies - 45,191 companies in total in 2018 - is justified by the implementation of a substantial number of political measures, addressing 9 out of 10 principles established by the main flagship policy initiative called Small Business Act (SBA). The SBA’s performance in Portugal is largely in line with the EU average, with an emphasis on above-average criteria in Entrepreneurship, Environment, ‘Second chance’, ‘Responsible administration’, ‘Skills & innovation’; and negative performance in the areas of ‘Internationalization’ and ‘State aid & public procurement’ (see Fig. 2). In turn, Denmark has a very strong SBA profile with emphasis on the best performance in Internationalization and the third-best performance in ‘Access to finance’ among European countries. Its performance is above the EU average in six SBA principles, and it only scores slightly below average in ´State aid & public procurement´ [2].
Project Management Practice in SMEs
99
Fig. 2. SBA profile. Adapted from European Commission [2, 11].
The average productivity of Danish SMEs is e79,200, substantially higher than the EU average of e44,600 [2]. As a highlight of the Danish economy pointed out by the European Commission [2], medium-sized companies boosted a 37.3% growth of the added value of SMEs in 2014–2018. Denmark intends to promote business in the next three years by improving the access of companies to skilled labor, strengthening entrepreneurship, and encouraging digitalization, automation, and internationalization as a fundamental priority [13]. However, the challenge continues in research and innovation activities, which benefit from high levels of investment and remain concentrated in some key players [2]. In Portugal, 67% of SMEs with 10 or more employees reported innovation activity during the period 2014–2016, compared to 52% in Denmark during the same period [14] (see Fig. 3). Another study by Muller et al. [15] indicates that SMEs active in industries characterized by high or very high R&D intensity represented 27.3% of the SME population in the EU-28 non-financial business sector (NFBS) in 2018.
Fig. 3. SMEs innovation activity over the period 2014–2016. Adapted from community innovation survey [14].
The gap between SMEs and large companies according to the types of innovation is evident (see Fig. 4). The marketing sector is an exception for presenting less innovation by large enterprises [15]. CIS-2016 explored the innovation-hampering factors (Fig. 5) and the lack of required skills within SMEs to carry out R&D and/or innovation activities [15]. Other factors, such
100
C. S. Watanabe et al.
Fig. 4. Share of innovating enterprises, by type of innovation and size [15].
as the high costs associated with this typology of activities, can be partially overcome by good project management practice, which leads to cost savings.
Fig. 5. Factors that hamper innovation activities, (2014–16) - large enterprises and SMEs [15].
3.2 Priority Policies and Measures for SMEs An important aspect analyzed by Villa and Taurino [16] is that the various types of government aid assumed very similar characteristics in terms of automation, integration, and interconnection of the production system with its management in different countries. The initiatives here highlighted are relevant measures that impact innovation in SMEs in Portugal and Denmark. Portugal, in recent years, has implemented a series of measures in areas of competence and innovation, with emphasis on two main keys denominated ‘Operational competitiveness and internationalization program’ (Programa Operacional Competitividade e Internacionalização) and the ‘National strategic reference framework programme’ (Portugal 2020). Other relevant initiatives are ‘Industry 4.0 Initiative’, ‘Technological and business innovation strategy for Portugal 2018–2030’ (Estratégia de
Project Management Practice in SMEs
101
Inovação Tecnológica e Empresarial para Portugal 2018–2030), ‘Collaborative laboratories’ (CoLABS), ‘Technological interface centres’ (Centros de Interface Tecnológico), ‘Suppliers club’ (Clube de fornecedores), ‘Research opportunities voucher’ (Vale Oportunidades de Investigação) and INCoDe.2030 Initiative [11]. The governmental efforts focus on solving the most relevant problem factors illustrated in Fig. 6. According to the World Economic Forum [17], the data highlight significant problems related to inefficient governance bureaucracy, access to finance, and insufficient capacity to innovate.
Fig. 6. Most problematic factors for doing business in Portugal [17].
Fig. 7. Most problematic factors for doing business in Denmark [17].
Denmark has also introduced significant measures in recent years with an emphasis on the Innovation Fund Denmark, National Innovation Networks (Innovationsnetværk Danmark), Decentralised Business Promotion Strategy, the Industrial Researcher program (Erhvervsforsker), ‘SME: Digital’ and ‘SME: Board’ [2]. Similarly, through measures such as the ‘SMV Pro concept’ (Dansk Erhvervsfremmebestyrelse), the Danish
102
C. S. Watanabe et al.
government seeks to address the problems related mainly to tax rates, tax regulations, inefficient government bureaucracy, and restrictive labor regulations, as illustrated in Fig. 7. 3.3 Project Management in SMEs Project management practices are viewed as those tools and techniques related to the day-to-day practice or to the things people do [18]. Another definition is the view of practice as social conduct, defined by history, context, individual values, and wider structural frameworks [19]. Turner and Ledwith [1] point out that management practices follow the level of maturity and growth of the company. Since SMEs carry out smaller projects and have different organizational and team structures, they need more informal, less bureaucratic, and people-focused project management practices [1, 8, 20]. In addition, SMEs are more susceptible to weak management, poor financial controls, and competition, which is the dominant dimension in the realization of changes [21]. They face substantial difficulties because of internal restrictions resources (given their size), external resources (due to market failures), and benefits of high levels of investment remain concentrated in some key players [22]. According to the study conducted by Turner, Ledwith, and Kelly [20], SMEs carry out many projects managed by amateurs, whose project management is not the first discipline. These projects represent more than 40% of the turnover of small and microenterprises and more than 60% in the first two years of existence. Kozlowski and Matejun [23] state that project management should be implemented, with consideration for the individual needs and possibilities of specific business entities. SMEs require tools that are different from the more traditional versions designed for medium or large-scale projects and proportional to the size of the company [8, 21]. Also, they should consider client consultation; planning, monitoring and control; and resource allocation as predominant success factors [8].
4 Discussion Based on existing economic data and studies, it is possible to identify an initial scenario design to foment future research and answer the three research questions. 1. What is the economic scenario in Portugal and Denmark in the context of SMEs, considering innovation and project management practice? The data point to the real participation of SMEs in the economies of both countries and the impact of each government’ measures to establish a commitment to this companies. Also, the importance of project management in the first years of the companies’ existence accounts for 60% of the turnover. The existing studies show the gap between project management theory and project management practice, the inability of companies to follow existing guidelines, and the lack of adaptations to promote competitiveness. There are exceptional potential and European political interests in encouraging innovation, as many SMEs operate in sectors characterized by low-knowledge or technology intensities.
Project Management Practice in SMEs
103
SMEs in Portugal and Denmark have a heterogeneous performance, showing the need to understand project management practice to group countries with strong performance, compare them, and at the same time, share knowledge with other countries that are struggling to achieve it. 2. What policies and measures have been implemented by Portugal and Denmark to support SMEs, and how do they impact project management practice? Innovation is driven by many actors, including companies, academic institutions, and individuals. It is up to governments to interconnect knowledge and make it accessible to SMEs with programs and incentives. The main recent government measures and initiatives in each country aim to minimize the most problematic factors indicated in Fig. 6 and Fig. 7. These measures impact project management as they promote entrepreneurship, access to finance, and innovation. The Portugal 2020 program exemplifies successful measures taken between Portugal and the European Commission to identify smart strategies to guide scientific, technological, and economic expertise. Similarly, the Danish initiative ‘Innovationsnetværk Danmark’ promotes innovation through a wide-ranging network of researchers, technology service providers, and private companies. Therefore, they are measures with similar objectives to meet the particularity of each country and that can be examples for other European nations. Denmark, which has a strong SBA profile, can share successful measures that have remedied past weaknesses with Portugal. The understanding of management in SMEs in each country can contribute mutually to the expansion of knowledge and innovation. 3. How to promote new project management practices and innovation to meet the needs of SMEs (opportunities for scientific research)? This study shows that the development of knowledge in project management for SMEs through innovation and government initiatives is a research opportunity. The understanding of the use of project management is vital for economic development since SMEs represent the majority of companies, have different sizes and resource constraints (internal and external). The usage of project management is strongly linked to innovation and requires the dissemination of knowledge among the best-performing countries, according to the World Economic Forum and the European Commission. Although there is a gap between large companies and SMEs, it is necessary to understand the practices and adaptations of existing guidelines. Organizations of different sizes and sectors have increasingly used the management of projects to achieve their tangible and intangible benefits [24]. The urgency for new knowledge and innovation to assist SMEs becomes more and more evident, and every aspect and positive result of each country should be analyzed in-depth and seen as a complementary opportunity.
5 Conclusions Portuguese and Danish SMEs depend significantly on government support as they face adversities [2, 11, 12, 25]. The policies and measures adopted by both countries to favor SMEs need to align the needs of entrepreneurs with the management practices appropriate for them. Furthermore, they depend on an understanding of the characteristics presented in the SBA profile and the problematic factors that hinder the business.
104
C. S. Watanabe et al.
Portugal is highlighted by the level of SMEs innovation activity, while Denmark is marked by the strong SBA profile. It is expected that this paper, which analyzes the scenario and briefly compares Portugal and Denmark, will be able to extract the positive effects of the initiatives to mitigate the deficiencies and problems faced by other European countries. It will also enable learning from countries that perform better on EU principles and have a strong SBA profile.
References 1. Turner, R., Ledwith, A.: Project management in small to medium-sized enterprises: fitting the practices to the needs of the firm to deliver benefit. J. Small Bus. Manag. 56, 475–493 (2018). https://doi.org/10.1111/jsbm.12265 2. EC: 2019 SBA Fact Sheet: Denmark (2020) 3. PORDATA: PORDATA - Pequenas e médias empresas: total e por dimensão. https://www. pordata.pt/Portugal/Pequenas+e+médias+empresas+total+e+por+dimensão-2927, Accessed 13 Jan 2020 4. Burns, P.: Entrepreneurship and Small Business: Start-up, Growth and Maturity. Palgrave (2016) 5. Moeuf, A., Pellerin, R., Lamouri, S., Tamayo-Giraldo, S., Barbaray, R.: The industrial management of SMEs in the era of Industry 4.0. Int. J. Prod. Res. 56, 1118–1136 (2018). https:// doi.org/10.1080/00207543.2017.1372647 6. Svejvig, P., Andersen, P.: Rethinking project management: a structured literature review with a critical look at the brave new world. Int. J. Proj. Manag. 33, 278–290 (2015). https://doi. org/10.1016/j.ijproman.2014.06.004 7. Alves, P.R., Tereso, A., Fernandes, G.: Project management system implementation in SMEs: a case study. In: Proceedings of 33rd International Business Information Management Association Conference IBIMA 2019 Education Excellent Innovation Management through Vision 2020, pp. 8322–8332 (2019) 8. Turner, R., Ledwith, A., Kelly, J.: Project management in small to medium-sized enterprises: a comparison between firms by size and industry. Int. J. Manag. Proj. Bus. 2, 282–296 (2009). https://doi.org/10.1108/17538370910949301 9. Fernandes, G., Moreira, S., Araújo, M., Pinto, E.B., Machado, R.J.: Project management practices for collaborative university-industry R&D: a hybrid approach. Procedia Comput. Sci. 138, 805–814 (2018). https://doi.org/10.1016/j.procs.2018.10.105 10. Jugdev, K., Thomas, J., Delisle, C.: rethinking project management: old truths and new insighs. Proj. Manag. 7, 36–43 (2001) 11. EC: 2019 SBA Fact Sheet: Portugal (2020) 12. OECD: OECD SME and Entrepreneurship Outlook 2019 (2019). https://doi.org/10.1787/349 07e9c-en 13. Danish Business Promotion Board: Erhvervsfremme I Danmark 2020–2023, 122 (2020) 14. CIS: Community Innovation Survey: latest results - Product – Eurostat. https://ec.europa.eu/ eurostat/web/products-eurostat-news/-/DDN-20190312-1, Accessed 01 Nov 2020 15. Muller, P., Robin, N., Jessie, W., Schroder, J., Braun, H., Becker, L.S., Farrenkopf, J., Ruiz, F.A., Caboz, S., Ivanova, M., Lange, A., Lonkeu, O.K., Muhlshlegel, T.S., Pedersen, B., Privitera, M., Bomans, J., Bogen, E., Cooney, T.: Annual Report on European SMEs 2018/2019 - Research & Development and Innovation by SMEs (2019) 16. Villa, A., Taurino, T.: SME innovation and development in the context of industry 4.0. Procedia Manuf. 39, 1415–1420 (2019). https://doi.org/10.1016/j.promfg.2020.01.311 17. WEF: The Global Competitiveness Index Report 2017–2018 (2018)
Project Management Practice in SMEs
105
18. Tereso, A., Ribeiro, P., Fernandes, G., Loureiro, I., Ferreira, M.: Project management practices in private organizations. Proj. Manag. J. 50, 6–22 (2019). https://doi.org/10.1177/875697281 8810966 19. Cicmil, S., Williams, T., Thomas, J., Hodgson, D.: Rethinking project management: researching the actuality of projects. Int. J. Proj. Manag. 24, 675–686 (2006). https://doi.org/10.1016/ j.ijproman.2006.08.006 20. Turner, R., Ledwith, A., Kelly, J.: Project management in small to medium-sized enterprises: tailoring the practices to the size of company. Manag. Decis. 50, 942–957 (2012). https://doi. org/10.1108/00251741211227627 21. Masurel, E., Van Montfort, K.: Life cycle characteristics of small professional service firms. J. Small Bus. Manag. 44, 461–473 (2006). https://doi.org/10.1111/j.1540-627X.2006.00182.x 22. EC: Annual Report on European SMEs. European Commission (2019) 23. Kozlowski, R., Matejun, M.: Characteristic features of project management in small and medium-sized enterprises. E a M Ekon. a Manag. 19, 33–48 (2016). https://doi.org/https:// doi.org/10.15240/tul/001/2016-1-003. 24. Fernandes, G., O’Sullivan, D.: Benefits management in university-industry collaboration programs. Int. J. Proj. Manag. 39, 71–84 (2020). https://doi.org/10.1016/j.ijproman.2020. 10.002 25. Davis, N., Galvan, C., Gratzke, P., Jelinek, T., Kiessler, A., Nurluel, M., Quigley, J., Ruppert, M.: Enhancing Europe’s Competitiveness Fostering Innovation-Driven Entrepreneurship in Europe (2014)
Critical Management Risks in Collaborative University-Industry R&D Programs Gabriela Fernandes1
, Joana Domingues2 , Anabela Tereso2(B) and Eduardo Pinto3
,
1 University of Coimbra, CEMMPRE, Department of Mechanical Engineering,
3030-788 Coimbra, Portugal [email protected] 2 University of Minho, Centre ALGORITMI, Production and Systems Department, 4804-533 Guimarães, Portugal [email protected] 3 CCG/ZGDV Institute, Campus de Azurém, 4804-533 Guimarães, Portugal [email protected]
Abstract. In university-industry collaborative research and development (R&D) programs and projects, Risk Management (RM) has assumed a preponderant importance for the success of programs and projects due to their innovative nature. This paper aims to present the management risks identified in R&D funded programs within a large university-industry collaboration (UIC) case study, as well as the prioritization of the risks identified, and the suggested responses to the most critical risks. Based on literature, document analysis and participant observation of the case study, a catalogue of management risks for R&D programs and projects was created. Seeking to focus on critical risks, a survey and a focus group were developed to analyze these risks by key stakeholders involved in the UIC. The prioritization of risks was conducted based on a qualitative risk analysis carried out by key stakeholder’s in the UIC, who take the role of Program Manager, Project Manager and Program and Project Management Officer. Then, the critical risks were discussed in the focus group to define risk response strategies. Additionally, the paper discusses strategies for the institutionalization of the catalogue of management risks in UICs. The research was conducted based on only one case study which may limit the generalization of results. Keywords: Risk management · R&D programs · University-industry collaborations · Risk planning · Risk identification and planning risk responses
1 Introduction In today’s economic context, the challenge of improving and increasing the positive impacts and opportunities inherent in the research and development (R&D) processes has become a priority [1]. One of the ways of supporting and responding to existing R&D challenges is through the establishment of relationships between industry, universities, © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 106–115, 2021. https://doi.org/10.1007/978-3-030-72651-5_11
Critical Management Risks in Collaborative University-Industry R&D Programs
107
research centers and other organizations [2]. In the globalization context, competition increased as well as the investment in R&D. Many countries have implemented policies to promote and sustain university-industry collaborations (UICs) to support technological growth [1]. UIC is seen as a temporary organization that presents a collaborative working environment, within a specific context, with heterogeneous partners, collective responsibilities and, in many cases, public financial support [3]. UICs are usually funded and named as projects by funding agencies but are often organized as programs by partners [4]. In spite of different organizational cultures between university and industry, the inherent benefits of these partnerships outweigh the barriers [5]. Most of the problems associated with the cultural gap between industries and universities can be mitigated through effective Project Management, as this discipline has already proven to bring tangible and intangible values to both organizations [6]. With the increasing occurrence of UIC projects, it has been possible to verify that there are some associated failure reports [7]. Hence, considerable research has emerged in the identification of management success factors, from which emphasis is on identification, analysis and management of risks [5, 8]. In fact, Risk Management (RM) is becoming a key factor within organizations, as it can minimize the probability and impact of threats and capture opportunities that may arise, during the program and projects’ life cycle [9]. Due to the low maturity of RM in the context of UIC, this paper aims to identify management risks associated to UIC programs and projects and identify the most critical, by prioritizing them throughout a qualitative risk analysis. Additionally, the paper explores typical response strategies to the most critical risks, on a case study in a large-scale UIC program between Bosch Car Multimedia and University of Minho, which focuses on the development and production of advanced multimedia solutions for cars. Consequently, the main contribution resulting from this research work is the improvement of RM in the context of collaborative university-industry R&D programs and projects. The following section presents a literature review on RM in R&D programs and projects. Subsequently, the research strategy and methods used, and the case study background are described. Then, the results are presented, namely the identification of management risks in UICs, the most critical ones and the recommended risk responses. Finally, the last section discusses the research practical implication and concludes indicating the research limitations.
2 Background In this research we adopt the Project Management Institute risk definition “an uncertain event or condition that, if it occurs, has a positive or negative effect on one or more project objectives” [10, p. 397], i.e., risk includes both uncertain events that may negatively affect the project (threats) and those that may have a positive effect on the project objectives (opportunities). Therefore, RM consists of a set of processes, techniques and tools that aim to identify, analyze and respond to the risks of a project, by reducing the nature of negative risks and potentiating positive risks [11]. RM in projects and programs aims to provide a systematic process to identify and manage risk, helping to define different project objectives, improve project control,
108
G. Fernandes et al.
increase the chances of project success, improve communication between project participants, facilitate decision-making and prioritize actions. Risks have the potential to cause the project to deviate from the plan and fail to achieve the objectives [12]. Consequently, the relationship between RM and the success of a project has been the subject of several investigations and recent studies suggest that moderate levels of RM planning have a positive impact on project performance [13]. This relationship results from the fact that all risks and uncertainties are inherent to all projects. According to the existing literature, the chances of a risk occurring are greater in the design, planning and start-up project phases, and the impact of the cost of a risk on the project is lower if it occurs at an early stage in the project life cycle. The early stages of the project represent the time when there is an opportunity to minimize the impact or to work on potential risks [14].Therefore, the risk identification phase is particularly important during the implementation of RM processes and is considered one of the success factors [13]. Due to the novelty of the typology of R&D programs and projects, the identification of risks in these cases becomes even more vital [10, 11, 15]. The identification of the critical management risks, based on the risk assessment parameters probability and impact [11], and their typical risk responses based on previous R&D programs and projects brings an added value for the collaboration’s success [15].
3 Research Methodology 3.1 Research Strategy and Methods The research followed a case study design, aiming to depart from existing knowledge and then learn from the experience of the program and project stakeholders of one large case study between University of Minho (UMinho) and Bosch industry in Portugal. Case study is perceived by researchers as one of the most used research strategies [16]. By using it, researchers can focus on a particular phenomenon and discover crucial knowledge. The research methods used in the case study were document analysis, participant observation, survey and focus group. Based on the literature review, participant observation, as well as document analysis, the context of the case study was studied through various documents that support the management of the programs and its projects. A survey was carried out, aiming to qualitatively classify the risks identified according to the reality experienced in the UIC between Bosch and UMinho. Subsequently, risks were classified and prioritized according to their risk level. Then, a focus group was conducted to discuss the most critical risks, as well as the response strategies to these risks, for key program stakeholders: Program Manager; Project Managers; and Program and Project Management Officers. 3.2 Case Study Background With approximately eight years of partnership, the strategic collaboration established between Bosch and UMinho includes so far five programs in three sequent investment phases, with the sponsorship of the Portuguese Government by competitive public funds.
Critical Management Risks in Collaborative University-Industry R&D Programs
109
These two partners from an early stage have perceived the added value associated with project management in supporting the collaboration, so they have established a Governance Model based on a purposefully developed approach, especially devoted to the management of collaborative R&D funded programs and projects, named Program and Project Management (PgPM) approach. The PgPM approach divides the program life cycle in four phases: Program Preparation, Program Initiation, Program Benefits Delivery and Program Closure [17]. The first investment phase was the HMI-EXCEL program, the second investment phase was named Innovative Car HMI program, and the third phase is known as Crossmapping the Future. Data on the number of staff involved, the investment made, the number of patents and publications created, for the three phases of the investment, can be found in Table 1. Table 1. Three investment phases of the Bosch and UMinho partnership. Bosch–UMinho partnership
Innovation HMI EXCEL
Innovative CAR HMI
Crossmapping the future
Duration
2013–2015
2015–2018
2018–2021
Investment
19 Million e
54.7 Million e
108 Million e
Staff assigned to the program
205
288
516
Deliverables
162
411
656
Scientific publications
32
109
130
Patents
12
24
33
The main risk tool used in the UIC of Bosch and UMinho is the risk register. Each project of the program develops a risk register, which details all the information about the risks (date, description, causes, impacts, qualitative analysis, responses, responsible and current status). The project risk register is updated regularly during the progress meetings. All the project risk registers are compiled in the program risk register.
4 Results Based on literature, participant observation and the documentary analysis of HMI-Excel and IC-HMI programs (1st and 2nd investment phase), a risk catalogue was developed, which presents many of the identified risks in both R&D programs. The developed risk catalogue portrays the risks categorized by the life cycle phase of both programs and projects, the risk causes, impacts, type of risk, based on a Risk Breakdown Structure (RBS), as well as the identified responses to each of the risks. During the risk categorization process, the simplified RBS adopted by Bosch and UMinho was considered. The analysis of the risks identified included eight categories (Scope, Time, Cost, Quality, Human Resources (HR), Purchasing, Management and Patents). Technical risks were
110
G. Fernandes et al.
not considered due to the specificity of this category, which is unique for each project, program or organization. The developed catalogue consists of a database of 40 risks. Then, a survey was carried out aiming to make a risk qualitative analysis, by classifying the risk assessment parameters probability and impact, and consequently evaluating its risk level. Note that these data represent only the reality experienced in the Bosch and UMinho case study. Additionally, a focus group was conducted to identify new risks beyond the 40 risks already surveyed, which resulted in the emergence of two new risks. Table 2 presents the final risk catalogue resultant from the focus group, containing 42 potential management risks of UIC programs. Table 2. Management risks of UIC programs. ID 1
Risk Description Lack of stakeholders' commitment
2
Unclear roles and responsibilities
3 4 5 6
ID 22
Risk Description Resource allocation errors Cross-trained HR that can easily change from 23* project to project
Lack of transparency in the management flow of information between stakeholders Misalignment among stakeholders on the project’s expectations Lack of executive and management leadership Deficit of technical and management competences
7
Governance mechanisms institutionalized
not
fully
8
Limited exploitation of the knowledge and technology developed
9
Rapid changes in the organisation structure
10∆ Attract talented human resources to research * Deviation between the official and the 11 effective program's start date Lack of decision-making procedures, 12 especially in the program preparation phase 13
Delays in the program financial execution
14
Conflicts related to the patent’s authorship Delays in recruitment of human resources (HR) Lack of alignment between the projects of the program
15 16
17* Public recognition of the partnership 18∆ 19 20 21
Wrong assumptions taken during program preparation phase Non-compliance with the consortium contract terms Insufficient allocation of HR Limited collaboration between project team members
24
Conflict between different stakeholders
Lack of alignment of the project team with the 25 project objectives No exploitation of the project results by the 26 industrial partner Changes in the procurement plan defined in 27 the funding application Difficulties in obtaining equipment and 28 materials from suppliers not belonging to the organizations supplier’s database Necessity of new resources not previously 29 planned Resignation of key HR during the project life 30 cycle Misalignment between the project plan in the 31 funding application and the project execution Non-adoption of project management tools 32 and techniques Leveraging of innovative outputs not initially 33* planned Development of new unplanned technologies 34* (serendipity) 35 Delays in orders placed to suppliers 36 37 38 39 40 41 42
Delays in the scheduled project’s activities Difficulty in aligning the project deliverables with those defined in the funding application Non-submission of the number of patent applications contracted Non-execution of the funded budget Non-compliance with quality of the project results Non-compliance with project requirements Reluctance of the consortium to stop projects, when they are no longer aligned with the organizational strategy
Identified and qualified risks in the focus group/* Opportunities or positive risk.
Critical Management Risks in Collaborative University-Industry R&D Programs
111
About 88% of the risks identified are threats and 12% are opportunities. In fact, during the focus group, it was questioned why there were identified a small number of opportunities compared with the number of threats. One participant stated that “the teams are not yet trained to identify opportunities, a factor resulting from the lower level of maturity in RM”. In addition, other participant mentioned that “this is related to the aversion to loss - the tendency of individuals to be more affected by losses than by gains”. The risks identified were qualified by participants mostly as medium and high impact, implying that most of the risks identified may highly impact the success of the programs and projects and the achievement of the objectives initially proposed. Table 3 identifies the risk qualification obtained for the 42 risks, through the probability and impact estimates (Probability and Impact Matrix). Table 3. Probability and impact matrix of the 42 management risks. Impact 0.05
0.10
0.20
0.40
0.80
Probability
0.9 0.7 0.5
8, 11, 15, 31, 36, 42 17*
0.3 0.1
10 ∆*,
3, 12, 19, 22, 23*, 27, 29 7, 16, 28
18 ∆,
2, 4, 6, 9, 13, 14, 20, 21, 24, 25, 32, 34*, 35, 39
1, 30 5, 40
26, 37, 38, 41
33*
Identified and qualified risks in the focus group/* Opportunities or positive risk.
Table 3 shows several risks that need response plans to be implemented to minimize their negative impact or potentiate their positive impact on the project. Identified risks that are located in the red zone of the matrix should be considered a priority and it is necessary to ensure that stakeholders pay adequate attention to reduce their impact on the project or program. Risks located in the yellow zone require periodic supervision by team members, who should develop adequate response plans for them. Finally, risks located in the green zone require monitoring and control, and these generally do not affect the success of programs and projects [10, 11]. Attending the qualitative data analysis, the ten most critical risks, which need more attention by key stakeholders, are presented and categorized in Table 4. All risks identified are threats to the programs and projects. Table 4 shows that most of the critical risks belong to the category of Human Resources (HR), evidencing the key role that HR play in the management of projects and programs [18]. Most of the critical risks occur during the project execution phase, or the program benefits delivery phase, according to the phases identified in PgPM approach adopted in the UIC between Bosch and UMinho [19]. Yet, according to the existing literature,
112
G. Fernandes et al. Table 4. The ten most critical management risks. ID
Risk Description
Risk Category
Risk Level
1
Lack of stakeholders' commitment
HR
0.56
30
Resignation of key HR during the project life cycle
HR
0.56
5
Lack of executive and management leadership
HR
0.40
40
Non-compliance with quality of the project results
0.40
11
Quality Limited exploitation of the knowledge and technology develManagement oped Deviation between the official and the effective program's start Time date
15
Delays in recruitment of human resources (HR)
HR
0.28
31
Misalignment between the project plan in the funding application and the project execution
Scope
0.28
36
Delays in the scheduled project’s activities
8
42
Time Reluctance of the consortium to stop projects, when they are no Management longer aligned with the organizational strategy
0.28 0.28
0.28 0.28
the chances of a risk occurring are higher in the design, planning and start-up phases of projects [14]. However, in this research work, we just focused on management risks, and might technical risks be more critical during the design, planning and start-up phases of projects. Negative risks or threats, in the red zone, should be treated with some urgency to avoid them or to turn them into more residual risks, to decrease their impact on programs or projects’ objectives. So, it is important to define response plans to these risks. Table 5 presents recommended actions discussed during the focus group for responding to the ten most critical risks.
Critical Management Risks in Collaborative University-Industry R&D Programs
113
Table 5. Risk responses for the ten most critical management risks identified. ID Risk responses 1 1) Establish a program vision, mission, and values; 2) Communicate the benefits resultant from participating in R&D UICs for career progression; 3) Establish a maximum number of projects to each project leader to avoid overload; 4) Demonstrate the strategic importance of the program’s expected benefits; 5) Promote moments for sharing results among program and project stakeholders 30 1) Create employment contracts for key researchers to eliminate the precarious nature of research grants; 2) Create a career perspective in industry and/or continue their research work on new projects at university; 3) Integrate research fellows into degree cycles at university; 4) Promote collaboration between team members and their commitment; 5) Provide better working conditions; 6) Provide technical training cycles; 7) Provide a period of transmission of knowledge for the new researcher, in case of resignation; 8) Archive developed knowledge 5 1) Establish a maximum number of projects to each project leader to avoid overload; 2) Create moments for sharing ideas between leaders and team members; 3) Demonstrate the strategic importance of the program’s expected benefits 40 1) Consult regularly the program management team on their technology roadmaps to adapt the product to the market needs; 2) Hire senior research fellows; 3) Request project sponsorship; 4) Require project scope changes when necessary; 5) Promote the writing of scientific articles before the end of the project 8 1) Develop internal and external management policies to recruit key HR; 2) Collect, analyze and archive lessons learned, risks and issues for future use in new R&D programs 11 1) Prepare the consortium for self-funding until the funding contract is signed; 2) Prepare the funding application before 12 to 18 months before the planned project start date 15 1) Improve the visibility of available research grants; 2) Improve employment contract conditions 31 1) Identify the project leader as soon as possible, ideally during the development of the funding application; 2) Ensure the project leader’s commitment to the set of deliverables foreseen in the funding application; 3) Identify the necessary project changes as soon as possible, minimizing its impact 36 1) Involve stakeholders in decision-making; 2) Predict future delays and understand whether the date of delivery of results is compromised 42 1) Explain the reasons for the project closure; 2) Understand whether the development of a new project or different from the proposed could be more beneficial to the partners
5 Discussion and Conclusions Based on the literature review, document analysis and participant observation of the case study context, it was possible to develop a catalogue of management risks for universityindustry collaborative R&D programs and projects. Due to the novelty of this typology of programs and projects, there is a large degree of associated risk and uncertainty. The developed risk catalogue, which also includes the suggested risk responses, based
114
G. Fernandes et al.
on previous R&D programs and projects, gives an important contribution to practice, allowing future UICs to take this as an input during the process of identifying risks, pondering them also as potential risk of their UIC, and include them or not in their risk register. The risk catalogue, including 42 management risks, has been analyzed and validated by key stakeholders of the UIC between Bosch and UMinho, through the response to a questionnaire survey and the conduction of a focus group. From the 42 risks identified (Table 2), it was possible to classify ten risks as being the most critical (Table 4), and it was defined the plan response strategies for the treatment of these ten risks (Table 5). ‘Lack of stakeholder’s commitment’ and ‘Resignation of key HR during the project life cycle’ were identified as the most critical risks. This database of risks can be applied to organizations with a similar context of the collaborative R&D programs established between Bosch and UMinho. However, it is worth of mentioning that throughout the case study analysis, the project teams showed resistance regarding the investment of time in RM activities and their training in this area. The project teams should be encouraged to conduct a proactive identification of risks related to their work tasks, as well as raising interest in the risks related to their project. It would be important to study what factors cause project teams to find no significant advantage in implementing RM practices, and to show them the value of RM practices and other project management practices [6]. In this context, there is a need to standardize the RM processes and tools, such as the use of the risk catalogue here proposed, to facilitate the institutionalization of RM practices, to ensure the effective management of programs and projects developed within a public funding R&D UIC. In order to assist all stakeholders in performing their RM functions, it is necessary to provide training on RM to the project teams, emphasizing its importance, and to involve all stakeholders in the RM process. In fact, in the case of Bosch and UMinho, the Project Management Office played here an important role [17]. Regarding the evaluation of the generic catalogue of management risks by stakeholders, although it has been validated in a focus group, there have sometimes been disagreements between participants. As an example, the controversy between stakeholders in the analysis of the risk ‘Non-compliance with project requirements’ stands out; some of the stakeholders questioned whether it made sense for this to be considered a risk, as it jeopardized the entire development and expected results of the project as a whole. However, it was considered a risk, as non-compliance with the aims of the project can be pointed as a project risk. This discussion happens because of the low level of RM maturity among some stakeholders. Unfortunately, due to the global pandemic of COVID-19, which spread rapidly in 2020, it was not possible to carry out the focus groups in person, generating a greater distance between the program participants, which might have limited the depth of discussion among participants. Furthermore, like any research based on just one case study, the generalizability of its findings is limited. In this regard, future studies can induce multiple case studies and cross-check the conclusions among them, thereby increasing the generalizability of the results. Anyway, the RM catalogue here presented can support the risk identification and the risk response planning activities in other similar R&D UIC programs and projects, bringing an important contribution to practice.
Critical Management Risks in Collaborative University-Industry R&D Programs
115
Acknowledgements. This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.
References 1. Fontana, R., Geuna, A., Matt, M.: Factors affecting university-industry R&D projects: the importance of searching, screening and signalling. Res. Policy 35(2), 309–323 (2006) 2. Pereira, C.: University-industry cooperation: the case of the University of Minho. Geogr. Notebooks 23, 283–293 (2004) 3. Brocke, J., Lippe, S., Barnes, T., Pashby, I., Gibbons, A.: Effective university - industry interaction: a multi-case evaluation of collaborative R&D projects. Eur. Manag. J. 20(3), 272–285 (2015) 4. Fernandes, G., Pinto, E., Machado, R., Araújo, M., Pontes, A.: A program and project management approach for collaborative university-industry R&D funded contracts. Procedia Comput. Sci. 64, 1065–1074 (2015) 5. Rybnicek, R., Königsgruber, R.: What makes industry–university collaboration succeed? a systematic review of the literature. J. Bus. Econ. 89(2), 221–250 (2019) 6. Fernandes, G., O’Sullivan, D., Pinto, E., Araújo, M., Machado, R.: Value of project management in university–industry R&D collaborations. Int. J. Manag. Proj. Bus. 14(4), 819–843 (2020) 7. Bruneel, J., D’Este, P., Salter, A.: Investigating the factors that diminish the barriers to university-industry collaboration. Res. Policy 39(7), 858–868 (2010) 8. Plewa, C., Quester, P.: Key drivers of university-industry relationships: the role of organisational compatibility and personal experience. J. Serv. Mark. 21(5), 370–382 (2007) 9. Alhawari, S., Karadsheh, L., Talet, A., Mansour, E.: Knowledge-based risk management framework for Information Technology project. Int. J. Inf. Manage. 32(1), 50–65 (2012) 10. PMI: A Guide to Project Management Body of Knowledge: PMBoK Guide. 6th edn. PMI, Pennsylvania (2017) 11. PMI: Practice standard for project risk management. Project Management Institute (2009) 12. Peixoto, J., Tereso, A., Fernandes, G., Almeida, R.: Project risk management methodology: a case study of an electric energy organization. Procedia Technol. 16, 1096–1105 (2014) 13. Carvalho, M., Jr. Rabechini, R..: Impact of risk management on project performance: the importance of soft skills. Int. J. Prod. Res. 53(2), 321–340 (2015) 14. Larson, E., Gray, F.: Cross Reference of Project Management Body of Knowledge (PMBOK) Concepts to Text Topics (2011) 15. Stosic, B., Mihic, M., Milutinovic, R., Isljamovic, S.: Risk identification in product innovation projects: new perspectives and lessons learned. Technol. Anal. Strateg. Manag. 9(14), 1–6 (2016) 16. Yin, R.: Case study research: design and methods. Can. J. Progr. Eval. (5) (2014) 17. Fernandes, G., Pinto, E., Araújo, M., Machado, R.: The roles of a programme and project management office to support collaborative university–industry R&D. Total Qual. Manag. Bus. Excell. 31(5–6), 583–608 (2020) 18. Huemann, M., Keegan, A., Turner, J.: Human resource management in the project-oriented company: a review. Int. J. Proj. Manag. 25(3), 315–323 (2007) 19. Fernandes, G., Domingues, J., Tereso, A., Pinto, E.: A stakeholders’ perspective on risk management for collaborative university-industry R&D programs. Procedia Comput. Sci. 181, 110–118 (2021)
A Project Management Hybrid Model of Software Development in an Academic Environment Cláudia Dinis1
, Pedro Ribeiro2
, and Anabela Tereso3(B)
1 Engineering Project Management, Centre ALGORITMI, University of Minho,
4804-533 Guimarães, Portugal 2 Department of Information Systems, Centre ALGORITMI, University of Minho,
4804-533 Guimarães, Portugal [email protected] 3 Production and Systems Department, Centre ALGORITMI, University of Minho, 4804-533 Guimarães, Portugal [email protected]
Abstract. The project management paradigm is changing. Agile methodologies are part of this change. They came in opposition to traditional standards and their dissemination and intertwining is resulting in the adaption of traditional models to hybrid models, especially when dealing with small projects. They allow to balance flexibility and predictability. There is no standard hybrid model, it must be adapted to the project being developed. This paper aims to analyze the performance of a hybrid model were the initiation, planning and finalization followed the traditional methodology and the development followed the agile methodology (scrum). This model was used by a team responsible for the management and development of a software project in an academic setting. The project consisted of the development of a digital platform to support project management in Small and Medium Enterprises (SME). The team was composed by university students and the hybrid model was defined by the teachers of the course. Data were collected through observation and the application of a questionnaire to the team. The results show that the model used was effective for the management and development of this project, done in an academic environment, since it was less formal and more adaptable to people with little experience in project management than a purely traditional model. Keywords: Project management · Hybrid model · Software development
1 Introduction The concept of project management is in evolution, however the use and application of new processes and practices, according to statistics, have little impact on success rates. According to The Standish Group International’s Chaos Summary 2009 [1] and Chaos Report 2015 [2], on the Information and Technology (IT) activity sector, between 2000 and 2015 the increase of project success (on time, on budget, with the required © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 116–125, 2021. https://doi.org/10.1007/978-3-030-72651-5_12
A PM Hybrid Model of SW Develop. in an Academic Environment
117
features and functions) was from 28% to 29%, the increase of challenging projects (late, over budget, and/or with less than the required features and functions) was from 49% in 2000 to 52% in 2015, and percentage of failed projects (cancelled prior to completion or delivered and never used) was decreased from 23% to 19%. These results can be explained, among other reasons, by inappropriate project management practices. Projects are essential for the creation of value and benefits in organizations, so it is necessary that the management and execution of the project is efficient and effective [3]. For project management we can consider traditional models, agile models and hybrid models. The traditional models proposes a cycle development, unilateral and without interactivity [3, 4]. The agile models, which emerged as a faster, more effective and more reliable way for software development [5], propose an iterative development management [4]. The hybrid models blend the two models, which seek to bring together in one model only the best practices of traditional models and agile models [6]. Software development is a professional activity, in which the software is developed for a specific purpose. Software development processes seek to fully specify the requirements, design, develop and test the system [4]. The research described in this paper was based on the development of a software to support project management in SMEs defined by a student of the Master’s in Engineering Project Management (MEPM). The project was developed in an academic environment, by a team of six students of the Integrated Master’s in Engineering and Management of Information System (IMEMIS). This study focuses on the analysis of the practices used in a hybrid project management model, applied in an academic environment, in a software development project executed and managed by a university team. The final goal is to analyze the performance of the defined hybrid model, by addressing the following specific goals: 1. What are the features of the hybrid model used in this project? 2. Is this hybrid model adequate for this project? The paper follows a simple structure. Section 2 gives the theoretical background of the topics under study, by making a synopsis of the main concepts. Section 3 describes the research methodology applied – case study. Section 4 presents the results and discussion. Finally, Sect. 5 presents the conclusions and suggestions for future work.
2 Literature Review 2.1 Project Management Concepts A project is “a temporary endeavor undertaken to create a unique product, service or result” [3]. With projects organizations move from one state (before the project) to another (after the project), being a driver of change in the organization [3]. It is expected that it will achieve an objective and that it will be able to produce the agreed deliverables, in accordance with the previously defined requirements and restrictions [3, 7]. Project management is defined as the application of knowledge, skills, methods, tools, competences and techniques in order to meet the requirements of a project [3, 7]. Organizations like Project Management Institute (PMI), International Project Management Association (IPMA) or Scrum.org, that deal with project management. They aim
118
C. Dinis et al.
to add value to professionals, through standards, guidelines, training and certifications. Project management practices, tools and techniques should be tailored to the organization and project context [3]. The choice is influenced by several factors: sector, work experience, age, work load and educational level [8, 9]. 2.2 Project Management in IT Sector A software process model is a simplified representation of a software process. A software process is a set of related activities that lead to the production of a software product. Software processes are characterized as directed by plans or as agile processes. In the processes directed by plans (waterfall model), activities are planned, and progress is evaluated by comparison with the initial planning. In agile, processes planning is gradual, the system is developed as a series of versions (increments), so that each version adds functionality to the previous one. The process is changed to reflect customers’ changing needs [4]. These models are not exclusive, and are sometimes used together (hybrid model), combining the features of the waterfall model with the features of the incremental (agile) development models. Parts of the system that are well defined can be specified and developed based on the waterfall model, and parts that are difficult to specify in advance should always be developed using an agile approach [4]. Summing up, for the management of a software development project, one can consider traditional, agile or hybrid methodologies. Waterfall Model The waterfall model proposes the development in a single cycle [3]. The activities of each phase must be planned in advance [4]. The start of each activity depends on the conclusion of the previous one, unilaterally, without interactivity, and they do not overlap [3, 4]. The phases are: analysis and definition of requirements (in this phase the system specifications or requirements are defined); system and software design (here a general system architecture is established); implementation and unit test (the software, as a set of programs or program units, is developed and tested to verify if meet specifications); system integration and testing (the system is integrated and tested as a whole and is delivered to the customer); operation and maintenance (errors that had not been discovered are corrected) [4]. This is a model with little flexibility, iterations between phases are synonymous of rework and increased costs and it is difficult to satisfy changing customer requirements. When a new requirement or problem is encountered, its solution is ignored or postponed, sometimes making the system unsuitable for users. This practice makes the model suitable only for projects with well-defined requirements, well understood and with low probability of change during product development [4]. Agile (scrum) Agile methodologies emerge as a lightweight method as opposed to the waterfall model, considered heavy. The scrum is an agile framework where people can deal with complex adaptive problems and deliver products with the highest possible value, in a productive
A PM Hybrid Model of SW Develop. in an Academic Environment
119
and creative way [10]. Uncertainty and creativity are characteristics of scrum; scrum teams must always be prepared for change [5, 11]. The focus of the scrum approach is in managing iterative development. This framework has three phases: general planning (the goals of software design and architecture are established); cycle of sprints (each cycle develops an increment of the system); project closure (complete the required documentation, system help boards and user manuals, and evaluate the lessons learned from the project) [4]. The initial increments of the project allow customers to verify the requirements requested in practice and propose changes to be considered in the later increments of the project. In this way, it is possible to identify flaws and request new changes or corrections throughout the project, with no need to reach the end of the development to know and correct them [4, 5]. Hybrid Model A survey of 800 participants showed that agile models are associated with the intention of improving efficiency and effectiveness in the market and technological innovations. However, although institutions consider themselves to be users of agile models, they also resort to traditional practices and, together, the mix of practices used leads to the desired results [12]. Hybrid is defined as what has different elements in its composition. In project management, a hybrid model can be defined as the combination of principles, practices, techniques and tools from different approaches that aims do adapt management to the specific business and project context. The objective is to reduce risks and increase innovation, to deliver better business results and added value. It is also intended to maximize the performance of the project and product thus providing a balance between predictability and flexibility [12]. The model must be aligned with the reality of people, scenarios and goals [6], so here is no set of “silver bullet” practices. The hybrid model must be built specifically for each project, team and institution, in order to obtain a robust, efficient model, which allows to achieve the project goals. These models have proven to be effective, particularly in dynamic and flexible scenarios, such as in the IT sector, however, it is necessary to critically analyze the moment when each of the practices should be applied. The balance between the two is the responsibility of the project leader, as well as necessary adaptations throughout the project [12].
3 Research Methodology 3.1 Choices Made The research strategy chosen was the case study. For certain research tasks in the social sciences, the case study is the adequate method [13]. The case study investigates a context from which detailed and intensive knowledge is developed, so, this choice allows to gain a rich understanding of the processes and of the research context. But in a given context, there are a limited number of variables, so the ability to explore, collect data and understand is limited [14].
120
C. Dinis et al.
3.2 Case Study Background The case study researches the “how” and the “why” of a contemporary event [15]. Allows to cover the case context, focus in the case study topics and obtain details. The techniques and procedures for collecting data selected were: observation; document analysis; and questionnaire. The documentation selected to analyze were the documents produced by the team throughout the development of the project. The questionnaire was applied to the team and it is intended to know their opinion in relation to the methodology used for project management. The observation was carried out from a product owner perspective in the sprint planning meetings. In this study the “case” is “the software development in an academic environment using a project management hybrid model”. This project was developed by a group of students of the 4th year of the IMEMIS, doing the course named Information Systems and Technologies Project, which has the weight of 10 ECTS out of a total of 60 ECTS (total ECTS for the 4th year). The scrum team consisted of the researcher, from the MEPM, who assumed the role of Product Owner, and a group of students from the IMEMIS, who assumed the role of the development team, one of them being Scrum Master.
4 Results and Discussion 4.1 Initial Setting and Documents Produced The hybrid model followed was defined by the teachers of the course where this project was integrated in order to provide practical learning experiences of project management to the students. The initial deadline assigned to this project was 15 weeks, however, due to the Covid-19 pandemic, the deadline was extended by one week, adding up to 16 weeks. Two weeks for project initiation, four weeks for planning, eleven weeks for execution and one week for conclusion. Planning and execution were phases that overlapped during the initial two weeks of project execution (first sprint). The 11 weeks allocated to the execution were divided into 5 sprints, each sprint lasting 15 days (2 weeks), except for the last sprint which lasted 3 weeks. The project management support documents produced by the team were: project charter; status report; final status report; execution report; progress report; minutes of meetings and product documentation. 4.2 The Features of the Hybrid Model Used The first research question was: “What are the features of the hybrid model used in this project”? So, this section will answer this question. The distribution of tools and techniques by phases can be seen in Table 1. The project initiation started with the first meeting with the product owner. After the kick-off meeting, the team proceeded to prepare the project charter and started the requirements analysis. The project planning started with the stakeholders’ management. The team started with the identification of the stakeholders. A stakeholders’ analysis was done in order
A PM Hybrid Model of SW Develop. in an Academic Environment
121
Table 1. The tools and techniques used at each phase of the project Phases
Tools and techniques
Initiation
Kick-off meeting and project charter
Planning
Requirement analysis; stakeholders analysis; WBS; work-package stakeholders perspective; activity list; baseline plan; Gantt chart; milestone planning; dependencies; PERT; critical path model; budget work package representation; RAM; quality plan; quality inspection; risks list; ranking of risks; work-package risk perspective; probability and impact matrix; risk response plan; communication plan
Execution
Sprint; backlog; work review; progress report; state report; execution report; minutes of meetings
Conclusion Final report; product documentation
to determine stakeholders’ power and interest and define the appropriate management strategies. A work-package perspective of the role of the stakeholders in the project was created. For the scope management a Work Breakdown Structure (WBS) was done. In schedule management, the team has developed the activity list, the milestone plan, the dependencies between the activities, to obtain the baseline plan and the Gantt chart. Program Evaluation and Review Technique (PERT) was used to calculate the durations, however it was used in very simplified way, because it was an academic environment, using a fixed variation for pessimistic and optimistic scenarios, and then they evaluated the critical path, using the method. To calculate the budget, the team developed a resource list, using again the PERT technique to determine the cost of each resource and the appropriate reserves. They made a representation of the budget work package to understand how the budget is distributed across the different stages of the project. The team created work rules and made an organization chart. They also developed the Responsibility Assignment Matrix (RAM). They also developed a quality plan based on ISO/IEC 25010-2011 and used the Likert Scale to evaluate the result of the functionalities. In order to assess the quality of the platform they carried out one quality inspection. Communication plan where defined for the purpose of the communication, specifying frequency and communication channels. Then, a list of risk was developed, with information about the qualitative levels of impact and probability of occurrence, allowing to evaluate the severity of the risk and the description of the potential impact on the project. They represented the impact of the risks in time and work-packages and did a graphic representation of risks information (Probability and Impact Matrix), determining the ranking of risks. Finally, a response plan was made. For success management, they identified the success factors and characterized them. In the execution the team used the scrum methodology, by using sprint planning and the backlog. During this phase, it was developed a progress report, a status report, an execution report and minutes of meetings. In the conclusion they developed a final report with all the work done, and the product documentation. Although the project planning
122
C. Dinis et al.
was carried out in a waterfall perspective, that plan was not used, and the agile planning (scrum) was followed. Summing up, the hybrid model used in this project is characterized by the application of the traditional model in the initiation, planning and conclusion phases. And the application of the agile model (scrum methodology) in the execution (see Fig. 1).
Fig. 1. Hybrid model used in this project (Authors own elaboration)
4.3 Hybrid Model and Team Performance Evaluation Discussion The second research question was: “Is this hybrid model adequate for this project?” To answer this question, the quality of the model must be defined, so it is necessary to evaluate the quality and success of the project. The quality of a result or performance is defined as the degree of compliance with the requirements of a set of inherent characteristics defined by the stakeholders [3]. In scrum, quality is defined as the ability that products or completed deliveries have to meet the acceptance criteria and to achieve the business value expected by the customer [16]. One of the quality assumptions applied to this project is related to the requirements. The absence of “bugs”, performance and ease of use are also criteria for assessing quality. Considering all the quality evaluation criteria, the software developed was considered to have quality, by the team and professors. When discussing the results, it should be noted that more than half of the time dedicated to the development of this platform was under the influence of the Covid-19 pandemic. Thus, the team was unable to meet in person, namely with the product owner, deprived of adequate infrastructures, which was reflected in an increased difficulty in solving problems. Due to the pandemic, the schedule was extended by one week, which was integrated in the last sprint. As it is an academic environment, the team made some simplifications, one of which when using PERT. For academic purposes, the team used a unique formula to evaluate the pessimistic and optimistic estimates, which eliminates the effects of the uncertainty associated with each activity and therefore had an impact on the calculation of schedule and reserve estimates. This simplification presents itself as a limitation of the project, as the PERT was limited and skewed to a single value for all activities, regardless of the
A PM Hybrid Model of SW Develop. in an Academic Environment
123
level of uncertainty associated with each one, which can be reflected in a less realistic planning and schedule. As previously mentioned, and only for academic reasons, the team carried out the planning of the project’s development using the traditional methodology and used an agile methodology (scrum) for project execution. Comparing the two methodologies, if the team had followed the traditional planning, and as new requirements were integrated in the backlog, the project would have to be replanned. But as it was intended to use an agile methodology, this aspect was addressed by this methodology. Thus, the use of scrum in the development of the project proved to be essential. The team’s perspective on the applied methodology is essential for the analysis of its performance. To this end, a questionnaire was applied to collect, and process data related to their considerations regarding the methodology used. The team had 6 elements. The data collected (see Fig. 2) revealed that four elements (66.7%) consider that 20% of the time used was in the application of traditional methodologies, while only two (33.3%) considered that 50% of the time was used for that. Regarding agile methodologies, four elements (66.7%) consider that these methodologies had a percentage of time used in their application of 20%, while the remaining 2 elements (33.3%) refer to values between 50% and 90% of the time. 20%
20% 50%
(a)
Yes
50%
No
90%
(b)
(c)
Fig. 2. a) % of time spent with waterfall model b) % of time spent with agile model c) Do you consider that this % were adequate?
The technical implementation component was, from the perspective of half of the team (50%), the one that corresponds to the highest percentage of time used, about 60% of the project time. Thus, it is concluded that, in the majority, the team considers that the technical component of the project was the one that absorbed the largest percentage of the project time, and, considering the previous data, that the two methodologies used in the hybrid methodology distributed the time equally among themselves. When questioned if they consider that the percentages of time used were adequate for the organization and management of the project, 5 elements (83.3%) responded yes, while only 1 (16.7%) responded no, claiming that it would be necessary adjustment to the methodologies. The team’s self-assessment was unanimous. The team appreciated its overall performance from very good to excellent, and it is reflected in the evaluation of the project, which was awarded the final grade of 18 (eighteen) values, on a scale from 0 (zero) to 20 (twenty).
5 Conclusions The most common project management practices are related to the traditional methodology (rigid, based on heavy and inflexible processes). In opposition emerged the agile
124
C. Dinis et al.
methodologies (brought flexibility and are suitable for very unpredictable environments), to which Scrum belongs. However, these practices are not exclusively used, so companies that live in environments where a formal process is required, but also more informal practices, adapt these two methodologies, according to their needs, giving rise to a hybrid model. A hybrid model allows for moderate flexibility and unpredictability. This paper answers two research questions. To answer the first, that was “What are the features of the hybrid model used in this project?”, the documents produced by the team were analyzed and observation was made, during team meetings, despite the limitations. The traditional practices, tools and techniques used were: kick-off meeting and project charter; requirements analysis; WBS; stakeholders analysis; work-package stakeholders perspective; activity list; baseline plan; Gantt chart; milestone planning; dependencies; PERT; critical path model; budget work package representation; RAM; quality plan; quality inspection; communication plan; risks list; work-package risks perspective; probability and impact matrix; ranking of risks; graphic presentation of risk information; risk response plan; and final status report. The scrum practices, tools and techniques used were sprint, backlog, work review, progress report, state report, execution report. The initiation, planning and conclusion followed the traditional methodology and the execution followed the agile methodology (scrum). All these elements were used in a correct way, considering the evaluation of the team and professors, having in account the simplifications and conditions. The second question is “Is this hybrid model adequate for this project?”. The choice of the methodology to be used for this project was made by the teachers of the curricular unit, in order to meet all needs and to be used for academic learning purposes. Even though, the use of scrum in the development phase proved to be fundamental and relevant for this project because, due to its flexibility, it enabled the team to overcome the difficulties felt due to the pandemic, namely the impossibility of holding face-to-face meetings, more distant monitoring by teachers and allowed the introduction of new requirements in the product backlog throughout the project. The scrum allowed the team to overcome the difficulties, namely the technical difficulties which, due to the flexibility of the scrum, were realistically planned and overcome without causing relevant delays in the schedule. Increments were delivered at the end of all sprints, and the final product was of good quality. Therefore, the adopted methodology proved to be appropriate. The results support the idea that the hybrid models are effective [12] and that are appropriate for less formal environments such as the academic environment, and adaptable to people with little experience in the area of project management. This paper represents an interesting scientific contribution since it presents the details of an hybrid model used in an academic context that can be replicated in other similar contexts, allowing similar learning experiences and also because we could not find many studies focusing these issues. For future work, it would be interesting to analyze this hybrid model adapted to organizations, see the differences between size and activity sector and, despite there is no set of “silver bullet” practices [12], develop new hybrid frameworks based on practical experience. Acknowledgement. This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.
A PM Hybrid Model of SW Develop. in an Academic Environment
125
References 1. The Standish Group International. CHAOS Summary 2009 (2009) 2. The Standish Group International. CHAOS Report 2015 (2015) 3. PMI. A Guide To The Project Management Body of Knowledge (PMBOK GUIDE), 6th ed. Pennsylvania: Project Management Institute, Inc. (2017) 4. Sommerville, I.: Engenharia de Software, 9th edn. Pearson Education do Brasil, São Paulo (2011) 5. Sutherland, J.: SCRUM: A arte de fazer o dobro do trabalho com metade do tempo, Texto Edit. LeYa, São Paulo (2014) 6. Silva, R.F., Melo, F.C.L.: Modelos híbridos de gestão de projetos como estratégia na condução de soluções em cenários dinâmicos e competitivos. Rev. Bras. Gest. e Desenvolv. Reg. 12(3), 443–457 (2016) 7. IPMA. Referencial de Competências Individuais para Gestão de Projetos, Programas e Portefólios, vol. 4. International Project Management Association (2015) 8. Tereso, A., Ribeiro, P., Fernandes, G., Loureiro, I., Ferreira, M.: Project management practices in private organizations. Proj. Manag. J. 50(1), 1–17 (2019) 9. Ferreira, M., Tereso, A., Ribeiro, P., Fernandes, G., Loureiro, I.: Project management practices in private Portuguese organizations. Procedia Technol. 9, 608–617 (2013) 10. Schwaber, K., Sutherland, J.: The Scrum Guide: The Definitive The Rules of the Game (2017) 11. Romano, B.L., da Silva, A.D.: Project management using the scrum agile method: a case study within a small enterprise. In: 12th International Conference on Information Technology: New Generations, pp. 774–776 (2015) 12. Conforto, E., Barreto, F., Amaral, D.C., Rebentisch, E.: Modelos híbridos. Rev. Mundo Proj. Manag. 64, 10–17 (2015) 13. Flyvbjerg, B.: Five misunderstandings about case-study research. Qual. Inq. 12(2), 219–245 (2006) 14. Saunders, M., Lewis, P., Thornhill, A.: Research Methods for Business Students, 5th ed, vol. 30, no. 1. Pearson Education Limited, Harlow (2009) 15. Yin, R.K.: Case Study Research and Applications: design and methods, 6th edn. Cosmos Corporation, London (2018) 16. SBOK. Conhecimento em ScrumTM (Guia SBOK ) 3rd Edição. SCRUMstudy, Arizona (2017)
Computerised Sentiment Analysis on Social Networks. Two Case Studies: FIFA World Cup 2018 and Cristiano Ronaldo Joining Juventus Nuno Pombo1(B) , Miguel Rodrigues2 , Zdenka Babic3 , Magdalena Punceva4 , and Nuno Garcia1 1
Instituto de Telecomunica¸co ˜es, Universidade da Beira Interior, Covilh˜ a, Portugal {ngpombo,ngarcia}@di.ubi.pt 2 Universidade da Beira Interior, Covilh˜ a, Portugal [email protected] 3 Faculty of Electrical Engineering, University of Banja Luka, Banja Luka, Bosnia and Herzegovina [email protected] 4 University of Applied Sciences and Arts of Western Switzerland, Del´emont, Switzerland [email protected]
Abstract. Sentiment analysis on social networks plays a prominent role in many applications. The key techniques here are how to identify opinions, to classify sentiment polarity, and to infer emotions. In this study, we proposed a sentence-level sentiment analysis based on a microblogging platform like Twitter. Comprehensive evaluation results on realworld scenarios such as the FIFA World Cup 2018, and the Cristiano Ronaldo’s transfer to Juventus in the summer of 2018 demonstrate a correlation between the polarity (negative or positive) and fan’s sentiments. In addition, the evaluation of several machine learning techniques; applied to identify the polarity and related emotions, revealed that the SVM outperforms other models such as Naive Bayes, ANN, kNN, and Logistic Regression. Additional studies should be addressed to evaluate the proposed system on different sport events, and scenarios. Keywords: Social network analysis · Sentiment analysis · Topic modeling · Latent semantic indexing · Polarity · Microblogging · Twitter
1
Introduction
Moscow, Wednesday the 11th of July, Mario Mandzukic scored the goal that broke English hearts on 109 min as Croatia emulated their heroes of 1998 by securing their place in football’s greatest game for the very first time1 . 1
https://www.skysports.com/football/croatia-vs-england/385230.
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 126–140, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_13
Computerised Sentiment Analysis on Social Networks
127
Turin, sunday the 16th of september, Cristiano Ronaldo scored his first goal with Juventus in the 50th minute and added a second goal 15 min later as Juventus defeated Sassuolo 2-12 . Thousands miles away, and sixty-seven days in between these were remarkable events in 2018. It is estimated that there are around 3.5 billion football fans all over the world3 . Congruently, this popularity lead to accumulate the attention of the public, and the media. On the one hand, the explosive growth of social media has given football fans a space for expressing and sharing their thoughts, opinions and emotions regarding the strengths, weaknesses, and events which occur during matches with respect to their team and its opponent. On the other hand, the presence and awareness in social media ensures football players a global reach, and offers the ability to interact with their fans as well as to attract new ones [1–3]. The proliferation of smartphones and tablets, which has boosted content sharing [4]. In addition, online interaction is becoming more real, because people can discuss and give information about football on social media (e.g. Twitter). During the FIFA World Cup 2018 was observed 900 million tweets related with the event4 . This evidenced that Twitter enables people to publish messages to express their interests, favorites, opinions, and emotions towards the football. These messages are called tweets, which are real-time and at most 140 characters. Social media has proven to be a fast growing online tool and is still continuing to grow along with its users, namely, it is part of many individuals everyday lives. In line with this, the recent advances in social media and textual resources paved the way for information retrieval and sentiment analysis in data mining and natural language processing (NLP). In fact, when applied in a large-scale, the sentiment analysis may provide an insight into how people react emotionally intensified events such as win or lose. Sports fans’ emotions change during games that may manifest in their writing on social media. As example, of social media analysis tasks include prediction of political elections, and measurement of economic indicators. However, sentiment analysis over social media are challenging because they are usually based on diverse and unstructured sources of data. Although the existing methods have achieved encouraging performance, it is difficult for them to directly deal with both the heterogeneous content and the interconnection information for effective sentiment analysis. In contrast to standard texts with many words that help gather sufficient statistics, social media content comprise few characters that it may include informal languages such as idioms, jargon, slang, abbreviations or acronyms, and spelling and grammar errors [5]. These factors may lead to a less accurate classification models dependent on the representation of text in documents and/or sentences [6]. Finally, due to the social media wide spread usage and the valuable information that can be produced
2 3 4
http://www.espn.com/video/clip?id=3635549. https://www.topendsports.com/world/lists/popular-sport/fans.htm. https://footballcitymediacenter.com/news/20180613/2403377.html.
128
N. Pombo et al.
from it, is an emergent task to build a sentiment analysis tool which highlights the relevance and timeliness of this study. The main contributions of this work are: (i) a design of a computerised system for sentiment analysis on social media; (ii) a comprehensive experiment which includes a benchmark on several machine learning models to assess their ability in identifying emotions, and (iii) an extensive evaluation of the proposed approach on two real-world scenarios, namely FIFA World Cup 2018, and Cristiano Ronaldo joining Juventus in 100m deal from Real Madrid. The rest of the paper is organized as follows: in Sect. 2 we present some related work on Sentiment Analysis on social media. Section 3 introduces the requirements and the design methodology, and describes it implementations, while in Sect. 4 presents and discusses the results of an experimental evaluation on both scenarios. Finally, Section 5 draws conclusions and discusses future work.
2
Background
Sentiment analysis and opinion mining have been studied since the early 2000s, and several methods have been introduced to analyse emotions and opinions from social media. It should be noted that an emotion is the feeling or reaction that individuals have on a certain event (e.g. angry, fear, joy, ...), and the sentiment is the polarity of such emotion (e.g. positive, or negative). A sentiment enables individuals to convey their emotion through expression. In fact, in their daily life individuals express their emotions using various communications methods through speech, body language, and facial expression just to mention a few. Due to the progress of technology and social media individuals also may express their emotions by text, audio, and video. In a brief, sentiment analysis is the automatic analysis of the attitude of a speaker/writer regarding some topic. Usually, it requires a variety of Natural Language Processing (NLP) tasks. The cornerstone of sentiment analysis are the detection of subjectivity and polarity, the classification of intensity, and the extraction of opinion holder and targets. In our work, we are interested in the polarity’s analysis of a text, that is the evaluation of positiveness or negativeness of the author’s view towards a particular entity. However, since human language is intrinsically complex then several challenges are raised, such as, the interpretation either of words having multiple meaning, or the irony, or the negation [7]. In addition, the social media interaction is transacted by means of short format messages (e.g. Twitter) which impels users to express themselves in a more creative and less intelligible way. Sentiment analysis may be performed at three different granularity levels: (1) document, (2) sentence, and (3) aspect. The broader approach is the document level that aims to determine the overall opinion (e.g. positive, or negative) expressed in a document. On the contrary, the sentence level is focused to get the opinion expressed in a sentence. Finally, the aspect level is related with the topic to which it refers either at document or sentence level. To deal with these different levels, the literature in sentiment analysis presents two types of methods: methods based on semantic rules, and statistical
Computerised Sentiment Analysis on Social Networks
129
methods [8]. First, the semantic rule methods are focused on definition of a set of rules for the extraction and labeling of sentiment-related expressions presented in texts. This activity depends not only on the occurrence of words-related with a sentiment lexicon which includes words and the emotions they convey, but also on the morpho-syntactic analysis which encompasses words form, part of speech tags, among others features. Second, the statistical models are based on either unsupervised or supervised machine learning methods. In the supervised methods it is assumed that the user knows beforehand the classes (i.e. concepts) and the instances of each class. They get the knowledge through a process or training which includes a data set called the training sample which is structured according to the knowledge base supported by human experts, and databases [9]. On the contrary, in the unsupervised learning it is assumed that the user; due to the lack of information, is unaware of the classes. The unsupervised learning treats all variables the same way to determine the different classes based on diverse features observed in the collection of unlabeled data that encompass the sample set [10]. Nowadays, people have a special interest in sharing their opinions on social networks regarding many topics. Due to the proliferation of microblogging platforms, like Twitter, has transmuted the use of the Internet into a tool with which data are shared instantly, the collaboration is more effective, and it enhanced the communication. In line with this, a social information discovery system must assure a fast analysis of social network contents, that are characterized by high variability of topics, and hence should be designed as a general purpose application. Thus, applying machine learning techniques challenges for the real-time (or nearly-real-time) analysis of social network contents, due to the fact that they usually can not generalize well, and they usually require a time-consuming training phase for each domain of interest jointly with a costly manual annotation procedure for every new training set. The existing literature on microblogging platforms; namely on Twitter, sentiment analysis encompasses various features and methods. In [11], authors proposed a predictive Bayesian model for learners tweets emotional state (e.g. joy, fear, anger, sadness, and unknown) classification, and visualization into an elearning environment. In [12], a lexicon was applied to determine the USA sports fan emotional feedback during the FIFA World Cup 2014 tournament. Authors detected several emotions such as anger, fear, joy, sadness, disgust, surprise, trust, and anticipation. In [13], a stock market prediction model was proposed based on financial-related tweets. The Support Vector Machine (SVM), k-Nearest Neighbors (kNN), and a Bayesian model were tested being that the SVM revealed to be the most accurate. In political context [14] implement a aspect-level analysis to detect polarity of the top political topics in Twitter. In [15] a sentiment analysis model was proposed based on the combination of several existing lexicons aiming at to determine the polarity, strength, and emotion of each sentiment. In [16], authors proposed a multi-modal sentiment analysis based on text and image together aiming at to conduct the sentiment classification with Logistic Regression (LR). The same principle was followed by [17]
130
N. Pombo et al.
which proposed method combines visual and textual content using an Artificial Recurrent Neural Network. In [18], a model have been proposed for the sentiment analysis of tweets specific to a topic based on mixed labeled data and its features. Following that principle, [19] proposed a model based on Term Frequency Inverse Document Frequency as feature extraction methods and applied these features to SVM algorithm for classifying tweets. Unsurprisingly, the literature overview confirms that implementing a proper method is a key factor for achieving an effective and timely sentiment analysis.
3
Methods
Our experiments are focused on tweets content, since Twitter offers a microblogging service that allows to collect users’ data. In line with this a data collector was programmed in Python language. This program implements the search criteria and removes duplicated tuples composing an individual data set related with each case study. The world cup scenario encompasses five national teams such as: Belgium, Croatia, England, France, and Portugal. The corresponding data set was populated based on the following queries: (Q1) Belgium National Team World Cup 2018, (Q2) Croatia National Team World Cup 2018, (Q3) England National Team World Cup 2018, (Q4) France National Team World Cup 2018, (Q5) Portugal National Team World Cup 2018, (Q6) France Croatia match World Cup 2018, (Q7) France Belgium match World Cup 2018, (Q8) Croatia England match World Cup 2018, (Q9) England Belgium match World Cup 2018. The queries Q6-Q9 were included in the search due to the fixtures among included national teams. Thus, four additional games were considered, namely the England vs Belgium (0–1), in group stage - day 3, the semi-finals: France vs Belgium (1-0), and Croatia vs England (2-1), and the tournament final: France vs Croatia (4-2). The data obtained from these games are prone to provide relevant and representative information not only due to the clash among different studied teams, but also because they were crucial to determine the world cup winner. On the contrary, the Cristiano Ronaldo scenario was defined based on the following queries: (Q1) Cristiano Ronaldo Real Madrid, and (Q2) Cristiano Ronaldo Juventus. The methodology was complemented in Orange35 encompassing three stages: (1) Processing, (2) Text mining, and (3) Classification. First, the collected data are processed in a three-step algorithm including the stop words removal, the tokenization, and the topic modelling. The key idea is to filter out some common words a priori irrelevant for our experiments. Otherwise, the most common words in a text, such as “is”, “by”, and “the”, will lead to decrease the accuracy rate of the sentiment classification. In addition, a token is imputed in all the remaining words, aspiring to highlight the most significant words. Finally, a topic modelling technique whose main objectives are to determine the abstract topics presented in tweets, and to rank all the words based in its frequency, i.e. number of occurrences, in the text. In our experiments the Latent Semantic Indexing 5
https://orange.biolab.si/.
Computerised Sentiment Analysis on Social Networks
131
Table 1. Lexicon examples for emotional coding Emotions Lexicon sample Anger
Hell, shit, wft
Disgust
Sick, slop, suck
Fear
Afraid, nervous, worry
Joy
Bliss, happy, wonderful
Sadness
Sad, sorrow, depress
Surprise
Amazing, cool, surprise
(LSI) was used. This is because LSI identifies the semantic textual patterns in tweets content and it also identifies the impact of each content on each semantic textual pattern. Second, the text mining includes the acquisition and the segmentation of each message in terms of its polarity. On the one hand, polarity may varies among positive, and negative regarding the inferred sentiment in the processed message. The polarity is directly correlated with expressed emotions in a sentence which means for example when positive refers to a positive emotion. On the other hand, the obtained polarity must be decomposed into different emotions such as: anger, disgust, fear, joy, sadness, and surprise. Examples of how emotions were coded are presented in Table 1. Finally, several classification models were implemented such as: Logistic Regression (LR) [20], Artificial Neural Network (ANN) [21], Naive-Bayes (NB) [22], k-Nearest Neighbor (k-NN) [23], and Support Vector Machine (SVM) [24]. The training set was 70% of the sample, and the remaining 30% encompassed the test set. The data were classified into the different emotions above mentioned. Thus, several metrics assessed each classifier in terms of Precision, Recall, Accuracy, and F-Measure. TP (1) P recision = TP + FP TP (2) Recall = TP + FN TP+TN (3) Accuracy = P+N precision · recall F − M easure = 2 · (4) precision + recall where TP stands for True Positive and TN stands for True Negative. Similarly, FP, and FN means False Positive and False Negative, respectively. Finally, P stand for Positive and N for Negative evaluation. Thus, the Precision refers to the proportion of positive identifications that were correct. The Recall refers to the proportion of positives that were identified correctly. In addition, the Accuracy refers to the proportion of positive identification in the overall dataset. Finally, the F-Measure aims to measure a test’s
132
N. Pombo et al.
performance based on both precision and recall, and it ranges between 0 (lowest precision), and 1 (best precision). The schematic flowchart of the proposed methodology is depicted in Fig. 1.
Fig. 1. Proposed system architecture
In summary then, related with the World Cup case study we formulated two research questions (RQ): – RQ1: What is the national team with a higher average positive polarity of all tweets? – RQ2: There is a correlation between the national team polarity and the fans’ sentiments? On the contrary, concerning the transfer from Cristiano Ronaldo from Real Madrid to Juventus, a single research questions was defined: – RQ3. What kind of sentiments were observed in both fans’ of Real Madrid and Juventus?
4
Results and Discussion
Our experiments were based in 9694 tweets from which 2230 (23%) are related with the FIFA World Cup 2018 and 7464 tweets concerning to the Cristiano Ronaldo’s transfer. This case study encompassed 3574 records (48%) related with Real Madrid fans, and 3890 records related with the Juventus 3890 (52%) records. On the contrary, the tweets related with the FIFA World Cup 2018 are segmented as follows: 288 records associated with the Belgian national team, 212 records related with the French national team, 1136 concerning with the Croatian national team, 92 records concerning with the English national team, and 148 records corresponding with the Portuguese national team. In addition, 354 records were related with several crucial games in the tournament such as: France vs Belgium, France vs Croatia, Croatia vs England, and England vs Belgium. Tweets were represented by two-dimensional vectors (x, y): the value of the positive polarity variable (x) of each tweet was obtained as a number of positive polarity words divided by the total number of words in a tweet. Congruently, a
Computerised Sentiment Analysis on Social Networks
133
value of the negative polarity variable (y) was obtained as the number of negative polarity words divided by the total number of words in a tweet, both after text processing. In line with this, Fig. 2 gives a scatter plot of tweets as (x, y) vectors. Tweets in scatter plot are just all over the place and there does not appear to be any linear trend. Nonetheless, for further analysis we define a slope (r) of the line that best fit data in a scatter diagram is based on the mean of the negative polarity variable divided by the mean of positive polarity variable. Thus, as depicted in Fig. 2, related with polarity of each national team in terms of tweets, revealed that the French team got the most positive polarity ones (r = −0.46). In the case when a slope is negative (negative correlation), increasing of values of positive polarity variable decreases values of negative polarity variable in a scatter plot, and vice versa. That means that the number of negative polarity words in tweets decreases as the number of positive polarity words increase. That also means that the number of positive polarity words in tweets decreases as the
(a) Belgium
(b) Croatia
(c) England
(d) France
(e) Portugal
Fig. 2. National teams scattered plot
134
N. Pombo et al.
number of negative polarity words increase. Congruently, when the r − value is positive, means that the correlation between the variables is positive. This is observed with the Belgium national team (r = 0.02). Contrarily, was observed a positive polarity in all studied matches: Belgium vs England (r = −0.13), France vs Belgium (r = −0.13), England vs Croatia (r = −0.09), and France vs Croatia (r = −0.06). In addition, Fig. 3 depicts the scattered plot related with the Cristiano Ronaldo’s transfer in terms of supporters perspectives of both Real Madrid, and Juventus. In these perspectives a positive polarity was observed as evidenced by r = −0.19, and r = −0.29 related with Real Madrid and Juventus, respectively.
(a) Real Madrid perspective
(c) Juventus perspective
Fig. 3. Scattered plot of the Cristiano Ronaldo’s transfer
Comparison of different classification models’ performance to the polarity detection is shown in Tables 2, 3, and 4. The SVM is the most accurate classifier among the five national teams ranging from 0.797 (Croatia) to 0.968 (France). In addition, its precision varies from 0.838 to 0.971 related with the Belgian and French national teams, respectively. Similarly, the SVM, as depicted in Table 3, obtains also the best performance related with the analysed world cup games
Computerised Sentiment Analysis on Social Networks
135
ranging from 0.882 (Croatia x England) to 0.938 (England x Belgium). In both situations, the LR model produces the same accuracy as the SVM. On the contrary, the LR performs worst in the France x Belgium, and France x Croatia matches with an accuracy of 0.759. However, the SVM reveals higher precision than the LR whose metrics vary from 0.776 to 0.944, against the interval 0.897 to 0.944 observed in the SVM. Finally, when different classifiers are applied in the Cristiano Ronaldo’s transfer dataset, it is observable that the kNN (0.773) and the Naive Bayes (0.707) perform better in Juventus and Real Madrid fans, respectively. In both situations the SVM reveals its ineffectiveness to classify this dataset as evidenced by the reduced accuracy (Juventus fans: 0.453, and Real Madrid fans: 0.485), and precision (Juventus fans: 0.480, Real Madrid fans: 0.505). As depicted in Fig. 4 the most usual emotion observed during the analysed world cup games is the joy ranging from 38.98% to 64.81%, followed by the surprise whose varies from 16.95% to 31.63%. The fear is also presented in the detected emotions ranging from 3.70% to 32.20%. The anger, disgust, and sadness are reduced since the maximum percentage observed is 6.78%, 1.02%, and Table 2. Performance of classification models for National Teams’ polarity National team Classification model Metrics Precision Recall Accuracy F-Measure Belgium
Croatia
England
France
Portugal
SVM
0.916
0.907
0.907
0.901
Naive Bayes
0.892
0.884
0.884
0.878
kNN
0.898
0.884
0.884
0.876
LR
0.884
0.884
0.884
0.877
ANN
0.338
0.093
0.093
0.055
SVM
0.838
0.797
0.797
0.791
Naive Bayes
0.750
0.744
0.744
0.741
kNN
0.615
0.503
0.503
0.429
LR
0.821
0.791
0.791
0.786
ANN
0.831
0.785
0.785
0.777
SVM
0.965
0.963
0.936
0.961
Naive Bayes
0.965
0.963
0.936
0.961
kNN
0.890
0.926
0.926
0.908
LR
0.965
0.963
0.963
0.961
ANN
0.789
0.852
0.852
0.812
SVM
0.971
0.968
0.968
0.968
Naive Bayes
0.937
0.937
0.937
0.937
kNN
0.939
0.937
0.937
0.936
LR
0.936
0.937
0.937
0.936
ANN
0.850
0.825
0.825
0.827
SVM
0.938
0.932
0.932
0.928
Naive Bayes
0.884
0.886
0.886
0.885
kNN
0.865
0.864
0.864
0.862
LR
0.865
0.864
0.864
0.862
ANN
0.835
0.841
0.841
0.832
136
N. Pombo et al.
Table 3. Performance of classification models for World Cup Games’ polarity Game
Classification model Metrics
France x Belgium
SVM
0.897
0.897
0.897
0.895
Naive Bayes
0.871
0.793
0.793
0.793
Precision Recall Accuracy F-Measure
France xCroatia
England xCroatia
kNN
0.846
0.828
0.828
0.815
LR
0.776
0.759
0.759
0.749
ANN
0.711
0.793
0.793
0.749
SVM
0.897
0.897
0.897
0.895
Naive Bayes
0.871
0.793
0.793
0.793
kNN
0.846
0.828
0.828
0.815
LR
0.776
0.759
0.759
0.749
ANN
0.711
0.793
0.793
0.749
SVM
0.912
0.882
0.882
0.885
Naive Bayes
0.809
0.765
0.765
0.770
kNN
0.840
0.824
0.824
0.827
LR
0.912
0.882
0.882
0.885
ANN
0.882
0.824
0.824
0.826
England xBelgium SVM
0.944
0.938
0.938
0.937
0.900
0.875
0.875
0.873
Naive Bayes kNN
0.864
0.812
0.812
0.806
LR
0.944
0.938
0.938
0.937
ANN
0.250
0.500
0.500
0.333
Table 4. Performance of classification models for Cristiano Ronaldo Transfer’s polarity
Perspective
Classification model Metrics
Juventus
SVM
0.480
0.453
0.453
Naive Bayes
0.795
0.759
0.759
0.759
kNN
0.837
0.773
0.773
0.774
LR
0.832
0.772
0.772
0.774
ANN
0.833
0.772
0.772
0.774
Real Madrid SVM
0.505
0.485
0.485
0.493
Naive Bayes
0.756
0.707
0.707
0.647
kNN
0.754
0.700
0.700
0.632
LR
0.798
0.705
0.705
0.632
ANN
0.615
0.645
0.645
0.599
Precision Recall Accuracy F-Measure 0.403
5.10% respectively. These results; namely the “joy” that is broader expressed in tweets, are consistent with the positive polarity obtained on all the national teams. This may be interpreted as a generalized sentiment of happiness in the fans since their favourite teams reached an advanced stage in the tournament. Moreover, three matches presented an identical trend in terms of emotions;
Computerised Sentiment Analysis on Social Networks
137
(a) World Cup Games
(b) Cristiano Ronaldo’s Transfer
Fig. 4. Observed emotions
France x Belgium, France x Croatia, and England x Belgium. On the contrary, in the clash between Croatia and England it is observed different emotions since the “fear” is higher, and the “joy” is lower compared with the other matches. This may be interpreted due to the expectation of a well-balanced match between these teams. Finally, related with the Cristiano Ronaldo’s transfer we can observe a similarity in terms of “anger”, “disgust”, “fear”, and “joy”. The emotion “surprise” is greater for Juventus fans than those are supporters of the Real Madrid. On the contrary Real Madrid fans reveal an higher “sadness” for the leaving of Cristiano Ronaldo. In this scenario the significance of “fear”, and “sadness” emotions is congruently with the negative polarity observed related with the Cristiano Ronaldo’s transfer. In addition, it seems that the Real Madrid fans are less surprised with the transfer maybe due to the speculation and/or rumours on Ronaldo’s future promoted by the Spanish media.
138
N. Pombo et al.
The main goal of this study was achieved since our experiments revealed the suitability of the proposed method to classify polarity and emotions based on data collected on the social media. Despite its contributions, this study has certain limitations. On the one hand it was observed an unbalanced data due to the heterogeneity, and the different granularity of data sources. On the other hand, it was observed that the information embedded in the social network is abundant but disorganised in nature. In addition, the proposed sentiment analysis method is also limited in the detection of metaphors, sarcasm, irony, and ambiguities, just to mention a few, due to the intrinsic complexity of the human language.
5
Conclusions
The goal of this work is the development of a computerised system for information discovery from multiple sources of social media, which allows for the sentiments analysis of users. Furthermore, the paper introduces the sentiment analysis observed in two relevant sport events happened in 2018, the FIFA World Cup 2018, and the spotlighting transfer of Cristiano Ronaldo from Real Madrid to Juventus. This paper presented a sentence-level sentiment analysis based on a microblogging platform like Twitter. Our experiments revealed that France is the national team with the higher average positive polarity. There was observed a correlation between polarities of each national team and fan’s sentiments, namely the positive polarity resulted in the observed happiness. In addition, “fear”, “sadness”, and “surprise” were emotions observed by supporters of either Real Madrid or Juventus when Cristiano Ronaldo joined Juventus. Moreover, we conducted an extensive experiment to test the performance of five machine learning algorithms (SVM, Naive Bayes, kNN, LR, and ANN) in recognizing polarity and emotions in football conversations on social media. The results have shown that the SVM has achieved the best performance and thus revealed its robustness and consistency. As future work we are going to study different social media platforms, namely those where social content also may includes images. In addition, we plan to extend the proposed system to other languages, and also to implement enhanced algorithms that may enable sarcasm, and/or irony detection. Finally, complementary studies should be performed to evaluate the proposed system on other scenarios, and other machine learning techniques. Acknowledgments. This work is funded by FCT/MCTES through national funds and when applicable co-funded EU funds under the project UIDB/EEA/ 50008/2020. This article is based upon work from COSTNET (COST Action CA15109), supported by COST (European Cooperation in Science and Technology). More information in www.cost.eu
References 1. Aloufi, S., Saddik, A.E.: Sentiment identification in football-specific tweets. IEEE Access 6, 78609 (2018). https://doi.org/10.1109/ACCESS.2018.2885117
Computerised Sentiment Analysis on Social Networks
139
2. T. Tiago, F. Tiago, S.D. Faria, J.P. Couto, Who is the better player? off-field battle on facebook and twitter, Business Horizons 59(2), 175(2016). https://doi. org/10.1016/j.bushor.2015.11.003. http://www.sciencedirect.com/science/article/ pii/S0007681315001457 3. Korzynski, P., Paniagua, J.: Score a tweet and post a goal: social media recipesfor sports stars. Bus. Horiz. 59(2), 185 (2016). https://doi.org/10.1016/j.bushor.2015. 11.002. http://www.sciencedirect.com/science/article/pii/S0007681315001445 4. Nuno, P., Pedro, A., Joaquim, V.: Evaluation of a ubiquitous and interoperable computerised system for remote monitoring of ambulatory post-operative pain: a randomised controlled trial. Technology and HealthCare, 63–75 (2014). https:// doi.org/10.3233/THC-130774 5. Giahanou, A., Crestani, F.A.: Like it or not: a survey of Twitter sentiment analysis methods. ACM Comput. Surv. 49, 28:1 (2016) 6. Zdravevski, E., Lameski, P., Trajkovik, V., Chorbev, I., Goleva, R., Pombo, N., Garcia, N.M.: Automation in systematic, scoping and rapid reviews by an NLP toolkit: a case study in enhanced living environments, pp. 1–18. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-1075291 7. Diamantini, C., Mircoli, A., Potena, D., Storti, E.: Social information discovery enhanced by sentiment analysis techniques, Future Gener. Comput. Syst. 95, 816 (2019). https://doi.org/10.1016/j.future.2018.01.051. http://www. sciencedirect.com/science/article/pii/S0167739X17308634 8. Clavel, C., Callejas, Z.: Sentiment analysis: from opinion mining to human-agent interaction, IEEE Trans. Affect. Comput. 7(1), 74 (2016). https://doi.org/10.1109/ TAFFC.2015.2444846 9. Pombo, N., Rebelo, P., Ara´ ujo, P., Viana, J.: Design and evaluation of a decision support system for pain management based on data imputation and statistical models, Measurement 93, 480 (2016). https://doi.org/10. 1016/j.measurement.2016.07.009. http://www.sciencedirect.com/science/article/ pii/S0263224116303682 10. Pombo, N., Garcia, N., Bousson, K.: Handbook of Research on Artificial Intelligence Techniques and Algorithms, by V. Pandian. IGI Global, Hershey, pp. 183–203 (2015) 11. Ravichandran, M., Kulanthaivel, G.: Twitter sentiment mining (TSM) framework based learners emotional state classification and visualization for e-learning system. J. Theoretical Appl. Inf. Technol. 69, 84 (2014) 12. Yu, Y., Wang, X.: World cup 2014 in the twitter world: a big data analysis of sentiments in u.s. sports fans’ tweets. Comput. Human Behav. 48, 392 (2015). https://doi.org/10.1016/j.chb.2015.01.075. http://www.sciencedirect.com/ science/article/pii/S074756321500103X ˇ 13. Smailovi´c, J., Grˇcar, M., Lavraˇc, N., Znidarˇ siˇc, M.: Stream-based active learning for sentiment analysis in the financial domain. Inf. Sci. 285, 181 (2014). https://doi.org/10.1016/j.ins.2014.04.034. http://www.sciencedirect.com/science/ article/pii/S0020025514004885. Processing and Mining Complex Data Streams 14. Rill, S., Reinel, D., Scheidt, J., Zicari, R.V.: Politwi: early detection of emerging political topics on twitter and the impact on concept-level sentiment analysis. Knowl. Based Syst. 69, 24 (2014). https://doi.org/10.1016/j.knosys.2014.05.008. http://www.sciencedirect.com/science/article/pii/S0950705114001920 15. Bravo-Marquez, F., Mendoza, M., Poblete, B.: Meta-level sentiment models for big social data analysis. Know.-Based Syst. 69(1), 86 (2014). https://doi.org/10.1016/ j.knosys.2014.05.016
140
N. Pombo et al.
16. Wang, M., Cao, D., Li, L., Li, S., Ji, R.: Proceedings of International Conference on Internet Multimedia Computing and Service, ICIMCS 2014, pp. 76:76–76:80. ACM, New York (2014). 10.1145/2632856.2632912. https://doi.org/ 10.1145/2632856.2632912 17. You, Q., Cao, L., Jin, H., Luo, J.: Proceedings of the 24th ACM International Conference on Multimedia, MM 2016, pp. 1008–1017. ACM, New York (2016). https://doi.org/10.1145/2964284.2964288 18. Liu, S., Cheng, X., Li, F., Li, F.: Tasc:topic-adaptive sentiment classification on dynamic tweets. IEEE Trans. Knowl. Data Eng. 27(6), 1696 (2015). https://doi. org/10.1109/TKDE.2014.2382600 19. Windasari, I.P., Uzzi, F.N., Satot, K.I.: 2017 4th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE) (2017), pp. 266–269. https://doi.org/10.1109/ICITACEE.2017.8257715 20. Harrell Jr., F.E., Lee, K.L., Califf, R.M., Pryor, D.B., Rosati, R.A.: Regression modelling strategies for improved prognostic prediction, Stat. Med. 3(2), 143 (1984). https://doi.org/10.1002/sim.4780030207. https://onlinelibrary.wiley.com/ doi/abs/10.1002/sim.4780030207 21. Haykin, S.: Neural Networks: A Comprehensive Foundation, 2nd edn. Prentice Hall PTR, Upper Saddle River (1998) 22. Hand, D.J., McLachlan, G.J., Basford, K.E.: Mixture models: inference and applications to clustering. Appl. Stat. 38(2), 384 (1989). https://doi.org/10.2307/ 2348072 23. Dudani, S.A.: The distance-weighted k-nearest-neighbor rule. IEEE Trans. Syst. Man Cybern. SMC-6(4), 325 (1976). https://doi.org/10.1109/TSMC.1976. 5408784 24. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273 (1995). https://doi.org/10.1023/A:1022627411411
Modelling Academic Dropout in Computer Engineering Using Artificial Neural Networks Diogo M. A. Camelo1 , Jo˜ ao C. C. Santos1 , Maria P. G. Martins1,2(B) , and Paulo D. F. Gouveia1 1
2
Instituto Polit´ecnico de Bragan¸ca, Campus de Santa Apol´ onia, 5300-253 Bragan¸ca, Portugal CISE – Electromechatronic Systems Research Centre, University of Beira Interior, Cal¸cada Fonte do Lameiro, 6201-001 Covilh˜ a, Portugal [email protected]
Abstract. School dropout in higher education is an academic, economic, political and social problem, which has a great impact and is difficult to resolve. In order to mitigate this problem, this paper proposes a predictive model of classification, based on artificial neural networks, which allows the prediction, at the end of the first school year, of the propensity that the computer engineering students of a polytechnic institute in the interior of the country have for dropout. A differentiating aspect of this study is that it considers the classifications obtained in the course units of the first academic year as potential predictors of dropout. A new approach in the process of selecting the factors that foreshadow the dropout allowed isolating 12 explanatory variables, which guaranteed a good predictive capacity of the model (AUC = 78.5%). These variables reveal fundamental aspects for the adoption of management strategies that may be more assertive in the combat to academic dropout. Keywords: Educational data mining Academic dropout · Predictive model
1
· Artificial neural network ·
Introduction
Due to the importance of education for the progress of society as well as for citizens’ well-being and prosperity, effective measures are required to combat high school dropout in the current context of higher education institutions (HEI). In the majority of studies on the portuguese HEI, about the factors of success and academic failure, an exploratory and descriptive record is more privileged than an explanatory and propositional record [1] which indicates a poor prognosis for the design of effective early interventions. In order to assist the institutional decision-makers in this process, a predictive classification model is presented that allows the prediction, at the end of the first school year, of the dropout tendency of a student of computer engineering (CE) of the Polytechnic Institute c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 141–150, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_14
142
D. M. A. Camelo et al.
of Bragan¸ca (IPB). The main explanatory factors for dropping out were identified with the help of data mining techniques and artificial neural networks (ANNs), applied to records of 653 CE students of the IPB, characterized by 22 factors of students’ academic and demographic dimensions and, in particular, based on intermediate (un)success patterns of their academic career. By presenting a model with a considerable success rate (AUC = 78.5%), when applied to real data, the knowledge obtained may prove crucial to improve decision-making to combat student dropout. After this introduction, the present paper is composed of the following sections Sect. 2 – outline of related studies; Sect. 3 – presentation of the methodology and of the data model developed; Sect. 4 – presentation and discussion of results of the prediction model proposed; Sect. 5 – final conclusions and perspectives of future work.
2
Educational Data Mining
The use of data mining techniques in modelling performance and academic dropout is a promising area of research called Educational Data Mining (EDM). Major reviews of the state of the art [2–5] prove the importance and usefulness of EDM, as a tool for analysis and management support. According [6] the main goal of EDM is to generate useful knowledge which may ground and sustain decision-making targeted at improving student communities’s learning as well as educational institution’s efficiency. In fact, most EDM studies are subject to performance prediction and academic dropout, which identify the factors that can influence them and constitute fundamental aspects in the definition of management strategies focused on promoting success and preventing school dropout [5]. Within this context, the main aim of the studies [7–9] was to conduct a review to identify the most commonly studied factors that affect the students’ performance, as well as, the most common data mining techniques applied to identify these factors. In the study by [7] was pointed out that the most common factors are grouped under four main categories, namely, students’ previous grades and class performance, e-Learning activity, demographics, and social information. Additionally, Shihari [8] showed that the cumulative grade point average, and internal assessments (marks after entering higher education such exams, assignments, etc.) are the most frequent attributes used for predicting the student’s performance. Furthermore, other important attributes were also identified, including student’s demographic and external assessments (pre-university achievement classifications), extra-curricular activities, high school background, and social interaction network. Also, from the analysis of 10 EDM studies with an emphasis on foreseeing academic dropout, the author [9] concluded that in most cases, the average access to higher education, the level of education, and the parents’ profession and poor methodology are the main factors that affect academic dropout. Among the DM techniques most used these studies concludes that the Decision Tree, Artificial Neural Networks (ANN), Naive Bayes (NB) were, in descending order, the DM techniques most often used for forecasting purposes in EDM [7,8].
Modelling Academic Dropout Using Artificial Neural Networks
143
Regarding school dropout, the scope of the present study, the authors of the study [10], through the development of models that integrated the Random Forest (RF), support vector machine (SVM) and ANN techniques, concluded that there are essentially 2 factors related to students’ curricular context which mostly account for the academic dropout of the undergraduates attending the 50 degree courses taught in the institution used as a case study. The models presented, whose predictive accuracy was 76% when applied to new data, were developed with records of 3344 students, characterized by more than 4 dozens of potential predictors of dropout concerning students’ academic and demographic dimension, and their socio-economic status. In the study [11] considering demographic, socio-economic and learning performance information of 972 students at Riddle Aeronautical University, processed by learning algorithms, Naive Bayes, k-Nearest neighbors, RF, ANN, decision trees and logistic regression, concluded that the grades average, the classifications in the entrance exams, the average grade of the first year of college and the financial contribution provided by the family are, in descending order, the factors that best explain the student dropout. Classification rules and the algorithms, Naives Bayes, support vector machines, instance based Lazy Learning and Jrip were used in the study by [12], which results showed, with success rates above 80%, that it was possible to identify the propensity to drop out of high school students in Mexico. In the development and validation of the proposed model, information from 419 students was used in a very comprehensive way in academic and socioeconomic dimensions.
3 3.1
Data and Methodology Data Model
The dataset that supports the creation of the model capable of predicting the academic dropout of the students of CE at Institute Politecnic of Bragan¸ca was obtained from the Information System of that same educational institution and contains records of 635 students, enrolled in CE between 2006 and 2019. In the data selection and pre-processing phases, which preceded the applications of ANNs, there was a need to clean and transform the data. In particular, removing the enrollment of students from the pre-Bologna study plans, of those who have changed their course or educational institution, of those who are still enrolled in this school year (still of undefined outcome) and also, for each student, data was deleted from all enrollments subsequent to the first (since the prediction is only made with information available until the end of the 1st year). The target variable of prediction is of boolean type (1 or 0), whose value can take one of the following meanings: ‘dropout’, if the student did not complete the degree, or ‘not dropout’ if they concluded the degree. All students who did not have a valid registration in the year 2019/2020 and who, simultaneously, had not yet completed their course, had their enrolment classified as ‘dropout’. As the main operations of variable transformation, it was necessary to normalize
144
D. M. A. Camelo et al.
the predictors of the numerical type, using, for this purpose, the Min-Max scaling approach, through which all the values of the numerical variables became included between 0 and 1. There was also the need to calculate new variables, such as, for example, the predictive variable ‘n ects done’, which was obtained from the curricular units in which the student was approved. After the cleaning of the data and the performing of other pre-processing tasks, each of the IE students was characterized by a total of 22 potential explanatory variables of dropout, categorized into 3 main groups: demographic (D), curricular (C) and of matriculation (M). The variables in question are represented in Table 1. Table 1. Predictive variables considered in the abandonment prevision. Attribute
Type
Category Meaning
gender
nominal D
student’s gender
nationality
nominal D
student’s nationality
cod district
nominal D
code of the district where the student lives
cod district n
nominal D
code of the district where the student was born
ALGA
discrete C
grade in Linear Algebra and Analytical Geometry
CI
discrete C
grade in Calculus I
F
discrete C
grade in Physics
PI
discrete C
grade in Programming I
SD
discrete C
grade in Digital Systems
AC
discrete C
grade in Computer Architecture
CII
discrete C
grade in Calculus II
MD
discrete C
grade in Discrete Mathematics
PII
discrete C
grade in Programming II
age
discrete D
age at first registration
phase
ordinal
phase of the entry into the degree
cod type entry
nominal M
code of the type of entry of the student at the degree
scholarship
logical
C
where the student had a scholarship
associative leader logical
C
where the student was associative leader
M
cod freq type
nominal C
code of the type of frequency of the student
cod status
nominal C
type of the status of the student
n ects done auto
discrete M
number of credits done automatically on degree entry
n ects done
discrete C
number of credits completed done
3.2
Methodology
In the development of the model proposed, an approach using Artificial Neuronal Networks was chosen because of the ability and proven effectiveness demonstrated in some studies in the area of EDM (e.g. [3,4,7]).
Modelling Academic Dropout Using Artificial Neural Networks
145
All of the computational calculations inherent to the study are performed in R, in the Rstudio environment and specific packages for applying ANNs are used as an extension to R itself. All the pre-processing that preceded the application of the ANNs in the development of the new predicting model was carried out in Mysql. In the assumption of the established objectives, a random partitioning of the dataset is performed in 3 subsets: training, validation and testing. Such partitioning is characterized in Table 2. Table 2. Dimensions of the subsets used for training, validation and testing. Total Train Validation Test 653
391
130
132
With the training and validation subsets, and the processes of learning and refining, respectively, the ANNs is performed. Once trained and properly refined, from the initial set of 22 potential predictive variables, the identification of the most explanatory factors of academic dropout is conducted, through the selection of those that show greater impact to the intended prediction (feature selection). For that purpose, a progressive selection method (forward search) is adopted, in which the variables are selected one by one, in an iterative process, always joining to the ones previously selected the remaining ones that lead to a greater increment. Of course, the process ends when this increment becomes null or negative. After the set of variables most explanatory of dropout has been identified, the influence of each of them in the target variable of prediction is measured by means of a simple technique of sensitivity analysis, which attempts to define the importance of each variable by measuring the loss of accuracy resulting from its non-inclusion in the model. The subset of data left for testing is used only in the final assessment of the model in order to measure its generalization capacity. As a predictive evaluation metric for the different ANN configurations, the Area Under ROC Curve (AUC) is used, in which the ROC curves (Receiver Operating Characteristic) form a graph that illustrates the performance of a binary classification model, based on the specificity and sensitivity of the same classifier model.
4
Implementation and Results
For an easier understanding of the results presented in the next sections, a scheme is shown in Fig. 1 intended at illustrating the 2 predictive classification models developed and, in particular, the different subsets of predictors that support them.
146
D. M. A. Camelo et al.
Fig. 1. Illustrative diagram of the predictive model.
4.1
Training and Refinement of ANNs with All the Independent Variables
In order to obtain a first perception of the predictive capacity of the model, the first step consisted of training the model with the complete set of independent variables available. For easier referencing, the predictive model that integrates all these variables will be named mp22. As recommended in the literature, and with the objective of maximizing the predictive capacity, the model was refined, through the evaluation of its performance for different values of its most important hyperparameters. In the specific case of the present study, the hyperparameters used were the size (number of neurons of the hidden layer of ANNs) and Decay (learning rate decay). The 10 best values obtained in this refinement process can be found in Table 3. The prediction model was then supported by 1 an ANN of 6 neurons (in its hidden layer) and with a decay rate of 10− 3 . Table 3. The 10 best refinement results. Size
6
4 −1 3
Decay 10 AUC
4.2
8 −1 3
10
2 −1 3
10
3 −1 3
10
3 −2 3
10
10 −1 3
10
−1 3
10
20 −1 3
10
5
15 −1 3
10
1
10− 3
0.847 0.846 0.846 0.846 0.844 0.844 0.844 0.843 0.843 0.843
Selection of the Main Explanatory Factors of Dropout
After completing the training and refinement phases of the predictive model, an adjustment of the set of variables that support the model was made with the same training and validation data, by selecting, through the forward search method, those that had proven to be the most accountable ones for dropout.
Modelling Academic Dropout Using Artificial Neural Networks
147
The result obtained by the iterative process of variable selection is reported in Table 4. From the table, it is possible to conclude that the number of variables most explanatory of academic dropout in CE is 12, from which 4 belong to the students’ curricular dimension, 3 represent matriculation data and 5 are of demographic nature. This new model will henceforth be called mp12. Table 4. Variables selected by applying the forward search method.
4.3
Order Attribute
Type
Category
1
phase
ordinal
M
2
AC
discrete C
3
age
discrete D
4
n ects done
discrete C
5
cod type entry
nominal M
6
F
discrete C
7
cod district n
nominal D
8
cod district
nominal D
9
n ects done auto discrete M
10
gender
nominal D
11
nationality
nominal D
12
MD
discrete C
Evaluation of the Generalization Capacity of the Model Found
After the training, refinement and feature selection processes were completed, the new mp12 model was evaluated with the test subset in order to verify its true generalization capacity. The results of the respective previsions are shown in Table 5, which also includes, for comparison purposes, the results previously obtained with the validation data. Table 5. Comparison of the models performance. model AUC (validation) AUC (test) mp22 84.7%
76.7%
mp12 85.9%
78.5%
The comparison between the AUC obtained with the test data and what had been obtained with the validation data allows seeing that, as expected, the AUC values decrease – 8% in the model that uses all the variables available (mp22 ) and
148
D. M. A. Camelo et al.
7.4% in the model where the most explanatory variables were selected (mp12 ). These results reveal that the reduction of variables has greatly increased the generalization capacity of the model. In addition to having fully achieved what is the supreme purpose of this type of study (maximizing the capacity for generalization), the reduction of variables also translates a set of other advantages when the model is applied in real contexts, namely, by allowing to identify the main factors that explain the target predictive variables, helping eliminate redundant variables, decreasing the computational complexity of the model and facilitating its application in real contexts. In order to have another perspective on the behavior of each dropout prediction model, the respective ROC curves are overlapped in Fig. 2. Although the differences in performance between the two models are not easily perceptible in the figure, it is indeed possible to notice that the curve that is closest to the vertex characterizing the condition of optimality (specificity = 100% and sensitivity = 100%) is the model with the fewest variables (mp12 ).
Fig. 2. Final ROC curves of the models.
4.4
Relative Importance of the Main Explanatory Factors
After finding the set of variables which are most explanatory of dropout, and in order to distinguish the influence of each of them, an order of relevance was established among them, according to their importance for the prediction. The Fig. 3 shows the values of the relative importances in question. It is relevant to point out the great influence that the ‘entry phase’, the moment in which the student entered the course, has on the explanation of academic dropout. Other variables that also reveal great explanatory power are the classification in the course unit of ‘computer architecture’ and the ‘student’s
Modelling Academic Dropout Using Artificial Neural Networks
149
Fig. 3. Importance of the predictive variables.
age’ at the time of entering the course. It would be expected, however, that the variables ‘number of ECTS completed at the end of the first school year’ and ‘number of ECTS automatically credited upon entry’ would have shown greater explanatory power regarding dropout. The reason why this did not occur is most likely due to the high degree of correlation between these variables and other potential predictors of dropout, such as those related to the classifications obtained in certain course units.
5
Conclusions and Future Work
With the application of data mining techniques and, in particular, of artificial neural networks, a model of classification was developed, which allows predicting, at an early stage of the academic career, whether an IPB computer engineering student will be an academic dropout or a graduate. From a set of 22 predictive factors of dropout, the process of knowledge discovery in a database led to the conclusion that 12 factors are the most explanatory of dropout in CE. Subsequently, a sensitivity analysis technique was applied to the model with the 12 variables in order to calculate the relative importance of each one of them, after which it was found that the ‘phase of entry into the course’, ‘the classification in the course unit of computer architecture’ and the ‘student’s age at the moment of entering the course’ are the 3 variables with greater influence on the dropout of CE students. Since the classification obtained in the course unit of computer architecture is a strong predictor of dropout, some possible future work to be considered is the development of a predictive regression model that allows estimating students’ grades in this course unit. With such knowledge, institutional decision-makers could define ways to promote the educational success of students for whom a lower performance may have been estimated by the model. In order to complement this research, the cross-validation technique could be used, for instance, in the partitions of training, validation and testing. The results obtained in this study may support the definition of effective measures to combat academic dropout in the Computer engineering degree course, where
150
D. M. A. Camelo et al.
dropout rates are worrisome, since they reach about 50% in the dataset analyzed. For instance, setting up study support groups for the course unit of computer architecture might be one of the guidelines to suggest. Since EDM literature demonstrates that decision trees and artificial neural networks are the most widely used learning techniques in student modelling, it will be possible to explore the relevance of using other techniques in future work, that also often display very competitive results, such as committee-based ones, also called ensembles methods or mixture of experts. Acknowledgments. This work was supported by the Portuguese Foundation for Science and Technology (FCT) under Projects UIDB/04131/2020 and UIDP/04131/2020.
References 1. Costa, A., Lopes, J., Caetano, A.: Percursos de estudantes no ensino superior. Fatores e processos de sucesso e insucesso. Lisboa: Mundos Sociais. (2014) 2. Bakhshinategh, B., Zaiane, O.R., Elatia, S., Ipperciel, D.: Educational data mining applications and tasks: A survey of the last 10 years. Educ. Inf. Technol. 23(1), 537–553 (2018) 3. Pe˜ na-Ayala, A.: Educational data mining: a survey and a data mining-based analysis of recent works. Expert Syst. Appl. 41(4), 1432–1462 (2014) 4. Romero, C., Ventura, S.: Educational data mining: a review of the state of the art. IEEE Trans. Syst. Man Cybern. Part C (Applications and Reviews) 40(6), 601–618 (2010) 5. Martins, M.P.G., Migueis, V.L., Fonseca, D.S.B.: Data mining educacional: uma revis˜ ao da literatura. In: 13th Iberian Conference on Information Systems and Technologies (CISTI). Institute of Electrical and Electronics Engineers (2018) 6. Martins, M.P.G., Migueis, V.L., Fonseca, D.S.B., Alves, A.: A data mining approach for predicting academic success-a case study. Int. Conf. Inf. Technol. 918, 45–56 (2019) 7. Saa, A.A., Al-Emran, M., Shaalan, K.: Factors affecting students’ performance in higher education: a systematic review of predictive data mining techniques. J. Technol. Knowl. Learn. 4, 567–598 (2019) 8. Shahiri, A.M., Husain, W., et al.: A review on predicting student’s performance using data mining techniques. Procedia Comput. Sci. 72, 414–422 (2015) 9. Kumar, A.D., Selvam, R.P., Kumar, K.S.: Review on prediction algorithms in educational data mining. Int. J. Pure Appl. Math. 118, 531–537 (2018) 10. Martins, M.P.G., Migueis, V.L., Fonseca, D.S.B., Gouveia, P.D.F.: Previs˜ao do abandono acad´emico numa institui¸ca ˜o de ensino superior com recurso a data mining. Revista Ib´erica de Sistemas e Tecnologias de Informa¸ca ˜o. Associa¸ca ˜o Ib´erica de Sistemas e Tecnologias de Informaca, vol. E28, pp. 188–203 (2020) 11. Lehr, S., Liu, H., Kinglesmith, S., Konyha, A., Robaszewska, N., Medinilla, J.: Use educational data mining to predict undergraduate retention. In Advanced Learning Technologies (ICALT), IEEE 16th International Conference on Technologies, vol. 61, pp. 428–430. IEEE (2016) 12. M´ arquez-Vera, C., Cano, A., Romero, C., Noaman, A., Fardoun, H., Ventura, S.: Early dropout prediction using data mining: a case study with high school students. Expert. Syst. 33(1), 107–124 (2016)
Evolution of the Data Mining and Machine Learning Techniques Used in Health Care: A Scoping Review Carmen Cecilia Sanchez Zuleta1(B) , Lillyana María Giraldo Marín2 , Juan Felipe Vélez Gómez1 , David Sanguino Cotte3 , César Augusto Vargas López3 , and Fabián Alberto Jaimes Barragán3 1 Faculty of Basic Sciences, Universidad de Medellin, Medellín, Colombia
{ccsanchez,jfvelez}@udem.edu.co 2 Faculty of Engineering, Universidad de Medellin, Medellín, Colombia
[email protected] 3 Hospital Universitario San Vicente Fundación, Medellín, Colombia
{david.sanguino,cesar.vargas}@sanvicentefundacion.com, [email protected]
Abstract. The purpose of this scoping review was to observe the evolution of using data mining and machine learning techniques in health care based on the MEDLINE database. We used PRISMA-ScR to observe the techniques evolution and its usage according to the number of scientific publications that reference them from 2000 to 2018. On the basis of the results, we established two search strategies when performing a query about the subject. We also found that the three main techniques used in health care are “cluster,” “support vector machine,” and “neural networks.” Keywords: Machine learning · Health care · Database research · Scoping review · MeSH
1 Introduction Different techniques and models have evolved to extract the underlying information to support decision making because of the rising data quantity generated daily in different knowledge fields. Health care has not been alien to this change. For several years, data mining and machine learning techniques have been articulated with statistical techniques to accompany the research processes. Hence, it is of particular interest to perform a chronological review on the evolution of different data mining and machine learning techniques in the scientific literature of health care. Given the disruptive process that data science has become since big data, there has been a generation of technologies and methods to efficiently utilize massive data amounts, seeking to support the decision-making and discovery of knowledge [1]. One of the most rapidly growing areas in computer sciences is machine learning, which is produced when a computer is taught to recognize patterns by providing it with © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 151–160, 2021. https://doi.org/10.1007/978-3-030-72651-5_15
152
C. C. Sanchez Zuleta et al.
data and an algorithm to help them understand [2]. In the last two decades, it has become a standard tool in almost any task that requires extracting information in big data sets [3]. The successful application of data mining in various areas, such as marketing, retail, engineering, or banking, has expanded its horizon to new fields, namely, medicine and public health. Nowadays, an increasing number of data mining applications focus on analyzing information obtained from healthcare centers for better health management, disease outbreaks detection in hospitals, patients’ deaths prevention, and fraudulent insurance claims detection. On the other hand, data mining is an interesting interdisciplinary field that interacts with machine learning, statistics, pattern recognition, databases, information retrieval, and other data. Over time, researchers tried to create software tools for improving the incorporation of data mining models, such as classification trees, regression trees, logistic regression, neural networks, and fuzzy rules [4]. Given the great variety of existing data mining and machine learning techniques and their wide use in health care, it becomes necessary, when developing state of the art about it, to depurate the techniques, the models, and their uses, by starting to identify which ones have been used with higher frequency in the scientific literature in health care (registered in the MEDLINE database). This scoping review aimed to identify the evolution of the incursion of using data mining and machine learning techniques in health care based on the MEDLINE database and recognize which of these techniques have been more utilized throughout the last years. Keeping in mind the year of release and the data mining or machine learning technique used, we established two search strategies to identify the number of scientific articles that have been published in health care and registered in the MEDLINE database. From this perspective, this study will be organized according to the protocol proposed by PRISMA-ScR [5].
2 Method This scoping review aimed to evaluate the data mining and machine learning techniques used in health care. Hence, we conducted a chronological study of the evolution of these techniques’ incursion in the health sciences literature to identify the most commonly used techniques throughout the last years. To guarantee that the observed literature is associated with health care, we used the MEDLINE database (the National Library of Medicine) because it has more than 25 million citations of articles in biological sciences and biomedicine. The following questions were kept in mind to accomplish the objective. 1. What has been the evolution of the incursion of data science techniques (data mining and machine learning techniques) in health care? 2. What are the most used data mining and machine learning techniques over the last five years in health care? 3. How the searches about medical subject headings (MeSH) terms are handled in the MEDLINE database? 4. How do the MeSH terms impact a search’s results when used on it?
Evolution of the Data Mining and Machine Learning Techniques
153
This document’s methodology is framed in the PRISMA-ScR proposal [5], which can be found in the following link: http://www.prisma-statement.org/Protocols/Protoc olGuidance. Other databases are used in health care, such as EMBASE, Cochrane Library, and SCOPUS. However, we considered the MEDLINE database because it is the most used database, and it contains articles related to the implementation of data mining and machine learning techniques in health care. The MEDLINE database also includes the following [6]: • • • •
It contains citations from more than 5,200 magazines globally in 20 languages. It includes literature published since 1966. The citations are added to PubMed every day. More than 904,636 citations were added to the MEDLINE database in 2018.
Initially, we considered 12 data mining and machine learning techniques. Their incursion through the health care’s scientific publications has been studied from 2000 to 2018. We defined the search strategy according to the questions asked for the method. However, on the basis of the results obtained in the MEDLINE database, we restructured it over the research development to search on MeSH terms, as described in the following. Search Strategy 1. Initially, we considered a search about the direct terms of data mining, in which we obtained a high number of results for some techniques whose names coincided with the health care’s technical names, causing noise over the results (Fig. 1).
Fig. 1. Search equation of search strategy 1.
Search Strategy 2. To solve the previous situation, we included the MeSH term “data mining” in the initial search equation, followed by each name of the desired techniques.
154
C. C. Sanchez Zuleta et al.
Hence, we found that from 2000 to 2018, the MeSH term “data mining” was included in the MEDLINE database in 2010. Thus, the articles published before 2010 were not included in the results (Fig. 2). After performing an analysis of the search strategy results, we found that some machine learning techniques were not considered by the MeSH term used in the search strategy. Hence, we generated a new search strategy in which the MeSH term was “machine learning techniques.” With the new search strategy, we obtained a more refined search, in which we found that the MeSH term “machine learning techniques” was included in the MEDLINE database in 2016. Finally, we performed the searching on two fronts, from the initially placed condition and the conjunction of the MeSH terms (Fig. 2).
Fig. 2. Search equation of the search strategy 2
Figures 1 and 2 describe the search equations used to answer the scoping review questions with the aforementioned search strategies and the previous analysis. Figure 3 describes the workflow applied to the search engine to the PubMed website.
Fig. 3. The search algorithm used to the PubMed website
To avoid confusion between medical and informatics terms, we decided the meaning of a term; for example, “neural networks” are understood as a computational technique or part of the nervous system. We reduced the queries by the title and abstract, but we performed these queries only when necessary. Instead, the indexation by MeSH terms was taken advantage. To improve the search strategy results, we used an expert’s judgment, medic, to assign each citation with the thesaurus’s appropriate terms.
Evolution of the Data Mining and Machine Learning Techniques
155
Once the search equations were executed to the search engine to the PubMed website, we exported the results in MEDLINE format. The exported file was processed by a script to compile and group the obtained data. We used Microsoft Excel (Microsoft, Co., Redmond, WA, USA) for presenting the results. After filtering by year, the articles were grouped according to the selected techniques: b-cluster, cluster, decision tree, k-means, KNN, logistic regression, naïve Bayes, neural network, random forest, regression, survival model, and support vector machine (SVM).
3 Results In PubMed, each registered article contains assigned terms (descriptors) defining the subject precisely. Hence, the articles covering the same subject are given the same descriptor [7]. An ambiguous MeSH term could lead to undesirable results, complicating the systematic review process that serves as the basis for a state of art in which you want to work. The results that will now be presented will give the reader an idea of what to find if the MeSH terms used as the tree’s head are not clearly defined. Figures 4 and 5 show the results according to the search strategies (Figs. 1 and 2, respectively). Consequently, we analyzed the findings through the MeSH terms that constitute the tree’s head in the search. Hence, we performed a descriptive analysis of the chronologic evolution of the number of published articles by considering the MeSH terms based on the two search strategies: (1) name of the data mining model of interest and (2) conjunction of the terms between data mining and machine learning techniques. We will discuss in the following the behavior of each case.
Fig. 4. Behavior over time of mining techniques with the name of the technique as MeSH term.
Fig. 5. Evolution of investigations with non-statistical mining techniques as MeSH terms.
Using search strategy 1, in which the MeSH term used is directly related to the data mining or machine learning models used, we found that the techniques do not exclusively belong to data science (data mining or machine learning technique). Moreover, the techniques present more inclusion in the scientific literature, such as the techniques emerging of these disciplines and how it happens with the techniques (regression, cluster, or logistic regression). This is evident in Fig. 4, where we slightly noticed the data science techniques. In the figure, we showed the comparative graphic of all of the techniques used for this study. Three techniques (regression, cluster, and logistic regression) stood out from the 12 techniques searched through the timeframe, with slightly rising tendencies
156
C. C. Sanchez Zuleta et al.
throughout the period and a slight fall during the last year. The curve of the “regression technique” is maintained through the selected timeframe over the remaining curves. This result is coherent with the influence that the statistical regression models have had in investigating health care. In Fig. 5, we can see the chronological behavior of implementing the techniques without including these models, thus diminishing the noise generated by the scientific production of techniques (such as regression, cluster, and logistic regression). By removing the curves mentioned in the previous paragraph, we found that the evolution through time of the new data sciences techniques in the scientific production of health care becomes more visible (Fig. 5). Hence, the stood out techniques are “neural networks” and “decision trees.” At the beginning period, the amount of publications involving these two techniques is the same. As time goes on, the behavior of the inclusion of the “decision trees” technique tends to become relatively stable around the starting value; meanwhile, the “neural network” techniques show an increase during the first four years of the period (until ~ 2004) and present a relatively stable tendency until 2017 and a new spike in 2018. Except for SVM, the techniques show growth over time, with a slight decline in the last year (Fig. 5). On the other hand, the SVM technique, which does not show a lot of influence in the scientific production in the knowledge field of interest until 2010, presents a spike starting from that point in such a way that the amount of scientific publications in this knowledge area that lean on this technique starts growing in an accelerated manner, overcoming the number of scientific publications with decision trees in just two years, and overcoming in 2014 the number of publications that involve the “neural networks” technique. During the first decade, neural networks, decision trees, and k-means are the leading techniques in the number of publications that use them, in which neural networks have the highest frequency in almost all the time threshold. The SVM’s position in the field of knowledge of interest is the third technique used in 2011, second in 2012, and first in 2014. The curve behavior since 2010 shows that “neural networks,” “decision trees,” and “SVM” techniques are the most applied in health care; however, at the end of the time frame, the “random forest” technique surpassed the number of publications using the “decision trees” technique in 2017. At the end of 2018, “neural networks,” “SVM,” and “random forest” are the three main techniques to the number of scientific publications in health care. Even though many data mining and machine learning techniques have existed for a long time, their applications in many fields of knowledge began to be noticed during the 1990s, and they took full strength through the middle of 2000. In health care, according to the searches described in previous paragraphs (Figs. 4 and 5), the publications involving techniques not related to statistics become more evident starting in 2010. This is because the MEDLINE database introduces the term “data mining” as a MeSH term in that same year, and only in 2016, the term “machine learning” is recognized as a MeSH term.
Evolution of the Data Mining and Machine Learning Techniques
157
The discovery of these MeSH terms’ inclusion dates into the MEDLINE database motivated the initiative of determining the incidence of its incursion in the search strategies. Consequently, we changed the search strategy by putting the tree’s head with the new MeSH terms (data mining and machine learning) and leaving in second place the techniques’ names and the concept of interest in this review. Some articles whose previous publication dates are before the new MeSH terms’ incursion have been re-assigned to this branch of the tree. However, for the most part, the articles published before 2010 that involve data mining or machine learning techniques kept the tree’s head that they initially had. For the “regression” technique (a data mining technique), it has zero publications under the new MeSH terms until 2008 (Fig. 6). Among the publications in 2009, eight were pulled up from this branch of the tree. At the beginning of 2010, an ascending behavior begins peaking until 2016, with 229 publications. However, as previously indicated, the technique’s inclusion in the publications about health care during the whole period of study leads the curve of the techniques, reaching a maximum of 25,284 and 23,833 publications in 2013 and 2016, respectively (Fig. 7). This situation shows that the technique is constantly shown as an independent technique for data mining and machine learning in many publications. The difference between the technique’s behaviors in each description is precisely shown in Figs. 6 and 7. In the figures, a behavior similar to the “regression” technique can be observed in the “cluster” and “logistical regression” techniques.
Fig. 6. Technique as MeSH term.
Fig. 7. Technique as a branch of the new MeSH
Fig. 8. Evolution of investigations with mining techniques and MeSH terms “data mining” and “machine learning”
When the technique is the proper data mining or machine learning technique, the curves’ behavior varies slightly between both cases that are used. For example, in Figs. 5
158
C. C. Sanchez Zuleta et al.
and 6, we can see that the curve’s behavior of the publications using SMV with the name of the technique as the initial MeSH term and the case where “machine learning” and “data mining” are the terms that head the search, respectively. Through time, the behavior in both curves is the same. Previous to the date of inclusion of the new MeSH terms, there are only seven publications (one in 2005, two in 2008, and four in 2009), in which the frequencies in both curves are the same, as it happens in the curves of “regression” and “cluster,” indicating that these have been dragged to the new branch of the tree. Generally, the techniques that can be assumed as proper of the newly included MeSH terms present very similar behaviors between before and after the inclusion of such terms. The search for the number of publications in health care using data mining or machine learning techniques as their initial MeSH terms can be found. The techniques present significant changes at the moment of using different MeSH terms. Generally, not all of the publications were migrated to the appropriate branches of the tree.
4 Discussion A scoping review is an exercise carried out preliminarily before carrying out a systematic examination to identify the key characteristics or factors related to a concept (Anderson et al. 2008). Hence, in this scoping review, the importance of knowing the search structure used by the search engine used (PubMed in this case) has been raised, in which the results found are as optimal as possible. We found that depending on the MeSH term used as the head of the search tree, one result or another can be obtained. Each PubMed record was assigned some terms (descriptors) called MeSH terms, precisely defining the search tree for the subject being analyzed [7]. In data science, the terms “data mining” or “machine learning” were included MEDLINE database as MeSH terms as of 2010 the first and 2016 the second. Before these dates, the tree’s head had to be defined by the technique’s name or model of interest. By identifying the dates of entry into the MeSH terms’ database, we can observe the impact of these when searching and expose the needs develop a search strategy that considers both the dates of inclusion of the MeSH terms to the database search target. To subsequently carry out an exhaustive examination of the techniques found, we presented a preliminary exercise in this material seeking to identify the data mining and machine learning techniques that have had the greatest impact in health care research in the first years of this millennium. Two search strategies were executed. First, with the model’s name in the head of the tree as the MeSH term, in this case, since the beginning of the millennium, there is a gradual and growing inclusion of the different data science models in scientific production processes in health care. The second strategy, in which we obtained zero publications in almost all techniques before 2010 (the date of inclusion of the MeSH term “data mining”), the tree’s head was “data mining.” These results present an invitation to researchers to consider in their search strategy the search characteristics of the search engine being used to reach more up-to-date information. The results of this scoping review showed that for the search periods and taking into account the criteria defined in the chosen strategies, the three main techniques used in research health care are “cluster,” “SVM,” and “neural networks.” This implies that these techniques could be the most appropriate to carry out the exhaustive review in
Evolution of the Data Mining and Machine Learning Techniques
159
the state of the art of health care. Thus, there is also a need to study their disconnection from the searches that have the MeSH term in the tree’s head with the word “statistics.”
5 Conclusions The results demonstrated the importance of an adequate choice of the MeSH term when defining the search strategy to allow a more reliable search. It is also recommended to consider the date of inclusion of the said terms in the database because these dates affect the research results. The curves’ behavior in the time threshold has shown that only since 2010 it be clear about the inclusion of data mining and machine learning techniques in scientific publications in health care. This does not imply that they had not been used before this date. It also reflects is that the ignorance of the major branch led to said pro-duction being classified in another branch of the tree. On the basis of the curves’ behavior, it can be established that in addition to the “regression” and “logistic regression” techniques, which present works strongly framed in the area of statistics, the three main data mining techniques used in scientific production in the health care are “cluster,” “SVM,” and “neural networks.” This implies that these techniques could be of interest for a more detailed review of state-of-the-art health care. Overall, none of the search strategies proposed in this review can be said to be better than the other one. Instead, they were found to complement each other. When data mining and machine learning techniques are included, both strategies should be implemented to provide further coverage to the health care scientific publications dealing with these techniques.
References 1. Anderson, S., Allen, P., Peckham, S., Goodwin, N.: Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Res Policy Syst. 6(7) 1–12 (2008) 2. Storey, V., Song, I.-Y.: Big data technologies and management: what conceptual modeling can do. In: Data & Knowledge Engineering. Science Direct, pp. 50–67 (2017). https://www.sci encedirect.com/science/article/abs/pii/S0169023X17300277?via%3Dihub. Accessed 06 May 2020 3. McCrae, I., Hempstalk, K.: Introduction to machine learning in healthcare. In: ORION HEALTH. 2015. https://orionhealth.com/uk/knowledge-hub/reports/machine-learning-in-hea lthcare/. Accessed 06 May 2020 4. Shalev-Shwartz, S., Ben-David, S.: Understanding machine learning: from theory to algorithms. In: Cambridge University Press. 2014. https://www.cs.huji.ac.il/~shais/Understandin gMachineLearning/understanding-machine-learning-theory-algorithms.pdf. Accessed 06 May 2020 ˘ I., IONIT¸ A, ˘ L.: Applying data mining techniques in healthcare. Stud. Inform. Con5. IONIT¸ A*, trol. 25(3), 385–394. 2016. https://sic.ici.ro/wp-content/uploads/2016/09/SIC-3-2016-Art12. pdf. Accessed 06 May 2020 6. Tricco, A.C., Lillie, E., Zarin, W., O’Brien, K.K., Colquhoun, H., Levac, D., Weeks, L.: PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann. Intern. Med. 169(7), 467–473 (2018)
160
C. C. Sanchez Zuleta et al.
7. National Library of the Medicine. MEDLINE®: Description of the Database. In: National Library of the Medicine. 2019. https://www.nlm.nih.gov/bsd/medline.html. Accessed 8. Sevilla, B.U.: MeSH Database. In: Biblioteca Universidad de Sevilla. 2019. http://fama2.us. es/bgu/ad/tfg/pubmed/PubMed_08.htm. Accessed 06 May 2020
Learning Analytics Metrics into Online Course’s Critical Success Factors Yosra Mourali1,2,3(B) , Maroi Agrebi2 , Ramzi Farhat3 , Houcine Ezzedine2 , and Mohamed Jemni3 1 Faculty of Economics and Management, University of Sfax, Sfax, Tunisia
[email protected] 2 Univ. Polytechnique Hauts-de-France, CNRS, UMR 8201 – LAMIH, Valenciennes, France
{Maroi.Agrebi,Houcine.Ezzedine}@uphf.fr 3 Research Lab. LaTICE, University of Tunis, Tunis, Tunisia
[email protected], [email protected]
Abstract. E-learning has gained tremendous popularity. In recent decades, major universities all over the world offer online courses aiming to support student learning performance, yet often exhibit low completion rates. Faced with the challenge of decreasing learners’ dropout rates, e-learning communities are increasingly in need to improve trainings. Machine learning and data analysis have emerged as powerful methods to analyze educational data. They can therefore empower the Technology Enhanced Learning Environments (TELEs). Many research projects have interest in understanding dropout factors related to learners, but few works are committed to refine pedagogical content quality. To address this problem, this paper proposes descriptive statistics analysis to evaluate e-course factors contributing to its success. We take up, at first, the task of analyzing a sample of successful online courses. Then, we manage to identify features that are able to attract a large number of learners, meet their needs and improve their satisfaction. We report findings of an exploratory study that investigates the relationship between course success and the strategy of pedagogical content creation. Keywords: E-Learning · Pedagogical content · Course content quality · Successful E-Course · Descriptive statistics
1 Introduction Information and Communication Technologies (ICT) are evolving every day. Their application in education could have enormous possibilities for simulation, presentation and visualization of learning materials [1]. Due to the rapid development of ICT and internet technologies, the concept of e-learning is becoming more and more important. It is emerging as a modern technology that may have some advantages over traditional teaching methods including cost-effectiveness, regular updates, flexibility to time and place and accessibility to instructional information [2]. Hence, many educational organizations offer online courses to promise the democratization of knowledge and lifelong learning. Nowadays, e-learning is gaining popularity worldwide [3]. The number of © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 161–170, 2021. https://doi.org/10.1007/978-3-030-72651-5_16
162
Y. Mourali et al.
learners enrolled in online courses is growing; anyone can access to online education anytime and anywhere to develop his skills in any domain. TELEs target a wide variety of learners, provide a multitude of audiovisual materials and services, and establish networks among participants. Thanks to data storage capacity and development of computing power, it is possible to generate from TELEs and through different devices and sensors employed around the world, a great volume of data related to courses, students and their interactions [4]. As the captured data are important sources of information, it is crucial to benefit and find meaning to all this data that exponentially multiply day by day [5]. Data science has existed for decades and has recently garnered significant attention. It allows for the combination of various approaches including techniques connected to data analysis in the field of statistics, machine learning, artificial intelligence, etc. [6]. E-commerce, e-education, and e-services have been identified as important domains for analytics techniques [7]. The science of data includes the processes of data clearing and integration, selection and transformation of data, knowledge extraction, and data analysis, evaluation and representation [6]. Machine learning and data analysis are proposed as possible solutions to explore the increased amounts of collected data generated in e-learning field with the goal of extracting knowledge and useful information, making the learning experience more efficient [8, 9]. This study aims to evaluate course related factors contributing to the success of online courses in attempt to help authors to implement high quality pedagogical content. In other words, the subject of our research is to identify e-course related elements impacting e-learning content quality. This paper is organized as follows. In Sect. 1, we introduce the general context of the paper and describe the problem. In Sect. 2 we highlight e-learning challenges, present a range of interesting contributions using data analysis methods to improve TELEs and trace the factors impacting the quality of online course content. In Sect. 3, we highlight the methodology followed to conduct this study and the different findings. In Sect. 4, we finish by a conclusion, in which we give recommendations for pedagogical content designer.
2 Background and Related Works New trends of researches in education consider machine learning and data analysis as powerful technologies to enhance e-learning. As educational data are important sources of information for decision support and improvement of TELE, the needs of exploring them are currently underrepresented in the literature. In this section we will present a range of research using data analysis methods to support e-learning. Next, we will examine elements impacting the quality of pedagogical content. 2.1 Improving Online Courses in Literature In the literature, there have been a lot of methods proposed to improve online courses. In this section we present a range of recent research works related to this topic. Farhat et al. [10] have proposed an approach to evaluate e-courses which are built by reuse of existing learning objects. They have identified and defined 18 metrics that
Learning Analytics Metrics into Online Course’s
163
can be extracted from learning objects’ metadata. Those metrics are used therefore to calculate 7 content quality indicators. Those indicators can be then visualized by the authors and can be compared to some successful courses indicators to have feedback about possible improvements. According to Suciu et al. [11], communication, contributes to foster the peer-to-peer knowledge sharing, and as a consequence, increases learners’ skills [12]. Mahadevan et al. [13] propose a framework to enhance and facilitate collaboration and cooperation between students by assigning them to groups based on their learning style. This framework can be integrated with MOOCs and complete the existing methods such as forums and meetups. Sekeroglu et al. [14] use four different machine learning algorithms Backpropagation (BP), Support Vector Regression (SVR), Long-Short Term Memory (LSTM) and Gradient Boosting Classifier (GBC) in order to predict and classify behaviour of students in two courses and provide early intervention to increase the performances. Pawade et al. [15] have developed a system which uses both Word Mover’s Distance (WMD) similarity measure and machine learning approach to automatically generate the scores of the answers and the comments posted on online discussion forums. This can be very helpful in assessing the participation of students in the discussion forums. It is well known that different people learn in different ways. Therefore, it becomes imperative to personalize the content, in such a manner that caters to the different learners’ styles, preferences and needs [16]. Hugo [17] introduces a hybridization approach of AI techniques and statistical tools to evaluate and adapt the e-learning systems. Based on different fuzzy clustering techniques such as FCM and KFCM, the researcher classifies the learner’s profiles into specific categories; the learners’ classes named as regular, workers, casual, bad, and absent. Results proved that the KFCM is much better than FCM. In the same sphere, Christos et al. [18] implement a system using a hybrid model for misconception detection and identification (MDI) and an inference system for the dynamic delivery of the learning objects tailored to learners’ needs. The MDI mechanism incorporates the Fuzzy String Searching (FSS) and The String Interpreting Resemblance (SIR) algorithms in order to reason between possible learners’ misconceptions. The inference system utilizes the knowledge inference relationship between the learning objects and creates a personalized learning environment for each student. Mejia [19] mixed exploratory methods to determine if the use of audio or video recorded online discussions in VoiceThread enhanced student engagement with their peers in an undergraduate hospitality course. Quantitative approach to attain means, frequencies, and regression was supported by two open-ended questions subject to qualitative analysis. The findings revealed that students’ use of the audio function, both for posting their own responses and for listening to others, was a statistically significant predictor of student engagement with their classmates. According to Ramlatchan and Watson [20], the design of multimedia elements used in video for online courses can increase student perceptions of their instructor’s credibility and immediacy. Ramlatchan and Watson provided an experimental research study comparing five treatments groups; instructor-only, slides-only, video-switching, dualwindows, and layered-video multimedia presentation designs. Variances and Tukey post hoc calculations were conducted to test for statistically significant differences between
164
Y. Mourali et al.
groups in relation to perceived instructor credibility, nonverbal immediacy, and cognitive load. Previous researches have certainly contributed to the improvement of e-learning. Nevertheless, they still present several limitations. At first, they have ignored the quality aspect of pedagogical content. They focused on adapting online courses to leaners’ preferences rather than improving the quality of the pedagogical content itself. Secondly, they did not give importance to success factors of online courses. In addition, they did not consider metrics to evaluation pedagogical content quality. For these reasons, we manage to propose metrics into online course’s critical success factors. 2.2 Factors Impacting Quality of Online Course Content Course content or pedagogical content is considered as one of the indicators to evaluate the performance of a TELE and learning quality. Therefore, it seems to be important to observe and examine successful online courses to detect strengths aspects and good practices related to the pedagogical content. It is crucial for educators, designers and instructors to improve learning performance. The improvements can be reached from the analysis of popular courses through data related to their design aspect, content support aspect and assessment aspect. Almaiah and Almulhem [21] said that well designed courses improve quality of e-learning system and thus; increase students’ acceptance and maximize e-learning system use. However, poorly designed courses have a tendency to be responsible for low usage of e-learning system. In addition, Almaiah and Alyoussef [22] confirm that use of multimedia features with solid content and appropriate instructional Table 1. Features and proposed metrics for online course evaluation. Features
Metrics
Course design
- Description - Trailer - Goals - Syllabus - Level - For who - Duration - Effort
Course content support - Learning objects (LO) types - The number of LO - The number of videos - Duration of the video - The number of articles - The number of photos Course assessment
- The number of quizzes - The number of quiz questions - The number of test steps - Final exam
Learning Analytics Metrics into Online Course’s
165
methods can significantly influence the learning process by keeping the students engaged and motivated to learn. Table 1 summarizes proposed features and metrics, that can conduct to study quality of online course content. Based on descriptive statistics analysis methods, we aim to evaluate factors impacting the quality of online course content in order to recommend strategies guiding instructors to plan, design and implement e-learning modules. As a first task, we have to investigate online courses and select those with significant number of enrolled learners and positive learning experience perceived. Once we have a sample of successful courses, we can move to the second task. At this stage, the main challenge will be to find qualitative and quantitative criteria characterizing those courses. By referring to those criteria it would be possible not only to estimate the success potential of an online course but also to assist and guide course designers in creating and improving their educational content. Therefore, to achieve our goal we must find answers to this following research question: What are the characteristics which can indicate the success potential of an online course?
3 Data Analysis and Interpretation To condact our research, four steps were required: (1) collection of successful online courses, (2) data preparation, (3) descriptive statistics analysis using both mesures of center and spread and (4) recommandation of best guidelines to course designers. 3.1 Data Collection This article utilizes Class Central as the main instrument for data collection. Class Central is a listing of online courses. It aggregates 15000 courses from about 1000 top universities such as Harvard University and MIT, more than 40 providers like edX, Coursera and France Université Numerique Courses, and nearly 400 institutions such as Google, IBM and United Nations. Class Central focus, primarily, on free courses offered through massive open online course (MOOC) platforms. Student satisfaction is an essential indicator of students’ overall academic experiences and achievement. That is why, it is considered as an essential element to measure the quality of online courses [23]. Each year, Class Central publishes a ranking of “The Best Online Courses of All Time”. This list includes 200 courses arranged in descending order, representing the highest rated online courses and MOOCs of all-time from top universities around the world, based on thousands of reviews written by Class Central users [24]. In this work, we have referred to the 2020 report to select a sample of the best 50 courses. Hence, a dataset, of 50 observations, 24 attributes describing courses design, content support, and assessment, was developed. 3.2 Data Preparation After collecting data, to check for any missing data, outliers, and strange values, data analysis was performed. Each time we have to eliminate a course, it was replaced by another one from the initial list of “The Best Online Courses of All Time” taking into consideration the ranking. As a result, 50 courses were considered valid for further analysis. Features selected are detailed in Table 1.
166
Y. Mourali et al.
3.3 Data Exploration As said previously, course design is one of the key factors that impact e-learning. By examining the 50 successful courses we were able to highlight similarities in the course structure. At first, we have noticed that it is unavoidable to start by a training opening in which learners can find clear answers about his learning. The description of the course, with or without a trailer, is very important. The definition of objectives, syllabus, duration and effort are also very important. The staple ingredients of descriptive statistics are the measures of center. They include the mean, median and mode and serve to give an idea of the typical value of a variable [25]. They were approached with Python to investigate course size and the weekly effort. From the histogram illustrated in Fig. 1, we can see that the mode is equal to 6, which means that most courses are spread out over 6 weeks.
Fig. 1. Duration of successful online courses.
Referring to Fig. 2, the histogram shows that 3 h’ effort per a week is the most common value.
Fig. 2. Effort distribution of successful online courses.
The integration of multimedia, which enhances content visualization and user interaction such as, static graphics, animations, video, digital games, interactive simulations, quizzes, or other widget e-learning activities, contributes to learners’ cognitive and affective development [26]. Figure 3 shows that 30 courses out of 50 best courses have incorporated 6 LO types. A number of statistics regarding the use of LO in successful online courses is given in Fig. 4. The pie chart of LO type distributions sheds some light on how to exploit
Learning Analytics Metrics into Online Course’s
167
Fig. 3. LO types used in successful online courses.
multimedia to create a pedagogical content that attract learners and keep them motivated and engaged during the training. Hens, about 31% of the course content takes the form of video. Text and photo occupy the second place with a percentage of 18%. In the third place, quiz represents 9% of the course. However, near 23% of the rest of the course content takes at least 2 other forms like audio, link, resource, simulation, etc.
Fig. 4. The pie chart of LO type distributions in successful online courses.
As measures of spread give sense of how much the data tends to diverge from the typical value, they were approached to describe the video duration’s dispersion. As shown in Fig. 5, the boxplots are visual representations of the five number summary. Since the video duration of the best 50 courses of Class Central varies between 2 min and 15 min, most videos are between 5 and 9 min long as shown in Fig. 5. In addition, we found that the median and the mean are the same and equal to 7 min. So, it seems that the optimal duration is close to 7 min. The best 50 courses we have explored are free courses without certificate. Therefore, we didn’t find final exams. According to [27], exams increase learner anxiety. In our case exams were replaced by auto evaluation throw quizzes, scheduling collaborative activities and debates to encourage discussion and sharing of ideas and experiences. 3.4 Recommendations for Successful Online Course The field of data science has been evolving at a dramatic pace and has recently been propelled to the forefront of education research. One of methods, which are holding an
168
Y. Mourali et al.
Fig. 5. Video duration.
important place, is descriptive statistical analysis. The major motivation of this research is to provide course designers with specific requirements in the form of suggestions to improve their courses content which allow, as result, to enhance learning and support TELEs. Based on literature review, elements related to course design, content support and assessment were distinguished as factors impacting the course effectiveness. This led us to propose metrics to evaluate the quality of pedagogical content. We cite as example the course duration, the effort that a learner has to put into learning activities during a week, the diversity of the course content measured by the number of activity types, and the video duration. Thanks to descriptive statistical analysis we have done with Python tools, we could identify features of a successful course. According to our exploratory study, a successful e-course must firstly be well-structured. It is crucial to begin by a description of the whole content, to define objectives, to present the syllabus and to mention information about duration and effort. A successful e-course must, secondly, be very diversified. It is so important to include rich and interactive contents. Use of 6 different media types of LO is really desired. In essence, it is preferable that video occupy 30% of the course, text and photo may constitute each one about 20%. To avoid the risk of test anxiety, quiz is the best way to assess learners instead of exams and can represent 10% of the content. The rest of the course which is near 20% may take several other forms such as link, resource, audio or animation. Finally, a successful course must be practical and always presented in a very pedagogical way. Here we can give a wink to draw attention to the video size, duration and effort.
4 Conclusion While most previous research works have neglected the quality aspect of pedagogical content, this study contributes to advancing knowledge on how to better develop online courses. We have outlined the criteria of success related to pedagogical content and suggested some recommendations to help instructors implement high quality of learning modules. To attempt this object, we have study elements impacting course content. Then, we have proposed metrics to evaluate them. A dataset of 50 best online courses from Class Central platform has been exploited for the descriptive statistics analysis.
Learning Analytics Metrics into Online Course’s
169
Future research may focus, in a first step, on the exploitation of a largest dataset from different providers in order to have diversified successful online courses. We can, therefore, refine our results. Successful courses sometimes may not include the effectiveness of how students learn and how well the learned materials can be used in practice. That is why, in a second step, we may benefit, from machine learning approaches to enable automatic evaluation of the online courses based on learners’ traces.
References 1. Nikoli´c, V., Petkovi´c, D., Deni´c, N., Milovanˇcevi´c, M., Gavrilovi´c, S.: Appraisal and review of e-learning and ICT systems in teaching process. Phys. Stat. Mech. Appl. 513, 456–464 (2019) 2. Alqudah, N.M., Jammal, H.M., Saleh, O., Khader, Y., Obeidat, N., Alqudah, J.: Perception and experience of academic Jordanian ophthalmologists with E-Learning for undergraduate course during the COVID-19 pandemic. Ann. Med. Surg. 59, 44–47 (2020) 3. Mourali, Y., Agrebi, M., Ezzedine, H., Farhat, R., Jemni, M., Abed, M.: A review on elearning: perspectives and challenges. In: ICIW 2020, The Fifteenth International Conference on Internet and Web Applications and Services, Lisbon, Portugal, September 2020 4. Rodrigues, M.W., Isotani, S., Zárate, L.E.: Educational data mining: a review of evaluation process in the e-learning. Telematics Inform. 35(6), 1701–1717 (2018) 5. Udupi, P.K., Sharma, N., Jha, S.K.: Educational data mining and big data framework for e-learning environment. In: 2016 5th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), pp. 258–261. IEEE, September 2016 6. Popchev, I.P., Orozova, D.A.: Towards big data analytics in the e-learning space. Cybern. Inf. Technol. 19(3), 16–24 (2019) 7. Sathiyamoorthi, V.: An intelligent system for predicting a user access to a web based elearning system using web mining. Int. J. Inf. Technol. Web Eng. (IJITWE) 15(1), 75–94 (2020) 8. Moubayed, A., Injadat, M., Nassif, A.B., Lutfiyya, H., Shami, A.: E-learning: challenges and research opportunities using machine learning & data analytics. IEEE Access 6, 39117–39138 (2018) 9. Farhat, R., Mourali, Y., Jemni, M., Ezzedine, H.: An overview of machine learning technologies and their use in e-learning. In: 2020 International Multi-Conference on: Organization of Knowledge and Advanced Technologies (OCTA), pp. 1–4. IEEE, February 2020 10. Farhat, R., Defude, B., Jemni, M.: Towards a better understanding of learning objects content. In: 2011 IEEE 11th International Conference on Advanced Learning Technologies, pp. 536– 540. IEEE, July 2011 11. Suciu, G., Militaru, T.L., Cernat, C.G., Todoran, G., Poenaru, V.A.: Platform for online collaboration and e-learning in open source distributed cloud systems. In: Proceedings ELMAR-2012, pp. 337–340. IEEE, September 2012 12. Rajam, S., Cortez, R., Vazhenin, A., Bhalla, S.: E-learning computational cloud (elc2): web services platform to enhance task collaboration. In: 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, vol. 3, pp. 350–355. IEEE, August 2010 13. Mahadevan, D., Kumar, A., Bijlani, K.: Virtual group study rooms for MOOCS. In: 2015 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1711–1714. IEEE, August 2015
170
Y. Mourali et al.
14. Sekeroglu, B., Dimililer, K., Tuncal, K.: Student performance prediction and classification using machine learning algorithms. In: Proceedings of the 2019 8th International Conference on Educational and Information Technology, pp. 7–11, March 2019 15. Pawade, D., Sakhapara, A., Ghai, R., Sujith, S., Dama, S.: Automated scoring system for online discussion forum using machine learning and similarity measure. In: Advanced Computing Technologies and Applications, Singapore, pp. 543–553. Springer (2020) 16. Jebali, B., Farhat, R.: Toward personalization in MOOCs. In: 2017 6th International Conference on Information and Communication Technology and Accessibility (ICTA), pp. 1–6. IEEE, December 2017 17. Hogo, M.A.: Evaluation of e-learning systems based on fuzzy clustering models and statistical tools. Expert Syst. Appl. 37(10), 6891–6903 (2010) 18. Troussas, C., Chrysafiadi, K., Virvou, M.: An intelligent adaptive fuzzy-based inference system for computer-assisted language learning. Expert Syst. Appl. 127, 85–96 (2019) 19. Mejia, C.: Using voice thread as a discussion platform to enhance student engagement in a hospitality management online course. J. Hosp. Leisure, Sport Tourism Educ. 26, (2020) 20. Ramlatchan, M., Watson, G.S.: Enhancing instructor credibility and immediacy in online multimedia designs. Educ. Technol. Res. Dev. 68(1), 511–528 (2020) 21. Almaiah, M.A., Almulhem, A.: A conceptual framework for determining the success factors of e-learning system implementation using Delphi technique. J. Theor. Appl. Inf. Technol. 96(17) (2018) 22. Almaiah, M.A., Alyoussef, I.Y.: Analysis of the effect of course design, course content support, course assessment and instructor characteristics on the actual use of E-learning system. IEEE Access 7, 171907–171922 (2019) 23. Rajabalee, Y.B., Santally, M.I.: Learner satisfaction, engagement and performances in an online module: implications for institutional e-learning policy. Educ. Inf. Technol. 1–34 (2020) 24. CLASS CENTRAL Homepage. https://www.classcentral.com/. Accessed 18 Nov 2020 25. Akram, A., Fu, C., Tang, Y., Jiang, Y., Lin, X.: Exposing the hidden to the eyes: analysis of SCHOLAT E-Learning data. In: 2016 IEEE 20th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 693–698. IEEE, May 2016 26. So, W.W.M., Chen, Y., Wan, Z.H.: Multimedia e-learning and self-regulated science learning: a study of primary school learners’ experiences and perceptions. J. Sci. Educ. Technol. 28(5), 508–522 (2019) 27. Rivers, M.L.: Metacognition about practice testing: a review of learners’ beliefs, monitoring, and control of test-enhanced learning. Educ. Psychol. Rev. 1–40 (2020)
The Influence of COVID-19 in Retail: A Systematic Literature Review Marisa Lopes1(B) and João Reis1,2 1 Higher Institute of Management and Administration of Santarém,
2000-241 ISLA-Santarém, Portugal [email protected] 2 Research Unit on Governance, Competitiveness and Public Policies (GOVCOPP), Aveiro University, Campus Universitário de Santiago, 3810-193 Aveiro, Portugal
Abstract. The objective of this research is to discuss the state-of-the-art with regard to the influence of COVID-19 in retail, especially in the light of the economic, social and health perspectives. To do so, we carried out a systematic literature review, a technique that allows an efficient description of the literature and that made it possible to provide a holistic perspective of the phenomenon trough the interpretation of relevant published articles. The results showed the sector had to find new measures to mitigate the effects of corona virus, ranging from social and physical distance, protection and hygiene measures, adjustments in distribution and communication channels, to the transmission of a clearer messages by retailers. In addition, the digital transformation phenomenon has also been enhanced by this crisis, by using measures aimed at reducing the spread of the corona virus, such as the use of several digital payment systems and new distribution methods to respond to the new sales and delivery rules. This article reinforces the importance of these measures for a new model of action in the retail sector, in order to respond to the market needs. Keywords: COVID-19 · Crisis · Digital transformation · Retail · Society · Systematic literature review
1 Introduction In the 21st century, and despite the advancement of technology in the current digital age, we believe the world is not really prepared to face disasters, viruses or crisis situations with global effects [1]. The COVID-19 virus emerges as a warning to societies and economies, as the latter have been affected globally and on a large scale due to their unpreparedness. The current outbreak has spread rapidly across the world and has had negative health, social, political and economic impacts. Countries will have to find new measures, practices and behaviors in order to survive in this new reality [1, 2]. Retail was one of the most affected sectors, since, in the short term, was forced to: be more concerned with the safety and health of its employees, support the increased demand for food and consequent pressure on the supply chain, improve hygiene of their physical © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 171–181, 2021. https://doi.org/10.1007/978-3-030-72651-5_17
172
M. Lopes and J. Reis
spaces, better management their workforce, overburdening their employees to ensure the livelihood of the civilian population, among others issues [2, 3]. Studies indicate that pandemics will continue to exist and, therefore, this article aims to understand the consequences that a pandemic outbreak, in this case of COVID-19, had on the retail, with a harmful or beneficial effect. It also intends to serve as a preparation guide to allow a permanent reaction from the sector in these contexts, thus avoiding situations of service closure. As the pandemic influenced the retail sector, it has become an opportunity to investigate this real-life phenomenon. Notable studies, such as the article by Pantano et al. [4] has already focused on the consumer´s perspective, but has put aside measures taken internally by retailers. That is, academics have investigated the perspective of the external environment, such as the change in consumer and management behavior in the retail sector through the context of the pandemic [4], but have not yet dedicated their research effort to the internal environment, which we identified as a research gap. To fill the identified gap in the literature, we suggest the following research questions (RQ): RQ1: How is the epidemiological outbreak influencing the retail companies? RQ2: What measures are being taken by retailers to prevent the spread of the corona virus? Based on a preliminary literature review, we present some emerging concepts. In the following section, we present the methodology used in this article and, finally, the results and conclusions based on the systematic review.
2 Emerging Concepts In this section we present an epidemiological description and some of the emerging concepts of COVID-19 and its relation to the retail sector. 2.1 Epidemiological Summary (COVID-19) In December 2019, 27 cases were identified in the city of Wuhan (People´s Republic of China), which underwent through hospitalization with symptoms of pneumonia of unknown origin. The patients presented symptoms such as cough, fatigue and fever, but others were already in a critical state with severe infections in the lungs. All these patients had in common to have been in a Wuhan market that is known for selling seafood and live animals [5]. On January 31, the World Health Organization (WHO) reported 9,826 confirmed deaths worldwide, with 9,720 in China and 106 deaths spread over 19 countries, such as Canada, United States, Japan, France, Italy and Germany [6]. It was then that the WHO, in mid-March, classified coronavirus disease (COVID19) as a pandemic, a severe acute respiratory syndrome-related coronavirus 2 (SARSCoV-2), which can cause severe pneumonia and, consequently, death [3, 7, 8]. The coronaviruses (CoV) are a family of viruses that infect animals and also humans [9]. Currently, seven of these viruses are already known and have as effect infections of the respiratory tract [9]. Three coronaviruses are known to be transmitted to man through an animal or a host: SARS-CoV (2003), MERS-CoV (2012) and SARS-CoV-2 (2019)
The Influence of COVID-19 in Retail
173
[9]. Although this new coronavirus, Severe Acute Respiratory Syndrome (SARS-CoV2), was first identified in December 2019, in the People´s Republic of China (SNS, 2020), its origin has not yet been fully confirmed; however, scientists have already concluded from laboratory experiments that this virus could not have been genetically manipulated, which leaves aside some insinuations made by the United States of America over the People´s Republic of China [10]. Mainly, because studies reveal that five of the six residues compared between SARS-COV and SARS-COV2 are different, the latter demonstrating a greater affinity with humans, proving to be the result of a natural selection of a human ACE2 [10]. The enzyme ACE2 is a protein that allows the virus to infect the human organism, since it is directly linked to the cardiovascular system. Therefore, the transmission of SARS-CoV-2 occurs when this protein (receptor) binds to the virus protein and spreads in the human body, affecting the respiratory system [11]. Some scientific studies [10, 12] point out two possible origins of the virus: (1) natural selection in an animal host before zoonotic transfer; (2) natural selection in humans after zoonotic transfer. Research results have already been published and reveal that SARS-CoV-2 presents 96% similarity to a bat coronavirus, which could mean the origin of this animal as host [10]. However, Malaysian pangolin is also indicated as a possible host, since they are potential transmitters of this type of virus [5, 9]. As for the rate of transmission, recent studies show that people aged 60 and over have a higher risk of acquiring the virus (especially those with hypertension, coronary heart disease or diabetes) and children at lower risk, and even infected children have lighter symptoms [3, 5]. It is fundamental to be clairvoyant the origin of the virus in order to control its spread and future outbreaks. SARS-CoV-2 is mainly transmitted through respiratory droplets and contact, while individual protection measures are currently crucial and essential to mitigate the spread of this new coronavirus. In addition, the vaccine and group immunity is still a distant reality, because, if we know that a significant number of laboratories are trying to develop a potential vaccine so far, on the other hand it is still in testing phase and its effectiveness has not yet been fully proven [13]. 2.2 The Supply Chain Management and Its Relation to COVID-19 This outbreak had an impact on societies, as it has forced many sovereign states to declare the “state of emergency” that restricted freedom of movement to mitigate COVID-19 and relieve health systems. Social distancing has emerged as the new social paradigm and is introduced as the new term in the daily equation. Through an epidemiological context of this scale, the strategy of several countries has been to confine their citizens to minimize social contact, since, allegedly, the greatest danger is still among asymptomatic people, who can transmit the virus without them knowing [8]. The ubiquity of this outbreak has installed an environment of uncertainty and fear in the minds of consumers, which has led to a drastic change in consumption patterns, especially for non-perishable goods [4]. The increase in demand in a hoarding way was prompted by fear of the effect of the scarcity of some goods and, therefore, most retailers had to rethink their supply chains to avoid the lack of products on the market [4]. The supply chain networks suffered a negative impact also due to another challenge imposed by COVID-19 – border restrictions and closure, which affected the global economy. This has limited the compliance with deadlines and storage of some products and reduced economic profitability for some companies in the
174
M. Lopes and J. Reis
sector [14]. In turn, online purchasing options increased which, in a way, also stifled the supply chain that was not prepared for this increase, providing a transformation for both the manager and the consumer [15]. The COVID-19 revealed that the concern in retail became first and foremost the safety and health of its employees and customers and, thus, it was necessary to combine welfare and protection tools/strategies/practices to ensure the necessary safety distances for virus control and to avoid new contamination chains [15]. The focus of retailers had to be mostly limited to risk minimization through measures such as: management of physical versus online stock flow, hygiene and sanitation throughout the store, control and accounting of customers by physical space, signage of distance brands in stores, installation of barriers between employees, new payment options to avoid contact between customers and employees [16]. The hygiene and safety of commercial spaces were also an investment imposed and the challenge of retailers was to spread, through their communication channels, a reassuring message that there was stock of the product in the physical and online stores. This strategy made it possible to control consumer panic and, at the same time, guarantee some stability in the operations of these companies [14].
3 Methodology This article follows a systematic literature review, since it gathers relevant studies from the literature through its identification, selection and analysis, in a systematic way [17]. This method is very useful for objectifying answers the research questions, since it gathers several studies in the existing literature and, after the delimitation resulting from the critical analysis of these studies, it allows to arrive at evidence that can be reproductive and globally estimated [17, 18]. In order to reach a greater consensus on the subject, we combined two types of analysis: a quantitative and a qualitative one. The quantitative approach is based on a bibliometric analysis, since we explore the data from the selected literature on the subject; while, the qualitative approach, is based on a content analysis in a systematic way that facilitates the mapping of the main concepts and their reproducibility [17, 19]. Table 1 summarizes the research methodology. Table 1. Research methodology Approach
Description
Content
Quantitative approach
Bibliometric analysis
Distribution per journal, per country, and per research area
Qualitative approach
Content analysis of the selected articles
Discuss the state-of-the-art and to find the measures taken by retails to deal with Covid-19
The data search was conducted on October 7th, 2020, and the selected peer-reviewed database was Scopus, Elsevier. We started with the inclusion criteria by using “Retail”
The Influence of COVID-19 in Retail
175
and “COVID-19” terms in the topic (title, abstract and keywords). The type of documents selected apply exclusively to journal articles, because they have greater scientific credibility when compared with other types of documents, but also because this source is commonly used in systematic literature reviews (Table 2). Table 2. Systematic literature review process Institute for scientific information – web of science Criteria
Filters
Documents
Selected keyword
“Retail” AND “COVID-19”
83
Restriction
Topic (Title, Abstract, Author Keywords)
Document type
Journal articles
69
Language
English
67
The journal articles are entirely dated to 2020, since the origin of the pandemic occurred only this year. To avoid wrong interpretations, the selected documents had to be written in English. The systematic review literature may be limited by the fact that the studies are very recent, which may result from some lack of theoretical robustness. The study also presents a snapshot over this period of time, and it is likely that new evidence will strengthen the results of this article or stimulate discussion by presenting data that will prove the contrary. Nevertheless, the relevance of the study seems to be evident as a means to understand a current phenomenon and for which little is known. From a total of 83 manuscripts, we ended up with 67 articles, which we analyzed and presented the results in the next section.
4 Findings 4.1 Quantitative Analysis As we mentioned before, COVID-19 is a theme that has been recently discussed in the academic community, thus, we have made an analysis of the selected articles according their distribution: by journal, its quality (quartile) and the citations reported in Scopus’ database. As it is shown in Table 3, the Canadian Journal of Agricultural Economics (CJAE) is the most frequently cited journal to date (55%). In that regard, the CJAE represents more than half of the articles, since it directly deals with supplies/retail, namely, the food sector. Overall, most of the publications are from first-rate journals due to the interest in meeting economic and social needs and which, in addition to contributing to the theory, also affects the way professionals manage their businesses and which will end up affecting people’s lives. Most specifically, as the articles are located in quartiles Q1 and Q2, it indicates that they are a reliable source of evidence. We have also explored the distribution of documents per country, as we can see below in Fig. 1:
176
M. Lopes and J. Reis Table 3. Distribution per journal
Top 5 journals
Citations Quartile % of 67
Canadian journal of agricultural economics
37
Research in social and administrative pharmacy
Q2
55%
11
Q1
16%
International journal of environmental research and public health
7
Q2
10%
Journal of business research
6
Q1
9%
Continuum (Minneapolis, Minn.)
6
–
9%
Fig. 1. Distribution per countries (Top 10)
The identified articles mostly evoke studies on the United States of America, as it is somewhat evident that the largest economy in the world has a strong research relationship on COVID-19, enhanced mainly by its negative effects on GDP and also on the workforce that was forced to suspend some of its activities due to the effects of the quarantine [14]. Canada ranks second place and most of its studies are on the impact of the virus on one of its main economic activities - the primary agricultural sector (mostly grains and oilseeds) – which, in turn, has the United States of America as the largest exporter of these products and, therefore, both share this ranking for their trade and economic relations. Some research also shows that blocking borders and restricted goods transport between countries have resulted in a global concern: food shortages and price inflation of some food products in those countries [15]. We also did an analysis regarding the distribution of documents by research area, as illustrated in Fig. 2: On a global scale, the effect of COVID-19 was twofold, as it slowed the growth of developed economies and worsened the performance of emerging. Consequently, the economic effect has been cross-border to the world and thus it is no wonder that the vast
The Influence of COVID-19 in Retail
177
Fig. 2. Distribution per research area (subject research)
majority of studies focus on surveying the economic impacts on countries; it is followed by the social sciences studies, which mainly focus on consumer behavior and social distancing; finally, medicine also occupies a relevant percentage due to the genesis and development of the virus. 4.2 Qualitative Analysis In order to answer the research questions, we identified some categories that encompass the main challenges that the retail sector had to overcome, in order to mitigate the virus effects, both in the present and in the future. Due to the limited number of pages available for publication, it was not possible to list every author´s names for each identified measure. To mitigate that limitation, we decided to use percentages, which represents the number of articles that refer to a certain measure and the percentage of that measure over the total number of published articles, as shown in Table 4. The measures identified are common to all identified countries. Table 4. Measures taken by retails Themes
Articles
% of 67
Social and physical distancing
8
12%
Protection and hygiene measures for employees and customers
3
4%
Changes in consumption habits
15
22%
Adjustment of distribution channels
4
6%
Ecommerce/Automatization/Artificial Intelligence
5
7%
Use of communication channels, as a strategy to spread the message
2
3%
178
M. Lopes and J. Reis
The most commonly used measure to mitigate COVID-19 was, social distancing. The adoption of this measure by individuals reveals a sign of altruism, since, with a minimum of 1.5 m to 2 m, allows a control of the non-dissemination of involuntary outbreaks [20, 21]. However, this measure forced retailers to invest in business support practices and rules. That required redefining the layout of stores, such as the maximum number of people per space, the reduction of opening hours, the placement of signage in static or dynamic spaces (entrance and exit), sanitizers, and displays and acrylic protection barriers [22, 23]. If on the one hand, retailers were forced to invest in these individual protection measures and their spaces to subsist in the sector, on the other hand, they witnessed a food hoarding demand caused by fear and uncertainty in food shortages that imposed hourly overload, greater exposure to risk and the allocation of more human capital [24]. Thus, the retail sector has seen a change in consumption habits, especially in food categories, that is, an increase in demand for groceries. Previously, these goods did not have a significant share in household income; however, the aforementioned demand is mainly explained due to its long lifetime [4]. Product categories, such as: personal hygiene, kitchen products, packaged foods and toilet paper, experienced an increase in demand at the expense of perishable products (i.e., yogurt, cookies, meat and fish) [25, 26]. The economic factor (income) also started to influence the purchase act, both in quantity and quality of the good and the purchase at more competitive prices in substitute products become an option [27]. In order to guarantee products on supermarket shelves and meet the needs of the civilian population, it is important to redefine supply chains to meet the sudden demand. But also, the distribution channels, which are directly affected by the closure of borders and are unable to deliver the products in time to stores [28]. The effect of food scarcity can led to panic in the purchase of non-perishable goods, as government policies announce the blockade between the borders of some countries, and which mainly affect the food and agricultural sector [29]. Home deliveries have become the most convenient and safe option for consumers, since food services were forced to close during the quarantine period [30]. The corona virus has also accelerated digital transformation in the retail sector by replacing electronic payments over cash and e-commerce [31, 32]. One example is the 5G technology that assists in the identification, tracking, distribution of goods to the final customer [33, 34]. More and more the investment on automation, e-commerce and artificial intelligence should succeed in the investment plans of retail companies, because the future will probably hang for users, who privilege this type of faster, more comfortable and safer alternatives [35, 36]. In the retail sector there is an interdependence between stakeholders and, in an epidemiological context, the flexibility in the relationships of these parties is crucial for an effective response [37]. In times of crisis, retailers may have taken on an opportunistic role, to inflate prices of goods, with higher demand, and oligopolistic behavior has emerged through a break in the supply chain, by holding greater market power. As a result, retailers who showed greater flexibility with their stakeholders through ceding and efficient management in the chain became more successful and avoided product breakdowns [38].
The Influence of COVID-19 in Retail
179
Nowadays communication, regardless of format, comes to us in a fast, accessible and rough way. In times of pandemic crisis, the media have invaded our homes with alarming news about the corona virus, although with some uncertainties, they have undermined the minds of the populations with a tragic and frightening countenance that has globally fostered an environment of insecurity, fear and uncertainty. That is why retailers should be more cautious about the product availability on the market. To avoid panic buying and hoarding non-essential products, the decision to communicate and spread a clear and reassuring message about the availability of the products is a priority. On the other hand, it allows retailers to maintain and strengthen the loyalty of their current customers and to capture new customers, who value factors such as safety, hygiene and product availability in a retail store [4, 39, 40]. In the following section, we present the conclusions, which summarizes the most relevant findings by presenting the contributions to theory and practice, and suggestions for future research.
5 Concluding Remarks Although COVID-19 has caused an unprecedented crisis, retailers must take into account a number of shortcomings already identified and find strategies to mitigate its future effects. In times of pandemic, the retail sector suffered severe economic, social and health measures; however, from these scenarios, several opportunities may arise and the retail sector must take this into account when defining a future and sustainable strategy. Change and resilience are key success factors for survival and competitiveness in the retail sector. Due to COVID-19 there was a change in consumption habits, due to fear and uncertainty. The quarantine has leveraged online commerce and online product sales, groceries increased, and the stores were redefined in their layout with protection and hygiene measures. As we had the opportunity to identify in most of the published articles, Retail is considered one of the most essential sectors, reinforced by the fact that its workforce is vital to avoid food shortages. In light of the above, one of key recommendations to retail managers is to continue investing in safety and health measures, because the human capital will lead the fight against the pandemic with uncertain terms. In addition, digital transformation has allowed for a faster evolution, because physical contact was quickly identified as a danger and, in this regard, technologies were considered essential tools for the continuity of the retail commerce, whether through electronic payments, product tracking, online orders and delivery forecasts. The communication between retailers, employees and stakeholders should be also clear and the message must convey security and trust, as it will be on those bases that the customer will return to stores after the pandemic. The relationship between all stakeholders will mobilize flexible efforts with benefits for all, culminating in a smooth delivery to the last mile customer. As this article builds on a systematic review of the existing literature, it should be interesting to conduct an empirical research on the retail industry to validate our findings. Another issue that deserves to be investigated is the continued implementation of rules and new practices to mitigate COVID-19 effects, which, due to the effect of lassitude, the lack of implementation of control rules can have negative implications on productivity, motivation and well-being of the retail workforce.
180
M. Lopes and J. Reis
Acknowledgments. We would like to thank ISLA-Santarém for supporting and financing the researcher´s participation in the WorldCist´21–9th World Conference on Information Systems and Technologies.
References 1. Hall, M., Prayag, G., Fieger, P., Dyason, D.: Beyond panic buying: consumption displacement and COVID-19. J. Serv. Manage. (2020). https://doi.org/10.1108/JOSM-05-2020-0151 2. Donthu, N., Gustafsson, A.: Effects of COVID-19 on business and research. J. Bus. Res. 117, 284–289 (2020) 3. Li, Q., Guan, X., Wu, P., Wang, X., Zhou, L., Tong, Y., Xing, X.: Early transmission dynamics in Wuhan, China, of novel coronavirus–infected pneumonia. New Engl. J. Med. 282, 1199– 1207 (2020) 4. Pantano, E., Pizzi, G., Scarpi, D., Dennis, C.: Competing during a pandemic? retailers’ ups and downs during the COVID-19 outbreak. J. Bus. Res. (2020). https://doi.org/10.1016/j.jbu sres.2020.05.036 5. Zhang, T., Wu, Q., Zhang, Z.: Probable pangolin origin of SARS-CoV-2 associated with the COVID-19 outbreak. Curr. Biol. (2020). https://doi.org/10.1016/j.cub.2020.03.022 6. World Health Organization. Novel Coronavirus disease (2019-nCoV) Situation Report – 11. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200131-sit rep-11-ncov.pdf?sfvrsn=de7c0f7_4. Accessed 18 Oct 2020 7. World Health Organization. Coronavirus disease 2019 (COVID-19) Situation Report 51. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200311-sitrep51-covid-19.pdf?sfvrsn=1ba62e57_10. Accessed 18 Oct 2020 8. Lewnard, J., Lo, N.: Scientific and ethical basis for social-distancing interventions against COVID-19. Lancet Infect. Dis. 20(6), 631–633 (2020) 9. Wu, D., Wu, T., Liu, Q., Yang, Z.: The SARS-CoV-2 outbreak: what we know. Int. J. Infect. Dis. (2020). https://doi.org/10.1016/j.ijid.2020.03.004 10. Andersen, K.G., Rambaut, A., Lipkin, W.I., Holmes, E.C., Garry, R.F.: The proximal origin of SARS-CoV-2. Nat. Med. 26(4), 450–452 (2020) 11. Zheng, Y.Y., Ma, Y.T., Zhang, J.Y., Xie, X.: COVID-19 and the cardiovascular system. Nat. Rev. Cardiol. 17(5), 259–260 (2020) 12. SNS - Serviço Nacional de Saúde. https://www.sns.gov.pt/. Accessed 18 Oct 2020 13. Yang, P., Wang, X.: COVID-19: a new challenge for human beings. Cell. Mol. Immunol. 17(5), 555–557 (2020) 14. Fernandes, N.: Economic effects of coronavirus outbreak (COVID-19) on the world economy. Available at SSRN 3557504 (2020). https://dx.doi.org/10.2139/ssrn.3557504 15. Mollenkopf, D.A., Ozanne, L.K., Stolze, H.J.: A transformative supply chain response to COVID-19. J. Serv. Manage. (2020). https://doi.org/10.1108/JOSM-05-2020-0143 16. Bove, L., Benoit, S.: Restrict, clean and protect: signalling consumer safety during the pandemic and beyond. J. Serv. Manage. (2020). https://doi.org/10.1108/JOSM-05-2020-0157 17. Petticrew, M., Roberts, H.: How to appraise the studies: an introduction to assessing study quality. Syst. Rev. Soc. Sci. Pract. Guide, 125–163 (2006) 18. Cordeiro, A.M., Oliveira, G.M.D., Rentería, J.M., Guimarães, C.A.: Systematic review: a narrative review. Revista do colégio Brasileiro de Cirurgiões 34(6), 428–431 (2007) 19. Cheng, M., Edwards, D., Darcy, S., Redfern, K.: A tri-method approach to a review of adventure tourism literature: bibliometric analysis, content analysis, and a quantitative systematic literature review. J. Hosp. Tourism Res. 42(6), 997–1020 (2018)
The Influence of COVID-19 in Retail
181
20. Mejia, G.H., Hernandez-Vargas, E.A.: When is SARS-CoV-2 in your shopping list? (2020). https://doi.org/10.1101/2020.06.11.20128850 21. Huynh, T.L.D.: Does culture matter social distancing under the COVID-19 pandemic?. Safe. Sci. 130, 104872 (2020). https://doi.org/10.1016/j.ssci.2020.104872 22. Cohen, B.H., Busis, N.A., Ciccarelli, L.: Coding in the world of COVID-19: non–face-to-face evaluation and management care. CONTINUUM: Lifelong Learn. Neurol. 26(3), 785–798 (2020) 23. Martin-Neuninger, R., Ruby, M.B.: What does food retail research tell us about the implications of coronavirus (COVID-19) for grocery purchasing habits? Front. Psychol. 11 (2020). https://dx.doi.org/10.3389%2Ffpsyg.2020.01448 24. Hailu, G.: Economic thoughts on COVID-19 for Canadian food processors. Can. J. Agri. Econ./Rev. canadienne d’agroeconomie (2020). https://doi.org/10.1111/cjag.12241 25. Shetty, G., Nougarahiya, S., Mandloi, D., Sarsodia,T., Dabade, S.J.: Covid-19 and indian commerce: an analysis of fast moving consumer goods (Fmcg), and retail industries of tomorrow. Int. J. Curr. Res. Rev. 12(17) (2020). http://dx.doi.org/10.31782/IJCRR.2020.121715 26. Rude, J.: COVID-19 and the Canadian cattle/beef sector: Some preliminary analysis. Can. J. Agri. Econ./Rev. Can. d’agroeconomie (2020). https://doi.org/10.1111/cjag.12228 27. Gupta, V., Ivanov, D., Choi, T.: Competitive pricing of substitute products under supply disruption. Omega, 102279 (2020). https://doi.org/10.1016/j.omega.2020.102279 28. Gray, R.S.: Agriculture, transportation, and the COVID-19 crisis. Can. J. Agri. Econ./Rev. Can. d’agroeconomie (2020). https://doi.org/10.1111/cjag.12235 29. Brewin, D.: The impact of COVID-19 on the grains and oilseeds sector. Can. J. Agri. Econ./Rev. Can. d’agroeconomie (2020). https://doi.org/10.1111/cjag.12239 30. Richards, T.J., Rickard, B.: COVID-19 impact on fruit and vegetable markets. Can. J. Agri. Econ./Rev. Can. d’agroeconomie (2020). https://doi.org/10.1111/cjag.12231 31. Goddard, E.: The impact of COVID 19 on food retail and food service in Canada: preliminary assessment. Can. J. Agri. Econ./Rev. Can. d’agroeconomie (2020). https://dx.doi.org/ 10.1111%2Fcjag.12243 32. Kraenzlin, S., Meyer, C., Nellen, T.: COVID-19 and regional shifts in Swiss retail payments. Swiss J. Econ. Stat. 156(1), 1–20 (2020) 33. Siriwardhana, Y., De Alwis, C., Gür, G., Ylianttila, M., Liyanage, M.: The fight against the COVID-19 pandemic with 5G technologies. IEEE Eng. Manage. Rev. 48(3), 72–84 (2020) 34. Blit, J.: Automation and reallocation: will COVID-19 usher in the future of work? Can. Public Policy 46(S2), S192–S202 (2020) 35. Kumar, M.S., Raut, R.D., Narwane, V.S. Narkhede, B.E.: Applications of industry 4.0 to overcome the COVID-19 operational challenges. Diab. Metab. Syndr. Clin. Res. Rev. 14(5), 1283–1289 (2020) 36. Hailu, G.: Economic thoughts on COVID-19 for Canadian food processors. Can. J. Agri. Econ./Rev. Can. d’agroeconomie (2020). https://doi.org/10.1111/cjag.12241 37. Bretas, V.P.G., Alon, I.: The impact of COVID-19 on franchising in emerging markets: an example from Brazil. Glob. Bus. Organ. Excellence 39(6), 6–16 (2020) 38. Ihle, R., Rubin, O.D., Bar-Nahum, Z., Jongeneel, R.: Imperfect food markets in times of crisis: economic consequences of supply chain disruptions and fragmentation for local market power and urban vulnerability. Food Secur. 12(4), 727–734 (2020) 39. Wang, Y., Xu, R., Schwartz, M., Ghosh, D., Chen, X.: COVID-19 and retail grocery management: insights from a broad-based consumer survey. IEEE Eng. Manage. Rev. 48(3), 202–211 (2020) 40. Katrakazas, C., Michelaraki, E., Sekadakis, M., Yannis, G.: A descriptive analysis of the effect of the COVID-19 pandemic on driving behavior and road safety. Transp. Res. Interdiscipl. Perspect. 7, (2020). https://doi.org/10.1016/j.trip.2020.100186
Office Evolution from Ancient Age to Digital Age (E-working) Michal Beno(B) Institute of Technology and Business in Ceske Budejovice, Ceske Budejovice 370 01, Czech Republic [email protected]
Abstract. Evolution, organisational changes, technological development and modern ways of working are transforming the workplace. We spend more time in the work cubicle than ever before. The office seems to be a second home, and its structure and design have an important impact on our productivity, happiness and well-being. Throughout the evolution of mankind, we see natural development in different stages, changing from the Ancient Age to the Digital Age in cultural, economic, technological and design aspects. During this timeline, the emphasis on the office has shifted towards the employee’s work-life-balance. This has led to dramatic changes in the shape, purpose and design of our workplaces. But the appearance of Covid-19 has turned our personal and working lives upside down. The world’s biggest e-working experiment has begun. However, should we work from home indefinitely now? How is the world of the office evolving? What could the office look like after the ongoing historical transitions? In our paper, we have studied the history of the office and its future perspectives. Keywords: Office · Ancient world · E-working · Digital age
1 Introduction Technology changes everything. It is changing all aspects of where we work, when we work, how we work and what we use to work. This affects employers, employees and their work methods/processes. Just like human evolution, business survival is contingent upon transformation and adaptability. Generally, we spend more time at the office than at home. E-working is transforming the workplace. It seems to be our second home. But Covid-19 changed everything. The future of work is suddenly here. Today’s office is global and virtual. The virtual way of working is becoming increasingly popular due to its potential for cost savings; it is also a way for an organization to be more agile and adapt to crises such as global pandemics [1]. Additionally, the workplace is constantly changing and transforming. Recent history of office life shows a natural evolution of the workplace, along with cultural, economic growth, social, political and technological changes. The office has always been the place where people collect, process, archive, receive and pass on information from others. Everyday office work was and is determined by the use of paper. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 182–192, 2021. https://doi.org/10.1007/978-3-030-72651-5_18
Office Evolution from Ancient Age to Digital Age (E-working)
183
But with the introduction of modern technology, paper consumption has decreased. Over time, work-life-balance and flexibility have become real concerns for employers. The result has been dramatic change in the workplace. Modern workplace evolution has been aided by the internet, but nowadays far more advanced processes can be carried out online. Transformation has become a must-have core capability that determines success or failure. The office is not a typical modern invention. The concept of the office started hand in hand with the need for administration in the past, e. g. Sumerian accounting, later Egyptian scribes or sekhau, ancient Greece with society’s public offices and the Roman Empire with technological and societal administrations (scriba – notary or clerk). The office was born in the monastery, where books with ancient cultural assets were produced and the spread of Christian ideas took place. Further, strict monastic rules created a uniform institution, encouraging commitment and vita communis (communal life) [2] – a variant of modern-day team work. The word office stems from “burra” – a fabric from a monk’s robe to protect papyrus and paper. Three hundred years later, the French give the table the name bureau (from burra) [3]. Additionally, scriptorium was associated with the writing of religious books in a monastic context in the early Middle Ages, the notion of a place of communal work, workshop or atelier [4]. The office as we know it today solved the problem of the need to host many people and enable communication as well as documentation. This was done by creating the workplace, which reflected local culture, resources and customs in different architectural styles throughout history. Today, such requirements are becoming less relevant, with a different focus now on producing the best work possible. It does not matter where the work is done. We are living in the era of flexibility, mobility and the digital workplace. But telework has suddenly experienced a rebound, as a result of the measures to protect citizens from the coronavirus disease [5]. To understand this situation, we have evolved a simple scheme that can be applied to the timeline of human evolution from the Ancient Age to the Digital Age. Using this device, we have tried to explore the historical relationships between changed regimes and the changes in the office. These differences are important because the office is supposed to be a very rational construction (its structure, design) requiring functionality, efficiency, flexibility and adaptability. In the following section, we briefly outline the methodology used in our research. The third section provides an account of the e-working concept. The fourth section gives a short overview of office history. The fifth section closes with a discussion and the last section comes to our conclusions.
2 Methodology A literature review is an extensive critical analysis of the literature, or research, related to a specific topic or research question [6, 7]. It is simply a synopsis of other research. Literature reviews play an important role as a foundation for all types of research serving as a basis for knowledge development by creating guidelines for policy and practice, providing evidence of an effect, and, if well conducted, having the capacity to engender new ideas and directions for a particular field [8, 9]. Systematic literature
184
M. Beno
searching is recognized as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence [10]. The increasing volume and comprehensiveness of content within bibliographic databases and other electronic sources, access to them is becoming easier, faster and less expensive [11]. In order to gain a better understanding of this phenomenon, we carried out a comprehensive literature review focusing on the extent and nature of office transformation throughout human history. Additionally, the concept of e-working has been examined. The reviewed articles were classified into description, conceptual, empirical issues. The following research questions were posed: 1) How is the world of the office evolving? 2) What could the office look like after the ongoing historical transitions? 3) Should we work from home indefinitely? To clarify these research questions, we defined the key determinant office and analysed its interconnections and links with transformations of the workplace. To select higher quality articles and reduce the total quantity of articles we decided to select keywords. The main search items included, but were not limited to, “history of the office: space, design, standards” and “office invention”. We also used broad terms that are associated with the office, such as “workplace” and “virtual or digital office”. We followed a hermeneutic approach for conducting the literature review that emphasises continuous engagement with and gradual development of a body of literature during which increased understanding and insights are developed [12]. While most of the sources used were in English, the review also includes some German sources.
3 E-working: Conceptual Framework Even before the pandemic, the nature of work was changing because the nature of business is changing. Research indicates that remote work will equal, if not surpass, fixed-office locations by the year 2025 [13]. But the Covid-19 crisis has necessitated a rise in e-working. In European Union telework increased slowly in the 10 years before the Covid-19 outbreak, although mostly as an occasional work pattern [14] and as privilege granted to the employee and approved by the employer. “Flexible working arrangements” or “e-working” are replacing a range of different terms, such as “teleworking”, “telecommuting”, “networking”, “digital nomad” and “flexi space”, which seek to describe the ways in which new information and communication technologies have made it possible for information-processing work to be carried out at a distance [15]. The classic definition of teleworking is outdated [16]. Currently, there is no international statistical definition; working from a distance and working at home are not new phenomena, but the relevance of measuring them has increased, not least due to Covid-19 [17]. According to the latest data, 98% of people would like to have the option of working remotely for the rest of their careers [18]. The history of e-working is tightly interwoven with the history of information transfer and technology development. Before the era of industrialisation, working at home was the normal way of life for craftsmen and farmers. This period can be called the period of evolution and the boom
Office Evolution from Ancient Age to Digital Age (E-working)
185
period of working from home. Rybczynski [19] argues that the 17th century represented a turning point. The first industrial revolution at the end of the 17th century was possibly the first time in human history that sustained economic growth and technological progress collided with falling living standards and employment conditions, resulting in the great social upheavals [20]. The industrialisation era dramatically changed workplace models and arrangements [21]. The concept of the “electronic homeworker” was first proposed in the automation literature in 1957 [22]. It was not until the 1970s, however, that this idea received public attention, motivated primarily by the so-called energy crisis [23]. The father of e-working is considered to be Jack Nilles [16]. ICT plays an essential role in the spread of work and e-working. The Covid-19 pandemic creates an e-working tipping point. Millions of people around the world must now work from home. E-working uses information and communications technology to allow work to be carried out independently of location [16], including e-worker (working from home or remote office, full- or part-time), teleworker (part-time at home and part-time in the office) or mobile e-worker (on the move, outside the home or office, using modern ICT). Technology development and in particular high speed broadband enables much office based work to be conducted remotely or away from the office. This, coupled with increased journey times to work has led to a greater demand for the opportunity for staff to work remote from the office and closer to home on a one or 2 days a week basis [24]. We call these employees “hybrid workers (a mixture of home and cubicle working)” working in hybrid model combining remote and in-person work. Survey data from May 2020 in 21 countries demonstrates working remotely will not be temporary and hybrid workplace is likely to become a long-term business reality [25]. Basically, e-Work means the utilisation of ICT rather than commuting to work [26]. E-working can bring substantial benefits (economic and social) for management, staff and society. According to Beˇno & Ferenˇcíková [26] e-Work is eco-friendly with a triple win situation for business, society and our planet. It reduces carbon, greenhouse emissions, energy consumption, fossil fuel reliance, paper and plastic waste and promotes better care of the environment. Employees working from home will also save money for the organization [1, 26]. Telework has proved itself to be a key component of organizational realities that are aligned with social sustainability agendas [27].
4 The Evolution of the Office When most people think about work, the office comes to mind first. The ancient world was and remains one of the most interesting subject to be examined. The history of the office can be traced back to the ancient time in the form of administration. These administrative operations of ancient mankind reflect a very old part of intelligence in chronological order from the ancient age to the digital age. Throughout the centuries, the emergence of the ruling classes, religion, writing, cities, technology development and daily commute to work formed the basis of the office. The office has become inseparable from work. This illustrates not only the changing world of work, but also the physical spaces responding to the different stages of civilisation throughout history (economic,
186
M. Beno
cultural, technological and social). The history of workplaces and especially industrial workplaces is a well-established field within the social history of labour, labour movement and industrialisation [28]. The beginning of the office goes back to the Sumerians. The people of Sumer had an unusual flair for technological invention. They are credited especially with having originated a system of cuneiform writing on clay to record important details [29]. In Egypt, the role of the clerk (scribes or sekhau) was an important step for the “mobile office”. The Egyptian word for scribe was “sesh” [30], which describes not only clerks but also the small percentage of the population who were literate and worked in the bureaucracy in varying capacities. The scribal practices and the occupations held underwent significant changes during the various periods. The scribes dealt with different financial and administrative matters. They used papyrus and ostraca (stone flakes) for their work. Both materials were stored in temples and libraries and served as the first office furniture. Over the course of history, the three mediums of “P (papyrus, parchment and paper)” were the materials most often used in the invention of writing. The office has existed in different forms throughout history as an administrative adjunct to religion, state, banking and other affairs. Greece and Rome have become the address of history, culture, religion and language, as well as of politics, literature and business, as can be seen by the temples, sculptures, pottery and other archaeological findings (e.g. ancient Greek with society’s public offices [31], the Roman Latin term “officium”, which can be translated as duty, service or office [32], and the technological and societal administrations (scriba – notary or clerk)). The origin of the modern office goes back to mediaeval times, with large-scale organisations such as governments, trading companies and religious orders that required written records or documentation, e.g. ancient culture and the spread of Christian ideas [3]. The mediaeval monks used a “scriptorium”, a so-called writing room, where they copied old papyrus and later parchment rolls and turned them into unique books. The word office stems from “burra” – a fabric from a monk’s robe to protect papyrus and paper. Three hundred years later, the French give the table the name bureau (from burra) [3]. The earliest dated printed book was a Chinese copy of the so-called Diamond Sutra, one of the most important textbooks of Buddhism [33], later in 1440 it was the Gutenberg Bible. Monks, scriptorium, clerks, and universities sparked the development of a uniform type of space in the 18th century, the period of the Enlightenment. These concepts were, however, only turned into reality with the onset of industrialisation (scientific development and manufacturing, transition from craftsmanship to industrial manufacturing with the rise of new professions involving organisational and administrative duties). While these rooms were reserved for office functions, they were not yet specially furnished for office work. Nonetheless, the office became a strategic place, which promoted creativity and provided an adequate setting for streamlining, planning and developing various types of work [34]. Factories were the symbol of the industrialisation at the start of the 19th century, offices are emblematic of the current post-industrial era [35] (p. 9). In late 19th century, the boom in building construction started, and the first commercial offices appeared in the northern industrial cities of the United States. With the invention of the telegraph
Office Evolution from Ancient Age to Digital Age (E-working)
187
and telephone, offices could be situated away from the home and the factory [36]. The first office designs in the 20th century were based on factory floor spaces, aiming at total efficiency. The operation of the office consisted of routine work based on factory assembly line principles [37]. Van Meel [35] and Caruso St. John Architects [36] call them “white collar factories”. People needed to go to the office to enter and access paper documents and perform their tasks. Nowadays these offices are called “Taylorist offices” [38–40]. They were based on the need for high supervision of work done, and managers were seated around the open area in such a way that they could oversee the employees. New technologies such as electric lighting, the typewriter and the use of calculating machines allowed large amounts of information to be accumulated and processed faster and more efficiently than before. The concentration of wealth in the new corporations required an ever-greater proportion of an increasingly literate population to work in the office [35, 39]. With the improvement of building techniques in the early 20th century, offices became higher, more aesthetic and were designed for speeding up work processes [40]. A surge of “glass boxes” describes the period [35]. In Germany in the 1960s, the Quickborner team of management consultants developed the radical office layout idea of “Bürolandschaft” (in German) or office landscape [35, 36, 41, 42]. This broke the total control of employees by superiors in executive offices by dissolving the clear grid structures in which the office furniture had previously been organised, and dividing them into small, organic groups [41]. The next period is called “the rise of cubicles” or “evil twin” [41], where open layouts were separated by small partitions to provide some personal space for employees, providing a cost-effective way for organisations to use their space, e.g. Herman Miller and the operation of the office in the 1960s [40, 42], the cubicle farm in the 1980s [40, 42] and the combination offices with glass walls to open up vistas and avoid the feeling of isolation [41]. In 1985 workplace strategy was introduced, when Philip Stone and Robert Luchetti [43] declared “Your office is where you are”. Furthermore, in this period the rise of the personal computer resulted in the “electrification” of the office [35]. With the spread of the internet, modern office models have been created. From the mid-20th century to the present, the idea of the organisation and the workplace as a collaborative system continued to develop. Modern input-output models were created. Humans, finance, physical parameters and information were considered inputs, which, through a process transformation, were changed to products and services [44]. With the development of communication technologies, the speed, flexibility and independence of work increased [45]. Through the internet, the widespread occurrence of globalisation and digitisation, anyone can now conduct almost any business in virtual space rather than physically. The virtual office is a service that enables employees and business owners to work remotely [46]. Different workplaces became much more collaborative. Hot desking and desk sharing evolved out of the need to conserve space and provide a landing spot for employees who mainly work outside the office. Suddenly our workplaces have changed to reflect both work and individual demands, as well as to provide employees with more flexibility where, when, how they work and what they use for work. All of this has been enabled by e-working models.
188
M. Beno
Due to mounting social, public, national and global pressure, and employers’ and employees’ rising expectations, organisations are increasingly being expected to go beyond the legal issues and act immediately in the face of Covid-19. Creating flexible workplaces that protect employees and prevent further infection is a possible way in which organisations can meet their expectations of higher productivity and competitiveness. Since there are not many tools available for creating e-working workplaces, on the basis of our results we generalised a model that can guide organisations in managing their e-working in a balanced and positively expanding manner [47]. As the workplace and society adopt more flexible work arrangements, it is undeniable that e-working is here to stay. We have developed a simple basic three-dimensional model of future trends and factors affecting e-working (see Fig. 1) [47]. Each of the three dimensions, namely Technology, Global changes and Generational and demographic changes, is within the control of the individual country, and each one can be developed through appropriate support and guidance. Effective e-working is a matter of developing the ability within each of the three dimensions and then choosing to apply the eight controlling factors and barriers in a balanced way: 1) Talent acquisition and retention; 2) Eco-friendliness; 3) Cost reduction; 4) Modern technology; 5) Upgrading and security; 6) Big brother is watching you and legislation; 7) Digital natives and global citizens and 8) New labour market models and jobs. Our model provides a systematic way of incorporating best practices to overcome concerns and guide organisations through the process of building successful, sustainable and customised e-working programmes that meet the demands of a changing workplace.
Fig. 1. Out-of-office model
Our model is intended to help managers, employers and countries to obtain a better understanding of future e-working. It will be useful to everyone by describing the trends, factors and barriers, and it is organised in a way that will help everyone understand how e-working can develop. A flexible workplace is one in which employees and employers (managers) collaborate to use a model to promote e-working, well-being and the sustainability of the workplace by considering this Out-of-office Model. Working models will quickly shift from futuristic ideas to standard practice.
5 Discussion and Conclusions The office, the workplace and their management have always been connected to physical space and organisation. Throughout human evolution, the concept of the office has evolved in terms of social, economic and technological changes. However, it seems
Office Evolution from Ancient Age to Digital Age (E-working)
189
that employee requirements were not always included in the future development. Now, work is more and more based on projects, collaboration, the war of talent (talent is a critical driver for corporate performance) and the flexible workplace experience with its many advantages and some limitations. E-working will not be the same as in the past; the workforce is about to change dramatically. Does a sustainable workplace not mean “plus ça change, plus c’est la même chose”? Alphonse Karr’s words [48] remain true after 171 years. But we still seem to be getting the basics wrong, e. g. micromanagement and a culture of presenteeism (at work, but out of it) are not generally accepted for e-working and count as the biggest threats for a company’s productivity. Huge advances in technology have made the workplace more flexible and trustworthy. Six months into the global pandemic, organisations are now asking: Should we work from home indefinitely? And if they do decide to make major organisational changes about e-working, will they see the same productivity? The office of the future will most likely include highly networked, shared, multipurpose spaces that redefine boundaries between companies and improve everyone’s performance [49]. This paper covers a significant portion of powerful timelines through mankind’s evolution in relation to the office. The importance of adaptability in office design has evolved from Ancient Age to Digital Age, mostly due to many factors e.g. religion, culture, values, rapid change in our society, both in private and public organisations, modern technology, modern and innovative workplaces, growing personal and environmental and now pandemic concerns. Generally, today’s office was invented to bring people from their homes to the office so as to allow these employees to communicate easily and to work. Today, such needs are becoming less relevant, and in the time of Covid-19 the focus is very different. In the past, e-working was a privilege, now it is a lifebuoy. The world of work is constantly changing, new opportunities and requirements are emerging rapidly. Adapting the office landscape to future developments will be a challenge, especially after this pandemic. Whether the trends observed during Covid-19 (what organisations learnt about their employees working from home) will continue into the future, stabilise at their current levels, or reverse direction cannot be easily predicted. Workplaces have to become “learnplaces” [50]. However, taken together, the changes in the context of work during the course of history are having significant effects on the content of work. No single trend captures the changes in how work is done today. Nothing in the data we have examined would support the conclusion that all the changes in today’s workplace add up to “the end of the office” in any sense of this term. The conditions and content of work are certainly changing, sometimes in dramatic ways, but e-employees who want or need to work remain employed. Generally, the changes or trends we have discussed in our paper constitute something that could be characterised as a transformation of the office. When combined, as we have seen during Covid-19, they may lead to new conditions and possibilities, and these factors can be exploited to the benefit of all the parties concerned. Today, most companies see the workplace as a source of competitive advantage, not simply a cost burden [51] (p. 57). We agree with the Williams [51] that well-designed offices (including e-working models) increase productivity, save money for companies, attract top talent and have an impact on the environment. Finally, understanding the role of policies at the workplace level is crucial when it comes to fostering employability
190
M. Beno
over the life course. Workplace-based measures of reskilling and upskilling have to be available for all workers, not only for those in standard forms of employment [52, 53]. To enhance human capabilities and skills development, technology must be designed with human factors in mind [54], employees’ demands and labour market requirement.
References 1. Raišien˙e, A.G., et al.: Working from home—who is happy? a survey of Lithuania’s employees during the COVID-19 quarantine period. Sustainability 12(13), 5332 (2020) 2. Barrow, J.: Vita Communis or Separate Houses? Where did Canons Live in the 10th and 11th Centuries? https://eprints.whiterose.ac.uk/135347/3/J%20Barrow%20VOR.pdf. Accessed 29 Sept 2020 3. Eickhoff, H., Teunen, J.: Burra: Der fabelhafte Aufstieg der Büroarbeit. Friedrich Blaha, Germany (2018) 4. Stones, A.: Scriptorium: the term and its history. Perspective 1, 113–120 (2014) 5. Belzunegui-Eraso, A., Erro-Garcés, A.: Teleworking in the context of the Covid-19 crisis. Sustainability 12(9), 3662 (2020) 6. Clare, J., Hamilton, H.: Writing Research: Transforming Data into Text. Churchill Livingstone, Edinburgh (2003) 7. Machi, L.A., McEvoy, B.T.: The Literature Review: Six Steps to Success. Sage Publications, Corwin (2008) 8. Snyder, H.: Literature review as a research methodology: an overview and guidelines. J. Bus. Res. 104, 333–339 (2019) 9. Van Wee, B., Banister, D.: How to write a literature review paper? Transp. Rev. 36(2), 278–288 (2016) 10. Cooper, C., et al.: Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med. Res. Methodol. 1(1), 85 (2018) 11. Page, D.: Systematic literature searching and the bibliographic database haystack. Electron. J. Bus. Res. Methods 6(2), 171–180 (2008) 12. Boell, S.K., Cecez-Kecmanovic, D.: A hermeneutic approach for conducting literature reviews and literature searches. Commun. Assoc. Inf. Syst. 34(1), 257–286 (2014) 13. Kennedy, J.: Remote working will rival fixed office locations by 2025. https://www.siliconre public.com/careers/remote-flexible-working-fixed-offices-future-of-work. Accessed 29 Sept 2020 14. Telework in the EU before and after the COVID-19: where we were, where we head to. https://ec.europa.eu/jrc/sites/jrcsh/files/jrc120945_policy_brief_-_covid_and_telework_ final.pdf. Accessed 29 Sept 2020 15. Bates, P., Bertin, I., Huws, E.: E-work in Ireland. https://www.employment-studies.co.uk/sys tem/files/resources/files/394.pdf. Accessed 29 Sept 2020 16. Beno, M.: Transformation of human labour from stone age to information age. In: Mobile Web and Intelligent Information Systems, Lecture Notes in Computer Science, vol. 10995, pp. 205–216. Springer, Cham (2018) 17. ILO, COVID-19: Guidance for labour statistics data collection. https://ilo.org/wcmsp5/gro ups/public/---dgreports/---stat/documents/publication/wcms_747075.pdf. Accessed 29 Sept 2020 18. The 2020 State of remote Work. https://lp.buffer.com/state-of-remote-work-2020. Accessed 29 Sept 2020 19. Rybczynski, W.: Home: A Short History of an Idea. Penguin Books, New York (1987) 20. Polanyi, K.: The Great Transformation. Beacon Press, Boston (2001)
Office Evolution from Ancient Age to Digital Age (E-working)
191
21. Beno, M.: Working in the virtual world - an approach to the home office business model analysis. AD ALTA 8(1), 25–36 (2018) 22. Jones, J.C.: Automation and Design (1–5), Design, Nos. 103, 104, 106, 108, 110 (1957–1958) 23. Huws, U.: Telework: projections. Futures 23(1), 19–31 (1991) 24. WDC, e-Working in the Western Region, WDC policy briefing no. 7. https://www.wdc.ie/ wp-content/uploads/WDC_Policy-Briefing-no-7.pdf. Accessed 29 Sept 2020 25. Arubanetworks, Preparing for the post-pandemic workplace. https://www.arubanetworks. com/assets/eo/Hybrid-Workplace-Report.pdf. Accessed 29 Sept 2020 26. Beˇno, M., Ferenˇcíková, S.: The future of work is e-work. In: 14th IWKM 2019, 7–8 Nov 2019, Bratislava - Trenˇcín, Slovakia, pp. 6–20 (2019) 27. Gálvez, A., Tirado, F., Jesús Martínez, M.: Work-life balance, organizations and social sustainability: analyzing female telework in Spain. Sustainability 12(9), 3567 (2020) 28. Rainhorn, J., Bluma, L.: History of the workplace: environment and health at stake – an introduction. European Review of History: Revue européenne d’histoire 20(2), 171–176 (2013) 29. Krammer, S.N.: The Summerians. https://oi.uchicago.edu/sites/oi.uchicago.edu/files/upl oads/shared/docs/sumerians.pdf. Accessed 29 Sept 2020 30. Shaw, I., Nicholson, P.: British Museum Dictionary of Ancient Egypt. British Museum Press, London (1997) 31. Lane, M.: The idea of accountable office in ancient Greece and beyond. Philosophy 95(1), 19–40 (2020) 32. Officium. https://www.nihilscio.it/Manuali/Lingua%20latina/Verbi/Coniugazione_latino. asp?verbo=officium. Accessed 30 Sept 2020 33. Sack, H.: The very first Printed Book – The Diamond Sutra. https://scihi.org/diamond-sutra/. Accessed 30 Sept 2020 34. Bene, Inventing the office: Living and working in the days of the Enlightenment. https://bene.com/en/office-magazine/inventing-office-living-and-working-days-enligh tenment/. Accessed 30 Sept 2020 35. Van Meel, J.: The European Office: Office Design and National Context. COIO Publishers, Rotterdam (2000) 36. Caruso St. John Architects, Mozas, J., Per, A. F.: The Office on the Grass. The Evolution of the Workplace. A+t Architecture publishers, Vitoria-Gasteiz (2017) 37. Roy, B.: A Brief History of Management. https://medium.com/open-participatory-organized/ a-brief-history-of-management-23361290a08b. Accessed 30 Sept 2020 38. Kuang, C.: Design: Office Spaces. Wired 17(4) (2009). 39. Gou, Z.: Workplace design revolution: the inside-out Urbanism. In: book: Design Innovations for Contemporary Interiors and Civic Art, Chapter 12. IGI Global Editors, Italy (2017). 40. Lovell, M.: The evolution of office design. https://www.morganlovell.co.uk/the-evolution-ofoffice-design. Accessed 30 Sept 2020 41. Büro der Zukunft. https://world-architects.com/files/static_pages/micropages/paperworl d19_impressions/sonderausstellung/MATTER-BDZ-Flexibilitaet-175x250mm26NOV2 018-V27-LINKED%20BILD.pdf. Accessed 30 Sept 2020 42. Branch, The Rise and Fall (and Rise) of Office Cubicles. https://www.branchfurniture.com/ blogs/turn-key/history-of-office-cubicles. Accessed 30 Sept 2020 43. Stone, P.J., Luchetti, R.: Your office is where you are. Harvard Bus. Rev. 63(2), 102–112 (1985) 44. Liebler, J.G., McConell, C.: Management Principles for Health Professionals. Jones & Bartlett Learning, Sudbury (2012) 45. The impact of ICT on job quality: Evidence from 12 job profiles. https://eskillsjobsspain. com/wp-content/uploads/2016/06/TheimpactofICTonjob.pdf. Accessed 30 Sept 2020
192
M. Beno
46. Jerry, U.A.: Impact of virtual office on business environment. Int. J. Bus. Manage. 8(2), 76–84 (2020) 47. Beno, M.: Out-of-office model – the eight factors and barriers controlling the future of eworking. ACTA UNIVERSITATIS DANUBIUS 16(4), 150–163 (2020) 48. Plus ça change, plus c’est la même chose. https://www.histoire-en-citations.fr/citations/Karrplus-ca-change-plus-c-est-la-meme-chose. Accessed 30 Sept 2020 49. Waber, B., Magnoffi, J., Lindsay, G.: Workspaces that Move People. https://hbr.org/2014/10/ workspaces-that-move-people. Accessed 30 Sept 2020 50. Neufeind, M., O’Reilly, J., Ranft, F.: Work in the Digital Age. Challenges of the forth Industrial Revolution. Rowman & Littlefield International Ltd., London (2018) 51. Williams, J.: Looking Backward, Moving Forward: The Near Future of the Workplace. https:// fmj.ifma.org/publication/?m=30261&i=378064&p=56&pp=2. Accessed 30 Sept 2020 52. Dekker, F., Keune, M.: Netherlands: the sectoral impact of digitalization on employment and job quality. In: Work in the Digital Age. Challenges of the forth Industrial Revolution. Rowman & Littlefield International Ltd., London (2018) 53. Doellgast, V. L.: Rebalancing Worker power in the networked economy. In: Work in the Digital Age. Challenges of the forth Industrial Revolution. Rowman & Littlefield International Ltd., London (2018) 54. Taylor, M.: Good work: the Taylor review of modern working practices. https://assets.pub lishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/627671/ good-work-taylor-review-modern-working-practices-rg.pdf. Accessed 30 Sept 2020
Measuring Variability in Acute Myocardial Infarction Coding Using a Statistical Process Control and Probabilistic Temporal Data Quality Control Approaches J. Souza1,2(B)
, I. Caballero3 , J. V. Santos1,2,4 , M. F. Lobo1,2 J. Viana1,2 , C. Saez5 , and A. Freitas1,2
, A. Pinto1,2
1 Department of Community Medicine, Information and Health Decision Sciences
(MEDCIDS), Faculty of Medicine, University of Porto, Porto, Portugal 2 Center for Health Technology and Services Research (CINTESIS), Faculty of Medicine,
University of Porto, Porto, Portugal 3 University of Castilla-La Mancha, Castilla-La Mancha, Ciudad Real, Spain 4 Public Health Unit, ACES Grande Porto VIII–Espinho/Gaia, Vila Nova de Gaia, Portugal 5 Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y
Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Valencia, Spain
Abstract. Acute Myocardial Infarction (AMI) is frequently reported when coding hospital encounters, being commonly monitored through acute care outcomes. Variability in clinical coding within hospital administrative databases, however, may indicate data quality issues and thereby negatively affect quality assessment of care delivered to AMI patients, apart from impacting health care management, decision making and research as a whole. In this study, we applied statistical process control and probabilistic temporal data quality control approaches to assess inter-hospital and temporal variability in coding AMI episodes within a nationwide Portuguese hospitalization database. The application of the present methods identified affected data distributions that can be potentially linked to data quality issues. A total of 12 out of 36 institutions substantially differed in coding AMI when compared to their peers, mostly presenting lower than expected hospitalizations of AMI. Results also indicate the existence of abnormal temporal patterns demanding additional investigation, as well as dissimilarities of temporal data batches in the periods comprising the recent transition to the International Classification of Diseases, 10th revision, Clinical Modification (ICD-10-CM) and changes in the Diagnosis-Related Group (DRG) software. Hence, the main contribution of this paper is the use of reproducible, feasible and easy-to-interpret methods that can be employed to monitor the variability in clinical coding and that could be integrated into data quality assessment frameworks. Keywords: Data quality · Clinical coding · Data variability · Acute myocardial infarction · Clinical classification software · International classification of diseases
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 193–202, 2021. https://doi.org/10.1007/978-3-030-72651-5_19
,
194
J. Souza et al.
1 Introduction Ischemic heart disease remains the leading cause of death worldwide [1], with Acute Myocardial Infarction (AMI) being frequently reported when coding hospitalizations [2]. Inpatient coded data has been widely used for research, management and for measuring the processes of care through quality indicators, with AMI being one of the standard conditions for reporting of quality in acute care [3]. Monitoring AMI is important as substantial variations in practices exist across hospitals, despite being a condition for which treatments have gone through several advances over the last decades and despite attempts by medical societies to standardize AMI care through guidelines [4]. Nevertheless, it is critical to understand whether variation in AMI outcomes found in administrative databases are reflecting actual care performance. Assessment of care through quality indicators, as well as other processes related to health care management and research, are highly dependent on the quality of coded data in these databases [5]. In Portugal, information concerning the patient’s health condition, disease progression, medical procedures and treatments during hospitalizations are routinely reported in health records, discharge notes, pathology and surgical reports [6]. These sets of unstructured information are further translated into standard clinical codes by trained medical coders, according to the International Classification of Diseases, Clinical Modification, ninth and tenth revisions (ICD-9-CM/ICD-10-CM), with the latter one being issued in 2016 and put into practice in January 2017 at national level [7]. All produced coded inpatient and outpatient data from all public hospitals within the Portuguese National Health System (NHS) is gathered and maintained at the National Hospital Morbidity Database, resulting in a multisite data repository which is mainly used for hospital reimbursement, but also in several subdomains, ranging from health care management and decision-making, resource use monitoring, research and assessment of care by means of quality indicators [8]. Study designs involving multiple sites and databases can help to evaluate data quality by means of data variability, which in turn reflects the degree to which data is credible to fulfill its purposes [9]. In the context of this study, data variability among multiple sites (e.g., hospitals) may occur under natural circumstances, but it can also indicate data quality issues [10]. Besides variability between sources, temporal variability must also be considered, which occurs when differences are found over time, potentially leading to inaccurate, irreproducible or invalid information [11]. Concerning coded clinical data from hospitals, variability could be related to systematic failures, such as lack or nonadherence to guidelines or data definitions, transition and adaptation to different coding systems and versions of Diagnosis-Related Groups (DRGs) grouper software [6, 12, 13]. It can also occur due to random circumstances, such as typing or transcription errors, as well as failures in coding diseases, considering the highly subjective nature of this process, which is also critically linked to the level of quality of inpatient documentation [14]. Our hypothesis is that the variability in the diagnostic assignment of AMI may manifest as differences between hospitals regarding their distribution of AMI cases at the same point in time or as marked fluctuations over time. Although some degree of variability is expectable and may be determined by several factors, an expressive variability is commonly resulted from instability and inconsistency in the diagnostic assignment process [15]. In this study, inter-hospital and temporal variability in coding AMI within
Measuring Variability in Acute Myocardial Infarction Coding
195
the Portuguese National Hospital Morbidity Database was assessed. A Statistical Process Control (SPC) and a probabilistic temporal data quality control approaches were used to detect extreme variations that could indicate possible data quality issues in outlying hospitals.
2 Methods 2.1 Data Sources Data assessed in this study was extracted from the National Hospital Morbidity Database. We abstracted data from all inpatient episodes with a discharge date between January 1st 2011 and December 31st 2017. We included the years comprising the transition to ICD10-CM, which in turn could be a potential source of data variability. Variables retrieved included the patient’s age, sex, hospital, admission and discharge date and principal diagnosis. We included only inpatient episodes that were considered statistically valid, that is, with a hospital stay of at least 24h, or shorter than 24h for patients who died, left against medical advice or were transferred to another institution. Since hospitalization data was anonymized through an arbitrary episode identification number, there was no need for ethical approval. 2.2 Multisource and Temporal Data Variability Assessment We assessed multisource variability in terms of standardized AMI hospitalization rates across hospitals, defined as the ratio between observed and expected number of AMI discharges. Annual standardized rates were calculated using indirect standardization. A Logistic Regression model including age, sex and an interaction between age and sex was used to estimate the number of expected AMI cases. The Agency for Healthcare Research and Quality (AHRQ) Clinical Classification Software (CCS) was used to group each episode into meaningful and mutually exclusive groups representing specific diseases [16]. Episodes were classified based upon a principal diagnosis code within the subchapter 410 of the ICD-9-CM, or within the I21/I22 subchapters of the ICD-10-CM, following the CCS definitions for AMI. We considered standardized rates to account for differences in the populations admitted in each hospital. To further minimize the influence of hospital specific characteristics on multisource variability, we also performed the analyses by groups of hospitals according to an official categorization proposed by the Central Authority for Health Services (ACSS) for the Portuguese NHS hospitals, which clustered the institutions into five groups (Groups B, C, D, E and F). This grouping is based upon a hierarchical clustering method following the standardization of variables explaining hospital costs, thereby related to the casemix of the institutions [17]. Hospitals that are not categorized according to the ACSS method and oncology hospitals (group F) were excluded from the analysis, as well as those institutions with missing data for some years. A SPC method based on funnel plot assessment, which has been widely used for institutional comparisons in healthcare [18], was adopted to assess multisource variability (inter-hospital) in coding AMI. The SPC concept defines statistical boundaries
196
J. Souza et al.
to separate common-cause variation from special-cause variation. When funnel plot is used for institutional comparisons, a scatterplot of a given indicator against a measure of its precision (e.g., sample size) is drawn along with a horizontal line representing the group average or equal to 1 for standardized rates. It includes control limits at 95% (approximately two standard deviations) and 99.8% confidence levels (approximately three standard deviations) for a p-value assuming that indicator Y (standardized AMI rate) presents a known distribution with its respective parameters (8, σ). In this study, we assumed Poisson distribution to characterize AMI standardized hospitalization rates across the hospitals, using the exact formula to draw Poisson control limits. Overdispersed limits using hierarchical models at hospital level according to the method described by Spiegelhalter (2012) [19] were also generated, in which a second variance component for the “between-hospital” standard deviation is calculated, which in turn is added to the “within-hospital” standard deviation (σ). To assess temporal variability, we applied the method proposed by Sáez et al. (2015, 2016) [11, 20], in which probability distributions are compared across different periods of time by calculating the Jensen-Shannon distance (JSD). The JSD is bounded between zero and 1, thereby comparable across studies, in which value 1 represents distributions with no common values. In databases with low variability, JSDs among distributions tend to be small, whereas different or anomalous data distributions would mean higher JSD. The main advantage of using JSD distance is that the measurements are not affected by large sample sizes or multimodal distributions, offering a robust alternative to classical statistical tests for distribution comparison [20]. Temporal heat maps and InformationGeometric Temporal (IGT) plots based on JSD [11] were employed. These visualization techniques that helps to uncover temporal trends and to identify abrupt or recurrent changes in distributions over time. All analyses were performed using RStudio version (RStudio Team, Boston, Massachusetts, United States), version 1.2.1335, in particular the R packages “FunnelPlotR”, that implements the methods described by Spiegelhalter (2005) [18], and package “EHRtemporalVariability” for temporal variability assessment by means of JSD measurements and IGT plot [21].
3 Results Data on 6,218,657 hospitalizations occurred between 2011–2017 were abstracted from the Portuguese Hospital Morbidity Database. From this number, a total of 77,459 AMI admissions distributed across the 36 included hospitals were identified. Analyses were also performed for four groups of similar institutions based upon a classification provided by ACSS. Group B was composed of 8 hospitals (Hospital B1 to B8), with 4,055 AMI admissions; Group C was composed of 15 hospitals (Hospital C1 to C15), with 23,908 AMI admissions; Group Dwas composed of 7 hospitals (Hospital D1 to D7), with 22,906 AMI admissions; Group E was composed of 6 hospitals (Hospital E1 to E6), with 26,590. Figure 1 summarizes the funnel plot analysis of standardized hospitalization rates of AMI, considering all 36 hospitals and using data for the whole studied period. Institutions that were out of the control limits established at 95% Poisson adjusted for overdispersion (purple lines) are labelled. A total of 10 distinct hospitals were considered outliers in
Measuring Variability in Acute Myocardial Infarction Coding
197
AMI standardized hospitalization rates following the construction of the funnel plots, from which 8 presented lower than expected hospitalizations of AMI, whereas only 2 presented significantly higher rates in comparison with their peers.
Fig. 1. Funnel plot of the standardized AMI hospitalization rates in 36 Portuguese hospitals during 2011–2017
Funnel plots per hospital group using annual data were also generated in order to identify possible changes in coding patterns that could be introduced by modifications related to clinical coding that typically occurred from one year to another (e.g. transition to ICD-10-CM, changes in DRG grouper software). When assessing funnel plots by year and hospital group, the total number of distinct outlying institutions increases to 12. Among group B, 3 hospitals were labelled as outliers, presenting substantially lower than expected AMI admissions during the studied period (hospitals B1, B3 and B4). Regarding Group C, hospital C7 presented an extreme hospitalization of AMI during the entire studied period, contrasting with hospitals C8, C12 and C13, which presented lower than expected rates of AMI in some years. Hospital C14 also presented a higher than expected AMI hospitalization rate in 2011. Hospital D7 presented considerably higher AMI rates among Group D hospitals, being out of the control limits in 2013, 2014 and 2017. Considering Group E, hospital E1 was placed out of the control limits during the entire period, with a substantially higher hospitalization of AMI in comparison with its peers. Moreover, hospital E5 was also an outlying institution, with a substantially lower than expected hospitalization of AMI in 2015. To assess temporal variability, monthly relative frequencies of AMI episodes (considering the discharge date) in each hospital were assessed. Figure 2 presents the temporal heat map of monthly AMI relative frequencies for the studied period, in which is possible to detect affected distributions in several institutions over the years. The heat map is ordered by AMI frequency, where institutions located at the top are those presenting lower frequencies of AMI episodes, being mostly from group B. A sudden increase was observed for hospital C12 in the end of 2016, contrasted with an abrupt and prolonged decrease in AMI frequency in hospital D4 between 2012 and middle 2013. Another
198
J. Souza et al.
noticeable anomalous pattern was found for hospital E2 for the entire year of 2016, in which a much lower frequency of AMI cases can be observed in comparison with the remaining years.
Fig. 2. Temporal heap map of monthly AMI relative frequency across Portuguese hospitals, 2011– 2017
Furthermore, we constructed IGT plots proposed by Sáez et al. to assess temporal variability, considering data from all hospitals. In Fig. 3, the IGT plot of the monthly AMI relative frequencies can be seen for the entire period. The distance between data points represents the JSD between their values (AMI relative frequency). The flow of data points seems to evolve irregularly over the studied period, and two abrupt changes were observed between the transition 2013–2014 and 2015–2016, forming three temporal subgroups (I - before 2014, II - between 2014 and 2016, and III - after 2016).
Fig. 3. IGT plot of AMI monthly hospitalization across Portuguese hospitals, 2011–2017
Measuring Variability in Acute Myocardial Infarction Coding
199
4 Discussion The emphasis of this article was to detect possible data quality issues reflected in anomalous variability in coding AMI over time and between sources (hospitals). Assessing variability in coding AMI is relevant in a sense that it is a condition with a high hospitalization burden and which is often used in institutional comparisons for acute care quality assessment. Under the clinical perspective, inter-hospital variability in coding AMI can be linked to several factors. How and where the Myocardial Infarction (MI) occurred is what typically drives ICD code assignment. The site where AMIs occurred corresponds to the coronary artery involved (e.g., inferior wall, anterior wall), and thereby is highly dependent on the quality of inpatient documentation in order to capture the most specific ICD codes available [2]. In 2013, a new universal definition of MI was introduced, with changes covering the distinction between myocardial ischemia and myocardial injury and adding new criteria to define MI based on the setting of cardiac interventions [22]. This new definition may partially explain the observed variability after 2014 (Fig. 3, batch II). Furthermore, the recent transition to ICD-10-CM in Portugal adds yet more complexity into this task as it offers new codes to further specify type and cause of MI [23]. The application of funnel plots with standardized rates was useful to detect several outlier coding patterns potentially related to data quality issues. Most of the outlier institutions presented lower than expected rates. It is important to point out that AMI might occur in an inpatient episode as comorbidity or subsequent diagnosis following admissions for other causes [24]. In this sense, hospitals with substantially lower frequencies may present a higher AMI code distribution in the secondary diagnoses fields. That may severely affect measurements of quality of care, such as 30-day AMI in-hospital mortality, which often includes only patients with a principal diagnosis of AMI [24]. The analysis of IGT plots highlights a change in data patterns after 2014, comprising the transition to APR-DRG from AP-DRG [25], which is a more refined DRG version, with a novel patient stratification concept that is more dependent on secondary diagnoses coding, and thus may have driven different coding behaviors [26]. Furthermore, distinct temporal patterns can be observed for the years 2016 and 2017, corresponding to the transition to ICD-10-CM, which started at national level in January 2017. Other temporal batches of data seem to be affected by systematic processes, namely abrupt decrease and increase in AMI frequency in hospitals D4, C12 and E2, requiring thus additional investigation on data quality issues. The use of these methods has underlying advantages in the context of data quality assessment in health care, namely its usability for any type of data and clinical domain, capacity to detect anomalous patterns that can be potentially linked to data quality issues, as found in our study, as well as easy-to-interpret visualization tools to detect affected data batches, providing a feasible, reproducible and usable method to be integrated into data quality assessment frameworks. However, some limitations should be discussed. Multisource comparison was based upon standardized rates calculated by the indirect method, and the individual rates compares a hospital with the reference population, comprised by the sum of all hospitals. Therefore, rates are not stable if a specific hospital presents an age distribution that substantially differs from the reference population. Moreover, only age and sex were used to estimate the expected number of cases and we
200
J. Souza et al.
did not consider other factors that can influence variability, such as differences in coding practices and other hospital structural features (capacity, available medical technology, specialty level, etc.) that can be not reflected in the ACSS grouping.
5 Conclusions We described and applied a method based on SPC and probabilistic temporal data quality control for monitoring multisource and temporal variability in coding AMI, using a nationwide Portuguese hospital database. Analyses were also performed by groups of hospitals sharing similar case-mix characteristics. Outlying hospitals mostly reported less AMI admissions than expected, which might be related to a tendency of coding AMI as comorbidity or as an acquired diagnosis to other admission causes. Three distinct time batches were observed and appear to coincide with changes impacting data generation, such as transition to ICD-10-CM and changes in DRG grouper software. We employed feasible, reproducible and easy-to-interpret methods based on previous literature that can be useful for discovering abnormal patterns in health care data, helping to detect potential systematic issues affecting the quality and usability of coded data for health care management, quality assessment and research. Future works should include the further investigation and a possible association between abnormal data patterns found through these methods and key factors affecting data quality. Acknowledgements. The authors would like to thank the Central Authority for Health Services, I.P. (ACSS) for providing access to the data. The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was financed by FEDER-Fundo Europeu de Desenvolvimento Regional funds through the COMPETE 2020Operacional Programme for Competitiveness and Internationalisation (POCI) and by Portuguese funds through FCT-Fundação para a Ciência e a Tecnologia in the framework of the project POCI01–0145-FEDER-030766 (“1st.IndiQare-Quality indicators in primary health care: validation and implementation of quality indicators as an assessment and comparison tool”). In addition, we would like to thank to project GEMA(SBPLY/17/180501/000293)- Generation and Evaluation of Models for Data Quality, funded by the Department of Education, Culture and Sports of the JCCM and FEDER; and to ECLIPSE project (RTI2018–094283-B-C31) funded by the Spanish Ministry of Science, Innovation and Universities, and FEDER.
References 1. World Health Organization. The top 10 causes of death. https://www.who.int/news-room/ fact-sheets/detail/the-top-10-causes-of-death. Accessed 19 Nov 2020 2. Carr, K., Romanello, L.: Coding an acute myocardial infarction: unravelling the mystery. https://www.prismhealthcare.com/library/coding-an-acute-myocardial-infarctionunravelling-the-mystery/. Accessed 19 Nov 2020 3. Schiele, F.: Quality indicators for acute myocardial infarction. Eur. Heart J. 6(1), 34–59 (2016) 4. Rehman, S., Li, X., Wang, C., Ikram, M., Rehman, E., Liu, M.: Quality of care for patients with acute myocardial infarction (AMI) in Pakistan: a retrospective study. Int. J. Environ. Res. Public Health 16(20), 3890 (2019)
Measuring Variability in Acute Myocardial Infarction Coding
201
5. Shepheard, J.: Clinical coding and the quality and integrity of health data. Health Inf. Manage. J. Health Inf. Manage. Assoc. Australia 49(1), 3–4 (2020) 6. Alonso, V., Santos, J.V., Pinto, M., Ferreira, J., Lema, I., Lopes, F., Freitas, A.: Health records as the basis of clinical coding: Is the quality adequate? a qualitative study of medical coders’ perceptions. Health Inf. Manage. J. 49(1), 28–37 (2020) 7. Administração Central dos Sistemas de Saúde. Implementação do sistema de codificação clínica ICD-10-CM/PCS em Portugal, em substituição da atual ICD-9-CM. https://www2. acss.min-saude.pt/Portals/0/Circular%20Informativa%20n%C2%BA24_2016_DPS_ACSS. pdf. Accessed 19 Nov 2020 8. Freitas, A., et al.: A framework for the production and analysis of hospital quality indicators. In: Information Technology in Bio- and Medical Informatics, pp. 96–105. Springer, Heidelberg (2011) 9. Sáez, C., Romero, N., Conejero, J.A., García-Gómez, J.M. Potential limitations in COVID-19 machine learning due to data source variability: a case study in the nCov2019 dataset. J. Am. Med. Inf. Assoc. ocaa258 (2020) 10. Freitas, J.A., Silva-Costa, T., Marques, B., Costa-Pereira, A.: Implications of data quality problems within hospital administrative databases. In: XII Mediterranean Conference on Medical and Biological Engineering and Computing 2010, pp. 823–826. Springer, Heidelberg (2010) 11. Sáez, C., Rodrigues, P.P., Gama, J., Robles, M., Garcıa-Gomez, J.M.: Probabilistic change detection and visualization methods for the assessment of temporal stability in biomedical data quality. Data Min. Knowl. Disc. 29(4), 950–975 (2015) 12. Alonso, V., Santos, J.V., Pinto, M., Ferreira, J., Lema, I., Lopes, F., Freitas, A.: Problems and barriers in the transition to ICD-10-CM/PCS: a qualitative study of medical coders’ perceptions. In: World Conference on Information Systems and Technologies, pp. 72–82. Springer, Cham (2019) 13. Lobo, M.F., Oliveira, M., Oliveira, A.R., Santos, J.V., Alonso, V., Lopes, F., Ramalho, A., Souza, J., Viana, J., Caballero, I., Freitas, A.: Protocol for analysis of root causes of problems affecting the quality of the diagnosis related group-based hospital data: a rapid review and Delphi process. In: World Conference on Information Systems and Technologies, pp. 93–103. Springer, Cham (2019) 14. Cheng, P., Gilchrist, A., Robinson, K.M., Paul, L.: The risk and consequences of clinical miscoding due to inadequate medical documentation: a case study of the impact on health services funding. Health Inf. Manage. J. 38(1), 35–46 (2009) 15. Nordgaard, J., Jessen, K., Sæbye, D., Parnas, J.: Variability in clinical diagnoses during the ICD-8 and ICD-10 era. Soc. Psychiatry Psychiatr. Epidemiol. 51(9), 1293–1299 (2016) 16. Clinical Classification Software (CCS). https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ ccs.jsp. Accessed 19 Nov 2020 17. Administração Central do Sistema de Saúde. Grupos e Instituições. https://benchmarkingacss.min-saude.pt/BH_Enquadramento/AbordagemMetodologica. Accessed 19 Nov 2020 18. Spiegelhalter, D.J.: Funnel plots for comparing institutional performance. Stat. Med. 24(8), 1185–1202 (2005) 19. Spiegelhalter, D.: Statistical methods for healthcare regulation: rating, screening and surveillance. J. Roy. Stat. Soc. 175, 1–47 (2012) 20. Sáez, C., Zurriaga, O., Pérez-Panadés, J., Melchor, I., Robles, M., García-Gómez, J.M.: Applying probabilistic temporal and multisite data quality control methods to a public health mortality registry in Spain: a systematic approach to quality control of repositories. J. Am. Med. Inform. Assoc. 23(6), 1085–1095 (2016) 21. Sáez, C., Gutiérrez-Sacristán, A., Kohane, I., García-Gómez, J.M., Avillach, P. EHRtemporalVariability: delineating temporal data-set shifts in electronic health records. GigaScience 9(8), giaa079 (2020)
202
J. Souza et al.
22. Thygesen, K., Searle, J.: Update on the universal definition of acute myocardial infarction in the light of new data. In: Conference Papers in Medicine, pp. 1–5 (2013) 23. US National Center for Health Statistics. ICD-10-CM Official Guidelines for Coding and Reporting FY2020. https://www.cdc.gov/nchs/data/icd/10cmguidelines-FY2020_final. pdf. Accessed 19 Nov 2020 24. Asaria, P., Elliott, P., Douglass, M., Obermeyer, Z., Soljak, M., Majeed, A., Ezzati, M.: Acute myocardial infarction hospital admissions and deaths in England: a national follow-back and follow-forward record-linkage study. Lancet Public Health. 2(4), e191–e201 (2017) 25. Agrupador de GDH All Patient Refined DRG. https://www2.acss.min-saude.pt/Portals/0/ CN22.pdf. Accessed 08 Nov 2020 26. Souza, J., Santos, J.V., Canedo, V.B., Betanzos, A., Alves, D., Freitas, A.: Importance of coding co-morbidities for APR-DRG assignment: focus on cardiovascular and respiratory diseases. Health Inf. Manage. J. 49(1), 47–57 (2019)
Business Decision Making Based on Social Media Analysis C. Santiago Morales1(B) , M. Mario Morales2 , S. Glenda Toala1 B. Alicia Andrade1 , and U. Giovanny Moncayo2
,
1 University of Alicante, Alicante 03080, España {smorales,gtoala,aandrade}@uce.edu.ec 2 Central University of Ecuador, Quito 170129, Ecuador {mmoralesm,mmoncayo}@uce.edu.ec
Abstract. In this article, an analysis of the current potentialities of social media and their impact on the daily life of human beings will be done. Leading to the presentation of software that helps to the generation of new knowledge based on different tools analysis applied to the users. The use of social media their strengths and orientation are analyzed by reviewing the advantages of the main ones used in the world and their focus. In this way, an analysis is presented on a global and local scale, looking for important acceptance indicators for users who remain active in the media. Finding the fields of interest, age groups, and gender according to the network and indistinct parameters that ultimately let us see the impact from the user. Base on different SM search engines on the main trends found we can define a friendly interface that can be implemented in the software model to be proposed. Finally, we put forward a proposal whose main orientation is to analyze personal and business interest groups, performing a benchmarking that has as its main input social media. It allows companies to innovate and improve in a certain line of business by obtaining knowledge of those who do things better at a given time. Organizations will be able to make decisions more efficient, agile, and appropriate to their own business. Keywords: Social media · Benchmarking · Network algorithms · Decision making
1 Introduction Nowadays social media have become a trend of use worldwide, to the point that, according to a study carried out by the companies “We Are Social” and “Hootsuite” at the beginning of January 2019, there were around 4.54 thousand million people who used at least one social network, which constitutes 59% of the world’s population (Social & Hootsuit, 2019). As a consequence of the pandemic generated by Covid-19 in the current year 2020, this indicator should have suffered a significant increase. This validated information will be used and transform into meaningful knowledge in favor of those who receive it and use it (Duarte 2020). © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 203–213, 2021. https://doi.org/10.1007/978-3-030-72651-5_20
204
C. S. Morales et al.
There are various forms to use social media such as maintaining contact with friends or family, finding out about local and world events, sharing preferences or personal photos, exchanging ideas, keeping updated information on current events such as those in the health sector, among others. All this information must be taken from official sources and to use them they must be validated (Ramirez 2019). These behaviors in social media (SM) are a guide and a trend for several companies and for the state that seeks to learn about their potential and current clients. It helps to create a guideline of where their efforts in sales or services should be focused and what will be the best business strategies that must be implemented to effectively reach a certain market niche. If we refer to the political sphere, it can be said that although the interaction that takes place in SM is important, the more relevant information will be the impact that its content has (Silva and López 2014).
2 Theoretical Framework Deciding is a continuous activity of the human being in all aspects of life. Choosing between several options can be a very simple task, but sometimes it is so complex that it becomes a major concern. Decision-making involves numerous cognitive processes, the stimuli present in the task, the memory of previous experiences, and the estimation of the possible consequences of the different options (Martínez-Selva et al. 2006). “Artificial intelligence” (AI) is a science with a wide field of study dedicated to the understanding of the human brain and intelligence, to generate a mathematically logic model and processes that help facilitate and automate problems in different areas of knowledge (D’Addario 2019). “Web scraping” is a technique that serves to obtain information from web sites, which indexes the information of a web page using a robot that focuses on the transformation of unstructured data on the web (Rouse 2020). Finally, “Benchmarking” is a process by which the products, services, or work processes of leading companies are taken as a reference, to compare them with those of your own company, and later make improvements based on these references (Espinosa 2019). Therefore, we must recognize the importance of the data generated by SM for business decision-making by reviewing the impact they have around the world. For which we have carried out a study of the use of social media from a global perspective and the particularity in Ecuador. For the present study and due to the experience in different strata, we will focus on SM such as Facebook, YouTube, Instagram, and TikTok, of which we can emphasize the following: Instagram: It has the largest number of young users, integrating different photo and video options simply and attractively for the user. In August 2016, it integrated the so-called Instagram Stories into its strategies, allowing the user to share short videos or photos that were deleted 24 h after their publication. This has been taken advantage of by several companies that have made their products or services known through this strategy thus creating Instagram Shopping. “In 2019, the platform announced that it had more than one billion monthly active users” (Social & Hootsuit, 2019). Facebook: Considered one of the most important SM today with approximately 2,400 million users, launched in 2004 by Mark Zuckerberg; initially focused on students
Business Decision Making Based on Social Media Analysis
205
and currently with multiple uses for society. According to WeAreSocial and Hootsuite, “44% of the potential reach of Facebook ads are women and 56% are men. 32% of the world’s people over 13 years old access Facebook” (Social & Hootsuit, 2019). One of the central axes is based on the creation of profiles that are capable of generating Fanpages to advertise content, mainly commercial or image, focused on reaching a target audience and improving the perception of the product or person. YouTube: It is defined as a video search engine, where users create a channel where they can upload videos focused on any topic with an indefinite duration. The rest of the users can comment and even follow the person who has uploaded the video, known as YouTubers, through a subscription to their channels. “It is positioned as the most used video platform in most Spanish-speaking countries” (Social & Hootsuit, 2019). Several important companies have held live events, where they present their latest products through this platform. TikTok: This platform allows you to create, edit, and publish video clips of a maximum of 60 s. Its focus is the teenage audience as the content is simple to use but powerful. The company in charge of this platform is ByteDance, hosted in Douyin China since September 2016. By 2018 it already had 75 languages available in more than 150 markets reaching an approximate of 500 million users. In the first quarter of 2019, TikTok exceeded 1 billion downloads and not necessarily just teenagers (Estanga 2019).
3 Growth and Advantages in the Use of Social Media Nowadays, SM is one of the types of communities with the highest growth, thanks to the widespread of information and communication technologies (Latorre 2018). With the generation of Web 2.0, approximately in 2004, there is an explosion where the themes and content cover all areas and where new ways of establishing and maintaining social relationships are generated (Morales 2019). Remember that Web 2.0, or Social Web, refers to all those sites that allow information to be shared, contributing to collaboration between users and collective intelligence. Nowadays it is very easy to find out news and world events with just one click, for which, what today is known as trends are created on social media, which represent a topic that everyone is talking about. All this generates many comments within the SM, depending on their acceptance or rejection, which leads many companies to make important decisions, make modifications, and even eliminate proposals that were not to the popular liking. Concluding that what an organization is looking for is that both they and their products become a trend. Nowadays, the volumes of data that are generated and consumed stand out for their accelerated growth, which implies that the repositories that contain a collection of data are not only considered to perform traditional queries. They are also used as repositories from which you can obtain relevant information that is useful for decision making if you apply the data analytics methodologies existing today (Morales 2019). In Table 1, you can see the evolution that has occurred on the advancement of the web in its different generations, highlighting its main characteristics (a + g 2019).
206
C. S. Morales et al. Table 1. The Web’s history. (Champagnat 2019)
Generation/start Description Web 1.0/1990
Information that could only be accessed but not interacted, one-way
Web 2.0/2004
It allows us to share information, the appearance of blogs, forums, and social media
Web 3.0/2010
Semantic Web as the use of language in the network. Ex: searches using keys
Web 4.0/2016
Offers smarter behavior and predictive algorithms
3.1 Global Analysis As of July 2019, the world population was around 7.713 million people (Naciones Unidas 2019), of which around 4.3 billion worldwide now use the Internet, which represents more than half of the world’s population (Social & Hootsuit 2019).As seen in Fig. 1, from the internet users 80% have at least one social network, and six platforms claim to have more than one billion monthly active users.
Fig. 1. Digital data around the world. (Social & Hootsuit 2019)
Important in this study and that companies should take into account as shown in Fig. 2, is that the average active Internet users worldwide spend more than 6½ h online each day. Out of this 6½ h, 2 h, and 16 min are dedicated exclusively to social media. However, this data varies depending on the culture to which we refer, for example, in the Philippines, the average use of social media per user is 4 h while in Japan it is only 36 min a day (Social & Hootsuit, 2019). This ranking is based on the monthly growth rate of users worldwide where the data presented in front of each social network is in millions of people. Also, bars with a lighter mark are properly recognized as social media, and those that have a darker mark are considered as exclusive platforms for instant messaging. We can undoubtedly observe that Facebook is the SM with most users worldwide, followed by YouTube and Instagram.
Business Decision Making Based on Social Media Analysis
207
Fig. 2. Active users worldwide. (Social & Hootsuit 2019)
3.2 New Airs on Social Media With the absence of Twitter within the main SM and the appearance of networks such as TikTok and QZone should be considered for those companies that wish to make themselves known in front of potentbnm* bxial clients. However, Facebook is still the most recommended platform to show the world new ventures and this should take an important turn in the post-pandemic era. GlobalWebIndex is a market research SaaS company founded by Tom Smith in 2009 that provides audience insights to publishers, media agencies, and marketers around the world. In their studies, they have shown that more than three-quarters of Internet users in the world buy online every month, showing considerable rise in the use of mobile devices to make these types of purchases rather than desktops and laptops (Social & Hootsuit 2019). According to data from Statista, around 2.82 billion people around the world bought consumer goods such as fashion, groceries, or household appliances online, marking a 3% growth compared to 2017 and generating more than $ 1.75 billion per year. 3.3 Ages in Social Media In Fig. 3, there are data based on the average audience of Facebook, Instagram, and Messenger. The largest number of users by gender is from 18 to 35 years old, which should be the audience in which each company must focus on to carry out for marketing in social media. 3.4 Analysis in Ecuador Doing the same analysis within Ecuador and until 2019 Facebook is the platform for companies to make their products and services known by excellence. There is a quarterly growth of advertising, which is expected to increase in the next years allowing Ecuadorian
208
C. S. Morales et al.
Fig. 3. Age range of majority audience in social media. (Social & Hootsuit, 2019)
companies to make their service known. According to Social & Hootsuit stadistics 52% of facebook users are male and 48% represent female. Instagram, has shown continuous growth and the main content are images and videos. We can see the growth in advertising of 2.6% that reflects that Instagram has gradually become a platform that has gained the trust of several Ecuadorian companies to make themselves known. In this platform females users represent the mayority with 55% and male 45%. Twitter, for the common user in Ecuador, Twitter is not a platform classified as “preferred”, however, it is a good social media to publicize ideas focused on politics, government order, sports, and entertainment. The growth of the advertising level within Twitter has gone down with a −6.3%, which means that the confidence of companies to advertise on this network has decreased. Finally male population represent the mayority of users with 70% and female representing the 30% (Social & Hootsuit 2019). 3.5 2companies that Bet on the Use of Social Media The following information is based on the “Susty Social Media Ranking” made by the InContext company (Mejía-Llano 2020). This company is dedicated to supporting companies worldwide with solid strategies and convincing communications. The ranking reveals 100 leading companies that have maintained sustainable growth in social networks while keeping their corporate reputation and the constant impact of their brands. From the information found in this study these are some of the main ideas: • • • •
Within the ranking, there are 66 companies from the United States and 34 in Europe. No company of Latin American has been taken into account. Twitter is the predominant platform for sustainability companies Facebook and Instagram are more suitable for communication and consumer services, while Twitter and LinkedIn are more suitable for communication experts, however, there is no clear trend. • Companies with a higher budget can improve their audience due to the facilities of each platform that, through various sentiment analysis algorithms, can target users with specific profiles very precisely although this service is not free.
Business Decision Making Based on Social Media Analysis
209
According to this study, three success metrics were taken into account: a) Frequency of publications related to sustainability, which is key to building an audience and its future growth. b) The number of influencers, or influential followers, which represents an indication that your posts are credible and relevant. c) Amount of participation generated (retweets, likes, shares, responses) that symbolizes the general commitment to the company. Table 2 shows the use made by the most socially active companies of the different existing platforms by March 2019, taking into account the three previous metrics: Table 2. Business use od different social networks Twitter Linkedin Instagram Facebook Adobe
X
X
X
AT&T
X
X
Autodesk
X
X
Cisco
X
CVS
X
Dell
X
General Electric
X
HP
X
IBM
X
Intel
X
X
X
Johnson & Johnson X
X
X
Marks & Spencer
X
Mars, Inc
X
Microsoft
X
Nestlé
X
X
X
X
X
X
X X
X
SAP
X
X
Schneider Electric
X
X
In Table 3, the ranking made by the InContext company is displayed, which results in the following: There are several interesting facts within this information which are: • Cisco undoubtedly occupies the first place, due not only to the marketing it carries out of its products and the quality they have but also for its desire to share knowledge to everyone by generating many conferences that make its brand know at the same time.
210
C. S. Morales et al. Table 3. Companies with the Best Results in SM Compañía
Ranking País
Cisco
1
Microsoft
EEUU
2
EEUU
Johnson & Johnson 3
EEUU
Autodesk
4
EEUU
IBM
5
EEUU
SAP
6
Alemania
HP
7
EEUU
Nestlé
8
Suiza
Marks & Spencer
9
Inglaterra
Schneider Electric
10
Francia
• Six of the top seven positions are held by digital technology companies. • Johnson & Johnson is the highest-ranked consumer brand followed by Marks & Spencer in ninth place. • Overall companies like Intel do not appear in the top 10 despite having the most influencers in terms of digital technology companies. • Only one food company enters this ranking and it is Nestlé in position eight. In Table 3, it is represented which are the largest companies in terms of the use of social media and how they manage them. Therefore, there is a guide to make a solid proposal that helps companies in making decisions while maintaining business sustainability thanks to social media.
4 Proposal 4.1 Everything is on Social Media An analysis that uses a series of methodologies and theories can determine the link strength that people have with one or more social media. Their “profile” represents the personality of each user, their social connections, preferences, among others. It is intended to present a Big Data service that optimizes the user’s time by providing them with immediate information from social media according to their interests and needs. A search algorithm that provides information to the user about companies that could have the same needs and interests (a + g, 2019). What we are looking for is a friendly and dynamic interface that can categorize the different needs of the user and present several options that could meet their demand. By analyzing all possible social media or at least those that have the most interaction in posts people are searching for.
Business Decision Making Based on Social Media Analysis
211
a) Interface: After analyzing different interfaces, as shown in Table 4, with the greatest functionality we could present a similar to the one provided by Netflix. Since it offers several alternatives according to the different categorizations available. With the generated interface the user would like to stay in the proposed software. It will maintain the versatility of current platforms to facilitate the solution to their needs. In addition to the possibilities of interacting in several SM at the same time and receiving suggestions from new posts made based on what has been searched lately. Table 4. Visual Interfaces Application
Subscribers
Monthly cost
Interface acceptance
Basic characteristics
Netflix
182 million
7,99
9/10
Slight changes from one device to another Agile search
HBO Go
140 million
12,99
7/10
On Android, it is a problem Problems resuming videos
Amazon Prime Video
150 million
12,99
7/10
Complex interface Looks more like the store interface
b) Sentiment analysis and APIs: Within all this, several questions arise: – How can companies know their opinions about themselves? – How does this method help in decision making? The companies that could use software with these characteristics as a basis for decision-making would come across a sentiment analysis. Sentiment analysis is the process of defining the emotional tone behind a series of words. It is used to try to understand the attitudes, opinions, and emotions expressed in an online mention (Bannister 2019). In social networks, this type of analysis is important because it allows us to get an idea of public opinion on certain issues. For companies, the change of sentiment within social media corresponds to changes in the stock market value worldwide. Some companies already make use of this method using algorithms based on machine learning to predict if a phrase found in certain social media about them is positive or negative. Even the same social media provide a series of APIs, (set of functions and/or services on the network that can be used by any other software), which facilitate users to acquire certain information about the posts that are made on these platforms to be analyzed in the manner previously described.
212
C. S. Morales et al.
c) Architecture and programming: The following will be used in the proposal: • The microservices architecture, which aims to build an application as a set of small services on social media and for which the use of APIs and ApiRest Engine Analytics for Facebook, Youtube, Instagram, and further. • Programming languages such as Python that have exclusive libraries for downloading publications made on social networks, for example: – Tweepy, with which the tweets of different users are found. – Facebook-py obtains information from the publications made. – Instagram-API-Python, locate the text of posts made on Instagram. • For java language, there are specialized frameworks for connecting to the previously described APIs and the ones we will consider are: – Twitter4j offers facilities to generate the connection between the user and the endpoints provided by Twitter. – Facebook4j, a powerful framework for requests to the official Facebook API. – JInstagram, Framework that has access to Open Graph API (previous Instagram API). With the proposal already launched what is wanted is that by determining the approval or disapproval of a certain idea published on social media. For decision-making levels in companies, those results can be brought up to an internal business debate to make timely and precise decisions.
5 Discussion As a result of the growth of information and communication technologies in situations like the one we currently live in due to the pandemic, it has caused many people to research in social media in a way of distraction, this coupled with the ubiquity of mobile devices, creates new opportunities for companies to easily reach their users. As it can be seen in this article, there is a large number of people who interact daily on social media and who become a potential target audience for large companies. The proposed methodology offers an alternative for an adequate treatment based on the large amount of information generated every second on internet and it will contribute to the goals set by Ecuadorian and Latin American companies looking forward to satisfy the needs of their costumers.
6 Conclusions The proposed software LEARN FROM THE GREATEST (A + G) will be a platform that helps to improve the performance of companies by analyzing the behavior of potential clients towards a brand, product, or service. Sentiment analysis has proven to be a powerful decision-making tool.
Business Decision Making Based on Social Media Analysis
213
Social media (SM) are very important platforms for various types of companies to improve the marketing of their products and services to reach their customers in a much more dynamic and approachable way. Based on the experience obtained from other companies, it is possible to determine that the most successful companies in this field are those that invest in social media technology. A strategy for companies to learn about the experience of others is precisely the proposal presented by A + G due to the ease use, dynamic interface, and possibility to perform a complete analysis of different (SM) at the same time.
References a+g. (2019). aprende de los grandes Bannister, K.: Brandwatch (2019) Champagnat, U.M.: Issuu (2019) D’Addario, M.: Inteligencia Artificial: Tratados. Aplicaciones, Usos y Futuro (2019) Duarte, F.: BBC. Obtenido de (2020). https://www.bbc.com/mundo/noticias-49634612 Espinosa, R.: Benchmarking (27 de 03 de 2019) Estanga, G.: Blog de Marketing (17 de 05 de 2019) Latorre, D.M.: umch. Obtenido de (03 de 2018). https://umch.edu.pe/arch/hnomarino/74_His toria%20de%20la%20Web.pdf Martínez-Selva, J., Sánchez-Navarro, J., Bechara, A., Román, F.: Amazona WS (2006) MediaSustySocial. (s.f.). Greenbiz. Obtenido de https://www.greenbiz.com/article/here-are-com panies-use-social-media-best-sustainability-marketing Mejía-Llano, J.C.: Marketing Digital, Social Media y Transformación Digital (26 de 02 de 2020). Obtenido de https://www.juancmejia.com/marketing-digital/estadisticas-de-redes-sociales-usu arios-de-facebook-instagram-linkedin-twitter-whatsapp-y-otros-infografia/ Morales, S.: Metodología para procesos de inteligencia de negocios con mejoras en la extracción y trasnformación de fuentes de datos, orientado a la toma de decisiones. Naciones Unidas (2019) Ramirez, J.: Lifeder (2019). Obtenido de https://www.lifeder.com/para-que-sirve-internet/ Rouse, M.: TeachTarget (04 de 2020) Silva, T., López, M.: Dialnet (12 de 03 de 2014) Social, W.A., Hootsuit. DataReportal - Global Digital Insights (2020)
The Effects of Abrupt Changing Data in CART Inference Models Miriam Esteve(B) , Nuria Mollá-Campello, Jesús J. Rodríguez-Sala, and Alejandro Rabasa Center of Operations Research (CIO), Miguel Hernandez University of Elche (UMH), 03202 Elche (Alicante), Spain {miriam.estevec,jesuja.rodriguez}@umh.es, [email protected], [email protected]
Abstract. The continuous input of data into an Information System makes it difficult to generate a reliable model when this stream changes unpredictably. This continuous and unexpected change of data, known as concept drift, is faced by different strategies depending on its type. Several contributions are focused on the adaptations of traditional Machine Learning techniques to solve these data streams problems. The decision tree is one of the most used Machine Learning techniques due to its high interpretability. This article aims to study the impact an abrupt concept change has on the accuracy of the original CART proposed by Breiman, and justify the necessity of detection and/or adaptation methodologies that update or rebuild the model when a concept drift occurs. To do that, some simulated experiences have been carried out to study several training and testing conditions in a changing data environment. According to the results, models that are rebuilt in the right moment after a concept drift occurs obtain high accuracy rates while those that are not rebuilt or are rebuilt after a change occurs, obtain considerably lower accuracies. Keywords: Concept drift · CART · Regression rules · Data streaming
1 Introduction Nowadays, the data embedded in the Information Systems from the majority of the companies and organizations is gigantic and changes very quickly, so its analysis is complicated to follow. Continuous data streams arise naturally in companies, for example, those that need to continuously gather detailed information on the use of different parts of a network or the weather and analyze it to detect interesting trends. These changing huge data environments make traditional machine learning algorithms face new problems regarding evaluation methods or accuracy drop. The study of continuous data streams instead of the classical batch approach has introduced a whole new analysis process for data mining. In traditional data mining techniques computational resources, such as memory, time or hard disk space, are a major obstacle to memory-driven techniques [1]. In real life a new class can appear without previous warning, making traditional models © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 214–223, 2021. https://doi.org/10.1007/978-3-030-72651-5_21
The Effects of Abrupt Changing Data in CART Inference Models
215
unable to detect it and adapt to this change [2–4]. A data stream is a potentially infinite sequence of elements and is used to represent data elements that are available over time. There are different models of data flows that adopt different approaches depending on their mutability and structure. Their analysis consists of producing new results as new input data becomes available [5]. This continuous and unpredictable data change is mostly known as concept drift and the prevention of its consequences is searched through forgetting mechanisms. Regarding the concept drift characterization, several studies have defined the nature of the different changes and proposed strategies to detect and approach this changing environment [6–8]. In this new scenario, there are new methods for supervised classification presented both as trees [9] and as rules [10], unsupervised clustering [11], association rules [12] or other common tasks in data mining as outlier treatment [13]. Concretely, decision trees are one of the most used methods for data mining. These models are non-parametric, distribution-free and robust to the presence of outliers and irrelevant attributes. Also, the tree structure of their models has higher interpretability. It is for this reason that this technique has several adaptations to data stream context [14–16]. These previous works on the decision trees adaptation, specifically with the Classification and Regression Trees (CART) algorithm [17], start from the premise of this need with no formal proof that the static CART is deficient in streaming contexts. For this reason, this work aims to measure the impact that those changing environments have on the performance of the CART algorithm in order to justify the need for a stream adaptation. The paper is organized as follows. Section 2 is devoted to briefly introducing the standard CART techniques and streaming adaptations in CART technique. In Sect. 3, the influential factors for adaptive algorithm performance in the typical concept drift scenarios are explained. Conduction of this influence is investigated via experimental computations in Sect. 4. Finally, Sect. 5 concludes.
2 Related Work CART [17] is a machine learning technique with two main objectives, depending on the nature of the response variable. CART works like a nonparametric classification model when the response variable is categorical and as a nonparametric regression when it is numerical. The behavior of CART is simple: a criterion is chosen recursively generating a binary partition of the data until an ending rule is achieved. The result of this process is a tree structure. The tree begins at the root node, develops intermediate nodes and ends at the terminal nodes, which are those that comply with the ending rule, called leaves node. The split procedure consists of trying all possible combinations of the data which minimizes the sum of the mean squared error (MSE) of the two child nodes. Specifically, D , y ), given a data sample S composed of n elements sm = (xm , ym ) = [x1m , . . . , xm m m = 1, . . . , n, where xm ∈ R is the value of the attribute j = 1, . . . , D, for data element sm and ym ∈ R is a response variable of data element sm , CART has as an objective to predict the response variable ym through the predictors x1 , . . . , xm . To do this, CART partitioned into subsets where in case of left son, denoted by tL , satisfies xm < xD m and in . Finally, CART selects the case of right son, denoted by tR , its subset satisfies xm ≥ xD m
216
M. Esteve et al.
2 2 j the best xm value by minimizing 1n xm ∈tL ym − ytL 1n xm ∈tR ym − ytR , where ytL is the average prediction y in the subset tL node and ytR is the average prediction y in the subset of tR node. CART has the advantages of inherently performing multiclass regression, its interpretability is very simple due to its tree structure and the nonlinear relationship among nodes does not affect the performance of the decision tree [18–21]. However, CART may have unstable decision trees because an insignificant modification in the sample, such as eliminating several observations or changes in the data, may cause changes in decision trees: increase or decrease of tree complexity or changes in splitting variables and values [18, 22, 23]. So, CART technique is designed for static datasets and it is not possible to store an entire data stream or to scan it due to its volume [24–30]. Moreover, in the data stream case, the data elements income to the system continuously may occur the concept drift over time [31–33, 37] and an effective learner should be able to monitor such changes and to adapt quickly to them [7]. Several contributions to solve data streams problems are focused on incremental learning, i.e. continuously review the incoming data and adapt the model to it. The problem is that to guarantee the model trained in incremental mode is equal to the model trained in batch mode, the learning time is costlier [16, 30]. Many algorithms have been proposed to solve these problems, such as algorithms based on ensemble methods [33–35], support vector machines [36] and decision trees [9, 25, 38–41]. Specifically, among the algorithms that have adapted the decision trees we can find Hoeffding’s tree algorithm, also known as Very Fast Decision Tree (VFDT) [41] or its adaptive versions [9, 42], where the Hoeffding’s bound [43] is used to determine the least number of examples needed in a node to select a division attribute and another works; or in [44] where the McDiarmid’s bound is used to check whether the probabilistic model is adequate to the descriptions of tree algorithms. Another incremental algorithm is the namely Ultra Fast Forest Tree system (UFFT) by [45], where applying the idea of VFDT a forest of trees is built from data streams. The dsCART is introduced in [46], where this approach is based on all the above algorithms to make it applicable to both numerical and categorical data and does not require pre-pruning.
3 Influential Factors for Adaptive Algorithm Performance Over time there are different patterns of change [8]. These changes could arise at any point of the dataset with no warning, so detecting and handling it is a challenge. These changes over time are called concept drift. A concept drift could emerge at any time in any attribute of the dataset, as well as in the objective variable. The most extended patterns of concept drift are represented in Fig. 1. In the case that these changes occur in the class or objective variable, some of them could be associated with classification problems as the abrupt drift, the gradual change or the reoccurring concepts. In the same scenario, another pattern, as the incremental change, could be associated with regression problems when working with continuous variables. Machine learning is often used for these types of problems where data is collected over an extended period of time. As the data tends to change over time and all the training data is not available at once [47, 48], these models should be updated or retrained
The Effects of Abrupt Changing Data in CART Inference Models
217
Fig. 1. Changing concept pattern over time [8]
accordingly to increase the performance of their predictions. Thus, the information of the old data needs to be adapted to fit to the new data. In machine learning, the changes in data are often handled by windows size on the training data [49]. Furthermore, when these models are built and face a change that affects the quality of its predictions, normally an action is required. Since the data in a real continuous environment tend to be infinite but memory is not, the adaptive algorithms need to establish a forgetting method to define which examples are relevant to test or modify the model. The most extended forgetting method is the sliding window which can be used to detect change, to obtain updated statistics from the most recent examples, and to decide which data is used to rebuild or revise a model after a concept drift occurs [50]. In this study we used this forgetting method in its simplest version, the fixed-size sliding window to evaluate the quality of the model and rebuild it when necessary. Normally, the change degree influences the window size. For one hand, a small window size reflects accurately the current data distribution, which is better for changing environments. For other hand, a large window size has more examples to build a model with, which is better for stable periods.
4 Experimental Results Experiments are carried out in a computer with Itanium-2 dual-core. The code was written in C. Some experiments were carried out to tune certain parameters of the algorithm. In all of them, CART performance is tested in multiple changing situations with several scenarios defined by different training and testing boundaries. These changing datasets are simulated using SEA Concepts Generator, proposed by [40], in the form of abrupt changes. This simulator provides datasets with three independent real attributes (x1, x2, x3) and values between 0 and 10 being only the first two attributes relevant for the prediction. The class decision boundary is defined as x1 + x2 ≤ θ, where θ is a threshold value different for each concept. The original data model can produce four different concepts: (C1) θ = 9, (C2) θ = 8, (C3) θ = 7 and (C4) θ = 9.5. Each scenario evaluates 100 simulated datasets generated from different seeds. The average accuracies obtained by CART algorithm in each scenario are compiled in Table 1.
218
M. Esteve et al.
Fig. 2. Scenarios to be simulated and executed.
All the datasets generated to test the CART algorithm have a total of 10k examples, three descriptive attributes and one binary class to predict. The concept for study variables changes every 2,5k examples, dividing the sample in 4 concepts regarding its θ. First concept (C1) contains the examples from 0 to 2,5k, second concept (C2) from 2,5k to 5k, third concept (C3) from 5k to 7,5k and fourth concept (C4) from 7,5k to 10k. All designed experiments are shown in Fig. 2. Table 1 reports the average of the accuracy from CART predictions for the scenarios considered. The table is divided in two blocks to improve its visualization. Therefore, the first four columns correspond to the last four. In this way, the first column indicates the scenario simulated. The second column shows the training sample used, denoted by Tr., the change of concept used, denoted by C1, C2, C3 or C4 according to the θ value, and the interval of observations that have been used as a training sample. The third column indicates the same as the previous one, but this time the test sample, denoted by Ts, has been used. The last column shows the accuracy achieved for each proposed scenario. Regarding the results, the first scenario accuracy proves that CART algorithm can generate high precision models by rebuilding and testing inside a stable concept. Comparing this first with the second scenario, the accuracies drop from 98% (Ts 1.3) and 97% (Ts 1.4) to 92% (Ts 2.4) and 79% (Ts 2.5). In the third scenario, as it can be seen in Table 1, the accuracies for same-concept models are considerably better than the changed concepts with differences from more than 5 point (Ts 3.3 vs. Ts 3.4) to almost 19 points (Ts 3.5 vs. Ts 3.6) depending on the concept difference (θ). The rest of scenarios (from 4 to 8) do not rebuild the model, so their accuracies are directly linked to the differences between training and testing concepts. As it can be seen in Table 1, the models tested
The Effects of Abrupt Changing Data in CART Inference Models
219
Table 1. Results for scenarios. Scn
Tr./Concept (obs)
Ts./Concept (obs)
Acc.
Scn
Tr./Concept (obs)
1
Tr1.1 / C1 [0, 2000)
Ts1.1 / C1 [2000, 2500)
97,75
4
Tr4.1 / C1 + C2 Ts4.1 - C2 [0, 4500) [4500, 5000)
94,97
Tr1.2 / C2 Ts1.2 / C2 [2500, 4500) [4500, 5000)
97,97
Ts4.2 - C3 [5000, 7500)
88,12
Tr1.3 / C3 Ts1.3 / C3 [5000, 7000) [7000, 7500)
98,24
Ts4.3 - C4 [7500, 10000)
91,15
Tr1.4 / C4 Ts1.4 / C4 [7500, 9500) [9500, 10000)
97,72
Tr2.1 / C1 [0, 2000)
Ts2.1 / C1 [2000, 2500) Ts2.2 / C2 [2500, 3000)
2
Acc.
Tr5.1 / C1 + C2 Ts5.1 - C2 [2000, 4500) [4500, 5000)
97,07
97,75
Ts5.2 - C3 [5000, 7500)
90,80
91,08
Ts5.3 - C4 [7500, 10000)
88,32
Ts2.3 / C2 + 94,42 C3 [2000, 3000)
3
Ts./Concept (obs)
5
6
Tr6.1 / C2 [2500, 5000)
Ts6.1 - C3 + 89,68 C4 [5000, 10000)
Tr2.2 / C2 Ts2.4 / C3 [3000, 5000) [5000, 5500)
92,23
Ts6.2 - C3 [5000, 7500)
92,40
Tr2.3 / C3 Ts2.5 / C4 [5500, 7500) [7500, 8000)
79,48
Ts6.3 - C4 [7500, 10000)
86,96
Tr3.1 / C1 [0, 2000)
Ts3.1 / C1 [2000, 2500)
97,75
Ts3.2 / C2 [2500, 3000)
91,08
Ts7.2 - C3 [7000, 7500)
95,65
Tr3.2 / C2 Ts3.3 / C2 [2500, 4500) [4500, 5000)
97,97
Ts7.3 - C4 [7500, 10000)
83,31
Ts3.4 / C3 [5000, 5500)
92,20
Ts8.1 - C4 [9500, 10000)
89,37
Tr3.3 / C3 Ts3.5 / C3 [5000, 7000) [7000, 7500)
98,24
7
8
Tr7.1 / C2 + C3 Ts7.1 - C3 + 89,36 [2500, 7000) C4 [7000, 8000)
Tr8.1 C1 + C2 + C3 + C4 [0, 9500)
(continued)
220
M. Esteve et al. Table 1. (continued)
Scn
Tr./Concept (obs)
Ts./Concept (obs)
Acc.
Ts3.6 / C4 [7500, 8000)
79,36
Scn
Tr./Concept (obs)
Ts./Concept (obs)
Acc.
with the same concept (Ts 4.1, Ts 5.1, Ts 7.2) get great accuracy rates and, specially, those that build the model with more same-concept observations (Tr 5.1, Ts 5.1).
5 Conclusion and Future Work In the experiments carried out in this article, it has been shown that changes in concept have a significant effect on the accuracy achieved. This accuracy descends in different degrees when the test set contains observations from another concept. This dropping degree is directly influenced by the selected threshold, being the greatest difference between C3 (θ = 7) and C4 (θ = 9.5). The obtained results show that those scenarios in which the model is rebuilt after a concept drift, the performer gets high accuracy rates, while those that do not rebuild or it does rebuild in a changing frontier get considerably lower accuracy. These findings justify the necessity of detection and/or adaptation methodologies that rebuild or update the model when a concept drift occurs. In this case, the simulated conditions are known but in an actual environment the data changing process is completely uncertain. A real-time abrupt changing context (as it could be UK or EU finances facing Brexit or social crisis during COVID-19 pandemic) may considerably impact the inference capacity of CART and its performance, so an adaptive version might be necessary. Finally, we finish by mentioning several lines that pose interest for future research. The first one is the possibility of extending the study presented in this paper in another concept drift. Another line of research, we could evaluate how implementing detection and adaptation methods affect the inference capacity of CART algorithm. In addition, we could extend this word to pure sorting methods.
The Effects of Abrupt Changing Data in CART Inference Models
221
Acknowledgments. The authors are grateful for the financial support from Spanish Ministry of Science, Innovation and Universities under Grant FPU17/05365 and Grant DIN2018-010101. This work was also supported by the Spanish Ministry of Science and Innovation and the State Research Agency under grant PID2019-105952GB-I00/AEI/https://doi.org/10.13039/501100011033 and Miguel Hernández University under grant 197T/20.
References 1. Marascu, A., Masseglia, F.: Mining sequential patterns from data streams: a centroid approach. J. Intell. Inf. Syst. 27(3), 291–307 (2006) 2. Govindaraju, N.K., Raghuvanshi, N., Manocha, D.: Fast and approximate stream mining of quantiles and frequencies using graphics processors. In: Proceedings of the International Conference on Management of Data, pp. 611–622 (2005) 3. Masud, M., Gao, J., Khan, L., Han, J., Thuraisingham, B.: Classification and novel class detection in concept drifting data stream sunder time constraints. IEEE Trans. Knowl. Data Eng. 23(6), 859–874 (2011) 4. Elwell, R., Polikar, R.: Incremental learning of concept drift in nonstationary environments. IEEE Trans. Neural Netw. 22(10), 1517–1531 (2011) 5. Zhou, A., Cao, F., Qian, W., Jin, C.: Tracking clusters in evolving data streams over sliding windows. Knowl. Inf. Syst. 15(2), 181–214 (2008) 6. Widmer, G., Kubat, M.: Learning in the presence of concept drift and hidden contexts. Mach. Learn. 23(1), 69–101 (1996) 7. Gama, J., Rodrigues, P.: An overview on mining data streams. In: Studies in Computational Intelligence, vol. 206. Springer, Berlin (2009). https://doi.org/10.1007/978-3-642-01091-0_2 8. Gama, J., Zliobaite, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM Comput. Surv. 16, 37 (2013). https://doi.org/10.1145/2523813 9. Hulten, G., Spencer, L., Domingos, P.: Mining time-changing data streams. In: Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2001, pp. 97–106 (2001) 10. Kosina, P., Gama, J.: Very fast decision rules for classification in data streams. Data Min. Knowl. Disc. 29(1), 168–202 (2013). https://doi.org/10.1007/s10618-013-0340-z 11. Aggarwal, C.C., Han, J., Wang, J., Yu, P.S.: A framework for clustering evolving data streams. In: Proceedings of the 29th International Conference on Very Large Data Bases, VLDB Endowment, vol. 29, pp. 81–92 (2003) 12. Jiang, N., Gruenwald, L.: Research issues in data stream association rule mining. ACM SIGMOD Rec. 35(1), 14–19 (2006) 13. Subramaniam, S., Palpanas, T., Papadopoulos, D., Kalogeraki, V., Gunopulos, D.: Online outlier detection in sensor data using non-parametric models. In: Proceedings of the 32nd International Conference on Very Large Data Bases, VLDB Endowment, pp. 187–198 (2006) 14. Utgo, P.: ID5: an incremental ID3. In: 5th International Conference on Machine Learning, pp. 107–120. Morgan Kaufmann Publishers (1988) 15. Utgo, P., Berkman, N.C., Clouse, J.A.: Decision tree induction based on efficient tree restructuring. Mach. Learn. 29(1), 5–44 (1997) 16. Wang, H., Fan, W., Yu, P., Han. J.: Mining concept-drifting data streams using ensemble classifiers. In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 735–740. ACM Press (2003) 17. Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. CRC Press (1984)
222
M. Esteve et al.
18. Singh, S., Gupta, P.: Comparative study ID3, CART and C4.5 decision tree algorithm a survey. Int. J. Adv. Inf. Sci. Technol. (IJAIST) 27(27), 97–103 (2014) 19. Wang, Y., Witten, I.H.: Induction of Model Trees for Predicting Continuous Classes (1996) 20. Fan, G., Gray, J.B.: Regression tree analysis using TARGET. J. Comput. Graph. Stat. 14(1), 206–218 (2005) 21. Shao, Y., Lunetta, R.S.: Comparison of support vector machine, neural network, and CART algorithms for the land-cover classification using limited training data points. ISPRS J. Photogramm. Remote. Sens. 70, 78–87 (2012) 22. Bhavsar, H., Ganatra, A.: A comparative study of training algorithms for supervised machine learning. Int. J. Soft Comput. Eng. (IJSCE) 2(4), 2231–2307 (2012) 23. Lewis, R.J.: An introduction to classification and regression tree (CART) analysis. In: Annual Meeting of the Society for Academic Emergency Medicine in San Francisco, California, vol. 14 (2000) 24. Aggarwal, C.: Data Streams, Models and Algorithms, Springer, LLC., New York (2007) 25. Bifet, A., Kirkby, R.: Data Stream Mining a Practical Approach, Technical report, University of Waikato (2009) 26. Fan, W., Huang, Y., Wang, H. and Yu, P.: Active mining of data streams. In: Proceedings of the SDM (2004) 27. Gaber, M., Zaslavsky, A., Krishnaswamy, S.: Mining data streams: a review. Sigmod Rec. 34, 18–26 (2005) 28. Gao, J., Fan, W., Hang, J.: On appropriate assumptions to mine data streams: analysis and practice. In: 2007 IEEE International Conference on Data Mining, ICDM 2007, Omaha, NE (2007) 29. Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Elsevier (2006) 30. He, H., Chen, S., Li, K., Xu, X.: Incremental learning from stream data. IEEE Trans. Neural Netw. 22, 1901–1914 (2011) 31. Franke, C.: Adaptivity in Data Stream Mining, Ph.D. thesis, University of California, Davis (2009) 32. Tsymbal, A.: The problem of concept drift: definitions and related work, Technical Report TCD-CS-2004-15, Computer Science Department, Trinity College Dublin, Ireland (2004) 33. Kuncheva, L.I.: Classifier ensembles for changing environments. In: Multiple Classifier Systems, p. 15. Springer, Heidelberg (2004) 34. Kolter, J.Z., Maloof, M.A.: Dynamic weighted majority: An ensemble method for drifting concepts. J. Mach. Learn. Res. 8, 2755–2790 (2007) 35. Street, W.N., Kim, Y.: A streaming ensemble algorithm (SEA) for large-scale classification. In Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 377–382 (2001) 36. Ertekin, S., Bottou, L., Giles, C.L.: Nonconvex online support vector machines. IEEE Trans. Pattern Anal. Machine Intell. 33, 368–381 (2011) 37. Klinkenberg, R.: Learning drifting concepts: Example selection vs. example weighting. Intell. Data Anal. 8(3), 281–300 (2004) 38. Gama, J., Medas, P., Rodrigues, P.: Learning decision trees from dynamic data streams. In: Proceedings of the 20th Annual ACM Symposium on Applied Computing, pp. 573–577 (2005) 39. Gama, J., Fernandes, R., Rocha, R.: Decision trees for mining data streams. Intell. Data Anal. 10(1), 23–45 (2006) 40. Natwichai, J., Li, X.: Knowledge maintenance on data streams with concept drifting. In: International Conference on Computational and Information Science, pp. 705–710. Springer, Heidelberg (2004)
The Effects of Abrupt Changing Data in CART Inference Models
223
41. Domingos, P., Hulten, G.: Mining high-speed data streams. In: Proceedeings of the 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 71–80 (2000) 42. Liu, J., Li, X., Hong, W.: Ambiguous decision trees for mining concept-drifting data streams. Patt. Recogn. Lett. 30(15), 1347–1355 (2009) 43. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58, 13–30 (1963) 44. Rutkowski, L., Pietruczuk, L., Duda, P., Jaworski, M.: Decision trees for mining data streams based on the McDiarmid’s bound. IEEE Trans. Knowl. Data Eng. 25(6), 1272–1279 (2012) 45. Gama, J., Medas, P., Rocha, R.: Forest trees for on-line data. In: Proceedings of the 2004 ACM Symposium on Applied Computing, pp. 632–636 (2004) 46. Rutkowski, L., Jaworski, M., Pietruczuk, L., Duda, P.: The CART decision tree for mining data streams. Inf. Sci. 266, 1–5 (2014) 47. Fisher, D., Schlimmer, C.L.: Models of incremental concept learning: a coupled research proposal. Technical report CS-88-05, Vanderbilt University (1988) 48. Giraud-Carrier, C.L.: A note on the utility of incremental learning. AI Commun. 13(4), 215– 223 (2000) 49. Widmer, G., Kubat, M.: Learning in the presence of concept drift and hidden contexts. Mach. Learn. 23, 69–101 (1996) 50. Bifet, A., Gavaldà, R.: Learning from time-changing data with adaptive windowing. In: Proceedings of the 7th SIAM International Conference on Data Mining, vol. 7 (2007). https:// doi.org/10.1137/1.9781611972771.42.
Log Data Preparation for Predicting Critical Errors Occurrences Myriam Lopez1(B) , Marie Beurton-Aimar1(B) , Gayo Diallo1,2(B) , and Sofian Maabout1(B) 1
2
University of Bordeaux, LaBRI, UMR 5800, Talence, France {myriam.lopez,marie.beurton,sofian.maabout}@labri.fr, [email protected] BPH INSERM 1219, University of Bordeaux, 33000 Bordeaux, France
Abstract. Failure anticipation is one of the key industrial research objectives with the advent of Industry 4.0. This paper presents an approach to predict high importance errors using log data emitted by machine tools. It uses the concept of bag to summarize events provided by remote machines, available within log files. The idea of bag is inspired by the Multiple Instance Learning paradigm. However, our proposal follows a different strategy to label bags, that we wanted as simple as possible. Three main setting parameters are defined to build the training set allowing the model to fine-tune the trade-off between early warning, historic informativeness and forecast accuracy. The effectiveness of the approach is demonstrated using a real industrial application where critical errors can be predicted up to seven days in advance thanks to a classification model. Keywords: Predictive Maintenance (PdM) · Machine Learning (ML) · Classification · Log data · Data preparation
1
Introduction
Predictive maintenance is of paramount importance for manufacturers [6,10]. Predicting failures allows anticipating machine maintenance and thus enabling financial saving thanks to shortening the off-period of a machine and anticipating the spare parts orders. Nowadays, most modern machines in the manufacture domain are equipped with sensors which measure various physical properties such as oil pressure or coolant temperature. Any signal from these sensors is subsequently processed later in order to identify over time the indicators of abnormal operation [12]. Once cleaned, historical data from sensors can be extremely valuable for prediction. Indeed, their richness makes it possible to visualize the state of the system over time in the form of a remaining useful life estimation [5]. In parallel, machines regularly deliver logs recording different events that enable tracking the conditions of use and abnormalities. Hence, event-based prediction is a key research topic in predictive maintenance [4,13]. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 224–233, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_22
Log Data Preparation for Predicting Critical Errors Occurrences
225
Our contribution in this paper is a data preparation approach whose output is used to learn a model for predicting errors/failures occurrences. It relies on the exploitation of historical log data associated to machines. The purpose is to predict sufficiently early the occurrence of a critical malfunction in the purpose of maintenance operations facilitation. The proposed framework is applied on a real industrial context and the empirical obtained performances show its effectiveness. After introducing in the next section key definitions and parameters, we detail the data preparation process. Afterwards, we report on some of the experiments conducted in the context of our case study. Thereafter, we provide an overview of related work. Finally, we conclude and outline some indications for future work.
2
Dataset Collection and Preliminary Definitions
We suppose that we are given a log file F emitted by a machine M where the errors that have been observed by sensors associated to M are reported. We assume that the errors are logged in regular time intervals. Moreover, we consider a set of errors E = L ∪ H where L is the set of low errors j and H is the set of critical errors hj (high errors that will be called target in the following). Our aim is to predict high errors wrt other errors. Each record in F is a pair i, E where i is a timestamp expressed in days and E ∈ N|E| is a vector where E[j] represents the number of occurrences of error ej ∈ E at timestamp i. Hence, F is a temporal ordered sequence of such records. The following is the running example we use throughout the paper as an illustration of the approach. Example 1. Let the errors set be E = {1 , 2 , 3 , 4 } ∪ {h1 , h2 } and consider the following sequence (Table 1): Table 1. Example of a log event dataset with timestamp Timestamp 1 2 3 4 h1 h2 1
0
12 6
1
0
0
2
0
0
3
2
0
0
3
0
1
4
1
1
1
4
1
0
1
2
0
0
5
0
1
1
2
0
0
6
0
1
1
1
0
0
7
0
1
1
0
0
1
8
1
0
1
8
0
1
9
0
0
6
1
1
0
10
1
0
7
1
1
0
226
M. Lopez et al.
For example, the first row/record whose timestamp is 1 says, among others, that at day 1, there have been 12 occurrences of low error 2 and no occurrence of high error h1 . To come up with a prediction model and collect the data, three parameters, depicted in Fig. 1, are applied: – Predictive Interval: it describes the history of data used to perform predictions and, its size in days is defined by the parameter P I. The information contained in that interval is gathered in a structure referred as a bag. – Responsive Interval: since, from a practical point of view, making a prediction for the next day is of little interest, a common strategy is to constrain the model to forecast sufficiently early. The Responsive Interval is located immediately after the Predictive Interval and its size which controls how much early we want the model to forecast, is defined by the parameter RI. – Error Interval: intuitively, this interval, whose size is defined by parameter EI, allows to make error occurrence prediction wrt a time interval instead of a time unit. It therefore introduces a degree of uncertainty of the prediction’s temporality.
Fig. 1. The key parameters of model prediction
Example 2. Let P I = 3, RI = 4 and EI = 2. With this setting, we want to predict high errors wrt bags of size 3, and this prediction concerns a period whose duration is 2 days starting 4 days after the end of the observed bag. More precisely, the fact that EI = 2 means that a predicted error occurrence may happen either at the first, the second, or both days within the error interval. Consider the bag B1 defined between timestamps i = 1 and i = 3 in the Table 1. Then the prediction period (Error Interval) concerned by B1 is in timestamp [8, 9]. That is, at i = 3, we observe the bag B1 and we should be able to say whether, or not, a high error will occur between i = 8 and i = 9.
3
Proposed Methodology
The current section describes how training and test sets are prepared. Indeed, in order to train a binary predictive model, a prerequisite is to build a training set where each sample is labeled YES/NO and corresponds to a situation where, respectively, a high error occurred or not. Definition 1. Let Bi be a bag and given a target high error hj . Bi is labeled YES iff time interval [i + P I + RI; i + P I + RI + EI − 1] contains an occurrence of hj . Bi is labeled NO otherwise.
Log Data Preparation for Predicting Critical Errors Occurrences
227
Example 3. Let P I = 3, RI = 2 and EI = 2. Consider again the running example and let us focus on error h1 . To decide the label of bag B1 , we need to check whether in [6; 7] error h1 has been observed. Since target has not been observed, then B1 takes label NO. Hence, referring to Table 1 and applying the definition above, the bags B1 and B2 are labeled as NO and the bags B3 and B4 are labeled as YES. Observe that since their corresponding EI period is outside the bounds of the available history, bags 5, 6, 7 and 8 cannot be labeled. 3.1
Summarization Strategy
The reader may notice that the above configuration (labeled bags) is close to the one encountered in the Multiple Instance Learning (MIL) [3]. However, our setting cannot be in conformance with MIL because the objective is a prediction based on the whole bag rather than individual samples. Thus, we choose an alternative approach, by summarizing every bag Bi by maximizing values and transform it into a single features vector1 . Example 4. Let P I = 3 and consider the bag B1 starting at i = 1 and ending at i = 3. The bag is summarized by the vector 0, 12, 6, 2. That is, for each j we take the maximum of its values in B1 . So, with RI = 2 and EI = 2 and when considering the target error h1 we obtain the data set presented in Fig. 2. Timestamp 1 2 3 4 h1 1 0 12 6 1 0 0 1 0 0 6 0 1 1 1 0 7 0 1 1 0 8 1 0 1 8 9 0 0 6 1 10 1 0 7 1 1
h2 0 0 1 0 0 0 1 1 0 0
Bag 1 2 3 4 h2 Label h1 B1 0 12 6 2 1 NO B4 1 1 1 2 0
YES
Fig. 2. Bag summarization strategy by MAX() function
Each sample in the right pane of Fig. 2 on one hand, is the total number of occurrences of every j during a period of size P I = 3. The last columns testify whether h1 has occurred (YES) or not (NO) after this period at a precise time interval. The table on the left pane reminds the original data set.
1
Other methods of summarization can be used but in our use case, the MAX() function allowed us to obtain good prediction performances.
228
3.2
M. Lopez et al.
Overall Data Preparation Process
Figure 3 and Algorithm 1 depict our preprocessing stage. From a log file T , the consecutive records are gathered into bags using a sliding window of size P I. The label of each bag Bi depends on the content of interval EIi . The starting point of EIi is RI time units after the end point of Bi . Once the bags are identified and labeled, they are summarized by MAX() function to obtain the input data set T for model learning. As it may be seen from Algorithm 1, the complexity of data preparation is linear with the size of the sequence: the outer most loop is executed O(|T |) times. At each iteration, P I data rows are summarized and the label is assigned after checking EI row records. Hence, the overall complexity is O(|T |×(P I +EI)). From a computing point of view, the outer loop can be easily made parallel because the different iterations are independent of each others.
Algorithm 1: DataPrep Input: Sequence T , EI, P I, RI, Error h Output: Table T of labeled meta-instances. begin for i = P I + RI + 1 to |T | − EI + 1 do s ← Summarize(T , i − RI − P I, i − RI − 1, S) //A bag of size P I starting at index i − RI − P I is summarized using MAX() or another function S s.label ← F alse for j = i to i + EI − 1 do if T [j] contains an occurrence of h then s.label ← T rue Insert s into T return T
Fig. 3. Data Preparation Process. The bags of the training set are built by sliding windows with the following model parameters: P I = 3; RI = 4; EI = 2
Log Data Preparation for Predicting Critical Errors Occurrences
4
229
Experiments
4.1
Dataset Description
We collected raw data from a one-year history of several log files. Each file is associated to a specific machine. All machines belong to the same model2 . However, they are not subject to the same conditions of use. Given a combination of parameters (P I; RI; EI), each log file is processed using Algorithm 1. Output files are then merged to constitute our input dataset. The latter is partitioned into a 80/20 ratio of training and test sets respectively. The partitioning relies on a stratified sampling to keep a similar proportion of positive samples both in the test and training sets. The dataset contains 193 features (i.e., distinct low errors) and 26 high errors (our targets). For every high error, we designed several models, each of which is obtained by combination of three parameter values (P I; RI; EI). As part of our study, we were interested in the 11 most frequent target errors which occur in the dataset. Depending on the target, we ended up with an average of 1000 samples, among which, from 4% to 30% are positive. Although a balance correction technique is in general recommended in such a case, our experiments showed that subsampling did not provide any substantial accurracy gain3 . Then, we decided not to make use of subsampling. In the following experiments the default value for both parameters P I and RI is 7 days. The value of the RI parameter is chosen as constraint because the idea is that the model anticipates 7 days before the error target occurs. 4.2
Settings
All experiments were conducted with the RapidMiner software4 . Several classifier algorithms were explored, e.g., SVM, Naive Bayes, Random Forest, Decision Tree and the multi-layer feed-forward artificial neural network (NN) algorithm of the H2o framework [1]. NN algorithm outperformed the others and was subsequentely chosen as the actual predictive model. Experiments were set with 4 layers of 100 neurons each, 1000 epochs and a Cross Entropy loss function for the learning phase. To assess the performance of a prediction model, different metrics can be used. Among these, Accuracy and F1-score are suitable one. Accuracy is the ratio of correct predictions. F1-score wrt to the positive class is given by the harmonic mean of precision and recall5 : 2(precision × recall) 2 × TP = (precision + recall) 2 × TP + FP + FN 2 3 4 5
(1)
Regretfully, for confidentiality reasons we cannot share the data used for this study. Due to lack of space the results will not be detailed in this paper. Rapidminer.com. P P precision= T PT+F and recall= T PT+F . P N
230
M. Lopez et al.
T P , T N , F P , F N represent respectively, true positives, true negatives, false positives and false negatives. Because negative cases are much more frequent than positive one, accuracy alone is not sufficient to assess the performance of the model. F1-score allows to observe in more details the behavior of the model regarding the class of interest, in our case, the positive one. 4.3
Impact of the Error Interval Choice
Parameter EI introduces a flexibility wrt positive predictions as well as a rigidity for negative ones. Indeed, a positive prediction says that at some point in a future time interval, a high error may occur while a negative prediction says, no occurrence at any point in a future time interval. Intuitively and regarding positive predictions, one may expect that the higher EI, the more correct the model’s predictions are. To analyse this hypothesis we varied EI from 1 to 4 days and for each target error, we computed F1-score wrt the positive class6 The obtained results are depicted in Fig. 4.
Fig. 4. F1-score evolution wrt EI
Overall, our initial assumption is confirmed. We observe that for about half of the errors, setting EI to 1 day leads to a significant degradation of the model’s performance. In these cases the model yields to F1 score equal to zero (see errors E3 ; E4 ; E5 ; E10 and E11) explained by a null value of true positive prediction. For E8 and E9, the results show a different phenomenon from overall behaviour. Indeed, the largest F1 score for these errors is obtained when EI is set to 1. F1-score gives a global picture of the predictive model behavior wrt positive class in that it combines precision and recall measures. From the application perspective, false alarms, i.e., false positives ratio (F P R) is a key indicator7 . 6 7
Due to space limitation, we do not present here the prediction for negative class. P F P R = F PF+T = 1 − precision. P
Log Data Preparation for Predicting Critical Errors Occurrences
231
Fig. 5. False positive (false alarm) ratio wrt EI
Figure 5 shows the evolution of F P R with respect to EI. We observe that in general, increasing EI tends to decrease F P R, hence it increases the precision. Increasing the size of the EI concomitantly increases the number of positives examples in the training dataset. That can explain that we obtained better performances with a greater EI. Although these performances may also in some cases mask an over-expression of false positives, the phenomenom is not present here, as proven in Fig. 5. Despite the general trend of F1-score and precision wrt EI, the model performance is not monotonic. For example, the learned model for error E1 has the highest F1-score when EI = 2. This indicates that for every error, one needs to calibrate the most convenient EI regarding applications/F1-score needs.
5
Related Work
Fault anticipation for predictive maintenance has attracted a large body of research (please see e.g., recent surveys [8,9,14]) and several paradigms have been used for this purpose. Our solution belongs to the class of approaches based on supervised machine learning. As such, we focus our comparison to these techniques and more specifically on [11] and [7] that are, to our knowledge, the closest previous works. Let us start with [11]. Despite some subtle differences, the task of tagging the bags as negative or positive instances is similar to that of [11]. However, in the latter, negative bags propagate their label to each individual instance belonging to them. By contrast, every positive bag is aggregated using the average measure and gives rise to a new positive meta-instance. Therefore, the training data set is a mix of meta-instances (positive in this case) and simple instances (negative ones). The rational behind the use of the average score is that the so obtained instance is considered as a representative of a daily behavior inside a bag. In fact, and unless having a small standard deviation inside positive bags, a daily
232
M. Lopez et al.
record can be very different from the average of the bag it belongs to. Hence learning from bag averages and predicting from single instances may present some discrepancy. The data for our case study present high deviations inside the bags, which makes it hopeless to identify positive records using averaged records. In our case, the data preparation process considers positive and negative bags equally. In that way, consider the occurrence or not of a high error/failure is explained by the whole bag associated to it, and not just some instances belonging to that bag. Moreover, in contrast to what has been suggested in the work of [11], this way of proceeding reduces the issue of unbalanced classes that we are confronted with. Now let us consider the prediction phase. In [11], prediction is instance based (i.e., daily records) and these predictions are then propagated to bags. More precisely, during the prediction phase, successive instances belonging to the unclassified bag, are provided to the learned model. If one of them is classified as positive then the whole bag is positive. It is negative otherwise8 . So fundamentally, predictions are daily based in the sense that the real explanation of a possible error/failure occurrence depends actually on what happens during a day while ours needs are to combine events of larger intervals. This is also the reason why, unlike [11], our work is not in conformance with Multi-Instance Learning paradigm (MIL). Indeed, the MIL main hypothesis states that a bag is positive iff one of its instances is positive [2,3], while our approach does not impose any condition on individual instances. We have applied the methodology of [11] over our use case, but could not obtain F1-score better than 0.2. A more recent work with a similar data preparation process as [11] is [7]. The main difference is that its learned model is a regression that estimates the probability of a future error occurrence. If this probability is higher than some fixed threshold, then an alert is fired. Hence, during data preparation, the labels associated to bags are probabilities instead of YES/NO labels. Intuitively, the closer a bag is to an error, the higher is its probability. To handle this property, the authors used a sigmoid function. The reported results show that the proposed techniques outperform those of [11] by a large margin. We believe that this is mainly due to the data nature defining the use case and in general there is no absolute winner. As it has been shown in the experiments, our targeted use case does not need such complex techniques to achieve high prediction accuracy.
6
Conclusion and Future Works
We described an approach to exploit log data for predicting critical errors that may cause costly failures in the industry manufacture domain. The approach is based on aggregation of temporal intervals that precede these errors. Combined with a neural network, our solution turns to be highly accurate. Its main advantage, compared to other techniques, is its simplicity. Even if we do not claim that 8
Notice that in [11], during data preparation, bags are first labeled and then, their labels are propagated to instances while for bag label prediction, the labels of its instances are first predicted and then the bag label is deduced.
Log Data Preparation for Predicting Critical Errors Occurrences
233
it should work for every similar setting (log based prediction), we argue that it could be considered as a baseline before trying more sophisticated options. We applied our solution to a real industrial use case where a dozen of highly critical errors were to be predicted. The obtained accuracy and F1-score belong to respectively [0.8; 0.95], and [0.6; 0.9] which can be considered as highly effective compared to the reported results on log based failures prediction litterature. So far, we designed one model per target error. In the future, we plan to analyze in more depth the errors to see whether it would be possible to combine different predictions in order to reduce the number of models. In addition, we are interested in automating the parameters setting: given a (set of) target(s), to find the optimal values of P I, EI and RI such that the learned model performances is maximized.
References 1. Candel, A., Parmar, V., LeDell, E., Arora, A.: Deep learning with H2O (2016), h2O AI Inc 2. Carbonneau, M., Cheplygina, V., Granger, E., Gagnon, G.: Multiple instance learning: a survey of problem characteristics and applications. Pattern Recognit. 77, 329–353 (2018) 3. Dietterich, T.G., Lathrop, R.H., Lozano-P´erez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89(1–2), 31–71 (1997). https:// doi.org/10.1016/S0004-3702(96)00034-3 4. Gmati, F.E., Chakhar, S., Chaari, W.L., Xu, M.: A Taxonomy of event prediction methods. In: Proceedings of IEA/AIE Conference. Springer (2019) 5. Guo, L., Li, N., Jia, F., Lei, Y., Lin, J.: A recurrent neural network based health indicator for remaining useful life prediction of bearings. Neurocomputing 240, 98–109 (2017) 6. Hashemian, H.M.: State-of-the-art predictive maintenance techniques. IEEE Trans. Instrum. Meas. 60(1), 226–236 (2011) 7. Korvesis, P., Besseau, S., Vazirgiannis, M.: Predictive maintenance in aviation: failure prediction from post-flight reports. In: Proceedings of ICDE Conference (2018) 8. Krupitzer, C., Wagenhals, T., Z¨ ufle, M., et al.: A survey on predictive maintenance for industry 4.0. CoRR abs/2002.08224 (2020) 9. Ran, Y., Zhou, X., Lin, P., Wen, Y., Deng, R.: A survey of predictive maintenance: Systems, purposes and approaches. CoRR abs/1912.07383 (2019) 10. Salfner, F., Lenk, M., Malek, M.: A survey of online failure prediction methods. ACM Comput. Surv. 42(3), 10:1–10:42 (2010) 11. Sipos, R., Fradkin, D., M¨ orchen, F., Wang, Z.: Log-based predictive maintenance. In: SIGKDD Conference, pp. 1867–1876. ACM (2014) 12. Wang, C., Vo, H.T., Ni, P.: An IOT application for fault diagnosis and prediction. In: IEEE International Conference on Data Science and Data Intensive Systems, pp. 726–731 (2015) 13. Wang, J., Li, C., Han, S., Sarkar, S., Zhou, X.: Predictive maintenance based on event-log analysis: a case study. IBM J. Res. Dev. 61(1), 11 (2017) 14. Zhang, W., Yang, D., Wang, H.: Data-driven methods for predictive maintenance of industrial equipment: a survey. IEEE Syst. J. 13(3), 2213–2227 (2019)
Running Workshops to Identify Wastes in a Product Development Sample Shop Gabriela R. Witeck1(B) , Anabela C. Alves1 , Joana P. de Almeida2 , Ana J. Santos1 , Ana L. Lima1,3 , and Ricardo J. Machado1,3 1 Centro ALGORITMI, University of Minho, Guimarães, Portugal [email protected], [email protected], [email protected], [email protected] 2 Bosch Car Multimedia Portugal, S.A, Braga, Portugal 3 CCG/ZGDV Institute, Guimarães, Portugal
Abstract. Lean Thinkingaligned with Information technology can promote competitiveness for modern enterprises, while improve the performance of technological activities and eliminate no value-added tasks to the design process of product. The paper is a contextualization of the current wastes evidenced during the research work carried out in a company that supplies electronic components to automotive industry. Workshops were developed to map the wastes in the sample production processes, as all interested parties are seeing, discussing and learning together in a freely-open environment. The workshops promoted the identification of 60 types of waste, and 26 roots cause. In order to solve scientifically the root causes problems, the production procedures will be readjusted by adopting production guidelines according to the principles of Lean, and some digitization measures will also be considered. The evidence from the workshops demonstrates that the main types of waste reported in the present literature can also be observed in complex projects. Keywords: Lean Thinking · Information technology · Product development · Workshops
1 Introduction The key to enhance the core competitiveness of modern enterprises is to make full use of the advantages brought by information technology, industrial technology, modern science and management technology [1]. To improve the performance of technological activities, one must is to eliminate the activities without value-added to the design process of the product, or service. These activities are considered waste, once customers are not willing to pay for [2]. As the peculiar characteristics of the automotive industry, the product development (PD) process is responsible for influencing the competitiveness between companies. The application of Lean Thinking (LT) principles [3] to PDis labeled as Lean Product Development (LPD) [4, 5]. LPD is the practice of “creating value through a process © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 234–243, 2021. https://doi.org/10.1007/978-3-030-72651-5_23
Running Workshops to Identify Wastes in a Product Development Sample Shop
235
that builds on knowledge and learning enabled by an integrated product development system, consisting of people, processes, and technology” [6]. The application of “lean” to the services also started in Toyota Production System [4], as Lean Production [7], in this case in its Toyota Product Development System. Since them, Lean Thinking principles are applied in all services or administrative areas of goods companies, as the name of Lean Office, or even services companies such as hospitals, under the name of Lean Healthcare, or Higher Education Institutions, among others [8]. Additionally, since 2004the International Council for Systems Engineering (INCOSE) has been advocating this application in systems engineering and complex product development to reduce the wastes, delays, cost overruns and frustrations from this process. At the same time, value and stakeholders satisfaction are improved. Nevertheless, in many companies this process continues presenting wastes as any other type of service inside the companies. This was the case in the Sample Shop department of the company presented in this paper. The company supplies electronic components to the automotive industry. The Sample Shop is the name of the company department responsible for the new product development. Thus, this paper presents part of an ongoing project to reduce the product development lead-time by Lean Thinking philosophy and information technology. In a previous phase, it was necessary to diagnose the current process through mapping the whole process, from the order to the produced samples, involving all departments. The main purpose of this paperis to present the wastes and root causes problems evidenced through workshops ran to map the current process. This project is being developed in a context of a partnership between a university and the company focuses the new manufacturing processes, communication technologies and information and decision support systems for the management of factory operations. For this, a multidisciplinary team with members from the company and from the university was formed to develop the project. The paper is structured in five sections. After a brief introduction presenting the study, section two presents a literature review about Lean Thinking and Lean Product Development. Third section clarifies the research methodology and the fourth section describes the wastes mapping. Finally, the last section draws the conclusions.
2 Literature Review Introducing early new products to the automotive market has several strategic and operational advantages. To sustain competitiveness the world-class manufacturers must develop competencies to innovate, design, and reduce the wastes in the process. This brief literature review presents some of the Lean Thinking principles and Lean Product Development concept. 2.1 Lean Thinking Principles Toyota Production System (TPS) consists in socio-technical management practices developed by Toyota Company. This system was developed by the engineer TaiichiOhno that applied original concepts such as Jidoka from the mentor of Toyota Company, Sakiichi Toyoda and others improvements of his son, Kiichiro Toyoda. According to Ohno
236
G. R. Witeck et al.
[2] the main TPS goals were: Remove overloads (muri), search for flexible processes; Remove inconsistencies (mura), search for smooth processes; Eliminate waste (muda). Detailing the third concept, there are seven types of waste (mudas). According to TPS, eliminating waste consists in eliminate tasks/elements with no added-value from the client point of view. These were originally identified for a manufacturing environment [2] as: Excess motion – people or equipment walking or moving more than necessary to perform the process; Excess transportation – the movement of materials not required to perform the processing; on utilized potential (Personnel) – not using the full abilities of people/employees; Inventory – all parts, work-in-progress and finished goods not being processed; Defects – all work associated with identifying and correcting defects; Over production – production ahead of demand; Waiting – for the previous process step to deliver; Over processing – doing things that add no value for the customer. Waste was interpreted [3] as “any human activity that absorbs resources but creates no value”. Based on the fundamentals of TPS, Lean Thinking philosophy can be described as a set of principles aimed at reducing waste, adding value and maximizing the results of an organization through a systematic process of continuous improvement [3, 9]. This means that Lean goes beyond a set of tools and methods. Lean is based on five core principles to be followed when the intention is to implement Lean, based on the premise that organizations are made up of processes. Such principles were defined [3] as: Specify value from the standpoint of the customer; Identify the value stream for each product and eliminate those steps that do not create value; Take the product flow smoothly and continuously down the value stream; Let customers pull value between all steps; Aim for a state of perfection by reducing the number of steps and the amount of time needed to serve the customer, i.e., search continuous improvement. The Lean philosophy is a consistency of knowledge whose essence is the ability to reduce costs and increase productivity by eliminating waste through problem solving, in a systematically way. This implies rethinking how to lead, manage and develop people [10]. Lean practices must promote, in addition to greater customer satisfaction, an ideal environment for learning: without guilt, less errors, availability of information, opportunities, growth and respect for workers [11]. Lean manufacturing emphasizes heavily on the empowerment of employees. The employees are responsible for actually working and creating products and services, hence they should be given adequate flexibility and importance of acknowledging their ideas and suggestions. Incorrect allocation of employees to different tasks, improper performance evaluation and training and monotonous work are significant contributors for poor morale in work environment [12]. In many cases, workers also find it difficult to portray their suggestions and feedback in current workplaces. This is actually building a personal agreement with all the principles of Lean Thinking philosophy. It means going deeper into a learning cycle, experiencing improvements, reflecting and internalizing insights, finally resulting in lean improvement lessons [9]. The challenges factories face to implement lean due to lack of resources such as proper communication, monitoring, integration among others are analyzed according to these dimensions.
Running Workshops to Identify Wastes in a Product Development Sample Shop
237
Under a roof of Industry 4.0, production workers provide immediate feedback of production conditions via real time data through their own smart phones and tablets. The key workers are equipped with a smart handheld device, which is integrated with the company’s network [13]. Therefore, the concepts extended on product development not only help the front end users who collect the consumer needs, brainstorm, and develop concepts but also provide input to the back end where transition from design to production occurs [14]. 2.2 Lean Product Development Companies that successfully implement lean manufacturing programs also generally launch LPD programs, despite acknowledging the complexities related to their implementation [4]. In fact, the application of lean principles to PD offers, at least, three important advantages to companies: 1) It improves PD performance by reducing time to market and development costs [4]; 2) It further boosts the efficiency in lean manufacturing [4]; 3) It is a step forward towards the lean enterprise, which means exploiting the increased productivity deriving from coupling LPD with lean manufacturing [3]. The moderating effect of LPD on the relationship between LM practices and operational performance was studied [15]. The findings concern that LPD practices positively moderate the effect of LM practices on quality performance improvement. The research suggests that lean implementation will have a greater effect on operational performance improvement if the implementation spans across both manufacturing and product development areas [15]. 2.3 Industry 4.0 in the Sample Shop Industry 4.0 is revolutionizing the way organizations operate and consequently their systems. One of the most affected areas is manufacturing industry that is increasingly competitive and demanding, customers no longer require only quantity, but also quality, flexibility and individualization of their orders. To keep up with the market companies have started to implement new technologies. The Cloud platform increases mobility and information availability to a company, since it is device and location independent [16]. With this the organization can easily use mobile apps with more adaptable interfaces. In the future manufacturing execution systems (MES) may only consist of different “apps” made specifically for each equipment or department. The same combination of mobile devices with the increased reliability of positioning systems (e.g. RFID) will allow the representation of real time 3D maps and, maybe in a near future, augmented reality. These can bring considerable advantages in areas like identification and localization of materials or containers, or in maintenance related activities [17]. Through Internet of Things, Cloud manufacturing, cyber-physical production systems and other cutting-edge technologies that emerged with the Industry 4.0 like Agentbased techniques, and Big Data it is possible to obtain intelligent manufacturing shop floor where interconnection, interoperation, and interaction is achieved [18]. Intelligent shop floors enable the creation of a network that connects machines, processes and workers, and utilizes data, collected in real-time from different organization activities,
238
G. R. Witeck et al.
in analytics applications. The results of these applications can provide a system feedback loop that can interconnect with an automated system (e.g. controlling the machine directly, or be displayed to manufacturing operators through a mobile application) [19]. Concluding, the manufacturing systems are evolving to interoperability throughout the whole organization. Ideally, every equipment and worker are connected to each other, the central system is able to know the position and task of every one of them, the machines can operate by themselves and share information without needing external intervention. Moreover, stakeholders may be able to check the shop-floor and production state with a simple click in their mobile device.
3 Research Methodology The methodology used for this ongoing project is an action research, which engaged company’s collaborators and the papers’ authors in a teamwork multidisciplinary project. This methodology consists in five phases: (1) diagnosing; (2) action planning; (3) action taking; (4) evaluating and (5) specifying learning [20]. Following the action-research methodology, to define the problem is necessary to diagnose the system and processes. The data for this first phase was gathered through workshops, which can introduce a new concept, encourage participants to investigate it further on their own. Moreover, it is a way of introducing new practices [21]. This paper discusses in detail how this phase was processed and tools used to it. Main tool used was to run workshops among all stakeholders involved in the Sample Shop. The following phases of action-research are to be developed in future work. The advantages of using action-research methodology are related with its focus on change and implementation of solutions to several issues that are experienced by employees.
4 Running the Workshops The attention in this work was for the Samples Shop in a supplier of electronic components of automotive industry. The company matrix, located in Germany, covers all prototype building activities starting with the first idea of a conceptual sample in mechanical and electrical design until the project is transferred to the production plant studied. The main goal of the Sample Shop department is to produce and access the Sample according to the time schedule and product engineering process, as a contribution for launching of new products in the plant, in time with quality. The Sample Shop unit chart is divided in four areas: Sample Coordinators, Manufacturing Mechanical Sample Production, Manufacturing Electrical Sample Production and Sample Planning. Such Lean methodology shows as a comprehensive strategy that involves various approaches to eliminate waste from Sample Shop activities. Through the workshops was identifying the wastes in the whole process, with a focus on new Sample projects management. Also, was applied the quality tool Ishikawa Diagram [22], for the analysis and evaluation of problems in different activities throughout the Sample Shop production.
Running Workshops to Identify Wastes in a Product Development Sample Shop
239
4.1 Workshop Preparation and Results The workshops were conducted in order to access the necessary information and carry out a correct analysis of the current situation of Sample Shop Process production. The group on focus was induced to evidence casual problems in their assignment that could generate wastes over the process, as well as to develop and critically evaluate new interventions. To engage the participants in the wastes mapping, it was used a blackboard as shown in Fig. 1, and the Excel file. Systematically nominees the wastes, root cause of problems and prevent them from reoccurring. The workshop allowed a learning experience while providing solutions to address important local problems.
Fig. 1. Interactions during the workshops
Four rounds of standardized workshops were developed, as shown in Table 1, focused on industrialization projects, more specifically on the instrumental cluster business unit, and facilitated by two moderators. A Sample, of five or six, employees for each area in the Sample Shop department it was conveniently nominated, in order to its attendees to compose an audience that is actively engaged in workshop activities. However, as the groups are directly related to the department tasks and their relevant experience, they could be generalized to the wider population. It was created an Excel file with seven columns, as presented in Fig. 2, each one with classification attribute header. The first column named “Sample Production” represents the Sample Building stage: Planning, Insertion, Intermediate Processes and Final assembly. In the column “Wastes” must be inserted the type of waste as was based on literature presented above [2]. The third column was named “Root Cause Classification” and the researchers should introduce the root cause classification based on literature (Ishikawa Diagram) [21]. The root cause was classified in the fourth column. The fifth column named “Root Cause Description” presents a brief description regarding the root cause. The “Negative impact” represented in the sixth column recognize the issues and impacts caused by the problem. The “Measures to solve” column (seventh column) evaluate and decide available alternatives, also generate decisions in the absence of viable alternatives. Following this discussion, the moderators asked the groups which were the high impacts of root causes, in regards to the main problem: impaired Sample delivery. Due to this main problem the process target will not be achieved; it means Samples will not be
240
G. R. Witeck et al. Table 1. Workshops groups. Sample building
Number of participants Workshop duration (min) Number of wastes identified
Electrical
Mechanical
Sample planning
5
6
6
5
120
90
120
110
25
11
13
11
Fig. 2. Print of Excel file for wastes classifications.
delivered on time, on quantity, on specification, and according to the agreed conditions with the client. In total, as shown in Table 2, 60 root causes of sample delivery impaired were identified. Following the concepts of eliminating wastes and creating value must be a constant concern; the main wastes were identified as: inventory, motion, non-utilized talent, defects, extra-processing, but mostly waiting Table 2. Workshops groups results. Process wastes
Sample building
Electrical
Mechanical
Sample planning
Total
Waiting
15
10
6
8
39
1
3
7
Extra-processing
3
Defects
1
Personnel
4
1
3
5
3
7
Motion
1
1
Inventory
1
1
Total number of wastes identified
25
11
13
11
60
4.2 Root Causes for the Sample Delivery Impaired The data collected during the workshops, was illustrated in the Ishikawa diagram of Fig. 3. The benefit of this diagram is that shows preliminary structuring with higher transparency in a certain problem field. The clusters’ nominees were: material, methods, machines and personal. Thus, it is a first and essential step for the actual problem analysis. This standard approach leads to quickly identifying relevant root causes. In special cases, of course, these nominators have to be adjusted to the particular problem.
Running Workshops to Identify Wastes in a Product Development Sample Shop
241
Fig. 3. Workshop Results in Ishikawa Diagram for the impaired sample delivery
The diagram exposes, in an organized manner, the cause-effect chain, which connects the different root-causes to the problem under study. It is a systematic approach to analyze the relations and interdependencies among a big number of factors. Based on a matrix, the relation of each element with each other is assessed systematically. The 60 root causes of problems obtained in Table 2 were clustered in wastes categories. Then, it was possible to compact them into 26 root causes as showed in Fig. 3. 4.3 Lean and Technological Solutions In order to solve the 26 root causes problems, it will be necessary readjust the production procedures with adoption of production guidelines according to the principles of Lean, some digitization measures for the project were considered. Two different platforms are being developed nominated as I-Power and Saminfo, below follows description and details. I-Power: Project management processes platform, which will systematize the specific procedures of each project management, identify, analyze and manage all costs involved in industrialization, and promote communication management in project teams by developing communication skills. The Platform will create the conditions to develop the software solution that will support the project management work in a Lean way. Saminfo: Mobile applicative of the IT management system that seeks to integrate the current tools computer support (SAMOS and SAP).This application allows managing the priorities and resources of the production process. Also, the users will be notified when any tasks is pending, reducing the lead time of the prototype production and consequently speeding up deliveries to customers.
242
G. R. Witeck et al.
5 Conclusions The first step of an ongoing project was presented in this paper. Process mapping through workshops allowed gathering data that will be the base for the next steps of the project. These also allowed discussing properly the wastes identified through a workshop. These strategies were important to think about some solutions for sample shop production to follow good strategy for eliminating wastes. It will help to save time along the process, promoting higher effectiveness, even in the process automation, if this will become necessary. Workshops are an interesting way to map the processes as all interested parties are seeing, discussing and learning together in a freely-open environment. This allows realizing about the misunderstandings that each person has about what the other do or not do. At the same time, the activity aligned with Lean Thinking principles can be leveraged to address specific problems in these environments, for example poor project schedule adherence and significant extra-costs. The evidence from the workshops demonstrates that the main types of waste reported in the present literature can be observed also in complex projects. Also, Lean is a very vast area to study and implement. In a second stage, another workshop will be held with all participants at the same time, to carry out the information process design, and thus carry out the triangulation methodology in order to confirm the data - wastes and root causes - previously collected. As a future work, a value stream mapping (VSM) will be developed to calculate the actual lead-time. It is expected also to design a future state map by eliminating non value-adding activities. By using this method potential for improvements can be captured, which become clear when the whole process production is considered. Acknowledgments. This work is supported by: European Structural and Investment Funds in the FEDER component, through the Operational Competitiveness and Internationalization Programme (COMPETE 2020) [Project nº 039479; Funding Reference: POCI-01-0247-FEDER039479].
References 1. Gao, Q., Shi, R., Wang, G.: Construction of intelligent manufacturing workshop based on lean management. Procedia CIRP 56, 599–603 (2016) 2. Ohno, T.: Toyota Production System: Beyond Large-Scale Production. CRC Press (1988) 3. Womack, J., Jones, D.: Lean Thinking: Banish Waste and Create Wealth in Your Corporation. Free Press, New York (2003) 4. Liker, J.K., Morgan, J.M.: The Toyota way in services: the case of lean product development. Acad. Manag. Perspect. 20(2), 5–20 (2006) 5. Mascitelli, R.: The Lean Product Development Guidebook. Technology Perspectives (2007) 6. Rossi, M., Morgan, J., Shook, J.: Lean product development. In: Netland, T.H., Powell, D. (eds.) The Routledge Companion to Lean Management. Routledge, New York (2017) 7. Womack, J., Jones, D., Roos, D.: The Machine that Changed the World. Rawson Associates, New York (1990) 8. Alves, A.C., Flumerfelt, S., Kahlen, F.-J. (eds.): Lean Education: An Overview of Current Issues. Springer, Cham (2017)
Running Workshops to Identify Wastes in a Product Development Sample Shop
243
9. Singh, M., Rathi, R.: A structured review of Lean Six Sigma in various industrial sectors. Int. J. Lean Six Sigma 10, 622–664 (2019) 10. Witeck, G.R., Alves, A.C.: Developing lean competencies through serious games. In: International Symposium on Project Approaches in Engineering Education (PAEE) & Active Learning in Engineering Education Workshop, vol. 10, pp. 179–186 (2020) 11. Sremcev, N., Lazarevic, M., Krainovic, B., Mandic, J., Medojevic, M.: Improving teaching and learning process by applying Lean thinking. Procedia Manuf. 17, 595–602 (2018) 12. Sanders, A., Elangeswaran, C., Wulfsberg, J.P.: Industry 4.0 implies lean manufacturing: research activities in industry 4.0 function as enablers for lean manufacturing. J. Ind. Eng. Manage. (JIEM) 9(3), 811–833 (2016) 13. Schuh, G., Gartzen, T., Rodenhauser, T., Marks, A.: Promoting work-based learning through industry 4.0. Procedia CIRP 32, 82–87 (2015) 14. Tyagi, S., Choudhary, A., Cai, X., Yang, K.: Value stream mapping to reduce the lead-time of a product development process. Int. J. Prod. Econ. 160, 202–212b (2015) 15. Marodin, G., Frank, A.G., Tortorella, G.L., Netland, T.: Lean product development and lean manufacturing: testing moderation effects. Int. J. Prod. Econ. 203, 301–310 (2018) 16. Mourtzis, D., Doukas, M., Lalas, C., Papakostas, N.: Cloud-based integrated shop-floor planning and control of manufacturing operations for mass customisation. Procedia CIRP 33, 9–16 (2015) 17. Almada-Lobo, F.: The Industry 4.0 revolution and the future of Manufacturing Execution Systems (MES). J. Innov. Manage. 3(4), 16–21 (2016) 18. Zhong, R.Y., Xu, C., Chen, C., Huang, G.Q.: Big data analytics for physical internet-based intelligent manufacturing shop floors. Int. J. Prod. Res. 55(9), 2610–2621 (2017) 19. Åkerman, M., Stahre, J., Engström, U., Angelsmark, O., McGillivray, D., Holmberg, T., Bärring, M., Lundgren, C., Friis, M., Fast-Berglund, Å.: Technical interoperability for machine connectivity on the shop floor. Technologies 6(3), 57 (2018) 20. Susman, G.I., Evered, R.D.: An Assessment of the scientific merits of action research. Adm. Sci. Q. 23(4), 582 (1978) 21. Gersing, K., Oehmen, J., Rebentisch, E.: Designing workshops for the introduction of lean enablers to engineering programs. Procedia Comput. Sci. 28, 643–652 (2014) 22. Luca, L., Pasare, M., Stancioiu, A.: Study to determine a new model of the Ishikawa diagram for quality improvement. Fiability Durab. 1, 249–254 (2017)
Evolutionary Dynamics in Azorean Landscapes: The Land-Use Changes in Forests and Semi-natural Areas in the Archipelago from 1990 to 2018 Rui Alexandre Castanho1,2,3,4(B) , Gualter Couto2 , José Manuel Naranjo Gómez3,4,5 , Pedro Pimentel1 , Célia Carvalho6,9,10 Áurea Sousa7 , Maria da Graça Batista1 , and Luís Loures4,8
,
1 School of Business and Economics and CEEAplA, University of Azores,
9500-321 Ponta Delgada, Portugal 2 Faculty of Applied Sciences, WSB University, 41-300 D˛abrowa Górnicza, Poland
[email protected] 3 CITUR - Madeira - Centre for Tourism Research, Development and Innovation,
9000-082 Funchal-Madeira, Portugal 4 VALORIZA - Research Centre for Endogenous Resource Valorization, Polytechnic Institute
of Portalegre (IPP), 7300 Portalegre, Portugal 5 Agricultural School, University of Extremadura, 06007 Badajoz, Spain 6 Faculty of Social and Human Sciences and CEEAplA, University of Azores,
9500-321 Ponta Delgada, Portugal 7 Faculty of Sciences and Technologies and CEEAplA, University of Azores, 9500-321 Ponta
Delgada, Portugal 8 CinTurs—Centre for Spatial and Organizational Dynamics, University of Algarve, 8005-139
Faro, Portugal 9 CINEICC -Cognitive and Behavioural Centre for Research and Intervention Faculty of
Psychology and Education Sciences, University of Coimbra, Coimbra, Portugal 10 Institute for Genomic Health - Department of Psychiatry/College of Medicine SUNY
Downstate , Brooklyn, N.Y, USA Abstract. The particular richness and singularity of Azorean Landscapes demand a strong, well-defined, and extensive conservation policy planning. Accordingly, and recognizing the importance of the problem in the enlightenment of the sustainability thought, those planning approaches should be based and sustained by multiple studies and thematic fields to fully comprehend the issue. Thereby, the present research through Geographic Information Systems (GIS) tools and methods enables discussing the evolution of Forest and Semi-natural areas in the Azores Islands in the last three decades. Contextually, this investigation allows us to confirm that the land uses related to Forests and Semi-natural Areas experienced multiple changes – increasing and decreasing periods. For example, some of those reducing dynamics are concerning and should have a closer monitorization by the territorial government actors to provide protection preservation and conservation to these incomparable Ultra-peripheral Landscapes, Environments, and Ecosystems. Keywords: Azores region · Land-use changes · Regional studies · Landscape planning · Sustainable development © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 244–252, 2021. https://doi.org/10.1007/978-3-030-72651-5_24
Evolutionary Dynamics in Azorean Landscapes
245
1 Introduction Nowadays, there is a broad typology of several mapping models. According to Bartholomé and Belward [1]: “Multiple worldwide or mainland informational databases identifying with land cover have been created, planned, and produced.” Within all of the existing Geographical Information Systems (GIS), it should be highlighted the Global Land Cover (GLC2000) designed for the year 2000 or the Pan-European Land Cover Monitoring Project (PELCOM). Additionally, there is the Coordination of Information on the Environment (CORINE) cartography for the territorial and regional scales [2]. Land-use and regional occupation changes are a matter of extraordinary relevance at global, national, and regional levels once they create consequences on the biological, natural, and financial frameworks [3–5]. So, land-use and regional occupation changes are vital signs of human activities over the autochthonous habitat [6]. The assessment of land-use changes has been vital in thematic areas, such as spatial and urban planning, regional management and governance, ecosystems preservation and conservation, strategy planning, economic and social level, among many different disciplines [5]. Consequently, it is necessary to have a piece of progressively relevant information and understand how to promptly deal with the current regional assets in a more proficient and coordinated manner. Besides, it is indispensable to make the region more solid, improving its adjustment face to the developing behavior of the elements of globalization and external influences, despite their whether financial or climatic [7]. Contextually, the necessity to gain more information about the earth’s surface and its territorial occupation fostered several initiatives to analyze the use and land cover and their associated dynamics and patterns [5]. Therefore, Europe has managed a monitorization process of the land-use changes and territorial occupation in a combined way. Several authors as Pinto-Correia and Kristensen [8], Naranjo Gómez et al. [5] recognize that: “the diversity of land use and occupation maps results from the application of several methodologies and models underlying their achievement - many layouts, syntactic issues, schematic heterogeneity, and semantic aspects.” Also, land cover and land-uses cartography can represent an essential function in regulating a singular area’s economic, political, social, and ecological variables [8]. Various researches have been performed in European territories involving land-use changes, their components, and patterns (see: Santana-Cordero et al. [9]; Castanho et al., [10, 11]; Naranjo Gómez et al., [7, 8]; Vizzari et al. [12]; Pérez-Hernández et al. [13]; Varga et al. [14]; are just a few examples). According to Castanho [15]: “ (…) a regional planning strategy is an essential instrument for ascribing pre-conditions of well-being to their populations”. Consequently, to seek success for future generations who will live in that region, promoting the reduction of social unbalances and spatial disparities, and being an incentive instrument for sustainable development and territorial governance [16–21]. In this regard, the current study will analyze and assess the changes and evolution of Forest and Semi-natural areas in the Azores Archipelago in the last three decades, based on the CORINE Land-Use (CLC) data. Therefore, the evaluation of those evolutionary dynamics empowers us to produce some guidelines and considerations for the future territorial planning and management policies and strategies to be designed and conducted over Azorean landscapes.
246
R. A. Castanho et al.
2 The Azores Landscape in Brief The nine islands of the Azores Archipelago are located in the triple junction between the Eurasian, American, and African plates (Nubia), in a complex geodynamic context. Given the intense seismic and volcanic activity observed in the Azores, historical documents from the island population recorded more than 20 volcanic eruptions. The presence of strong natural forces translates into a mystical landscape and peculiar beauty, reflected in the volcanic morphology, from which stands out the mountain of Pico, the diversity of boilers and lagoons that are found in several islands, the expression of ephemeral landscapes such as the Capelinhos volcano or the Furnas fumaroles. The sea is also an almost ubiquitous presence on all the islands and directly and indirectly influences the characteristics of the coastal strip, concerning the height of the cliffs, the cut of the coast, the presence of vegetation on the slopes and the presence of Fajãs, with particular highlight to São Jorge and Flores. The Azores archipelago climate is essentially dictated by the geographic location of the islands in the context of the global atmospheric and oceanic circulation and by the influence of the aquatic mass from which they emerge. Generally, the climate of the region is characterized by mild temperature, high levels of humidity in the air, low insolation rates, regular and abundant rains, and a regime of strong winds that hover the archipelago as it evolves of atmospheric circulation patterns at the scale of the North Atlantic basin. However, the islands’ climate presents a seasonality of intermediate level, which is reflected in the different elements of the climate, like the rate of precipitation, the average temperature, and the level of relative humidity. Nevertheless, the four seasons, typical of temperate climates, are recognizable. One of the most remarkable characteristics of the Azorean climate is observed in the wide variety of weather conditions in periods as small as an hour. It is usual to observe in a relatively short period, at random, periods of rain or sun and wind or calm. The simultaneous occurrence of combinations of these same conditions in a single zone is also expected in the archipelago. The climate affects some of the Azores Autonomous Region’s economic sectors, namely, Tourism, Agriculture and Livestock, Transport, and Energy. In addition to the impacts above-mentioned, natural disasters caused by storms tend to repeat themselves over the years, requiring the gathering and allocation of resources, already scarce, to assist the affected populations and the reconstruction of the areas affected.
3 Land-Uses Changes in Forests and Semi-natural Areas in Azores Archipelago in the Period of 1990–2018 For a precise classification of the land-use changes in forest and semi-natural areas in the Azores Archipelago, it was essential to use the data provided by CLC (Corine Land Cover) through the European Environment Agency (EEA) platform; however, some of the level 3 land-uses were not possible to be obtained. Thus, it was employed a working scale of 1:100,000, with the minimum cartographic unit of 25 ha. So, the geometric accuracy was always greater than 100 m. Additionally, the representation of the land uses was created by polygonal graphical items. Polygons
Evolutionary Dynamics in Azorean Landscapes
247
are linked with a land-use code to which they refer. Hence, specific coding is produced in three different levels of detail (Table 1). Table 1. CLC nomenclature for Forest and Semi-natural Areas . (Source: [22]) Level 1
Level 2
Level 3
3 Forest and semi-natural areas
31 Forests
311 Broad-leaved forest 312 Coniferous forest 313 Mixed forest
32 Scrub and/or herbaceous vegetation associations
321 Natural grasslands 322 Moors and heathland 324 Transitional woodland-shrub 332 Bare rocks 333 Sparsely vegetated areas
Throughout the use of the GIS mentioned above, it was conceivable to assemble the total surface of forest and semi-natural areas according to CLC in the Azores Region between 1990 and 2018 (Table 2). Table 2. Surface (%) of forest and semi-natural areas according to Corine land cover in Azores Archipelago in 1990, 2000, 2006, 2012 and 2018. Source: authors. Code Years (%) 1990
2000
2006
2012
2018
311
9,36%
9,57% 9,40% 9,38% 9,88%
312
3,63%
4,10% 4,35% 4,45% 4,11%
313
0,89%
0,95% 1,02% 1,02% 1,01%
321
8,02%
7,97% 7,87% 7,76% 7,38%
322
8,65%
8,63% 8,42% 8,42% 9,10%
324
4,41%
4,08% 4,62% 4,72% 4,10%
332
0,25%
0,25% 0,25% 0,25% 0,25%
333
0,82%
0,82% 0,79% 0,78% 0,72%
248
R. A. Castanho et al.
Forests Focusing on Level 2 of CLC for Land Uses 3 (Forests), there are three typologies of land uses Broad-leaved forest (311), a Coniferous forest (312), and Mixed forest (313). In Azores Region, in the analyzed period (1990–2018), for the Land Use (LU) 311 the highest value was found in 2018 with 9,88% of surface and the minimum value in the year of 1990 – with 9,36% of the surface occupied by broad-leaved forest. Regarding LU 312, the maximum occupied surface was identified in the year 2012 (4,45%), and the lowest in the year of 19990 (3,63%). If we focus on the LU 313, it is possible to verify a smoother dynamic change in mixed forest surfaces in Azores Region - with a maximum of 1,02% (in 2012) and a minimum of 0,89% (in 1990). Scrub and/or Herbaceous Vegetation Associations Regarding the level of Scrub and herbaceous vegetation associations, there are four types of LU: Broad- Natural grasslands (321), Moors and heathland (322), Sclerophyllous vegetation (323) (however in this preliminary study this code was not obtained), and Transitional woodland-shrub (324). In this regard, for the LU 321, the highest value was found in 1990 (8,02%) and the minimum value in the year 2018, with 7,38% of the Azores Region surface occupied by Natural grasslands – which represents a decrease of 0,64 from 1990 to 2018. Considering LU 322, the maximum occupied surface by it was verified in the year 2018 (9,10%), and the lowest in the year 2012 (8,42%). Therefore, moors and heathland surfaces increased 0,68% from 2000 (lowest) to 2018 (maximum) in the Archipelago. Looking to the LU (324) transitional woodland-shrub, the maximum value of occupied surface (4,72% in the year 2012) and the minimum (4,08% in the year 2000). Open Spaces with Little or No Vegetation Considering open spaces with little or no vegetation, CLC has five different typologies of LU, nevertheless, in the present study only two were studies: Bare rocks (332), and Sparsely vegetated areas (333). The LU 332 (Bare rocks) remained stable in the Azores Region in the period of1990 to 2018. For LU 333, we verified a small decrease, in this case, of 0,10% of surfaces occupied by Sparsely vegetated areas. With the maximum in the year of 1990 (0,82%) and the minimum in 2018 (0,72%).
4 Discussion and Conclusions The current Azorean landscape results from a profound humanization that took place over five centuries, with dynamics influenced by decisive historical events that insularity faded in some cases, delaying the effects of cultural and technological evolution with generally negative repercussions - however, sometimes also positive. This evolution was part of significant landscape changes based on long cycles dominated by some crops such as cereals, indigo, vines, oranges, tea, pineapples, cryptomeria, or pastures. In more recent times, we have witnessed more intensive and rapid transformations and more localized, such as constructing large infrastructures – i.e., airports, ports, highways or urban expansions of the leading centers. The improvement of the means of communication with the outside, both with the Continent and with the world in general, and the
Evolutionary Dynamics in Azorean Landscapes
249
bet that has been made to promote and publicize the Archipelago in the last decades, has been reflected in a set social, economic, and cultural dynamics that directly or indirectly interact with the landscape and are the source of essential problems. Generally, as a result of climate change, in addition to an increase in average temperature, more frequent and severe droughts are expected to occur in the median-terrestrial basin and a greater frequency of extreme climatic phenomena. In the Azores, a redistribution of precipitation is expected with an increase in rainfall in winter, offset by a reduction in other seasons, giving rise to a shorter and more concentrated rainy season. However, it is to be expected that when extreme precipitation events occur in winter, mass movements on unstable slopes may be more frequent, increasing a danger that is already a reality in this Region. The climate can get warmer around 1–2 ºC. However, it is expected that the endemic vegetation itself may be part of the solution. This is because the laurel forest’s physiological characteristics lead this type of vegetation to be suitable for capturing the water present in the clouds and slowly causing it to condense and infiltrate the aquifers. This efficacy of endemic laurisilva vegetation, with broader and leatherier leaves, such as the species Laurus azorica, Ilex azorica, and Viburnum treleasei, distinguishes it favorably from other types of soil cover, such as pastures or forests of cryptomeria (Cryptomeria japonica) or Sphagnum peat bogs (Sphagnum sp.) Found in the islands’ highlands also play an essential role in the slow infiltration of water and underground aquifers’ supply. Therefore, endemic vegetation is a vital ground cover in areas intended to promote the infiltration of water in the subsoil. The fact that precipitation is more concentrated in winter may predispose the islands to a higher frequency of drought episodes. The solutions may include the adequate storage of precipitation, both in ponds and reservoirs and in cisterns, and mainly by adequate spatial planning that protects areas with more significant water infiltration and the sources of watercourses. On islands such as Graciosa, where there have been frequent episodes of water shortages, the recovery of elements of traditional architecture linked to water may have a dual function of storing this precious resource and recovering cultural heritage. Concerning the repercussions of the phenomena referred to in land-use changes, the changes should be highlighted at various levels. Changes in the pasture area, which until 1999 underwent an evident expansion, occupying lower and higher areas than those traditionally used for this purpose, eliminating agricultural, forest, and pond areas, some of which were used for water supply. The management of drainage basins for these lake systems also requires measures that necessarily involve reducing the pasture area or applying good practices. The basin plans have been implementing, seeking to correct them, as was the drainage basin’s case. From Lagoa das Furnas in São Miguel, the National Landscape Prize was awarded in 2012 to the Furnas Landscape Laboratory. The exploitation of quarries and gravel should also be worthy of special attention, in order to lessen the impacts on the landscape and the worsening of erosion problems, whose activity is already covered by a sectorial land-use plan, published in 2015 in the Plan For Extractive Activities in the Autonomous Region of the Azores. We have also seen an increase in forest areas, emphasizing cryptomeria settlements, a tree originally from Japan, first used in shelters and hedges. Later, given its easy adaptation to the Azorean climate, it was the most used conifer in the Azores’ forestry.
250
R. A. Castanho et al.
The construction of structures and infrastructures with poor landscape integration as the over-dimensioned roads in São Miguel and Terceira, related to the valorization of the car as a social symbol, when the current trends are in the sense of considering public transport as an alternative, represents more rational management from an environmental and economic perspective. It is also essential to ensure the integration of other structures and infrastructures since adopting balanced solutions in harmony with the surrounding spaces is essential for enhancing the landscape, the natural, and built heritage. The abandonment of agricultural areas and the consequent degradation of traditional systems and some of the associated built and cultural heritage, namely the vineyards in biscuit pens, the old cellars, some compartmentalization walls, the windmills, the watermills, despite the effort inherent to the occasional recovery of some specimens and the protection of landscapes such as the Pico Island Vineyard Culture Landscape, designated by UNESCO in 2004. Ultimately, the increase in tourist demand and the risk of pressure on specific places, namely for equipment construction, is time to plan correct management of this expanding sector. The investigation of the land-use changes patterns and dynamics is fundamental to comprehend the tendencies and developments of regions [17–20]. Throughout this preliminary research, it was possible to recognize the changes in the forest and semi-natural areas according to Corine land cover in the Azores Region in the Period 1990–2018. Consequently, it was conceivable to prove that these land uses suffered some changes, characterized by increasing and decreasing periods. In this sense, some of those decreasing values are concerning and should have special attention by the autonomous government authorities to provide preservation and conservation towards these unique Azorean landscapes and environments. Therefore, considering the uniqueness of these ultra-peripheral territories, their landscapes and environments, the main-actors and some of their policies and actions over the forest and semi-natural areas require to be re-think, re-designed, and administered in a more reasonable manner.
5 Prospective Research Lines Even if this preliminary study provides us some insights on the patterns, dynamics, and specificities of the land uses changes in forest and semi-natural areas in the Azores Region, more studies that allows us to intersect more variables and details critical to evolving in this thematic field should be developed. Furthermore, we should consider the fast changes in regional policies and societal behaviors that these ultra-peripheral territories present. These changes, simultaneously with the different administrative systems of the Portuguese mainland and the Autonomous Government by the other, lead us to the necessity for close monitoring of the trends and dynamics of the land-use changes and the territorial governance, in a way that it follows the so-desired sustainable development strategies.
Evolutionary Dynamics in Azorean Landscapes
251
Moreover, in this preliminary research, only the forest and semi-natural areas were studied. So, to better understand the LUC dynamics, all the LU of the Azores Archipelago should be analyzed. Funding. This paper is financed by Portuguese national funds through FCT – Fundação para a Ciência e a Tecnologia, I.P., project number UIDB/ECO/00685/2020 and also by the project GREAsT - Genuine Rural Experiences in the Azores Tourism with the code: ACORES-01-0145FEDER-000089.
References 1. Bartholomé, E., Belward, A.S.: GLC2000: A new approach to global land cover mapping from earth observation data. Int. J. Remote Sens. 26(9), 1959–1977 (2005). https://doi.org/ 10.1080/01431160412331291297 2. García-Álvarez, D., Camacho Olmedo, M.T.: Changes in the methodology used in the production of the Spanish CORINE: uncertainty analysis of the new maps. Int. J. Appl. Earth Obs. Geoinf. 63, 55–67 (2017). https://doi.org/10.1016/j.jag.2017.07.001 3. Gao, P., Niu, X., Wang, B., Zheng, Y.: Land use changes and its driving forces in hilly ecological restoration area based on gis and rs of northern China. Sci. Rep. 5, 11038 (2015) 4. Santos, R., Castanho, R.A., Lousada, S.: Return migration and tourism sustainability in Portugal: extracting opportunities for sustainable common planning in Southern Europe. Sustainability 11, 6468 (2019). https://doi.org/10.3390/su11226468 5. Naranjo Gómez, J.M., Lousada, S., Garrido Velarde, J., Loures, L., Castanho, R.A.: Land use changes in the Canary Archipelago using the CORINE data: a retrospective analysis. Land 9, 232 (2020). MDPI 6. Fadigas, L.: Urbanismo e Natureza: Os Desafios. Sílabo. Lisbon (2010). ISBN 978-972-618595-6 7. Naranjo Gómez, J.M., Loures, L., Castanho, R.A., Cabezas, J., Fernández-Pozo, L., Lousada, S., Escórcio, P.: Assessing land use changes in European territories: a retrospective study from 1990 to 2012. IntechOpen (2018). https://doi.org/10.5772/intechopen.80827 8. Pinto-Correia, T., Kristensen, L.: Linking research to practice: the landscape as the basis for integrating social and ecological perspectives of the rural. Landscape Urban Plan. 2013(120), 248–256 (2013). https://doi.org/10.1016/j.landurbplan.2013.07.005 9. Santana-Cordero, A.M., Monteiro-Quintana, M.L., Hernández-Calvento, L.: Reconstruction of the land uses that led to the termination of an arid coastal dune system: the case of the Guanarteme dune system (Canary Islands, Spain), 1834–2012. Land Use Policy 55, 73–85 (2016) 10. Castanho, R.A., Kurowska-Pysz, J., Naranjo Gómez, J.: Assessing land use changes in polish territories: patterns, directions and socioeconomic impacts on territorial management. Sustainability 11, 1354 (2019). https://doi.org/10.3390/su11051354. Special Issue: Spatial Analysis and Geographic Information Systems 11. Castanho, R.A., Lousada, S., Naranjo Gómez, J.M., Escórcio, P., Cabezas, J., FernándezPozo, L., Loures, L.: Dynamics of the land use changes and the associated barriers and opportunities for sustainable development on peripheral and insular territories: the Madeira Island (Portugal). IntechOpen (2018). https://doi.org/10.5772/intechopen.80827 12. Vizzari, M., Hilal, M., Sigura, M., Antognelli, S., Joly, D.: Urban-rural-natural gradient analysis with CORINE data: an application to the metropolitan France. Landscape Urban Plan. 171, 18–29 (2018)
252
R. A. Castanho et al.
13. Pérez-Hernández, E., Santana-Cordero, A.M., Hernández-Calvento, L., Monteiro-Quintana, M.L.: Beach surface lost historically: the case of the eastern coast of Las Palmas de Gran Canaria (Canary Islands, Spain). Ocean Coast. Manage. 185, 105058 (2020) 14. Varga, O.G., Pontius, R.G., Jr., Szabó, Z., Szabó, S.: Effects of category aggregation on land change simulation based on Corine land cover data. Remote Sens. 12(8), 1314 (2020) 15. Castanho, R.A.: Identifying processes of smart planning, governance and management in European Border cities. Learning from city-to-city cooperation (C2C). Sustainability 11, 5476 (2019). https://doi.org/10.3390/su11195476 16. Loures, L., Panagopoulos, T.: Reclamation of derelict industrial land in Portugal - greening is not enough. Int. J. Sustain. Dev. Plann. 5(4), 343–350 (2010) 17. Loures, L.: Post-industrial landscapes as drivers for urban redevelopment: public versus expert perspectives towards the benefits and barriers of the reuse of post-industrial sites in urban areas. Habitat Int. 45(2), 72–81 (2015) 18. Naranjo, J.M.: Impacts on the social cohesion of mainland Spain’s future motorway and high-speed rail networks. Sustainability 2016(8), 624 (2016) 19. Ulucak, R., Yücel, A., Koçak, E.: The process of sustainability: from past to present. In: Environmental Kuznets Curve (EKC) (2019) 20. Vulevic, A., Castanho, R.A., Naranjo Gómez, J.M., Loures, L., Cabezas, J., Fernández-Pozo, L., Martín Gallardo, J.: Accessibility dynamics and regional cross-border cooperation (CBC) perspectives in the Portuguese—Spanish borderland. Sustainability 12, 1978 (2020) 21. Bilgili, F., Ulucak, R., Koçak, E., ˙Ilkay, S.Ç.: Does globalization matter for environmental sustainability? Empirical investigation for Turkey by Markov regime switching models. Environ. Sci. Pollut. Res. 27(1), 1087–1100 (2020). https://doi.org/10.1007/s11356-019-069 96-w 22. European Environment Agency (EEA): CORINE LAND COVER (2020). www.eea.europa. eu/publications/COR0-landcover
A Comparative Study of Classifier Algorithms for Recommendation of Banking Products Ivan F. Jaramillo1(B) , Ricardo Villarroel-Molina1 , Bolivar Roberto Pico1 , and Andr´es Redchuk2 1
2
Engineering Sciences Department, Universidad T´ecnica Estatal de Quevedo, Quevedo, Ecuador {ijaramillo,ricardormol.villarroel,rbpico}@uteq.edu.ec Superior Technical School of Computer Science, Universidad Rey Juan Carlos, Madrid, Spain [email protected]
Abstract. Currently, the processing and storage capacities of computers have facilitated the handling of large volumes of data. Robust classification models are used by organizations in various sectors, but the algorithms are not efficient with all data sets; in fact, high-volume data can significantly change the most promising algorithms. In this work, a comparative study of classification algorithms was developed; To do this, we use a high-volume data set related to banking products purchases. We selected eight important algorithms from the literature in the first stage, but only five reached a second stage after the tree-based algorithms presented memory overflow problems. The criteria for quality evaluation were based on AUC (Area under the curve) measurements and training and testing time; in this investigation, the MARS algorithm was shown to be slightly superior to SVM in classification quality; while in execution times, both for training and for testing, MARS reached lower times. Both the MARS and SVM algorithms could be used as classifiers in this data set; however, the model construction costs should be considered when fractional construction methods are not used.
Keywords: Classification algorithms Algorithms comparison
1
· Banking product classification ·
Introduction
Nowadays, different types of companies are looking at strategies to support the needs and interests of their customers; therefore, goal setting is the key to meet those needs effectively. There are many ways to achieve those goals; however, many of them are supported by large amounts of data; thus requiring to make c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 253–263, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_25
254
I. F. Jaramillo et al.
models and techniques to help decision making at a higher level of the organization [11], and data mining techniques are an alternative. Data mining is a detailed process to discover patterns and extract knowledge from large volumes of data. An important area in data mining is classification techniques; they have been used to solve many problems such as recommendation systems based on quantitative data as social content. Also, direct marketing strategies are applying by exploiting data on banking products; Targeted telemarketing campaigns and bank credit evaluation are several examples in the commercial and business sector [4]. In practice, there is not an ideal technique to solve all optimization tasks [21]; therefore, it is always necessary to analyze the behavior of the algorithms against a new data set. In this research, we use of Santander Bank dataset, which is available on Kaggle website1 . It was used in a competition to build a recommendation system to predict which products their existing customers might purchase in the future based on their past behavior and similar customers. To develop our research, the following steps were performed. Firstly, the pre-processing tasks are performed to guarantee quality data. Secondly, several experiments with selected algorithms are developed, and the k-fold crossvalidation experimental method is used to obtain a stabilized model. Third, we use AUC metric and response time as quality measures. Finally, the measurements are compared using non-parametric statistical tests to find significant differences. According to quality measure Area Under Curve (AUC), the classifier built by the MARS algorithm shows superiority in comparison with Neural Net (NNET), Na¨ıve Bayes (NB) and Support Vector Machine (SVM) classifiers.
2
Overview
The studies based on the Gradient Boosting Model are achieving satisfactory results for classification problems [1,13]. Researchers contribute with some variants of algorithms in this line; thus, they have high-quality methods based on the LightGBM algorithm and combined with Bayesian optimization, where comparative studies with other classics derived from SVM, XGBoost, and Random Forest have achieved better results in problems related to credit card fraud [12]. On the other hand, Ensemble is one of the most popular and successful machine learning techniques, and decision trees are widely used algorithms for building these models. Interesting comparative analyzes have been developed in this context; the work [7] evaluates the performance of important algorithms such as XGBoost, LightGBM, and Random Forest. The result places XGBoost slightly above the others; classification trends are definitely biased toward using these techniques, and the reduced computational costs have contributed significantly to their popularity. Also, classical techniques continue to achieve significant results. Support Vector Machine Spark is a supervised learning technique, 1
Founded in April 2010, Kaggle is a website specialized in data science competitions. https://www.kaggle.com.
Classifiers Comparison
255
and in the study, they demonstrate superiority over Back Propagation Networks in the field of bank fraud detection [8]. The experimental setting is a predominant criterion in evaluating algorithms; we consider a simple evaluation environment given the computational capabilities. Next, we describe the relevant characteristics of the main classification algorithms evaluated in this work. 2.1
MARS Algorithm
Multivariate Adaptive Regression Splines (MARS) [5] is an adaptive method to define a non-parametric regression model in which the functional relationship between dependent and independent variables is not assumed; however, that relationship is made from a set of coefficients and functions basic, which are greatly influenced by the regression of the data. Comparative studies conducted by [9,22] indicate that MARS is suitable for problems with some significant variables, non-linearity, multi-collinearity or a high degree of iteration among predictors, in addition to presenting advantages over other techniques such as NNET, and can cover different classification types. 2.2
Na¨ıve Bayes Classifier (NBC)
This classifier is based on Bayes theorem [17]; some research on classification algorithms comparing has found that a simple Bayesian classifier’s performance can be comparable with decision trees and neural networks. Bayesian classifiers have achieved high precision and speed when applied to large databases. The algorithm combines apriori and conditional probabilities in a single formula, which is used to calculate each possible classification; once that was done, then classification with the largest value is chosen. NBC, despite being considered a highly biased classifier because it uses a simple model for inference, has shown robustness, both in its applications and in performance; likewise, it is to be assumed that it works better in small data sets due to its limited generalizability produced by elements not considered in training. A study based on the assumption of independence of the attributes which also limits this technique. From some models obtained, the relationship between the error and the dependency of the attributes was analyzed. Based on this study, it was shown how this affects both the probability of estimation and the quality of classification; furthermore, by forming a combined set of attributes, an excellent Naive Bayes approximation could be achieved [19]. 2.3
Support Vector Machine (SVM)
Through Support Vector Machine (SVM) [3], binary classification tasks are done by linear separation, and it consists of maximizing the margin between linear separation by using a quadratic programming formulation. When the classes are
256
I. F. Jaramillo et al.
not linearly separable, it needs to be solved in a very high dimensional space. Due to the high computational costs of moving to a space of high dimensionality, the kernel trick is used as a solution An unfavorable feature of this technique is its computational cost. It requires large matrix operations and numerical calculations; both training and classifier testing requires a lot of time and memory. 2.4
Neural Net (NNET)
The first mathematical model of an artificial neuron was created to carry out simple estimation tasks. It was presented in 1943 by joint work between the psychiatrist and neuroanatomist Warren McCulloch and the mathematician Walter Pitts [10]. The techniques based on artificial neural networks are non-linear models that can be used for classification tasks. A network of artificial neurons comprises a set of processing units (neurons) connected, with each of these connections having an associated weight (weighted connection). Each of these units has an activation level and performs a linear calculation that provides a means to update this activation level. Different kinds of algorithms use this method as a learning technique, and they use a simple single-layer structure or multiple layers. 2.5
Classification Metrics
By measuring the quality of the classifier is essential to establish a comparison criterion. An important concept applied to this task is the definition of truepositives, true-negatives, false-positives, and false-negatives; hence the most used evaluation measures are the rate of error, sensitivity, false alarm rate, specificity, and accuracy. Also, for accuracy, classifiers can be evaluated to speed, robustness, scalability, and interpretability. When we want to analyze the accuracy of classification between two or more algorithms, we can use certain measures, such as a confusion matrix, ROC analysis, or Area Under the Curve (AUC). AUC is considered one of the best methods to compare classifiers; an algorithm with high accuracy may not be sufficient because there may be a high number of false-positives in the results. It has been shown that AUC is statistically consistent, measures discrimination accuracy, and does not provide information that is redundant with diagnostic [16].
3
Methodology
This research collects contributions of important bibliographic databases for the study of the problem. In experimentation, we use a set of banking origin data, together with the software R Project for the analysis; next, we show the stages carried out in the research.
Classifiers Comparison
257
– Theoretical and conceptual bases: The bibliographic research was carried out in recognized databases (Scopus, Springer, IEEE and Science Direct). Topics covered are about algorithms and classification techniques applied to banking product problems. – Definition of objectives of the data mining process. In this stage of the process, the objectives to be achieved with the data analysis were established. – Data comprehension: It is a stage where an exploratory process is carried out with the data’s characteristics, identifying problems of the quality of the data, and carrying out a descriptive analysis. – Data Pre-processing: Debugging the data set. Some pre-processing tasks require sophisticated algorithms when the data set is large, which leads to the use of techniques based on evolutionary computation to achieve adequate times. In this phase, relevant tasks were carried out, such as a correction in data entry, elimination of irrelevant attributes (three attributes were identified), and finally handling missing values. – Modelling: Firstly, we proceed to the selection of a group of relevant classification algorithms according to the published results in bibliographic references; after of first selection, we start the experimental phase consisting of training and test. Two iterations were needed to fit the data set. In the first iteration, a simple random statistical sampling was carried out; for the sampling, it was in mind data were obtained in periods of seventeen different months, and each of the twenty-four products is an individual classification problem; therefore, five samples were obtained for each product per month, achieving 2040 sets of samples. A second pre-processing was carried out after identifying factors that produced failures in the execution of the algorithms in the first iteration; thus, transformation and combination of variables helped to reduce the number of categories in some cases. – Evaluation: Classifiers were compared using two quality indicators, the AUC metric and the computational time on construction/evaluation of the classifier; this allowed selecting the technique that on average achieved the most optimal values. The statistical comparison of the classifiers was also carried out using the Friedman and post hoc Friedman-Nemenyi hypothesis tests. – Analysis and interpretation of results: The tables and graphs obtained summarize the results achieved. These are a base of criteria that, together with the theoretical principles, allow the inference of assertions and conclusions. – Research report: Report is based on all the events that occurred and procedures performed in the research, from the documentary search phase to the interpretation of the results stage.
4
Materials
Experimental tests were carried out on HP ProLift server with 4 processors of 2.5 GHz and 10 cores each, 32 GB of RAM and 2 TB of hard disk; CentOS operating system 6.4.0. Microsoft R Open v3.3.3, Machine Learning package in R (MLR) v2.11.0.
258
5
I. F. Jaramillo et al.
Results
This section shows the series of results obtained, according to our proposed methodology; for this, we use tables and graphs to achieve a greater understanding. The data set is made up of forty-three categorical variables, two integers, and one real variable. The data is divided into two groups: customer (twenty-four attributes) and bank product(twenty-four attributes). It contains forty-eight attributes and nearly fourteen million instances. Table 1 shows a section of the data set with the products the customer has accessed, the data type of these attributes is binary (categorical), if the attribute value is one, then the customer owns such a product. For the treatment of missing values, the following tasks were performed: elimination of instances that have more than 40% of missing values in their attributes, a total of 27734 instances eliminated, attributes with more than 40% missing values are discarded and finally, attributes with a missing value rate of less than 20% were imputed by classical completion methods. Table 1. Data set section with products that bank customers have access to Num Attribute name Negative class Positive class 25
Saving acc.
13’645913
1396
26
Guarantees
13’646993
316
27
Current acc.
4’701721
8’945588
28
Derivative acc.
13’641933
5376
29
Payroll acc.
12’543689
1’103620
30
Junior acc.
13’518012
129297
31
Private acc.
13’514567
132742
...
...
...
...
For the selection of algorithms, the conditions that were established in the research design were taken into account; several comparative studies carried out by different authors have allowed us to analyze the most significant algorithms in the field of the problem [4], Table 2 details the algorithms that participated in the selection process.
Classifiers Comparison
259
Table 2. Algorithms studied in the literature Comparison work
Data type Featured algorithm
[14] NB, Tree Decision, SVM
C,N
[4] MLPNN, log. regres., NB and C5.0
C,N
C5.0
[18] J48, NB, K-NN and NNET
C
J48
[20] LDA, NNET and SVM
C,N
NNET
[15] C5.0, EvTree, CART
C,N
C5.0
[2] log. regression, gradient busting, KNN, C4.5,
C,N
Random
NNET, LDA, QDA, LS-SVM and Random forest
SVM
Forest
In the first selection, we have considered SVM, NB, NNET, C5.0, MARS, Random Forest, K-NN, and EvTree. After submitting the algorithms to first experimentation with a significant stratified sample of the data set, we obtain C5.0, Random forests, EvTree, and K-NN are not suitable for this classification problem; they caused an overflow of memory in the construction of classifier due to the high dimensionality despite the high resources of the server. On the other hand, SVM, NB, NNET, and MARS, achieve the construction and evaluation process without problems, so we reduced our selection to four algorithms. Ten processors are set in the configuration, and each resampling iteration (train/test) is a parallel job. Data subsets were constructed through the manipulation for both instance quantities and rate class imbalance, then experimentation is carried out for which seven independent tests are adjusted, sending as a parameter the number of instances of 1K, 5K, 10K, 20K , 50K, 100K and 200K respectively; so the process of construction and evaluation of the classifiers were made 2800 times: (20 products) ∗ (4 algorithms) ∗ (5 iterations in the resampling) ∗ (7 tests) = 2800. In the same way, evaluation measures of the seven independent tests are obtained and they are saved in files for visualization stage. For the comparative study, tables and graphs of the performance measures have been obtained; nonparametric statistical tests of Friedman and post hoc Friedman-Nemenyi were made to determine significant differences between the results thrown. Table 3. Output of measures taken by algorithm Algorithm AUC
AUC Min. AUC Max. Training time (sec.) Test time (sec.)
MARS
0.94318 0.79361
1.00000
124.18022
0.29275
NB
0.92778
0.72058
1.00000
0.40704
47.01965
NNET
0.80311
0.41546
0.99968
9.59462
0.50354
SVM
0.94041
0.74331
0.99994
567.64585
38.87510
260
I. F. Jaramillo et al.
The average results obtained from the AUC metric are shown in Table 3, together with the estimated times for the four selected algorithms; Fig. 1 show graphically the behavior of the measure, as a function of the number of records, NNET does not appear because it has a value well below the scale.
Fig. 1. Average AUC measure by each classifier
The algorithms NB and NNET gain an advantage in training times as can be seen in Fig. 2, on the contrary, in the validation tests, it takes MARS advantage, very close of Neural net, Fig. 3 clearly illustrates this trend depending on the number of instances.
Fig. 2. Average time measure of training by each classifier
Classifiers Comparison
261
Fig. 3. Average time measure of test by each classifier
For statistical verification of the existence or non-existence of significant differences in the quality of algorithms, we use the Friedman test and FriedmanNemenyi post hoc; it usually applies to this type of measurements [6]. The individual values of p − value for the classifier pairs are showed in the Table 4. The hypothesis test (Friedman test) for AUC measures determined that there is a significant difference between the four algorithms(significant p − value is 2.616E − 10; and Friedman-Nemenyi post hoc test shows us that the difference in AUC measure, with a significance level of 0.01 between NNET and MARS (p − value = 1.2E − 08), and also with the difference between SVM and NNET (p − value = 5.2E − 08). Table 4. Friedman-test matrix AUC measure NB
Train-time
MARS
NB
NNET
MARS
NNET
MARS
0.012
–
–
1.6E-06 –
–
2.5E-09 –
0.05
–
0.06812 0.00084 –
NNET 1.2E-08 0.017 – SVM
0.995
0.025 5.20E-08 0.26
NB
Test-time
0.05
NB
NNET –
1.8E-11 6.1E-05 1.2E-08 0.99484 0.00217
The training-time values achieved by the classifiers are statistically significant with a value α = 0.05 and p−value = 5.063E−12; it proceeded to apply post-hoc test, which obtained a critical difference with a significance level of α = 0.01, between NB and MARS (p − value = 1.6E − 06), and it is also remarkable difference between SVM and NB (p − value = 1.8E − 11), notice also that SVM and NNET are significantly different. Finally, Friedman test for testing-time measure shows us significant difference between algorithms with a value α = 0.05 and a p − value = 1.089E − 11.
262
I. F. Jaramillo et al.
Likewise, the Friedman-Nemenyi post-hoc test was applied. We obtained differences with a level of significance α = 0.01, between MARS and NB (p − value = 2.5E − 09); also, there are remarkable significant differences between SVM and MARS, also notice between NB and NNET.
6
Conclusions
In this investigation, we selected four classifier algorithms to test their applicability on a large data set from banking products. The NET, NB, SVM, and MARS algorithms were the most relevant in this problem, while others based on trees did not achieve a single model. MARS classifier showed superiority in AUC quality values compared with NNET, NB, and SVM; however, this superiority is small concerning SVM, which increases for NB and is even more noticeable for NNET. When comparing the values of training and test time metrics, it was found that SVM and NB need more time for classifiers building; and, the MARS classifier needs the shortest test time. Therefore, regarding these two metrics, the best performing classifiers were constructed by the MARS algorithm and those with the lowest performance by the SVM algorithm. Concerning the literature, once again, MARS shows a better classification than NNET. The case of NB can be explained because it works better in small data sets, and patterns could be outside the model. The difference with SVM maybe because this technique has disadvantages in large and unbalanced data sets, which would cause very long training and test times.
7
Future Works
The biggest challenge facing the diversity of currently existing techniques is bigdata. Several research types focus on the insertion of strategies to achieve better response times and higher quality results. This work provides an alternative method that can be implemented under collaborative filtering in the recommendation systems for banking products.
References 1. Barik, S., Mohanty, S., Mohanty, S., Singh, D.: Analysis of prediction accuracy of diabetes using classifier and hybrid machine learning techniques. Smart Innov. Syst. Technol. 153, 399–409 (2021) 2. Brown, I., Mues, C.: An experimental comparison of classification algorithms for imbalanced credit scoring data sets. Expert Syst. Appl. 39(3), 3446–3453 (2012) 3. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995) 4. Elsalamony, H.A.: Bank direct marketing analysis of data mining techniques. Int. J. Comput. Appl. 85(7), 12–22 (2014)
Classifiers Comparison
263
5. Friedman, J.: Multivariate adaptive regression splines. Ann. Stat. 19(1), 1–67 (1991) 6. Garc´ıa, S., Fern´ andez, A., Luengo, J., Herrera, F.: Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inf. Sci. 180(10), 2044–2064 (2010) 7. Gonz´ alez, S., Garc´ıa, S., Del Ser, J., Rokach, L., Herrera, F.: A practical tutorial on bagging and boosting based ensembles for machine learning: algorithms, software tools, performance study, practical perspectives and opportunities. Inf. Fusion 64, 205–237 (2020) 8. Gyamfi, N., Abdulai, J.D.: Bank fraud detection using support vector machine. In: 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference, IEMCON 2018, pp. 37–41 (2019) 9. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning Data Mining, Inference, and Prediction, 2nd edn. Springer, Stanford (2009) 10. Haykin, S.: Neural Networks A Comprehensive Foundation, 2nd edn. Pearson Education, Singapore (1999) 11. He, Z.Z., Zhang, Z.F., Chen, C.M., Wang, Z.G.: E-commerce business model mining and prediction. Front. Inf. Technol. Electron. Eng. 16(9), 707–719 (2015) 12. Huang, K.: An optimized lightgbm model for fraud detection. In: Journal of Physics: Conference Series. vol. 1651 (2020) 13. Mazumder, R., Salman, A., Li, Y.: Failure risk analysis of pipelines using datadriven machine learning algorithms. Struct. Safe. 89 (2021) 14. Moro, S., Laureano, R.M.S., Cortez, P.: Using data mining for bank direct marketing: an application of the CRISP-DM methodology. In: Novais, P., Machado, J., Analide, C., Abelha, A. (eds.) 25th European Simulation and Modelling Conference- ESM’2011, pp. 117–121. Guimaraes, Portugal (2011) 15. Pang, S.L., Gong, J.Z.: C5.0 classification algorithm and application on individual credit evaluation of banks. Syst. Eng.-Theory Pract. 29(12), 94–104 (2009) 16. Rotello, C.M., Chen, T.: ROC curve analyses of eyewitness identification decisions: an analysis of the recent debate. Cogn. Res.: Princ. Implic. 1(1), 10 (2016) 17. Russell, S.J., Norvig, P.: Learning in Neural and Belief Networks. Alan Apt, New Jersey (1995) 18. Khare, S.: A Comparative Analysis of Classification Techniques on Categorical Data in Data Mining. Int. J. Recent Innov. Trends Comput. Commun. 3(8), 5142– 5147 (2015) 19. Stephens, C.R., Huerta, H.F., Linares, A.R.: Why the naive bayes approximation is not as naive as it appears. In: 2015 6th International Conference on Information, Intelligence, Systems and Applications (IISA), pp. 1–6 (2015) 20. S ¸ tefan, R.M.: A comparison of data classification methods. Proc. Econ. Finance 3, 420–425 (2012) 21. Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997) 22. Zhang, W., Zhang, R., Wu, C., Goh, A.T.C., Lacasse, S., Liu, Z., Liu, H.: State-ofthe-art review of soft computing applications in underground excavations. Geosci. Front. 11(4), 1095–1106 (2020)
New Capabilities of the Geometric Multidimensional Scaling Gintautas Dzemyda(B)
and Martynas Sabaliauskas
Vilnius University Institute of Data Science and Digital Technologies, Vilnius, Lithuania {gintautas.dzemyda,martynas.sabaliauskas}@mii.vu.lt
Abstract. Multidimensional scaling (MDS) is a prevalent method for presenting multidimensional data visually. MDS minimizes some stress function. We have proposed in [1] and [2] to consider the stress function and multidimensional scaling, in general, from the geometric point of view, and the so-called Geometric MDS has been developed. Geometric MDS allows finding the proper direction and step size forwards the minimum of the stress function analytically when coordinates of a separate point of the projected space are varied. In this paper, we examine the stress function theoretically and experimentally when simultaneously changing the position of all the points of the projected space in the directions, defined by the Geometric MDS strategy for a separate point. Several new properties and capabilities of the Geometric MDS have been discovered. The obtained results allow us to extend the understanding of properties and ideas of Geometric MDS for the future development of a class of new effective algorithms for multidimensional data visualization and its dimensionality reduction.
Keywords: Dimensionality reduction MDS · SMACOF · Geometric MDS
1
· Multidimensional scaling ·
Introduction
Multidimensional scaling (MDS) is one of the most popular method for multidimensional data dimensionality reduction and visualization [3,4]. Consider the multidimensional data set as an array X = {Xi = (xi1 , . . . , xin ), i = 1, . . . , m} of n-dimensional data points Xi ∈ Rn , n 3. Data point Xi = (xi1 , . . . , xin ) is the result of observation of some object or phenomenon dependent on n features. Dimensionality reduction means finding the set of coordinates (layout) of points Yi = (yi1 , . . . , yid ), i = 1, . . . , m, in a lower-dimensional space (d < n), where the particular point Xi = (xi1 , . . . , xin ) ∈ Rn is representd by Yi = (yi1 , . . . , yid ) ∈ Rd . If d 3, dimensionality reduction results may be presented visually for human decision.
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 264–273, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_26
New Capabilities of the Geometric Multidimensional Scaling
265
An example of dimensionality reduction using MDS is presented in Fig. 1 by visualizing 5-dimensional (n = 5) data of customers shopping behavior during an online advertising campaign (for details, see description of the data in [5]). We do not present legends and units for both axes in Fig. 1 with visualization results because we are interested in observing the interlocation of m = 2644 sales and finding some regular structures of data. Results in Fig. 1 allow deciding on the efficiency of the campaign.
Fig. 1. Example of dimensionality reduction, customers shopping data, n = 5
Data for MDS is the symmetric m × m matrix D = {dij , i, j = 1, . . . , m} of proximities,where dij = dji . MDS tries to hold the proximities dij between pairs of multidimensional points Xi and Xj , i, j = 1, . . . , m, as much as possible. Proximity dij can be the distance between points Xi and Xj , but other notions of proximity are available [3]. If consider a distance, it is a proximity that indicates how two objects Xi and Xj are dissimilar. MDS looks for coordinates of points Yi representing Xi in a lower-dimensional Euclidean space Rd by minimizing some stress function. Several realizations of MDS with different stress function have been proposed (see review in [3]), seeking less dependence of the resulting stress value on the absolute magnitude of the proximities (dissimilarities). However, their minimization is more complicated. We have proposed a new approach, Geometric MDS, in [1] and [2]. We have disclosed its new properties in [6]. The advantage of Geometric MDS is that it can use the simplest stress function, and there is no need for its normalization depending on the number of data points and the scale of proximities. The performance was compared with popular multidimensional scaling using majorization (SMACOF [7]). It is shown in [2] that the Geometric MDS does not depend
266
G. Dzemyda and M. Sabaliauskas
on the scales of dissimilarities and, therefore, may use a much simpler stress function like the raw stress function [8]: S(Y1 , . . . , Ym ) =
m m
(dij − d∗ij )2 ,
(1)
i=1 j=i+1
where d∗ij is the Euclidean distance between points Yi and Yj in a lower dimensional space: d 2 ∗ (yil − yjl ) . (2) dij = l=1
The optimization problem is to find minimum of the function S(·), defined by (1), and optimal coordinates of points Yi = (yi1 , . . . , yid ), i = 1, . . . , m: min
Y1 ,...,Ym ∈Rd
S(Y1 , . . . , Ym ).
(3)
The motivation of this research is that the recent realization of Geometric MDS needs to be faster. Therefore, it is necessary to find new useful capabilities of the method and implement them in further realizations. The novelty of this paper is that several new properties and capabilities of the Geometric MDS have been discovered theoretically and experimentally. We have found a possibility to minimize the MDS stress by using ideas of Geometric MDS when all the points Y1 , . . . , Ym change their coordinates at once at the same time during a single iteration of minimization of the stress. The obtained results allow us to extend the understanding of properties and ideas of Geometric MDS for the future development of a class of new effective algorithms for multidimensional data visualization and its dimensionality reduction.
2
Overview of Geometric MDS
Geometric MDS for minimization of the stress function (1) proposed in [1] and [2] is reviewed in this section briefly. Let’s have some initial configuration of points Y1 , . . . , Ym . The main idea of Geometric MDS focuses on optimizing the position of one chosen point (let it be Yj ) when the positions of remaining points Y1 , . . . , Yj−1 , Yj+1 , . . . , Ym are fixed. In case of one point position optimization, we minimize a part of the global stress S(·) in (3). This part is named by local stress function S ∗ (·) in [2]. S ∗ (·) depends on Yj , only: S ∗ (Yj ) =
m i=1 i=j
⎞2 d 2 ⎝dij − (yil − yjl ) ⎠ . ⎛
l=1
(4)
New Capabilities of the Geometric Multidimensional Scaling
267
Denote a new position of Yj by Yj∗ . Let Yj∗ be chosen so that Yj∗ =
m
1 Aij , m − 1 i=1
(5)
i=j
where the point Aij lies on the line between Yi and Yj , i = j, in a distance dij from Yi . Equation (5) is the main formulae of Geometric MDS, when defining transition from Yj to its new position Yj∗ . It is reasoned in [2]. Further eight propositions of this section are proved in [2]. They disclose the advantages of the transition defined by (5). Proposition 1. The gradient of local stress function S ∗ (·) is as follows: ⎛ ⎞ m ∗ dij − dij ⎜ ⎟ ∇S ∗ |Yj = ⎝2 (yik − yjk ) , k = 1, . . . , d⎠ . ∗ d ij i=1
(6)
i=j
Proposition 2. The coordinates of point Yj∗ are equal to: ⎛ ⎜ ⎝
1 m−1
m i=1 i=j
dij (yjk − yik ) + yik d∗ij
⎞ ⎟ , k = 1, . . . , d⎠ .
(7)
Proposition 3. The step direction from Yj to Yj∗ corresponds to the antigradient of the function S ∗ (·) at the point Yj : Yj∗ = Yj −
1 ∇S ∗ |Yj . 2(m − 1)
(8)
Proposition 4. Size of a step from Yj to Yj∗ is equal to ⎛ ⎞2 d m ∗ d − d 1 ij ⎜ ⎟ ij (yik − yjk )⎠ . ⎝ m − 1 d∗ k=1
i=1 i=j
ij
Proposition 5. Let Yj does not match to any local extreme point of the function S ∗ (·). If Yj∗ is chosen by (5), then a single step from Yj to Yj∗ reduces a local stress S ∗ (·): S ∗ (Yj∗ ) < S ∗ (Yj ).
268
G. Dzemyda and M. Sabaliauskas
Proposition 6. The value of the local stress function S ∗ (·) (4) will converge to a local minimum when repeating steps (8) and Yj ← Yj∗ . Proposition 7. Let Yj does not match to any local extreme point of the function S ∗ (·). Movement of any projected point by the geometric method reduces the stress (1) of MDS: if Yj∗ is chosen by (5), then the stress function S(·), defined by (1), decreases: S(Y1 , . . . , Yj−1 , Yj∗ , Yj+1 , . . . , Ym ) < S(Y1 , . . . , Yj−1 , Yj , Yj+1 , . . . , Ym ). Proposition 8. The local stress function S ∗ (·) defined by (4) could be multimodal for dimensionality 1 d < ∞. Two algorithms realizing the idea of Geometric MDS are presented in [2]. The simplest way to minimize the stress S(·) by Geometric MDS is a consecutive changing of positions of points Y1 , . . . , Ym many times. The stress is minimized, namely by a consequent changing the positions of separate d-dimensional projected points. The position of each selected point Yj is changed once consecutively.
3
Joint Recalculation of the Coordinates of all Points
Geometric MDS recalculates the coordinates of a single d-dimensional point Yj at each step. The result is a new point Yj∗ . The properties of such transition from Yj to Yj∗ are disclosed on Propositions proved in [2] and given above. The the raw stress function (1) decreases after movement Yj to Yj∗ (see Proposition 7). Let’s consider a generalized way to update the entire set of points Y = {Y1 , . . . , Ym } to Y ∗ = {Y1∗ , . . . , Ym∗ }. The basic equation of Geometric MDS (5) is used for defining the transition from Yj to its new position Yj∗ . However, in this case, all the points Y1 , . . . , Ym change their coordinates to Y1∗ , . . . , Ym∗ at once at the same time during a single iteration of minimization of the stress. Therefore, we perform a simultaneous change of the position of all the points Y1 , . . . , Ym of the projected space in the directions, defined by Geometric MDS strategy for separate point. The goal is to evaluate the stress function changes (1) after such joint recalculation of projected points Y1 , . . . , Ym . When we recalculate coordinates of one d-dimensional point (e.g. of Yj ) in Geometric MDS, we take into account coordinates of all m points Y1 , . . . , Ym . After recalculation, we will have an updated set Y = {Y1 , . . . , Ym }, where Yj will be different as compared with the previous content of this set. Afterwards, when we recalculate coordinates of another d-dimensional point (e.g. of Yi , i = j), we will use all Y1 , . . . , Ym including the updated point Yj . There is no such dependence among data for separate point recalculation in the case of joint recalculation of coordinates of Y1 , . . . , Ym .
New Capabilities of the Geometric Multidimensional Scaling
269
Natural question arises – whether the value of stress function (1) decreases after such joint recalculation of projected points Y1 , . . . , Ym . Moreover, it is expedient to explore the properties of stress function (1) not only in the case of joint transition from the set Y = {Y1 , . . . , Ym } to Y ∗ = {Y1∗ , . . . , Ym∗ } but in some intermediate cases. Let us define the term intermediate case. Each point Yj has its own equivalent Yj∗ defined by eq. (8). Let us introduce some parameter a whose values determine a gradual linear transition from Y = {Y1 , . . . , Ym } to Y ∗ = {Y1∗ , . . . , Ym∗ }. If a = o we have no transition. If a = 1, we have a transition from Y = {Y1 , . . . , Ym } to Y ∗ = {Y1∗ , . . . , Ym∗ }. If a = 0 or a = 1, we have a transition to some set Y + = {Y1+ , . . . , Ym+ }. We will define Yj+ more in detail. The idea comes from Propositions 1, 2, 3 + + + and 4. Let Yj+ = (yj1 , . . . , yjk , . . . , yjd ) be such: + yjk = yjk −
m a dij − d∗ij (yik − yjk ) m − 1 i=1 d∗ij
(9)
i=j
Proposition 9. The weight center of the data set Y + = {Y1+ , . . . , Ym+ } is independent of the parameter a and coincides with the weight centre of the data set Y = {Y1 , . . . , Ym }. Proof. Let us consider k-th component of the weight center of the data set + + + Y + = {Yj+ = (yj1 , . . . , yjk , . . . , yjd ), j = 1, ..., m}. It is equal to average m values of k-th component of the data set Y + : ⎛ ⎞ m m m ∗ 1 + 1 ⎜ a dij − dij ⎟ yjk = (yik − yjk )⎠ ⎝yjk − m j=1 m j=1 m − 1 i=1 d∗ij i=j
=
m m m dij − d∗ij a 1 yjk − (yik − yjk ) m j=1 m(m − 1) j=1 i=1 d∗ij i=j
⎞ ⎛ m m m m m ∗ ∗ d − d d − d 1 a ij ij ⎟ ⎜ ij ij = yjk − yik − yjk ⎠ . ⎝ ∗ m j=1 m(m − 1) j=1 i=1 d∗ij d ij j=1 i=1 i=j
i=j
Taking into account the symmetry of proximities and distances, i.e. dij = dji and d∗ij = d∗ji , we get that ⎛ ⎞ m m m m ∗ ∗ dij − dij dij − dij ⎜ ⎟ yik − yjk ⎠ = 0. ⎝ ∗ ∗ dij dij j=1 i=1 j=1 i=1 i=j
i=j
270
G. Dzemyda and M. Sabaliauskas
Then
and
m
m
1 + 1 yjk = yjk . m j=1 m j=1 m
m
1 + 1 Yj = Yj . m j=1 m j=1
(10)
Partial case of (10) is m
m
1 ∗ 1 Yj = Yj . m j=1 m j=1 Proposition 9 discloses very interesting and useful property of such the set Y + = {Y1+ , . . . , Ym+ }. By changing a, we will move from Y = {Y1 , . . . , Ym } to the new configuration Y + = {Y1+ , . . . , Ym+ } continuously without changing the weight center of data set. How does the value of the stress function S(Y1+ , . . . , Ym+ ) change depending on a? In fact we have single-variable function S(a), which value is equal to S(Y1 , . . . , Ym ) as a = 0, and it is equal to S(Y1∗ , . . . , Ym∗ ) as a = 1. Where is the minimum of S(a)? The experimental investigation is performed and presented in the next section seeking to get an answer.
4
Dependence of Stress S(a) on a: Search for Minimum
In this section, we examine the dependence S(a) on a experimentally. Two examples are presented indicating the existence of several local minima of S(·) and such ao , where S(a = ao ) < S(a = 0). The experiments were performed using the same m = 6 multidimensional data points X1 , . . . , X6 . The symmetric 6 × 6 matrix D = {dij , i, j = 1, . . . , 6} of proximities is as follows: ⎞ ⎛ 0 1.908 3.789 3.997 3.982 3.949 ⎜1.908 0 2.167 2.099 2.185 2.562⎟ ⎟ ⎜ ⎜3.789 2.167 0 1.302 2.501 3.627⎟ ⎟ D=⎜ ⎜3.997 2.099 1.302 0 1.278 2.512⎟ . ⎟ ⎜ ⎝3.982 2.185 2.501 1.278 0 1.269⎠ 3.949 2.562 3.627 2.512 1.269 0 Data sets Y = {Y1 , . . . , Ym }, from which the optimization starts (case a = 0), are presented in Fig. 2a and Fig. 2b, m = 6.
New Capabilities of the Geometric Multidimensional Scaling
271
The results are presented in Figs. 3 and 4. Figure 3 shows the transition from Y = {Y1 , . . . , Ym } to Y ∗ = {Y1∗ , . . . , Ym∗ } (case a = 1) by one step of the simultaneous changing the position of all the points Y = {Y1 , . . . , Ym } of the projected space in the directions and by steps, defined by Geometric MDS strategy for separate point. Figure 4 shows the dependence S(a) on various a for two different starting layouts of Y = {Y1 , . . . , Ym }.
(a) Starting layout 1
(b) Starting layout 2
Fig. 2. Data set Y from which the optimization starts
(a) Step from starting position 1
(b) Step from starting position 2
Fig. 3. Step from Y to Y ∗ : case a = 1
272
G. Dzemyda and M. Sabaliauskas
(a) Dependence S(a) from position 1
(b) Dependence S(a) from position 2
Fig. 4. Dependence S(a) on a
The curves in Figs. 4a and 4b indicate, that – function S(a) is multimodal, i.e. it may have several local minima, – the local minima are when a > 0, – transition from Y = {Y1 , . . . , Ym } to Y ∗ = {Y1∗ , . . . , Ym∗ } does not guarantee the reaching local minimum of S(a), – after transition from Y = {Y1 , . . . , Ym } to Y ∗ = {Y1∗ , . . . , Ym∗ } (case a = 1), the first local minimum is almost reached and is similar to the local minimum: in Fig. 4a, the local minimum is S(a = 0.964) = 45.365 while S(a = 1) = 45.464; in Fig. 4b, the local minimum is S(a = 1.050) = 28.771 while S(a = 1) = 28.966.
5
Conclusions
This paper allows us to extend the understanding of ideas and capabilities of Geometric MDS ([1,2,6]) for the future development of a class of new effective algorithms for multidimensional data visualization and its dimensionality reduction in general. Several new properties and capabilities of the Geometric MDS have been discovered theoretically and experimentally. We have found a possibility to minimize the MDS stress by using ideas of Geometric MDS when all the points Y1 , . . . , Ym change their coordinates at once at the same time during a single iteration of minimization of the stress. Several essential findings may be drawn from the results in this paper. The transition from some starting layout of lower-dimensional points by one step of the simultaneous changing of the position of all the points in the directions and by steps, defined by Geometric MDS strategy for separate points, decreases the raw stress value. The defined above gradual single-variable dependent linear
New Capabilities of the Geometric Multidimensional Scaling
273
transition from the starting layout serves as a basis to search for the layout that is optimal in a global sense. The peculiarity of the transition among layouts is that the weight center of the layouts of projected points remains the same. It is proved theoretically in this paper. Dynamic geometry program GeoGebra was used in performing the experiments and visual presentation of results in all the figures of this paper. It is a non-commercial and interactive software for the visual representation of algebra and geometry [9]. Acknowledgements. This research has received funding from the Research Council of Lithuania (LMTLT), agreement No S-MIP-20-19.
References 1. Dzemyda, G., Sabaliauskas, M.: A novel geometric approach to the problem of multidimensional scaling. In: Sergeyev, Y.D., Kvasov, D.E. (eds.) Numerical Computations: Theory and Algorithms, NUMTA 2019, vol. 11974 of Lecture Notes in Computer Science, pp. 354–361. Springer, Cham (2020) 2. Dzemyda, G., Sabaliauskas, M.: Geometric multidimensional scaling: A new approach for data dimensionality reduction. Applied Mathematics and Computation (2020) https://doi.org/10.1016/j.amc.2020.125561 (in press) 3. Dzemyda, G., Kurasova, O., Zilinskas, J.: Multidimensional Data Visualization: Methods and Applications, vol. 75 of Springer Optimization and its Applications. Springer, Cham (2013) 4. Borg, I., Groenen, P.J.F., Mair, P.: Applied Multidimensional Scaling and Unfolding. 2nd edition, Springer, Cham (2018) 5. Pragarauskait˙e, J., Dzemyda, G.: Visual decisions in the analysis of customers online shopping behavior. Nonlinear Anal. Model. Control 17(3), 355–368 (2012) 6. Sabaliauskas, M., Dzemyda, G.: Visual analysis of multidimensional Scaling Using GeoGebra. In: Dzitac, I., Dzitac, S., Filip, F., Kacprzyk, J., Manolescu, M.J., Oros, H. (eds.) Intelligent Methods in Computing, Communications and Control. ICCCC 2020, Advances in Intelligent Systems and Computing, vol. 1243, pp. 179–187. Springer, Cham (2021) 7. De Leeuw, J., Mair, P.: Multidimensional scaling using majorization: SMACOF in R. J. Stat. Softw. 31(3), 1–30 (2009) 8. Kruskal, J.B.: Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika 29(1), 1–27 (1964) 9. The GeoGebra Team (2020). https://www.geogebra.org/
Hope Amid a Pandemic: Is Psychological Distress Alleviating in South America While Coronavirus Is Still on Surge? Josimar E. Chire-Saire1(B) , Khalid Mahmood2 , Jimy Oblitas-Cruz3 , and Tanvir Ahmed4 1
2
4
Institute of Mathematics and Computer Science (ICMC), University of S˜ ao Paulo (USP), Sao Paulo, Brazil [email protected] Department of Information Technology, Uppsala University, Uppsala, Sweden [email protected] 3 Research and Development Department, Universidad Privada del Norte, Trujillo, Peru [email protected] Department of Computer Science, University of Central Florida, Florida, USA [email protected]
Abstract. As of November 17, 2020, the COVID-19 pandemic has over 55 million reported cases, causing more than 1.3 million deaths. To prevent this pandemic, some countries placed severe resection in the form of full-scale lockdown, while others took a moderate approach, e.g., mass testing, prohibiting large-scale public gatherings, restricting travels. South America adopted primarily the lockdown strategies for not having a sophisticated public-health infrastructure. Since the social interactions between people are primarily affected by the lockdown; psychological distress, e.g., anxiety, stress, fear are supposedly affecting the South American population in a severe way. This paper aims to explore the impact of lockdown over the psychological aspect of the people of all the Spanish speaking South American capitals. We have utilized infodemiology approach by employing large-scale Twitter data-set over 33 million feeds in order to understand people’s interaction over the months of this ongoing coronavirus pandemic. Our result is surprising: at the beginning of the pandemic, people demonstrated strong emotions (i.e. anxiety, worry, fear) that declined over time even though the actual pandemic is worsening by having more positive cases, and inflicting more deaths. Therefore, the result demonstrate that the South American population is adapting to this pandemic thus improving the overall psychological distress. Keywords: COVID-19 · Coronavirus · Infoveillance · Psychology Natural Language Processing · South America · Twitter · Google Trends · Social media analysis
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 274–283, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_27
·
Hope Amid a Pandemic
1
275
Introduction
The COVID-19 pandemic is one the of modern day’s calamity which is affecting the entire population of almost every country in the world. The disease caused by the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) a.k.a coronavirus was first identified in the Chinese city of Wuhan. Just over a month, the pandemic is recognized as Public Health Emergency of International Concern by the World Health Organization (WHO) [WHO 2020]. As of November 17, 2020 coronavirus infected 55 million people causing over 1.3 million deaths. The coronavirus demonstrated a fast growth of infections within and outside of China and soon spread nearby Asian countries, Europe and North America. This virus overwhelmed the sophisticated public health infrastructures of the developed countries, e.g. Italy, Spain, the United Kingdom, and the United States, consequently took many lives. Numerous public safety measures have been taken in order to prevent the fast spreading of the virus. Some countries utilized severe forms of measurement such as full-scale lockdown strategy while others utilized moderate form of action including mass testing, drive through testing, social distancing, limiting mass gathering, travel restriction, etc. These strategies by no means have stopped the on-going pandemic. South America first observed a case on February 26, 2020 in Brazil. By 26th June, it has more than 2 million confirmed cases consisting of 81,000 deaths, eventually became on the main epicenter of coronavirus crisis in the world. Recently, the World Health Organization has indicated that the epicenter of the contagion is in Latin America [Kirby 2020] and, therefore, it is not known how it could affect the course of the disease evolution or how mitigation measures would be carried out, but, in any case, a significant impact on mental health is estimated [Caycho-Rodr´ıguez et al. 2020]. In addition, it should be considered that several countries in the region have taken a number of very rigid and highly anticipated measures of strict quarantine, which also implies variability in terms of psychosocial involvement. Being a continent consisting of mostly developing countries, South America faces a unique challenge in dealing with the pandemic. Deprived health care system and limited testing capability1 are among the main reasons for not dealing with COVID-19 crisis successfully. The primary measurement of most of the South American capitals to deal with the coronavirus crisis is to limit the restriction of human movement, e.g. partial to full-scale lockdown, prohibiting large-scale public gathering, suspending schools. Since the lockdown is not sustainable and in fact, greatly impacts the livelihood of the people of developing countries, eventually it’s needed to be lifted over in order to bolster the economy. Apart from the health and economic crisis due to the COVID-19 outbreak, limiting the freedom of movement significantly impacts people’s psychological well-being. Amid of lockdown, people are more vulnerable towards the 1
Uruguay is one of the few countries in South American to limit the outbreak due to extensive testing.
276
J. E. Chire-Saire et al.
apprehension of the unforeseeable future. Therefore, understanding the psychological hardship of the South American population during this pandemic will be a great means to facilitate the judicious public health decisions during this ongoing pandemic which is likely to be prevalent for some time period. Infodemiology [Eysenbach 2011] is a new research field, with the objective of monitoring public health [Kamel Boulos et al. 2010] in order to support public policies based on electronic sources, i.e. Internet. In infodemiology, the data that is used in research is primarily open, and textual, having no structure and comes from different internet services, e.g. blogs, social networks, and websites. One of the key aspects of making rapid and effective public health actions is to concurrently observe the situation that demanded a fast response. Infoveillance approach could perform real-time monitoring of the situation by utilizing so-called social sensors [Wang et al. 2018], where individual or communities of people share their present status through different social media platforms (e.g. Facebook, Twitter). The data from social media could effectively be utilized in real-time in order to make adaptive public health measures to alleviate crisis that demonstrates natural unpredictability such as a pandemic or a disease outbreak. The infoveillance approach, tailored towards surveillance proposals has previously been applied to monitor H1N1 pandemic with data source from Twitter [Chew and Eysenbach 2010], a Dengue outbreak in Brazil using Social Sensors and Natural Language Processing[Chire Saire 2019], COVID-19 symptoms in Bogota, Colombia using Text Mining [Chire Saire and Navarro 2020]. In this paper, we are trying to understand how COVID-19 is impacting the psychological health of the population in Spanish speaking South American capitals. We are utilizing the infodemiologic approach to understand our research question by analyzing social media data sets. In summery, the contributions of the paper are as follows: – We explore how the population living in the Spanish speaking South American capitals facing the psychological distress during COVID-19 pandemic. We demonstrated that the Infoveillance approach of analyzing social media data, i.e. Twitter, can be effectively utilized to understand people’s mental status during the pandemic. Surprisingly, this has not been considered before. – We have collected a large number of Twitter feeds of all the Spanish speaking South American capitals which are cumulatively around 33 million of data points and applied proper data mining methods in order to effectively understand our research question (discussed in Sect. 2). – Our results described in Sect. 3.2 is counter-intuitive and surprisingly demonstrated that people’s interest in the pandemic is decreasing, inferring that the psychological distress is alleviating over the months while the actual pandemic is worsening (discussed in Sect. 4). This is the most important finding of our work. – In Sect. 3.3, we have verified our findings with another social media platform, i.e. Google Trend, and demonstrated that similar pattern can also be observed in Google Search.
Hope Amid a Pandemic
277
– Since the data from Twitter can be publicly access with Twitter API, our findings discussed in Sect. 2 and 3 are reproducible. Therefore, similar strategy can be adopted to explore infoveillance approach for understanding the psychological impact due to COVID-19 pandemic for other countries as well.
2
Data Process Analysis
The present analysis is inspired by the Cross Industry Standard Process for Data Mining(CRISP-DM) [Shearer 2000] steps, which consists of the frequent phases of Data Mining tasks. The workflow for data analysis is presented in Fig. 1 consists of five steps, which explained in the following subsections.
Step 1 Select scope and Social Network
Step 2 Find relevant terms
Step 3
Step 4
Step 5
Build query
Cleaning Data
Visualization
Fig. 1. Data process analysis
2.1
Selecting the Scope and Social Network
South America, being one of the most impacted continents of the pandemic, consists of 10 countries where official languages are either Spanish or Portuguese. Among those, Brazil is the only county where the official language is Portuguese. The rest of 9 countries in South America where Spanish is the official language are Argentina, Bolivia, Chile, Colombia, Ecuador, Paraguay, Per´ u, Uruguay, Venezuela. Table 1 presents the corresponding data related to Spanish spoken capitals of South America. Table 1. Cartographic and demographic information of the Spanish speaking South American countries Country
Capital
Area(km2 ) Population People/km2
Argentina Buenos Aires 2,792,600
44,938,712 16.092
Bolivia
La Paz
11,383,094 10.361
Chile
Santiago
Colombia Bogot´ a Ecuador
Quito
Paraguay Asunci´ on Per´ u
Lima
Uruguay
Montevideo
Venezuela Caracas
1,098,581 756,102
19,107,216 25.270
1,141,748
50,372,424 44.118
283,561
17,300,000 61.009
406,752
7,152,703 17.584
1,285,216
32,950,920 25.638
176,215
3,529,014 20.026
916,445
28,067,000 30.625
278
J. E. Chire-Saire et al.
The analysis is conducted over the capitals of each country because there is more concentration of population having affordable access to the internet compare to other cities. In this context, the urban population act as social sensors can potentially be utilized to inspect an event using interaction in the social media platform through posts/publications. Twitter is the choice of social network used in this work since the population of all the Spanish speaking countries uses it. Users on Twitter can send messages up to 240 characters long. It also facilities and access to data through it’s Application Programming Interface(API), which data scientists prefer for the ease of collecting the data. 2.2
Find the Relevant Terms to Search
Initially, we are interested in finding the twitter feeds related of coronavirus pandemic. In order to perform this, we have chosen terms that are related to coronavirus such as: coronavirus, covid19, etc. Since the users do not follow a specific pattern to a keyword when writing a post, where @coronavirus, #covid19, #covid 19 all similar to keyword COVID-19, we have considered variations of having special characters (i.e. @, #, –, ) in our search terms. Similar strategy has also been utilized in the work of [Chire Saire 2020]. 2.3
Build the Query to Collect Twitter Data
The query to extract tweets using API uses the next parameters: – – – – –
date: 01-04-2020 to 30-06-2020 terms: the chosen words mentioned in previous subsection geolocalization: the longitude and latitude of 9 Spanish speaking capitals language: Spanish radius: 50 km
2.4
Preprocessing of the Data
We have performed the following pre-processing steps in order to filter out unwanted information that could impact our analysis. – Cleaning urls using regular expressions, i.e. https?:\S+|http?:\S|[^A-Za-z0-9@#]+ , to clean urls or links – Eliminating non-alphabet characters – Converting all text to lowercase only letters – Deleting stopwords, e.g. articles ( el(the), la(the) ) , conjunctions 2.5
Visualization
When we are considering visualization, our paper depicts data from two different sources: – Twitter data: filtering with 3 specific keywords related to people’s psychological distress which are anxiety, fear and stress. – Google Trend’s data: for displaying what is trendy for a particular keyword in google search for our chosen countries.
Hope Amid a Pandemic
3 3.1
279
Results Dataset Description
The collected data has three main attributes: date, text, user name. The Table 2 provided overview of the collected 33 million Twitter feeds from 9 different capital cities. Table 2. Dataset description of 9 South American capital cities Country Argentina Bolivia
5,973,746
737,765
8.097
260,405
21,364
12.189
Chile
5,616,438
303,700
18.493
Colombia
3,192,229
310,816
10.270
Ecuador
1,887,503
100,911
18.704
Per´ u
4,312,931
284,621
15.153
Paraguay
2,981,195
154,301
19.321
Uruguay
1,402,344
188,328
7.446
7,905,287
297,685
26.556
33,532,078
2,399,491
15.137
Venezuela Total
3.2
Number of tweets Unique users Tweets/user
Evolution of Interest on Twitter Related to Psychological Distress
In this section, we are exploring how the psychological distress that related to our three terms anxity, fear, and stress evolve over four months of time from March to June during this pandemic for the 9 capital cities presented in Fig. 2. In these figures, we present the months from March until June in the x-axis, while in the y-axis, we have reported the corresponding frequency that is appeared for the corresponding psychological terms for that particular month. Argentina, Chile, Colombia, Ecuador, Peru, Paraguay, Uruguay and Venezuela present a decreasing number of posts from March to June for both terms: anxiety, fear and stress. An exception is evident in Bolivia, a peak has reached in April followed by a downward pattern for all three terms. For term, stress, Chile demonstrates it’s peak in April while Peru in May. One clear conclusion can be made that all South American countries have a decreasing trend for these three terms related to psychological distress.
280
J. E. Chire-Saire et al.
(a) Argentina
(b) Bolivia
(c) Chile
(d) Colombia
(e) Ecuador
(f) Peru
(g) Paraguay
(h) Uruguay
(i) Venezuela
Fig. 2. Evolution of terms related to physiological distress for the capitals of 9 Spanish speaking South American countries.
Hope Amid a Pandemic
281
Fig. 3. Google Trends of search term coronavirus (vertical lines separate months)
3.3
Validating the Analysis Using Google Trend
In order to validate our finding, we have considered the Google Trends to have a perspective about how South American people’s interest over coronavirus generally appeared in the Google Search. In Fig. 3, we have displayed the frequency of coronavirus search term appeared in Google Search during March until July, relevant to our 9 countries. Surprisingly, all the countries present a decreasing interest of term coronavirus which is similar to our previous analysis using Twitter data. In fact, it is quite surprising that the interest of the South American population are similar when it comes to the interest in coronavirus and almost all the country follows a similar trend.
4
Discussion
Are people in South America have been accustomed to the pandemic? In order to understand such phenomena, observe the daily new cases and deaths of COVID19 pandemic from March until July, 2020 in Fig. 4.
(a) Daily new cases
(b) Daily new deaths
Fig. 4. New cases and deaths of COVID-19 of 9 South American countries.2
2
The graphic is generated using a platform from www.ourworldindata.org.
282
J. E. Chire-Saire et al.
From Fig. 4, it is clearly evident that in most South American countries, both the total number of new cases and death are low at the beginning which increases over time. Interestingly, at the beginning of March/April, most countries impose a stricter restriction on people’s freedom of movement, such as forced lockdown, however, people’s psychological distress was high (demonstrated in two previously Subsects. 3.2 and 3.3) even having relatively low number of active cases and deaths. As the coronavirus pandemic exacerbates by having more positive cases and deaths, people’s apprehension about the coronavirus is surprisingly alleviated. This might properly explain why enforced confinement enables psychological distress upon the population. Currently, only two instruments are widely used to determine anxiety, stress and fear. These are: Fear of COVID-19 Scale [Ahorsu et al. 2020] and the Coronavirus Anxiety Scale [Lee 2020], which are the scales with more versions available in different languages. However, not all of these versions have evidence of validity and reliability, since they were developed very quickly and without taking into account the diversity of population, so a more agile technique, such as text mining, can be a faster point of identification of these problems generated in the COVID 19 pandemic. Text mining is a starting point in determining these problems listed as impacts on mental health [Zhang 2020]. With this information obtained, the institutions in each country can make public health decisions and direct to more specific studies. Currently, in the context of Latin America, only one measurement model is being used. The dynamism of the pandemic confronts us with the need for systematic, rapid and consistent epidemiological data and, to this end, such instruments as text mining, is presented as an excellent resource for decision-making in the field of mental health. However, it should be noted that, on their own, they cannot exclusively guide decision-making on mental health.
5
Conclusions
In this paper, we have employed large scale social-media data (i.e. Twitter) in order to understand the psychological distress due to COVID-19 pandemic of the Spanish speaking South American population. We have found that even though coronavirus pandemic is aggravating with having more active cases and deaths, people’s interaction in social media related to anxiety, worry, and fear of coronavirus is decreasing. We are speculating that people are becoming more accustomed to the pandemic compare to the beginning. This leads us to an optimistic conclusion that the people mental distress is alleviating reflecting the innate nature of humanity for overcoming tough-time.
6
Future Research
In future, by utilizing our infodemiology approach, we will try to understand impact of covid-19 that stimulates particular psychological affliction; e.g., effects of lockdown or self-confinement promoting certain psychological disease such as
Hope Amid a Pandemic
283
depression. We are also excited to find-out if the similar trait for South America that we found in this paper, also occurs in other countries including developed one. Another interesting direction could be to perform sentiment analysis utilizing machine learning approaches to understand how certain public health decisions taken during COVID-19 pandemic was impacted public opinion negatively, thereby promoting psychological distress.
References Ahorsu, D.K., Lin, C.-Y., Imani, V., Saffari, M., Griffiths, M.D., Pakpour, A.H.: The fear of COVID-19 scale: development and initial validation. Int. J. Ment. Health Addict. 1–9 (2020) Caycho-Rodr´ıguez, T., Barboza-Palomino, M., Ventura-Le´ on, J., Carbajal-Le´ on, C., No´e-Grijalva, M., Gallegos, M., Reyes-Bossio, M., Vivanco-Vidal, A.: Traducci´ on al espa˜ nol y validaci´ on de una medida breve de ansiedad por la COVID-19 en estudiantes de ciencias de la salud. Ansiedad y Estr´ es (2020) Chew, C., Eysenbach, G.: Pandemics in the age of twitter: content analysis of tweets during the 2009 h1n1 outbreak. PloS One, 5(11), e14118 (2010) Chire Saire, J.E.: Building intelligent indicators to detect dengue epidemics in Brazil using social networks. In: 2019 IEEE Colombian Conference on Applications in Computational Intelligence (ColCACI), pp. 1–5 (2019)Z Saire, J.E.C.: Infoveillance based on social sensors to analyze the impact of covid19 in South American population. medRxiv (2020) Saire, J.E.C., Navarro, R.C.: What is the people posting about symptoms related to coronavirus in Bogota, olombia? (2020) Eysenbach, G.: Infodemiology and infoveillance: tracking online health information and cyberbehavior for public health. Am. J. Prevent. Med. 40(5, 2), S154–S158. Cyberinfrastructure for Consumer Health (2011) Kamel Boulos, M.N., Sanfilippo, A.P., Corley, C.D., Wheeler, S.: Social web mining and exploitation for serious applications: technosocial predictive analytics and related technologies for public health, environmental and national security surveillance. Comput. Meth. Programs Biomed. 100(1), 16–23 (2010) Kirby, T.: South America prepares for the impact of COVID-19. Lancet Respir. Med. 8(6), 551–552 (2020) Lee, S.A.: Coronavirus anxiety scale: a brief mental health screener for COVID-19 related anxiety. Death Stud. 44(7), 393–401 (2020) Shearer, C.: The CRISP-DM model: the new blueprint for data mining. J. Data Warehous. 5(4), 13–22 (2000) Wang, D., Szymanski, B.K., Abdelzaher, T.F., Ji, H., Kaplan, L.M.: The age of social sensing. CoRR, abs/1801.09116 (2018) WHO: Who statement regarding cluster of pneumonia cases in wuhan, china, p. 9. WHO, Beijing (2020) Zhang, T.: Data mining can play a critical role in COVID-19 linked mental health studies. Asian J. Psychiatry 54, 102399 (2020)
An Entropic Approach to Assess People’s Awareness of the Health Risks Posed by Pesticides in Oenotourism Events Ana Crespo1 , Rui Lima2 , M. Rosário Martins3 , Jorge Ribeiro4 José Neves2,5 , and Henrique Vicente5,6(B)
,
1 Departamento de Fitotecnia, Escola de Ciências e Tecnologia,
Universidade de Évora, Évora, Portugal 2 Instituto Politécnico de Saúde do Norte, CESPU, Famalicão, Portugal
[email protected], [email protected] 3 Departamento de Química, Escola de Ciências e Tecnologia, Laboratório HERCULES,
Universidade de Évora, Évora, Portugal [email protected] 4 Instituto Politécnico de Viana do Castelo, Rua da Escola Industrial e Comercial de Nun’Álvares, Viana do Castelo, Portugal [email protected] 5 Centro Algoritmi, Universidade do Minho, Braga, Portugal 6 Departamento de Química, Escola de Ciências e Tecnologia, REQUIMTE/LAQV, Universidade de Évora, Évora, Portugal [email protected] Abstract. Wine production and vineyard work are seasonal activities as there is a period when no intervention is required and oenotourism may fill that void. On the other hand, given that grapevines are tied to this area of activity, it is of paramount importance to assess customer and staff awareness of the health risks of pesticide use, that emerge from people’s responses to specific questionnaires. Therefore, a workable problem-solving method is proposed which enables one to assess the level of awareness of people who are taking on the risks, evaluated in terms of an estimation of the individuals entropic state with respect to this particular issue. The analysis and development of such a model is based on a number of Logic Programming formalisms for Knowledge Representation and Reasoning, tha are consistent with an Artificial Neural Network approach to computing. The data collection process involved 173 participants. The proposed system presents an accuracy of about 90% and enables the diagnosis of risk awareness and the correspondent fragilities among customers and staff for a particular pesticide. Keywords: Oenotourism · Health risks · Pesticides · Entropy · Logic programming · Knowledge representation and reasoning · Artificial Neural Networks
1 Introduction Portugal has a strong wine tradition and the high quality of Portuguese wines is recognized worldwide. Several awards and prizes have been won in international competitions. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 284–296, 2021. https://doi.org/10.1007/978-3-030-72651-5_28
An Entropic Approach to Assess People’s Awareness of the Health Risks
285
Although researchers are trying to find ways to grow vines without the phytosanitary products (i.e. organic production), it is still very expensive these days, and not all agricultural explorations are associated with these practices. In addition, wine is a culture that is very susceptible to fungal diseases (mainly downy mildew and powdery mildew), and once these fungi tend to persist after treatment, it is not always possible to meet the maximum doses allowed for organic production. Pesticides are compounds that are primarily used in agricultural practices to protect, maintain, mitigate and repel the plague that could damage crops. They are xenobiotics as soon as they are exogenous compounds with high toxicity, not only against humans, but also against other animal spices that can be harmed by their presence in pastures, water and even on the air [1]. Minimizing the risks posed by the use of pesticides is therefore one of the greatest challenges where efforts must be made to contribute to environmental sustainability. Exposure to a pesticide can lead to acute toxicity (e.g. skin and/or eye irritation, death) in the short term and chronic toxicity (e.g. teratogenic effects, cancer, mutagenic and others). The effects of these xenobiotics on animals depend on many factors, such as their chemical structure, their mechanism of action and the toxicity they pose to living things. Not all organisms react in the same way. The response can be specific to a particular species or change from individual to individual [2]. Entry of xenobiotics into the organism affects not only its absorption in the bloodstream, but also its distribution in the body. This means that the same doses can be more or less toxic depending on the route of absorption. Pesticides can enter the human system mainly through the skin (skin exposure), the mouth (oral exposure) and the lounges (inhalation exposure). Once absorbed, the pesticide can be distributed to the various constituents and tissues of the human being. Those that are absorbed through oral exposure are then absorbed directly into the bloodstream in the gastrointestinal tract and reach the liver if metabolized immediately after absorption, while others may suffer from a different type of distribution in the body [3]. Those that are absorbed through skin or inhalation can be transported to the liver and directly bio transformed or excreted via the urinary system without hepatic biotransformation [4]. Given the exponential growth of tourism in Portugal, particularly in the Alentejo region, and given that most enotourism companies have vines attached, it is important to assess the Degree-of-Awareness (DoA) of the customers and workers about the health risks of using pesticides. Undeniably, the aim of this study was to evaluate the DoA about the health risks associated to the use of these type of chemicals. This paper develops through the bits, i.e., following the introduction, the fundamentals adopted in the study are set, namely the notion of Entropy and the use of Logic Programming (LP) for Knowledge Representation and Reasoning (KRR) [5–9], complemented with a view of the enforced methods. Next, a case study of data collection and processing related to the DoA of oenotourism users on the health risks associated with the use of pesticides will be presented and assessed based on the answers of people to a specific questionnaire that, once rewritten in terms of set of logical programs stands for the universe of discourse of the environment [6–9]. Then conclusions are drawn and future work is outlined.
286
A. Crespo et al.
2 Background 2.1 Thermodynamics and Knowledge Representation and Reasoning Aiming to explain the basic rules of the proposed approach, the First and Second Law of Thermodynamics were considered, attending that one’s system moves from state to state over time. The main principles of thermodynamics can be summarized in one sentence, viz. The total energy content of the universe is constant, and the total entropy increases steadily. Thus, energy cannot be generated or destroyed, but it can be converted from one form to another. Every time it is transformed, there is a penalty, i.e., a loss of available energy for future work. The former one stands for the Energy Saving Law, that refers that the total energy of an isolated system is constant, i.e., cannot change over time. This means that energy can be converted but cannot be generated or destroyed. The latter deals with Entropy, a property that quantifies the orderly state of a system and its evolution. These characteristics fit the proposed vision of KRR practices, which must be understood as a process of energy degradation. Indeed, it is believed that the universe of discourse is at a given entropic state, the energy of which can be decomposed and used in sense of degradation, but never used in the sense of destruction. It may be introduced as, viz. • Exergy, sometimes called available energy or more precisely available work, is the part of the energy which can be arbitrarily used after a transfer operation or, in other words, denoting the entropic state of the universe of discourse. In Fig. 1 it is given by the dark colored areas; • Vagueness, i.e., the corresponding energy values that may or may not have been transferred and consumed. In Fig. 1 are given by the gray colored areas; and • Anergy, that stands for an energetic potential that was not yet transferred and consumed, being therefore available, i.e., all of energy that is not exergy. In Fig. 1 it is given by the white colored areas [5, 7–9]. These terms refer to all possible processes as pure energy transfer and consumption practices. 2.2 The Logic Programming Framework Several advances to KRR are based on the LP architype, namely in Model and Proof Theory. The problem-solving method used in this work is based on Proof Theory and is applied to logic programs that use an extension of the LP language [6] in the form of a finite summative of clauses as depicted in program 1, viz.
An Entropic Approach to Assess People’s Awareness of the Health Risks
287
The first clause denotes predicate’s closure, “,” designates “logical and”, while “?” is a domain atom denoting “falsity”, the pi , qj , and p are classical ground literals, i.e., either positive atoms or atoms preceded by the classical negation sign ¬. Indeed, ¬ stands for a strong declaration and speaks for itself, while not denotes negation-by-failure, i.e., a failure in proving a certain statement since it was not declared in an explicit way. According to this way of thinking, a set of abducibles are present in every program [6]. In this work are given in the form of exceptions to the extensions of the predicates that make the program, i.e., clauses of the form, viz.
that denote data, information or knowledge that cannot be ruled out. On the other hand, clauses of the type, viz.
are invariants that make it possible to specify the context under which the universe of discourse should be understood [6].
3 Methods The place of study was an oenotourism facility, named Monte da Esperança (39º943.906"N 7º11 27.434"W), located at Esperança, on Portalegre district (Portugal). It is inserted on São Mamede Natural Park and includes an own production winery, and 17 ha of vineyard, producing 7 different grape varieties. One hundred and seventy-three (173) participants aged between 20 and 78 years of age (with an average of 51 ± 23 years old) were enrolled in this study. The gender distribution was 41.3% and 58.7% for male and female, respectively. The participants are workers of oenotourism enterprise and costumers who attended the enterprise between September 2018 and August 2019. The questionnaire request was selected to collect the data. The questionnaire consists of two parts, while the first part contains the general questions about age, gender, academic qualifications, place of residence, context
288
A. Crespo et al.
and frequency of exposure to pesticides. The second comprises statements about pesticide health risks, including the skills with which the information is found, understood, assessed and applied. The Action Research Methodology for problem solving is used in this work. It involves the conducting of systematic enquiries of the of those that belong to the universe of discourse to improve their own practices, which in turn can enhance their working environment and the working environments of those who are also part of it, i.e., customers and staff. The purpose of undertaking action research is to bring about change in specific contexts. Action research’s strength lies in its focus on generating solutions to practical problems and its ability to empower practitioners, by getting them to engage with research and the subsequent development or implementation activities.
4 A Thermodynamics View of Data Processing The participants were requested to tick the option(s) that best reflects their opinion about each question. Participants who select more than one option are also asked to indicate the evolution trend of their answer, i.e., increasing trend (Very Difficult → Very Easy) or the contrary (Very Easy → Very Difficult). Thus, the answer options were limited to the following scale, viz. Very Easy (4), Easy (3), Difficult (2), Very Difficult (1), Difficult (2), Easy (3), Very Easy (4) The issues under consideration were divided into four groups, namely (Find Information – Four Items (depicted as Find – 4 predicate), Understand Information – Four Items (depicted as Understand – 4 predicate), Assess Information – Four Items (depicted as Assess – 4 predicate), and Apply Information – Four Items (depicted as Apply – 4) predicate. The former one comprehends the questions, viz. Q1 – How easy would you say it is to find information on what to do in case of accidental contact with pesticides; Q2 – How easy would you say it is to find information on what to do in case of pesticide poisoning; Q3 – How easy would you say it is to find information on symptoms of pesticide poisoning; and Q4 – How easy would you say it is to find information on how to ask for medical help in an emergency. The understand information group includes the questions, viz. Q5 – How easy would you say it is to understand information on the pesticide labels; Q6 – How easy would you say it is to understand information about health risks of using pesticides; Q7 – How easy would you say it is to understand information on what to do in case of pesticide poisoning; and
An Entropic Approach to Assess People’s Awareness of the Health Risks
289
Q8 – How easy would you say it is to understand information provided by health professionals. The assess information group embraces the questions, viz. Q9 – How easy would you say it is to assess information conveyed by the media on the impact of pesticide use; Q10 – How easy would you say it is to assess the usefulness of the information on pesticide labels; Q11 – How easy would you say it is to assess the need to seek a medical emergency post; and Q12 – How easy would you say it is to assess whether the indications given by health professionals are adequate. Finally, the apply information group comprises the questions, viz. Q13 – How easy would you say it is to apply the information when deciding to use/not use pesticides; Q14 – How easy would you say it is to apply the information to ensure the proper dosage; Q15 – How easy would you say it is to apply the information to proceed correctly in case of poisoning by pesticides; and Q16 – How easy would you say it is to apply the information to correctly describe an eventual accident to health professionals. To illustrate the conversion process of qualitative information into quantitative one, the complete calculations for Find – 4 group are presented. Table 1 displays the answers of participant # 1 to the questions of these group (i.e., questions Q1 to Q4). Table 1. The participant # 1 s answers to Find– 4 questionnaire.
Questions Q1 Q2 Q3 Q4
(4)
(3)
(2)
(1)
Scale (2) ×
(3) ×
(4)
Vagueness ×
× ×
×
Leading to
Fig. 1
Leading to
The entries in Table 1 are to be read from left to right, from very easy (4) to very difficult (1), denoting an increasing entropy, or from very difficult (1) to very easy (4), which denotes a decreasing entropy). Once the input for Q1 matches (2) → (3) there is an improvement tendency of the system’s universe of discourse. Otherwise, the input for
290
A. Crespo et al.
Q4 matches (4) → (3). The system’s universe of discourse tends to deteriorate, i.e., the input for Q1 states a tendency to the degradation of the system’s universe of discourse. For Q2 no options were pointed out, which indicates a vague situation, i.e., the value of the system entropic state is unknown, although ranging in the interval 0…1. Finally, for Q3 the answer was difficult (2), a fact that it is clear and needs no further explanation. Figure 1 describes the responses presented in Table 1 in terms of the different forms of energy, i.e., exergy, vagueness and anergy. The markers on the axis correspond to any of the possible scale options, which may be used from bottom → top (from (4) → (1)), indicating that the entropic state of the system’s universe of discourse decreases as the entropy increases, or used from top → bottom (from (1) → (4)), indicating that the entropic states of the system’s universe of discourse increases as entropy decreases. Table 2 gives an evaluation of the entropic states of the system’s universe of discourse for the Best and Worst-case scenarios (BCS/WCS).
Leading to
Table 2
Leading to
Fig. 1. A graphical endeavor of participant # 1 perception of the influence of each question of the Find – 4 questionnaire on the entropic state of the universe of discourse.
An Entropic Approach to Assess People’s Awareness of the Health Risks
291
Table 2. An evaluation of the entropic state of the universe of discourse for the Best and Worst-case scenarios for Find – 4 questionnaire regarding participant # 1.
292
A. Crespo et al.
where the minus sign (−) indicates that the entropy has a tendency to decrease. The data presented in Table 3 can now be structured in terms of the extent of predicate find – 4 in the form, viz (Table 3 and Program 2). Table 3. The extent of find – 4 s predicate gotten from a participant # 1 s answers of Find – 4 questionnaire.
EX BCS 0.27
VA BCS 0
AN BCS 0.73
Leading to
DoA BCS 0.93
QoI BCS 0.73
EX WCS 0.27
Program 2
VA WCS 0.37
AN WCS 0.36
DoA WCS 0.77
QoI WCS 0.36
Leading to
The assessment of DoA and QoI for the find – 4 s scope is now carried out in the form, viz. • DoA is figured out using DoA = 1 − ES 2 (Fig. 2), where ES stands for the exergy’s that may have been consumed, a value that ranges in the interval 0…1. In the Best-case scenario, ES = exergy, while in the Worst-case scenario, ES = exergy + vagueness). DoABCS = 1 − 0.272 = 0.93; DoAWCS = 1 − (0.27 + 0.37)2 = 0.77 • QoI is evaluated in the form, QoI = 1 − ES/Intervallength(= 1), viz. QoIBCS = 1 − 0.27 = 0.73; QoIWCS = 1 − (0.27 + 0.37) = 0.36 Table 4 displays the answers of participant #1 to Understand – 4, Assess – 4, and Apply – 4 questionnaires. The computational process for each group is similar to that previously presented for the Find – 4 one. Table 5, in turn, presents the extensions of the respective predicates.
An Entropic Approach to Assess People’s Awareness of the Health Risks
293
Fig. 2. DoA evaluation.
Table 4. Participant # 1 answers to the Understand – 4, Assess – 4, and Apply – 4 questionnaires.
Group Understand – 4
Assess – 4
Apply – 4
Questions Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Leading to
(4)
(3)
(2) ×
(1) × ×
Scale (2) (3)
(4)
vagueness ×
× × ×
× × × ×
× × × Table 5
Leading to
where 0.39 = (0.27 + 0.53 + 0.34 + 0.41)/4. The procedures for the remain parameters are similar.
5 Awareness of Health Risks Assessment – A Logic Programming Approach Logic Programming (LP) is a problem-solving methodology that involve expressing problems and their solutions in a way that a computer can execute. In this section a mathematical-logical program is presented that considers the awareness of the oenotourism players with respect to health risks due the use of pesticides, with respect to the Best-case scenario (Program 3). This framework provides the basis for a symbolic
294
A. Crespo et al.
Table 5. The participant # 1 answers to the Find – 4, Understand – 4, Assess – 4 and Apply – 4 questionnaires for the Best and Worst-case scenarios.
EX BCS Find – 4 0.27 Understand – 4 0.53 0.34 Assess – 4 Apply – 4 0.41 Global 0.39 Questionnaires
VA BCS 0 0 0 0 0
Leading to
AN BCS 0.73 0.47 0.66 0.59 0.61
DoA BCS 0.93 0.85 0.94 0.91 0.91
QoI EX VA BCS WCS WCS 0.73 0.27 0.37 0.47 0.53 0.36 0.66 0.34 0.36 0.59 0.41 0 0.61 0.39 0.27
Program 3
AN WCS 0.36 0.11 0.30 0.59 0.34
DoA WCS 0.77 0.46 0.71 0.91 0.71
QoI WCS 0.36 0.11 0.30 0.59 0.39
Leading to
assessment of the DoA of oenotourism players, plus a measure of its sustainability (QoI), i.e., a set of truth values that range in the interval 0…1 [7–9].
It is now possible to generate the data sets that will allow one to train an ANN (Fig. 3) [9, 10], viz.
An Entropic Approach to Assess People’s Awareness of the Health Risks
295
• The input is made in the form of the employees/customers’ opinions on the Find Information (scope of predicate find – 4), Understanding Information (scope of predicate understand – 4), Assess Information (scope of predicate assess – 4), and Apply Information (scope of predicate apply – 4); and • The output relating to an assessment of the DoA evaluation and its sustainability (QoI), truth values that range in the interval 0… 1.
0.91
0.61 DoA Sustainability (QoI)
(DoA) Output Layer Hidden Layer
Bias
Input Layer
Bias
find - 4
0.59
0.91
0.59
0
0.41
0.73
0.93
0.73
0
0.27
Pre-processing Layer
apply - 4
Fig. 3. A creative view of the ANN topology for DoA assessment and a measure of its Sustainability (QoI) for the Best-case scenario.
To guarantee the significance of the results attained, 20 experiments were performed in all tests. In each simulation, the database was randomly split into two mutually exclusive partitions, i.e., the training set, with 2/3 of the data, used to build-up the model, and the test set, with the remaining cases to evaluate its performance. The model accuracy was 91.4% for the training set (106 correctly classified in 116) and 87.7% for test set (50 correctly classified in 57). In the classification process, high denotes DoA higher than 0.85, medium stands for DoA ranging in the interval 0.5…0.85, and low were assigned to DoA lesser than 0.5.
6 Conclusions and Future Work In this work, the assessment of the Degree-of-Awareness (DoA) of oenotourism enterprises players about the health risks of using pesticides was performed based on competencies like find, understand, assess, and apply information. The proposed model allows to predict not only the DoA but it also gives an opportunity to be proactive and increase the awareness of health risks that arises from the use of pesticides by identifying the
296
A. Crespo et al.
issues that the workers/costumers revel gaps, i.e., the points that negatively impact the workers/costumers’ entropic state. Future work will deal with the emotional state of customers and staff and its impact on the decision-making process about potential pesticide risks and their assessment. Acknowledgments. This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.
References 1. Aktar, W., Sengupta, D., Chowdhury, A.: Impact of pesticides use in agriculture: their benefits and hazards. Interdiscip. Toxicol. 2, 1–2 (2009) 2. Whitford, F., Fuhremann, T., Rao, K.S., Arce, G., Klaunig, J.E.: Pesticide Toxicology – Evaluating Safety and Risk. Purdue University Cooperative Extension Service (2003). https:// www.extension.purdue.edu/extmedia/PPP/PPP-40.pdf. Accessed 10 Nov 2020 3. Roberts, T., Hutson, D.: Metabolic Pathways of Agrochemicals – Part 2: Insecticides and Fungicides. The Royal Society of Chemistry, Cambridge (1999) 4. Krieger, R.: Introduction to Biotransformation. In: Hodgson, E. (ed.) Pesticide Biotransformation and Disposition, 3rd edn., pp. 53–72. Academic Press, New York (2012) 5. Wenterodt, T., Herwig, H.: The entropic potential concept: a new way to look at energy transfer operations. Entropy 16, 2071–2084 (2014) 6. Neves, J.: A logic interpreter to handle time and negation in logic databases. In: Muller, R., Pottmyer, J. (eds.) Proceedings of the 1984 Annual Conference of the ACM on the 5th Generation Challenge, pp. 50–54. ACM, New York (1984) 7. Neves, J., Maia, N., Marreiros, G., Neves, M., Fernandes, A., Ribeiro, J., Araújo, I., Araújo, N., Ávidos, L., Ferraz, F., Capita, A., Lori, N., Alves, V., Vicente, H.: Entropy and organizational performance. In: Pérez García, H., Sánchez González, L., Castejón Limas, M., Quintián Pardo, H., Corchado Rodríguez, E. (eds.) Hybrid Artificial Intelligent Systems. Lecture Notes in Computer Science, vol. 11734, pp. 206–217. Springer, Cham (2019) 8. Figueiredo, M., Fernandes, A., Ribeiro, J., Neves, J., Dias, A., Vicente, H.: An assessment of students’ satisfaction in higher education. In: Vittorini, P., Di Mascio, T., Tarantino, L., Temperini, M., Gennari, R., De la Prieta, F. (eds.) Methodologies and Intelligent Systems for Technology Enhanced Learning. Advances in Intelligent Systems and Computing, vol. 1241, pp. 147–161. Springer, Cham (2020) 9. Fernandes, A., Figueiredo, M., Dias, A., Ribeiro, J., Neves, J., Vicente, H.: A case-based approach to assess employees’ satisfaction with work guidelines in times of the pandemic. In: Florez, H., Misra, S. (eds.) Applied Informatics. Communications in Computer and Information Science, vol. 1277, pp. 183–196. Springer, Cham (2020) 10. Neves, J., Maia, N., Marreiros, G., Neves, M., Fernandes, A., Ribeiro, J., Araújo, I., Araújo, N., Ávidos, L., Ferraz, F., Capita, A., Lori, N., Alves, V., Vicente, H.: Employees balance and stability as key points in organizational performance. Log. J. IGPL (2021). https://doi.org/10. 1093/jigpal/jzab010
Knowledge Management Strategies Through Educational Digital Platforms During Periods of Social Confinement Romel Ramón González-Díaz1(B) , Ángel Eduardo Acevedo-Duque2 Katia Ninozca Flores-Ledesma1 , Katiusca Cruz-Ayala1 , and Santos Lucio Guanilo Gomez3
,
1 Centro Internacional de Investigación y Desarrollo - CIID, 230001 Monteria, Colombia
[email protected] 2 Universidad Autónoma de Chile, 1030000 Santiago de Chile, Chile
[email protected] 3 Universidad Nacional Jorge Basadre Grohmann, 23002 Tacna, Peru
[email protected]
Abstract. During COVID-19 the world experienced an unprecedented educational crisis, where conventional systems based on presence were fractured by long periods of confinement. In order to face this situation, the benefits of the educational digital platforms were used. In this sense, this research aimed to analyze knowledge management strategies through the use of educational digital platforms during periods of social confinement in Mexico, Colombia and Peru. A survey composed of 22 items with a Likert scale was applied to 396 teachers in the period from 01/09/2020 to 15/11/2020. It was concluded that schools had difficulties in connecting to the Internet, lack of electronic devices, lack of information technology skills for the use of educational digital platforms. Keywords: Knowledge management · Digital platforms · Social confinement · Technological skills
1 Introduction COVID-19 has plunged the world into a devastating and possibly unprecedented health crisis in recent decades. In 1918 the Spanish flu infected one third of the world’s population in two years with a mortality rate of 10% [1]. The 2003 Severe Acute Respiratory Syndrome (SARS), the 2009 H1N1 influenza and the 2014 Ebola in West Africa. COVID-19 (SARS-CoV-2), by mid-2020, has claimed over 603,000 deaths in seven months, keeping the world population on alert. Unlike other diseases, COVID-19 has one of the highest reproductive rates (R0), ranging from 1.5 to 3.5. Governments are trying to reduce this rate to values below 1, in order to make social and economic life, and thus, their health systems can attend to different emergencies [2]. In order to minimize this SARS-CoV-2 reproduction rate, Latin American governments have bet on social isolation measures in order to mitigate the wave of contagion that © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 297–303, 2021. https://doi.org/10.1007/978-3-030-72651-5_29
298
R. R. González-Díaz et al.
threatens public health systems. These actions have not only limited human interaction, but have also fractured the face-to-face behavior of the learning process in educational institutions. In Colombia, the national government has designed measures to mitigate this educational crisis, including: promotion of teleworking, use of interactive platforms, delivery of computer equipment and other events that encourage the interaction of users with materials and tools that generate an immersive and innovative experience [3]. In this sense, students and teachers provide a virtual meeting through spaces that aim to improve the development of communication skills, technology and training, allowing the student’s accessibility from anywhere, generating discussions of issues beyond the class schedule. However, the scientific literature Hernández García de Velazco, et al. [4], Vargas Herrera and Moya Marchant [5] have demonstrated the existence of resistance to change in the handling of interactive platforms, which awakens interest in the study of the use of interactive platforms and the process of knowledge management. Studying the use of interactive platforms, allows the academic community to determine the failures in the process of change in the modalities of studies, which must meet educational quality criteria. This situation commits educational entities to comply with efficiency, openness and flexibility as applied characteristics in virtual education. ICTs are a “sine qua non” component of a new paradigm of society: the information, knowledge, and learning society [6, 7]. Educational institutions have taken important steps towards digitalization by making a great effort to improve the telecommunications infrastructure and electronic administration [8–10]. Nowadays it is almost inconceivable to teach without technology, or without reference to methodologies such as gamification, inverted classrooms or bringing your own device [11–13], which can contribute to create new scenarios that facilitate and promote different processes within the classroom, and that connect and help transfer knowledge to the outside world [1, 14, 15]. In summary, transforming conventional teaching into new models mediated by ICT (blended learning or mobile learning) has a double implication for teachers: on the one hand, it requires practical knowledge to create active student communities [16, 17]. This task is complicated because it involves a series of skills that teachers do not possess, either because they do not know them well or because they are not adequately trained. In fact, among the dimensions of digital teacher competence, teachers have received a lower score in resource creation and problem solving, which includes selecting the best tools or resources available for a given purpose. Moreover, teachers often have difficulties in agreeing on basic aspects such as the cognitive demands of the activities and the items on the test [11, 12], as well as in the formulation of competence indicators [15] when applying competency-based assessment. Nevertheless, teachers face relevant challenges when selecting resources to support the development of competencies. Likewise, in order to achieve knowledge management strategies during periods of social confinement and educational digital platforms, a questionnaire is applied to teachers during periods of social confinement. Later, the data is analyzed and interpreted to determine the behavior of the object of study.
Knowledge Management Strategies Through Educational Digital Platforms
299
2 Materials and Methods For this research, a survey composed of 22 items with a Likert type scale was applied, validated in the opinion of 3 experts with a Cronbach’s Alpha coefficient of 0.96 (Excellent). A random sample of 396 teachers collected in international events was estimated between the period from 01/09/2020 to 15/11/2020. The data were analyzed by means of descriptive statistics to know the behavior of the variables under study. Then the chi-square statistic was applied in order to test the following hypotheses: – H0 = Null hypothesis: Knowledge management is not related to the use of educational digital platforms in Latin America. – H1 = Alternative Hypothesis: Knowledge management is related to the use of educational digital platforms in Latin America. – Asymptotic significance (bilateral): 0.005. If H1 is accepted, it is considered a contingency coefficient to measure the intensity of the relationship between variables.
3 Analysis and Discussion of the Results Once the statistical data has been processed, the results are presented and discussed by study variables. First, the behavior of knowledge management in educational institutions in Mexico, Colombia and Peru is described. Then, the use of educational digital platforms is observed in order to analyze knowledge management strategies through the use of educational digital platforms during periods of social confinement in Mexico, Colombia and Peru.
Fig. 1. Use of educational digital platforms in Latin America
300
R. R. González-Díaz et al.
Figure 1 shows that 93.67% of teachers consider as excellent or good the use of educational digital platforms in the development of significant learning, a situation that coincides with the approaches of Spain Bone and Vigueras Moreno [18] and Souza, et al. [19], who coincide in seeing the technological gap that exists in teachers and students, who express their denial of the use of educational platforms. Regarding Knowledge Management, the following results can be observed (see Fig. 2).
Fig. 2. Knowledge management behavior in Latin America
Figure 2 shows that 98.48% of the teachers surveyed consider that knowledge management in educational institutions is between excellent and good. These results are consistent with the statements of Guzmán and Arrieta [20], Acevedo-Correa, et al. [21], who agree on the existence of a significant contribution in the transfer of knowledge for the generation of significant learning. Finally, it can be observed that the variables described present a significant relationship according to the results by the Pearson Chi-square test statistic: 0.000 < 0.05. Decision: The null hypothesis is rejected and the researcher’s hypothesis “there is a relationship between knowledge management and the use of educational digital platforms” is accepted. This relationship presented a contingency coefficient with an approximate significance of: 0.763. Finally, Fig. 3 shows the relationship between the Latin American countries studied, knowledge management and the use of the educational digital platform. In Mexico the use of educational digital platforms was concentrated in Good (51.6%) and Excellent (42.30%) for a total of 93.9% of teachers using educational platforms efficiently. However, these results do not agree with those presented by Ramirez-Vera, et al. [22] and Fernandez and Romero [23] who in their studies make known the challenges in terms of technological teaching skills in Mexico. This situation is not different in Peru, whose results show a teacher with Good (59.8%) and Excellent (33.3%) for a total of 93.1% of those surveyed, who claim to have technological skills. On the contrary, studies presented by Porras Jaramillo and Quispe
Knowledge Management Strategies Through Educational Digital Platforms
301
Fig. 3. Grouped Knowledge Management Box Diagram by Latin American Country for Educational Digital Platforms
Huari [24] reveal the deep technological gaps between those who teach. In Colombia, the situation is similar because the results reflect Good (58.7%) and Excellent (34.9%) for a total of 93.6%. In contrast, the reflections of Benavides Albesiano [25] and Marín, et al. [26] show that there is still a lack of orientation and training processes for teachers at different levels to adapt to the change in learning methodology.
4 Conclusion In virtue of the above, and given the level of significance with knowledge management, it can be concluded that despite the fact that educational institutions and governments have made significant efforts and teachers consider themselves to have technological competencies to face the new challenges of educational modernity. There are still wide training gaps in the use of educational platforms. Likewise, there is resistance to this change and a challenge for schools to reconcile rigid positions in accepting new forms of learning experiences, revealing the imminent need generated by the educational crisis brought about by Covid-19.
5 Future Work Therefore, the findings of this paper provide a baseline against which the future development of this line of research on educational digital platforms and knowledge management in educational institutions can be evaluated. In addition, it is hoped that these findings can contribute to guide future research on the relationship between the use of digital educational platforms and knowledge management.
302
R. R. González-Díaz et al.
References 1. Hussain, F.N., Al-Mannai, R., Agouni, A.: An emergency switch to distance learning in response to the COVID-19 pandemic: experience from an internationally accredited undergraduate pharmacy program at Qatar University. Med. Sci. Educ. 30(4), 1393–1397 (2020). https://doi.org/10.1007/s40670-020-01079-9 2. Elgueta, M.F.: Desafíos de la educación jurídica latinoamericana en tiempos de pandemia. Rev. Pedagog. Univ. Didact. Derecho, Review 7(1), 1–5 (2020). https://doi.org/10.5354/07195885.2020.57763 3. McFadden, P., Russ, E., Blakeman, P., Kirwin, G., Anand, J., Lähteinen, S., Baugerud, G.A., Tham, P.: COVID-19 impact on social work admissions and education in seven international universities. Soc. Work Educ. 39(8), 1154–1163 (2020). https://doi.org/10.1080/02615479. 2020.1829582 4. García, J.J.H., de Velazco, R., Ripoll, R., Hernandez, A.C.C.: Relevance and social responsibility of sustainable university organizations: analysis from the perspective of endogenous capacities. Entrep. Sustain. Issues 7(4), 2967–2977 (2020). https://doi.org/10.9770/jesi.2020. 7.4(26) 5. Herrera, F.V., Marchant, L.M.: Analyzing the concept of spiritual development based on institutional educational projects from schools located in Valparaíso, Chile. Relig. Educ. 115(2), 201–214 (2020). https://doi.org/10.1080/00344087.2019.1677986 6. Soboleva, E.V., Chirkina, S.E., Kalugina, O.A., Mikhail, Y., Shvetsov, V.A., Kazinets, E.B., Pokaninova: Didactic potential of using mobile technologies in the development of mathematical thinking. EURASIA J. Math. Sci. Technol. Educ. 16(5), 1–20 (2020). https://doi.org/ 10.29333/ejmste/118214 7. Safsouf, Y., Mansouri, K., Poirier, F.: An analysis to understand the online learners’ success in public higher education in Morocco. J. Inf. Technol. Educ. Res. 19, 87–112 (2020) 8. Ruipérez-Valiente, J.A., Martin, S., Reich, J., Castro, M.: The unMOOCing process: extending the impact of MOOC educational resources as OERs. Sustainability 12(18), 7346 (2020). https://doi.org/10.3390/su12187346 9. Robles, D., Christian, G., Quintero, M.: Intelligent system for interactive teaching through videogames. Sustainability 12(9), 3573 (2020). https://doi.org/10.3390/su12093573 10. Rakic, S., Tasic, N., Marjanovic, U., Softic, S., Lüftenegger, E., Turcin, I.: Student performance on an e-learning platform: mixed method approach. Int. J. Emerg. Technol. Learn. (iJET) 15(02), 187 (2020). https://doi.org/10.3991/ijet.v15i02.11646 11. Mulenga, E.M., Marbán, J.M.: Prospective teachers’ online learning mathematics activities in the age of COVID-19: a cluster analysis approach. Eurasia J. Math. Sci. Technol. Educ. 16(9), em1872 (2020). https://doi.org/10.29333/ejmste/8345 12. Mithun Mohan, M., Pallavi Upadhyaya, K., Pillai, R.: Intention and barriers to use MOOCs: An investigation among the post graduate students in India. Educ. Inf. Technol. 25(6), 5017– 5031 (2020). https://doi.org/10.1007/s10639-020-10215-2 13. Maphosa, V., Dube, B., Jita, T.: A UTAUT evaluation of whatsapp as a tool for lecture delivery during the COVID-19 lockdown at a Zimbabwean University. Int. J. High. Educ. 9(5), 84 (2020). https://doi.org/10.5430/ijhe.v9n5p84 14. Liu, Z.-Y., Lomovtseva, N., Korobeynikova, E.: Online learning platforms: Reconstructing modern higher education. Int. J. Emerg. Technol. Learn. (iJET) 15(13), 4 (2020). https://doi. org/10.3991/ijet.v15i13.14645 15. Gil-Quintana, J., Malvasi, V., Castillo-Abdul, B., Romero-Rodríguez, L.M.: Learning leaders: teachers or youtubers? Participatory culture and STEM competencies in italian secondary school students. Sustainability 12(18), 7466 (2020). https://doi.org/10.3390/su12187466
Knowledge Management Strategies Through Educational Digital Platforms
303
16. Al-Maroof, R.S., Salloum, S.A., Hassanien, A.E., Shaalan, K.: Fear from COVID-19 and technology adoption: the impact of Google Meet during Coronavirus pandemic. Interact. Learn. Environ. 290(1), 1–16 (2020). https://doi.org/10.1080/10494820.2020.1830121 17. Adarkwah, M.A.: I’m not against online teaching, but what about us?: ICT in Ghana post Covid-19. Educ. Inf. Technol. (2020). https://doi.org/10.1007/s10639-020-10331-z 18. España Bone, Y.I., Vigueras Moreno, J.A.: La planificación curricular en innovación: elemento imprescindible en el proceso educativo. Revista Cubana de Educación Superior 40(1) (2021) 19. de Souza, K.R., et al.: Remote work, teaching health and virtual strike in a pandemic scenario. Trab. educ. saúde 19, e00309141 (2021) 20. Guzmán, J.E., Arrieta, D.B.: Gestión del conocimiento en Instituciones de Educación Superior: Caracterización desde una reflexión teórica. Rev. Cienc. Soc. 26(3), 83–97 (2020) 21. Acevedo-Correa, Y., Aristizábal-Botero, C.A., Valencia-Arias, A., Bran-Piedrahita, L.: Formulación de modelos de gestión del conocimiento aplicados al contexto de instituciones de educación superior. Información Tecnológica 31(1), 103–112 (2020) 22. Ramírez-Vera, L., Veytia, M.G., English, R.: Plataformas educativas de nivel básico usadas en homeschooling en México. Edähi Boletín Científico de Ciencias Sociales y Humanidades del ICSHu 8(16), 65–70 (2020) 23. Fernandez, A.D., Romero, A.C.G.-L.: Retos en tiempo de pandemia: plataformas educativas para la formación de agentes de cambio. In: Congreso Internacional de Innovación y Educación para el Desarrollo, GKA INNO 2020 (2020) 24. Porras Jaramillo, M.L., Quispe Huari, L.A.: Aplicación de la plataforma Schoology en estudiantes del cuarto grado de secundaria de una institución educativa pública en Lima, Perú (2020) 25. Benavides Albesiano, R.A.: Impacto del uso de las plataformas virtuales para los programas de formación de la Aeronáutica Civil de Colombia (2020) 26. Marín, S.Q., Cádiz, M.M.M., Pabón, N.D.: Retos de la educación superior: aprovechando las plataformas digitales. In: CIVINEDU 2020, p. 498 (2020)
Information Foraging on Social Media Using Elephant Herding Optimization Yassine Drias1(B) , Habiba Drias2 , Ilyes Khennak2 , Lydia Bouchlaghem1 , and Sihem Chermat1 1
University of Algiers, 16000 Algiers, Algeria [email protected] 2 USTHB, LRIA, 16111 Algiers, Algeria {hdrias,ikhennak}@usthb.dz
Abstract. In this paper, a new Information Foraging approach based on Elephant Herding Optimization (EHO) is proposed and tested on social media. We adapted the original EHO algorithm and combined it with the information foraging theory. In order to test our approach, we constructed a dataset containing more than one million tweets collected during the second semester of 2020. The results are very satisfying and show the ability of our approach to improve the information foraging process both in terms of relevance and response time. To further evaluate our system, we held a comparative study with another well-known metaheuristic applied to information foraging, namely Ant Colony Optimization. The outcomes show the superiority of our proposal. Keywords: Information foraging · Swarm intelligence herding optimization · Social media
1
· Elephant
Introduction
We witnessed recently a worldwide health crisis that impacted our daily routines and life habits. COVID-19 spread very quickly all over the world, which resulted in a situation of lockdown and social distancing. People turned to virtual platforms and social media to communicate and to get information. According to the Digital 2020 reports, more than 3.81 billion people are active social media users, representing by this mean 49% of the world’s population [1]. The data generated on social media represents a rich source of information and should therefore be exploited in an intelligent and effective way using modern techniques. In recent years, many efforts have been done in the domain of information foraging. Techniques and approaches such as game theory [2], deep learning [3], multi-agent systems [4], ontologies [5] and bio-inspired computing [6] were employed for this purpose. Besides, information foraging was applied to many domains like query auto-completion [7], recommender systems [8] and cyberattack prediction [9]. More recently, some works focused on tackling information c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 304–314, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_30
Information Foraging Using EHO
305
foraging on social media [10]. As social media platforms are becoming more popular among users, we believe it would be interesting to further explore these preforms using information foraging. In this paper, we propose a novel approach based on Elephant Herding Optimization to perform information foraging on social media.
2
Modeling Information Foraging on Social Media
Information Foraging is a recent paradigm, which aims at discovering paths leading to relevant information on the Web. The Information Foraging Theory (IFT) [11] is grounded on the analogy between the animal food foraging behavior and the behavior of humans while seeking information on the Web. It is based on the assumption that, when searching for information on the Web, users follow some hints and cues that help them to reach relevant information just like animals do when they follow the scent of their preys in order to catch them. In the following subsections, we give details on our proposed model that adapts the information foraging theory to social media platforms. 2.1
Social Networks Representation
In social media platforms, the information sources are the posts shared by the users. These posts, the users and the social interactions and relations are generally represented by a social graph. The term social graph was used for the first time during the Facebook F8 conference and became since then part of the internet vocabulary. In this work, we model a social network as an oriented graph G (V, E) where, V is the set of vertices representing the social media users and E is the set of directed edges representing different social relationships such as : a post, a re-post, a friendship, a mention, a reply and a follow. Figure 1 gives an example of a simplified social graph representation.
Fig. 1. Social graph
Note that the edges representing the relations post, re-post, mention and reply are the ones containing the social posts, and thus represent the information sources. We denote those edges as content-sharing edges.
306
2.2
Y. Drias et al.
User’s Interests
In addition to a collection of posts represented by a social graph, the information foraging process takes also the user’s topical interests as an input. These user’s interests can be either introduced explicitly by the user or implicitly extracted from the user’s activity (profile and interactions) on social media. The idea is to process the keywords provided by the user or the information available on their profile in order to create the user’s interests vector V following the bag-of-words model. 2.3
Information Scent
Information Foraging aims to find relevant information, while minimizing the time spent doing the search. This can be achieved thanks the information scent concept [12], which can be seen in real life as the user’s estimation of the value that a source of information will deliver to them. This value is mainly derived from the description/content of the source. For instance, if we consider the case of the Web, the information sources are the Web pages and they are described by a URL, a title and in some cases an icon. In our case, the goal is to reach posts that are relevant to a certain user based on their interests. We assume that a post is more appealing to the user if it is similar to their interests and that the information scent value should increase each time we get closer to a relevant post and decrease otherwise. We define the information scent as the difference in similarity between the current post with the user’s interests vector and the considered post to be visited with the user’s interests vector. Formula (1) estimates the information scent generated when considering to move from the current post located on the edge ei to one of its neighbors located on the edge ej . Inf oScent(ej ) = Sim(Ej , V ) − Sim(Ei , V )
(1)
where: – Sim(Ei , V ) is the cosine similarity between the user’s interests and the current post. – Sim(Ej , V ) is the cosine similarity between the user’s interests and the potential next post to be visited. Ej ∈ Ni , with Ni being the set of adjacent edges to the edge Ei , i.e. Ei ’s neighborhood. The role of the information scent is to guide the foraging. A positive information scent value means that we are getting closer to a relevant post in the social graph, while a negative value means the opposite.
Information Foraging Using EHO
307
2.4 Surfing Path The information foraging process starts from a post, then at each step it tries to reach a pot containing more relevant information compared to its predecessor. For this purpose, a surfing path is constructed starting from an initial contentsharing edge and can then be enriched by adding other edges, in such a way as to build a chain of connected posts. In other words, at each step of the foraging process the system should select one content-sharing edge to visit among the reachable edges from the current post. This decision is based on formula (2). 0, if Inf oScent(ej ) ≤ 0 (2) P (ei , ej ) = Inf oScent(ej ) , otherwise Inf oScent(el ) el ∈N pi
where: – P (ei , ej ) is the probability to choose the edge ej among the reachable edges from the current edge ei – N pi is the set of adjacent content-sharing edges of the edge ei with a positive information scent value, i.e. ∀el ∈ N pi Inf oScent(el ) > 0.
3
Using EHO to Forage Information on Social Media
Elephant Herding Optimization (EHO) is a swarm intelligence meta-heuristic that takes its origins from the herding behavior of elephants in nature. It was first introduced in [13] to solve hard continuous optimization problems. In order to efficiently find relevant information in a big social graph, we decided to combine the information foraging theory with EHO. This way, each elephant will search relevant social posts by browsing a part of the graph, and thus construct surfing paths leading to relevant information. To be able to use EHO, we made some important adaptations to the original algorithm beforehand, without altering its general structure. Those adaptations mainly concern the fact that EHO was originally developed to address continuous problems while the problem we are dealing with is a discrete one. 3.1
Elephant Population and Positions
To generate a population of elephants with p clans, we first create p elephants with respect to distClan, representing the minimal distance between clans. Then, using the positions of the p elephants, the rest of the elephants are generated in each clan with respect to distElephant, representing the maximal distance separating the elephants of the same clan. Figure 2 gives an example of a population of 3 clans distributed over a social graph.
308
Y. Drias et al.
Fig. 2. Example of a population of elephants
Considering a social graph with m content-sharing edges, we should assign m distinct positions, one to each content-sharing edge. Each elephant j belonging to a clan ci , is identified by a position that we denote xci ,j . The position of an elephant at time t is the position of the edge it is visiting at that time. 3.2
Building a Surfing Path
During a generation of the EHO algorithm, each elephant is given the task of constructing a surfing path with the goal of reaching relevant information. Algorithm 1 is used for this purpose. Algorithm 1. Surfing path construction Input: xci ,j : the elephant’s position, V : the user’s interests, G: the social graph; Output: SP : a surfing path leading to a relevant post; 1: SP ← {∅}. 2: Fetch ei the edge with the position xci ,j in the social graph. 3: Append ei to surfing path SP 4: Ni ← {∅} 5: for each adjacent content-sharing edges ej do 6: Compute Inf oScent(ej ) using formula (1) 7: if Inf oScent(ej ) > 0 then append ej to surfing path Ni 8: end if 9: end for 10: if Ni = {∅} then return SP 11: else 12: Choose the next edge to visit according to formula (2) 13: Go to 2 14: end if
Information Foraging Using EHO
3.3
309
Evaluating the Elephants’ Solutions
The best way to evaluate the solutions found by the elephants is to calculate the similarity between the user’s interests and those solutions, i.e. the surfing paths. The fitness function is given by function (3). f (xci ,j ) = Sim(Ei , V )
(3)
where: – Ei : is the last post on the surfing path constructed by elephant j in clan ci . – V : is the user’s interests vector. – Sim(P, V ) is the cosine similarity between Ei and V 3.4
Updating Operator
At the end of each generation after the evaluation of the foraged solutions, the elephants’ positions are updated using formula (4). xnew,ci ,j = xci ,j + α(xbest,ci − xci ,j ) × r
(4)
Where: – – – –
xnew,ci ,j : is the elephant’s new position. xci ,j : is the current elephant position. xbest,ci : is the position of the matriarch. α ∈ [0, 1]: is a scale operator that defines the influence of the matriarch on the new position of the elephant j. – r ∈ [0, 1]: is a stochastic distribution that improves the diversity.
The position of the matriarch of each clan is also updated throughout the generations, by calculating the average fitness of each clan using formula (5). After that, the position of the elephant with the closest fitness value to the average is used in formula (6) to calculate the new matriarch’s position. nc 1 i = f (xci ,j ) nci j=1
(5)
xnewbest,ci = xavg,ci × β
(6)
favg,ci
Where: – favg,ci : is the average fitness of the clan ci . – xnewbest,ci : is the new position of the matriarch of the clan ci . – xavg,ci : is the position of the elephant with the fitness value closest to the favg,ci value. – β ∈ [0, 1]: is a factor that determines the influence of the average position on the matriarch’s new position. – nci : is the number of elephants in the clan ci . – xci ,j : is the position of the elephant j in the clan ci .
310
3.5
Y. Drias et al.
Separating Operator
At the end of each generation, the elephant with the worst fitness value will leave the clan. This behavior is implemented using formula (7). xworst = xmin + (xmax − xmin + 1) × r
(7)
Where: – xworst : represents the position of the elephant with worst fitness value. – xmin and xmax : are the upper and lower bounds of the positions interval. – r: is a stochastic and uniform distribution parameter. The overall process of information foraging on social media using EHO is shown in Algorithm 2. Algorithm 2. Elephant Herding Optimization for Information Foraging Input: V : the user’s interests, G: the social graph; Output: a list of surfing paths ranked by relevance; 1: Set the generations counter k ← 1 and the solutions list sols ← {∅}. 2: Initialize the Maximum generation (M axGen), empirical parameters α and β, clans number nClans, population size and number of elephants in each clan nci . 3: In the search space, randomly initialize the positions of the elephants according to distClan and distElephant. 4: while k ≤ M axGen do 5: for i ← 1 to nClans do for all clans in elephant population for all elephants in clan ci 6: for j ← 1 to nci do 7: Build the elephants’ surfing paths using algorithm (1) 8: Calculate the elephants’ fitness using formula (3). 9: Update the positions of the elephants xci ,j using formula (4). 10: Update the matriarch’s position using formula (6). 11: end for 12: end for 13: for i = 1 to nClan do for all clans in elephant population 14: Locate the worst elephant individual of clan ci using formula (7). 15: Generate a new elephant in the clan ci . 16: end for 17: Append the best surfing paths found in generation k to sols 18: Update the generation counter, k ← k + 1. 19: end while 20: Return the best surfing paths ranked by their relevance.
4
Experiments
The experiments presented in this section were implemented using Java programming language and were held on a laptop running Windows 10 with an Intel Core i5-4300M CPU at 2.60 GHz and 6 GB of RAM.
Information Foraging Using EHO
4.1
311
Dataset Description
We tested our algorithm on Twitter, which is one of the most popular social networks and a microblogging platforms. We constructed a dataset composed of 1 027 876 tweets that we grouped in a big social graph. The data crawling was performed using NodeXL [14] and took place during the second semester of 2020. Figure 3 showcases the main topics covered by our dataset. 4.2
Empirical Parameters Setting
In order to get the best outcomes we conducted extensive tests with the aim of tuning the empirical parameters. Figure 4 gives an example of how we set the values of α and β based on the similarity variation. Table 1 shows the values of the empirical parameters that yielded the best results.
Fig. 3. Topics covered by the dataset Table 1. Empirical parameters values Parameter Value α
0.9
β
0.4
nClans
8
nElephants 90 MaxGen
Fig. 4. Setting α and β
40
312
4.3
Y. Drias et al.
Foraging Results
Table 2 presents some examples of information foraging results with 7 different users’ interests (column one) generated for evaluation purposes. Column two shows the surfing path containing the most relevant tweet, column three gives the similarity value between the surfing path and the user’s interests and column four and five exhibit respectively the response time in seconds and the surfing path’s length. Note that for the cases where the surfing depth is superior to 1, we show the whole surfing path by chronological order of visit as it is the case for the user’s interest “diabetes type 2, intermittent fasting” for instance. Table 2. Information foraging results User's interests
Most relevant surfing path
Python for Machine Learning and Data Mining #DeepLearning #datamining Machine Learning, IA, #learning via https://t.co/qcC4wrx6m6 https://t.co/kLvs68HzEQ Python American Express, free American Express https://t.co/o9suYDdsV3 speech, democracy COVID19 immunity @CoocoLa_Vrej WHO is still not sure if those who recovered from COVID 19 develop a certain immunity that they will not get COVID 19 virus again. transmission 1 Does anyway here do intermittent fasting? How do you do it? diabetes type 2, 2 Can intermittent fasting make you diabetic? intermittent fasting 3 Intermittent fasting has proven to help cure Type II diabetes Bitcoin price within about 3% of gold price https://t.co/GwjcMSB9Jp Bitcoin prices market But who believes Joe had anything to do with deciding this, or preparing the 1 doc? Who is in charge? https://t.co/x8RBCz4H6D Joe Biden and Bernie 2 Folks mention Biden's past plagiarism True Senders @LyndaMo85130479 @BugOffDear Biden positions are literally just 3 copy/pasted from Bernie Sanders Samsung IoT Smart City https://t.co/Xnf5JHnOq9 via @YouTube Smart City, 5G, IoT @_funtastic5_ #TelkomFuntastic5 #RWSTREG5 #smartcity
Similarity
Time Surfing (s) depth
0.71
26
1
0.64
27
1
0.57
26
1
0.74
28
3
0.67
24
1
0.46
26
3
0.7
25
1
We observe that in almost all cases, our approach is capable of finding relevant tweets in a short amount of time considering the size of the dataset and the fact that no preprocessing or offline steps were performed beforehand, since the foraging process happens online. We can also notice that the surfing depth is to a certain extent small, which can be explained by the fact that our social graph is not strongly connected. Furthermore, during the surfing path construction, a tweet is inserted into the surfing path only if it is more relevant than its predecessors already existing in that same path. 4.4
Comparative Study
For the sake of comparing our approach (EHOIF ) to other similar information foraging approaches, we implemented the Web Information Foraging Using Hybrid Ant Colony Optimization and Tabu Search (WIFACO) approach proposed in [15] and adapted it to the social networks environment. We believe that it would be interesting to compare the two approaches since they are both based on bio-inspired metaheuristics. The empirical parameters of ACO were set as follows: α = 0.2, β = 0.4, ρ = 0.8, q0 = 0.8, N bAnts = 50, gen = 50. Results of the comparison are reported in Fig. 5.
Information Foraging Using EHO
313
Fig. 5. Comparison with ant colony optimization
5
Conclusion and Perspectives
A novel information foraging approach based on elephant herding optimization was proposed in this paper. We first adapted EHO to information foraging by doing several changes to the original algorithm. In order to evaluate our approach, we created a dataset containing more that 1 million tweets. The information foraging results were very satisfying both in terms of relevance and response time. Finally, we performed a comparative study with an already existing information foraging approach based on the well-known metaheuristic “ant colony optimization”. The results showed that our approach has an overall better performance. As far as we know, this is the first time an information foraging approach based on elephant herding optimization is proposed and one of the few destined to work on social media, which gives this work a remarkable originality. We also think that the results are really promising. In the near future, we plan to undertake a preprocessing phase using clustering in order to optimize the foraging in large datasets. This will allow a better placement of the elephants on the search space starting from the first generation based on the user’s interests. We also aim to do a more in depth comparative study with ACO and other metaheuristics and even compare our approach to other information access approaches such as Information Retrieval.
References 1. We Are Social, Digital 2020 reports. (2020). https://wearesocial.com/blog/2020/ 01/digital-2020-3-8-billion-people-use-social-media 2. Drias, Y., Kechid, S.: Dynamic Web information foraging using self-interested agents: application to scientific citations network. Concurrency Comput. Pract. Experience J. 31(22), e4342 (2019). https://doi.org/10.1002/cpe.4342 3. Niu, X., Fan, X.: Deep learning of human information foraging behavior with a search engine. In: International Conference on Theory of Information Retrieval, Santa Clara, CA, USA, pp. 185–192. ACM (2019). https://doi.org/10.1145/ 3341981.3344231
314
Y. Drias et al.
4. Drias, Y., Kechid, S.: Dynamic Web information foraging using self-interested agents. In: Recent Advances in Information Systems and Technologies - WorldCIST 2017, vol. 569, pp. 405-415. Springer, Cham (2017). https://doi.org/10.1007/ 978-3-319-56535-4 41 5. Nguyen, V., Rabby, G., Sv´ atek, V., Corcho, O.: Ontologies supporting researchrelated information foraging using knowledge graphs: literature survey and holistic model mapping. In: Knowledge Engineering and Knowledge Management: 22nd International Conference, EKAW, Bolzano, Italy. Springer, Cham (2020). https:// doi.org/10.1007/978-3-030-61244-3 6 6. Drias, Y., Kechid, S., Pasi G.: Bee swarm optimization for medical web information foraging. In: Journal of Medical Systems. 40(2), Springer (2016). https://doi.org/ 10.1007/s10916-015-0373-5 7. Jaiswal, A., Liu, H., Frommholz, I.: Utilising Information Foraging Theory for User Interaction with Image Query Auto-Completion. In: Advances in Information Retrieval, ECIR, pp. 666–680. Springer (2020). https://doi.org/10.1007/978-3-03045439-5 44 8. Schnabel, T., Bennett, P., Joachims, T.: Shaping feedback data in recommender systems with interventions based on information foraging theory. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne VIC, Australia, pp. 546–554. (2019). https://doi.org/10.1145/3289600. 3290974 9. Dalton, A., Dorr, B., Liang, L., Hollingshead, K.: Improving cyber-attack predictions through information foraging. In: International Conference on Big Data, Boston, MA, USA, pp. 4642–4647. IEEE Computer Society (2017). https://doi. org/10.1109/BigData.2017.8258509 10. Drias, Y., Pasi, G.: Credible information foraging on social media. In: Trends and Innovations in Information Systems and Technologies - WorldCIST 2020, pp. 415– 425. Springer (2020). https://doi.org/10.1007/978-3-030-45688-7 43 11. Pirolli, P., Card, S.: Information foraging. Psychol. Rev. 106–4, 643–675 (1999) 12. Budiu, R., Royer, C., Pirolli, P.: Modeling information scent: a comparison of LSA, PMI and GLSA similarity measures on common tests and corpora. In: ComputerAssisted Information Retrieval, 8th International Conference, Pittsburgh, PA, USA, 31-(22), (2007) 13. Wang, G., Deb, S., Gao, X., Coelho, L.: A new metaheuristic optimisation algorithm motivated by elephant herding behaviour. Int. J. Bio-Inspired Comput. 8(6), 394-409 (2016). https://doi.org/10.1504/IJBIC.2016.10002274 14. Smith, M., Ceni, A., Milic-Fraylin, N., Shneiderman, B., Mendes Rodrigues, E., Leskovec J., Dunne, C.: NodeXL: a free and open network overview discovery and exploration add-in for Excel 2007/2010/2013/2016. The Social Media Research Foundation (2010) https://www.smrfoundation.org 15. Drias, Y., Kechid, S., Pasi, G.: A novel framework for medical web information foraging using hybrid ACO and Tabu search. In: Journal of Medical Systems. 40(1), Springer (2016). https://doi.org/10.1007/s10916-015-0350-z
Social Vulnerability Segmentation Methodology Based on Key Performance Indicators Nuria Mollá-Campello1(B) , Kristina Polotskaya1 , Esther Sobrino1 , Teresa Navarro2 , and Alejandro Rabasa1 1 Center of Operations Research (CIO), Miguel Hernandez University of Elche (UMH), 03202
Elche (Alicante), Spain [email protected], {k.polotskaya,a.rabasa}@umh.es, [email protected] 2 Red Cross Spain - Valencian Community, 46014 Valencia, Spain [email protected]
Abstract. Detecting social vulnerability is a complex process that involves many personal and environmental factors to take into account. In this paper, a new methodology is proposed to detect which individuals are more likely to live under vulnerable situations merging all those personal and environmental factors in one metric. This methodology assigns each person who asks The Red Cross for help different normalized key performance indicators (KPI) regarding several areas as personal, social, health, environmental, economic, labour and familiar. All these fields are combined to represent the person in two axes: the internal or personal vulnerability axis (composed by personal, health and economical indexes) and external axis (composed by environmental, labour, social and familiar indexes). Based on these axes and through unsupervised machine learning techniques, this methodology assigns each person a vulnerability group which may be related with a series of actions to cover their needs and act upon their situation. This way, the proposed methodology allows us to go from a high dimensionality to a reduced problem that considerably simplifies the study. This process permits The Red Cross act quicker in those high-vulnerable or more-likely-vulnerable situations, improving the assistance process and helping the estimation of needed resources. Keywords: Vulnerability · Segmentation · Key performance indicators · Dimensionality reduction
1 Introduction When we talk about vulnerability, many factors are involved in the process of its definition. We do not simply refer to social situations that affect a person but the environmental, health, personal, economic, professional and familiar conditions. All these dimensions try to deeply describe a personal situation that, obviously, is changing over time, to act upon the most vulnerable cases giving an accurate and quick answer. In the social vulnerability field, several factors must be considered, and they are absolutely dependent on personal circumstances, that are on continuous changing. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 315–323, 2021. https://doi.org/10.1007/978-3-030-72651-5_31
316
N. Mollá-Campello et al.
So, although Red Cross Spain, with many actions and plans, provides direct help to people at risk through a vast accumulated expert knowledge, the organization is strongly committed to discovering new processes that facilitate decision-making about the care of its users, reinforcing its own technical experience with the indisputable value of the data itself. It is important to remember that we are working with people, so the way actions are carried out is really relevant to the attendance process. The authors believe that hybrid expert-driven/data-driven approach becomes especially important in the context of social care and attention to vulnerability. In this sense, Red Cross and Miguel Hernández University face this vulnerability segmentation project with this doble approach. The paper is organized as follows. Section 2 is dedicated to briefly explain the objectives of this research both from the methodological and the practical point of view. In Sect. 3, we explain the current situation and the previous works made in the fields to contribute. Formal methodology is presented and detailed in Sect. 4 and tested via computational experiences in Sect. 5. Finally, Sect. 6 concludes and proposes future works.
2 Objectives From the methodological point of view, the objective is to define a methodology that, based on the definition of vulnerability indicators (in the form of KPI), can significantly reduce the dimensionality of a particularly “wide” dataset. This way, a problem with a large number of attributes could be reduced simplifying the application of Machine Learning techniques and its interpretability. In this specific case, we want to segment the Red Cross’ users under different vulnerability dimensions and provide the experts interpretable representation to decide which program needs to be applied. Joining all user information in two or three measurements makes possible to generate a two-or-three-axis plot, that is more understandable since we are familiar with 2d or even 3d graphs.
3 Current Situation Nowadays, the Information Systems contain more and more knowledge based on Data Analysis techniques. These increasingly used methods provide models based either on data (data-driven) or on expert criteria (expert-driven) and aim to speed up the decisionmaking process. In the social intervention field, a quick and accurate response is especially important since we deal with people at risk. Several contributions in this field use Machine learning techniques to predict the socio-economic vulnerability [1] or to plan the allocation of resources [2]. The Red Cross vulnerability programs are annually evaluated in the Social vulnerability reports [3]. These memories describe the taken actions and their results, measuring the impact and improvement achieved in the personal situation of the users. In this sense, this work set a precedent by applying unsupervised Machine Learning techniques, specifically a segmentation method, to vulnerability Red Cross datasets.
Social Vulnerability Segmentation Methodology
317
In these decision-making and resources planning processes, it is necessary to highlight the importance of KPI in a wide variety of cases; from industrial processes [4, 5] to energy production [6] or supply chain management [7]. All these cases based their systems in a series of Key Performance Indicators (KPI) that support the decision-making process [8]. In most cases, these key indicators are defined by the organization though a strategic and complex consulting process based on their own data and needs [9]. Regarding the segmentation process, several techniques are considered to study the user vulnerability data. The reference algorithm is Kmeans [10], based on iterative items reassignment to the closest group centers. Other methods such as Kmedoids [11], based on medoid instead of mean concept; or CLARA [12], that incorporates random selection of items, are also widely applied. These segmentation methods have already been applied to multiple fields as marketing [13], tourism [14], medicine [15] or the social sciences field [16]. In this case, the Kmeans algorithm is selected to segment since it is the most extended method and allows the comparation of this study with similar ones.
4 Methodology A new 4-step analytical methodology is proposed in this paper, whose main contribution is the generation of synthetic KPI that summarize complex personal situations though numerous vulnerability variables. All these attributes are linked to a specific dimension or category for which a representative KPI is created. In the next step, the proposed methodology joins this dimensional KPI in several axes based on data (data-driven) or expert criteria (expert-driven). Using these axes, the methodology segments the users generating groups with similar characteristics and needs. In an expert-driven environment, it is desirable for these segmentation groups to be deeply studied to determine which actions might have an impact on their situation. On the other hand, in a data-driven approach, the necessary actions could be determined by data itself, assigning a program based on the most-used action for that group. Figure 1 shows a description of the proposed methodology. In the first step, initial user data is preprocessed and loaded in the system. This data comes originally from surveys made by Red Cross experts to their users. In the second step, several KPI are defined, attending to the different vulnerability dimensions. These dimensions are social, environmental, familiar, economic, personal, health and labour, and all them compile a set of variables that describe the situation of a user. This way, we end up with seven indicators that summarize information and represent the vulnerability level in each field for every person. In the next phase, the vulnerability axes are generated under data-driven criteria and expert-driven criteria. Data-driven approach study the data correlation matrix to group the dimension’ KPI while the expert-driven criterion forms the axes following the advice of Red Cross professionals. Both approaches are studied, and it is decided to use the expert criteria since less axes are generated and they are meaningful for the professionals that are going to interpret the results. After this, in the fourth and last step, the segmentation takes place using the proposed axes to group similar vulnerability profiles. Below, the different phases of the proposed methodology are explained in more detail applied to the Red Cross case of study.
318
N. Mollá-Campello et al.
Fig. 1. Segmentation methodology. General overview.
4.1 Step 1: Data Acquisition and Preprocessing The initial survey collects up to a maximum of 245 variables, out of a total of 20202 users. All these variables collect information about different areas that define the social, environmental, familiar, economical, personal, health and working situation. These data correspond to the period of time between 2015-01-02 and 2020-01-28. The data has been formally pre-processed to erase all erroneous or out-of-range values (outliers). 4.2 Step 2: Generation of KPI From this multidimensional dataset, a total of 7 KPI have been generated, one for each vulnerability area: personal, social, economic, familiar, labour, environmental and health. Each key indicator has been created by weighing the importance of some dimension variables combining them in a meaningful and normalized way. Table 1 summarizes the KPI compilation result. The criteria followed to select variables is based on both data quality and expert opinion. After a first attributes filter based on the quality of data (nonnull values), Red Cross professionals through a strategic consulting process decide which variables define each dimension. This way, we end up summarizing numerous complex categorical variables in 7 numerical KPI that indicate the dimensional vulnerability level of a user. 4.3 Step 3: Generation of Vulnerability Axes As it is mentioned above, two different ways of creating vulnerability axes are proposed. The first one responds to a data-driven approach, in which a correlation matrix is used to group the KPI forming the most correlated axis. The second way responds to an expert-driven approach where the criteria of the Red Cross experts are used to create two axes: (1) one related to the internal vulnerability and (2) another related to external
Social Vulnerability Segmentation Methodology
319
Table 1. KPI dimensions and variables KPI
# variables # selected variables
Economic
45
Labour
22
8
Familiar
55
4
Environment 19
10
2
Social
31
8
Health
43
20
Personal
21
7
vulnerability. Figure 2 shows how vulnerability factors or dimensions would form the axes in each of these approaches. In both cases, a very similar distribution of factors is obtained, being especially noteworthy that in the data-driven approach, the health component should constitute a third axis by itself. Finally following the Red Cross specialists’ opinion, the research stuck to the expert-driven approach, two axes that compile vulnerability from internal (personal, economic, health) and external (social, familiar, labour, environment) factors. At this point, the methodology has reduced a 245-variable problem to two meaningful axes.
Fig. 2. Vulnerability axes under data-driven and expert-driven criteria
320
N. Mollá-Campello et al.
4.4 Step 4: Users Vulnerability Segmentation At the last step, the segmentation is performed according to the internal and external axes proposed by the NGO. Several grouping criteria are studied to offer the Red Cross scenarios with 2 or 3 clusters to apply actions depending on the situation. Segmentations with a high number of groups always imply more resources but it is more probable to get better results and satisfaction rates. It is important to get a balance between dedicated means and personal needs to guarantee quality and sustainable programs.
5 Computational Experience and Results The segmentation process has been designed considering the internal and external vulnerability axes, formed as follows: 1. Internal Vulnerability Axis (X): personal_kpi, health_kpi, economic_kpi 2. External Vulnerability Axis (Y): enviromental_kpi, labour_kpi, social_kpi, familiar_kpi. The clustering process represented in two dimensions makes results easier to interpret for NGO experts. In this sense, several clustering scenarios have been proposed. The computational experience is run with R language and the Kmeans algorithm [10], specifically the Clustering function from the MachineLearning package [17], based on the Kmeans implementation in base package [18]. The segmentation experiments generate 2 and 3 groups based on the expert recommendations. Both cases are represented in Fig. 3 (2 groups) and Fig. 4 (3 groups).
Fig. 3. User’s vulnerability segmentation in 2 groups
Social Vulnerability Segmentation Methodology
321
Fig. 4. User’s vulnerability segmentation in 2 groups
The Red Cross must decide how many vulnerability groups want to create depending on the number of strategies the NGO wants to keep in order to address the individual vulnerable situations. As it has been said, two scenarios are studied, segmenting data in 2 and 3 groups. In the first scenario, we segment the data in 2 groups as it can be seen in Fig. 3. Group 1 (blue) accumulates 42.30% of the sample while the group 2 (red) gathers the remaining 57.90%. The most representative individual or centroids for internal and external vulnerability in this segmentation are (0.27, 0.09) for group 1 and (0.05, 0.06) for group 2. In the second scenario, all the users are divided in 3 groups as it can be seen in Fig. 4. These groups gather 17.29% of users in group 1 (green), 51.26% in group 2 (blue) and 31.45% in group 3 (red). In this case, the centroids of each cluster correspond to the following internal and external vulnerability values: group 1 (0.34, 0.11); group 2: (0.03, 0.05); group 3: (0.2, 0.08). Considering the differences between scenarios, the Red Cross decides that the smallest group (group 1 with 17.29%) is relevant and therefore, this group remains. This way, this second segmentation in 3 groups is the one that will be used by the NGO to plan their actions. It can be noticed that the variation in the internal vulnerability axis (X) is greater than the variation of the external vulnerability axis (Y). It can be clearly seen in both figures Fig. 3 and Fig. 4 that the internal vulnerability axis (X) is predominant when the group partitions take place. This means that the elements in each group are divided based on the internal vulnerability criteria (personal, health and economic KPI).
322
N. Mollá-Campello et al.
6 Conclusions and Future Work The definition of KPIs during the analysis process is shown as a very efficient technique to reduce the dimensionality of a problem, making the descriptive models easier to interpret by social experts. The expert-driven and data-driven hybrid approach is presented as a magnificent alternative in the field of complex social data analysis. Decisions based on the experience of sociologists can easily be confirmed or reoriented thanks to the analysis of relatively long series of data. This approach is especially relevant in data contexts that change significantly with the economic and environmental circumstances of the studied individuals. In this sense, the segmentations offered as a final result of the study were valued very positively by the Red Cross and those responsible for the attention to the vulnerable cases. Based on this study new reallocation of resources and personal plans designs are proposed. This work is a first approach to a complex problem that has been simplified and which future works go through implementing predictive models. These methods, provided with schematized KPI, would forecast the necessary actions to prevent vulnerable situations or to minimize their impact. The predictive models could be also used to infer the personal satisfaction of the users with their assigned plans. It is expected that the reduction in dimensionality obtained with this methodology will offer precise and still interpretable predictive models. Finally, the authors intend to integrate this methodology into an early warning vulnerability system, which allows the NGO and the authorities in the field of social welfare, to act quickly in the most difficult cases of exclusion risk. Acknowledgments. The authors are grateful for the financial support from Red Cross Valencian Community under collaboration agreement 104/19 with Miguel Hernández University. This research has been also supported by the Spanish Ministry of Science, Innovation and Universities, under grant DIN2018-010101 and Miguel Hernández University under grants 207T/20 0500/5421007/22609 and ISOCIAL program 0500/5421006/22609.
References 1. Engelmann, G., Smith, G., Goulding, J.: The unbanked and poverty: predicting area-level socio-economic vulnerability from m-money transactions. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 1357–1366 (2018) 2. Farrokhvar, L., Ansari, A., Kamali, B.: Predictive models for charitable giving using machine learning techniques. PLoS ONE 13(10), e0203928 (2018) 3. Spanish Red Cross, Departament of Studies and Social Innovation. Social vulnerability report (2018) 4. Sun, Q., Ge, Z.: Deep Learning for Industrial KPI prediction: when ensemble learning meets semi-supervised data. IEEE Trans. Ind. Inf. 17(1), 260–269 (2021). https://doi.org/10.1109/ TII.2020.2969709 5. Yang, X., Zhang, Y., Shardt, Y.A.W., Li, X., Cui, J., Tong, C.: A KPI-based soft sensor development approach incorporating infrequent, variable time delayed measurements. IEEE Trans. Control Syst. Technol. 28, 2523–2531 (2019)
Social Vulnerability Segmentation Methodology
323
6. May, G., Barletta, I., Stahl, B., Taisch, M.: Energy management in production: a novel method to develop key performance indicators for improving energy efficiency. Appl. Energy 149, 46–61 (2015) 7. Rodríguez-Rodríguez, R., Alfaro-Saiz, J.-J., Carot, J.M.: A dynamic supply chain BSC-based methodology to improve operations efficiency. Comput. Ind. 122, 103294 (2020) 8. Rodriguez, R.R., Saiz, J.J.A., Bas, A.O.: Quantitative relationships between key performance indicators for supporting decision-making processes. Comput. Ind. 60(2), 104–113 (2009) 9. Ohlig, J., Hellebrandt, T., Metzmacher, A.I., Pötters, P., Heine, I., Schmitt, R.H., Leyendecker, B.: Performance management on the shop floor – an investigation of KPI perception among managers and employees. Int. J. Quality Serv. Sci. 12, 461–473 (2020) 10. MacQueen, J.B.: Some methods for classification and analysis of multivariate observations. In: Le Cam, L.M., Neyman, J. (eds.) Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 281–297. University of California Press, California (1967) 11. Chen, Z.: Data Mining and Uncertaining Reasoning. An Integrated Approach. Wiley Interscience (2001) 12. Wei, C.-P., Lee, Y.-H., Hsu, C.-M.: Empirical comparison of fast partitioning-based clustering algorithms for large data sets. Exp. Syst. Appl. 24(4), 351–363 (2003) 13. Bloemer, J.M., Brijs, T., Vanhoof, K., Swinnen, G.: Comparing complete and partial classification for identifying customers at risk. Res. Market. 604, 1–5 (2003) 14. Rabasa, A., Pérez Martín, A., Giner, D.: Optimal clustering techniques for the segmentation of tourist spending. Analysis of tourist surveys in the Valencian community (Spain): a case study. Int. J. Des. Nat. Ecodyn. 12, 482–491 (2018) 15. Bruse, J.L., Zuluaga, M.A., Khushnood, A., McLeod, K., Ntsinjana, H.N., Hsia, T.Y., Sermesant, M.: Detecting clinically meaningful shape clusters in medical image data: metrics analysis for hierarchical clustering applied to healthy and pathological aortic arches. IEEE Trans. Biomed. Eng. 64(10), 2373–2383 (2017) 16. Trovato, M.R., Clienti, C., Giuffrida, S.: People and the city: urban fragility and the real estate-scape in a neighborhood of Catania, Italy. Sustainability 12(13), 5409 (2020) 17. Perez-Martin, A., Perez-Torregrosa, A., Rabasa-Dolado, A., Molla-Campello, N., RodriguezSala J.J.: MachineLearning: Machine Learning Algorithms for Innovation in Tourism. R package version 0.1.3 (2020). https://datascienceumh.github.io/MachineLearning/ 18. R Core Team R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2018). https://www.R-project.org/
A Requirements Catalog of Mobile Geographic Information System for Data Collection Badr El Fhel1 , Lamyae Sardi1 , and Ali Idri1,2(B) 1 Software Project Management Research Team, ENSIAS, Mohammed V University in Rabat,
Rabat, Morocco [email protected] 2 MSDA, Mohammed V Polytechnic University, Ben Guerir, Morocco
Abstract. Mobile Geographic Information System (mGIS) for data collection are designed to capture, analyze and manage geographical data. The aim of this paper is to define a requirements catalog for mGIS for data collection based on the main software engineering standards, GIS standards and literature. The catalog contains functional requirements in addition to requirements about usability, internationalization (i18n), performance efficiency, reliability and sustainability. This catalog can be useful for stakeholders and developers to evaluate their apps and/or identify potential requirements for new mGIS apps for data collection. Keywords: Requirements · Reusable catalog · Geographic information system · Data collection · Apps · I18n · Sustainability
1 Introduction Mobile computing systems and hardware have changed the way mobile mapping technology is used by moving Geographic Information System (GIS) from desktop into user’s hands [1]. Mobile GIS applications (GIS apps) improves Location-Based Services (LBS) [2] and play an important role in mobility mapping [3]. Users participate in GIS mapping via apps [4]. This phenomena is widely known as Volunteered Geographic Information (VGI) and is defined by Goodchild as individuals who used the web to create, assemble and disseminate geographic information [5]. In fact, most of smartphones are nowadays equipped with a variety of built-in sensors that allow geo-location and orientation [6]. These sensors concern mainly the Global Positioning System (GPS) which is considered as an crucial component in the general mGIS architecture [7]. Thereby, mGIS for data collection allow user to capture and store geo-spatial data in the field. Several technologies have also supported the development of mGIS such as wireless communication technology and mobile positioning technology [8]. These technologies made it possible to integrate more functionalities and develop more GIS apps [9]. Many GIS apps for data collection have been developed to be used in various domains. For instance, Maciej M et al. presented a review of twelve GIS apps for environmental field survey [10]. In the electrical industry, a GIS app for gathering, integrating, © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 324–336, 2021. https://doi.org/10.1007/978-3-030-72651-5_32
A Requirements Catalog of Mobile Geographic Information System
325
analyzing and visualizing spatial data was developed by Poorazizi et al. [11]. In the agriculture domain, a research of real-time agriculture information collection system based on mobile GIS was carried out by Chen et al. [12]. In addition to these applications, many other solutions were proposed regarding arable lands [13], fire hydrant mapping [14] and geological data acquisition [15]. Despite the heterogeneity of the spatial information to be collected, two approaches are adopted to simplify and model space [16]; i) Vector model which is useful to treat discrete geographic phenomena like rivers, roads and building. ii) Raster model that is usually used to represent continuous phenomena like satellite images and maps. Besides the spatial information, non-spatial characteristics can be collected, so that they can be assigned to the spatial data. From a functional point of view; GIS allow storing, checking, and displaying data related to positions on earth’s surface, thus functionalities that allow geoprocessing and spatial analysis can be provided by mGIS for data collection [16]. Along with these requirements specific to GIS data collection, non-functional requirements should also be considered to describe how the software operates. In this respect, MGIS apps should implement internationalization (i18n), reliability, usability and performance aspects. For a mGIS to be used by a great number of users, it should implement a multi-language support and In order to target a large number of users, the multi-language support feature has to be considered. Moreover, positioning system coordinates should be defined locally and globally across the world [7]. Quality assurance integrated into the data acquisition workflow is very important in mobile GIS [17], functionalities that allow spatial data quality control have to be implemented in mGIS in order to tell user how reliable the collected datasets are [17, 18]. From a user satisfaction point of view, usability is considered as an important attribute for the system acceptance by the end users. In the ISO 9241-15: 2008 standard, usability is defined as the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments [19]. Given that measuring usability includes both objective and subjective metrics [20], ISO/IEC 25010 defines six sub-characteristics of the usability: Appropriateness, Learnability, Operability, User inter-face aesthetics and Accessibility [32]. Moreover, Performance is an important issue in mGIS, many researchers discussed and presented solutions about mGIS designs [21–23] ISO/IEC 25010 defines performance efficiency as the performance relative to the amount of resources used under stated conditions. Three dimensions of the performance efficiency were therefore stated: Time behavior, Resource utilization and Capacity. Requirements engineering is considered as a key task in software development [24]. Requirements activities consist of eliciting, analyzing, specifying and validating requirements, thus, the requirements engineering process has to be carefully performed [25]. The ISO/IEC/IEEE 29148 standard claims that good requirements should be unambiguous, complete, consistent, feasible, comprehensible and testable [26]. Hence, a Software Requirements Specification (SRS) document is required to communicate requirements to stakeholders and define details for developers. Previous requirements catalog were carried out by researchers in various disciplines. Regarding GIS applications, Israr and Syed Shahab Ali carried out a study of usability
326
B. El Fhel et al.
requirements for GIS applications and tested the usability requirements on user performance and satisfaction [30]. Moreover, other studies presented specifications regarding spatial data quality assurance during mobile GIS data collection [17, 18]. To the best of our knowledge, a requirement catalog specific to mGIS for data collection has not yet been developed. Hence, this paper aims to propose a catalog that includes both functional and quality requirements. The functional requirements determine what a mobile GIS for data collection is able to provide while the quality requirements determine how well the software product performs. These quality requirements mainly focus on the usability, i18n, performance efficiency, reliability and sustainability of mGIS for data collection. This paper is organized as follows: Sect. 2 describes the methodology followed to develop the requirements catalog. Section 3 reports the results with an illustration example on an existing mobile GIS application. Section 4 discusses the findings of this illustration. Finally, conclusions and future work are presented in Sect. 5.
2 Method 2.1 Requirements Specification In order to identify the SRS for mobile GIS for data collection, this study relies upon the following sources: (i) Studies about features and functionalities of mobile GIS [10, 31] (ii) Analysis of functionalities in GIS apps [11–15] (iii) Studies on software quality concerns for field data collection in Mobile GIS [17, 18, 34] (iv) Studies usability requirements for GIS apps [30, 32] (v) Existing software requirements catalogs that focus on sustainability and i18n [27–29]. Moreover, requirements from the following standards were also extracted: • ISO/IEC 25010 standard for system and software product quality requirements and evaluation [33] • W3C standards for Web and mobile devices. • OGC Web Services Common Standard [34] • ISO 19113 Geographic information – Quality principles [35] • ISO/TS 19158: 2012 Geographic information- Quality assurance of data supply [36] • ISO 9241-15: 2008 standard for ergonomics of human-system interaction [37] • OGC specifications regarding Hierarchical Data Format [38] • ISO 19133:2005 Geographic information- Location-based services - Tracking and navigation [39] 2.2 Catalog Development In order to develop a requirements catalog that is unambiguous, complete, consistent, feasible and testable as recommended by ISO/IEC/IEEE 29148, a process of three steps was followed. Figure 1 describes the generation process which consists of: (1) Identification of relevant sources (2) Requirements Extraction from these sources (3) Generation of the catalog in alignment with the ISO/IEC/IEEE 29148. This standard contains a set of provisions for the processes and products related to requirements engineering for systems and software products and services throughout their life cycle.
A Requirements Catalog of Mobile Geographic Information System
327
Fig. 1. The generation process of the catalog
3 Result 3.1 Identified Requirements The extracted requirements are presented in this section. Table 1 shows the functional requirements of mGIS for data collection. Table 2 contains quality requirements regarding i18n, reliability, performance efficiency, sustainability and usability. Note that i18n and sustainability requirements were retrieved from the existing catalogs in [27–29] and were thoroughly adapted to the mobile GIS for data collection. Table 1. Functional requirements Functionalities 1) The application shall allow drawing on the map 2) The application shall allow markers edition 3) The application shall allow users to add place note 4) The application shall allow users to access attributes 5) The application shall allow users to save their position 6) The application shall allow users to take Geo located pictures 7) The application shall allow users to record videos 8) The application shall allow users to do audio recording 9) The application shall allow users to do track recording 10) The application shall allow users to measure area and distance 11) The application shall implement buffer generation 12) The application shall implement Voronoi diagrams 13) The application shall implement data envelopment analysis 14) The application shall provide the bounding box (BBOX) tool 15) The application shall implement a spatial data filter 16) The application shall support known GIS File Format 17) The application allow user to export their geographic data as known GIS file format
3.2 The Generated Catalog The requirement catalog was structured according to the IEEE 29148:2011 standard as shown in Table 3. The aim of this catalog is to provide a comprehensive set of
328
B. El Fhel et al. Table 2. Quality requirements identified
Internationalization requirements 1) The application shall support multiple coordinate systems 1.1 The application shall support coordinate systems’ conversion 1.2 The application shall allow users to find a place using coordinates 1.3 The application shall allow the conversion of coordinates to a readable address 2) The application shall support multiple distance units 3) The application shall support multi-language 3.1 The application shall be adapted to the user language selection 3.2 The application shall provide name places in maps according to the user language 3.3 The application shall allow users to find a location using voice search in their preferred language 4) The application shall allow Geo Positioning in the world wide 4.1 The application shall implement the WHAT3WORDS locator 4.2 The application shall allow users to find location using place name 4.3 The application shall allow the conversion of a point in map to a readable address 4.4 The application shall implement the find nearby feature 4.5 The application shall allow users to find routes 4.6 The application shall allow users to customize routes Reliability requirements 1.1 The application shall allow users to define data integrity constraints 1.2 The application shall inform users about errors regarding data quality 2) The application shall inform users about the accuracy of data 2.1 The application shall display current GPS Satellites’ number 2.2 The application shall display GPS Satellites status 2.3 The application shall display user position accurately Performance efficiency requirements 1) The application shall use the device resources 1.1 The application shall use the GPS system 1.2 The application shall use the GNSS resource 1.3 The application shall use the compass feature 2) The application shall allow user to parameter the capacity 2.1 The application shall use the device local storage 2.2 The application shall allow users to clear data cache 2.3 The application shall allow users to check internal memory space (continued)
A Requirements Catalog of Mobile Geographic Information System
329
Table 2. (continued) Internationalization requirements 3) The application shall synchronize user position with the marker in the map within an acceptable time Sustainability requirements 1) The application shall have a positive individual impact 1.1 The application shall respect users’ privacy and security 1.2 The application shall include different types of animations 2) The application shall operate in diverse mode 2.1 The application should allow offline mode 2.2 The application should allow night mode 3) The application shall have a positive environmental impact 3.1 The application should allow user to display points of interest 3.2 The application should display points of interest review 4) The application shall allow interaction between users 4.1 The application shall allow users to share their current position 4.2 The application shall support the messaging feature 4.3 The application shall have an SOS message button 4.4 The application shall allow users to share routes 4.5 The application shall provide information about travel 4.6 The application shall allow users’ rating 5) The application shall be able to connect to different web map services 5.1 The application shall support OGC standard with regards web services 5.2 The application shall support remote access to databases 5.3 The application shall support ESRI services 5.4 The application shall integrate dropbox Usability requirements 1) The application shall allow users to change their visualization preferences 1.1 The application shall allow users to switch layers 1.2 The application shall allow users to customize graphical properties 1.3 The application shall allow users to clear last reference 1.4 The application shall support the labelling feature 1.5 The application shall allow the users to center the map to their positions 2) The application shall keep the user informed 2.1 The application shall display map (continued)
330
B. El Fhel et al. Table 2. (continued)
Internationalization requirements 2.2 The application shall display user location 2.3 The application shall display map Scale 2.4 The application shall display device speed 2.5 The application should display coordinates
functional requirements and quality attributes with respect to usability, performance efficiency, i18n, reliability and sustainability for mGIS for data collection. Requirements related to usability and performance efficiency are structured in the Subsects. 3.3 and 3.4, respectively. The remaining requirements are organized in the Subsect. 3.7. 3.3 Illustration In order to validate GIS apps for data collection using the developed catalog, an existing app in the Google Play Store called “Locus GIS - offline geodata collecting, SHP edits” was evaluated. This application was installed in more than 100000 devices, lastly updated on 16 November 2020 and the evaluation was performed on 20 November 2020. Note that on the day of the evaluation, this application was rated 4.2 stars. The evaluation was performed based on a questionnaire of 44 items as shown in Table 3. This assessment questionnaire was developed according to the requirements in the catalog. The application was installed on the author’s phone (Android) and a checklist was generated in response to the questionnaire. The evaluation was conducted by the first author and verified by the second. Note that some requirements that are not applicable to this application have been discarded. Each question in the assessment questionnaire was answered by Yes (1 point) if the functionality/feature is supported, No (0 points) if the application does not provide this feature/functionality or partially (0.5 point) when the attended feature/functionality is not fully satisfied. The results show that 32 functionalities or features (72.73%) in the assessment questionnaire were fully provided by the application. Partially, only the question about the OGC web services is satisfied (2.27%). The results also show that 11 questions in the assessment questionnaire were answered by No, which means that 25% of the functionalities/features are not supported. Finally, the total score of the application “Locus GIS - offline geodata collecting, SHP edits” is calculated as follow: (32 * 1 + 1 * 0.5 + 11 * 0)/44 = 73.86% The application obtained a very high score of 73.86%. Note that requirements in the questionnaire related to the usability are implemented at 100%. Requirements about Reliability are supported at 83.33%, therefore the requirements about spatial data integrity needs to be considered. The assessment show that requirements about Sustainability (50%) and i18n (29.41%) needs to be developed for this application. Thus, the requirements catalog can be useful for developers to identify missing functionalities in order to improve this application.
A Requirements Catalog of Mobile Geographic Information System Table 3. SRS outline (IEEE 29148:2011) 1. Introduction 1.1 Purpose 1.2 Scope 1.3 Product overview 1.4 Product perspective 1.5 Product functions 1.6 User characteristics 1.7 Limitation 1.8 Definitions 2. References 3. Specific requirements 3.1 External interfaces 3.2 Functions 3.3 Usability requirements 3.4 Performance requirements 3.5 Logical database requirements 3.6 Design constraints 3.7 Software system attributes 3.7.1 Reliability 3.7.2 Availability 3.7.3 Security 3.7.4 Maintainability 3.7.5 Portability 3.7.6 Internationalization 3.7.7 Sustainability 3.7.8 Acceptability 3.8 Supporting information 4. Verification 5. Appendices 5.1 Assumptions and dependencies 5.2 Acronyms and abbreviations
331
332
B. El Fhel et al.
4 Discussion According to the results of the evaluation, the application “Locus GIS - offline geodata collecting, SHP edits” implements most of the functionalities/features in the assessment questionnaire, especially, the usability requirements. However, the assessment shows that 25% of requirements are not fulfilled. These questions are: • Q5: Taking geo-located pictures is a functional requirement that shall be fulfilled to allow users to collect data in raster mode, geo-located images can be useful in projects like monitoring construction site. • Q8: Spatial data analysis is a crucial function in mGIS for data collection, minimum features of data analysis like intersection and buffering shall be provided to users. • Q16: Search location using place name is important to help users to find places. • Q17: Nearby search should be provided, this functionality helps users to operate in word wide and locate important places if needed. • Q19: Data collection should be performed carefully to avoid returning to the field, the application should allow users to define integrity constraints to verify data according to their needs (Table 4). • Q25: The GNSS should be supported to improve the accuracy of the positioning. • Q28: The application should allow user to clear data in cache, this functionality is important to manage storage resources and free up space upon the mobile device. • Q29: The application should allow users to check if the internal memory space is free enough to store data collected. • Q33: The application should allow users to display points of interest to facilitate mobility during data collection in the field. • Q34: The application shall display points of interest review to help users to find places which mash their needs. • Q35: The application shall allow users to share their positions to keep interaction between them.
A Requirements Catalog of Mobile Geographic Information System
333
Table 4. Assessment questionnaire ID
Question
Result
Q1
Does the application allow drawing on the map?
Yes
Q2
Does the application allow markers edition?
Yes
Q3
Does the application allow feature attributes access?
Yes
Q4
Does the application user to save here/her position?
Yes
Q5
Does the application user to take Geo Located Photos?
No
Q6
Does the application user to record Track?
Yes
Q7
Does the application distance and area measure?
Yes
Q8
Does the application spatial data analysis?
No
Q9
Does the application data filter?
Yes
Q10
Does the application support known spatial File Format?
Yes
Q11
Does the application support multiple coordinate systems?
Yes
Q12
Does the application allow search place by coordinates?
Yes
Q13
Does the application adapt its content to the user’s linguistic preferences?
Yes
Q14
Does the application names places in maps according to the user language?
Yes
Q15
Does the application allow measurement units selection?
Yes
Q16
Does the application allow search location using place name?
No
Q17
Does the application allow nearby search?
No
Q18
Does the application allow user to apply data quality assurance?
Yes
Q19
Does the application allow user to define the data integrity constraints?
No
Q20
Does the application inform user about errors regarding data quality?
Yes
Q21
Does the application inform user about the accuracy of data?
Yes
Q22
Does the application display GPS Satellites status and number?
Yes
Q23
Does the application display user position accuracy?
Yes
Q24
Does the application use the GPS?
Yes
Q25
Does the application use the GNSS resource?
No
Q26
Does the application use the COMPASS?
Yes
Q27
Does the application use the device local storage?
Yes
Q28
Does the application allow clear data cache?
No
Q29
Does the application user to check internal memory space?
No
Q31
Does the application allow privacy policy access?
Yes
Q32
Does the application allow offline mode?
Yes
Q33
Does the application allow user to display points of interest?
No (continued)
334
B. El Fhel et al. Table 4. (continued)
ID
Question
Result
Q34
Does the application allow/display points of interest review?
No
Q35
Does the application allow user position share?
No
Q36
Does the application allow application rating?
Yes
Q37
Does the application support OGC standard regarding web services?
Partially
Q38
Does the application allow user to customize visualization preferences?
Yes
Q39
Does the application allow layers switch?
Yes
Q40
Does the application allow user to custom graphical properties?
Yes
Q41
Does the application allow user to center map to his/her position?
Yes
Q42
Does the application display map?
Yes
Q43
Does the application display map scale?
Yes
Q44
Does the application display coordinates?
Yes
5 Conclusion and Future Work In this study, a requirements catalog regarding mGIS applications for data collection was carried out. This catalog focus on requirements about usability, i18n, performance efficiency, reliability and sustainability. In order to develop the catalog, previous studies and recommendation standards were explored. An illustration was presented to show stakeholders how to apply the catalog to evaluate their applications. As future work, we intend to conduct an audit on a set of GIS apps for data collection, to analyze their degree of compliance with the requirements of our catalog. The present catalog can also be improved by proposing a prioritization requirements method.
References 1. Hunter, A.: Mobile GIS as if field users mattered [microform]: small is ubiquitous but can speech be recognized (2019) 2. Chen, R.: Ubiquitous Positioning and Mobile Location-Based Services in Smart Phones. IGI Global, Hershey, pp. 1–361 (2012) 3. Jayasinghe, A., Sanjaya, N., Chemin, Y.: Application of mobile GIS for mobility mapping (2014) 4. Brovelli, M.A., Minghini, M., Zamboni, G.: Public participation in GIS via mobile applications. ISPRS J. Photogrammetry Remote Sens. 114, 306–315 (2016) 5. Goodchild, M.F.: Citizens as sensors: the world of volunteered geography. GeoJournal 69, 211–221 (2007) 6. Zhizhong, M., Yuansong, Q., Lee, B., Fallon, E.: Experimental evaluation of mobile phone sensors. In: 24th IET Irish Signals and Systems Conference, pp. 1–8 (2013) 7. Lu, Z.: Reference ellipsoid and the geodetic coordinate system. In: Geodesy, pp. 165–263 (2014)
A Requirements Catalog of Mobile Geographic Information System
335
8. Drummond, J., Billen, R., Joao, E., Forrest, D.: Dynamic and mobile GIS: investigating changes in space and time (2007) 9. Xu, Z., Xie, Z.: « Research on key technology of general embedded GIS”. J. Geograph. Inform. Syst. V2, 15–18 (2010) 10. Nowak, M.M., Dziób, K., Ludwisiak, Ł., Chmiel, J.: Mobile GIS applications for environmental field surveys: a state of the art. Global Ecology and Conservation, p. e01089 (2020) 11. Poorazizi, E., Alesheikh, A.A., Behzadi, S.: Developing a mobile GIS for field geospatial data acquisition. J. Appl. Sci. 8, 3279–3283 (2008) 12. Chen, X., Zhao, J., Bi, J., Li, L.: Research of real-time agriculture information collection system based on mobile GIS. In: 2012 1st International Conference on Agro-Geoinformatics Shanghai, pp. 1–4 (2012) 13. Ye, S., Zhu, D., Yao, X., Zhang, N., Fang, S., Li, L.: Development of a highly flexible mobile GIS-based system for collecting arable land quality data. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 7(11), 4432–4441 (2014) 14. Shadin, M.S., Tahar, K.N.: The implementation of mobile GIS for fire hydrant mapping. In: 2015 International Conference on Space Science and Communication (IconSpace), Langkawi, pp. 65–70 (2015) 15. Chen, H., Xiao, K.: The design and implementation of the geological data acquisition system based on mobile GIS. In: 19th International Conference on Geoinformatics, pp. 1–6 (2011) 16. Yuan, M.: Geographical Information System and Science. 2nd edn. Part. 2, pp. 74–80 17. Wang, F., Mäs, S., Reinhardt, W., Kandawasvika, A.: Ontology based quality assurance for mobile data acquisition. In: EnviroInfo, pp. 334–341 (2005) 18. Wang, F., Reinhardt, W.: Spatial data quality concerns for field data collection in mobile GIS. In: Proceedings SPIE 6420, Geoinformatics 2006: Geospatial Information Science, p. 64201C (2006) 19. ISO 9241–11: International Standard, ISO 9241-11, Ergonomic requirements for office work with visual display terminals (VDTs) (1998) 20. Hornbæk, K.: Current practice in measuring usability: challenges to usability studies and research. Int. J. Hum. Comput. Stud. 64, 79–102 (2006) 21. Elsidani Elariss, H., Khaddaj, S.: A time cost optimization for similar scenarios mobile GIS queries. J. Vis. Lang. Comput. 23(5), pp. 249–266 (2012) 22. Shi, W., Kwan, K., Shea, G., Cao, J.: A dynamic data model for mobile GIS. Comput. Geosci. 35(11), 2210–2221 (2009) 23. Steiniger, S., Weibel, R.: GIS software - a description in 1000 words (2010) 24. Asghar, S., Umar, M.: Requirement engineering challenges in development of software applications and selection of Customer-Off-The-Shelf (COTS) components. IJSE 1(1), 32–50 (2010) 25. Bourque, P., Richard, E.F., et al.: Guide to the Software Engineering Body of Knowledge (SWEBOK (R)): Version 3.0. IEEE Computer Society Press, New York (2014) 26. ISO/IEC/IEEE 29148: 2011 Systems and software engineering — Life cycle processes — Requirements engineering 27. Ouhbi, S., Alemán, J.L.F., Idri, A., Toval, A., Pozo, J.R., El Bajta, M.: A reusable requirements catalog for internationalized and sustainable blood donation apps. In: ENASE, pp. 285–292 (2017) 28. Ouhbi, S., Fernández-Alemán, J.L., Toval, A., Rivera Pozo, J., Idri, A.: Sustainability requirements for connected health applications. J. Softw. Evol. Process. 30, e1922 (2018) 29. Bachiri, M., Idri, A., Redman, L.M., Fernandez-Aleman, J.L., Toval, A.: A requirements catalog of mobile personal health records for prenatal care. In: International Conference on Computational Science and Its Applications, pp. 483–495 (2019)
336
B. El Fhel et al.
30. Ali, I., Shah, S.S.A.: Usability requirements for GIS application : comparative study of google maps on PC and smartphone. Dissertation (2011) 31. Miguel Montesinos, J.C.: GIS Mobile Comparison (2010). https://bit.ly/2MF8kqV. Accessed 20 Nov 2020 32. Lew, P., Zhang, L., Olsina, L.: Usability and user experience as key drivers for evaluating GIS application quality. In: 18th International Conference on Geoinformatics, pp. 1–6 (2010) 33. ISO/IEC 25010: 2011 System and software quality models 34. Web Service Common. https://bit.ly/2JSW9ps. Accessed 20 Nov 2020 35. ISO 19113: 2002 Geographic information — Quality principles 36. ISO/TS 19158: 2012 Geographic information — Quality assurance of data supply 37. ISO 9241-151: 2008 Ergonomics of human-system interaction — Part 151: Guidance on World Wide Web user interfaces 38. OGC Hierarchical Data Format Version 5 (HDF5®) Core Standard. https://bit.ly/3hODh7G. Accessed 20 Nov 2020 39. ISO19133: 2005 Geographic information -Location-based services-Tracking and navigation
Keeping the Beat on: A Case Study of Spotify Inês Gomes1 , Inês Pereira1 , Inês Soares1 , Mariana Antunes1 , and Manuel Au-Yong-Oliveira2(B) 1 Department of Languages and Cultures, University of Aveiro, 3810-193 Aveiro, Portugal
{ines.m.gomes,inesmaria.p,i.soares,mariana.antunes}@ua.pt 2 INESC TEC, GOVCOPP, Department of Economics, Management, Industrial Engineering
and Tourism, University of Aveiro, 3810-193 Aveiro, Portugal [email protected]
Abstract. The music industry has faced a tremendous change ever since the emergence of streaming. People now had access to unlimited music to listen to and share with others, which is a great concept. As companies started to invest in the future of streaming, many platforms were created, one of them being Spotify. Standing amidst a crowd of competitors, Spotify managed to climb to the top and comfortably remain there over the years (listed on the NYSE and having 120 employees; IPO share price (3rd April 2018) = 169.9 US Dollars and share price of 319.77 US dollars on 4th December 2020). However, it was not easy. Being one of the top streaming platforms in the industry requires effort, investment, adaptation and innovation, and exceptional management. This study analyses in-depth the aforementioned aspects, as well as the company’s business models and its revenue, which combined, led Spotify to where it is today. A survey with 498 answers was performed regarding Spotify, and the results were analyzed by using descriptive and inferential (Chi-Square test) statistics. We thus conclude very confidently that there does seem to be an association (statistically significant at the 0.1% level) between age and the use of Spotify. Younger respondents (18–35 years) are more likely to use Spotify than older respondents (> 35 years). Overall, Spotify’s business model is one of the best in its field and is constantly evolving, which is what is to be expected of a company that seeks the top spot on the podium. Keywords: Spotify · Streaming platforms · Business models · Freemium · Premium
1 Introduction “Thanks for listening to Spotify. No, really you could’ve listened to the radio. You could’ve spun some vinyl. You could’ve played a cassette tape. You could’ve listened to an 8 Track tape if you knew what an 8 Track tape looked like, but you listen to Spotify. Thanks for that and you still have hundreds of more playlists to enjoy.”– Spotify commercial quote [this is a direct transcription from a Spotify ad]. Since the appearance of streaming, society became more demanding and eager to know how it can take the next step towards innovation and improvement. Streaming is © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 337–352, 2021. https://doi.org/10.1007/978-3-030-72651-5_33
338
I. Gomes et al.
now a new reality growing every minute, and companies started to spend more money on streaming services to adapt to this new era. The music industry was one of the most capable to evolve with the new tendencies– it adapted itself over time. From radio to streaming, the music industry showed us how good it is to be flexible and how beneficial it can be to think alongside advancement, and not against it. After the CD, the music industry had to adapt once more, due to the increasing levels of piracy - a result of digital development - and then streaming platforms emerged. In this paper, our main focus will be on analyzing Spotify’s business model, and how they make it work while using a freemium model, how they managed to be successful, how it was created, and how they can improve. The literature review will mainly focus on streaming music services and why they have grown and what made streaming platforms a blooming service. A survey was also launched, mainly to see how Spotify customers react to the service provided. The analysis of this survey was done using descriptive statistics, as well as inferential statistics (the chi-square test of independence).
2 Literature Review 2.1 Online Music and Streaming Platforms In the past 20 years, we watched a significant increase in online streaming platforms’ popularity due to the massive influence of digitalization in the commercial music market. “Online streaming platforms now represent one of the most significant distribution channels for recorded music, accounting for more than half of the industry revenues worldwide” [1]. The most recent global music report showed that “streaming revenue grew by 34.0% and accounted for almost half (47%) of global revenue, driven by a 32.9% increase in paid subscription streaming. There were 255 million users of paid streaming services at the end of 2018 accounting for 37% of total recorded music revenue” [2]. “Today, the only growing relevant format of music consumption is streaming” [3]. Since the consumer became more demanding due to the large diversity and easy access, streaming gained some power over the digital CD physical format. Streaming platforms provide the consumer with a large diversity of libraries of songs that could be listened to at any time and place [3] while buying CD’s only allowed us to listen to whatever they were about, and the consumer had to have a device that could read the CD format. According to [4] Spotify and Apple which “offer at least a 30 million-song library”. Nowadays people do not buy CD’s, and soon they will become as obsolete as tapes. For how much longer will the CD industry be maintained? Despite the increased use of streaming platforms, “physical formats still occupy a fraction of the market” [5]. 2.2 Music Consumption– Traditional vs Digital Innovation and the fast development of technology have had an incredible impact on the way music is consumed by people. A direct consequence is “the flourishing of digital music and the development of internet-based services such as mp3, peer-to-peer
Keeping the Beat on: A Case Study of Spotify
339
networks, and online music stores” [6]. Over the past two decades, right after embracing the new world of digital music, CD’s and audiotapes sales watched a huge decline. The shift from buying a physical album to downloading digitally released songs is now almost everyone’s favorite way to consume music [5]. During the first ten years of the new millennium, with the mp3 music format, the release of the iPod, and broadband internet connections, online music format distribution had an incredible increase. This was proved in 2008 when “(…) Apple’s online music store iTunes became the largest seller of music (including physical music formats) in the USA.” [6]. “iTunes was a radical change in the music industry. It was the first online retailer that was able to offer the music catalogs from all the major music companies, it used an entirely novel pricing model, and it allowed consumers to de-bundle the music album and only buy the tracks that they actually liked” [7]. It has not been a good time for physical music. It started declining and we cannot see a light at the end of the tunnel for traditional music consumption to come back. At the beginning of the millennium, physical music was running the music industry revenue but, by 2010, it started declining until 2015 when digital music took over the rest of it and it has been an increasing world since then [8]. However, “online streaming platforms have become one of the largest components of the music industry, accounting for 43% of digital music sales in the USA in 2015, while download revenues decreased by 10.5%” [5]. Spotify is the best and most unique example to prove this as it has been replacing legitimate digital downloading since it was born in 2006. Comparably, in 2015, Apple launched a paid streaming service (Apple Music) [5]. Digital music consumption “began the decade in a very different place to how it ended it” [8]. Digital downloads and piracy were the main reason for the almost ‘extinction’ of physical music sales. However, it has taken on many ways and shapes throughout the journey. First, iTunes was the key to its early success, yet by the end of the decade Apple, aiming to focus only on its streaming service, launched Apple Music. By that time iPod “was still a popular tool for music consumption” that saw its end in 2015 when consumers moved to smartphones, even though “50.31 million iPods were sold in 2010” [8]. The shift, from CD’s to Spotify, “has brought about a mix of challenges and opportunities for practitioners, altering the relationship between digital and physical music consumption” [5]. 2.3 Business Models on Streaming Platforms The magic world of online streaming contains a wide range of business models that allow companies, in this case streaming platforms, to convert their listeners into fixed and loyal customers. To achieve this, the target audience enters the market following the two most used business models by music streaming platforms. A good example is Google, Apple, and Netflix that began to offer only ad-free plans and, on the other side of the coin, services like Spotify or Deezer following a mixed free and paid business model [4].
340
I. Gomes et al.
Freemium is undeniably the most popular business model used in web services. This model allows digital platforms to provide their customers with both the free version and the paid version. The free version is known as the “method of attracting users by giving a percentage of your product or service for free and offering a more interesting, deeper experience for a price” [9]. On the other hand, the paid version offers the full experience to customers and, at the same time, means revenue to companies [10]. One of the benefits of using freemium is that free services tend to reach a very large number of users, which, eventually, culminates in the notably famous and powerful marketing tactic: word-of-mouth communication. Former research that aimed to identify the most important element that affected customers’ willingness to pay for online music services showed that the price “was identified as the most crucial attribute” [11]. The power of the network should not be underestimated, in marketing terms. People especially trust people they know so a word-of-mouth recommendation will mostly be more trustable than advertising [9, 10]. “The so-called basic subscription entails frequent commercial interruptions after a few songs. Somehow, users are compensated for the nuisance of ads with free access to music. Contextually, users are allowed to upgrade to a paying solution with quality improvements and the absence of commercial interruptions. This business model is commonly called Freemium” [4]. A study around the effects of music streaming showed that “free streaming services - because they do not offer users full mobility in their music consumption - can lead to a stimulation in alternative music consumption channels that offer mobility, such as licensed and unlicensed downloading” [12]. Alternatively, the famous premium and high-quality plan, is nothing but a “purely subscription-based business model” [4]. Companies like Apple and Google Music have adopted this model in which people have 100% access to the offered platform music stock, without any ads or commercials. “The absence of commercials is usually associated with quality improvements” [4]. These two different business models have different advantages. The freemium free model can be more attractive to new consumers, yet the premium model is often more profitable [11]. For a streaming online industry, the key to success depends on consumer preferences, and this will define which business model will bring success. It is necessary that companies must understand their consumers’ insights to bring value and formulate the most optimal business model [11]. Price, above all, is paramount regarding online music services [11]. 2.4 Piracy With the advancement in technology surrounding the Internet, making it available and accessible, digital forms of music have expanded. The MP3 format became popular, but the mainstream recording industry (e.g., EMI, Sony, BMG, Universal, and Warner Bros) did not take advantage of this knowledge and early online music sites were too difficult to use [13]. When Napster appeared in 1999, it made it easier for users to download and share files. However, the users only paid for the program itself, while no royalties were given to the artists. There was no need to buy expensive CD’s with limited music anymore, as people were able to download music into blank ones. A lot of
Keeping the Beat on: A Case Study of Spotify
341
legal problems came from the misuse of Napster, and in 2001, it was declared to enable its users to commit piracy and it “was ordered to remove copyrighted material from its servers.” [13]. Digital piracy became the new global phenomenon due to “the public’s demand for fast, convenient, and easy-to-access content” [14]. Although legal actions took place to dismantle the illegal download of music and “discourage users from downloading unauthorized material” [14] this had little success and failed to “significantly reduce file-sharing traffic” [14]. Moreover, all of this brought severe consequences to the music, movies, and software industry revenues. Several industries had to adapt to and innovate with a new business model to be able to survive. With the problem surging in the digital world, companies like Netflix, Apple, and Spotify adapted to streaming platforms to keep their customers happy and with the easy access to content that they desired. Both Netflix and Spotify were able to curb [14] “the rise of digital piracy, and (for the first time) managed to even reduce illegal downloads”. Daniel Ek, the founder, and CEO of Spotify said that Spotify was created with the purpose to stop digital piracy. Although Spotify was created to, in a certain way, stop digital piracy, we can still see that people persisted to’illegally’ download music. Even the platform itself has been targeted, and people have managed to get Spotify Premium without paying, although we should consider this more a digital hack, than digital piracy.
3 Spotify Spotify is a streaming platform founded in 2006 in Stockholm, Sweden, by Daniel Ek and Martin Lorentzon. Their purpose was to develop a legal digital music platform to respond to the growing piracy the music industry was facing in the early 2000s. The platform is a music-streaming service providing both free and premium services [15]. Free Spotify access comes with lower sound quality, advertisements, and requires an internet connection to play music. Premium access comes with the payment of a monthly fee and users can listen uninterruptedly, without needing to have an internet connection and are also able to download songs to any device that has the Spotify app. To find its initial users, Spotify contacted influential musicians in Sweden, asking them to try out the new app. This was their initial strategy and it proved to be a success. The initial users were called beta testers and were awed at how good the company’s product was, even though it was at an early stage of development. In one year, Spotify had an app that musicians were excited about, and so they helped in spreading the news of the new music app. Having raised several million US dollars in funding, Spotify invested almost all that money into hiring top engineering talent, to keep up with the development of the app. In 2007, Ek and Lorentzon approached Pär-Jörgen Pärson, a general partner of venture capital fund Northzone, to invest in the app. “They had a level of ambition that I hadn’t seen before in Sweden and were very aware of what they were building. They weren’t settling for the good enough; they were going for the best. They were looking for the best people, the best talent, and were not easily impressed by the regular giants.” [16].
342
I. Gomes et al.
While many startups believe the only way to evolve is to grow quickly, Spotify preferred to focus on the development of a consistent and robust product and growing slowly by restricting how many invites users could give away to their friends. This was also a way of creating and maintaining a certain exclusivity around the app. In a field of constant innovation, Spotify is pressured to adapt and keep up with the new trends, whilst fulfilling the needs of its users. Spotify has also improved the app and created new features that no other competitor has, to keep the lead. Moreover, the company has been collecting users’ feedback and considering it and is analyzing the ideas described below. 3.1 Karaoke Feature As of right now, Spotify was granted a patent for a new feature that allows users to “overlay a music track with their own vocals” [17] through “devices, systems, and methods for overlaying audio data for user vocals and media content received from distinct devices and systems” [18]. The vocals would be “captured using a microphone of a client device” [18] and then “transmitted to a media presentation system, while corresponding media content, such as a music track, is transmitted to the media presentation system from a remote server distinct from the client device” [18]. Which means, in other words, that the users could use their phones as the device, and the “media presentation system” [17] could be a television or speakers. 3.2 Tracking Behavioral Patterns The company has also been granted a patent for “methods and systems for personalizing user experience based on [user] personality traits.” [19], arguing that “there is a need for systems and methods for personalizing media content in accordance with one or more personality traits associated with a user” [19]. It also states that “it is possible to identify a personality trait of a user based on the content (e.g., music) the user consumes (e.g., listens to) and the context in which they consume the content.” [19], and that “behavioral variables such as a user’s mood, their favorite genre of music, or their demographic can all correspond to different personality traits of a user” [19]. To do this, they would use questionnaires, such as the “Big Five Inventory” or the “MeyersBriggs personality survey” [19]. With this knowledge, Spotify could promote content such as audio advertising, music, and podcasts, based on the users’ personality traits it has detected [20]. In conclusion, “for example, the tone of voice may be more upbeat, high-pitched and/or exciting for users that have been assigned the personality trait of extroversion” [19] and “the tone of voice may be quiet and/or soft-toned for users that have been assigned the personality trait of introversion (…) thereby improving the user experience” [19]. Therefore, in the future, Spotify may be able to use neuroscience to indirectly manipulate people into using the app. 3.3 Podcasting as a Key Driver for Future Growth As Spotify keeps investing in this trajectory, podcasting is becoming an indicator of future growth. 19% of the total monthly active users engage with podcast content, and
Keeping the Beat on: A Case Study of Spotify
343
as reported, consumption is growing at triple-digit rates [21]. Spotify’s CEO says that “podcast users spend almost twice the time on the platform, and spend even more time listening to music” [22]. As of 2019, Spotify invested in the podcast market by acquiring Gimlet Media and Anchor, two podcasting networks, which enhances the “potential to grow much faster with more original programming– and to differentiate Spotify by playing to what makes us unique – all with the goal of becoming the world’s number one audio platform.” [22]. 3.4 Evolution of the Platform: Will It Include Video? As of June 2020, Spotify announced the global launch of video podcasts. The new feature will allow, at launch, both free users and paid subscribers to watch the video content from a select group of creator podcasts. These video podcasts are supported on both the desktop and mobile app, and video will serve as an additional component [23]. The video and audio will be synced, and the audio will keep playing even if listeners lock their phones. Advert spots will still play as well, but with the video showing up as a single, static shot [24]. Video podcasts began rolling out on the 21st of June in supported markets, so the rest of the world should see it soon [23]. 3.5 Spotify as a Label: Representing Artists Spotify is now a key platform for artists to promote and advertise their music. Their biggest playlist, “Today’s Top Hits”, has over 26 million followers and more than 20 billion streams. It features newly released songs, the playlist image is changed to the top artist of the moment, and it has a “top of the Hottest 50” chart that artists use for reference of how good their song is doing. Due to its popularity, the playlist alone is shaping music culture and has already helped kickstart the careers of many artists [25]. In addition to this, artists have access to dashboards that allow them access to data related to their fans and other artists, helping them with decision-making and marketing strategies when it comes to planning a song and/or album releases and upcoming tours [26]. This being said, with its reputation and visibility, the question arises: will Spotify become a label and represent and promote artists? 3.6 Business Models When analyzing Spotify’s business model, there are crucial elements to focus on. The company’s strategy to be profitable, which includes the Premium plans and the way it manages advertisements; and the general revenue it is getting from it. There are a few ways the company found to make money. As soon as the app is opened on any phone or laptop, the first noticeable thing is the ads. However, as predominant as they can be, they are not the reason behind Spotify’s scale. The main source of income is the subscription model, more specifically, Premium. The Premium and Ad-Supported Services work independently, yet they are critical to each other [21]. Spotify makes 91% of its revenue from subscriptions, and the other 9% comes from advertisements. Out of the revenue it generates, Spotify keeps 30% and splits the remaining amount between licensing, music deals, and paying the artists [27].
344
I. Gomes et al.
When it comes to the Premium plans, Spotify has five plans to offer: the Free version, which gives you access to podcasts, videos, and songs on shuffle-only playback (on mobile) and has unlimited ad-supported streaming; Premium Individual offers ad-free streaming with extreme quality soundtracks (320 kbps), offline listening, and on-demand playback, for 6.99 EUR/month; Premium Duo, which has the same features but for two premium accounts, and generates the “Duo Mix”, a regularly-updated playlist for two, for 8.99 EUR/month; Premium Family also offers ad-free and offline streaming, on-demand playback, and extreme quality soundtracks (320kbps), for up to 6 premium accounts for family members living under the same roof, with the particularity of allowing to block explicit content and offering a family mix, all for 10.99 EUR/month; and lastly, the Premium Student Version, for 3.49 EUR/month, is the same as the individual version, but cheaper for university students [28]. All Premium plans come with an offer period of 1 month for free. On the web version, you can listen to their content for free with no shuffle nor skipping restrictions, and Spotify takes that feature away on mobile. In addition to this, they do bi-annual campaign promotions, offering deals to get Spotify Premium for a fraction of the regular price, or even for free, during a specific time (usually threemonths long). Most of these will only be available for new users who have never had a Premium account, to attract new customers. An interesting particularity is that once you register with a trial offer, the Premium subscription will automatically continue after the trial period expires, so if you do not want to be charged for it, you have to cancel before it ends. By doing all the above, they create a habit in the users, who then find it hard to cope with the limitations and are more likely to pay for one of the Premium plans, driving the growth of paid memberships [21]. As far as advertisements go, Spotify’s ad-supported service has no subscription fees, offering limited on-demand online access to their catalog. The ad-supported service is a critical ingredient. It is crucial for Spotify in terms of the acquisition of paid members, while being a viable option for users that cannot afford the paid plan. Through this service, the company monetizes from ad-types such as the sale of display, audio, and video advertising delivered through advertising impressions [21]. More specifically, Spotify makes use of the following types of advertisement: Branded Moments, where brands can get the users’ attention by choosing a specific moment in which they can watch their ad in exchange for 30 min of ad-uninterrupted music; Sponsored Sessions, which are similar to Branded, but more effective for the brands, as the users have to click the display in order to be able to enjoy the 30-minute listening session; Audio, that consists in showing a clickable banner while the audio ad is playing for 30 s; Video Takeover, for computer devices only, gets the users’ attention with a full focus video ad; Overlay, in which Spotify shows a full-page image ad when the app is opened after being minimized; Homepage Takeover, shows a full-width banner right on the home screen; Sponsored Playlist, in which the playlist image is replaced by an ad for the duration of the playlist; Leaderboard, a 30-s clickable banner ad [29]; and Welcome Back Ads [30], shown when the users return to the app. Spotify has around 329 million users, 144 million of which are Premium users [31]. However, it has never been known for its profit. Ever since the beginning in 2008 up until 2018, the company was never able to make a profit. This seems to be a common
Keeping the Beat on: A Case Study of Spotify
345
problem among streaming services, as their competitors, for instance, Apple Music and Amazon, have been struggling with the same issue. Even though they are generating over $1 billion in annual revenue, they all are far from profitable. A couple of years ago, Generator Research published an analysis that concluded that “no current music subscription service…can ever be profitable, even if they execute perfectly” [32] Despite this, in the last quarter of 2018, Spotify was able to profit for the first time [33]. However, according to experts, this profit did not come to last, even though it is a big accomplishment not only for Spotify but for the music industry as well [31]. Spotify has since 3rd April 2018 been listed on the NYSE. The IPO share price was 169.9 US dollars and on 4th December 2020 that value had practically doubled to 319.77 US dollars, which is an indicator of an expectation of future profits.
4 Methodology This paper is a result of the desire and curiosity to understand how a company like Spotify, which at first glance does not have a big presence in advertising, attracts so many users, as well as how it can profit from having a freemium option [9]. It also caught our attention due to the increasing use of streaming platforms, and how people resort much more to them than normal TV or CD’s in today’s day and age. The topics mentioned were searched in scientific databases, using specific keywords related to the subject, such as Streaming platforms, Spotify, freemium, and business models. Some other online research was also performed on websites specific to the subject at hand. The Spotify platform was also studied, in which the focus was mainly on: a) how the company started [15, 16], b) how it works, and c) their business model [21, 27, 28], and d) how the company is innovating to keep their customers happy, and finally e) how they are keeping up with the latest trends [23, 24], showing their capacity to adapt to market demands. Our survey consisted of twenty questions, the first ones being related to personal and demographic information, such as age, gender, and nationality. The other questions were based on receiving respondents’ opinions and perspectives on consumption behavior regarding Spotify. The questions were structured in the following formats: multichoice, Likert scales, open-ended questions, and dichotomous yes or no answers. All the questions were tested between the workgroup and validated by our supervisor before the survey was launched to the public. The survey was released between the 30th of October 2020 and the 7th of November 2020. We disclosed it through personal contacts and social media. The sample is thus a convenience sample (accessible to the researchers), quite prominent in business and management research [36], though not allowing for definitive findings [36]. We were highly motivated by the fact that we found a lot of information and studies about Spotify, with very different points of view and opinions. To accomplish our objective we are going to analyze some of the most relevant questions according to our Literature Review, by doing descriptive and inferential statistics, more specifically, the Chi-Square test, according to [34, 35], where the question that is going to be tested is “does age affect the use of Spotify”?
346
I. Gomes et al.
5 Results and Discussion With our survey, a sample of 500 answers was obtained, from which two answers were removed due to a lack of accuracy. Consequently, 498 answers were validated. Taking into consideration that the survey mainly had responses from ages between 18 to 25 years, and that it also had a majority of answers from Portuguese people, this could be a limitation regarding Spotify’s user-spectrum. We will use this to our advantage, in the sense that our sample will show promising results regarding the age range of 18 to 25 in relation to another factor, as will be shown during the analysis using the Chi-Square test. Another limitation is that not everyone answered all the questions, so the universe of respondents varies from question to question. To discuss and analyze the results, descriptive statistics were used as well as inferential statistics (the chi-square test of independence). Firstly, using descriptive statistics, the survey sample comprises an age range of under 18 to over 65 years. Additionally, 85% of the sample uses Spotify, of whom 75.4% are between the ages of 18 and 25 years old (Table 1). Only 15% of the sample do not use Spotify. Since this survey was created by Portuguese students, the majority of answers, 92.7%, was also from Portuguese people. Some other nationalities are represented (there are 15 nationalities in total), such as Brazilian, with 4.3% of the answers. Table 1. Ages that use spotify Age < 18
No. People Percentage 26
6.1%
18–25 319
75.4%
26–35
29
6.9%
36–45
22
5.2%
46–55
20
4.7%
56–65
6
1.4%
> 65
1
0.2%
Total: 423
100%
When asked about having Spotify Premium, only 481 out of the 498 participants answered. From that sample, 184 declared to have it, meaning that only 184 are paying for the service (Table 2). 62% use the Free plan. Out of the Spotify Premium users, it was verified that about 83% of the users are between the ages of 18 and 35 years old, from which a large percentage of 73.9% are between 18 and 25 years old (Table 2). Out of those who pay for the Premium plan, 45% have the Family plan, 32% have the Student plan, 20% have the Individual plan, and 4% have the Duo plan (Table 3). Since it has been seen that most of Spotify’s profit comes from subscriptions, these numbers present interesting evidence. Although there is a large incidence of answers from student participants, it is verified that they find it more advantageous to share a
Keeping the Beat on: A Case Study of Spotify
347
Table 2. Ages of people who have Spotify premium Age
No. People Percentage
< 18
7
3.8%
18–25 136
73.9%
26–35
16
8.7%
36–45
11
6.0%
46–55
10
5.4%
56–65
4
2.2%
Total: 184
100%
Table 3. Types of plans that Spotify Premium users chose Plans Duo
No. People Percentage 7
4%
Student
59
32%
Family
82
45%
Individual
35
19%
183
100%
Total:
Family Premium plan, than to use the Student one. Taking this into consideration, due to a large number of answers from Portuguese people, it is normal, in the Portuguese culture, that students stay at home until they finish their studies and start to work and/or get married, due also to the low salaries registered in the country [38], which means that financial independence will often only be reached later in life. We also verified that besides Spotify, people use other streaming platforms: out of a universe of 372 answers 70% uses YouTube, 27% uses YouTube and other platforms, and only 2% uses other streaming services that are not YouTube or Spotify, such as Deezer, TIDAL, etc. (Table 4). Table 4. Other streaming platforms used Streaming platform No. People Percentage YouTube
262
70%
YouTube + Others 102
27%
Others Total:
8 372
2% 100%
348
I. Gomes et al.
To analyze by way of the chi-square independence test, we have to put the values in a table to determine the expected values for each category [34, 35]. In this survey, 498 people answered the question “Do you use/or not use Spotify”. It was decided that it would be interesting to see if the demographic information regarding the age of the participants would show some promising results when mixed with the information regarding the use or not of the App. Therefore, the null hypothesis (H0) is “Age does not affect the use of Spotify” and the alternative hypothesis (H1) is “Age does affect the use of Spotify”. In Table 5 we can see the observed values of our survey. Table 5. Values observed (in the survey) Age 46
27
65.85%
14
34.15%
41
Total:
423
75
498
To calculate the expected value that is necessary to see if we can reject or not the H0, the following formula is used: Expected value = (Row total × Column total)/Grand total In this case, the sum of chi-square values is 24.069, but to check if our H1 is valid, we have to calculate the degrees of freedom, which is achieved through the following formula: (Number of columns – 1) × (Number of rows −1). In Table 5 there are 2 columns (uses/does not use) and 5 rows (5 age groups) of data, so we can apply that to the formula above to get our degree of freedom: (2–1) × (5–1) = 4. Hence, for this calculation, we have 4 degrees of freedom. The chi-square test statistic from Table 6 [34] is 24.069 which is greater than the critical value at the 5% significance level, which is 9.49. Additionally, the test statistic is also greater than the critical value at the 0.1% significance level, where the critical value is 18.47. With these results, we can reject our H0 (null hypothesis) that age does not affect the use of Spotify and approve the H1 (alternative hypothesis) that age does affect the use of Spotify. We thus conclude very confidently that there does seem to be an association between age and the use of Spotify (there is only 1 chance in 1000 of falsely rejecting the null hypothesis) [36]. We can also see row percentages in Table 5 – each cell is shown as a percentage of the total number in that row [36]. In this case we are presuming that age influences using or not using Spotify. Younger respondents (18–35 years) are more likely to use Spotify than older respondents (> 35 years). These findings confirm previous research by [37].
Keeping the Beat on: A Case Study of Spotify
349
Table 6. Expected values and the chi-square statistic O 26
EXP VALUE
(O-E)
(O-E)ˆ2
28.0301205
– 2.0301205
4.121389
(O-E)ˆ2/E 0.1470343
7
4.96987952
2.03012048
4.121389
0.82927346
319
304.084337
14.9156627
222.477
0.73162924
39
53.9156627
– 14.915663
222.477
4.12638891
29
28.8795181
0.12048193
0.014516
0.00050264
5
5.12048193
– 0.1204819
0.014516
0.00283487
22
27.1807229
– 5.1807229
26.83989
0.98746048
10
4.81927711
5.18072289
26.83989
5.56927711
27
34.8253012
– 7.8253012
61.23534
1.75835777
14
6.1746988
7.8253012
61.23534
9.91713782
TOTAL:
24.0698966
6 Conclusions and Future Research We have been witnessing, in recent decades, the evolution of technology and the shift to a digital world. All industries tried and are trying to go along and adapt to the fast development and increase of the digital and technological wave, and the music industry is included. It is one of the industries that is the most capable of evolving and adapting to new tendencies. From audio tapes and CD’s to mp3 and live streaming platforms, music has never stopped being innovative, creative, and adaptable. However, things were not always easy. For instance, at the beginning of the new digital era, problems like piracy expanded into a huge phenomenon due to the urgent feeling of wanting to access the most content possible. To survive, music industries had to adapt, and especially innovate by using new business models to replace the old ones. Spotify is one of the most successful examples, and it was created to respond to the growing music piracy phenomenon. By using a freemium business model, Spotify offers its consumers the opportunity to get access to an infinite music catalog. Regarding the benefits and the low cost of premium plans, it is more attractive to pay for the whole package, yet consumers still have the opportunity to access all of the content for free, with the disadvantage of listening to ads between songs. To understand the success Spotify has achieved just by using a freemium business model, a survey with 498 responses was performed. The results indicate that having a free and a paid option can bring profit and benefits for both the company and users. The authors of this study perceive that this business model is one of the best strategies for new millennial companies and has become very important recently. However, future research should be conducted to better understand what other business models could work for music streaming platforms, as nowadays things change very quickly and there is a constant need to adapt.
350
I. Gomes et al.
The fact that Spotify can do personalized playlists [19, 20] is something that the customers are really satisfied with. Regarding the current piracy linked to online streaming platforms and on how to erase it, research is lacking. This case study showed that a considerable number of users use a hacked version. To explain this, there is the fact that people are not willing to pay much money for the Premium plan when they can hack it and have it for free. A future study could focus on analyzing and understanding, if it is indeed possible, how to improve and adapt Spotify’s prices to get people to use the app legally and, preferably, the Premium plan. Additionally, a lack of information was felt whilst researching for evidence on the prices charged by Spotify, and the authors believe that the company would benefit from sharing such details, as consumers would be able to know the exact value of what they are paying for. In future studies, the sample may be extended to different generations and the appropriate tests may be performed to observe the same phenomena in relation to those generations (showing more depth in the older-than-25-year-old categories). The generations’ characteristics may provide greater explanatory power for the trends observed in the study of the use of digital platforms such as Spotify. Finally, the cultural factor may be used as an element of analysis. Perhaps, starting with an analysis between countries with similar historical trajectories as Brazil and Portugal and, in the sequence, extrapolating the analyses to quite distinct cultures such as North America, Eastern Europe and Asia. This could provide valuable information for the discussion of the phenomena under analysis.
References 1. Kaimann, D., Tanneberg, I., Cox, J.: “I will survive”: online streaming and the chart survival of music tracks. Managerial and Decision Economics, pp. 1–18 (2020) 2. IFPI: Global Music Report (2019). https://www.ifpi.org/ifpi-global-music-report-2019/. Accessed 28 Oct 2020 3. Fly, B.: How does music consumption impact the music industry and benefit artists? Accounting Undergraduate Honors Theses 20 (2016) 4. Carroni, E., Paolini, D.: Business models for streaming platforms: content acquisition, advertising, and users. Inf. Econ. Policy 52, 1–13 (2020) 5. Lee, M., Choi, H.S., Cho, D., Lee, H.: Can digital consumption boost physical consumption? The effect of online music streaming on record sales. Decis. Support Syst. 135(6), (2020) 6. Magaudda, P.: When materiality “bites back”: digital music consumption practices in the age of dematerialization. J. Consum. Cult. 11(1), 15–36 (2011) 7. Wikström, P.: The Music Industry in an Age of Digital Distribution. OpenMind. https://www.bbvaopenmind.com/en/articles/the-music-industry-in-an-age-of-digitaldistribution/. Accessed 11 May 2020 8. Murphy, S.: Here’s how music consumption has changed over the last decade. The Music Network. https://themusicnetwork.com/music-consumption-decade/. Accessed 02 Nov 2020 9. Kingsnorth, S.: Digital Marketing Strategy: An Integrated Approach to Online Marketing. Kogan Page, London (2016) 10. Khong, T.: Customer perceived value freemium business model: Case study: Spotify in Finnish and Vietnamese markets (Master’s Thesis in Economics and Business Administration Master’s, University of VAASA) (2019)
Keeping the Beat on: A Case Study of Spotify
351
11. Kim, J., Nam, C., Ryu, M.H.: What do consumers prefer for music streaming services?: A comparative study between Korea and the US. Telecommun. Policy 41(4), 263–272 (2017) 12. Aguiar, L.: Let the music play? Free streaming and its effects on digital music consumption. Inf. Econ. Policy 41, 1–14 (2017) 13. Pelusi, M.: Napster Is Released. Salem Press Encyclopedia. http://widgets.ebscohost.com/ prod/customerspecific/ns000290/authentication/index.php?url=https%3a%2f%2fsearch. ebscohost.com%2flogin.aspx%3fdirect%3dtrue%26AuthType%3dip%2cshib%2cuid% 26db%3ders%26AN%3d89315521%26lang%3dpt-pt%26site%3deds-live%26scope% 3dsite. Accessed 30 Oct 2020 14. Aversa, P., Hervas-Drane, A., Evenou, M.: Business model responses to digital piracy. Calif. Manag. Rev. 61(2), 30–58 (2019) 15. Zennyo, Y.: Freemium competition among ad-sponsored platforms. Inf. Econ. Policy 50, (2020) 16. Farmbrough, H.: How This Swedish Venture Capitalist Became Spotify’s Lead Investor. https://www.forbes.com/sites/heatherfarmbrough/2018/03/27/how-this-swedish-venturecapitalist-became-spotifys-lead-investor/?sh=4bfd2e469075. Accessed 04 Nov 2020 17. Stassen, M.: New Spotify Patent Sheds More Light On Potential Karaoke Mode – Including Auto-Tuned Vocals - Music Business Worldwide. Music Business Worldwide. https://www.musicbusinessworldwide.com/new-spotify-patent-sheds-more-light-onpotential-karaoke-mode-including-auto-tuned-vocals/. Accessed 04 Nov 2020 18. Spotify AB, Stockholm (SE): Methods and Systems for Overlaying and Playback of Audio Data Received from Distinct Sources. U.S. Patent 10,771,521 B2 (2020) 19. Spotify AB, Stockholm (SE): Methods and Systems for Personalizing User Experience Based on Personality Traits. U.S. Patent 10, 798,214 B2 (2020) 20. Stassen, M.: Spotify Has A Patent For Personality Tracking Technology –And It’s Pretty Creepy Stuff - Music Business Worldwide. Music Business Worldwide. https://www.musicb usinessworldwide.com/spotify-has-a-patent-for-personality-tracking-technology-and-its-pre tty-creepy/. Accessed 10 Nov 2020 21. Cuofano, G.: How does Spotify make money? Spotify Freemium Business Model in a Nutshell. https://fourweekmba.com/spotify-business-model/. Accessed 04 Nov 2020 22. Spotify. Audio-First - Spotify. https://newsroom.spotify.com/2019-02-06/audio-first/. Accessed 11 Nov 2020 23. Perez, S.: Techcrunch is now a part of verizon media. https://techcrunch.com/2020/07/21/ spotify-launches-video-podcasts-worldwide-starting-with-select-creators/?guccounter=1. Accessed 11 Nov 2020 24. Carman, A.: Spotify is testing video podcasts with two youtube stars. https://www.theverge. com/2020/5/6/21249389/spotify-video-podcast-test-upload-content-test-youtube. Accessed 11 Nov 2020 25. Spotify. Spotify’S Biggest Playlist, Today’S Top Hits, Celebrates 25 Million Followers — Spotify. https://newsroom.spotify.com/2019-12-16/spotifys-biggest-playlist-todays-top-hitscelebrates-25-million-followers/. Accessed 04 Nov 2020 26. Google Cloud. Spotify Case Study| Google Cloud. https://cloud.google.com/customers/spo tify. Accessed 04 Nov 2020 27. Desai, M.: How Spotify makes Money - Business Model. https://medium.com/dissectingmusic-tech/how-spotify-makes-money-business-model-ca0a71a19163. Accessed 10 Nov 2020 28. Spotify. Listening Is Everything. https://www.spotify.com/pt/premium/?checkout=false# PLANS. Accessed 10 Nov 2020 29. Ads.spotify.com. Soluções De Publicidade| Spotify Advertising. https://ads.spotify.com/ptBR/experiencias-publicitarias/. Accessed 10 Nov 2020
352
I. Gomes et al.
30. Kakar, K.: How Spotify Works: Business Model and Revenue Streams. https://insights. daffodilsw.com/blog/how-spotify-works-business-model-and-revenue-streams. Accessed 10 Nov 2020 31. Seitz, P.: https://www.investors.com/news/technology/spot-stock-buy-now/. Accessed 14 Nov 2020 32. Richter, F.: Infographic: Will Music Streaming Ever Be Profitable? Statista Infographics. https://www.statista.com/chart/577/revenue-and-net-loss-of-pandora-and-spotify/. Accessed 10 Nov 2020 33. Business Wire. Spotify Technology S.A. Announces Financial Results for Fourth Quarter of (2018). https://www.businesswire.com/news/home/20190206005298/en/Spotify-Tec hnology-S.A.-Announces-Financial-Results-Fourth. Accessed 10 Nov 2020 34. Oakshott, L.: Essential Quantitative Methods - for Business, Management, and Finance. 6th edn. Macmillan International Higher Education, London (2016) 35. Saunders, M.N.K., Cooper, S.A.: Understanding Business Statistics – An Active-Learning Approach. The Guernsey Press, Guernsey (1993) 36. Bryman, A., Bell, E.: Business Research Methods, 4th edn. Oxford University Press, Oxford (2015) 37. Statista – Share of Spotify users in the United States as of March 2018, by age. https://www. statista.com/statistics/475821/spotify-users-age-usa/. Accessed 06 Dec 2020 38. PORDATA – National minimum wage – monthly figure (PPS). https://www.pordata.pt/en/ DB/Europe/Search+Environment/Table. Accessed 28 Dec 2020
Democratic Talent Management in an Industry 4.0 Environment of Digital Revolution Alberto Rendo1 , Manuel Au-Yong-Oliveira1,2,3(B) , and Ana Dias1,2 1 Department of Economics, Management, Industrial Engineering and Tourism, University of
Aveiro, 3810-193 Aveiro, Portugal [email protected], {mao,anadias}@ua.pt 2 GOVCOPP, Aveiro, Portugal 3 INESC TEC, Porto, Portugal
Abstract. The authors propose to study an innovative approach to talent management based on the need for change that organizations are facing due to a disruptive environment brought on by the fourth industrial revolution: the digital revolution. Its nature and impact on society, on management systems, and on business organizations are described as the stage where a significant change movement is happening to an extent and intensity never seen before. To deal with this, the authors elaborate on the need for increased agility and flexibility as determinant factors for the competitiveness of organizations and businesses. In that sense, a new approach to talent management is proposed based on an innovative concept of talent – which is democratic. By describing the key elements of this innovative concept, a new approach to human resource management is also described, arguing for the need of moving towards disruption in traditional policies, processes, and practices. Five interviewees approached during the Covid-19 pandemic, in late 2020, share their views herein on their talents and of digital disruption and opportunity. Keywords: Leadership · Technology · Change · Innovation · Human resource management · VUCA · Transformation · Digitalization · Competitiveness · Talent
1 Introduction Organizational competitiveness is an essential aspect of current global markets in the industry 4.0 environment. Industry 4.0 means “a new type of progress that excels at resource optimization” [1, p. 214]. Industry 4.0 is about radical disruption in increasingly intelligent industries where autonomous and interconnected technologies function [1], though where the human element is still central. To be more competitive firms must take advantage of their talent. This article is made up by inner reflections [2, 3] and is a result of deep and prolonged thought [4] on the subject of talent management. The article is personal in its approach [5], reflecting a stance on democratic talent management. We share a perspective that is unique, resulting from decades of experience in the realm of human resource management and human resource collaboration. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 353–365, 2021. https://doi.org/10.1007/978-3-030-72651-5_34
354
A. Rendo et al.
Talent per se is not an attribute of a select few [6, 7]. We all have talents which firms must capitalize on. To make firms more competitive and employees more fulfilled. This will involve a turnaround in human resource practices, especially in Portugal, where academic qualifications are paramount and somewhat limiting in a competence-based view. The article continues with a look at the literature and a section on understanding the context. We then discuss the methodology followed in the article. We performed five interview interactions and collected primary data on talents and motivation in technological environments. The final section discusses results and then concludes.
2 A Look at the Literature – A Global Disruption Brought on by Technology Our first theme is related to the nature and effects of the fourth industrial revolution (digitalization) on organizations. It seems consensual that the above is causing a global disruption process in the ways of organizing both people (structure) and management processes. For organizations operating in VUCA environments (an acronym for volatility, uncertainty, complexity and ambiguity), dynamic capabilities are crucial to longterm success because they allow for a flexible reconfiguration of a company’s resource set according to its needs [8, 9]. In this sense, the perspective of dynamic capabilities emphasizes the continuous reconfiguration of organizational resources, taking into account environmental dynamics [10]. There have been many attempts to define what dynamic capabilities are, but one of the most used definitions was suggested by [8, p. 1107]: “Company processes that use resources – specifically processes to integrate, reconfigure, obtain and free up resources – to combine and even create market change. Thus, dynamic capabilities are the organizational and strategic routines by which companies obtain new resource configurations as markets emerge, collide, divide, evolve and die.” Thus, dynamic capabilities are seen as processes that address the evolutionary nature of organizational resources in a changing environment and explicitly linking them to the external environment in order to create sustainable competitive advantage for the company. This competitive advantage is not so related to the dynamic capabilities themselves, but to changes in resources according to the requirements of the dynamic environment [8]. The relevance of this approach to the authors’ work is justified not only because dynamic capabilities are fundamental in the adaptive process of organizations to VUCA environments, but also, and above all, because talent management itself is a dynamic capacity. When it comes to talent management, the literature indicates a significant and unanimous agreement around the relevance of talent management practices for companies’ competitiveness [11]. It is also possible to verify a large consensus around two other dimensions of talent management (TM): (a) TM should consider work practices which include also the configuration and structuring of work – and thus the “how” of working together; (b) TM practices should be contextualized: aligned with the company’s
Democratic Talent Management in an Industry 4.0 Environment
355
long-term goals, strategy, and organizational culture [12]. However, when it comes to the talent concept, the literature has a lack of clarity and the discussion is whether TM should focus on all employees or concentrate on high potentials and high performers. When the question is “what is talent?” the answer in the literature is neither clear nor consensual. The concept is approached in a generic perspective, as a human characteristic that, when applied to management, translates into a global capacity of an organization linked to the potential and performance of all people [11]. We see this lack of clarity in the definition of the talent concept as the main differentiating opportunity for our study and knowledge production. Additionally, the need for change in Human Resource Management (HRM) within the disruptive movement brought by Industry 4.0 is mentioned in the literature. Traditional forms and unquestionable truths of recent decades in this discipline of management are called into question because they no longer respond to the characteristics and needs of businesses in the current context [13]. Talent involves what an employee is able to do well (capacities and skills) supported by the employee’s experience; intelligence (largely innate), compounded by knowledge (gained through study) and character; these are all paramount; as is intrinsic commitment, and the capacity for development [14]. The question is how to discover employees’ talents and get them to work in favor of the enterprise. The result will be added competitiveness in the market and more satisfied employees, who should then remain more loyal to the firm (increased retention) as well as acting as beacons to attract additional talent to the firm [14] in a never-ending cycle of organizational learning and evolution. In sum, our perspective is that all of us have talents, talent is not a restricted attribute of a select few [6]. Instead of simply attributing roles to employees, according to qualifications and competences, we need to identify the most capable people for those roles, according to the firm’s needs but also according to the employees’ talents. We need to be focus on talents, so as to best energize the firm.
3 Understanding the Context 3.1 A Brave New World - Acting in the Unknown and the Unpredictable The acronym VUCA aims to characterize a certain nature of the context in which decisions and actions take place. Its applicability to the current global context – social, political and economic – largely allows not only to understand the forces, movements and phenomena that are unfolding, but also the demanding and intense level of adaptation and change to which people, organizations and systems are exposed. In most areas of this adaptive movement the processes will be even of transformation, and will require new forms of interpretation and analysis, of organizing human and technological systems, new areas of knowledge and competence, new professions, and, finally, new patterns of social and professional behavior. Furthermore, in addition to the depth and breadth of change, it is happening at a speed never seen before. Its pace is frantic and exponential. Technology and its extraordinary evolution are the main catalysts for change and thus the platforms on which the VUCA environment of today’s world operates and develops. In the past, one innovation led to the next. Now, an innovation leads exponentially to new innovations. The pace of creating new data is in the order of the quintillions of bytes per
356
A. Rendo et al.
day: how should we deal with the explosion of data? This data, stored in the “cloud”, not so much teaches human beings but above all makes machines smarter instantly. Artificial intelligence is changing the world, and the best artificial intelligence can be smarter than a human being. The world will change more in the coming decades than it has changed so far. What we do is going to change. The way we do it will change even more. 3.2 Industry 4.0 – A New Way of Performing In terms of the economy and organizations, the Industry 4.0 (i4.0) phenomenon condenses the changes that are happening in the world. Industry 4.0 arises from the extraordinary technological development around digitization and the rise of machines (automation, robotization, artificial intelligence, Internet of things, big data, and block chain, among others), creating new possibilities for organizing production and administrative processes, altering the standards of efficiency, productivity and competitiveness, transforming the organization of logistics chains and thus opening up new possibilities in the way companies relate to their suppliers, customers and consumers and, of course, also to their employees. All this at high speed [15]. Additionally, within this network are the people who, more than machines and processes, will most likely be the “chess piece” that suffers the most impact of this sweeping movement. What role is reserved for them? What risks and opportunities does i4.0 bring to their performance and development? It is important to try not only to answer these questions, but above all to demonstrate that this transformation of technology and machines is, above all, a movement in which people are at heart. In order to better understand the nature of the transformation caused by i4.0, it is useful to understand the nature and effects of the three industrial revolutions that preceded it: the 1st Industrial Revolution: steam; the 2nd industrial revolution: electricity; the 3rd industrial revolution: computers. Let us begin with the speed of these revolutions: the 1st revolution (c. 1760) and the 2nd revolution (c. 1870–1970) lasted for a century; the 3rd revolution was already significantly shorter (1970–2012). All of these revolutions developed in a stable and predictable context: around a single technology, evolving incrementally, and with a lot of time given to people to discover, learn, develop, practice and become masters in subareas of a single large area of knowledge. When we use these same analysis parameters to understand the 4th industrial revolution (the digital one) it is clear that the size and depth of the transformation is tremendous and, consequently, this distinguishes it from previous revolutions. From the outset, as a founding point, the fact that it develops around many technologies (production, mobility, information, communication, among others) increases the spectrum of areas of knowledge that we have to deal with. Moreover, the evolution of these technologies is exponential, that is, an innovation immediately leads to new discoveries and “new innovations”. The mastery of information and knowledge creation processes moves from people to machines because, through the Internet of Things (IoT) and Artificial Intelligence (AI), they will be able to autonomously produce data and share it with each other, learn instantly and, in an absolutely disruptive way in the face
Democratic Talent Management in an Industry 4.0 Environment
357
of everything we know to date, teach humans new ways of acting and new solutions to their problems. This dimension of the exponentiality and the plasticity of the evolution drastically reduces the duration of the time cycles of organizations and systems. Today, and in the future, we will have much less time to plan, decide and act. On the other hand, the knowledge that allows us to perform professionally is itself in permanent and accelerated evolution. The time available to discover, learn, practice and become masters in a particular area of knowledge is reduced to levels hither-to-date unimaginable. That is why the disruptive movement that i4.0 brings primarily means a transformation into two aspects of human action: the way we process and use knowledge and the way we work [15]. Context volatility will no longer allow for attitudes or devices dependent on predetermined external orders. The determinants of individual success thus become intrinsic to the individual, based on the action, values, and talents of each. Clarified, although in a synthetic way, the nature and impacts of the 4th industrial revolution on organizations, systems and people, it is simpler to understand the pertinence and relevance of the acronym “VUCA” to characterize the environments in which societies, organizations and people will evolve from now on. And with that what is definitely at stake is the stability and predictability that man has always sought to sustain his evolution and prosperity. How can organizations respond to this disruptive phenomenon? From an evolutionary perspective, what factors will human systems and individuals need to adapt to this volatile, unpredictable, complex and ambiguous scenario? While integrating into the core of its action two essential pieces in the puzzle: agility and flexibility. 3.3 The Concept of Talent The concept of talent (in the new highly technological and digital world) is most likely the most relevant aspect in this study. We consider and want to demonstrate that talent is the most important dynamic capacity in the adaptive process and, therefore, in the competitiveness of a company in the era of i4.0. It is differentiating because the concept of talent has been approached in the literature and in the management of companies in a generic perspective, as a human characteristic that, when applied to management, translates into a global capacity of an organization linked to the potential and performance of all people [11]. In contrast, the concept that we shall defend is anchored in a perspective of depth, that is, talent seen as the manifestation of essential characteristics of individuals, of defining characteristics of a certain individual. Another differentiating dimension that we shall try to demonstrate in the concept of talent is related to the extent of its applicability. We argue that talent is a ‘universal individual property’ (which everyone owns) and not a ‘partial individual property’ (only within the reach of some privileged or predestined individuals) [6]. What is talent? We advance an answer. Talent is a human property, individualized, revealed in actions that demonstrate the combination of exceptional skills, appetite and a level of knowledge that distinguish and define the individuality of each of us. When properly structured and applied, talent translates into exceptional results in a given field of action or in a certain typology of tasks and/or contexts. In short, this human property has
358
A. Rendo et al.
a root in the essence of each person and has a direct relationship with their performance levels. 3.4 Talents and Competencies This pair of concepts – talents and competencies – is largely at the origin of the lack of clarity and lack of consensus regarding the concept of talent in organizations. In fact, they are distinct constructs and therefore represent equally distinct concepts. However, the disciplines of Organizational Psychology and HRM have not only not treated them differently, but, on the contrary, confuse them in terms of concepts and practices. Let us look at Table 1 and at the descriptions of each element in view of the two concepts. Table 1. Differences between talents and competencies – basic concepts Motivation
Knowledge
Development
Competencies Superficial and located at the behavioural level
Capacities
Stimulated externally in order to create a reason and a will for action
Taught and transmitted, non-personal (facts, data and theory)
Externally demanded. Long-term cycles of practice needed
Talents
Intrinsic motivation for certain areas of personal interest
Based on self-learning. Autonomous and proactive search to deepen a certain area of interest
Internally demanded. Short-term cycles of practice needed
Intrinsic and deep individual skills and abilities (cognitive, physical and emotional)
We see, therefore, that both in terms of their nature and with regard to the factors of mobilization and development, the two concepts (of talents and competencies) differ substantially. However, in addition to the relevance of the distinction of the concepts in itself, we understand that the scope in which the divergence is most accentuated is in the approach and practices of HRM to which they give rise. In conclusion, we consider that the competency management paradigm reduces the potential contribution of people to the competitiveness of companies in VUCA environments and in i4.0 contexts. This happens, in our view, because the creative potential and the agile and adaptive nature of people is constrained by external regulatory requirements. Individuality is primarily considered and valued to the extent that it demonstrates alignment with these normative requirements. This static view is therefore antagonistic to the plasticity required for the success and competitiveness of today and, to that extent, the policies and practices of HRM that are anchored in it are confronted with obsolescence. We propose to demonstrate that Talent Management is the paradigm that best responds to the current challenges of organizations.
Democratic Talent Management in an Industry 4.0 Environment
359
4 Methodology This exploratory research study will mainly follow the interpretive paradigm and philosophical stance [16] – concerned with how human beings make sense of the world and consequently attach meanings to events in day-to-day and organizational life. The methodological options are anchored in two factors that we consider as being structural: 1) the innovative nature of the central theme of the study, both with regard to the concept of talent, and with regard to the context under study (Industry 4.0); 2) the profile of the authors who, in the subject in question, have significant professional experience and, at the same time, considerable research experience in the academic context. We intend to take advantage of the almost 60 years of combined experience that the authors have in human resource management and collaboration. We will let the data talk to us [17] to generate new exploratory theory. “Researchers must be open to various perspectives (and voices) from the field and “go-back-and-forth” between their “theorising” and their data collection.” [17, p. 145]. The choice of a qualitative methodology – and the search for rich data [18] – is also a form of adaptation to the experience and professional situation of the authors. This approach is particularly adjusted to the nature of the theme under study. As we will describe later, societies and organizations are living disruptive times, facing challenges associated with the transition to new paradigms of organization and management. We sought to gather narratives on these changes. We intend to use real facts and stories from the daily experience of the research participants’ professional contexts and, thus, approach two realities that usually coexist with difficulty: theory and professional practice. We thus performed five interviews (in a purposive sample) during November and December of 2020. The interviews occurred over several, face-to-face, informal interactions, with the participants, whereby they developed a narrative on their talents and on how they see the digital environment. Due to the Covid-19 pandemic safety measures were followed. It was, however, difficult to gather more data from more informants as many firms were closed or in a lay-off period and given our time and budget constraints we had to be content with five in-depth interactions. Notes were taken during the interviews by the researcher. The objective was the collecting of narratives on talents (and on weaknesses) while also gaining insights into the digital realm. During the unstructured interviews, specific questions were posed, such as: 1) what are your talents? 2) what are your weaknesses? 3) what is your relationship with technology?
5 Interviews – Field Work The following are personal narratives given by five individuals, about their talents, difficulties and perspectives of the digital world. These narratives were gathered during informal contacts and meetings with the participants. 5.1 A Talent for Digital Marketing and Everything Online “I am talented for anything digital. I work as a marketing manager and specialize in digital marketing. I work on a firm’s website to increase visits by existing as well as
360
A. Rendo et al.
new customers (search engine optimization, or SEO). I give greater visibility to a firm online. I post on social media, for the firm, every day of the week. I never stop working completely… I have a passion for everything digital. I am an expert with a digital camera. I take photos but also make videos; and I am adept at editing content (using Adobe Premiere Pro). I will also use e-mail to contact potential clients, after finding their contacts. I will persist until I get a positive answer. I have gained important new business through electronic means. On a small budget (as little as 40 thousand euros a year), I can get excellent results, with an excellent return on investment. I use Google Ads (for online advertising campaigns) and Google Analytics (which provides free tools for me to study and better understand my customers and our website usage). I use Microsoft Dynamics 365 (an app) for customer relationship management (CRM). I also use Microsoft Bookings as a scheduling tool and to book meetings. We often communicate and meet at a distance using Microsoft Teams. Google Tag Manager is good for online events. I can see how many people are attending, in real-time. Google Trends is used to see organic search tendencies. This is all due to my passion for what I do and achieve in the digital world. In the physical world, I am somewhat a shy person…” (Participant A, female, Millennial, marketing manager, undergraduate degree in marketing management, post-graduate degree in business and digital environments, resides in Portugal). 5.2 A Talent for People and Establishing Empathy in the Digital World “I took an undergraduate degree in international management. However, I am a creative individual and there was not much room for creative individuals, nor their development, in my degree. I worked as a creative strategy executive, after graduating. I enjoy producing material, first as an assistant, now as a freelance photographer. My greatest asset is that I establish empathy and I am approachable – my subjects can relate to me very easily and I can establish a connection with them. They open up to me and show me their real selves, their souls. My talent is creativity in so far as I will use new props, new environments, to capture new and different photos. I used to use only analog cameras. Recently, I started using digital cameras. An obvious advantage is that a digital camera comes out cheaper. Analog camera film and its development are expensive, especially considering that a photoshoot for a textile fashion catalog may involve hundreds of photos. The evolution of the photograph industry has also made possible the immediate viewing of the material created, rather than having to wait for the photos to be developed. That is another great advantage, aside from the economic aspect that I mentioned. My true love is analog, but we need to adapt. People prefer digital because there is more room to play with. In analog, you never know how the photoshoot is going to turn out, so there is more risk.” (Participant B, male, Millennial, freelance photographer, undergraduate degree in international management, resides in Portugal). 5.3 A Talent for the Spoken and Written Word “I am very talented regarding the spoken and written word. My natural oral communication talent has been acknowledged by various people and speaking in public comes very naturally to me. I can talk for hours on a specific topic I have minimally prepared.
Democratic Talent Management in an Industry 4.0 Environment
361
I have even spoken for more than an hour, with no preparation, when I was asked to fill in for an absent conference speaker. Furthermore, I am good at writing. This is an important asset for me to have, as a researcher, and I thus publish a lot of academic work. I publish very prolifically, every year, including academic journal Scopus/ISI Web of Science peer-review articles, national and international book chapters, books, posters, and conference papers. My Ph.D supervisor had a very special talent for drawing and creating visual images and content (using a personal computer and various apps) which I do not have. Hence, as a team, we worked well. However, I am generally not good at interacting with people, socializing, at an informal level. I am not good at small talk, as perhaps I am too serious. I am somewhat a loner outside the classroom. I find personal interaction challenging and quite tiring. Preparing for a lecture is what I enjoy doing; or writing an article, or book. I am thus in my element. Anything that involves words is my talent, in more formal communication. I also consider myself to be an apt deputy-leader, but I do not have the profile of a corporate CEO (Chief Executive Officer). I cannot decide on the spur of the moment certain [difficult] paths which need to be taken, and I would be unable to fire people without losing sleep.” (Participant C, male, Generation X, researcher and university lecturer linked to the social sciences, resides in Portugal). 5.4 Leader by Design “I am a born leader. I am used to being the boss. I decide very quickly and can do that by myself. I also can build a team around me, and my employees come to me for guidance. From a young age, I had to fend for myself against my two older brothers. We would fight, I learned how to kick low [laughs]. I need to be surrounded by people. It is through people that I work the best. That is my talent – leadership. I am also kind and generous, no-one has a more tender heart than me. I think that people sense that. Which means that I get let down now and again, by so-called “friends”. My brothers and I inherited my father’s firm, and we have made it grow tremendously. But there are also a lot of risks involved in the day-to-day management of the firm. Which makes me want to retire early. In a globalized world we never really know what is going to happen. This Coronavirus situation is an example. It has pretty much put the international side of our business “on hold”. As concerns technology, I am very active on social media – Facebook, Instagram, and WhatsApp. I belong to many e-communities with different groups of people and have always embraced technology. I love my Apple watch (great for keeping up with what is going on, during boring meetings) and I use several apps for professional and personal purposes. Technology is everywhere. I use my iPhone to do mostly everything. I also enjoy streaming services – Spotify, HBO, Netflix. I have paid for family subscriptions in each of these. We have to keep up with the times. I consider myself to be very apt with technology and tend to adhere to new technology before it becomes mainstream and before everyone else does. It makes life more efficient and I can thus multi-task – which is important when you are an executive-owner of a business and have a large family.” (Participant D, female, Generation X, an executive-owner of a family business in the private sector, an undergraduate degree in computer science, resides in Portugal).
362
A. Rendo et al.
5.5 Firms and Employee Motivation “I noticed some apathy in those individuals who worked solely for short-term rewards. Their motivation being suspect - only working for themselves and their results – led to an obsession with rewards – rewards which should be accessory rather than the “be-all” and “end-all”. Of course, the fault in all of this lies in management. As organizations try and entice employees to work harder and longer for short-term rewards, they thus take all of the enjoyment out of the task, as well as removing all of the autonomy. The external reward is the focus, and not the intrinsic reward of the task itself. Having worked for some time in the industry, I have noticed that the motivation element in an Industry 4.0 environment is decisive. New heights of exposure are now possible in the digital era – and the yearning for advancement may even be surpassed by the yearning for recognition. Let us consider social networks as an example. Never before, and practically since 2004, when Facebook was launched, have people had so much exposure in their networks and the networks of their friends and peers. Posting our successes on social networks has become an objective of many, even if those posts show but a small sample of the lives and lifestyles of those involved. Facebook is like a shop window where we only show what we want to promote. LinkedIn may be even more extreme, in so far as it is a medium for professional advancement and may give us a simple or even important boost in professional terms. The above has led to unprecedented levels of jealousy, as a consequence. The answer lies in leadership and autonomy. To speak of emphasizing intrinsic motivation, in Portugal, rather than emphasizing the “carrot” and the “stick”, which traditionally have been used to make people work towards organizational objectives, is new in so far as we in Portugal see work as a means to put food on the table, rather than seeing beyond that, to work as a means of liberation and self-actualization. Consider the student who studies for a good grade versus the student who studies to learn and evolve as a human being. We shall need to help people see past the next paycheck (or students to see past their grades) and hence make work less tiring and boring, rather enticing people to use their talents – for their good and for the good of the organization that they serve. Social networks and other realities of the digital era need to be used for good rather than for creating jealousy amongst colleagues and friends. If there was no search for external recognition and rewards, but rather a search for intrinsic rewards, would not organizations be a more equal and just setting, for fun, industrious, talented, and enthusiastic individuals to thrive in (as the late Sean Connery stated, at the AFI Awards, describing the part of his job as an actor that he loved)? Those others know who they are (the late Sean Connery continued). Is working in an i4.0 environment all about making it all more worthwhile? HRM processes need to focus on talent management. Like other aspects of organizational management, processes, and practices today are faced with disruptions that will force leaders to behave substantially differently from those standards in the future. To manage talents, leaders must relinquish their central role of direction, holding answers and certainties, for a role of opening and stimulating creativity and entrepreneurship, of employee involvement in decisions, of the identification and development of individual talents. They must be, above all, bearers of issues, enablers of freedom and autonomy
Democratic Talent Management in an Industry 4.0 Environment
363
that allow the participation and commitment of all of the elements of their team, making individual diversity the basis of their success, the success of people, and the organization.” (Participant E, male, Generation X, a manager in the public sector, post-graduate studies in management, resides in Portugal).
6 Discussion and Conclusions Figure 1 summarizes the contribution from the interviews. The field work makes visible several distinct talents: a talent for digital marketing and everything online (participant A), a talent for establishing a human connection (participant B), a talent for the written and spoken word (participant C), a talent for leadership (participant D), and, finally, a talent for recognizing the importance of motivation and how the digital world has brought on change (participant E). In all of the interviews the technological component in society and business is present.
Fig. 1. Examples of talents from the field research
We would like to emphasize how each participant in the primary data collection effort would only be happy working through their unique talents. Participant A is somewhat shy and prefers interacting with technology and online. Participant B enjoys establishing empathy, while leveraging technology. Participant C has a talent for words, but not for drawing and images. Participant D is a leader by design and enjoys working through people, rather than alone. Participant E is a pioneer and discusses how Portugal needs to change, to make work more interesting and enjoyable. Work should be about worthwhile accomplishment, and this will only be possible if we use and follow our talents.
364
A. Rendo et al.
According to [19] jobs are now perceived by the new generation as a way to find new meanings for their lives and to reinvent themselves. We have the perspective that the future of work needs to change. Moreover, this will require planning and career development, earlier on in the process – namely at school - and not only for the gifted [20]. This change in paradigm – to talent development and usage - needs to be embraced by the education system – and later continued in the workplace. Future research should use a larger sample. Due to the Covid-19 situation, this limitation was hard to overcome. Acknowledgements. We would like to thank our five interviewees for having shared their views with us on talent and on the digital world. We also thank them for having validated the content of their contributions herein.
References 1. Dantas, T.E.T., de-Souza, E.D., Destro, I.R., Hammes, G., Rodriguez, C.M.T., Soares, S.R.: How the combination of circular economy and Industry 4.0 can contribute towards achieving the sustainable development goals. Sustain. Prod. Consumption 26, 213–227 (2021) 2. Lazard, L., McAvoy, J.: Doing reflexivity in psychological research: what’s the point? what’s the practice? Qual. Res. Psychol. 17, 159–177 (2020) 3. Deo, C., Gouzouasis, P.: To build a home. Qual. Res. Psychol. 17, 178–181 (2020) 4. Ellis, C., Adams, T.E., Bochner, A.P.: Autoethnography: an overview. Forum Qual. Soc. Res. 12, 1–18 (2011) 5. Au-Yong-Oliveira, M.: Using reflexive, introspective and storytelling tools: towards becoming more autoethnographic in academia. Educ. Sci. 10(4), 120 (2020) 6. Leite, M.P., Mihajlovski, T., Heppner, L., Branco, F., Au-Yong-Oliveira, M.: The impact of the digital economy on the skill set of high potentials. In: Rocha, Á., et al. (eds) New Knowledge in Information Systems and Technologies, Advances in Intelligent Systems and Computing (Book of the AISC series), vol. 931, pp. 726–736. Springer Nature Switzerland AG (2019) 7. Au-Yong-Oliveira, M.: A liderança do departamento de marketing na era da tecnologia digital. RISTI – Rev. Ibér. Sistemas Tecnologias Informação E24, 168–183 (2019) 8. Eisenhardt, K.M., Martin, J.A.: Dynamic capabilities: what are they? Strateg. Manag. J. 21, 1105–1121 (2000) 9. Wang, C.L., Ahmed, P.: Dynamic capabilities: a review and research agenda. Int. J. Manag. Rev. 9(1), 31–51 (2007) 10. Schilke, O.: On the contingent value of dynamic capabilities for competitive advantage: the nonlinear moderating effect of environmental dynamism. Strateg. Manag. J. 35(2), 179–203 (2014) 11. Gallardo-Gallardo, E., Thunnissen, M.: Standing on the shoulders of giants? A critical review of empirical talent management research. Empl. Relat. 38(1), 31–56 (2015) 12. Gallardo-Gallardo, E., Scullion, M., Thunnissen, H.: Talent management: context matters. Int. J. Hum. Resour. Manage. 31(4), 457–473 (2019) 13. Claus, L.: HR disruption – time already to reinvent talent management. BRQ Bus. Res. Q. 22, 207–215 (2019) 14. Alves, P., Santos, V., Reis, I., Martinho, F., Martinho, D., Sampaio, M.C., Sousa, M.J., AuYong-Oliveira, M.: Strategic talent management: the impact of employer branding on the affective commitment of employees. Sustainability 12(23), 9993 (2020)
Democratic Talent Management in an Industry 4.0 Environment
365
15. Skilton, M., Hovsepian, F.: The 4th Industrial Revolution: Responding to the Impact of Artificial Intelligence on Business. Springer International Publishing, New York (2017) 16. Saunders, M., Lewis, P., Thornhill, A.: Research Methods for Business Students, 7th edn. Pearson Education, Harlow (2016) 17. Elharidy, A., Nicholson, B., Scapens, R.W.: Using grounded theory in interpretive management accounting research. Qual. Res. Acc. Manag. 5(7), 139–155 (2008) 18. Mason, J.: Qualitative Researching, 2nd edn. Sage, London (2002) 19. Nezami, S., Bakker, C., Tinga, D.: Security and the new generation workforce. In: Advanced Sciences and Technologies for Security Applications, pp. 255–272 (2021) 20. Smith, C.K., Wood, S.M.: Supporting the career development of gifted students: new role and function for school psychologists. Psychol. Schools 57(10), 1558–1568 (2020)
Stimulating Components for Business Development in Latin American SMEs Romel Ramón González-Díaz1,2(B)
and Luis Armando Becerra-Perez2
1 Centro Internacional de Investigación y Desarrollo–CIID, 230001 Monteria, Colombia
[email protected] 2 Universidad Autónoma de Sinaloa, 80015 Culiacan, Mexico
[email protected]
Abstract. This study aims to analyze the stimulating components for business development in Latin American SMEs (case: Colombia).For this purpose, the unit of analysis was the companies listed in the Legiscomex database (trade intelligence) as of December 31, 2019, which integrates the national databases of companies registered and validated by the National Tax and Customs Administration the Confederation of Chambers of Commerce of Colombia. For this exercise of the impact of the four components of business development, 100,000 iterations were defined (Monte Carlo method), which represents a large amount of random data on the information parameters collected and accumulated to estimate a probability distribution of the success that the companies would have in real life, having as input the specific information of financial management,knowledge management, innovation processes and marketing. The results reflect that business development is influenced by four components in the following proportion: Financial Management (30%), Knowledge Management (25%), Innovation Processes (25%) and Marketing (20%). Keywords: Business development · SMEs · Latin America · Marketing · Financial management · Knowledge management · Innovation processes
1 Introduction Today, the dynamics of world markets have changed the way we do business, the number of suppliers has increased, and customers are more demanding [1]. In Latin America, 92% of SMEs go bankrupt in the first year of their creation, and only 16% of them make it to the third year [2–4]. This situation has worsened in the last year with the arrival of the economic crisis caused by the COVID-19 [5]. SMEs are essentially constituted with a dynamic and flexible organizational structure [6]. Therefore, they have chosen to modify their business management system to survive the vicissitudes of the new modernity, and have quickly prepared their staff in information technology and digital marketing [7–9]. This facilitated SMEs’ participation in the digital market, and they have been recovering their economic income [8]. This economic recovery process has been the result of the articulation of components for the business development of SMEs. The scientific literature describes some models © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 366–374, 2021. https://doi.org/10.1007/978-3-030-72651-5_35
Stimulating Components for Business Development
367
for organizational development, such as Organizational models related to behavioural change, which refer to those that generate changes in an organization’s personnel’s behaviour, use training to promote greater participation and communication in the organization. Organizational Models related to structural change are related by the organization’s senior management and vary according to the situation, the work environment, and the organisation’s structure and technology [10]. Furthermore, the Models of organizational development related to structural and behavioural changes, where managers must create awareness of their contribution in driving changes to improve their performance, must also clarify who is responsible for each task to clarify the distribution of responsibilities [11, 12]. Organizational Models are strategies that involve restructuring traditional organizational systems, which implies the idea of participation and development of people through education and behavioural sciences. This is integrated with the postulates of entrepreneurial development expressed by Blázquez Santana, et al. [13]. They propose that the interest in the theory of the entrepreneurial function from the perspective of growth is a planned effort of the entire organization, administered from the top management, to increase the effectiveness and well-being of the enterprise through planned interventions in the processes of the entity, which apply the knowledge of the behavioural sciences. In this sense, this study makes an effort to determine the components with the most significant relationship in Latin American SMEs’ business development. This information will allow the entrepreneurs to prioritise those components that intervene in the development of their business. To this end, an analysis is presented of the components used by Pérez Uribe, et al. [14] through the Modernization Model for Organizational Management (MMGO), which is widely accepted in Mexico and Colombia. In this regard, an exercise was carried out to determine its specific weight in the business development of SMEs. The results of the study show the generation of emerging components where the effectiveness of business management is concentrated: Financial Management, Innovation Processes, Marketing and Knowledge Management.
2 Materials and Methods Based on the studies of González-Díaz, et al. [15], Hernandez-Julio, et al. [16] as empirical evidence from studies conducted in Colombia, Mexico and Panama, where the relevance of the following components can be verified: Financial Management, Knowledge Management, Innovation Process and Marketing. To corroborate the impact of these components, we have carried out an exercise to determine their specific weight in the business development of SMEs. We took as an analysis unit the companies listed in the Legiscomex database (business intelligence) as of December 31, 2019, which integrates the national databases of companies registered and validated by the National Tax and Customs Administration and the Confederation of Chambers of Commerce of Colombia, which makes it a reliable source, on this information we collected the data required to know the impact on business development. Likewise, statistical parameters such as arithmetic means, upper and lower limits of each of the business components were created, and different distribution types were
368
R. R. González-Díaz and L. A. Becerra-Perez
determined according to each component’s behaviour. Data normalization techniques and the elimination of inconsistencies were applied to determine the sample of companies and proceed to the simulation and analysis of 100 thousand iterations or synthetic companies of the different economic sectors. The tool applied for the analysis of the selected information was @risk, a software (Palisade Corporation, U.S.) robust in risk estimation in the business environment, which uses the Monte Carlo simulation technique. As it is widely known in the field of simulation models, Monte Carlo follows an entirely random process where first the variables are selected with their probabilities of occurrence; then, random numbers are generated in a large sample and quantity; third, each random number is applied over the area of the cumulative probability frequency; and fourth, the final variables are obtained which are used as if they were variables offered by reality.
3 Analysis and Discussion of the Results The importance of building simulation models, especially with Monte Carlo, is that it moves from making business analysis with deterministic numbers to probabilistic behaviour, and that implies a substantial advance that changes the way of interpreting the results from a constant or purely deterministic approach, where only one scenario exists, to a predictable oscillation approach where there can be several scenarios and each of them with probabilities of occurrence. For this impact exercise of the four components of business development, 100,000 iterations were defined, which represents a large amount of random data on the information parameters collected and accumulated to estimate a probability distribution of the success that the companies would have in real life, having as input the specific information of financial management, knowledge management, innovation processes and marketing. The results found from the simulation are presented in Table 1. Table 1. Incidence of the components for business development Financial management
Knowledge management
Innovation processes
Marketing
Business development
Degree of Var. ± Degree of Var. ± Degree of Var. ± Degree of Var. ± influence influence influence influence 0,3
4,192
0,25
4,689
0,25
3,752
0,2
4,689
31,34
In the section of finances, the results show that the financial management has a healthy weight on its success, affecting it in a value of 0.3, that is to say, a third of the excellent development of this one. In other words, the good or bad performance that the company has in this area affects at least 30% of the results. The novelty of this output is not really that financial management has a significant weight on the company’s results, but to know the value in which it affects. In order of importance, the second most important component was the innovation process, which impacted a quarter (0.25) of
Stimulating Components for Business Development
369
the company’s success. In that order of ideas, the knowledge management component had a specific weight of 0.20, as did the marketing component. Once the components’ incidence parameters and their impact on business development were established, a sensitivity analysis was carried out. This type of analysis tries to determine how changes in input variables influence the output variables. If we have only one input variable and we want to find its incidence value in the output variable, the’data tables’ of Microsoft Excel and its’target search’ function solve the problem; but if what we have is a combination of imputed variables, which must be searched to optimize (maximize or minimize) an output variable. For this simulation, 100,000 iterations (random samples) were defined, setting as input variables financial management, knowledge management, innovation processes and marketing, while the output variable was business development, the results being as follows: The tornado figure (Fig. 1) shows the sensitivity of the “input” variables to the “output” objective, that is, to the business development of SMEs. In this case, it is found that financial management, innovation processes and knowledge management, in that order, are the components that can most impact the development of the company. Financial management is the most correlated variable with the average output value (33,211), which can be raised to 45,217 or lowered to 24,583. Secondly, innovation processes can make the average value rise to 44,899 or fall to 25,605; while knowledge management can make this same average value rise to 41,084 or fall to 26,715. We can also appreciate from this brief sensitivity analysis that the marketing variable is poorly correlated with the average result of business development.
Fig. 1. Impact of the components on business development.
3.1 Financial Management According to Bravo, et al. [17], financial management is considered one of the fundamental management areas and necessary in any organization. Its main competencies are the analysis, decisions, and actions related to financial resources in SMEs’ activity, that
370
R. R. González-Díaz and L. A. Becerra-Perez
is, such management integrates all tasks related to the achievement, use, and control of financial resources. Specifically, on this component and applying the same methodology described above, a simulation was carried out, which yielded the following results represented in Fig. 2.
Fig. 2. The behaviour of the normal distribution of the business component: financial management
This figure shows the business component of financial management. It results from a simulation of 100,000 iterations (samples) of SME data, constituting a minimum of 1,0000, maximum of 9,9992, mean of 4,1925, the confidence level of 90%, ± 0.0479, the standard deviation of 2.0577, asymmetry of 0.5277 and kurtosis of 2.5484. In general terms, according to the data interpretation scale for financial management, it is validated that 90% of the SMEs are in the range of 1.33–8.03, representing between very bad and bad management of their financial resources. 3.2 Innovation Processes According to Muñoz [18], it is the set of activities that address the process of organizing and directing the company’s resources (human, material, economic) in order to increase the creation of new knowledge, generate ideas to develop new products, processes and services or improve existing ones, and transfer that knowledge to all areas of the organization’s work (Fig. 3). As for the business component of knowledge management, the 100,000 iterations (samples) of data from SMEs have a minimum of 1,0000, a maximum of 9,9998, a mean of 3,7519, a confidence level of 90%, ± 0.0447, a standard deviation of 1.9206, an asymmetry of 0.7188 and kurtosis of 2.9206. In this sense, to determine the impact on business development through the data interpretation criteria, 90% of the SMEs (1.25–7.43), have deficient levels and low innovation processes.
Stimulating Components for Business Development
371
Fig. 3. The behaviour of the normal distribution of the business component: innovation processes
3.3 Knowledge Management Knowledge management is considered the ability to effectively manage the flow of knowledge within the organization to guarantee its permanent access and reuse, thus stimulating innovation, improvement of decision-making processes, and the generation of new knowledge. This process would be mediated by the facilitating nature of information technologies, enabling the flow of information throughout the organization and optimising internal and external communication channels [19]. Figure 4 shows the results of the business component knowledge management. For this, the model was simulated with 100,000 iterations of data from SMEs, resulting in a minimum of 1,0000, a maximum: 9,9999, an average of 4,6889, a confidence level of 90%, ± 0.0503, a standard deviation of 2.1628, an asymmetry of 0.3227 and kurtosis of
Fig. 4. The behaviour of the normal distribution of the business component: knowledge management
372
R. R. González-Díaz and L. A. Becerra-Perez
2.2808. The above implies that the organizations studied are in 90% (1.47–8.57) of the cases in low and moderate knowledge management levels. 3.4 Marketing There are various definitions, and interpretations of marketing and this concept have undergone epistemological changes, however, in this work, we assume the position according to the interpretation of Marín López and López Trujillo [20], who consider it to be the set of relevant practices and processes to create, communicate, release and exchange offers that have value for customers, partners and society in general. Figure 5 shows the behaviour of marketing as a business component. One hundred thousand iterations of data of the studied SMEs were analyzed, resulting in a minimum of 1.0018, a maximum of 9.9957, with an average of 4.6889, a confidence index of 90%, ± 0.0503, a standard deviation of 2.1631, an asymmetry of 0.3226 and kurtosis of 2.2806, which means that 90% of the organizations maintain between 1.47 and 8.57, according to the criteria of interpretation of the data, are considered low to moderate marketing actions.
Fig. 5. The behaviour of the normal distribution of the business component: marketing
4 Conclusion In general, it is concluded that the four pillars on which the present model of organizational intervention rests demonstrate at a conceptual and statistical level their ductility in the established fields. In this scenario, each pillar responds to a trend but impacts the global component, so they complement each other by integrating into a single and consistent network of management possibilities that ensure the best business performance. On the other hand, the authors are convinced that each company is unique in its branch, taxonomy, structure and managerial diversity, that they function as living organisms, that they face a complex and dynamic reality to which they have to adapt in order
Stimulating Components for Business Development
373
to survive; therefore, the built model is adapted to each one of them, exposing their strengths and making visible their weaknesses, with which, the necessary corrections can be taken to implement actions tending to correct the errors that otherwise would not have been detected.
5 Future Work This research leaves a gap in the field of business development in SMEs, giving the opportunity to deepen the work related to the construction of business development models contextualized to the economic, political, social and cultural situation of the Latin American region. Likewise, the proposed method is far from being complex and its results are evident and easy to interpret, given that the resulting records allow the traceability of the achievements made and the activities pending execution, thus obtaining a management log that tends to optimize the resources based on the above factors.
References 1. Arora, P., De, P.: Environmental sustainability practices and exports: the interplay of strategy and institutions in Latin America. J. World Bus. 55(4), 101094 (2020). https://doi.org/10. 1016/j.jwb.2020.101094 2. Cardoza, G., Fornes, G., Farber, V., Gonzalez Duarte, R., Ruiz Gutierrez, J.: Barriers and public policies affecting the international expansion of Latin American SMEs: Evidence from Brazil, Colombia, and Peru. J. Bus. Res. 69(6), 2030–2039 (2016). https://doi.org/10. 1016/j.jbusres.2015.10.148 3. Hernandez, M.C., Montoya, M.V., Martínez, J.F.: Development of a model for evaluating the npd process in smes: a latin american experience. Procedia CIRP 21, 449–454 (2014) https:// doi.org/10.1016/j.procir.2014.03.123 4. Valdez-Juárez, L.E., Solano-Rodríguez, O.J., Martin, D.P.: Modes of learning and profitability in Colombian and Mexican SMEs. J. High Technol. Manag. Res. 29(2), 193–203 (2018). https://doi.org/10.1016/j.hitech.2018.09.007 5. Sharma, P., Leung, T.Y., Kingshott, R.P.J., Davcik, N.S., Cardinali, S.: Managing uncertainty during a global pandemic: an international business perspective. J. Bus. Res. 116, 188–192 (2020). https://doi.org/10.1016/j.jbusres.2020.05.026 6. Korpysa, J.: Entrepreneurial management of SMEs. Procedia Comput. Sci. 176, 3466–3475 (2020). https://doi.org/10.1016/j.procs.2020.09.050 7. Falahat, M., Ramayah, T., Soto-Acosta, P., Lee, Y.-Y.: SMEs internationalization: the role of product innovation, market intelligence, pricing and marketing communication capabilities as drivers of SMEs international performance. Technol. Forecast. Soc. Change 152, (2020). https://doi.org/10.1016/j.techfore.2020.119908 8. Fraccastoro, S., Gabrielsson, M., Pullins, E.B.: The integrated use of social media, digital, and traditional communication tools in the B2B sales process of international SMEs. International Business Review, p. 101776 (2020). 10.1016/j.ibusrev.2020.101776 9. Hirvonen, J., Majuri, M.: Digital capabilities in manufacturing SMEs. Procedia Manufact. 51, 1283–1289 (2020). https://doi.org/10.1016/j.promfg.2020.10.179 10. Arranz, N., Arroyabe, M.F., Li, J., de Arroyabe, J.C.F.: An integrated model of organisational innovation and firm performance: generation, persistence and complementarity. J. Bus. Res. 105, 270–282 (2019). https://doi.org/10.1016/j.jbusres.2019.08.018
374
R. R. González-Díaz and L. A. Becerra-Perez
11. Bocken, N.M.P., Geradts, T.H.J.: Barriers and drivers to sustainable business model innovation: organization design and dynamic capabilities. Long Range Plann. 53(4), (2020). https:// doi.org/10.1016/j.lrp.2019.101950 12. Slatten, L.A., Bendickson, J.S., Diamond, M., McDowell, W.C.: Staffing of small nonprofit organizations: a model for retaining employees. J. Innov. Knowl. 6(1), 50–57 (2020). https:// doi.org/10.1016/j.jik.2020.10.003 13. Blázquez Santana, F.J., Dorta Velázquez, A., Verona Martel, M.C.: Concept, perspectives and measurement of business growth. Cuad. Administración 19(31), 165–195 (2006) 14. Pérez Uribe, R.I., Ocampo Guzmán, D., Ospina Bermeo, J., Cifuentes Valenzuela, J., Cubillos Leal, C.A.: MIIGO-Intervención e innovación para el direccionamiento estratégico, 1 ed. (no. 1). Universidad EAN (2017) 15. González-Díaz, R.R., Acosta-Moltó, E., Flores-Ledesma, K., Vargas, E.C., Menacho-Rivera, A.: Marketing experience in non-profit organizations: a look at experience pro-viders. Rev. Iberica Sist. Tecnol. Inf. Article vol. 2020, no. E36, pp. 186–202 (2020). (in Spanish) 16. Hernandez-Julio, Y.F., Merino-Fuentes, I., Gonzalez-Diaz, R.R., Guerrero-Avendano, A., Toledo, L.V.O., Bernal, W.N.: Fuzzy knowledge discovery and decision-making through clustering and dynamic tables: application in colombian business finance. In: Rocha, A., Perez, B.E., Penalvo, F.G., del Mar Miras, M., Goncalves, R. (eds) 15th Iberian Conference on Information Systems and Technologies (CISTI 2020), vol. 2020. IEEE Computer Society (2020). https://doi.org/10.23919/cisti49556.2020.9141117 17. Bravo, C.C.C., Zurita, M.P.F., Segovia, G.W.C.: La gestión financiera aplicada a las organizaciones. Dominio Cien. 3(4), 220–231 (2017) 18. Muñoz, L.D.C.: Elementos clave de la innovación empresarial. Una revisión desde las tendencias contemporáneas. Rev. Innov. ITFIP 6(1), 50–69 (2020) 19. Guzmán, J.E., Arrieta, D.B.: Gestión del conocimiento en instituciones de educación superior: caracterización desde una reflexión teórica. Rev. Cienc. Soc. 26(3), 83–97 (2020) 20. Marín López, J.C., López Trujillo, M.: Análisis de datos para el marketing digital emprendedor: caso de estudio del parque de innovación empresarial de manizales. Rev. Univ. Empresa 22(38), 65–78 (2020)
Handling Industrial Consumer Rights by Using Blockchain M. A. El-dosuky1(B)
and Gamal H. Eladl2
1 Computer Science Department, Faculty of Computer and Information,
Mansoura University, Mansoura, Egypt [email protected] 2 Information Systems Department, Faculty of Computer and Information, Mansoura University, Mansoura, Egypt [email protected]
Abstract. The main objective of this research paper is to propose a business model to deal with warranties of products bought by clients and present a prototype implementation for a vehicle supply chain. Many customers have fake warranties which reduce customer loyalty to large companies. Therefore, the blockchain features can be used here as a technical solution. It can connect each device data and product lifetime to preserve consumer rights, by arranging companies with the highest guarantee under the blockchain. Furthermore, a proposed blockchain business model (BBM) is presented that regulates the life cycle of any product, the manufacturer under the control of the blockchain. The paper provided a case study for vehicle industry. After implementing blockchain structures, it was proven and observed that the blockchain is scalable to huge number of devices. This leads to obvious results and transparency among the giant companies in the interest of the client, and eliminating consumer rights issues such as black market and over prices. Keywords: Blockchain · Governance · Consumer rights · Supply chain · Warranties
1 Introduction Blockchain offers business value in 3 main areas that are value transfer, smart contract, and recordkeeping [1]. Recordkeeping considers creating immutable records under a consensus protocol without reliance on a third party. Governments can adopt blockchain to start a cycle of trust in the legal and financial system, in areas such as identity management, voting and protecting sensitive data [2]. This paper proposes a business model that regulates the life cycle of the consumer with governance of the Ministry of Industry and the manufacturer under the existence of blockchain. Blockchain can be used as a preservation of consumer rights and customer satisfaction feedback.
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 375–384, 2021. https://doi.org/10.1007/978-3-030-72651-5_36
376
M. A. El-dosuky and G. H. Eladl
The rest of this paper is organized as follows. Section 2 summarizes an overview of interdisciplinary blockchain and supply chain governance. Section 3 is the proposed business model in details and its implementation, and showing blockchain structure. Finally, Sect. 4 contains conclusion and future directions.
2 Overview 2.1 Blockchain Basic Concepts Blockchain is basically a decentralized peer-to-peer (P2P) network of transactions, without a central authority [3]. There are four types of blockchains as shown in Fig. 1 [4]. Public permissionless blockchain can be read, written and committed by anyone such as Bitcoin. Public permissioned blockchain can be read by anyone, written and committed by authorized participants such as supply chain ledger for retail brand. Consortium blockchain is read by a restricted set of participants, written and committed by authorized participants such as a set of banks with a shared ledger. Private permissioned blockchain can be read, written and committed by network operator only such as an external ledger shared by an enterprise and its subsidiaries.
Blockchain
Closed
Open
Public permissionless
Public permissioned
Consorum
Private permissioned
Fig. 1. Blockchain types [4]
Despite the supremacy of blockchains and their potential, there are three weaknesses, namely scalability, privacy and governance [3]. The paper in hand focuses on governance problem for two reasons. First, governance problem emerges after the collapse of widelyadvertised platforms. Second, many solutions are proposed to scalability and privacy problems but they all seem to need “some degree of centralization” of governance as in a “Masternode” [3].
Handling Industrial Consumer Rights by Using Blockchain
377
2.2 Blockchain-Based Governance There are twelve principles of blockchain-based governance [5]. First, for ages the State and bureaucracy have been a remedy of scaling problem in order to achieve consensus and coordination. Second, State as a hierarchical organization can be modeled as a Single Point of Failure (SPOF), which means that the failure of a component yields to the failure of the entire system. Third, distributed architectures introduce trust-bycomputation concept. Forth, Decentralized Autonomous Corporations (DACs) in which relationships between individuals and the State can be automated by “a series of instant atomic interactions.” Fifth, a nation can be put on the blockchain in a Starbucks-style public administration. Sixth, globalized government services without borders that interact with each others in order to achieve citizen satisfaction. Seventh, blockchain can help achieving direct democracy. Eighth, futarchy in which individuals vote twice, one for general outcomes, then for proposals to achieve those aforementioned outcomes. Ninth, decentralized services are still based on the authority of the State. Tenth, a new social contract characterized by DACs, with no need for the State as a novel direction affecting the citizen culture. Eleventh, franchulates that is a mixing of “franchise” and “consulate”, which means availability of the blockchain anytime anywhere. Twelfth, authority floating freely that is already happening in information industry, in which works operate in open areas. 2.3 Blockchain for Supply Chain A Supply Chain consists of all stakeholders involved in achieving a customer’s request [6]. Figure 2 shows the product journey from order to delivery. Blockchain has a great impact in achieving supply chain six objectives of cost, speed, dependability, risk reduction, sustainability, and flexibility as narrated in [7]. However, there are many supply chain due diligence challenges regarding to blockchain such as supply chain fragmentation, the traceability of goods flow, transparency and risk information [8]. Challenges of adapting blockchain are divided into a dichotomy of technical and nontechnical [8]. Technical ones consider asset digitization, interoperability, and privacy versus transparency. Non-technical ones consider data model standardization, governance, embedding responsible business conduct and including informal actors. 2.4 Blockchain as a Business Model (BBM) Blockchain can disrupt the existing business models by three ways [9]. First way is by authenticating traded goods. Second way is by facilitating disintermediation. Third way is by enhancing operational efficiency. Appropriateness of BBM is based on many conditions [3]: such as verification of both transactions and data is required; feasibility of disintermediation in a technical and economical way; many users need data sharing; and business processes need trust in transactions.
378
M. A. El-dosuky and G. H. Eladl
Fig. 2. Product journey
Furthermore, there are five best BBMs that are [10]: 1. Blockchain as a Service (BaaS) which is the most popular business model; 2. Utility token business model which involves driving the functionality in business through the use of tokens, such as Ripple and Stellar; 3. Blockchain-based software products, where any business should buy and integrate a blockchain solution, such as MediaChain from Spotify; 4. Blockchain professional services; 5. Peer-to-peer (P2P) blockchain business model.
3 Proposed Blockchain Business Model 3.1 Stakeholders and Information Flow Figure 3 shows the stakeholders of the proposed blockchain business model, in which blockchain is a backbone acting as the single source of truth. On one side there is the business as usual showing the flow of goods from the factory to the consumers through distributers. On the other side is the representative of bureaucracy such as the ministry of industry and the chamber of commerce. Figure 3 also shows the information flow. The factory can publish production details such as location, time and quantity. Distributers can publish shipped product details
Handling Industrial Consumer Rights by Using Blockchain
379
while in transit. Both warehouse and retailers can publish store details. Consumers can publish anonymous feedback on the products. Figure 3 shows vehicle supply chain with arbitrary bar codes. A factory may use Global Location Number (GLN) to identify a location of products. An exporter/importer can use Serial Shipping Container Code (SSCC) to identify logistics units. A warehouse
Fig. 3. Vehicle supply chain with accumulative blockchain bar codes
380
M. A. El-dosuky and G. H. Eladl
can use Universal Product Code (UPC) usually in USA. The Japanese version of UPC is Japanese Article Numbering (JAN), while the European version is European Article Numbering (EAN). Finally, the retailer may use QR code. Mining blockchain for consumers is composed of accumulative blockchains of all the factory, importer, warehouse and retailer. 3.2 Mathematical Model The owner and curator of the blockchain is “Gov” which keeps track of all transactions of products. The key activities are adding and validating blocks. A factory “F” can add blocks regarding the production of product “P”. Then, an importer “I” can add blocks regarding the importing process of the product “P”. Then, a warehouse “W” can add blocks regarding the warehousing process of the product “P”. Finally, a retailer “R” can add blocks regarding the retailing, selling, and reselling process of the product “P”. This can be modeled as shown in Eq. 1: Gov ← R(W(I(F(P))))
(1)
This can be easily implemented as aggregation in any object-oriented programming language such as Java. Any block contains a data “D”, and a hash “H”. The data accumulates all product data and labels. 3.3 Proposed Blockchain Business Model (BBM) Figure 4 shows proposed blockchain business model depicted as a business model canvas. The key partners are internet service providers. The key activities are adding and validating blocks. The value propositions are tracking products from production to delivery. The customer relationships are mainly for warranties. The targeted customer segments are any customer having access to the Internet. The cost structure is mainly for blockchain server operation and Internet access. Revenue streams are mainly taxes. 3.4 Blockchain Structure Implementation Implementation of the blockchain means blockchain mining procedure. The private blockchain implementation can be later extended into another blockchain type. This is an acceptable limitation to build a prototype that is programmed by using Java language. Figure 5 shows Graphical User Interface (GUI). Product form, for instance, encapsulates product details which are: ID, name, type, price, manufacturer, bar code, quantity, production date, and expiry.
Handling Industrial Consumer Rights by Using Blockchain
Key Partners Key Activities
Value Propositions
Customer Relationships
Customer Segments
Ministry of Industry, Chamber of Commerce
Tracking products from production to delivery
Warranties and Satisfaction feedback
Internetbased customer
Postdelivery services
Internet service providers
Factory, importer, warehouse and retailer
Mining, reading and validating blocks
Resources Blockchain Databases and servers
Channels
Cost Structure • • •
381
Blockchain server operation Blockchain mining cost Internet access
Revenue Streams Fees for the services Taxes
Fig. 4. Proposed blockchain business model canvas
Fig. 5. A prototype of supply chain GUI (Product Form)
Figure 6 shows blockchain structure. It composes all supply chain phases as shown in Fig. 3. At the first run, the genesis block is created. Adding any subsequent block that encapsulates the data shall generate a hash to the previous block.
382
M. A. El-dosuky and G. H. Eladl
Genesis Block
Hash: 00000d99d5b74ae4083b0dc 72eb037911c7e4fc3b57cdf96
1st product data Previous Hash: 00000d99d5b74ae4083b0dc 72eb037911c7e4fc3b57cdf96 Hash: 0000061ed3da10ae227de915a6 5c1aaff1b77653d234a6af
2nd product data
Previous Hash: 0000061ed3da10ae227de915a65c1aaff1 b77653d234a6af Hash: 00000e6065aed4d98294c7858edca7652b2 d8fd070568a6e Fig. 6. Blockchain structure mining
3.5 Comparing Proposed BBM with Traditional Supply Chain Figure 6 shows blockchain structure mining. It composes all supply chain phases. At the first run, the genesis block is created. Adding any subsequent block that encapsulates the data shall generate a hash to the previous block. Table 1 shows an observed comparison between proposed BBM with traditional supply chain. Traditional supply chain has loose integration and separated phases, while proposed work is accumulative. Proposed BBM is verifiable, easily to trace, and transparent. Proposed BBM has no over price from black market. It can also eliminate brokers. Table 2 shows hash and time of creating 10 blocks for 10 products (vehicles).
Handling Industrial Consumer Rights by Using Blockchain
383
Table 1. Comparison between proposed BBM with traditional supply chain. Aspect
Traditional supply chain Proposed BBM
Integration
Loose integration
Tight integration
Phases
Separated phases
Accumulative
Verifiability
Non-verifiable
Verifiable
Tracing
Needs effort to trace
Easy to trace
Transparency No transparency
Transparent
Price
Over price
No over price
Broker
Brokers found
No brokers
Table 2. Hash, time (in nanoseconds) of creating 10 blocks for 10 products (vehicle data). Block
Hash
Genesis
00000d99d5b74ae4083b0dc72eb037911c7e4fc3b57cdf96
Time 897281524
1
0000061ed3da10ae227de915a65c1aaff1b77653d234a6af
21908751754
2
00000e6065aed4d98294c7858edca7652b2d8fd070568a6e
39943522060
3
0000077d8138ec359b329597b4b0fefecd416bc1745f718b
4687554783
4
000008e389fd736f507fac5150befa0add1c329eb0d344e6
26650053492
5
00000415fa85279deb717159c0891989d80748b0833dbe9d
10545790901
6
000007dfea04c9b44719753d8c3342f557c1499f8e32ea85
44904216538
7
0000069537fa7bd97875936b31157f5daee6ce89b780854a
15871547062
8
000004df9037cf9b31300d7f5ee7e2e621757454db6c7494
15934173508
9
000002f135aa78738578d6640a28bb18bcc8f1d339ae9eeb
33368280209
10
00000e5ce98a36e3ed9b861b7e67c4dce28adb6d794efdef
9031627590
4 Conclusion and Future Directions A blockchain business model (BBM) that regulates the life of the consumer with a government unit is proposed which can be used to preserve consumer rights and customer satisfaction feedback. The paper proposed a prototype design of a business model that eliminates consumer rights issues by using blockchain. After applying the proposed model, interactions are verifiable and transparent. In the future, many directions may focus on enhancing the security of the proposed model. Another possible direction is to integrate tokens in the implementation. Another direction may consider measuring supply chain performance [11–14]. Another direction may be considering security of blockchain [15].
384
M. A. El-dosuky and G. H. Eladl
References 1. White, M., Killmeyer, J., Chew, B.: Will blockchain transform the public sector?, Deloitte Insights, Deloitte University Press. https://bit.ly/3f2ANAo. Accessed 5th July 2020 2. Bhardwaj, C.: What are the Benefits of Blockchain for Government Services?. https://bit.ly/ 2NWAnzJ Accessed 5th July 2020 3. Akgiray, V.: The Potential for Blockchain Technology in Corporate Governance, OECD Corporate Governance Working Papers No. 21 (2019) 4. Hileman, G., Rauchs, M.: Global blockchain benchmarking study. Camb. Centre Altern. Finance. University of Cambridge 122 (2017) 5. Swan, M.: Blockchain: Blueprint for a New Economy. O’Reilly Media, Inc., Newton (2015) 6. Sunil, C., Meindl, P.: Supply Chain Management: Strategy, Planning, and Operations. Pearson Prentice Hall, Upper Saddle River (2007) 7. Kshetri, N.: Blockchain’s roles in meeting key supply chain management objectives. Int. J. Inf. Manage. 39, 80–89 (2018) 8. OCED 2019: Is there a role for blockchain in responsible supply chains? OECD Publishing (2019) 9. Nowi´nski, W., Kozma, M.: How can blockchain technology disrupt the existing business models? Entrepreneurial Bus. Econ. Rev. 5(3), 173–188 (2017) 10. Sharma, T.K.: The Best Blockchain Business Models, Blockchain Council, 2019. https://bit. ly/2ZjzgQV. Accessed 5th July 2020 11. Beamon, B.M.: Measuring supply chain performance. Int. J. Oper. Product. Manage. (1999) 12. Shepherd, C., Günter, H.: Measuring supply chain performance: current research and future directions. In: Behavioral Operations in Planning and Scheduling, pp. 105–121. Springer, Berlin, Heidelberg (2010) 13. Lapide, L.: What about measuring supply chain performance. Achieving Supply Chain Excellence Through Technol. 2(2), 287–297 (2000) 14. Wong, W.P., Wong, K.Y.: Supply chain performance measurement system using DEA modeling. Ind. Manage. Data Syst. (2007) 15. El-Dosuky, M.A., Gamal, H.E.: DOORchain: deep ontology-based operation research to detect malicious smart contracts. In: World Conference on Information Systems and Technologies. Springer, Cham, (2019)
How Health Data Are Managed in Mozambique Lotina Burine , Daniel Polónia , and Adriana Gradim(B) University of Aveiro, Campus Universitário de Santiago, 3810-193 Aveiro, Portugal {lburine,dpolonia,adrianacoutinho}@ua.pt
Abstract. Data is the basis for designing information. In the healthcare, the management of data can improve patients’ treatment and contribute to better provide healthcare services. This study aims to explore how health data are treated in the national health system in Mozambique. The methodological strategy was based on a qualitative analysis, through a semi-structured interview made to the manager of the health information department, supported by a bibliographic search based on PRISMA. The keywords used were data, value, and healthcare, with a limitation of the articles researched. For the work, only articles referring to these terms and with access were included. After screening the articles, considering the criteria used, 49 articles were analyzed. 17 articles were then included in this study. The results of this article are important, as they offer information about the treatment of health data. Through the study it was possible to verify how essential the treatment of health data is, as it helps in decision making. It was also possible to conclude that it is necessary to computerize data and information on health, in Mozambique. The use of the health information system serves as a mechanism for the collection, processing, analysis, and transmission of information to plan, organize, operationalize, and evaluate health services. In this way, health data, when in computerized systems, assist in decision support, decreasing the hypothesis of errors and accelerating the decision-making process by health professionals. Keywords: Data · Value · Health systems · Mozambique
1 Introduction The data is one of the most important elements of the health information systems, and the information is composed of data with meaning for those who see it [1]. The data alone do not speak, they are as raw material and need to be worked on to produce information that translates into knowledge, an interpretation and a judgment about a given situation. Data collection and sharing through the healthcare system is needed and has several benefits: it can help to identify patients’ profiles and improve care; can help to identify geographical trends and groups of individuals and; can help to better allocate resources and improve healthcare service provision [2, 3]. In Mozambique, with a population of over 30 million people [4], the paper information system continues to be used in all health units. This constitutes an obstacle for © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 385–393, 2021. https://doi.org/10.1007/978-3-030-72651-5_37
386
L. Burine et al.
the correct management of health data through a global approach in the country. The government has been trying to fit the electronic system, through the development of systems that allow the collected data to flow until reaching the level of the Department of Health Information such as SIS-MA. In this way, the present article intends to analyse how health data are treated in the national health system of Mozambique to find flaws and/or improvements. The treatment of health data is extremely important, as it is through them that the information that the Ministry of Health (MISAU) uses to design strategic actions is obtained, thus supporting decision making. The information obtained helps MISAU to take measures to resolve certain cases in the health units. Considering the above, this article is structured as follows: first, we present the introduction followed by the theoretical foundation. Third, the methodology for this study is presented, followed by main results and discussion about the findings, the limitations of the study and the paths for future research. Finally, the conclusion is presented with final considerations around the practical/theoretical implications and limitations of the study.
2 Theoretical Background 2.1 The Case of Mozambique With an increase of the average life expectancy (63 years for women and 57 years for men), the way the health service in Mozambique is being improved over the years is an important topic for analysis [6]. The National Health System (NHS) in Mozambique comprises the public sector, the private sector for profit, the private sector for non-profit and the community. Of these, the public sector, that is, the NHS, is the main provider of health services at the national level. As for the for-profit private sector, it is gradually developing, especially in large cities. However, the growth of these providers is conditioned by the increase in household income. The NHS is organized into four levels. Levels I and II are the most peripheral, with the mission of providing primary care and referring patients with more serious clinical conditions - complications in childbirth, trauma, medical and surgical emergencies, among others - to the next levels. Levels III and IV are basically aimed at providing specialized curative health care. In general, primary care continues to be the dominant strategy in health intervention, aiming to reduce the high mortality rates (e.g. the child mortality rate in 2020 is 67,4%) imposed by communicable diseases such as HIV/AIDS, tuberculosis, malaria, and neglected tropical diseases [4, 7]. Health problems associated with high maternal mortality rates, around 490 deaths for 100.000 inhabitants in 2015, are also priority areas for intervention [6, 8]. In Mozambique, the Ministry of Health (MISAU) is the central body of the NHS. This is responsible for defining the policies, guidelines and standards that must be followed by each Health Unit (HU). Each region has a MISAU representation that is called the Provincial Health Directorate (PHD). Still in each district there is a representation of the NHS which is the District Service for Health, Women and Social Action (SDAMAS). The Health System Diagram is represented in Fig. 1:
How Health Data Are Managed in Mozambique
387
Fig. 1. Health system diagram. proper elaboration
MISAU has a department called the Cooperation Planning Division (CPD) in which there is a department called the Department of Health Information (DHI). This is responsible for coordinating the general health information process from validity, treatment and processing until the data are used not only at MISAU level, but also at other levels. Also, MISAU is responsible for making strategic decisions, as there are some decisions that are taken at each level, and by the HU. 2.2 Management of Health Data in Mozambique This study is based on the theoretical assumption that health data are important for obtaining information [9, 10]. In the healthcare sector, information is a fundamental source for improving processes and to help improve the decision-making process. To have information of quality, data must reflect the reality found. Data collection methodologies need to take into account the socio-cultural context and investigate in detail how perceptions and practices can facilitate or prevent data collection, to improve the quality of data [11]. Also, there is the need for continuous training on data recording procedures at all levels. To maintain data quality, health care data must be appropriate, organized, timely, available, accurate and complete [12]. Regarding healthcare services, there is a relation between a proper health system and the provision of better health services, because with this practice, patients and the hospital can gain [13, 14]. The use of electronic health records is fundamental for improving patient care [2]. Health Information Systems are composed of a structure capable of guaranteeing the collection and transformation of data into information. In this way, information systems (IS) are systems or processes that transform data into information that assist decision-making processes, efficiently, objectively and quickly, which is why they are increasingly implemented [15]. For the results to be successful, collaborative interaction between people, technologies and procedures is necessary.
388
L. Burine et al.
Also, IS are increasingly necessary in organizations and this reality is also inserted in health services [16]. IS are a strategic resource, regarding actions in decision making, both by health professionals and government officials involved in public health policies. Considering the importance of data in health, as mentioned above, data alone does not constitute information [17]. They must be treated to be conceived as information that will assist in the decision-making process of MISAU. Information systems must be simple, allowing an easy and quick access to data. They must be well organized so that they can process relevant, useful information that will facilitate decision making. It is in this sense that the strategies of using tools that assist in the work of health professionals must seek to improve the quality of care provided to the patient [16]. As an example, the SIS-MA is a flexible, open-source, web-based monitoring and evaluation platform that collects, validates, analyses, and presents aggregated statistical data, adapted to integrated health information management activities. It works as an online system and intermittent with the following features. The data is stored on a centralized server that allows access from any device in any region. Also, it allows the collection of data through aggregated data forms, which results from the combination of various items such as sex and age. At the same time, it allows the capture of the various events that occur during the patient care process based on their individual processes. This system provides the list of health units in the NHS, interactively. Through it, several reports related to the proportion and types of units can be made. Lastly, the Public Data Portal allows reporting of data collected and processed by SIS-MA without having to log in. This presentation is made using graphs, tables and maps [5]. However, some obstacles still emerge. Namely, SIS-MA is used by hospitals from the NHS even though majority of the hospitals are from the private sector. Therefore, it remains the doubt about to what extent the healthcare institutions from the public and private sectors share data. The use of health data shared among participants of the healthcare ecosystem (e.g., hospital, universities, AIDS) is useful to identify geographic trends. So, data sharing is increasingly important, especially in developing countries, such as Mozambique [2]. This sharing of data, turned in information, helps in the various spheres of MISAU. To make decisions and outline strategies at the MISAU level, data sharing is necessary. However, data must be of quality and reliable, since MISAU will take decisions based on this information. At the same time, the reduced capacity of the Mozambican health system, both in terms of human resources (0.7 clinicians per 10000 people between 2010 and 2018) and quality infrastructure, reduces access to health care by the population, especially in rural areas, which leads to an unknown reality of data [6, 8]. In the analysis of how Mozambique is evolving there is the need to considerer that the country remains among the poorest countries in the world. While in 2011 it was ranked in the 184th position out of 187 countries, according to the UNDP Human Development Report, after eight years, in 2019 it remains in the 180th position [6].
How Health Data Are Managed in Mozambique
389
3 Methodology For the theoretical basis of this article, an online search was made using Scopus and ISI Web of Knowledge. The keywords “data”, “value” and “healthcare” were used to identify all articles published on this topic. From this search, 49 articles were found. There were no duplicates of the selected articles. Then, the records were sorted by keywords and the summary with a total of 17 articles included in this study. The screening also included the keyword “Mozambique” considering the context of the research. In this screening, the PRISMA method was used, as shown in Fig. 2. PRISMA is useful for the critical evaluation of published systematic reviews to reduce the risk of bias [18].
Fig. 2. PRISMA method application
After, a pilot sheet was also developed in Excel for data extraction to gather all the information necessary to produce this article. It allows to systematize the information found, dividing it by year, title, authors, journal, keywords, summary, research questions, variables, hypotheses, methods, application context, results, future research and DOI. From the research of the databases, we can conclude that in Mozambique there are few studies regarding the treatment of health data. Of the researched articles, it was difficult to find one that addressed the topic. Information on the matter was found on the MISAU website. Another important fact observed was that, although there are few
390
L. Burine et al.
articles addressing the theme in the Mozambican case, it was possible to support the study using some articles, as example [19, 20]. After, the methodological strategy was based on a qualitative analysis, with the aid of a semi-structured interview. The manager of the health information department was interviewed to understand how data processing works in the health sector, supported by what was found in the literature.
4 Results and Discussion From the interview, it was possible to verify that data are important in health. They should be worked on for the design of information, and this information, in turn, will assist in decision making. It should be noted that MISAU, although using IS, has not yet released the paper in its entirety, having tried to computerize some procedures. The use of paper is still very present. This occurs in many locations, where there are books of records in the HU for several programs, because there are no conditions for the use of information and communication technologies. The periodicity of the information depends on the programs, but most of the forms are monthly. Those that are weekly, refer to epidemiological bulletins of diseases of mandatory notification (e.g., cases of malaria, cholera), which must be reported weekly. Some cards are still of a quarterly nature. After the registration of all events, a monthly summary is produced at the end of each month. The person responsible for the program, at the local HU is responsible for producing this summary, referring to the 30 past days. Regarding the treatment of data, after obtaining information, MISAU will assist in the decision-making process. The goal is to be able to outline some intervention strategies for the HU, thus supporting decision making. Note that great care must be taken at all levels when analyzing and verifying data to avoid disparities. However, there are flaws while making decisions and there is a risk of allocating a program in a location that does not need it. This means that one can make a wrong decision if the data is not checked effectively and efficiently. So, data from computerized systems help support decision making in health, reducing the chance of errors, and speeding up decision making by professionals. In summary, there is still the need to improve the health system in Mozambique in relation to the treatment of health data. However, the government has tried to overcome this situation. The study made it possible to verify that the use of paper is still frequent, and it should be eliminated. In the literature, several authors speak of the importance of using computerized systems [21, 22] and the need to computerize the entire system, from the bottom to the top. By computerizing data and information, it is expected that the processes are well organized and allow to assist effectively and efficiently in decision making. Having the computerized system, it is possible to have a structured database that will support health professionals. The lack of studies carried out in this area of study is also a flaw, which needs to be improved to be able to share information worldwide. In the Strategic Social Health Plan, one of the established principles and which must be obeyed is that of “Transparency and Accountability”. It appears that in Mozambique
How Health Data Are Managed in Mozambique
391
the healthcare sector seeks to be transparent and understandable. While analyzing statistics on the country, statistics on performance are out of date and there is not a clear definition the role of each person who works in the sector. Also, it seeks to provide information on the progress of the population’s health and on the efficiency, effectiveness and impact of the actions developed by the NHS. The disclosure of the sector’s statistical data is done at any level of health care provision or health system management, aggregated or not according to the level that disseminates them (district, provincial or central).
5 Conclusions From the research, it was found that information systems in health are necessary and indispensable. However, it is dependent on data, information, and knowledge. In addition, health information systems are composed of a structure capable of guaranteeing the collection and transformation of data into information. Information is power, and competitive organizations today are those that have information. Also, in health it is necessary that the information is there and that it is computerized, increasing its quality to facilitate the decision-making process. 5.1 Practical and Theoretical Implications Considering that in some cases the data processing is still done based on the use of paper, it is suggested that the government find alternative ways of transform the entire process to the computerized system. This is a practical application that was found in the literature and corroborates the idea that computerized information systems are better than paper [23, 24]. In turn, information systems play a vital role in assessing quality, as they allow timely access to records collected, processed, and documented in health. Regarding the implications, after conducting the study, there is a need to conduct more case studies in different regions, in terms of data processing, to analyse what the real situation is. With this study it was found that, in terms of literature, there is little information regarding the management of data in the healthcare sector in Mozambique. Also, there is a lack of articles regarding the treatment of data in health. 5.2 Limitations and Future Research Avenues Like any study, this investigation has some limitations. Only one interview was made. It is suggested that future studies may include interviews with various representatives of the different levels of the system to enrich the study. This would allow to assess other perceptions in relation to the treatment of health data. It is also suggested that in future investigations, interviews can be conducted with people from different entities that collaborate in the system. The involvement of different stakeholders can help to understand the network of contacts and how it can be operationalized to improve data sharing.
392
L. Burine et al.
References 1. Siqueira, M.C.: Gestão estratégica da informação. Rio de Janeiro (2005) 2. Neves, A.L., et al.: Health care professionals’ perspectives on the secondary use of health records to improve quality and safety of care in England: qualitative study. J. Med. Internet Res. 21(9), 1–10 (2019) 3. König, S., et al.: In-hospital mortality of patients with atrial arrhythmias: insights from the German-wide Helios hospital network of 161 502 patients and 34 025 arrhythmia-related procedures. Eur. Heart J. 39(44), 3947–3957 (2018) 4. Instituto Nacional de Estatística Moçambique, INE Destaques — Instituto Nacional de Estatistica (2020). http://www.ine.gov.mz/. Accessed 13 Jul 2020 5. Ministério da Saúde, “SISMA - Sistema de Informação para Saúde em Monitoria e Avaliação,” (2019). http://sisma.saudigitus.org/. Accessed 13 Jul 2020 6. United Nations Development Program, Human Development Report 2019 (2019) 7. World Health Organisation, Communicable Diseases Cluster| WHO| Regional Office for Africa (2020). https://www.afro.who.int/about-us/programmes-clusters/communicablediseases-cluster. Accessed 13 Jul 2020 8. Martins, A., Freitas, R.P., Ribeiro, S.: Benchmarking: Ficha de País, Moçabique (2013). http://www.healthyn.pt/Images/Documentos/Benchmarking_Moçambique.pdf. Accessed 13 Jul 2020 9. Vélez, M., Wilson, M.G., Abelson, J., Lavis, J.N., Paraje, G.: Understanding the role of values in health policy decision-making from the perspective of policy-makers and stakeholders: a multiple-case embedded study in Chile and Colombia. Int. J. Heal. Policy Manage. 1–13 (2019) 10. Weng, R., Huang, J., Kuo, Y., Huang, C.: Determinants of technological innovation and its effect on hospital performance. Afr. J. Bus. Manage. 5(11), 4314–4327 (2011) 11. Allison, T., Peter, S., Jonathan, C.: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int. J. Qual. Heal. Care 19(6), 349–357 (2007) 12. Surodina, S., Lam, C., de Cock, C., van Velthoven, M., Milne-Ives, M., Meinert, E.: Engineering requirements of a Herpes simplex virus patient registry: discovery phase of a real-world evidence platform to advance pharmacogenomics and personalized medicine. Biomedicines, 7(4), (2019) 13. Ezran, C., et al.: Assessing trends in the content of maternal and child care following a health system strengthening initiative in rural Madagascar : A longitudinal cohort study no. DiD, pp. 1–23 (2019) 14. Igira, F.T.: The dynamics of healthcare work practices Implications for health management information systems design and implementation. Emerald Gr. Publ. Ltd. 12(3/4), 245–259 (2012) 15. Rodrigues, R.J.: Information systems: the key to evidencebased health practice. Bull. World Heal. Organ. Geneve 78(11), 1344–1351 (2000) 16. da; Mota, L.A.N., Pereira, F.M.S., de. Sousa, P.A.F.: Sistemas de Informação de Enfermagem: exploração da informação partilhada com os médicos. Rev. Enf. Ref. Coimbra IV(1) (2014) 17. Sihvo, S., Ikonen, T., Mäkelä, M.: Implementing health technology assessment-based recommendations in Finland: managed uptake of medical methods. Int. J. Technol. Assess. Health Care 33(4), 430–433 (2017) 18. Liberati, A., et al.: The PRISMA statement for reporting systematic reviews and metaanalyses of studies that evaluate health care interventions: explanation and elaboration. J. Clin. Epidemiol. 6(7), 1–28 (2009)
How Health Data Are Managed in Mozambique
393
19. Rama, N.J., Pimentel, J.M.P., Raposo, V.: A importância das bases de dados na gestão do conhecimento em saúde. Rev. Port. Ciurgia 36(2), 436–437 (2016) 20. Roque, A.C.: Disease and cure in Mozambican health service reports from the end of the nineteenth century. Hist. Ciencias, Saude - Manguinhos 21(2), 515–537 (2014) 21. Graciano, M.M.C., Araújo, E.W., Nogueira, D.A.: Sistema de informação em saúde e atuação do profissional médico. Rev. Médica Minas Gerais 19(3), 198–205 (2009) 22. dos Santos, T.O., Pereira, P.L., Silveira, D.T.: Implantação de sistemas informatizados na saúde: uma revisão sistemática. Rev. Eletrônica Comun. Informação e Inovação em Saúde, 11(3), 1–11, (2017) 23. Guedes, A.: A aceitação do registo de saúde eletrónico pelos profissionais de saúde das instituições hospitalares. Escola Nacional de Saúde Publica, Universidade Nova de Lisboa (2011) 24. Pinto, L.: SI e profissionais de enfermagem., Universidade de Trás-os-Montes e Alto Douro (2009)
Multi-perspective Conformance Checking Applied to BPMN-E2 Rui Calheno1(B) , Paulo Carvalho1 , Solange Rito Lima1 , Pedro Rangel Henriques1 , and Mateo Ramos-Merino2 1
Centro Algoritmi, Universidade do Minho, Braga, Portugal [email protected], {pmc,solange,prh}@di.uminho.pt 2 Universidade de Vigo, Vigo, Spain [email protected]
Abstract. One of the most used notations for process modelling is the Business Process Model and Notation (BPMN), being the expressiveness in representing processes its strongest attribute. However, this notation has shortcomings when dealing with some specific domains (namely Hazard Analysis and Critical Control Points systems), struggling to model activity duration, quality control points, activity effects and monitoring nature. For this particular purpose, an extension named Business Process Model and Notation Extended Expressiveness (BPMN-E2 ) was developed to tackle the limitations found in the original notation. In this paper, a multi-perspective conformance checking algorithm is proposed focusing on detecting non-conformities between an event log and a process model, regarding the information provided by the new elements within BPMNE2 . Despite being based on this new notation, the proposed algorithm can be applied to other process model notations as it follows a twostep approach that starts by converting the model into a directly follows model (annotated with conformance rules), which is then used in the second phase to perform the conformance checking task effectively.
Keywords: BPMN Process mining
1
· BPMN-E2 · Conformance checking ·
Introduction
Nowadays, people and organisations depend more and more on technology and, as a consequence, more data is constantly being collected [1]. Considering this, today’s organisations aim to extract information and value from data stored in their information systems to better improve their business and gain a competitive advantage over other organisations [1,8]. Process mining is a discipline that combines data mining and process modelling to properly analyse the ever growing event data. It aims to discover, monitor and improve real processes, by extracting meaningful insights and knowledge from event logs [9]. To do so, process mining c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 394–404, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_38
Multi-perspective Conformance Checking Applied to BPMN-E2
395
relies on techniques that can be grouped into one of three areas considering its intended purpose [1]: Discovery, Conformance Checking and Enhancement. BPMN-E2 is an extension for the BPMN language that gives to the process model designer the possibility of describing in more detail the workflow behaviour, the activities being performed and the context of one particular process [14]. The need of a more context-aware approach surfaced when dealing with process control environments, particularly Hazard Analysis and Critical Control Points (HACCP) based environments where it is vital to ensure the safety and security of products manufactured in a given industry (e.g. food, pharmaceutical or cosmetics) [14]. Such a process description is exemplified in Fig. 1.
Fig. 1. BPMN documentation excerpt of PN mixtures elaboration process [14].
Fig. 2. Extending the model in Fig. 1 using BPMN-E2 notation.
In this paper, we propose the development of a new conformance checking algorithm that takes into account the extended expressiveness of BPMN-E2 notation. This data-aware multi-perspective technique allows for a richer process analysis, providing insights on activity durations, activity effects, and decision points related non-conformities. Furthermore, it takes advantage of BPMN-E2 distinction of monitored and non-monitored activities to provide reliable conformance results by appropriate filtering. This is expected to reduce possible false-negatives conformance errors from being detected when working with partially monitored environments. This paper is organised as follows. After introducing the research topic and motivation, key background concepts are presented in Sect. 2. The conformance checking proposal is introduced and explained in Sect. 3. The proposal implementation is explained and evaluated in Sect. 4. This proposal is also framed in
396
R. Calheno et al.
the context of related work, as discussed in Sect. 5. Finally, the main conclusions and future work prospects are provided in Sect. 6.
2
Background
To model a business process with process mining tasks in mind, the notation to use should ensure that process mining techniques can be executed flawlessly, without compromising the understandability of the results (representational bias) [9,18]. Over the years, several process model notations where proposed, such as Petri nets, causal nets, process trees and BPMN, being this the standard notation for business process modelling, used by a variety of professionals in their everyday jobs (business analysts, product managers, technical designers, system architects and more) [9]. However, despite having a great support for process modelling, the BPMN notation has some drawbacks when dealing with particular domains, namely HACCP systems [14]. In these scenarios, the business processes are explained in detail using both a BPMN diagram and a natural language description that explains the activity behaviour, quality controls or how to proceed in case of hazards. Jointly, a BPMN model and a natural language description provide a rich and complete representation of the business process, however, only the model can be used during process mining tasks, which leads to the following issues [14]: (i) difficulty in representing specific context details, complete workflow activities, and the semantics of the path selection to be taken during a process instance. Temporal features, identification of quality control and monitoring points and the effects of an activity on the characteristics of a product cannot be modelled, neither from a visual nor from a machine-readable perspective; and (ii) possible misleading conformance checking results when monitored and non-monitored activities are present in the same model. Table 1. New elements introduced by the BPMN-E2 notation [14]. Elements Monitoring Point Activity Effect Activity duration
Advanced decision point
Definition Represents the measurement of a variable or a set of variables in a specific point in the workflow. Represents the activity effect, i.e. how the activity affects a product or how it can change the product characteristics. Represents an estimation of the expected execution time of an activity. Represents a decision point that exposes the reasons of a particular choice, connecting the possible choices and paths with the characteristics of the product and the measured variables. It also distinguishes Normal (above) and Quality (below) decision points.
Graphics
Multi-perspective Conformance Checking Applied to BPMN-E2
397
Furthermore, human auditors should study in detail both documents, namely the BPMN and the one expressed in natural language, which could over complicate auditing tasks. To solve these limitations, the authors of BPMN-E2 notation aimed to extend the BPMN element set with new human and machine readable elements (Table 1) that better represent the contextual information of HACCP processes, formerly only present in natural language documents. Note that, despite the priority given to HACCP systems, this notation can be easily applied to other domains requiring the same level of richness and expressiveness. Considering these new elements, and the fact that they convey not only a graphical but also a machine-readable representation, richer context-aware process models can be designed carrying information about the activities being performed and the overall process. For example, the process modelled in Fig. 1 can be modelled using BPMN-E2 providing a centralised source of information about every activity of the process (see Fig. 2).
3
Proposal
This section explains the conformance checking proposal advocated in this work, starting by highlighting its main design goals, detailing then the rational and mechanisms sustaining the proposal. 3.1
Design Goals
Considering that BPMN-E2 focuses on a process’ data flow, the developed conformance checking mechanism must primarily be focused on the data perspective of process mining and, thus, providing an efficient and viable way to pinpoint and warn about the following deviations: 1. inconsistent activity effect - assuming the existence of the “Activity effect” element, it should be possible to verify if one or more data variables are being properly affected by an activity; 2. inconsistent activity duration - assuming the existence of the “Activity duration” element, it should be possible to compare the observed activity duration with the expected duration; 3. wrong path selection - assuming the existence of the “Advanced decision point” element, it should be possible to verify if a specific case took the right path according to the values of one or more variables. Moreover, distinguishing between “quality” and “normal” decision points, will enable taking different measures regarding which type of decision point was broken. Furthermore, the mechanism should be able to distinguish between monitored and non-monitored activities and act accordingly. Non-monitored activities do not produce event log records, and so it is important to disregard these activities during conformance checking in order to reduce false negatives and provide more viable results. To achieve this, an initial solution was devised considering two phases: 1) conversion, from BPMN-E2 notation to a more suitable structure; and 2) conformance checking, replaying through the event log while checking with the previous structure for eventual non-conformities.
398
R. Calheno et al. temperature = Int('temperature') s = Solver() s.add(temperature > 7) s.add(temperature == 9) s.check() # satisfied
Listing 1: Process of rule verification in Python using Z3 SMT solver.
3.2
Conversion Phase
To abstract the conformance checking algorithm from the initial model, an intermediate representation of the rules to be followed is needed when replaying the log. One viable structure that can accurately store this information is a Directly Follows Model (DFMs) where the nodes represent the modelled activities and the edges represent a sequence flow between activities. In [10], DFMs are syntactically described as directed graphs in which the nodes are either an activity, start or end. Therefore, the language of DFM consists of all traces that can be obtained when flowing from the start node to the end node. Considering this, it is possible to extend a DFM by annotating each edge with a set of conformance rules that must be followed when flowing from node n1 to node n2 . Consequently, this approach is here defined as Directly Follows Rules Model (DFRM). These rules can then be checked during conformance checking tasks to detect possible non-conformities and deviations during process execution. SMT Solvers as Conformance Rules Satisfiability Modulo Theories (SMT) addresses the problem of deciding the satisfiability of a first-order formula with respect to some background theory [17]. An SMT Solver is a tool for deciding the satisfiability of formulas in these theories [12]. Attending to its logical nature, the problem of verifying a conformance rule can be seen as an SMT problem considering, in this case, the following background theories: Arithmetic, Real and Arrays. Consequently, a conformance rule can be mapped into an SMT Solver instantiated with an initial set of assertions. Thus, a conformance rule can be seen as an SMT Solver instantiated with an initial set of assertions that refer to the conditions of a BPMN-E2 element. During conformance checking, these rules can be verified by adding a set of new assertions based on the event log data and solving the SMT problem. In case of satisfiability, the rule is also satisfied, otherwise, the rule is not verified. In the code snippet in Listing 1, the process of verifying a rule is demonstrated3 . The SMT Solver is initialised with an assertion (temperature > 7), this represents the conformance rule to check. Then an equality assertion is added (temperature == 9), this represents the actual value provided by the event log. Finally, the rule is checked. In this case, since the provided temperature value is bigger than 7, the conformance rule is satisfied. With DFRM’s and SMT Solvers as conformance rules in mind, the conversion phase consists of parsing the BPMN-E2 source file whilst converting the new
Multi-perspective Conformance Checking Applied to BPMN-E2
399
elements into first order assertions and associating them with the respective edge of the DFRM. This conversion process is exemplified in Fig. 3.
Fig. 3. Excerpt of a BPMN-E2 model, extended with advanced decision points and activity duration (left), and the respective DFRM graph after conversion (right).
Dealing With Non-Monitored Activities One of the biggest advantages of BPMN-E2 notation is the identification and distinction of monitored and non-monitored activities, thus providing a way to graphically represent processes with partially monitored activities without influencing conformance-checking results. Consider, for instance, the non-monitored activity A27 modelled in Fig. 2; traditional conformance checking approaches would be expecting records of this activity execution in an event log, which would lead to the expected yet incorrect identification of a conformance error. To overcome this drawback, non-monitored activities must be filtered during the conversion phase assuring that the inputs of non-monitored activities become connected to the corresponding outputs. The automation of this process allows to maintain the consistency of the diagram and to prevent the loss of information in the workflow [14]. 3.3
Conformance Checking Phase
The second phase comprises the conformance checking algorithm. This algorithm will receive an event log and a previously generated DFRM as inputs, and will produce a detailed report concerning the data-flow of the process as output. The event log is parsed case-by-case, activity-by-activity keeping track of the most recent attribute updates. It is expected that the process’ activities will create and alter these attributes. By comparing changes in their values, it is possible to: 1) extract activity’s duration; 2) extract activity’s effect; and 3) extract the path that was taken. These data is then fed to the previously generated DFRM, being used to check the satisfiability of the respective set of conformance rules. All detected non-conformities are recorded, storing the conformance rules that were broken, the cases they occurred in, and the activities and the parameters responsible for them. Considering the four quality criteria (fitness, precision, generalisation and simplicity) that can be used to access the quality of a given model [1], we propose
400
R. Calheno et al.
a way to compute the fitness of a model, since it measures the proportion of the event’s log valid behaviour according to the model. Firstly, we can consider the fitness of an individual trace τ to be the percentage of rules that were satisfied. This way, assuming a BPMN-E2 model, the fitness of a given trace can be computed using Eq. (1) weight(ς) ς∈srules , if trules = [] ς∈trules weight(ς) f (τ, m) = (1) 1 , if trules = [] where srules is the list of conformance rules that were satisfied, trules is the list of conformance rules that were tested and weight is a function that retrieves the weight for a particular conformance rule. At the event log level, fitness can be computed by averaging the fitness values of each log trace. For an event log l and a model m, the fitness is computed using Eq. (2): f (τ, m) (2) f (l, m) = τ ∈l length(l) Note that, in Eq. (1), if no conformance rules are tested it is considered as if there are no data-flow deviations. However, it can be the case that the trace does not follow the correct control-flow in the first place, leading to certain rules not being tested. Thereby, we encourage to complement our approach with stateof-the-art control-flow conformance checking techniques, such as token replay and alignments. This “hybrid” approach will provide insights on both control and data-flow that can be leveraged to attain a more complete conformance analysis.
4
Proof of Concept
Considering the increasing support for Process Mining, Process Modeling and Data Science, we opted to develop the library in Python, thus taking advantage of the existing libraries, namely Process Mining for Python (PM4Py [3]). This further contributes to python’s ecosystem by providing the following utilities for the BPMN-E2 notation and the proposed conformance checking approach: (i) import BPMN-E2 diagrams; (ii) convert BPMN-E2 diagrams to the respective DFRMs; (iii) visualise BPMN-E2 diagrams and DFRMs using Graphviz; (iv) execute the proposed data-aware conformance checking task; and (v) produce pandas dataframes based on conformance checking results, enabling a customized analysis according to the user’s business needs, such as the report in Fig. 4. To evaluate the conformance checking proposal, four BPMN-E2 diagrams were tested against three event logs. The used diagrams are composed of three activities (A, B, C) and three monitoring groups. The first diagram (referred to as AD) also contains an activity duration element, the second diagram (AE) also contains an activity effect element, the third diagram (DP) also contains a decision point element, finally, the forth diagram (ALL) contains all three elements from the previous diagrams.
Multi-perspective Conformance Checking Applied to BPMN-E2
401
Fig. 4. Example of a conformance checking HTML report.
To evaluate the scalability of the solution, each log had an increasingly number of cases (1k, 10k and 100k) with no deviations regarding both control-flow and data-flow. For each log, the conformance checking task was executed five times and its duration recorded. The final time duration was obtained by averaging all the recorded time durations, as shown in Table 2. It is important to note that the event logs and diagrams used are synthetic and were intended to evaluate the performance of testing each type of rule separately and together. Table 2. Conformance checking task duration (in seconds). Diagram Log 1k Log 10k Log 100k AD
0.373
3.697
37.283
AE
0.355
3.696
35.953
DP
0.371
3.703
37.243
ALL
1.085
10.843
108.351
Analysing the obtained results by isolating the event log, it is possible to notice that verifying a rule takes approximately the same time regardless of its type. This was expected as all rules share the same logic. Furthermore, it is expected that the conformance checking task scales linearly regarding the number of rules being tested per case. This can be backed up by the obtained results, considering a linear increase in the task duration for bigger logs and BPMN-E2 diagrams with more rules to be verified. Therefore, the solution may underperform for larger event logs since that every log trace must be considered,
402
R. Calheno et al.
given that cases with the same execution trace can yield events with different attribute values. A solution to this problem is suggested in Sect. 6 as future work.
5
Related Work
The lion’s share of process mining research focuses on control-flow, i.e., the ordering of activities [6]. Therefore, the majority of the effort put on conformancechecking techniques is pointing towards the control-flow perspective, ignoring other perspectives such as data, resources and time [5]. In this way, there can be several deviations that are not caught during a conformance-checking task, e.g., activities that take longer than what is expected, activities that should not be executed by a certain resource or activities that fail to make specific changes. Considering that focusing only on one perspective can lead to incomplete diagnosis, data-aware (also called multi-perspective) conformance-checking techniques were developed [5,11]. In [5] a technique is proposed that extends controlflow alignments to incorporate other perspectives by constructing an Integer Linear Programming (ILP) problem and, consequently, solving it. In [11], a different approach is proposed using Compliance Rules Graphs (CRG) to declare a set of rules that the process execution must obey, each rule is bound to an activity, thus enabling to pinpoint the cause of a violation. More recently, in [10], process mining discovery and conformance approaches using Directly follows Models were proposed, given its intuitiveness and simplicity; however, these techniques focused on control-flow instead of data-flow. As explained in Sect. 3, DFMs served as the baseline for this work, being extended with conformance rules to accommodate data-aware conformance checking. In the presented proposal, the chosen strategy takes into account data reflecting domain knowledge information represented in conjunction with the process model. In this regard, the used BPMNE2 extension offers the possibility to detail the workflow behaviour, the activities being performed and the context of one particular process. A literature analysis shows that there is a trend to incorporate different knowledge in the process model to improve the applicability of process management techniques. In this line, [13] emphasises the importance of understanding the application context information for enhanced process analysis. In [19], a pre-warning analysis system that analyses product abnormalities from a data-centric perspective is developed. The work described in [2] highlights the formal definition of the new modelling elements for the monitoring events generated by processes. Both [7] and [15] discuss the idea of further developing the description of decision points. Finally, other works such as [16] and [4] develop the concept of providing additional information related to specific activities. Nevertheless, these works analyze specific issues of data representation; meanwhile, the BPMNE2 extensions offers a multi-point of view perspective of different types of information. This aspect allows the creation of multi-perspective conformance checking techniques as the one proposed in this paper.
Multi-perspective Conformance Checking Applied to BPMN-E2
6
403
Conclusions and Future Work
In this paper, a new data-aware conformance checking approach is proposed aiming at satisfying the two main objectives stated by the authors of BPMN-E2 : a) avoid the generation of false-negative conformance errors in partially monitored processes; and b) take advantage of all information related to the process model that was previously only available in natural language, and was then translated to a machine-understandable representation. The two-step conformance checking algorithm described in this paper tackles both objectives by filtering the nonmonitored activities and by producing a detailed report for each new element of the BPMN-E2 , respectively . To provide a more complete analysis regarding both control-flow and data-flow, the combination of our approach with state-ofthe-art alignment-based conformance checking algorithms is also encouraged. A implementation of the proposal was also carried out allowing for the execution of multi-perspective conformance checking and the generation of dataframes for subsequent analysis. Although the core functionalities are set, additional adjustments may be required to improve its performance and/or results. Therefore, future work will consist in improving the developed conformance checking library, tackling its current limitations regarding the scalability issue stated in Sect. 4 we intend to develop a case clustering algorithm that groups cases based on the attribute’s values of their events thus reducing the event log size and complexity for multi-perspective approaches. Acknowledgement. This work has been supported by FCT – Funda¸ca ˜o para a Ciˆencia e Tecnologia – within the R&D Units Project Scope: UIDB/00319/2020.
References 1. van der Aalst, W.: Data Science in Action. In: Process Mining, pp. 3–23. Springer, Berlin (2016) 2. Baumgrass, A., Herzberg, N., Meyer, A., Weske, M.: BPMN extension for business process monitoring. Enterp. Model. Inf. Syst. Architect. EMISA 2014, 85–98 (2014) 3. Berti, A., van Zelst, S.J., van der Aalst, W.: process mining for python (PM4Py): bridging the gap between process- and data science. In: CEUR Workshop Proceedings vol. 2374, pp. 13–16 (May 2019). http://python.org http://arxiv.org/ abs/1905.06169 4. Braun, R., Schlieter, H., Burwitz, M., Esswein, W.: BPMN4CP: Design and implementation of a BPMN extension for clinical pathways. In: 2014 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 9–16. IEEE (2014) 5. De Leoni, M., Van Der Aalst, W.M.: Aligning event logs and process models for multi-perspective conformance checking: An approach based on integer linear programming. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8094 LNCS, pp. 113–129 (2013) 6. De Leoni, M., Van Der Aalst, W.M.: Data-aware process mining: discovering decisions in processes using alignments. In: Proceedings of the ACM Symposium on Applied Computing, pp. 1454–1461 (2013)
404
R. Calheno et al.
7. Friedenstab, J.P., Janiesch, C., Matzner, M., Muller, O.: Extending BPMN for business activity monitoring. In: 2012 45th Hawaii International Conference on System Science (HICSS), pp. 4158–4167. IEEE (2012) 8. Grupe, F.H., Owrang, M.M.: Data base mining: discovering new knowledge and competitive advantage. Inf. Syst. Manage. 12(4), 26–31 (1995) 9. Kalenkova, A.A., van der Aalst, W.M., Lomazova, I.A., Rubin, V.A.: Process mining using BPMN: relating event logs and process models. Softw. Syst. Model. 16(4), 1019–1048 (2017) 10. Leemans, S.J., Poppe, E., Wynn, M.T.: Directly follows-based process mining: exploration & a case study. In: Proceedings - 2019 International Conference on Process Mining, ICPM 2019, pp. 25–32 (2019) 11. Ly, L.T., Rinderle-Ma, S., Knuplesch, D., Dadam, P.: Monitoring business process compliance using compliance rule graphs. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 7044 LNCS (PART 1), pp. 82–99 (2011) 12. de Moura, L., Bjørner, N.: Z3: An Efficient SMT Solver. In: Lecture Notes in Computer Science, pp. 337–340. Springer-Verlag (2008), http://link.springer.com/ 10.1007/978-3-540-78800-3 24 13. Musa, A., Gunasekaran, A., Yusuf, Y.: Supply chain product visibility: methods, systems and impacts. Expert Syst. Appl. 41(1), 176–194 (2014) ´ 14. Ramos-Merino, M., Santos-Gago, J.M., Alvarez-Sabucedo, L.M., Alonso-Roris, V.M., Sanz-Valero, J.: BPMN-E2: a BPMN extension for an enhanced workflow description. Softw. Syst. Model. 18(4), 2399–2419 (2019) 15. Rodr´ıguez, A., Fern´ andez-Medina, E., Piattini, M.: A BPMN extension for the modeling of security requirements in business processes. IEICE Trans. Inf. Syst. 90(4), 745–752 (2007) 16. Saeedi, K., Zhao, L., Sampaio, P.R.F.: Extending BPMN for supporting customerfacing service quality requirements. In: 2010 IEEE International Conference on Web Services (ICWS), pp. 616–623. IEEE (2010) 17. Sebastiani, R., Armando, A., Barrett, C., Bozzano, M., Bruttomesso, R., Cimatti, A., Franzen, A., De Moura, L., Ghilardi, S., Griggio, A., Krstic, S., Nieuwenhuis, R., Oliveras, A., Ranise, S., Roveri, M., Strichman, O., Tacchella, A., Tinelli, C.: Lazy satisfiability modulo theories important discussions with. J. Satisfiability Boolean Model. Comput. 3, 141–224 (2007) 18. Van Der Aalst, W., Adriansyah, A., De Medeiros, A.K.A., et al.: Process mining manifesto. In: Lecture Notes in Business Information Processing. vol. 99 LNBIP, pp. 169–194 (2012) 19. Zhang, K., Chai, Y., Yang, S.X., Weng, D.: Pre-warning analysis and application in traceability systems for food production supply chains. Expert Syst. Appl. 38(3), 2500–2507 (2011)
Event-Driven Ontology Population - from Research to Practice in Critical Infrastructure Systems David Graf1,2(B) , Wieland Schwinger1 , Werner Retschitzegger1 , Elisabeth Kapsammer1 , and Norbert Baumgartner2 1 Johannes Kepler University, Linz, Austria {david.graf,wieland.schwinger,werner.retschitzegger, elisabeth.kapsammer}@cis.jku.at,[email protected] 2 team GmbH, Vienna, Austria [email protected]
Abstract. In an interconnected world, the Systems-of-Systems (SoS) paradigm is prevalent in various domains, particularly in large-scale environments as being found in the area of critical infrastructures. Due to the characteristics of SoS and those of critical infrastructures, the realization of high level services for Operational Technology Monitoring (OTM), such as failure cause reasoning, is challenging, whereas interoperability and evolvability are most pressing. In this realm, the contribution of this paper is twofold: Firstly, we conduct a systematic literature review focusing on semantic technologies in areas like (i) semantic annotations, (ii) event log focused work in the IoT, (iii) organizational process mining, and (iv) complex event processing. Based thereupon, we elaborate towards a hybrid (semi)-automatic ontology population approach in the context of OTM by combining inductive and deductive methods. Keywords: Systems-of-systems · Critical infrastructure systems Ontology population · Operational technology monitoring
1
·
Introduction
Operational Technology Monitoring (OTM). In an interconnected world, the Systems-of-Systems (SoS) paradigm is prevalent in various domains, particularly in large-scale environments as being found in the area of critical infrastructures. One example of such a Critical Infrastructure System (CIS) are Intelligent Transportation Systems (ITS). By the interplay of various constituent systems (e.g., a video system, electronic road signs, or tunnel operating programs), ITS’s aim is to provide services on top (e.g., efficient monitoring and controlling of traffic) based on a wide range of technologies, aka. Operational Technologies (OT) This work is supported by the Austrian Research Promotion Agency (FFG) under grant FFG Forschungspartnerschaften 874490. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 405–415, 2021. https://doi.org/10.1007/978-3-030-72651-5_39
406
D. Graf et al.
being more and more based on Internet-of-Things (IoT) [25]. Due to the high demands on the reliability of these services, adequate techniques for monitoring the wide range of OT used, i.e., Operational Technology Monitoring (OTM) are needed. In this respect, OTM is responsible to ensure high availability of OT, for instance, to reactively or even proactively trigger maintenance actions in case of an identified (prospective) failure of OT. Challenges in CIS. The realization of high level services for OTM (e.g., service quality monitoring of OT [10] or failure cause reasoning) is, however, due to the characteristics of SoS [21] and those of critical infrastructures [6], challenging. Constituent systems, mostly geographically distributed, are focusing on very specialized and encapsulated tasks, often operating in an isolated manner, which lead to massive heterogeneity at different levels [27]. Hence, interoperability is lacking and thereby an integrated view of OTM across systems is very limited. This is aggravated by prevalent legacy systems, since building up the entire infrastructure from scratch is not the standard-case in CIS, predominately showing heterogeneity of data lacking structured and semantic information. In addition, the dynamic nature of such CIS environments lead to omnipresent evolution of systems comprising behavioral aspects (e.g., OT failures, OT maintenance) as well as structural aspects, meaning that the underlying OT is continuously added, removed or replaced. Thus, there is the need to deal with evolvability within OTM, which dramatically increases its complexity. These key challenges hamper an integrated and up-to-date view, i.e., a conceptual representation, of the entire OT infrastructure, being independent from the actual technology used, which is, however, an indispensable prerequisite for enabling efficient OTM and providing high level services for OTM on top. (Semi-)Automatic Ontology Population. A promising paradigm to address these challenges are semantic technologies in terms of ontologies [9]. While the ontology’s T-Box can be manually specified by domain experts through modeling the OT objects and the relationships in between at type-level, it is simply not feasible from a practical point of view to manually populate an ontology’s ABox with hundred thousands of objects and their links in between. To give an example from practice, the national highway network focused by our work and being the underlying example throughout this paper comprises more than 100.000 OT devices of more than 200 different types, ranging from simple sensing and actuating devices (e.g., a CO-sensor) to more complex systems consisting of many devices of various types (e.g., a video system), which are geographically distributed over 2.220 km highway and 165 tunnels (cf. our previous work, e.g. [11]). Thus, (semi-)automatic ontology population is a must, especially in the light of evolvability outlined above. Contribution and Paper Structure. In order to enhance (semi-)automatic population of an OT ontology’s A-Box, the current paper’s contribution is twofold: In Sect. 2, we conduct a systematic literature review of most promising approaches of related research areas. Based thereupon, we elaborate towards a hybrid (semi)automatic ontology population approach in the context of OTM by combining inductive and deductive methods in Sect. 3. While using implicit knowledge
Event-Driven Ontology Population
407
provided by OT event streams as a basis, we gain additional information on top by representing explicit knowledge provided by domain experts. We conclude and discuss future work in Sect. 4.
2
Promising Lines of Research
Addressing the primary goal of our work namely (semi-)automatic population of an ontology’s A-Box with OT objects and their links in between, related work can be found in various areas most relevant in those of (i) semantic annotation from text or semi-structured data, surveyed by [20], (ii) event log focused work in the IoT such as event log mining and event analysis, (iii) the area of organizational process mining using semi-structured log data as data source to extract knowledge about underlying resources, most promising mining approaches surveyed by [22], as well as (iv) complex event processing as a specific form of stream processing dealing with the streaming aspect in such environments, most promising approaches surveyed by [1]. The following literature review considering these areas and compares related work primarily based on the source data structure, the techniques applied, and the target data structure. Table 1 gives a summary along these comparison dimensions. Semantic Annotation. The work of [14] and [28] use data-driven techniques such as clustering and semi-supervised classification to populate a domain ontology, the latter being closely related regarding the target data structure by populating an ontology’s A-Box with resources in terms of web services. Both, however, use primarily unstructured text documents as data source and do not use semistructured stream data originating from event logs. Closely related to our work is the approach of [4] populating an event ontology’s A-Box for monitoring vineyards grounded on a heterogeneous IoT sensor network aiming to mine causality relationships between events occurred during the life cycle of a wine production. Although the data originates also from IoT sensors, the target ontology primarily focus on events and not on a representation of the underlying IoT objects. Related with respect to techniques used are approaches in the area of semantic annotation, i.e., the process to annotate entities in a given text with semantics [8] (e.g., using ontology classes), such as the work of [18,19], which, however, address primarily the input data source themselves (often in terms of text documents) as target data structure. Event Log Mining. While the work considered so far focus rather on the target data structure, event log mining and event analysis focus on the source data structure, i.e., dealing with huge amount of event log data generated on a daily basis in various systems. Data mining techniques and tools are commonly used, such as the work of [3] applying natural language processing and information extraction techniques by using the tool GATE in order to semantically enrich event log data. Moreover, [31] use the tool LogClusterC to mine and discover line patterns, similar to event types, from semi-structured textual event logs. Both, however, focus on enrichment of the log itself, rather than using information provided by the log for other purposes such as failure reasoning or populating
408
D. Graf et al.
an ontology. Closely related to ours is the work of [12] transforming air quality sensor data to a semantic-based representation in terms of a Ressource Description Framework (RDF) model. In addition, their presented “event and clustering analytic server”, considered as a middleware, is based on the OpenIoT platform [16] being a well-known semantic-based framework in the IoT context considered in one of our previous work [9]. The created ontology, however, models the event data itself, rather than representing the underlying resources, i.e., the individual sensors. With respect to mining correlations from event logs, the approach followed by [30] is worth to mention, introducing an abstract concept of a “service-hyperlink” which represents dependencies of data services based on the correlation of events in the log. Their focus, however, lies on dependencies between services rather than between individual devices of OT. Organizational Process Mining. Aiming to “derive the underlying organizational structure of a CPS” from event logs, most promising mining approaches are reviewed by [22]. Discussed techniques such as “metrics based on (possible) causality” focusing on temporal succession of activities, or “metrics based on joint cases” focusing on frequency and correlation of resources, being suitable to derive relationships between objects, seems to be promising for our work, thus being considered as widely related. This is also the case for variations of distance measures, e.g., those used by [26], and traditional clustering techniques applied to event logs, e.g., those used by [15] as well as, since “time is a key relation between pieces of information” [7], time-based approaches such as the organizational mining approach of [13]. With respect to the source data structure and the techniques used, closely related is the approach of [5] in terms of semantically annotating event log information in order to enhance the discovery of unknown dependencies, i.e., how activities, or events respectively, are connected or who performed the activity, i.e. the resources respectively. Although these parts of their work are similar to ours, their focus lies on enhancing medical decision making rather than representing underlying resources. Complex Event Processing. With respect to the streaming aspect prevalent in the environment considered by our work, we can find related approaches in the area of stream processing and one of its specialization namely complex event processing (CEP). The event based pattern matching approach TPStream based on Allens interval-algebra [2] proposed by the work of [17] identifies temporal relationships between events, however, their work focus on identifying complex temporal patterns rather than derive additional knowledge from rather simple patterns. Closely related is the work of [7] proposing a new semantic model for real-time reasoning from sensor data. As in our work they transfer event data to a semantic based model in terms of RDF triples, however, they represent context information of events, rather than derive knowledge of underlying resources. Related regarding identifying relationships between OT devices is the approach followed by [29] in terms of combining event information with background knowledge, however with the aim to improve processing quality rather than deriving relationships. Worth mentioning with respect to pattern learning is the approach of [23] using rule-based machine learning to learn new patterns prevalent in event data, thus considered as widely related to our work.
Event-Driven Ontology Population
409
As Table 1 shows, although approaches discussed so far are related to our work in some of the comparison dimensions, however, none of them is directly applicable to our requirements since none of them aim populating an ontology’s A-Box as target data structure based on event log data originating from message streams as data source structure. Hence, in the following, we propose a hybrid approach for populating an OT ontology’s A-Box inspired by approaches discussed so far.
Source
Structure
unstructured
semi-structured at-rest
semi-supervised
Data
distance-based
Analysis
similarity-based
others Other
tool-usage (e.g, GATE)
Methods
CEP
health events
roles
web services Other
org. model
Formalisms documents
RBAC model event patterns a
survey, b preliminary work and not yet validated
log events Ontology
IoT data OT
law events
Resource-
stream processing Ontology
specific data-mining wine events
Mehdiyev et al. [23]
temporal-based
Content-
Teymourian et al. [29]
unsupervised
Target
Korber et al. [17]
Endler et al. [7]
Detro et al. [5]b
Jin et al. [15]
Jafari et al. [13]
Ni et al. [26]
Matzner u. Scholta [22]a
IE Learning
NLP supervised
Zhu et al. [30]
in-motion
Machine
Techniques Information text-based Retrieval
CEP
structured Data
Hromic et al. [12]
Zhuge et al. [31]
Amato et al. [3]
Reyes-Ortiz et al. [28]
Mining
Belkaroui et al. [4]
Process
Mining
Jayawardana et al. [14]
Event Log
Liu et al. [19]
Semantic Annotation
Lin et al. [18]
Ganino et al. [8]
Table 1. Related approaches
410
3
D. Graf et al.
Hybrid Ontology Population Approach
Approach at a Glance. Our approach is designed for real-time use, i.e., to be applied to a stream of human interpretable service-messages (e.g., a sensor sends a CO2 value) and status-messages (e.g., a device notifies an error) consisting of (i) human interpretable message text, (ii) a unique identification of the affected OT device, (iii) the OT device type, and (iv) temporal information. The rational behind using human interpretable messages is that those messages can be seen as the lowest common denominator of various heterogeneous systems, since all of them somehow communicate and interact with a human operator, i.e., send messages and receive control actions. The main idea of our hybrid approach is to combine inductive and deductive methods in order to populate an OT ontology’s A-Box. Thereby, the inductive part is based on the event-information of logs, i.e., the messages, to (i) instantiate OT devices as objects, and (ii) compute the correlation between those devices allowing to (semi-)automatically derive potentially existing links among them. This inductive part is complemented by the deductive part of our approach in terms of gaining additional A-Box information based on the conceptualization of the domain, i.e., the T-Box (OT-types and their relationships in between), modeled by domain experts. Rational of our Approach. The rational behind our proposed approach, or respectively, the reasons why a semantically rich OT ontology A-Box is beneficial for OTM purposes, is the following. Firstly, reasoning at A-Box level provides more information with respect to the monitored environment than reasoning on T-Box level, only, since the A-Box can be considered as a concretization of the T-Box (e.g., location information of individual OT objects showing spatial proximity to other OT objects). Secondly, meta-data about individual OT objects can be used for reasoning, which is crucial for OTM (e.g., hours of operation impacting the probability of a certain failure category). Thirdly, the OT ontology A-Box can be linked with context-information highly relevant for OTM (e.g., current weather or traffic situation impacting the criticality of an OT failure). Fourthly, links between objects allow identifying concrete other objects being affected by a failure or being the potential cause of a failure (e.g., the camera failure is caused by a temporarily overloaded server). In the following, we discuss the details of our approach based on the three core phases (i) correlation analysis, (ii) link instantiation, and (iii) object instantiation (cf. Figure 1). Correlation Analysis (1). In order to identify the correlation between OT devices (which further allow to derive links in the link instantiation phase), we adhere on techniques for calculating the correlation between individual OT devices based on temporal information of events (adapting and extending most promising work discussed above such as [17] to our specific purposes). The rational behind using techniques of (temporal) correlation analysis to derive links is that events of OT devices, which have some kind of correlation in-between to fulfill certain services, potentially occur in a certain degree of simultaneity or even in recurring patterns (e.g., a failure of an energy supply device will cause corresponding timely events of those devices being connected to that energy supply).
Event-Driven Ontology Population
411
Fig. 1. Hybrid ontology population approach
Calculating the correlation (we adhere on similarity calculations based on “functional connectivity” of [24]) between the OT devices results in a n-times-n sized device-correlation-matrix (cf. Figure 1), where n is the number of OT devices, showing the correlation between all possible devices. At this point, the inductive part of our approach gets in touch with the deductive part in terms of a plausibility-check, i.e., restricting the correlation calculations between devices where their corresponding types have a relationship modeled in the T-box, only. The device-correlation-matrix is continuously updated (considering a stream of
412
D. Graf et al.
messages) since new messages potentially provide new information and might change the correlation-values (e.g., the correlation-value between two OT devices increases if their corresponding message behavior show similar patterns, for instance, “cam_1” always sends a message before “srv_a”). Link Instantiation (2). During link instantiation phase the device-correlationmatrix is used as a basis to instantiate links between two OT objects in the A-Box. A link is instantiated and annotated with a probability, only, if the correlation value is above a certain threshold since otherwise links are too unstable (links are instantiated and deleted frequently when correlation values slightly changes). Additionally, the link must be consistent with the corresponding relationship modeled in the T-Box (e.g., given multiplicities). Since our approach is designed for a stream of messages, already existing links in the A-Box have to be recurringly rechecked and possibly updated (or even deleted) based on the correlation values (cf. center of Fig. 1). Thereby, link instantiation works in a loop-wise way, i.e., focusing the OT object, each relationship in the T-Box, or link in the A-Box respectively, is processed (instantiated, updated, or deleted) one after another (cf. examples visualized in Fig. 1). Object Instantiation (3). While OT devices occurring in the message log can be instantiated straight forwardly to OT objects, existing but not yet populated OT devices (e.g. due to having not (yet) sent a message - aka “silent objects” and consequently “silent links”) can be inferred through explicit knowledge provided by domain experts. The main idea is to consider T-Box information in a way that mandatory objects and their links are instantiated in the OT ontology’s A-Box although not included in the message logs (e.g., based on T-Box information, a camera device is mandatory connected to a media server and therefore we are able to instantiate the “silent” media server object as well as the “silent link” to the camera object at the moment when the camera object is instantiated - cf. examples of Fig. 1). This method gives a series of benefits. Firstly, for monitoring purposes it enables to bridge „blind spots “ in the monitored environment being the consequence of lacking integration of systems or even of unmonitored OT areas. Secondly, it enables identifying objects being affected by a failure or being the potential cause of a failure although those objects do not exist in the log (so far). Thirdly, in a semantically richer A-Box, object and link meta-data of silent objects can be used for reasoning purposes, also. Fourthly, instantiating silent objects and silent links works towards completing the A-Box. On the downside, this however comes at the costs that new objects derived from the event-information of the message-stream have to be merged with existing silent objects in the A-Box if the silent object was already instantiated beforehand. At this point the device-correlation-matrix again is an indication whether a new object is “the same” as a silent object and as a consequence have to be merged. Secondly, by considering T-Box relationship information to derive links, multiplicities have strong impact (e.g., T-Box relationships of optional multiplicity (0..1 or 0..*) can not be considered to derive “silent objects and silent links”). At this point we have to emphasize that the approach discussed so far is still work-in-progress meaning that we have some parts already implemented as
Event-Driven Ontology Population
413
a prototype in a CIS domain adapted setting such as the correlation-analysis phase and the initial population of objects, whereas implementation of parts of the link instantiation phase is still ongoing. Nevertheless, first experimentations with real-world data containing 822k events (status- and service messages) from the year 2019 from a certain part of a highway network show promising results.
4
Conclusion and Future Work
The ontology population approach presented in this paper is grounded on a systematic literature review of related research areas dealing with similar requirements (especially regarding source and target data structure), which are (i) semantic annotations, (ii) event log focused work in the IoT, (iii) organizational process mining, and (iv) complex event processing. Based thereupon, we elaborate towards a hybrid (semi)-automatic ontology population approach in the context of OTM by combining inductive and deductive methods. Since the presented approach is work-in-progress, beside ongoing work at the core phases such as experimenting with configurations and parameters (e.g., correlation calculations or threshold values) as well as different techniques for A-Box updating strategies (e.g., interval-based methods), future work includes dealing with algorithm performance and computational complexity since the large amount of objects as well as the (dependent on the OT-type) high message frequency lead to high requirements on computation in real-world settings.
References 1. Alevizos, E., et al.: Probabilistic complex event recognition: a survey. ACM Comput. Surv. (CSUR) 50(5), 1–31 (2017) 2. Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983) 3. Amato, F., et al.: Detect and correlate information system events through verbose logging messages analysis. Computing 101(7), 819–830 (2019) 4. Belkaroui, R. et al.: Towards events ontology based on data sensors network for viticulture domain. In: Proceedings of the International Conference on the IoT, pp. 1–7. ACM (2018) 5. Detro, S. et al.: Enhancing semantic interoperability in healthcare using semantic process mining. In: Proceedings of the International Conference on Information Society and Technology, pp. 80–85 (2016) 6. Ellinas, G. et al.: Critical infrastructure systems: basic principles of monitoring, control, and security. In: Kyriakides E., Polycarpou M. (eds.) Intelligent Monitoring, Control, and Security of CIS, pp. 1–30. Springer, Berlin (2015) 7. Endler, M. et al.: Towards stream-based reasoning and machine learning for IoT applications. In: Intelligent System Conference, pp. 202–209. IEEE (2017) 8. Ganino, G., et al.: Ontology population for open-source intelligence: a GATE-based solution. Softw. Pract. Experience 48(12), 2302–2330 (2018) 9. Graf, D., Schwinger, W., Kapsammer, E., Retschitzegger, W., Baumgartner, N.: Cutting a Path Through the IoT Ontology Jungle - a Meta Survey. In: International Conference on Internet of Things and Intelligence Systems. IEEE (2019)
414
D. Graf et al.
10. Graf, D., Schwinger, W., Kapsammer, E., Retschitzegger, W., Pröll, B., Baumgartner, N.: Towards Operational Technology Monitoring in ITS. In: International Conference on Management of Digital Eco-Systems. ACM (2019) 11. Graf, D., Schwinger, W., Kapsammer, E., Retschitzegger, W., Pröll, B., Baumgartner, N.: Towards message-driven ontology population - facing challenges in real-world IoT. In: World Conference on Information Systems and Technologies, pp. 361–368. Springer, Cham (2020) 12. Hromic, H., et al.: Real time analysis of sensor data for the IoT by means of clustering and event processing. In: Proceedings of International Conference on Communications, pp. 685–691. IEEE (2015) 13. Jafari, M., et al.: Role mining in access history logs. J. Comput. Inf. Syst. Ind. Manage. Appl. 1, 258–265 (2009) 14. Jayawardana, V., et al.: Semi-supervised instance population of an ontology using word vector embeddings. In: Proceedings of International Conference on Advances in ICT for Emerging Regions, pp. 217–223. IEEE (2017) 15. Jin, T., et al.: Organizational Modeling from Event Logs. In: Proceedings of International Conference on Grid and Cooperative Computing, pp. 670–675. IEEE (2007) 16. J. Kim and J. Lee: OpenIoT: an open service framework for the internet of things. In: Proceedings of World Forum on IoT (WF-IoT), pp. 89–93 (2014) 17. Körber, M., et al.: TPStream: low-latency and high-throughput temporal pattern matching on event streams. Distrib. Parallel Databases 37, 1–52 (2019) 18. Lin, S., et al.: Dynamic data driven-based automatic clustering and semantic annotation for IoT sensor data. Sens. Mater. 31(6), 1789–1801 (2019) 19. Liu, F., et al.: Device-oriented automatic semantic annotation in IoT. J. Sensors 2017, 9589,064:1–9589,064:14 (2017) 20. Lubani, M., et al.: Ontology population: approaches and design aspects. J. Inf. Sci. 45(4), 502–515 (2019) 21. Maier, M.W.: Architecting principles for systems-of-systems. J. Int. Council Syst. Eng. 1(4), 267–284 (1998) 22. Matzner, M., Scholta, H.: process mining approaches to detect organizational properties in CPS. In: European Conference on Information Systems (2014) 23. Mehdiyev, N., et al.: Determination of rule patterns in CEP using ML techniques. Proc. Comput. Sci. 61, 395–401 (2015) 24. Messager, Antoine, et al.: Inferring functional connectivity from time-series of events in large scale network deployments. Trans. Netw. Serv. Manage. 16(3), 857–870 (2019) 25. Murray, G., et al.: The convergence of IT and OT in critical infrastructure. In: Proceedings Australian Information Security Management Conference, pp. 149– 155 (2017) 26. Ni, Z., et al.: Mining organizational structure from workflow logs. In: Proceedings of International Conference on e-Education, Entertainment a. e-Management, pp. 222–225. IEEE (2011) 27. Noura, M., et al.: Interoperability in IoT infrastructure: classification, challenges, and future work. In: International Conference on IoT as a Service, pp. 11–18. Springer, Cham (2017) 28. Reyes-Ortiz, J., et al.: Web services ontology population through text classification. In: Proceedings of Conference on Computer Science and Information Systems, pp. 491–495. IEEE (2016)
Event-Driven Ontology Population
415
29. Teymourian, K., et al.: Fusion of background knowledge and streams of events. In: Proceedings of International Conference on Distributed Event-Based Systems, pp. 302–313. ACM (2012) 30. Zhu, M., et al.: Service hyperlink: Modeling and reusing partial process knowledge by mining event dependencies among sensor data services. In: Proceedings of International Conference on Web Services, pp. 902–905. IEEE (2017) 31. Zhuge, C., Vaarandi, R.: Efficient event log mining with LogClusterC. In: Proceedings of International Conference on Big Data Security on Cloud, pp. 261–266. IEEE (2017)
Semantic-Based Image Retrieval Using Balanced Clustering Tree Nguyen Thi Uyen Nhi1,3 , Thanh The Van2(B) , and Thanh Manh Le1 1 Faculty of Information Technology, University of Science - Hue University, Hue, Vietnam
[email protected], [email protected] 2 Office of Scientific Research Management and Postgraduate Affairs, HCMC,
University of Food Industry, Hue, Vietnam [email protected] 3 Faculty of Statistics and Informatics, University of Economics, the University of Danang, Hue, Vietnam
Abstract. In this paper, we propose a model for semantic-based image retrieval (SBIR) on the clustering balanced tree, C-Tree, and ontology to analyze the semantics of an image and extract a similar set of images, in which the input is a query image. This structure is constructed rely on separating the nodes from the leaf node and growing towards the root to create a balanced tree. A set of similar images are searched on the C-Tree to classify the query image based on the k-NN (k-Nearest Neighbor) algorithm. Then, the SPARQL query is generated to query the semantics of the image on ontology. We experimented with image datasets such as COREL (1000 images), Wang (10,800 images), ImageCLEF (20,000 images). The results are compared and evaluated with the relevant projects published recently on the same datasets. Keywords: Clustering · C-Tree · SBIR · Ontology · SPARQL
1 Introduction According to IDC (International Data Corporation), in 2016 the world created 1,138 trillion images (700 times more than 2015). In 2017, digital data reached 17 trillion gigabytes and it is predicted that it will reach 175 trillion gigabytes by 2025 [3]. The problem of data mining, search and semantic analysis of images is becoming more common and is a research topic that is interested in many people. Therefore, it’s essential to gain the development of highly accurate image retrieval system using a structure for mapping low-level features to high-level semantics on ontology [5, 15]. We solve two problems: (1) search for a set of similar images by low-level features with input image; (2) convert low-level content into high-level semantics of the image based on SPARQL query and ontology. So, we propose C-Tree structure which can store large data on external memory and make query times faster. The leaf nodes contain the indexes of the image; the internal node contains only the centers of child nodes and the paths to its children. At the same time, C-Tree is grown in a balanced way from © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 416–427, 2021. https://doi.org/10.1007/978-3-030-72651-5_40
Semantic-Based Image Retrieval Using Balanced Clustering Tree
417
leaf to root. The query result on the C-Tree finds a set of images that are similar to the content with the query image. To analyze the semantics of the image, the k-NN algorithm [10] is proposed to classify the image as a basis for extracting high-level semantics for that image. The SPARQL query is generated based on the classifications and query on ontology. The result of this query is the set of descriptive semantics of the image and the set of similar images according to the semantic classification of the input image.
2 Related Works Many types of research focusing on the construction of multidimensional index have been developed such as B-tree [6], R-trees [4], KD-tree [11], S-tree [12], etc. In the indexing method, an image is represented by features vector to find the similarity between the query image and the image in the database [7]. KD-tree and R-tree are examples of the method of searching for images in space [19]. R*-Tree [2] has a better performance than R-Tree. This shows that researches related to tree structure image querying are paid much attention in recent years. Y. Liu, D. Zang, and G. Lu [9] proposed an image retrieval method based on highlevel semantics of each region on the image. During the retrieval, a set of images whose semantic concept matches the query is returned. The system was tested on the COREL image dataset. V. Vijayarajan et al. [23] performed an image retrieval based on natural language analysis to create a SPARQL query for the purpose of searching for an image set based on the triplet language RDF. M. Jiu, et al. [8] used deep multi-layer networks based on nonlinear activation functions for image annotation. SVM technique is applied to classify images at the output layer in order to extract a semantic level according to visual information for similar images based on the BoW (Bag-of-Words). This method is evaluated the experimental results on the ImageCLEF image data set. Van T.T. and Le MT. [12] proposed a binary signature clustering method of an image to create a clustering graph structure based on an S-Tree for constructing content-based image retrieval. They also evaluated the experiment the datasets including COREL, CBIR, Wang, MSRDI, and ImageCLEF. Wang, X., & Wilkes, D.M. [14] proposed the method of fast image retrieval based on the Quantization Tree and presented an effective way to index local image regions. Quantization Tree has a hierarchical organization of labeled descriptions from the image color chart. Quantization Tree combines the visual semantic content of an image into content-based image retrieval algorithms for semantic querying of images. The above researches mainly focus on tree data structures for efficient storage of image indexes that are low-level vector extracted from large image datasets for contentbased image retrieval but lack of semantic analysis of the image. This limitation makes the image query system not to meet the needs of the user, creating a “semantic gap” between content-based queries and high-level semantics. Therefore, many studies on semantic-based image retrieval on ontology are paid attention to.
418
N. T. U. Nhi et al.
3 Semantic-Based Image Retrieval on C-Tree 3.1 Architecture of SBIR_CT We describe the architecture of the semantic image retrieval system on C-Tree named SBIR_CT and its components. Figure 1 describes the architecture of a semantic image query system on C-Tree with two specific stages as follows: (1) Preprocessing phase: Each image from the dataset segmented and extracted features vector; create a balanced clustering tree structure named C-Tree from the training data samples; Build the ontology from the image dataset and WWW. (2) Image query phase: Execute a query in a C-Tree to obtain a set of similar images and an arrangement of similar images according to the measure; Classify image from a set of similar image based on the k-NN algorithm; Query the semantic image from SPARQL based on ontology.
Fig. 1. Model of semantic-based image retrieval on C-Tree
3.2 Extracting the Feature Vector We perform extraction of low-level features, which is described in Fig. 2, including MPEG7 color feature, shape feature, texture feature, detecting the Laplacian of Gaussian object, object recognition based on boundary and surface with Sobel filter, enhancing pixel intensity with Gaussian filtering.
Semantic-Based Image Retrieval Using Balanced Clustering Tree
419
Fig. 2. An example of the low-level feature extraction of the image
3.3 Structure of C-Tree In Fig. 3, the C-Tree consists of a root, a set of nodes I and a set of leaves L. Nodes are connected to each other through the path of the parent-child relationship. The element ED is stored in the leaf, and the center EC is stored in the node. The element ED = index, f , ID, file, cla contains index is the ED position on the file, f is the vector feature of an image, the image identifier ID, the image annotation file, and the cla are classes of the image. The center EC = index, fc , isNextLeaf , path contains index is the EC position on the file, fc is the center vector of the feature vectors f at a child node that has a path linking the path to the EC, and is Next Leaf is the value that checks whether the next subset is a leaf or not. Therefore, the C-Tree includes leaves L contain data elements ED: L = {ED1 , ED2 , . . . , EDn }, 1 ≤ n ≤ M , in which M is the maximum number of elements in a leaf; the nodes I are nodes with at least one child node, containing center elements EC: I = {EC1 , EC2 , . . . , ECm },1 ≤ m ≤ N , in which N is the maximum number of elements in the node; each element internal node-links to its adjacent child via its path; each leaf has the same height. The rules of building C-Tree are described as follows: a) When the root is empty, the root is a leaf containing at most M elements ED. b) Data elements are added to the tree according to the branch selection rule with the closest measure until a suitable leaf is found to add; Fig. 4 describes how to add a new element to the cluster. c) If a node in a C-Tree is full (the number of elements of node, leaf exceeds the threshold M , N ), the splitting based on K-means algorithm is performed; Fig. 5 shows an example of splitting a leaf on C-Tree. 3.4 Building the Ontology for Image Datasets Figure 6 is the proposed model consists of the steps: adding image annotations and semantic descriptions for images; extracting the resource identifiers URIs, descriptions of image taken from WWW. Figure 7 is an example of a semantic description for the class from the thesaurus. Figure 8 is an example of a ontology built on Protégé.
420
N. T. U. Nhi et al.
Fig. 3. Structure balanced clustering tree C-Tree
Fig. 4. Adding an element to the cluster on C-Tree
4 Search Algorithms on SBIR_CT System 4.1 Content-Based Image Retrieval on C-Tree We extract the feature vector of the query image and find a branch with closest similar measure. Image retrieval algorithm on the C-Tree is done as follows:
Semantic-Based Image Retrieval Using Balanced Clustering Tree
Fig. 5. Splitting a node on C-Tree
Fig. 6. A model for building the ontology
Fig. 7. An example of a semantic description for FLOWER class
421
422
N. T. U. Nhi et al.
Fig. 8. An example of a visual ontology built on Protégé
Fig. 9. A search results for CBIR of SBIR_CT system
4.2 Query Image Classification An input image is queried on the C-Tree to find a set of similar images by content. From there, the k-NN algorithm is implemented to classify the query image.
Semantic-Based Image Retrieval Using Balanced Clustering Tree
423
4.3 Image Semantic Retrieval and Analysis Relying on the visual word vector W generated from the CIKN algorithm, the SPARQL query is generated to execute the query on the ontology.
5 Experiment 5.1 Application We used datasets in the experiment including COREL, WANG, ImageCLEF. The result of this process is a sets of similar images and semantic descriptions as meta-data, URI, concept. Figure 9 shows a result of the SBIR_CT query system. 5.2 Experimental Results To evaluate image search efficiency, we use evaluation factors including precision, recall in Fig. 10, Fig. 11, Fig. 12. The average search time (milliseconds) and performance values on the datasets are summarized in Table 1, with 19.914 ms (COREL), 39.747 ms (Wang), 44.089 ms (ImageCLEF). Table 2 summarizes the results of classification accuracy and time on the C-Tree of COREL, Wang, ImageCLEF. We compared the obtained performance with other studies on the same set of image data. Table 3, Table 4, Table 5 show the results of the comparisons between our proposed method and the methods of the studies on the COREL, Wang, mageCLEF.
424
N. T. U. Nhi et al.
Fig. 10. Image retrieval performance on C-Tree of COREL dataset
Fig. 11. Image retrieval performance on C-Tree of WANG dataset
Fig. 12. Image retrieval performance on C-Tree of ImageCLEF dataset
Semantic-Based Image Retrieval Using Balanced Clustering Tree Table 1. Performance of SBIR_CT on image datasets Performances
COREL
WANG
ImageCLEF
Avg. precision
0.677655
0.607222
0.606217
Avg. recall
0.699885
0.489184
0.409401
Avg. F-measure
0.688521
0.545011
0.474718
Avg. query time (ms) 19.91437
39.74690
44.08912
Table 2. Image classification performance of SBIR_CT system on image datasets Dataset
Precision
Time (ms)
Corel
0.73017535 12.18645
Wang
0.68935259 14.23075
ImageCLEF 0.69135772 15.96235
Table 3. Comparison between methods on the COREL dataset Methods
Mean average precision (MAP)
A. Huneiti, 2015 [17]
0.559
Garg, M., 2019 [18]
0.602
Bella M. I. T., 2019 [19] 0.609 Our method
0.6777
Table 4. Comparison between methods on the WANG dataset Methods
Mean average precision (MAP)
Dos Santos, 2015 [20] 0.570 R. Das, 2018 [21]
0.559
P. Chhabra, 2018 [22] 0.577 Our method
0.6072
Table 5. Comparison between methods on the ImageCLEF dataset Methods
Mean average precision (MAP)
V. Vijayarajan, 2016 [13]
0.4618
H. Cevikalp, 2017 [16]
0.4678
Z. Seymour, 2018 [23]
0.420
Our method
0.6062
425
426
N. T. U. Nhi et al.
6 Conclusion In this paper, we have built a balanced clustering tree structure C-Tree and proposed creation and querying algorithm on C-Tree to store image data. A k-NN algorithm is used to classify images based on a set of similar images. Besides, we have built the ontology for the image sets to store the descriptive data for image sets based on the RDF trilogy language. The SPARQL queries are generated to semantic query on the ontology for the input image. Since then, a semantic image query system SBIR_CT is built on the basis of C-Tree and ontology. The average query accuracy per set of COREL, Wang, ImageCLEF are 0.6777, 0.6072, 0.6062, showing that our proposed query system is feasible for single-object and multi-object image datasets. In the future, we will build a semi-automatic ontology for image sets from WWW to enhance the efficiency of semantic-based image retrieval. Acknowledgment. The authors would like to thank the Faculty of Information Technology, University of Science - Hue University for their professional advice for this study. This work has been sponsored and funded by Ho Chi Minh City University of Food Industry under Contract No. 147/HD-DCT.
References 1. Bella, M.I.T., Vasuki, A.: An efficient image retrieval framework using fused information feature. Comput. Electr. Eng. 75, 46–60 (2019) 2. Beckmann, N. et al.: The R*-tree: an efficient and robust access method for points and rectangles. In: ACM SIGMOD Conference, pp. 322–331 (1990) 3. Reinsel, D., Gantz, J., Rydning, J.: The Digitization of the World: From Edge to Core sponsored by Seagate, IDC Technical Report (2018) 4. Elmarsson, A., A comparison of different R-tree construction techniques for range queries on neuromorphological data, School of Elec. Engi. and Comp. Sci. (2020) 5. Gombos, G., Kiss, A.: P-Spar (k) ql: SPARQL evaluation method on Spark GraphX with parallel query plan. In: IEEE 5th International Conference, pp. 212–219 (2017) 6. Graefe, G., et al.: Modern B-tree techniques. In: IEEE International Conference, pp. 1370– 1373 (2011) 7. Jiang, W., Er, G., Dai, Q., Gu, J.: Similarity-based online feature selection in content-based image retrieval. IEEE Trans. Image Process. 15(3), 702–712 (2006) 8. Jiu, M., Sahbi, H.: Nonlinear deep kernel learning for image annotation. IEEE Trans. Image Process. 26(4), 1820–1832 (2017) 9. Liu, Y., Zhang, D., Lu, G.: Region-based image retrieval with high-level semantics using decision tree learning. Pattern Recogn. 41(8), 2554–2570 (2008) 10. Ma, Y., Xie, Q., Liu, Y., Xiong, S.: A weighted KNN-based automatic image annotation method. Neural Comput. Appl. 1–12 (2019) 11. Procopiuc, O et al.: Bkd-tree: a dynamic scalable kd-tree. In: International Symposium on Spatial and Temporal Databases, Springer, pp. 46–65 (2003) 12. Van, T.T., Le, T.M.: Contentbased image retrieval based on binary signatures cluster graph. Expert Syst. 35(1), e12220 (2018) 13. Vijayarajan, V., et al.: A generic framework for ontology-based information retrieval and image retrieval in web data. Hum.-centric Comp. Infor. Sci. 6(1), 18 (2016)
Semantic-Based Image Retrieval Using Balanced Clustering Tree
427
14. Wang, X., et al.: A fast image retrieval method based on a quantization tree. In: Machine Learning-based Natural Scene Recognition, Springer, Berlin, pp. 195–214 (2020) 15. Zhong, B., et al.: Ontology-based semantic modeling of knowledge in construction: classification and identification of hazards implied in images. J. Constr. Eng. Manage. 146(4), 04020013 (2020) 16. Cevikalp, H., Elmas, M., Ozkan, S.: Large-scale image retrieval using transductive support vector machines. Comp. Vision Image Underst. 173, 2–12 (2018) 17. Huneiti, A., Daoud, M.: Content-based image retrieval using SOM and DWT. J. Softw. Eng. Appl. 8(02), 51 (2015) 18. Garg, M., Singh, H., et al.: Fuzzy-NN approach with statistical features for description and classification of efficient image retrieval. Mod. Phys. Lett. 34(03), 1950022 (2019) 19. Bella, M.I.T., Vasuki, A.: An efficient image retrieval framework using fused information feature. Comput. Electr. Eng. 75, 46–60 (2019) 20. dos, Santos, J.M. et al.: A signature-based bag of visual words method for image indexing and search. Pattern Recogn. Lett. 65, 1–7 (2015) 21. Das, R., Thepade, S. et al., Novel feature extraction technique for content-based image recognition with query classification. Int. J. Comp. Vis. Robot. 7(1–2), 123-147 (2017) 22. Chhabra, P., Garg, N.K., Kumar, M.: Content-based image retrieval system using ORB and SIFT features. Neural Comput. Appl. 32(7), 2725–2733 (2020) 23. Seymour, Z., et al., Image annotation retrieval with text-domain label denoising. In: ACM on International Conference on Multimedia Retrieval, pp. 240–248 (2018)
NoSQL Comparative Performance Study Pedro Martins1(B) , Paulo Tom´e1 , Cristina Wanzeller1 , Filipe S´ a1 , and Maryam Abbasi2 1
CISeD - Research Centre in Digital Services, Polytechnic of Viseu, Viseu, Portugal {pedromom,ptome,cwanzeller,filipe.sa}@estgv.ipv.pt 2 CISUC - Centre for Informatics and Systems of the University of Coimbra, Coimbra, Portugal [email protected]
Abstract. This article aims to introduce the concept of NoSQL, describe the database systems Cassandra and MongoDB, but above all, perform a comparative study (practical) of both systems, on the same dataset. The YCSB benchmark tool is to test different types of operations - reading, writing, and reading and writing - by performing these tests on various workloads, through progressive increases of clients to perform the operations, in order to compare the two solutions in terms of performance. Preliminary results show that Cassandra generally performs better in all types of operations, but it depends more on the capacity in terms of resources of the machine. On the other hand, MongoDB is able to achieve better results in a scenario where hardware resources are lower, with performance levels slightly higher than Cassandra. Keywords: NoSQL Vertical scalability
1
· Cassandra · MongoDB · YCSB · Performance ·
Introduction
The traditional Database (DB) Systems, better known as Relational Data Base Management Systems (RDBMS) have been based for several decades on the relational model, being known as SQL (Structured Query Language) databases and their variants. Over time, other alternatives to these traditional systems have emerged, and in recent years databases based on non-relational models, better known as NoSQL (Not only SQL), have been increasingly used and popular [8,9]. These systems are distinguished by the fact that there is no relationship between entities, and most of them are based on the storage of simple key-value pairs. With the increase in the use of the Internet, and the need for high data availability and scalability, distributed systems capable of handling huge volumes of data have become necessary [8,9]. To achieve and match all these parameters, NoSQL databases have become the preference for Big Data, as they allow easier handling of large volumes of data [8]. In terms of NoSQL systems, MongoDB and Apache Cassandra stand out, which will be compared throughout this article c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 428–438, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_41
NoSQL Comparative Performance Study
429
due to some differences between them. These differences are mainly because of the fact that Cassandra stands out for being based on a type of non-relational model oriented to a family of columns (attributes of the table), maintaining similarities with a relational DB in terms of rows and columns [3]. On the other hand, MongoDB uses a non-relational model oriented to documents in the form of JSON, in its BSON (Binary JSON) aspect, stored in collections [3]. The study and analysis performed in this article are focused on the comparison between Cassandra and MongoDB in terms of their performance, and this comparison will be performed through tests performed using the YCSB, in both systems in a single cluster node (single machine), and these benchmarks will be performed at 3 different types of operations (writing, reading, and writing and reading simultaneously), and at each new test there will be an increase in workload (more clients performing operations).
2
Related Work
The NoSQL databases emerged from the difficulties detected throughout the evolution of the Internet, more specifically in the Web 2.0 phase, due to the huge volume and variety of data, and the demand for workload related to the increase in users [3]. BigTable by Google and DynamoDB from Amazon, which allowed the use of horizontal scaling schemes to increase the performance of the databases through the increase of machines operating in the form of a cluster [3]. Since then, there has been a constant evolution, with more than 225 NoSQL solutions as of 2017 [3]. NoSQL databases are characterized by the fact that their data models are nonrelational and independent of a schema, making databases more flexible [3]. Cassandra is currently the 2nd most used NoSQL database technology, with companies like Apple and Netflix making use of this technology [3]. This technology is written in the Java language, and was initially developed by Facebook for research and discontinued in 2008, and in 2009 it became an Apache project until today, always remaining open-source [3]. As previously mentioned, Cassandra databases fit into the NoSQL columnfamily oriented data model. This means, in more detail, that a Cassandra database is composed of a keyspace, family of columns, rows and columns [1,3]. The keyspace represents the entire dataset, and more abstractly it is the entire database [3]. The column-family is equivalent to a table in the relational model, and is composed of rows (records), as can be seen in Fig. 1, where the family of columns (table) for fruits (“Fruits”) contains two records (1 for each type of fruit), each type of fruit containing three and two columns, respectively, with different types of data [3]. Each record has a unique identifier (row key) which in the case of the Fig. 1 they are represented by “Apple” and “Banana” and can also be represented by an incremental integer as in a relational database [3]. Each column contains a key-value pair along with a timestamp (e.g. ’123456789’), in order to ensure consistency in the case of multiple copies of the same data [3]. The language used to perform queries in Cassandra is called CQL, having a syntax quite similar to SQL, in terms of the terminology and structure of the queries [3].
430
P. Martins et al.
Fig. 1. Example of a family of columns (table) in Cassandra [3]
Today, MongoDB is the industry’s leading NoSQL database technology. This system is written in the C ++ language and developed by the company 10gen, being more recently known as MongoDB, Inc., and it was created back in 2007 to deal with scalability problems [3]. Like Cassandra, it is a free and open-source technology [3]. A database in MongoDB is represented by a set of collections, with each collection aggregating documents, as shown by Fig. 2. Usually the documents that are part of the same collection contain the same structure, but it is possible that each document has a different structure from the others [3]. In terms of the language to perform queries, MongoDB can be accessed through the mongo shell with commands completely based on JavaScript [3].
Fig. 2. Example of a document in MongoDB [3]
YCSB - Yahoo Cloud Serving Benchmark is an open-source tool, one of the most used for benchmarking and workload generator, in order to test NoSQL databases, with adapters for different database systems such as Cassandra, MongoDB, HBase, Couchbase, among others [4]. The YCSB client consists of 4 modules: the executor (generator) of the workloads, the client threads module, the DB interface module and the statistics module [4]. After generating the data to be loaded into the database in question, by the YCSB client, the workload executor will launch the client threads to perform a set of operations on the DB using the DB interface module. Finally, the statistics module will return statistics for each operation for analysis [4]. These statistics range from the measurement of the total execution time, throughput (operations per second), to the latency of writing and reading.
NoSQL Comparative Performance Study
3
431
Experimental Setup
In order to carry out the comparative study, this chapter presents the experimental setup used to obtain all the results necessary for the evaluation of the two database systems (Cassandra and MongoDB). The experimental setup configurations in terms of hardware and software, mainly YCSB configurations, were based on [1,2,5–7,11]. 3.1
Hardware and Software Specifications
All experiments were carried out with different test in a machine with the Windows 10 operating system (64-bit), PC with Windows 10 Pro (64-bit), 16 GB of RAM at 3200MHz, 512 GB of SSD and AMD Ryzen 5 3600 3.6 GHz CPU (6 Cores). After, Apache Casssandra 3.11.6, MongoDB 4.2 and MongoDB Compass were installed. For the correct functioning of these 2 DB systems, version 1.8.0 251 of JAVA and the respective system variable (‘JAVA OPTIONS’) were configured, so that both Cassandra and MongoDB used a minimum and maximum value (even value) of RAM memory, in order to guarantee the same conditions in terms of memory usage by the clients of these 2 databases. Then, version 0.17.0 of YCSB was installed to carry out the tests, and also Python 27 to execute all commands (in the command line interface). 3.2
Dataset
The dataset used in the databases is generated by the YCSB data generator, which is part of the YCSB client, and this dataset has a size of about 10.9GB. This dataset is exactly the same for both DB, being composed of 11 attributes. Each attribute is 100 bytes in size and is filled with a random string (binary in the case of MongoDB and VARCHAR in the case of Cassandra), making a total of 1.1 KB per record. The table consists of the primary key ‘y id’ and nine more attributes designated from ‘field0’ to ‘field9’, which corresponds to 10 million records in each of the databases: – Cassandra: Dataset 10.9 GB loaded into the keyspace ‘usertable’ database ‘ycsb’; – MongoDB: 10.9 GB dataset loaded for the ‘usertable’ collection that is part of the ‘ycsb’ database. 3.3
Workloads
The workloads used in this study were adaptations to the YCSB workloads, and these workloads are called: Workload 1 (100% reading), Workload 2 (100% writing) and Workload 3 (50% reading and 50% writing). It should be noted that all workloads were configured, through their files, so that operations could be carried out, both for writing and for reading, to all the attributes of each record.
432
4
P. Martins et al.
Experimental Results
This chapter will perform an analysis of all values obtained in the experimental tests performed, in order to analyze the performance. The analysis and interpretation of the results obtained, regarding all the benchmarks executed, were based on [4,10]. When executing the benchmark commands, in shell, related to the workloads in question, the following values will be used: 10 million records to be used in the execution of operations (‘recordcount’ attribute); 200,000 operations (‘operationcount’ attribute) to be executed in each test, this being a fixed value in all tests, making up a high number of operations. In each of the DB’s and workloads, the number of clients will be increased progressively (1 to 4, 4 to 8, and so on) in order to understand the workload that the DB system can support, that is, up to how many clients, simultaneously carrying out operations - until reaching the defined number of 200,000 - its performance remains stable or with increases in throughput until it reaches very high latency values and throughput values start to decrease. All tests (benchmarks) performed, with the exception of data loading, were performed 3 times in order to guarantee an average of the values obtained and therefore greater accuracy. These tests are focused on the graphic and written analysis of the following points, with greater emphasis on the last two points: – In each DB, understand whether it is the writing, reading or 50% writing-50% reading workload that has the best performance; – Make a qualitative comparison between the two databases, in order to understand which one has better performance in each of the workloads. 4.1
Data Loading
As can be seen from the Fig. 3, the loading of the 10 million records (10.9 GB) performed better in MongoDB, as it had an average of 31,414 operations per second (insertions per second), and in the case of Cassandra, lower throughput values were obtained. This demonstrates that in this data loading benchmark, there was a better performance of MongoDB. It should be noted that this insertion of records was made with a fixed total of 64 clients making the insertions simultaneously, hence high throughput values (operations/sec) were obtained. 4.2
Cassandra
Workload 1, 100% of read operations, in which it was intended to obtain throughput and average latency values for reading operations tests over 200,000 records (all attributes of each record), as shown in Fig. 4, as the number of clients (threads) to perform operations at the same time is increased, the latency also increases steadily up to impractical values from 64 clients, with the values of operations per second having little variation. This demonstrates that Cassandra is not able to deal with a high increase in workload (more clients) in a positive
NoSQL Comparative Performance Study
433
Fig. 3. Data loading - Throughput
way, since from 32 clients (inclusive) the performance starts to decrease and the latency values become very high. It is concluded that the best performance (peak throughput and good latency), is obtained with 4 clients.
Fig. 4. Workload 1
In relation to Workload 2, 100% of write operations, where lower performance is expected, with the increase in the number of clients (threads), latency also increases steadily and linearly to very high values from 256 clients (inclusive), being that the values of operations per second also start to decrease with this number of clients. This demonstrates that Cassandra is able to deal with an increase in workload (more clients) in a positive way through an excellent performance up to high client values. In comparison with Workload 1, it turns out that Cassandra is much more optimized to perform writing operations than reading. Thus, it is concluded that the best performance is obtained with 128 clients (peak throughput), taking into account the values analyzed and presented in the graph of the Fig. 5. Regarding Workload 3, 50% of read operations and 50% of write operations, as seen in the Fig. 6, with the increase in the number of clients (threads), the latency also increases steadily and linearly up to very high values from 512 clients
434
P. Martins et al.
Fig. 5. Workload 2
(inclusive). In terms of ops/sec they are also starting to decline at 256 clients and more dramatically than in previous workloads. This shows that Cassandra is able to deal with an increase in workload in a positive way through better performance, up to 16 clients simultaneously, as in Workload 1. Considering the average latencies presented, it appears that the latency obtained in reading operations is much higher than the latency in writing (up to 7x more), and consequently also the throughput in reading is lower.
Fig. 6. Workload 3
Conclusions. In order to finish the analysis of all tests performed on Cassandra, the following three points are concluded: – Better performance in Workload 2, only with write operations, with throughput values much higher than the other two workloads; – In Workload 3, where writing and reading operations are carried out simultaneously, Cassandra is very similar to Workload 1 in terms of performance. 4.3
MongoDB
Regarding Workload 1, 100% of read operations, as shown in the graph of the Fig. 7, MongoDB starts operations (1 client) with good performance, with both
NoSQL Comparative Performance Study
435
throughput and latency increasing linearly (the values double) up to 4 clients simultaneously, but from 8 clients (including) the increase in throughput is lower while average latency reaches very high and impractical values at 128 clients. This demonstrates that MongoDB handles the workload increase in a positive way up to 64 clients, with the best performance occurring at this point, and from this point on the performance starts to decrease.
Fig. 7. Workload 1 - MongoDB
In relation to Workload 2,100% of write operations, as shown in the graph of the Fig. 8, MongoDB starts with good throughput and latency values, and from 4 clients there are very unstable variations in performance increase and decrease. These aspects demonstrate some instability on the part of this DBMS in dealing with variations in the workload in writing operations, but, despite this, there are very low values of average latency up to 1024 clients. We can conclude that the point with the best performance is verified with 128 clients, given the peak throughput and the associated low latency, and the performance in terms of workload is acceptable up to 128.
Fig. 8. Workload 2 - MongoDB
436
P. Martins et al.
Workload 3, 50% of read operations and 50% of write operations, good throughput values are verified across the entire line (Fig. 9), with the decline occurring with 64 clients, but slightly. On the other hand, the observed latency values are very high for reading operations, which demonstrates that MongoDB is not so optimized to deal with a lot of workload when it comes to reading operations. Regarding the best performance, it is verified at the point where there are 16 clients to carry out the operations simultaneously, and the ability to handle workloads are found to be acceptable up to 32 clients, since thereafter latency is very high in reading operations.
Fig. 9. Workload 3 - MongoDB
Conclusions. In order to complete the analysis of all tests performed on MongoDB, the following four points are concluded: – Better performance in Workload 1 if we only look at throughput values, with values slightly higher than the other 2 workloads, despite higher latency values; – Much higher average latency in read operations compared to write operations, in contrast to the throughput that is always higher in Workload 1; 4.4
Comparative Analysis: Cassandra Vs. MongoDB
After interpreting and analyzing the results obtained in all benchmarks for the two systems, we can draw several conclusions in the form of a comparison between Cassandra and MongoDB. Both systems demonstrate different forms of optimization, as well as different levels of performance depending the workload in question. 1. In Workload 1, the tests demonstrated that MongoDB and Cassandra have practically the same performance level, taking into account the highest values of throughput (mainly with fewer clients) and similar average latency compared to Cassandra. This indicates that in terms of read-only operations MongoDB has a better performance than Cassandra, taking into account that
NoSQL Comparative Performance Study
437
MongoDB performs the mapping of records to volatile memory (RAM), which considerably increases performance, being called as a DBMS “in memory”, just like Cassandra, but with different mechanisms. Cassandra, on the other hand, presents slightly lower performance, since this database is optimized for writing operations, due to the use of sequential writing mechanisms and, therefore, parts of the same record that is being read, are housed in different files (in memory), increasing execution times; 2. In Workload 2, it is verified, that the database that presents the best performance is Cassandra; 3. In Workload 2, Cassandra demonstrates an evident superiority, with high throughput increases and more capacity to handle higher values of workload (512 vs. 256 clients). A very important factor: Cassandra puts all records in memory, followed by sequential writing on disk, which reduces the number of operations performed by the disk, and increases the overall performance, since the speed in memory is higher than the speed in disk. Cassandra, considering these mechanisms, is more optimized to perform writing operations. On the other hand, MongoDB uses concurrency control mechanisms (blocks), which reduces the speed of execution of operations, as well as the fact that it uses more disk than memory to perform these operations; 4. With regard to Workload 3, it was concluded, as in Workload 1, that the throughput values are much higher in MongoDB compared to Cassandra. But overall, MongoDB is the one with the best performance because, in addition to these aspects, it has lower average latency values than Cassandra and can handle more workload (stable values of operations per second up to 512 clients);
5
Conclusions
In this article, the concepts of NoSQL, Cassandra and MongoDB database systems, as well as the YCSB benchmarking tool were introduced. Then, a practical study was carried out by producing multiple benchmarks on the same conditions in both solutions (Cassandra and MongoDB), in order to analyze and draw conclusions about their performance in terms of throughput and latency with regard to operations of writing, reading, and writing and reading simultaneously, and thus realizing which of the systems proves to be more efficient and optimized. With this study, it was possible to conclude that both databases demonstrate good levels of performance, taking into account the hardware used, both with regard to the number of operations they can perform in a short time, as well as values of low latency, considering the number of clients to carry out operations simultaneously. It was found that in the case of MongoDB, due to the mechanisms and characteristics of this system, it has better performance in reading operations, while Cassandra obtained better results in terms of writing operations. The use of one or another database must be properly analyzed according to what it is intended to store in terms of information (data volumes and data type),
438
P. Martins et al.
as both solutions differ not only in terms of the type of data model for which they are intended but also in terms of the mechanisms used, which can affect the performance of each database in different types of operations and workloads. Acknowledgements. “This work is funded by National Funds through the FCT - Foundation for Science and Technology, I.P., within the scope of the project Ref UIDB/05583/2020. Furthermore, we would like to thank the Research Centre in Digital Services (CISeD), the Polytechnic of Viseu for their support.”
References 1. Apache Cassandra Documentation. https://cassandra.apache.org/doc/latest Accessed 10 Jul 2020 2. The MongoDB 4.4 Manual. https://docs.mongodb.com/manual Accessed 11 Jul 2020 3. K. Bhamra. A Comparative Analysis of MongoDB and Cassandra A thesis presented for the degree of Master of Science. PhD thesis, Nov 2017 4. Bousalem, Z., Guabassi, I.E., Cherti, I.: Relational databases versus HBase: an experimental evaluation. Adv. Sci. Technol. Eng. Syst. J. 4(2), 395–401 (2019) 5. Cooper, B.F.: Github - Yahoo! Cloud Serving Benchmark. https://github.com/ brianfrankcooper/YCSB Accessed 11 Jul 2020 6. Cooper, B.F.: How to Benchmark CassandraDB with YCSB Workloads on All-Flash Block Storage? https://www.msystechnologies.com/blog/how-tobenchmark-cassandradb-with-ycsb-workloads-on-all-flash-block-storage Accessed 22 Jul 2020 7. Kamsky, A.: Performance Testing Mongodb 3.0 Part 1: Throughput Improvements Measured with YCSB. https://www.mongodb.com/blog/post/performancetesting-mongodb-30-part-1-throughput-improvements-measured-ycsb Accessed 21 Jul 2020 8. Li, Y., Manoharan, S.: A performance comparison of SQL and NoSQL databases. In: 2013 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM), pp. 15–19, Vitoria, BC, Canada, IEEE August 2013 9. Mansouri, Y., Babar, M.A.: The Impact of Distance on Performance and Scalability of Distributed Database Systems in Hybrid Clouds, Jul 2020. arXiv:2007.15826 10. Matallah, H.: Experimental comparative study of NoSQL databases: HBASE versus MongoDB by YCSB. Int. J. Comput. Syst. Sci. Eng. 32(4), 307–317 (2017) 11. ScaleGrid. How to Benchmark MongoDB with YCSB? https://scalegrid.io/blog/ how-to-benchmark-mongodb-with-ycsb Accessed 05 Aug 2020
A Model for Designing SMES’ Digital Transformation Roadmap Lu´ıs Cunha1 and Crist´ ov˜ ao Sousa1,2(B) 1
CIICESI, Escola Superior de Tecnologia E Gest˜ ao, Instituto Polit´ecnico Do Porto, Felgueiras, Portugal {8130277,cds}@estg.ipp.pt, [email protected] 2 INESC TEC, Porto, Portugal
Abstract. Industry 4.0 confronts companies, in particular SMEs, with various technological, organizational and cultural challenges with great impact on traditional business models. This paradigmatic socio-technical shift, implies the redefinition of the role of people in the organisation, the integration of all organisational decision layers (from the factory floor to the decision support structures) and the digital connection of the entire value chain, including processes, people and machines. However, the lack of qualified resources and the lack of an holistic understanding of industry 4.0 derail SMES’ digital transformation journey. This research work discusses the need for industry 4.0 re-conceptualisation, tailored to SMES’ needs. A lightweight ontology is presented and discussed how it contributes to the organisation and structuring a Community Of Practice, to share knowledge in the context of SMES’ industry 4.0 initiatives. Despite of the discussed use case, the developed artefact might be used to assess SME’s digital readiness. Keywords: Industry 4.0 · Maturity models transformation · Lightweight ontology
1
· SME · Digital
Introduction
The globalization phenomenon, driven by the digitalization of services a processes automation, leverage industrial competitiveness to a new level of demand. In this new competitive environment, consumers are the main agents of market changes, pushing business models to the limit, demanding for: i) a greater variety and customization of products and services; ii) better quality on orders’ fulfillment (deadlines, quality, etc.). To cope with this, industry must increase production flexibility of both upstream and downstream collaboration, as well as, real-time monitoring of products and demands, to quickly adapt to market changes and meet the customers requirements. This, however, requires companies to change their working standards, adopting an organic-based stance, whose cognitive synchronization of the parts (all) contributes to the optimization of the c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 439–448, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_42
440
L. Cunha and C. Sousa
whole [1]. This is actually one of the main assumptions for this new edge industrial management paradigm, called industry 4.0. Yet, this paradigm shift is not an easy journey for most of the companies, specially SMEs (Small and Medium Enterprises). Industry 4.0 initiatives, enclose challenges and barriers for which SMEs are not ready [2]. Moreover, Industry 4.0 and the underlying business changes, should not be seen as “one-size” “fits all” recipe. SMEs needs affordable and uncomplicated models, whose adoption can be feasible, in the context of an iterative-based maturity road-map approach. Considering that small and medium enterprises represent 5,32% of the Portuguese industry [3], this paper discusses a model to support small enterprises to: i) assess digital readiness in the scope of industry 4.0 initiatives or projects; ii) retrieve useful information in the context of digital transformation activities; iii) design their digital transformation road-map strategy. Hence, a conceptual analysis was followed, decomposing and simplifying existing reference models into a semantic artefact [4]. This artefact aims at representing the needs, means and ends pipeline in the scope of industry 4.0 digital transformation initiatives.
2
Industry 4.0 Overview
Industry 4.0 might be seen as a socio-technical [5] recipe for industries to “cook” their own digital transformation “dish”. It includes an extensive list of interdependent ingredients (concepts), that should be dosed according to industry needs. Industry 4.0 is based on a networked holistic vision, whose business assets includes partners (suppliers and customers), people, machines, processes, devices and all other things, properly orchestrated towards a common business purpose, which might be to produce better products with less resources in an sustainable, flexible and environmental-friendly way [6]. But, while large companies are shifting, progressively, to this new paradigmatic business model, SMEs still facing difficulties in starting their digital transformation journey. SMEs must be able to understand the industry 4.0 concept in order to identify and map their needs to the means that will enable the company to install the appropriated capabilities to overcame digital transformation challenges. The truth is that Industry 4.0 encloses a conceptual quagmire that must be clarified. For this, some reference architectures emerged such as RAMI4.0 [4] and IIRA [4]. The idea behind these reference models is to provide a common view over industry 4.0 and guide efforts on the implementation of a unified strategy. Still, those architectures are hard to be instantiate for SMEs. 2.1
Industry 4.0 Most Common Reference Models
RAMI 4.0 is a reference architecture model for Industry 4.0, presented at the Hannover Messe 2015, with the authorship of the industrial associations BITCOM, VDMA and ZVEI [7]. Conceptualized on a 3D model and developed in a service-oriented architecture (SOA), its focus is on manufacturing, process automation, digitization and communication technologies to make the Smart Factory vision real and to address the problem of Industry 4.0 adoption [8].
A Model for Designing SMES’ Digital Transformation Roadmap
441
IIRA (Industrial Internet Reference Architecture) is an open, standards-based architecture for IIoT systems and maximizes its value for having wide applicability in industry to boost interoperability, map applicable technologies and guide the development of technologies and standards. The IIRA was developed by the IIC (Industrial Internet Consortium), witch is an organization formed by 5 companies: AT&T, Cisco, General Electric, Intel and IBM with the objective of facilitating the implementation of IIoT (Industrial Internet of Things) in companies. The collaboration of multiple stakeholders, energy, health, manufacturing, transport and public sectors resulted in the development of this reference architecture [9]. Both RAMI and IIRA triggered several technical-scientific discussions and studies, which resulted in several instances representing different views on industry 4.0, however, most of those instances aren’t SMEs-friendly [10]. Industry 4.0, in the context of small enterprises, call for a back-to-the-basis methodology, returning the necessary and sufficient simplicity to industry 4.0 strategy, in order to provide useful support SMES in their digital journey. For this we propose reconceptualising industry 4.0, tailored to SMES. 2.2
Assessing SMEs Readiness for I4.0
The concept associated with the fourth industrial revolution implies an increase in the complexity of manufacturing processes [11,12]. The realization of several strategic orientation workshops for several companies, revealed the existence of difficulties that companies have in understanding the technological stack associated with Industry 4.0 and the underlying concepts [5,11]. Companies are unable to establish the relationship between Industry 4.0 concepts and their particular business strategies. Determining the state of development of companies in relation to Industry 4.0 is a problem that results in the absence of strategic actions, projects and implementation programs [11] aligned with the assumptions associated with the fourth industrial revolution. Uncertainty is still common to many small and medium-sized companies, as they cannot even estimate the effort required to acquire the technologies around Industry 4.0 and cannot predict the impact on their business models [11]. The migration from the industrial paradigm to Industry 4.0 presupposes an evolutionary approach according to interactive and incremental models, the path of which depends on the state of the organization compared to the architectural stack of Industry 4.0. Considering, on the one hand, the complexity of the concept of Industry 4.0 and its architecture and, on the other hand, the lack of human resources with know-how in the area, work has emerged that sought to systematize the knowledge inherent in Industry 4.0 and its architectures reference in maturity models. Maturity models are used as an instrument for conceptualizing and measuring maturity of an organization or process in order to respect the defined objectives [11]. The maturity models aim to capture the starting point and allow the start of the development of strategies and processes aimed at the adoption of Industry 4.0 practices [11].
442
3
L. Cunha and C. Sousa
(Re)Conceptualisation of Industry 4.0
The re-conceptualisation process took into consideration the main reference models (RAMI and IIRA), as well as, some surveys developed to assess industry 4.0 readiness and maturity models [11]. These resources were analysed and five main categories were extracted, namely; i) Vertical Integration; ii) Horizontal Integration; iii) IT (Information Technologies); iv) Product Development, and; v) Marketing. As the cornerstone for these dimensions, people and business processes emerge. The smart integration of digital technologies in each of these dimensions, fosters organisational and operation changes, allowing companies to add new value to their processes, products and/or services. This is called digital transformation [13,14] - a continuum process to leverage industry 4.0 maturity. As a continuum, it means it encloses several stages or levels of digital capabilities, allowing business to overcome certain needs. For each dimension, three maturity levels were identified, which combined, represent SME’s digital transformation journey. Hence, we came to the conceptual framework depicted by Fig. 1.
Fig. 1. Industry 4.0 conceptual framework
Accordingly, each dimension encloses a digital path that must be covered to achieve a certain level of maturity, wherein the company obtain the necessary capabilities to embrace a certain I4.0 challenge. Table 1 summarizes the digital maturity levels associated to each dimension. Moreover, each I4.0 challenge might require different levels of digital transformation maturity in different dimensions. The conceptual framework encloses a static representation. In order to ensure its utility, it was developed a model, wherein based on organisations’ needs, it is possible to identify the means to implement the appropriated capabilities to support organisations’ industry 4.0 strategy. The model was developed from
A Model for Designing SMES’ Digital Transformation Roadmap
443
Table 1. Digital maturity levels of I4.0 conceptual framework Dimensions
Digital maturity levels L1 L2
L3
Vertical integration
OT-IT integration
Data integration
Data aggregation
Horizontal integration
Data standardisation
Architecture standardisation
Value chain automation
Product Dev.
Automation
M2M and H2M interaction
Digital twin
ICT
Business capabilities oriented development
Real-time and scalable IT setups IT setups integration intelligence
Marketing
Go social
Go intelligent
Go auto
a semantic perspective to represent, explicitly, the concepts and the relations between concepts on the framework. This semantic artefact allows organisations to: i)understand the dependencies between different digital transformation maturity levels; ii) become aware of their maturity level, and; iii) design their digital transformation journey towards industry 4.0. The semantic characterisation of the aforementioned conceptual framework, was performed to address the following competency questions (focus questions): – – – – – – – – – – – –
What kind of needs (typically) drives an I4.0I? What kind of capabilities enables an I4.0I? Which are the maturity Levels’ enabling capabilities? Which are the maturity levels’ (digital) enablers? Which are the maturity levels’ top needs? What dimension do maturity levels belong to? What kind of digital transformation processes/activities are triggered by what certain needs? What kind of enablers are required by a specific digital transformation process? What specific (practices/protocols/frameworks/standards) do support a specific digital transformation process? What kind of protocols does a framework use? What kind of standards does a protocol implements? What kind of practices does a framework allows?
At the end, the semantic artefact should be able to answer the competency questions above, guiding industrial organisations on their industry 4.0 initiatives. Figure 2 depicts a lightweight ontology, representing the conceptual framework. Accordingly, each of the Digital Transformation occurs at specific Business level and has needs, means and ends. The Needs of each level is a set of business requirements/entities of a organization for meet a organizational objective. The Means includes a set of protocols, standards, frameworks or practices to achieve
444
L. Cunha and C. Sousa
the specific need. The Ends contains the capabilities acquired by the organization after a need conclusion. Maturity models are used as an instrument for conceptualizing and measuring maturity of an organization or process in order to respect the defined objectives [11].
Fig. 2. lightweight ontology for describing SMEs industry 4.0 initiatives
4
Experiment
The developed lightweight ontology (or conceptual ontology) can have several contexts of use, namely: i) SMEs digital maturity assessment or digital readiness; ii) roadmap definition within the scope of digital transformation initiatives, and; iii) Knowledge Organisation System for structuring and organisation of a collaborative platforms. Considering that, one of the recommendations for SMEs to ignite their digital transformation journey is to get involved in virtual communities [15], in order to share information and experiences on digital transformation initiatives, the authors describe how the developed artefact was used as a Knowledge Organisation System [16] for the semantic organisation and structuring of a Community of Practice (CoP) web platform. Learning from our pairs can be a factor that enables change. It increase trust and provides relevant information to substantiate SMEs’ decisions towards industry 4.0. As far as we know, the existence of such communities - focused
A Model for Designing SMES’ Digital Transformation Roadmap
445
on SMEs - are scarce. Moreover, they lack on a common view to support information sharing. Following the need and the evidence, the CoP was designed to manage information and share knowledge about systems, cases, lessons and articles, carried out in the context of SMES’ industry 4.0 initiatives. Therefore, KOS assumed a twofold purpose: i) to ensure a common representation of the CoP content, and; ii) to support a content-based information retrieval approach. As for the common representation of information, let’s consider the example of a specific CoP content item, such as an article. The article might be textually described as follows: IoT architecture for SMEs is an article, addressing the need for monitoring of production process and events. It focus a Industry 4.0 vertical integration dimension, enclosing three maturity levels and the underlying digital transformation processes, namely: i) OT-IT integration; ii) Data Integration, and; iii) Data aggregation. These processes required digital enablers such as: Programmable logic controller (PLCs) - used to control different electro-mechanical processes for use in manufacturing, plants, or other automation environments; sensors, edge computing and cloud computing and automation technology. For these, MQTT 1 - a machine-to-machine (M2M) “Internet of Things” connectivity protocol - and OPC-UA2 standards were needed at the facility level. NodeRED 3 was the main integration framework following publish/subscribe communication pattern and SOA architecture pattern. For experimental purposes, KOS was implemented using Neo4j4 graph database platform, which allows to store, query, analyze and manage highly connected data more efficiently and to leverage relationships. Neo4j repository contains, in the same data-space, two interconnected graphs: the KOS itself, and; the data describing the CoP content. To distinguish the KOS model form the data graph it was consider the “is-a” relationship. Accordingly, when a node is connected by an “is-a” relation to another node, it means the first one is an instance of the second. Figure 3 depicts and excerpt of the article representation “IoT architecture for SMEs”. Additionally, it is possible to observe other instances such as the maturity levels, the digital transformation process and examples of standards, protocols, frameworks and ICT Tools. In the figure, instances are represented as rectangles and connected by an “is-a” to the respective concept or entity. Regarding information retrieval aspects, CoP is based on an visual-oriented architecture and graph-based discovery algorithms. Neo4j data access is offered trough a GraphQL5 query language API. The API was made available trough
1 2 3 4 5
https://mqtt.org. https://opcfoundation.org/about/opc-technologies/opc-ua/. https://nodered.org. https://neo4j.com. https://graphql.org.
446
L. Cunha and C. Sousa
Fig. 3. Semantic representation of and article in the CoP
Apollo server6 . The front-end, was implemented using Apollo Client, and the visualisation component was built upon PopotoJS7 . Popoto.js is a JavaScript library built with D3.js designed to create interactive and customizable visual query builder for Neo4j graph databases (see footnote 4). This technological stack allowed to implement rich visual data navigation functionalities and take advantage of advanced search and data discovery algorithms offered by Neo4j.
6 7
https://www.apollographql.com/docs/apollo-server/. http://www.popotojs.com.
A Model for Designing SMES’ Digital Transformation Roadmap
5
447
Conclusion
I4.0 implementation is still on its infancy for SMEs, and to make it a reality, several challenges and gaps must be addressed. From the SMEs point-of-view, the digital transformation roadmap cannot be trodden using existent reference architecture models and their “one-size” fits all approach. This paper presented a lightweight ontology representing the industry 4.0 conceptual framework, tailored to SMES’s needs. The scenarios for using this artifact are several and very promising, however, the authors explored, successfully, the design of a semanticbased community-of-practice to share experiences, lessons, articles and systems in a common and standardised way. The main purpose of the developed platform is to provide answers on how to move forward on the digital transformation roadmap, following and iterative approach grounded in digital maturity levels. The results discussed in this paper encourage the authors to develop a formal ontology an implement it in a tool to assess SME’s digital readiness in the scope of their intended industry 4.0 strategy. Acknowledgments. This work is financed by National Funds through the Portuguese funding agency, FCT – Funda¸ca ˜o para a Ciˆencia e a Tecnologia within project UIDB/50014/2020
References 1. Veile, J.W., Kiel, D., M¨ uller, J.M., Voigt, K.I.: Lessons learned from Industry 4.0 implementation in the German manufacturing industry. J. Manufact. Technol. Manage. 31(5), 977–997 (2019), https://www.emerald.com/insight/content/doi/ 10.1108/JMTM-08-2018-0270/full/html 2. Horv´ ath, D., Szab´ o, R.Z.: Driving forces and barriers of industry 4.0: do multinational and small and medium-sized companies have equal opportunities? 146, 119–132 (2019). https://linkinghub.elsevier.com/retrieve/pii/S0040162518315737 3. PORDATA - Base de Dados de Portugal, https://www.pordata.pt/Portugal/ Pequenas+e+m/%C3/%A9dias+empresas+total+e+por+sector+de+actividade+ econ/%C3/%B3mica-2928 4. Simth, B.: RAMI 4.0 and IIRA Reference Architecture Models a Question of Perspective and Focus (2015) 5. Erol, S., Schuhmacher, A., Sihn, W.: Strategic guidance towards industry 4.0 - a three-stage process model. In: Resource Efficiency for Global Competitiveness, pp. 495–500 (2016) 6. Industry 4.0: fourth industrial revolution guide to Industrie 4.0. https://www.iscoop.eu/industry-4-0/ 7. Zezulka, F., Marcon, P., Vesely, I., Sajdl, O.: Industry 4.0 – an Introduction in the phenomenon. IFAC-PapersOnLine 49(25), 8–12 (2016). https://linkinghub. elsevier.com/retrieve/pii/S2405896316326386 8. Contreras, J.D., Garcia, J.I., Diaz, J.D.: Developing of Industry 4.0 Applications. Int. J. Online Eng. (iJOE) 13(10), 30 (2017). http://online-journals.org/index. php/i-joe/article/view/7331
448
L. Cunha and C. Sousa
9. Paper, W., Summary, E.: Interoperability between IIC Architecture & Industry 4.0 Reference Architecture for Industrial Assets (2016). https://www. semanticscholar.org/paper/Interoperability-between-IIC-Architecture-%26-4.0for-Paper-Summary/397678aef42cc6bf2d447c8a41bf7a885d6fa920 10. Mittal, S., Khan, M.A., Romero, D., Wuest, T.: A critical review of smart manufacturing & Industry 4.0 maturity models: Implications for small and medium-sized enterprises (SMEs). J. Manufact. Syst. 49, 194–214 (2018). https://linkinghub. elsevier.com/retrieve/pii/S0278612518301341 11. Schumacher, A., Erol, S., Sihn, W.: A Maturity model for assessing industry 4.0 readiness and maturity of manufacturing enterprises. Procedia CIRP 52, 161–166 (2016). https://linkinghub.elsevier.com/retrieve/pii/S2212827116307909 12. Pessl, E.: Roadmap industry 4.0 – implementation guideline for enterprises. Int. J. Sci. Technol. Soc. 5(6), 193 (2017). http://www.sciencepublishinggroup.com/ journal/paperinfo?journalid=183&doi=10.11648/j.ijsts.20170506.14 13. Reis, J., Amorim, M., Mel˜ ao, N., Matos, P.: Digital Transformation: A Literature Review and Guidelines for Future Research. In: Rocha, , Adeli, H., Reis, L.P., Costanzo, S. (eds.) Trends and Advances in Information Systems and Technologies, pp. 411–421. Advances in Intelligent Systems and Computing, Springer, Cham (2018) 14. Verhoef, P.C., Broekhuizen, T., Bart, Y., Bhattacharya, A., Qi Dong, J., Fabian, N., Haenlein, M.: Digital transformation: a multidisciplinary reflection and research agenda. Journal of Business Research, vol. 122, pp. 889–901 (January 2021). https://linkinghub.elsevier.com/retrieve/pii/S0148296319305478 15. Freitas, R., Sousa, C., Sousa, C.: Industry 4.0 in tˆ amega e sousa’s region in a twofold perspective: Industry vs IT enterprise. In: 2018 13th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6. IEEE (2018) https:// ieeexplore.ieee.org/document/8399309/ 16. Friedman, A., Smiraglia, R.P.: Nodes and arcs: concept map, semiotics, and knowledge organization. J. Documentation 69(1), 27–48 (2013). http://www. emeraldinsight.com/journals.htm?articleid=17073332&show=abstract
Boosting E-Auditing Process Through E-Files Semantic Enrichment Crist´ov˜ ao Sousa1,2(B) , Mariana Carvalho1 , and Carla Pereira1,2 1
CIICESI, Escola Superior de Tecnologia e Gest˜ ao, Instituto Polit´ecnico do Porto, Felgueiras, Portugal {cds,mrc,csp}@estg.ipp.pt 2 INESC TEC, Porto, Portugal [email protected]
Abstract. E-auditing has been evolving along with the global phenomenon of digitization. Auditors are dealing with new technological challenges and there is the need for more sophisticated tools to support their activities. However, the digital processes used for identifying and validating inconsistencies in the organizations’ financial information are not very efficient. Due to the high number of violations occurrences of tax e-auditing rules, in which many of them turn out to be irrelevant; the auditors’ work is often hindered, which may lead to incomplete data analysis. In this paper, we propose an approach to the e-auditing process based on the SAF-T (PT) files semantics enrichment using a graph-based data structure representation format. Using a graph-based data representation, we can take advantage of another way to perform queries and discovery mechanisms to retrieve information and knowledge, easing the auditing process and consequently enhancing the outcome of the tax e-auditing rules application.
Keywords: Knowledge management Semantics
1
· Tax e-Auditing · SAF-T (PT) ·
Introduction
As organizations need to evolve and digitize all of the internal processes, the tax auditing process also needs to be digitized/computerized to keep up with this evolution. Tax auditing [8] is the process in which an auditor crosschecks the financial information of a specific organization to validate and verify if there are any inconsistencies. Nowadays, the tax auditing process still requires manual intervention of the auditor, which may turn this process slow and inefficient. Although the intervention of the auditor is essential, his role should be mainly focused on defining audit rules and actions for digital inspection. He/she should have less intensive role, especially in the phase when he/she checks the inconsistencies in the efile. With the increasing electronic information and due to the huge volume of c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 449–458, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_43
450
C. Sousa et al.
data in e-files, the auditors only select random data samples (instead of using all data) to validate and verify inconsistencies. Thus, inconsistencies in the nonselected data may not be discovered. To ease this process, intelligent mechanisms can be provided in order to adapt the process to identify inconsistencies in an automated way. After reviewing the literature, we realized that the e-audit process does not yet present tools that efficiently support auditors in this process. Therefore, in this paper, we propose a semantic enrichment-based decision support system represented using a graph-based data structure format. The e-files are converted in a semantic-enriched graph-based data structure and independently audited through the representation of tax audit rules defined by the Organization for Economic Co-operation and Development (OECD). This paper is organized as follows: Sect. 2 presents the concept of e-auditing process and review of literature on this subject, Sect. 3 contains the proposed decision support system developed for support the audit process. Finally, Sect. 4 concludes the paper and suggests future work.
2
e-Auditing Process and Standards
The e-auditing process can be characterized by 5 levels [3] represented in Table 1. The higher the level, the more independent, faster and efficient the audit process. Table 1. Maturity levels in e-auditing. Level 1 e-file
Organizations must use a structured file to issue their tax declarations
Level 2 e-accounting Organizations must communicate their tax data electronically within a defined period of time. Paradigm changing Level 3 e-match
Organizations must communicate their tax data electronically, which is crosschecked with all contributors data in real-time.
Level 4 e-audit
The submitted data in Level 2 is crosschecked in real time to identify inconsistencies. Organizations are notified and need to respond and rectify in a specific period time.
Disruptive Level 5 e-assets
Tax data is accessed and analyzed by auditors in real time without the use of files. If any inconsistencies are identified, organizations are notified and need to respond and rectify in a specific period time
When compared with more rudimentary and manual techniques, E-auditing has several advantages: [5] – Faster analysis: Since the entire analysis process is done digitally, all the analysis processes, including the most complex and time-consuming, become faster.
Boosting E-Auditing Process Through E-Files Semantic Enrichment
451
– Reliability: Given the automation of all analysis processes, the information derived from the analysis using the digital system is correct. As the hypothesis of human error is excluded, it does not contain errors. – Regularized Analysis: the audit process can be seen as a tax management tool for an organization. If the organization frequently validates its tax situation using an e-auditing system, when the organization is audited by the competent entities, its situation is regularized and it has, over time, the exact notion of the existing problems in its tax situation. As seen before, organizations must communicate their tax information to the tax authorities. In Portugal, the e-file that contains this tax information is called Standart Audit File for Tax Purposes (SAF-T (PT))1 . This file is a standardized file in XML, Extensible Markup Language, and allows to export accounting, billing and transport documents/records received and issued from organizations. There are two types of SAF-T (PT) files structures: billing and accounting. The billing SAF-T (PT) is a simple file that is sent monthly to the Autoridade Tribut´ aria e Aduaneira (ATA) and from which it is possible to extract monthly billing information from organizations such as products sold, IVA rates used, customer list and documents issued by the organization in that period of time. The other one, accounting SAF-T (PT), is a more complete version. This file is composed of all information of the billing SAF-T (PT) and other accounting entries that the organization made in the period requested by the auditors in the audit process. This file is independent of any software program, which facilitates its integration with e-audit systems. In order to streamline the entire e-audit process, the OECD has issued a document [7] with a set of good practices for analyzing tax documents. This document aims at facilitating and simplifying the discovery of inconsistencies and ease the e-auditing process. This analysis is done through the execution of tests, mostly logical propositions, using the financial information from the SAF-T (PT). These tests can be divided in three categories: 1. Compliance Tests – These tests are used to validate an organization’s internal transactions and to determine whether the transactions are being consistently and correctly performed; 2. Substantive Tests – In addition to the execution of transactions and their consistency (made by the compliance tests), the substantive tests validate the content of those same transactions, relating them to the best accounting practices and legal requirements. The degree of detail is influenced by the outcome of the compliance tests; 3. Sampling Techniques - Statistical inference and sampling are used to select data sets or patterns to validate. This validation might generate inconsistencies, which must be thoroughly analyzed by the auditor. 1
http://info.portaldasfinancas.gov.pt/pt/apoio contribuinte/SAFT PT/Paginas/ news-saf-t-pt.aspx.
452
2.1
C. Sousa et al.
Related Work on Tax E-Auditing
There are some available tools that aim at supporting the audit process. As stated before, one of the main goals of using tools for assisting in this process is to ease the audit process for the auditor, decreasing his/her workload. Tax e-auditing has been widely used [4,12–14] and in the last years, there has been an increasingly use of tools for supporting the audit process. In Portugal, the tax e-file SAF-T (PT) was first introduced in 2008. Until now, four additional data structures alterations have been published by the Portuguese Tax Authorities (PTA)2 . Recently, PTA has defined that the declaration of SAF-T (PT) accounting files will be mandatory in 20213 . But, curiously, there are only a few papers concerning how (or if) Portuguese auditors use electronic tools during the auditing process. [10] focused on understand how Portuguese auditors use information technologies in their auditing processes, they carried out in-depth interviews, discovering which were the used/preferred tools in several tasks during the auditing process. [2] conducted a study aiming at identifying the most used tools for helping Portuguese auditors in the audit process. [1] aims at identifying the most used computerized tools by the Portuguese Statutory Auditors. Both studies concluded that auditors are still reluctant in using computerized tools: “the longer the auditor work in auditing, the more willing to use these tools” [2]. Since most auditors are not skilled with an information technology background, they have a great difficulty in using computer-aided auditing systems. We perform a survey of the features of the mentioned tools in [2,10], which is represented in Table 2 As shown in Table 2 and referred in [9,11], col.bi 4 is the most complete tool when compared with other tools. As for the remaining tools, CentralGest5 , Filosoft6 , IDEA7 , Galvanise (former ACL)8 , Delloitte9 and Ey10 : they all present gaps/flaws when it comes to features, functionalities or even costs for acquiring the tool or licensing. However, despite all these tools fulfill their purpose (by helping the auditor to perform the audit in an easier and more efficient way) they do not present a satisfactory level of performance when processing large amounts of data. As these systems are not sufficiently optimized, this process turns out to be time consuming and cumbersome. It would be necessary to have an adequate support tool, such as a tool with integrated high processing capabilities, scalable, completely 2 3 4 5 6 7 8 9 10
https://info.portaldasfinancas.gov.pt/pt/apoio contribuinte/SAFT PT/Paginas/ news-saf-t-pt.aspx. https://dre.pt/home/-/dre/139348418/details/maximized. https://www.petapilot.com/colbi. https://www.centralgest.com/software/analisador-saft. http://www.filosoft.pt/. https://idea.caseware.com/products/idea/. https://www.wegalvanize.com/. https://www2.deloitte.com. https://www.ey.com/.
Boosting E-Auditing Process Through E-Files Semantic Enrichment
453
Table 2. Audit support tools. Tools/Feature Cloud-based Detail analysis Dashboard Tax structure and rules analysis
Cost Maturity level
col.bi
Yes
Yes
Yes
Yes
Low 3
CentralGest
No
No
Yes
Yes
Low 3
Filosoft
No
No
No
Yes
Low 3
IDEA
No
Yes
Yes
Yes
High 3
Galvanise
Yes
Yes
Yes
Yes
High 3
Delloitte
No
Yes
Yes
Yes
High 3
Ey
Yes
Yes
Yes
Yes
High 3
autonomous and intelligent that would allow inferring the data, detecting possible inconsistencies in a short period of time and consequently ease the auditor’s workload. But, as far as we know, the proposed solution, a semantic-based decision support system, has not yet been explored and it could be seen as a solution for the stated problem.
3
Semantic Representation of Tax E-Files
The Portuguese tax e-files (SAF-T (PT)), already enclose a semantic structure, which is denoted by the XML data representation formalism, itself. However, the semantic metadata is limited to containment hierarchical relations, wherein, the XML-elements (SAF-T concepts) might contain one or more sub-elements. This kind of data organisation is interesting for data drilling information retrieval purposes, yet, inefficient and time-consuming in a big data scenario. Rather than narrowing down a search, based on a containment hierarchy, a contextual information retrieval approach using a multidimensional data search architecture is addressed. This multidimensional perspective is powered by a set of other relations that emerge from the SAF-T data structure, according to domain knowledge analysis. Accordingly, we move to an intelligent information retrieval architecture that supports auditors on the decision about the selection of their target inspections, based on iterative query and discover actions. To cope with this view, SAF-T e-files need to be redesigned to a semantic structure. The e-files semantic enrichment will allow a better correlation between the SAF-T content and, thus, the development of intelligent search and discover mechanisms. However, this semantic endeavour cannot exclude the domain experts. The greater the auditor’s commitments to the vocabulary used to the semantic enrichment of e-files, the greater is the utility of the query and discover activities.
454
C. Sousa et al.
Accordingly, the semantic enrichment process included the following activities: 1. 2. 3. 4.
SAF-T analysis; “Qualification” of existing hierarchical relations; Discovery of new associative relations; Add new conceptual structures based on OECD audit rules.
SAF-T analysis (1) implies to know and understand the SAF-T structure, which requires to: i) study the domain; ii) interview domain experts; iii) analyse the current SAF-T vocabulary through the associated XSD11 files. Afterwards (2), SAF-T hierarchical relations need to be “qualified”, assigning specific meaning to each type of containment hierarchical relation. There are several types of containment relations, each of them enclosing a different rationale, from which different constraints emerge, and therefore, different results of the inference process. A containment relationship might denote a containment rationale or a part-hood rationale, wherein, the second carries a greater dependency between the container and the contained. This rational ambiguity among conceptual relations must be clarified. Sometimes, the use of the appropriated terminology for naming relations, is sufficient to clarify the SAF-T conceptual structure underlying meaning. The next activity (3) implies a deeper look into SAF-T in order to extract other kind of relations. The main task was to retrieve all the elements containing the ref attribute, whose content is pointing to another element and, additionally, all namespaces (if exist), which include other defined elements used within SAF-T vocabulary. From this analysis other associative relations might be gathered. Finally (4), it was considered the OECD audit rules. Those rules allow to identify other kind of relations between concepts having into account the auditors’ point-of-view. Moreover, it is also possible to identify some roles, that some concepts assume in the scope of specific transactions. Same examples of roles are: buyer, seller, supplier, customer, etc. Each of these roles were modeled as relations (is buyer, is seller, is customer, is supplier, etc.). The conceptual alignment between the audit rules and e-files structure and content, contributes to: 1. Semantic enrichment of the existent facts within the KB; 2. Improve data indexing and inference mechanisms; 3. Allow the design of an integrated environment for e-auditing, to support auditors on rules specifications and intelligent information retrieval activities oriented to auditor’s needs. Figure 1 depicts an excerpt of SAF-T semantic enrichment, focused on the invoice concept and OECD tests for sales output tax, which aim at confirm that all sales are invoiced. Thus, the SAF-T representation was extended to consider new concepts and meaningful conceptual relations. Concretely, the OECD’s sales output tests to confirm all sales are invoiced, are as follows: 11
XML Schema Definition.
Boosting E-Auditing Process Through E-Files Semantic Enrichment
455
– Report Dispatch note/ Invoice Status, in order to verify items supplied and Dispatch Note issued but without related invoice; – Report orders fulfilled but not invoiced and Report Order/Work in Progress status, in order to identify cancelled, held or long outstanding orders; – Report selection of credit notes with tax, in order to do credit notes crossreference to a previously issued sales invoice, and; – Report payments received with no related invoice.
Fig. 1. Excerpt of SAF-T Semantic Enrichment focused on invoice
Following the OECD suggested test, a new semantic representation emerge, considering a different conceptual structure for orders, invoices and dispatch notes, providing additional information about the domain, by explicitly represent the real scope of the relationship. In the original SAF-T, orders are referenced by its ID, within a sub-element of invoice. Other concepts were considered such as: sales; Payment; Revenue; Expense;...; and new roles were explicitly represented, namely: Buyer and Seller. Additionally, the invoice types are now represented as concepts instead of enumerations. This new model has the ability to answer,
456
C. Sousa et al.
more directly, to all the OECDs recommendations. The new SAF-T conceptual representation, reshapes the pure hierarchical vision of the SAF-T data to a graph-based approach, leveraging new perspectives to information management and retrieval for the purposes of digital auditing. 3.1
Exploring E-Files for Issues Findings
Once the SAF-T has been enriched semantically, it might be represented as a graph and, therefore, it is feasible to apply graph-based algorithms to query and discover activities. To perform this analysis and extraction of information we can resort to Neo4j12 , which is a graph database management system and provides a way to create and manipulate a graph data structure. As we intent to extract some kind of knowledge or suggestions that could be helpful to the auditors, we can use Neo4j algorithms [6]: – the Louvain modularity algorithm, which is a community detection algorithm, aims at finding clusters based on community density. It maximizes the modularity score of each cluster by evaluating how a specific node influences the density cluster. In our case study, this algorithm could let us to identify suspect relationships between entities. For example, it can be used to analyze and find possible doubtful relationships between companies and their associated transactions, which may imply a possible fraud ring (or cluster) of companies. – Local Clustering Coefficient is also a community detection algorithm. It aims at finding clusters by identify the local clustering coefficient for each node, which means that, the coefficient describes the likelihood that the neighbors of a specific node are also connected. This algorithm may allow us to find entities that “know” each other and, similar to the previous described algorithm, the Louvain algorithm, to found possible fraud rings. These two algorithms can complement each other, as the outcome of the Louvain algorithm provides more information than the local clustering coefficient algorithm. – PageRank algorithm is a centrality algorithm. It measures the transitive influence of nodes, which means, this algorithm identifies the most influential nodes based on the influence of its neighbours. The influence of a node is calculated based on the score (or weight) of the relationships with its neighbours. In our case, this algorithm could be useful to find the most influential companies, based on the available data of the SAF-T (PT). – Betweenness Centrality, as PageRank algorithm, is a centrality algorithm. This algorithm assigns a score to a specific node. This score represents the number of shortest paths (between every pair of nodes) that pass through the referred node. This algorithm could be advantageous to detect hidden patterns in the relationships between companies, or even to identify possible influential companies that could not be detected by the PageRank algorithm. 12
https://neo4j.com.
Boosting E-Auditing Process Through E-Files Semantic Enrichment
457
– Common Neighbors is an algorithm that predicts the existence of possible relationships between a pair of nodes. As its name implies, this algorithm is able to identify the nodes’ common neighbours. In our case, this algorithm could be suitable to find ghost companies, as it has the ability to find different companies that have the same information in specific attributes, like phone contacts or addresses.
4
Conclusions and Future Work
Either from academic and practitioner perspective, there is evidence on issues and needs related to the e-auditing process. Auditors need assistance, not only during the auditing process but, mainly, on the selection of the companies to audit. The more accurate is the selection of the audits to be performed, the more efficient the process and the higher the ROI. Considering the structure of e-files (SAF-T) supporting the e-auditing process, it is very difficult to obtain, earlier enough, useful information on companies accounting and financial behavior, in order to decide which to audit. To find candidate companies, the auditor needs to perform data drilling actions to find relevant issues. In this paper was discussed an approach to structure a knowledge base to store graph-based e-files and, a process to transform hierarchical based e-files to a new semantic graph. Based on this new conceptual representation, new query and discovery mechanism might be implemented in order to provide a contextual information retrieval approach using a multidimensional data search. This will conduct the auditor during the issues discovery activities. Despite of promising, this approach implies to store all e-files metadata in the same repository. Currently, the authors are developing an ontology for SAF-T, whose inference rules will provide much more data to be used by the query and discover algorithms. Additionally, is to be developed a component to allow users (auditors) to specify their our auditing rules and tests. Acknowledgments. This work was supported by the Northern Regional Operational Program, Portugal 2020 and European Union, trough European Regional Development Fund (ERDF) in the scope of project number 39900-31/SI/2017, and by FCT – Funda¸ca ˜o para a Ciˆencia e Tecnologia within project UIDB/04728/2020.
References 1. Amaral, B., Marques, R.P., In´ acio, H.: The use of computer-assisted audit tools in portuguese statutory auditors’ work. In: 2019 14th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6. IEEE (2019) 2. Dias, C., Marques, R.P.: The use of computer-assisted audit tools and techniques by portuguese internal auditors. In: 2018 13th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–7. IEEE (2018) 3. EY: Tax administration is going digital: Understanding the challenges and opportunities (2019) 4. Huseynov, T., Mammadova, U., Aliyev, E., Nagiyev, F., Safiyeva, F.: The impact of the transition to electronic audit on accounting behavior. Econ. Soc. Dev. Book Proc. 4, 378–384 (2020)
458
C. Sousa et al.
5. Maydew, E.L., Shackelford, D.A.: The Changing Role of Auditors in Corporate Tax Planning, p. 40 (2005) 6. Needham, M., Hodler, A.E.: Graph Algorithms: Practical Examples in Apache Spark and Neo4j. O’Reilly Media, Newton (2019) 7. OCDE: Guidance on Test Procedures for Tax Audit Assurance. https://www.oecd. org/tax/forum-on-tax-administration/publications-and-products/technologies/ 45045414.pdf 8. OCDE: Strengthening Tax Audit Capabilities: General Principles and Approaches. https://www.oecd.org/tax/administration/37589900.pdf 9. O’Neill, H., Vicente, L.B.M., Pinho, V., et al.: Reporting model for decision support based on the saf-t standard. In: 17a Conferˆencia da APSI, vol. 17, pp. 406–409 (2017) 10. Pedrosa, I., Costa, C.J.: Computer assisted audit tools and techniques in real world: Caatt’s applications and approaches in context. Int. J. Comput. Inf. Syst. Ind. Manag. Appl. 4, 161–168 (2012) 11. Reis, D.S.: Distributed and scalable architecture for saf-t analysis and processing (2018) 12. Ruliana, T., Indrawati, A., Febrina Lubis, S.: Audit electronic (e-audit) to the financial statements. Sci. Papers Manag. Econ. Eng. Agric. Rural Dev. 17(3), 363–368 (2017) 13. Supriadi, T., Mulyani, S., Soepardi, E.M., Farida, I.: Influence of auditor competency in using information technology on the success of e-audit system implementation. EURASIA J. Math. Sci. Technol. Educ. 15(10), em1769 (2019) 14. Tuti, T., Nzinga, J., Njoroge, M., Brown, B., Peek, N., English, M., Paton, C., van der Veer, S.N.: A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implementation Sci. 12(1), 61 (2017)
Integration of UML Diagrams from the Perspective of Enterprise Architecture Luís Cavique1(B)
, Mariana Cavique2
, and Armando B. Mendes3
1 Universidade Aberta, Lasige, Lisbon, Portugal
[email protected] 2 Universidade Europeia, Lisbon, Portugal
[email protected] 3 Universidade Açores, Algoritmi, Ponta Delgada, Portugal
[email protected]
Abstract. An integrated view of the information system has been an objective to deal with complexity. However, bibliography proposes many solutions with many synonyms depending on the layer, methodology, framework or tool used, that does not allow a broad view of the system. In this work we chose three basic elements of the information systems and we demonstrate how they are enough to integrate a set of essential UML diagrams. The proposed model firstly defines a set of UML diagrams for each layer of the Enterprise Architecture, and then heuristic rules are detailed in order to ensure vertical and horizontal alignment. Keywords: UML · CRUD · Enterprise architecture · Organization alignment
1 Introduction UML (Unified Modeling Language) (Fowler 2003) is a powerful tool that improves the quality of systems analysis and design. The use of UML iteratively in analysis and design, allows the fulfillment of the system requirements with object-oriented design, as well as with relational databases models. The bibliography on UML tools is vast and it is presented at different levels and formats. However, it is usually presented in separate chapters, where each chapter refers in detail to use-case diagrams, class diagrams, activity diagrams, state diagrams, sequence diagrams and physical diagrams. UML unified a set of diagrams of different authors, where each UML diagram corresponds to a partial view of the system, keeping the holistic view poorly developed. On the other hand, Enterprise Architecture (Lankhorst 2013) promises an integrated approach to deal with complexity, going beyond the symbolic models (such as the UML diagrams) and trying to achieve more coherent and meaningful tools, called semantic models. In this work, our goal is to give an integrated view of the system, linking the UML tools to the CRUD matrix, achromic of < create, read, update, delete > (IBM 1978, © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 459–468, 2021. https://doi.org/10.1007/978-3-030-72651-5_44
460
L. Cavique et al.
Martin 1983). An Enterprise Architecture perspective will be used, where we seek to find vertical and horizontal alignments, that ensures the consistency of the system. Enterprise Architecture is usually defined by the three-layers model: business/process architecture, information system architecture and technological architecture, as shown in Fig. 1.
Business / Processes Architecture Information System Architecture (IS)
Data Architecture
Applications Architecture
Technological Infrastructure Architecture
Fig. 1. Three-layers model of Enterprise Architecture
In Enterprise Architecture there are many synonyms depending on the layer, methodology, framework or tool used, so it is important to find a synthesis with a reduced number of names. In this work, we chose only three basic elements: the actors, the activities and the data. Some of the synonyms are as follows: (i) actors is synonym of lines-of-responsibility, (ii) activities is synonym of applications, tasks, uses-cases or operational-processes, (iii) data is synonym of classes or informational-entities. In the following examples we will use actors (α, β), activities (A, B, C) and data (X, Y, Z, W). This version of Enterprise Architecture with three layers, includes: • Business/Processes Architecture: where processes are made up of activities (A, B, C) and managed by human actors (α, β); • Information System Architecture: with two different software groups, the data (X, Y, Z, W) and the applications (A, B, C); • Technological Infrastructure Architecture: which consists of hardware components and basic software (operating systems and database management systems). The challenge of Enterprise Architecture is to create a vertical alignment that allows the communication among the business team, IS team and IT team, merging the three layers into a single architecture. The paper is organized in four additional sections. In Sect. 2, related work is presented. Section 3 introduces an integrated view of UML tools. Section 4 details the proposed method. Finally, in Sect. 5 conclusions are drawn.
Integration of UML Diagrams from the Perspective of Enterprise Architecture
461
2 Related Work In this section we introduce the subjects of: CRUD matrix, Enterprise Architectures and topic about alignment in information systems. CRUD Matrix CRUD matrix was popularized by Martin (1983) in his book Managing the Data-base Environment. CRUD matrix crosses information between applications and data classes. This approach intends to obtain a compact view of the system, as well as the ‘Design Structure Matrix’ techniques (Eppinger and Browning 2012), in order to avoid coupled sub-systems. Both techniques try to find a matrix with the main diagonal filled and the smallest number of elements in the remaining matrix. Similarly, ‘Axiomatic Design’ (Suh 2001) studies the transformation of the customer needs into functional requirements and related them to a set of design parameters using a design matrix. The customer needs correspond to the system requirements which are materialize in functionalities/applications. And, the data architecture corresponds to the design parameters and the CRUD matrix to the design matrix. Such as the axioms of axiomatic design, the goal of the CRUD matrix is to maintain the independence of the functional requirements in order to minimize the information content of the design.
Fig. 2. CRUD matrix with counters
In Fig. 2, the CRUD matrix is shown with CRUD counters, that validate possible inconsistencies (Cavique 2020a). The number 1311 indicates that there are 1 Create, 3 Reads, 1 Update and 1 Delete. There must be a single application that performs Create, Update and Delete, (CUD), and there may be multiple applications with Read operator. Preferably we will have CRUD counters with 1N11, i.e. a unique Create, Update and Delete and multiple Read operators. Enterprise Architectures Enterprise Architecture (Lankhorst 2013) reuses the term ‘architecture’ from building and construction, referring to a holistic view of the enterprise. ArchiMate is one of the most popular enterprise modelling language, created to be a meta-model of tools like UML or BPMN. However, the proposed ArchiMate language evolves into fields more and more specific, increasing the lexical complexity and losing the necessary simplicity of an architecture. In the book of Desfray and Raymond (2014) proposed the EAP (Enterprise Architecture Profile) language which extends the UML concepts in order to represent all TOGAF objects. In the sub-title, the authors promised a practical guide using UML and BPMN, which we believe has not been fully achieved.
462
L. Cavique et al.
In Barros et al. (2000) the authors modeled business processes, business entities, business roles and business events using UML language. However, only the business layer was dealt with, lacking the information system and technological layers. Silingas and Butleris (2009) proposed an approach to customizing UML tools for domain-specific modeling needs. The authors reused the generic Zachman framework for answering to the 6 Wh questions in Business, System, and Technology models. However, they do not provide a clear procedure to apply the approach. Perez-Castillo et al. (2019) mapped EA, concluding that the process is costly and subject to errors, which may discourage enterprises from adopting EA. In this work we will choose the UML diagrams, since they have been on the market for decades and there is a wide community of professionals already familiar. Information System Alignment The authors (Pereira and Sousa 2005) and (Vasconcelos 2017) presented a set of three heuristic rules that guarantee the alignment of the information system. The heuristic rules, shown in Fig. 3, can be summarized as follows: #1 The data architecture must support the architecture of the business processes; #2 Each process activity is automated by a single application; #3 Each data set is managed (CUD) by a single application in the CRUD matrix; Rules #1 and #2 guarantee vertical alignment and rule #3 guarantees horizontal alignment.
Fig. 3. Heuristic rules for enterprise alignment
3 Integrated View of UML Tools BSP (Business Systems Planning) (IBM 1978) presents four basic elements for Information Systems Planning, coined as the Iron Cross: organization, applications, data and technological systems (Rocha and Freixo 2015).
Integration of UML Diagrams from the Perspective of Enterprise Architecture
463
In this approach we use three of the four elements. Our UML analysis considers the following entities: the actors, the use-cases/applications and the data classes (Cavique 2020a). Next, we demonstrate that three elements are enough to guide the set of UML diagrams. Although CRUD matrix is not a UML tool, it can complete a system view as shown in Fig. 4.
Fig. 4. UML diagrams and CRUD matrix guided by the three essential elements
With the three vectors (actors, applications, classes) combinations of an element, of pair of elements and of a trio can be generated. The only diagram with a single element is the class diagram, which is a consequence of the business narrative. The pair (actors, applications/use-cases) is called a use-case diagram. A second pair (applications, classes) is represented by the CRUD matrix. A third pair (actors, classes) is represented by the sequence diagram. Finally, the set of all sequence diagrams represents the trio of (applications, actors, classes).
4 Proposed Model In the proposed model, firstly we define UML diagrams for each layer of the Enterprise Architecture, then we present the alignment heuristic rules for Enterprise Architecture and finally a procedure to check alignment is detailed. 4.1 UML Diagrams for Enterprise Architecture In this work we intend to detail the Enterprise Architecture using UML diagrams for modeling and systems analysis. Figure 5 shows the three layers of Enterprise Architecture associated with UML diagrams. In the 1st layer (above) it is shown the processes diagram
464
L. Cavique et al.
with a sequence of activities. In the 2nd layer, the list of applications (in the center on the right), the data diagram (in the center on the left), the CRUD matrix merging the applications and data items (in the center on the middle) are presented. Finally, in the 3rd layer, the infrastructure diagram is shown below. To represent the process diagram, the BPMN (Business Process Model and Notation), an activity diagram with partitions (‘swim-lanes’) in UML or a simplified version in a UML use-case diagram can be used. For the representation of the data, an Entity-Relationship diagram or a class diagram in UML can be used. Also, in the 2nd layer there is a list of applications and a CRUD matrix that unify applications and data. Finally, the infrastructure diagram can be represented by an architecture diagram or UML implementation using components and nodes. 4.2 Heuristic Rules to Align the Enterprise Architecture The set of diagrams must be aligned, in order to assure consistency in their articulation. Two alignments are generally considered, vertical and horizontal alignment. The whole architecture begins with a description of the business area, that we call ‘Business Area Narrative’. This narrative can be obtained through meetings, focus groups, interviews (structured or unstructured) or case studies. The narrative generally includes a survey of the ’as-is’ system (past and present) and the intended or ’to-be’ (future) system, associated with a set of functional requirements. The analysis of the system considers two possible views: the view of the functional analyst who deals with the end users oriented to the process architecture (1st layer) and the view of the organic/systems analyst oriented to the IS/IT architecture (2nd and 3rd layer). For that reason, from the business narrative two heuristic rules are applied, rule #1 related to data architecture and the new rule #0 is connected to the process architecture.
Fig. 5. Three layers of the Enterprise Architecture associated with the diagrams
Integration of UML Diagrams from the Perspective of Enterprise Architecture
465
Fig. 6. Heuristic rules to align the Enterprise Architecture
The activities that can be automated are called applications, following rule #2. And applying rule #3, the crossing of data with applications is represented in the CRUD matrix. The authors (Pereira and Sousa 2005) and (Vasconcelos 2017) presented a set of three heuristic rules that guarantee the alignment of the information system. Based on these works, we add rule #0 and rule #4 to our model, which connect the referred diagrams. Figure 6 illustrates a set of rules to achieve the Enterprise Architecture alignment. The heuristic rules are as follows: #0 The process architecture must support the narrative of the business area; #1 The data architecture must support the narrative of the business area; #2 Each process activity is automated by a single application; #3 Each data set is managed (CUD) by a single application; #4 Each infrastructure is associated with one or more applications. The development of the system begins with the Business Narrative. The application of rules #0 and #1 initiates the two possible views of the system. Then rule #2 is applied, which is responsible for the digital transformation of activities into applications. Rule #3 uses data and applications to create the CRUD matrix. Finally, rule #4, associates a technological infrastructure with one or more applications. Note that rules #0, #1, #2 and #4 guarantee vertical alignment and rule #3 guarantees horizontal alignment. Next, we will detail the procedure for modeling the Enterprise Architecture alignment in three layers.
466
L. Cavique et al.
4.3 Procedure to Check Alignment Procedure 1 checks the alignment in the Enterprise Architecture and numbered in the same way as the layer number: (1) for the 1st layer of Business processes, (2) for the 2nd layer of information system, IS, and (3) for the 3rd layer, of information technology, IT. Procedure 1: (0) define the business narrative. (1) define architecture processes: list activities and actors, activity diagrams. (2) define information system architecture. (2.a) define data architecture: class diagram. (2.b) define CRUD matrix: data versus applications. (2.c) detail CRUD matrix with sequence diagrams. (3) define physical architecture: infrastructure diagram. (0) Define the business narrative. The narrative should refer in a logical way, the actors, activities and data, as well as the way they are articulated in the business. (1) Define the process architecture (or sequence of activities). List the activities, list the actors and fill in the matrix activities versus actors. Develop a UML use-case diagram, or a UML activity diagram where lines of responsibility with the respective system actors are included. Apply rule # 0, where the process architecture must support the narrative of the business area. (2) The Information System Architecture we divided into three subpoints: a) define data architecture: class diagram. b) define CRUD matrix: data versus applications. c) detail CRUD matrix with sequence diagrams. (2.a) Define Data architecture. Create an Entity-Relationship diagram or a UML class diagram for the data architecture that represents the necessary data that correspond to the requirements referred to in the narrative. Apply rule # 1, where the data architecture must support the narrative of the business area. (2.b) Apply rule # 2 where each activity, which is automated, must correspond to a single application. Produce the CRUD matrix representative of the relationship between applications and classes (data). Apply rule # 3, where each data set is managed (CUD) by a single application. (2.c) Detail the CRUD matrix for each application, associating a sequence diagram, which details the message sequences between the classes. (3) Define technological architecture. The UML infrastructure includes technological platforms, servers, client computers, databases, operating systems, etc. Apply rule # 4, associating one or more applications to each infrastructure. Finally, the three lists of the essential elements of the system (actors, activities and data classes) should be reviewed and the heuristic rules for vertical and horizontal alignment checked.
Integration of UML Diagrams from the Perspective of Enterprise Architecture
467
5 Conclusions Since organizations are getting more global, bigger, with more relationships and consequently more complex, an effort to obtain a holistic view is increasingly necessary. Enterprise architecture, composed by three layers, promises companies to deal with digital transformation through a clear presentation and allowing horizontal and vertical alignment of business and IT in a holistic manner. However, to answer these challenges scientific literature proposes solutions with many synonyms depending on the layer, methodology, framework or tool used, that do not allow a broad view of the system. Additionally, new languages evolve into fields more and more specific, increasing the lexical complexity and losing the necessary simplicity of an architecture. Perez-Castillo et al. (2019) concluded that the EA process is costly and subject to errors, which may discourage enterprises from adopting this view. In this work we proposed an approach with the three layers of EA, the four elements of the iron cross with five rules for organization alignment, using UML diagrams, since they have been on the market for decades and there is a wide community of professionals already familiar. BSP (Business Systems Planning) presents four basic elements for Information Systems Planning, coined as the Iron Cross. In this work we chose three basic elements of the information systems and we demonstrate how they are enough to integrate a set of essential UML diagrams. Although CRUD matrix is not a UML tool, it can complete a system view. The proposed model firstly defines a set of UML diagrams for each layer of the Enterprise Architecture, and then heuristic rules are detailed in order to ensure vertical and horizontal alignment. The procedure to check the alignment in the Enterprise Architecture is based on the works of (Pereira and Sousa 2005) and (Vasconcelos 2017), which present a set of three heuristic rules that guarantee the alignment of the information system. In our model, we extended the set of rules in order to connect the UML diagrams and the CRUD matrix. The proposed procedure to check the alignment in the Enterprise Architecture is divided into three steps, for an equal number of organization layers. In a previous step the business narrative is defined. Then business architecture, the information architecture and the technological architecture, apply the (i) use-case/activity diagrams, (ii) class diagrams, CRUD matrix with sequence diagrams to detail the applications, and (iii) infrastructure diagram, respectively. The steps between diagrams are supported by the referred heuristic rules, allowing the alignment of the system. This work is an attempt to reduce and standardize the multiple synonyms in information systems. We also integrate UML diagrams with CRUD in an Enterprise Architecture point of view, allowing a holistic view of the organization. The presented approach is based on several years of teaching this UML approach to undergraduate students (Cavique 2020a, Cavique 2020b). In future work, we plan to establish key performance indicators, KPI, in order to measure the ability to adapt to real-world systems.
468
L. Cavique et al.
References Barros, A., Duddy, K., Lawley, M., Milosevic, Z., Raymond, K., Wood, A.: Processes, roles, and events: UML concepts for enterprise architecture. In: Evans, A., Kent, S., Selic, B. (eds.) UML 2000: The Unified Modeling Language. Lecture Notes in Computer Science, vol. 1939, pp. 62–77. Springer, Heidelberg (2000) Cavique, L.: Modelação de Sistemas de Informação: Elementos essenciais na visão integrada das ferramentas do UML com a matriz CRUD, Recursos Educativos, Universidade Aberta, Portugal (2020a) Cavique, L.: Arquitetura Empresarial: na perspetiva da integração de diagramas UMLRecursos Educativos, Universidade Aberta, Portugal (2020b) Desfray, P., Raymond, G.: Modeling Enterprise Architecture with TOGAF: A Practical Guide Using UML and BPMN. Elsevier Inc., Amsterdam (2014). ISBN: 978-0-12-419984-2 Eppinger, S.D., Browning, T.R.: Design Structure Matrix Methods and Applications. MIT Press, Cambridge (2012) Fowler, M.: UML Distilled: A Brief Guide to the Standard Object Modeling Language, 3rd edn. Addison-Wesley Professional, Boston (2003). ISBN: 978-032-119-368-1 IBM Corporation. Business System Planning Information - System Planning Guide, 2nd edn. International Business Machines Corporation, New York (1978) Lankhorst, M.: Enterprise Architecture at Work: Modelling, Communication and Analysis, 3rd edn. Springer, Heidelberg (2013). ISBN: 978-364-229-650-5 Martin, J.: Managing the Database Environment. Prentice-Hall, Englewood Cliffs (1983). ISBN: 013-550-582-8 Pereira, C.M., Sousa, P.: Enterprise architecture: business and IT alignment. In: Haddad, H., Liebrock, L.M., Omicini, A., Wainwright, R.L. (eds.) SAC, pp. 1344–1345. ACM (2005) Perez-Castillo, R., Ruiz-Gonzalez, F., Genero, M., Piattini, M.: A systematic mapping study on enterprise architecture mining. Enterp. Inf. Syst. 13(5), 675–718 (2019) Rocha, Á., Freixo, J.: Information architecture for quality management support in hospitals. J. Med. Syst. 39, 125 (2015) Silingas, D., Butleris, R.: Towards customizing UML tools for enterprise architecture modeling, In: Nunes, M.B., Isaías, P., Powell, P. (eds.) IADIS International Conference Information Systems (2009). ISBN: 978–972–8924–79–9 Suh, N.P.: Axiomatic Design: Advances and Applications. Oxford University Press, Cambridge (2001). ISBN: 0-19-513466-4 Vasconcelos, A.: Slides Aulas Arquiteturas Tecnológicas Empresariais, DEI, Instituto Superior Técnico, Universidade Lisboa, Portugal (2017)
MongoDB, Couchbase, and CouchDB: A Comparison Pedro Martins1(B) , Francisco Morgado1 , Cristina Wanzeller1 , Filipe S´ a1 , 2 and Maryam Abbasi 1
CISeD - Research Centre in Digital Services, Polytechnic of Viseu, Viseu, Portugal {pedromom,fmorgado,cwanzeller,filipe.sa}@estgv.ipv.pt 2 University of Coimbra, Coimbra, Portugal [email protected]
Abstract. With the rapid increase of cloud computing, big data and real time web applications, there is eminent need for databases that can handle and process large volumes of data, with high-performance and great scaling capabilities. Since its beginning this was the main goal for NoSQL databases. In this paper we will touch bases with three of the most popular NoSQL databases: MongoDB, Couchbase and CouchDB. We will compare them using the OSSpal methodology and the YCSB framework in a quantitative and qualitative matter. Also, we will cover the basics of NoSQL and its types, how it scales. Keywords: NoSQL YCSB
1
· CouchDB · Couchbase · MongoDB · OSSpal ·
Introduction
With the rapid increase of cloud computing, big data and real time web applications, came an underlying need for flexible database structures that differs completely from the relational database world. This approach is called NoSQL. It’s sometimes referenced in the literature as “non-SQL” , “not only SQL” or even “non-relational” . NoSQL databases utilizes a flexible data model, with no predefined schema. This is the complete opposite from a relational database perspective, with its fixed schema and table structure. This means in a relational database world every row in a table is very strictly defined. Thus, one of the advantages of NoSQL is that it supports unstructured data meaning there are no database level restrictions around what kind of data can be stored at any point [13]. Another issue that NoSQL databases can be of assistance is scaling. NoSQL databases scale horizontally opposed to the relational database that scales vertically. We will describe this in more depth later. The OSSpal methodology used in this paper to compare the databases is proven to be adequate for collaborative open source projects like the ones we are covering. It compares the databases in six categories: Overall Quality, Installability, Usability, Robustness, Security and Scalability [23]. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 469–480, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_45
470
P. Martins et al.
This paper is organized as follows. Section 2 describes in more depth the concept of NoSQL and its types (Document store, Key-value store, Graph, etc.). Section 2.1 covers the scaling capabilities of a NoSQL database. Section 3 describes each of the NoSQL databases MongoDB, Couchbase and CouchDB and its core features. Section 4 is dedicated to our analysis of those three databases using the OSSpal methodology as well as performance analysis. Finally, Sect. 5, points out the conclusions of the paper and future work.
2
State of the Art/Related Work(s)
On its core a database is just the store of a set of data related with each other. Through the times languages and applications have been developed to manage this data as easy and efficiently as possible. This development evolved by the need to follow data size, environment changes and performance requirements [16]. In 1974 the first version of SQL databases was released by IBM. A relational database that offers a mathematical way to structure, keep and use data. The introduction of relations brings the benefits of keeping the information as constrained collections, rows of records, used by tables. Has a disadvantage, if a table needs to be changed the entire database will be affected, due to his vertical scaling [18,19]. Carlo Strozzi in 1998 introduced the concept of NoSQL, has a non-relation model that aims to simplify databases, and at the same time, improve performance and flexibility. NoSQL is based on BASE (Basically Available, Soft State and Eventually Consistent) principle that sacrifices consistence to achieve a higher availability. By removing the relational feature, the database expansion and scalability becomes much easier. NoSQL scaling is horizontal, if more performance is needed the system will distribute the database to more machines and distribute the tasks between them. On relational models and vertical scaling, the performance is improved by upgrading the servers and machines. Due to its simple structure the performance on reading and writing processes is very good, together with the high availability, easy scaling and flexibility, NoSQL is very suitable for big data [9,18,19] NoSQL gained strength when Web 2.0 industry increased the need for data storage. Big players like Google and Amazon got involved with the movement, providing a big boost to his development, currently many other vendors provide differentiated NoSQL systems. Most of those systems will be included on one of the following three categories: – Key-values stores: a very simple but powerful and efficient model, that basically consists in an application programming interface (API). A key value data store allows the user to store data in a schema-less manner. It is divided in two parts, a string that represents the key and the actual data as the value typically stored as a BLOB. Some of the most well known systems that use this model are Amazon DynamoDB [15], Voldemort and RIAK. – Document oriented stores: this model of databases store their data in the form of a document indexed to a key. It offers great performance and horizontal scalability options. Documents can be compared to records in relational databases, but they are much more flexible due to the lack of schema.
MongoDB, Couchbase, and CouchDB: A Comparison
471
They are normally stored in standard formats like XML, JSON or BSON. The better known systems using this model are MongoDB, CouchDB and Couchbase. – Column oriented database: in some points similar to a relational database, this model also stores data in tables, but each row contains a dynamic number of attributes. Using nodes to distribute both rows and columns this model provides a better availability and scalability when compared to relational databases. Some examples of this system are Google Big-table and Cassandra. – Other non-relational database models like Graph databases or Object oriented databases are also considered part of the NoSQL category and don’t fit under the models above. Unlike the other models, data querying on these models involve complex object behaviour rather than simple key lookups. Being a bit more complex they aren’t as popular among NoSQL systems [19,22]. 2.1
Scaling Capabilities
Since NoSQL databases deal with big volumes of data and need to have high availability they have to implement forms of scalability. To do this, many databases implement some form of sharding and replication. Sharding is a way of partitioning data from one database in multiple servers, sometimes all over the world, so that it is available to as many users as possible. Once data gets too big on one shard the system can have auto-sharding so that it partitions the data again on other servers. This is a type of horizontal scaling where data from one database is distributed across different machines [8]. Another form of horizontal scaling that NoSQL databases implement is replication, where the whole database is copied to another server. This will create replicas that then will need to perform the same operations that the original database suffers so that the data keeps consistent. This will improve the availability of data to the users and also the copies can replace any other copy when a server failure occurs or data is lost [5]. Replication and sharding can, and often do, work together. With this, there could be a case where two shards with the exact same data exist. This gives the database more computing power and more places where data can be accessed and processed. Scaling then is a really important part of creating an efficient and highly available NoSQL deployment.
3
MongoDB, Couchbase and CouchDB
MongoDB is a free and open source NoSQL document-oriented database developed by MongoDB inc. It is a highly scalable operational database available onpremises and as a cloud service named MongoDB Atlas. It was initially released on February of 2009 [21]. Document oriented databases is one of many NoSQL designs that can be implemented and it is the main form of design used in MongoDB. The data is stored on JSON-like name-value documents that then are stored on collections. MongoDB doesn’t actually use JSON, it uses BSON an extension to JSON
472
P. Martins et al.
created for this project that also stores the type of data, int, string, float, date, and also the document size. By default, MongoDB uses dynamics schema, which means documents in a specific collection don’t need to have the same set of fields and data types. However it is possible to implement JSON schema validation. MongoDB also does not implement a SQL like query language [11]. To improve database performance, MongoDB uses MapReduce [14], Indexes and Server-Side scripts, it is also ACID compliant. MapReduce is a command that MongoDB makes available to condense large volume of data into useful aggregated results. Indexes are special data structures that condense a small portion of a collections data to help improve query performance. It implements Server-Side scripting through JavaScript execution. MongoDB clusters are formed by a primary node, secondary nodes and an arbitrary node, has shown on Fig. 1. The primary node receives all write operations. To provide redundancy and high data availability MongoDB implements Replication on clusters. For this purpose, the secondary nodes will have a replica of the data on the primary node and will also apply the same write operations of the primary node. The arbiter node will not have any data, it will be used to select a new primary node in case of failure [6].
Fig. 1. MongoDB clusters
MongoDB implements auto-sharding, an automatic and built in way of creating shards which makes it simpler for developers that don’t need to build sharding logic. Sharded clusters in MongoDB have three components: mongos, shard and config server. The shards will have a subset of the sharded data, mongos are query routers and config servers store metadata and configuration settings. With this, it’s no extra software is needed to apply sharding to a MongoDB solution and the implementation is a lot simpler [6]. At a minimum, a sharding configuration must have what is shown on Fig. 2.
Fig. 2. MongoDB sharding configuration
MongoDB, Couchbase, and CouchDB: A Comparison
3.1
473
Couchbase
Couchbase is an open source software to manage NoSQL, document oriented databases. Its build on an architecture that supports flexible JSON models, easy to scale, high performance, mobile synchronization, and elevated security. Developed by CouchBase Inc. it was initially released in August 2010. It is currently a base for worldwide companies from the most varied sectors. Being a software in constant evolution, its architecture is currently built over the following components: Cluster Manger, Data Service, Index Service, Query Service, Search Service, Analytics Service and Eventing Service as represented on Fig. 3. Its implementation is available in four different versions (Server, Cloud, Autonomous Operator with Kubernetes, Lite e Sync Gateway), it also provides tools and connectors to facilitate management, development, and integration with other softwares. CouchBase is available to download on https:// www.couchbase.com/downloads [1].
Fig. 3. Couchbase architecture [2]
The Fig. 4 presents a home dashboard from CouchBase Community Edition 6.5.1, with is simple and intuitive interface, it provides telemetry from a preselected server. This telemetry allows an easy visualization of the server and database performance, with dynamic and interactive charts updated with RAM usage in real time, read, write, search time, and other useful information for a database manager. It is also possible to identify other characteristic features of this software, like buckets, documents, and indexes. Buckets are the elements that connect to applications and store their information in JSON documents [17]. The documents can be compared to records, they store the information in JSON format. Indexes identify the data inside the buckets, they are a key element to run efficient queries. Couchbase provides a big range of indexes and management options. On the same figure, it is also possible to identify shortcuts to management features like logs, security, server management and query editor [24].
474
P. Martins et al.
Fig. 4. Couchbase dashboard
3.2
CouchDB
CouchDB was initially launched in 2005 and it is a Database Management System Software (DBMS) non-relational, document-oriented developed by Damien Katz using the Erlang programming language. As of 2008 CouchDB is part of the Apache Foundation. The DBMS administration is performed through a web interface, Fig. 5, provided by the web server embedded in CouchDB, called Fauxton. Database queries are processed through HTTP requests handled by the RESTful JSON API of CouchDB. This RESTful API was conceived with the use of industry standards; thus, it is universally compatible with any programming language without the need for additional customization [20].
Fig. 5. CouchDB fauxton administration interface
CouchDB is compatible with multiple Operating Systems, such as Windows, macOS, Unix (Debian, Ubuntu, RHEL, CentOS). A CouchDB database is composed by at least one or more documents identified by a unique ID. Documents are JSON objects. To illustrate, a CouchDB document is the same as a row in relational database table, Fig. 6.
Fig. 6. CouchDB document example
MongoDB, Couchbase, and CouchDB: A Comparison
475
Create, Read, Update, Delete (CRUD) operations are performed through HTTP requests, processed by the RESTful API and the response is in the JSON format. In Fig. 7 is an example of how to create a document in the database. In this example we used the HTTP PUT method to insert the document.
Fig. 7. CouchDB insert operation
Those are the requirements to insert a document: CouchDB address (http://127.0.0.1:5984), database name (teste), document unique ID (“1”) and the array JSON with the data, Fig. 8. In order to retrieve data, we should use the HTTP GET method. In this example we did directly from the browser. Type the CouchDB address, database name and document unique ID.
Fig. 8. Document visualization
4
Results and Analysis
After describing the three systems on the previous sections, on this division we will evaluate and compare MongoDB, CouchDB and Couchbase. To classify the systems, we used the OSSpal methodology, this is a project whose objective is to help organizations to find open source software of the best quality. The OSSpal methodology is widely accepted as one of the to combine quantitative and qualitative measures, and by consequence better classify open source software [10]. We defined six criteria based on OSSpal: Installability, Usability, Robustness, Security, Scalability and Overall Quality. For each criterion we defined a weight according to our considerations and research, the most relevant categories have bigger weights. The Overall Quality has the biggest weight, representing thirty percent of the total, we consider this category a global feeling for the system, an aggregation of the other categories and for this the most influential for the final evaluation. Usability with twenty percent of the weigh is a key factor for the success of a system and for this we defined a bigger weight than Installability and Security. The categories with smaller weights are Robustness and Scalability, they are not less important than the previous ones, we just consider them less relevant to the system evaluation, Table 1.
476
P. Martins et al. Table 1. Categories weight
Categories
Weights
Installability
15%
Usability
20%
Robustness
10%
Security
15%
Scalability
10%
Overall quality 30%
To classify each system on each category, we followed again the OSSpal methodology, we assigned a value from zero to five to each category, beginning with zero the lowest score and five the highest, Table 2. The evaluation for each system was based on our own experience working with each system, the information we researched, rankings, reviews, and analyses on G2 [4], DB-Engines [3] and [12]. To compile the information from Table 2 and achieve an end value that represents the classification of each system we used a mathematical method. Each category value is multiplied by his weight and all the results are added to achieve a final system evaluation. Couchbase: 3.5 ∗ 0.15 + 4.5 ∗ 0.2 + 4.5 ∗ 0.1 + 3 ∗ 0.15 + 3.5 ∗ 0.1 + 3.5 ∗ 0.3 = 3.725 CouchDB: 3 ∗ 0.15 + 4 ∗ 0.2 + 4 ∗ 0.1 + 3 ∗ 0.15 + 3.5 ∗ 0.1 + 3.5 ∗ 0.3 = 3.5 MongoDB: 4.5 ∗ 0.15 + 4.5 ∗ 0.2 + 4.5 ∗ 0.1 + 4 ∗ 0.15 + 3.5 ∗ 0.1 + 4.5 ∗ 0.3 = 4.325 Table 2. Application evacuations through OSSpal method
Categories
MongoDB Couchbase CouchDB
Installability
4,5
3,5
3
Usability
4,5
4,5
4
Robustness
4,5
4,5
4
Security
4
3
3
Scalability
3,5
3,5
3,5
Overall quality 4,5
3,5
3,5
MongoDB, Couchbase, and CouchDB: A Comparison
477
Following the benchmarks, our evaluation and research also classify MongoDB as the best system, from the three evaluated, with a final classification of 4.325, compared with 3.725 on Couchbase and 3.5 of CouchDB. 4.1
YCSB
To test and evaluate the performance of all 3 NoSQL databases the Yahoo Cloud Serving Benchmark (YCSB) tool was used. YCSB is an open source benchmark program initially developed by Yahoo to evaluate the performance of database management systems, primarily NoSQL databases. For this, the program offers many options to test and compare performance metrics, it gives a set of workloads to use, each of them with various read and write percentages and record or operation counts. This can be downloaded here https://github. com/brianfrankcooper/YCSB. To use this tool, the user will first need to execute the “load” command to create as many records as needed inside the database. Then a “run” command can be made to perform all read and write operations needed on the records previously created. By default, only one client-side thread is used to perform these operations but this can be changed on the console command. After every execution the program will output a report with performance statistics [7]. To make the performance evaluation on MongoDB, Couchbase and CouchDB an on-premise version of all these databases had to be installed on a local machine. The following are the version number of the databases installed: – MongoDB: v4.2.8 – Couchbase: v6.5.1 – CouchDB: v3.1 There are 2 scenarios being tested the insert records and read/write operations on a 50% split. For each scenario the test was done with 1000, 15000 and 50000 records and operations respectively. Only one client-side thread was used to make these performance tests. The PC used for the MongoDB and Couchbase tests had 16 GB of memory and a CPU Ryzen 3700x with 8 cores, 16 threads and base speed 3.6 GHz with Windows 10. Since YCSB does not have a direct integration with YCSB and all alternatives only worked on Linux it was not possible to test it on the same environment. Table 3, shows the results of runtime in milliseconds and Table 4 shows the throughput in operations per second during the inserts:
478
P. Martins et al. Table 3. Runtime of inserts
Insert runtime Rows Database
1000
15000
50000
MongoDB
1245 ms
4991 ms
12528 ms
CouchDB
44534 ms 549579 ms 1687161 ms
Couchbase
1614 ms
6289 ms
15189 ms
Table 4. Throughput of inserts
Insert throughput Rows Database
1000
15000
50000
MongoDB
803.2128514 ops/s 3005.409738 ops/s 3991.06 ops/s
CouchDB
22.45475367 ops/s 27.2936193 ops/s
Couchbase
619.5786865 ops/s 2385.116871 ops/s 3291.8559 ops/s
29.635583 ops/s
In Table 5 are the results of runtime in milliseconds and Table 6 shows the throughput in operations per second on read and write with 50% split: Table 5. Runtime of reads and writes
R/W runtime Operations Database
4.2
1000
15000
50000
MongoDB
1380 ms
5755 ms
14466 ms
CouchDB
25369 ms 329459 ms 1116037 ms
Couchbase
1638 ms
6105 ms
15074 ms
Results
Regarding Couchbase, MongoDB and Couchbase it can be seen that MongoDB always has better performance, lower run-times and higher throughput on insert and read/write. It can also be observed that the gap between MongoDB and Couchbase on the insert scenario is a lot higher than comparing with the read/write scenario. This means that MongoDB has much better performance than Couchbase while inserting a big quantity of records. On the read/write scenario, while MongoDB still getting consistently better values than Couchbase, the values are closer so the performance is as well.
MongoDB, Couchbase, and CouchDB: A Comparison
479
Table 6. Throughput of reads and writes
R/W Throughput Operations
5
Database
1000
15000
50000
MongoDB
724.6376812 ops/s 2606.429192 ops/s 3456.3805 ops/s
CouchDB
39.41818755 ops/s 45.52918573 ops/s 44.801382 ops/s
Couchbase
610.5006105 ops/s 2457.002457 ops/s 3316.9696 ops/s
Conclusions and Future Work
Per our analysis, both in a qualitative and quantitative matter, through the OSSpal methodology and the YCSB benchmark, we found MongoDB as the leading open-source NoSQL Database Management System (DBMS). MongoDB scored 4.325 on our scale, Couchbase and CouchDB, 3.725 and 3.5 respectively and on the performance tests MongoDB got better results than Couchbase. This does not mean companies or developer should always choose MongoDB over the others. As mentioned before, each NoSQL DBMS has it strengths and is more suitable for a different type of application. This study provides developers with more information to help them choose the most adequate NoSQL DBMS. As future work we recommend to do benchmarks on more NoSQL databases, as well as to make them all on the same environment. It would also be important to test different types of workloads as well as a different number of threads to see their impact. Acknowledgement. “This work is funded by National Funds through the FCT - Foundation for Science and Technology, IP, within the scope of the project Ref UIDB/05583/2020. Furthermore, we would like to thank the Research Centre in Digital Services (CISeD), the Polytechnic of Viseu for their support.”
References 1. Couchbase. https://www.couchbase.com, Accessed 25 Aug 2020 2. Couchbase under the hood. https://resources.couchbase.com/c/server-arcoverview?x=V3nd e, Accessed 25 Aug 2020 3. Couchbase vs. couchdb vs. mongodb comparison. https://db-engines.com/en/ system/CouchDB%253BCouchbase%253BMongoDB, Accessed 25 Aug 2020 4. Couchdb, couchbase server, and mongodb — g2. https://www.g2.com/compare/ couchdb-vs-couchbase-couchbase-server-vs-mongodb, Accessed 25 Aug 2020 5. Mongodb replication. https://docs.mongodb.com/manual/replication, Accessed 25 Aug 2020 6. Mongodb sharding docs. https://docs.mongodb.com/manual/core/shardedcluster-components/, Accessed 20 July 2020 7. Running-a-workload. https://github.com/brianfrankcooper/YCSB/wiki/ Running-a-Workload, Accessed 25 Aug 2020 8. Sharding. https://docs.mongodb.com/manual/sharding, Accessed 25 Aug 2020
480
P. Martins et al.
9. Abramova, V., Bernardino, J., Furtado, P.: Experimental evaluation of NoSQL databases. Int. J. Database Manag. Syst. 6, 10 (2014) 10. Balaguer, F., Di Cosmo, R., Garrido, A., Kon, F., Robles, G., Zacchiroli, S.: Open source systems: towards robust practices. In: 13th IFIP WG 2.13 International Conference, OSS, Buenos Aires, Argentina, 22–23 May 2017, Proceedings, vol. 01, no. 2017 (2017) 11. Banker, K.: MongoDB in action (2011) 12. Cal¸cada, A., Bernardino, J.: Evaluation of couchbase, couchdb and mongodb using osspal. In: KDIR, pp. 427–433 (2019) 13. Cutler, J., Dickenson, M.: NoSQL Databases, pp. 117–126 (2020) 14. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008) 15. DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: amazon’s highly available key-value store. ACM SIGOPS Oper. Syst. Rev. 41, 205–220 (2007) 16. Hick, J.-M., Hainaut, J.-L.: Strategy for database application evolution: the DBMAIN approach, pp. 291–306 (2003) 17. Hubail, M., Alsuliman, A., Blow, M., Carey, M., Lychagin, D., Maxon, I., Westmann, T.: Couchbase analytics: Noetl for scalable nosql data analysis. In: Proceedings of the VLDB Endowment, vol. 12, pp. 2275–2286 (2019) 18. Louren¸co, J., Abramova, V., Vieira, M., Cabral, B., Bernardino, J.: Nosql databases: a software engineering perspective. Adv. Intell. Syst. Comput. 353, 741–750 (2015) 19. Mahruqi, R.S.A.: Migrating web applications from SQL to nosql databases (2010). https://qspace.library.queensu.ca/handle/1974/27587, Accessed 25 Aug 2020 20. Manyam, G., Payton, M., Roth, J., Abruzzo, L., Coombes, K.: Relax with couchdb - into the non-relational DBMS era of bioinformatics. Genomics 100, 1–7 (2012) 21. Membrey, P., Plugge, E., Hawkins, D.: The Definitive Guide to MongoDB: The NoSQL Database for Cloud and Desktop Computing. Apress (2011). GoogleBooks-ID: 4ogx82US4fwC 22. Nayak, A., Poriya, A., Poojary, D.: Article: type of nosql databases and its comparison with relational databases. Int. J. Appl. Inf. Syst. 5, 16–19 (2013) 23. Wasserman, A., Guo, X., McMillian, B., Qian, K., Wei, M.-Y., Xu, Q.: Osspal: finding and evaluating open source software, pp. 193–203 (2017) 24. Zablocki, J.: Couchbase Essentials. Packt Publishing Ltd., Birmingham (2015). Google-Books-ID: EVvTBgAAQBAJ
Comparing Oracle and PostgreSQL, Performance and Optimization Pedro Martins1(B) , Paulo Tom´e1 , Cristina Wanzeller1 , Filipe S´ a1 , 2 and Maryam Abbasi 1
CISeD - Research Centre in Digital Services, Polytechnic of Viseu, Viseu, Portugal {pedromom,ptome,cwanzeller,filipe.sa}@estgv.ipv.pt 2 CISUC - Centre for Informatics and Systems of the University of Coimbra, Coimbra, Portugal [email protected]
Abstract. Relational databases are getting bigger and more complex. Also, current Database Management Systems (DBMSs) need to respond efficiently to operations on their data. In this context, database optimization is evident as a process of refining database systems, aiming to improve their throughput and performance. This paper evaluates and compares the performance of Oracle and PostgreSQL database systems with the TPC-H benchmark, following a strategy of adding column-based indexes to optimize query execution. Ten TPC-H queries are performed on tables without any restrictions, with primary and foreign keys and with index constraints. The performance in each set of executions is analyzed. The results allow inferring a positive impact when using constraints with a significant speedup as well as better throughput. Oracle has shown stability and robustness for queries, with best results in scenarios with poor performance conditions. However, PostgreSQL showed shorter execution times after the optimizations made and proved to be more sensitive. Global performance results show that Oracle can improve 7% performance with indexes and PostgreSQL 91%. When comparing the results of Oracle with PostgreSQL, no indexes, Oracle/PostgreSQL is 64% faster, and with indexes, PostgreSQL/Oracle is 75% faster. Keywords: Database optimization DSS · TPC-H
1
· DBMS · Oracle · PostgreSQL ·
Introduction
Currently, database information systems have spread globally and represented an essential class in various domains of the business world and society that require analysis of large amounts of data. Databases provide information stored with a hierarchical and related structure, which makes possible to extract the content and arrange it easily. The relational DBMS is the most widely used DBMS model and is based on the normalization of data in the rows and columns of the tables and c The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 ´ Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 481–490, 2021. A. https://doi.org/10.1007/978-3-030-72651-5_46
482
P. Martins et al.
manipulated using Structured Query Language (SQL). However, the efficiency of an SQL query is subject to different database functions and in particular to its computation time. Database optimization is the strategy of reducing the response time of the database system, but each DBMS has unique ways to be tuned. Systems Decision Support (DSS) enable companies to achieve profitable results from business information [2]. They involve workloads with the ability to process complex ad-hoc queries on large volumes of data. The focus is on reading as opposed to the transactional workload that focuses on modifying data, thus requiring a different approach. The TPC-H benchmark [7] illustrates DSSs that examine large volumes of data, execute queries with a high degree of complexity in order to stress the database server, and give answers to critical business questions. The time needed to execute ad-hoc DSS queries is related to the database design, size and its tuning [19,26]. This study aims to analyze the performance of the TPC-H benchmark in Oracle and PostgreSQL databases in three scenarios: tables with none constraint, with primary and foreign keys, and with index constraints. Ten TPC-H queries are analyzed, and indexes are created to improve the performance and throughput of the databases, seeking to reduce the queries execution time and cost and draw conclusions about the impact of the indexes. This document is organized in seven sections. Section 2, presents the stateof-the-art review. Section 3 presents an overview of DBMS. Section 4 expose the experimental methodology and Sect. 5 discusses the experimental setup. Section 6, shows the obtained results. Finally, in Sect. 7 conclusions are drawn, followed by the introduction of future work guidelines.
2
Related Work
Since the1980 s, the relational model has been the most popular database model [9]. In relational databases, the primary data structure is a table, with records consisting of attributes. Most current databases store records contiguously and are hence called row-oriented databases. Commonly, there are two types of queries execution: Online Transaction Processing (OLTP) and DSS [5,26]. OLTP applications most often use row-oriented relational databases and a significant portion of queries involves short-running update queries to update only a small number of rows [22]. In the opposite, On-Line Analytical Processing (OLAP), DSS and data mining among other query intensive applications, involve complex queries on extensive databases and summarize data in multiple dimensional structures, being critical the query optimization achieve short response times. However, systems optimized for OLTP vs DSS and index search workloads may lead to diverging designs [22]. The time needed to execute ad-hoc DSS queries is related to the database design and size, the Relational Database Management System (RDBMS) and its tuning [26]. Database systems have a large number of configuration parameters that control memory distribution, I/O optimization, costing of query plans, parallelism and other behaviours [10]. Query optimization is the overall process of choosing the most efficient means of executing a SQL statement. The database optimizes each SQL statement based
Comparing Oracle and PostgreSQL, Performance and Optimization
483
on statistics collected about the accessed data [15]. There are some query optimization techniques such as using the names of the actual columns in SELECT statement instead of the ‘*’, using the HAVING clause to filter the rows after all rows are selected, minimizing the number of subquery block in one’s query, using the operator ‘EXISTS’, ‘IN’. Table joins, using UNION ALL in place of UNION, among others [15,18,20]. Inefficient SQL queries may take hours or even days to complete or may fail to complete at all. Query optimization usually is the most cost-effective way of improving the performance of an existing application. DBMS configuration tuning is also essential for any data-intensive application effort. Other measures such as changing database parameters or altering disk layouts will usually be ineffective unless the application’s SQL is properly tuned [13]. There may be some divergences between DBMS due to non-standardization that can lead to different results and unwanted side effects [27]. Index tuning, when done correctly, is essential to high performance, allowing reducing the execution cost of queries [20] and can be the solution to many performance problems [15]. An index for a table is a data organization that enables specific queries to access one or more records of that table fast [5]. However, the usefulness of an index depends on how queries use the index. Unused indexes can entail overhead, and wrong types of index end up harming performance [25]. The DBMSs offer other options to aid performance such as partitioning large tables, using materialized views, storing plan outlines and many others [17]. Over the last decades, research literature proposes and investigates a wide variety of query optimization algorithms and techniques in relational database systems [4,14,16,19,24,28]. However, SQL Tuning activity is complicated, timeconsuming and requires much expertise. Leading database vendors have overcome these challenges and provide a way to perform automatic performance diagnosis and tuning, integrating advisory systems for tuning databases [22]. These advisors analyze a query statement and provide suggestions to improve the performance of the system to the given query. The ability to self-tune is a critical aspect of building a self-managed database [3,8]. A database can have two tuning goals, very different and sometimes mutually exclusive [1]: maximize throughput (minimizing hardware resources usage) and minimize response time (fast data response, regardless of hardware resources usage).
3
Database Management System
Oracle Database, developed and marketed by Oracle Corporation, and PostgreSQL, an open-source RDBMS, are two of the most common RDBMS today. TPC-H workloads run complex queries scanning large volumes of data. They, therefore, require the use of database features such as parallel query and inmemory column stores to maximize performance. PostgreSQL has improved over time and supports row-oriented parallel query since version 9.6, offering significant performance improvement over single-threaded queries and the ability to
484
P. Martins et al.
use many CPUs and finish faster. Regarding Oracle, only the Enterprise Editions support these features. Oracle and PostgreSQL provide several index types such as B-Tree, Bitmap and bitmap join, Hash, Function-based, among others. The B-Tree is the standard in both and can work with all data types and be used for equality and range queries efficiently [11]. An index gives the ability to retrieve a small set of randomly distributed rows from a table, allowing to reduce disk I/O.
4
Experimental Methodology
This study analyses the performance of Oracle and PostgreSQL and how much constraints impact it with the TPC-H workload. Before the experiment was performed warm-up runs, such as the execution of SELECT COUNT(*) statement, in order to collect statistical information about the tables, its records and data access patterns, and to preload data that is on disk into the database system buffer cache, which reduces disk I/O and speeds up the database [18]. For each database, we executed ten TPC-H queries on tables without any restrictions, with primary and foreign keys and with index constraints. We designed our experiment to create indexes on foreign keys columns, as allowed by documentation [6], and columns related to joining operator of the queries, for example in column “S SUPPKEY” for Query 5, since joins are the most costly relational operators. Also, were created indexes on columns in WHERE clauses [12,20], as examples the column “L SHIPDATE” for Query 1 and the column “R NAME” for Query 5. In the latter, and according to the generated query that looks up for filtering the region America, with an index in that column the database follows the index data structure until found the “AMERICA” entry, which is much less computationally expensive than a full table scan. The standard index type of both DBMS was used. B-Tree indexes work very well for Relational Database systems and can eliminate full table scans [20]. Five runs were performed per query. The average and standard deviation of the execution time were calculated, discarding maximum and minimum values. The queries execution plans were observed, resorting to the query statement prefixes EXPLAIN PLAN for Oracle and EXPLAIN (and EXPLAIN ANALYSE) for PostgreSQL.
5
Experimental Setup
This section briefs the specifications of the used hardware and software platforms that describes the TPC-H benchmark and its workload and discusses some details about data preparation. 5.1
Hardware and Software Platforms
As part of this work, we ran our experiments using Oracle Database 18c Standard Edition and PostgreSQL Database 12.3 on a 64-bit Windows 10 platform with
Comparing Oracle and PostgreSQL, Performance and Optimization
485
a dual-core Intel Core i5-430M processor with each core running at 2.2 GHz, a Solid-State Drive memory disk and 3GB RAM. 5.2
Database Workload
The TPC-H benchmark [7] is a DSS benchmark that models the activity of a product supplier company [6]. TPC-H illustrates DSSs that analyze large volumes of data and execute queries with a high degree of complexity in order to stress the database server. It consists of eight tables, namely SUPPLIER, PART, PARTSUPP, LINEITEM, CUSTOMER, ORDER, NATION and REGION. The REGION and NATION tables are of constant sizes and the others scale linearly as described by the Scaling Factor (SF), except the LINEITEM table whose size is prescribed by the benchmark. The TPC-H download includes scripts to create the database and primary and foreign keys constraints, 22 queries and the DBGEN and QGEN tools. The DBGEN is for generating the data, and the QGEN is for generating queries. These tools allow creating databases and queries for various SFs. The data is randomly generated for the tables and so do the parameter values for the 22 queries. DBGEN tool generates TBL files. These files have a “|” character at the end of each line and to make them compatible with PostgreSQL it has been removed and saved in CSV files. As the LINEITEM table is very large, it was necessary to split it into smaller chunks to import into PostgreSQL. The generated queries by QGEN are not fully compatible with PostgreSQL and “where rownum