Volume 8, Number 4, October 2022 IEEE Systems, Man and Cybernetics Magazine

Citation preview

CALL FOR PAPERS IEEE Systems, Man, and Cybernetics Magazine Special Issue on Human-centered Collaborative Systems I.

AIM AND SCOPE

Recent years have witnessed an increasing shift in interest from machine-machine collaborative systems toward humancentered collaborative systems (HuCS). The HuCS aims at offering user-friendly environments and requires less learning, but at the same time provides high throughput and interaction capabilities, which apparently is an interdisciplinary edge-cutting research field. It has gained immense popularity in many areas over the past decades, e.g., pervasive computing, context-aware computing, affective computing, artificial intelligence, social network dynamics, social intelligence and cognition, and social systems design. Tremendous applications of the HuCS today have spread into the industry, healthcare, and even entertainment domains. In this context, user-friendliness and human-centeredness are two vital parts of constructing a successful HuCS. To facilitate the development of human-centered collaborative systems, this special issue aims to provide an interdisciplinary platform to integrate current research achievements. This special issue is to provide a summary of recent research that advances human-centric software engineering and to also serve as a collection of current state-of-the-art approaches and technologies. While we encourage authors of top-quality papers accepted by the 2022 International Workshop on Human-Centric Software Engineering and Cyber Security (HCSE&CS 2022, co-located with ASE2022) to submit their extended articles, this special issue also has an open call to the wider research community.

II.

TOPICS

The topics relevant to the special issue include (but are not limited to) the following topics:         

Social network, social systems design and architectures, and socio-cultural modeling and representation Human-machine/AI collaboration systems, and human-vehicle collaboration systems Human-centered collaborative intelligence, crowd/ collective intelligence Pervasive/ubiquitous computing, context-aware computing, evolutionary computing, and affective computing Human factors in the Internet of Things (IoT)/ AIoT Role-based collaboration methodology, and agentbased collaboration methodology Human-centered recommender systems Secure & privacy-preserving HuCS Emerging AI approaches and their applications in the

Digital Object Identifier 10.1109/MSMC.2022.3211365

    

HuCS Human-centered design for education, transportation, industry, healthcare, finance, etc. Incorporating human factors into requirements and design e.g., emotions, bias, personality, and culture Tools and models for capturing and interpreting user behaviors in collaborative systems Impact of human factors on software development processes and software teams Other related topics

III.

SUBMISSIONS

Manuscripts should be formatted according to the IEEE SMC Magazine guidelines for authors (https://www.ieeesmc.org/publications/smc-magazine), and submitted through the IEEE’s Manuscript Central (https://mc.manuscriptcentral.com/smcmag). Please select “Special Issue” under Manuscript Category of your submission.

IV.

IMPORTANT DATES

Manuscript submissions due: 30 November 2022 First round of reviews completed: 31 December 2022 Revised manuscripts due: 31 January 2023 Second round of reviews completed: 31 March 2023 Final manuscripts due: 30 April 2023

V.

GUEST EDITORS

Dongning Liu Professor Guangdong University of Technology, China Email: [email protected] Xiao Liu Professor Deakin University, Australia Email: [email protected] Tun Lu Professor Fudan University, China Email: [email protected] Wei-Neng Chen, Professor South China University of Technology, China Email: [email protected] Dakuo Wang, Adjunct Associate Professor Northeastern University, U.S. Email: [email protected]

Smart Solutions for Technology

www.ieeesmc.org

Volume 8, Number 4 • October 2022

Features 6 On Blockchain

Design Principle, Building Blocks, Core Innovations, and Misconceptions By Wenbing Zhao

28

15 A Transfer Learning-Based Method to Detect Insulator Faults of High-Voltage Transmission Lines via Aerial Images Distinguishing Intact and Broken Insulator Images By Fatemeh Mohammadi Shakiba, S. Mohsen Azizi, and Mengchu Zhou

26 Prediction of Flooding Due to Heavy Rainfall in India Using Machine Learning Algorithms Providing Advanced Warning By R. Balamurugan, Kshitiz Choudhary, and S.P. Raja

Special Section on Explainable Artificial Intelligence for the Social Internet of Things 36 Blockchain Technology in Agriculture for Indian Farmers

50

A Systematic Literature Review, Challenges, and Solutions By Urvashi Sugandh, Swati Nigam, and Manju Khari 

44 Explainable Artificial-Intelligence-Based Privacy Preservation Approach for Information Dissemination on Social Networks

ABOUT THE COVER Blockchain technology’s features include decentralized consensus, privacy, security, and data immutability.

An Incremental Technique By Shoayee Dlaim Alotaibi and Kusum Yadav 

©SHUTTERSTOCK.COM/IURII MOTOV

48 An Explainable Artificial-Intelligence-Based CNN Model for Knowledge Extraction From the Social Internet of Things Proposing a New Model By Lulwah M. Alkwai

Mission Statement

Departments & Columns

The mission of the IEEE Systems, Man, and Cybernetics Society is to serve the interests of its members and the community at large by promoting the theory, practice, and interdisciplinary aspects of systems science and engineering, human–machine systems, and cybernetics. It is accomplished through conferences, publications, and other activities that contribute to the professional needs of its members. Digital Object Identifier 10.1109/MSMC.2022.3207038



O c tob e r 2022



4 Editorial 34 Special Section Editorial 54 Meet Our Volunteers 57 Conference Reports

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

1

CALL FOR PAPERS IEEE Systems, Man, and Cybernetics Magazine Special Issue on Cooperative design, visualization, engineering, and applications

I.

AIM AND SCOPE

Due to the fast development of the information technology applied to design visualization, engineering and many other fields, we can see a very strong demand in supporting the interactions and cooperation. The field of technological support to the cooperation within these fields has never been so active. This is due to the highspeed internet, availability of low-cost data storage and the advance of many emerging technologies such as web applications, social media, big data, block chain, crowd sourcing, internet of things (IOT), nano-networks etc. This special issue will focus on the technical advance and achievement of the support to the cooperation and interaction, both theoretical and practical.

II.

TOPICS

The topics relevant to the special issue include (but are not limited to) the following topics: Multi-user, multi-location, multi-modal cooperative design including system architecture, user interface, prototyping, design for total life cycle support etc.  Cooperative visualization for multiple users, multiple locations, multi-modal displays, web applications, textual and multimedia 2D, 3D desktop interfaces, 3D virtual world environments, and multiple user embedded systems etc.  Collaborative process planning, modeling, control, prototyping, manufacturing, and engineering. Integration and interoperability in engineering. Typical areas can be building construction, aerospace, mechanical engineering, software engineering etc.  Other cooperative applications that network, multiple units are involved such as cooperative learning, decision making, gaming, health caring, 

Digital Object Identifier 10.1109/MSMC.2022.3211386

2

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE October 2022

robotics ...  Basic theories, methods and technologies that support cooperation: big data, blockchain, Internet of Things (IOT), nano-networks, social media, crowd technology, knowledge management, ontology etc. applied to cooperative applications.

III.SUBMISSIONS Manuscripts should be formatted according to the IEEE SMC Magazine guidelines for authors (https://www.ieeesmc.org/publications/smcmagazine), and submitted through the IEEE’s Manuscript Central (https://mc.manuscriptcentral.com/smcmag). Please select “Special Issue” under Manuscript Category of your submission.

IV. IMPORTANT DATES Manuscript submissions due: OCT 30, 2022 First round of reviews completed: DEC 15, 2022 Revised manuscripts due: Jan 31, 2023 Second round of reviews completed: March 15, 2023 Final manuscripts due: April 30, 2023

V. GUEST EDITORS Professor Yuhua Luo Email: [email protected], [email protected] University of Balearic Islands, Spain Dr. Sebastia Galmer E-mail: [email protected] University of Balearic Islands, Spain

IEEE Systems, Man, and Cybernetics Magazine EDITOR-IN-CHIEF Haibin Zhu Nipissing University, North Bay, Ontario, Canada [email protected]

ASSOCIATE EDITORS

Okyay Kaynak, Vice President, Organization and Planning Shun-Feng Su, Vice President, Publications Ying (Gina) Tang, Vice President, Finance Vladik Kreinovich, Treasurer

Mali Abdollahian, Australia Mohammad Abdullah-Al-Wadud, Saudi Arabia Choon Ki Ahn, Korea Bernadetta Kwintiana Ane, India Krishna Busawon, UK György EIgner, Hungary Liping Fang, Canada Giancarlo Fortino, Italy Hossam Gaber, Canada Aurona Gerber, South Africa Jason Gu, Canada Abdollah Homaifar, USA Okyay Kaynak, Turkey Kevin Kelly, Ireland Kazuo Kiguchi, Japan Abbas Khosravi, Australia Vladik Kreinovich, USA Wei Lei, China Kovács Levente, Hungary Xiaoou Li, Mexico Darius Nahavandi, Australia Chris Nemeth, USA Vinod Prasad, Singapore Hong Qiao, China Ferat Sahin, USA Mehrdad Saif, Canada Bahram Shafai, USA Weiming Shen, Canada Liqiong Tang, New Zealand Ying Tan, Australia Yingxu Wang, Canada Margot Weijnen, Netherlands Peter Whitehead, USA Zhao Xingming, China Laurence T. Yang, Canada Qiangfu Zhao, Japan

Tom Gedeon, Secretary

SOCIETY BOARD OF GOVERNORS

Publications Ethics Committee Shun-Feng Su, Chair Imre Rudas Edward Tunstel Vladik Kreinovich Peng Shi Fei-Yue Wang Robert Kozma Ljiljana Trajkovic Haibin Zhu

Executive Committee Sam Kwong, President Imre Rudas, Jr. Past President Edward Tunstel, Sr. Past President Enrique Herrera Viedma, Vice President, Cybernetics Saeid Nahavandi, Vice President, Human–Machine Systems Thomas I. Strasser, Vice President, Systems Science and Engineering Yo-Ping Huang, Vice President, Conferences and Meetings Karen Panetta, Vice President, Membership and Student Activities

Valeria Garai, Asst. Secretary Editors Peng Shi, EIC, IEEE Transactions on Cybernetics Robert Kozma, EIC, IEEE Transactions on Systems, Man, and Cybernetics: Systems Ljiljana Trajkovic, EIC, IEEE Transactions on Human–Machine Systems Bin Hu, EIC, IEEE Transactions on Computational Social Systems Dongrui Wu, EIC, SMC E-Newsletter Industrial Liaison Committee Christopher Nemeth, Chair Sunil Bharitkar Michael Henshaw Yo-Ping Huang Azad Madni Rodney Roberts Organization and Planning Committee Vladimir Marik, Chair Enrique Herrera Viedma Mengchu Zhou Dimitar Filev Robert Woon Ferat Sahin Edward Tunstel Larry Hall Jay Wang Michael Smith C.L. Philip Chen Karen Panetta

History Committee Michael Smith Membership and Student Activities Committee Karen Panetta, Chair György Eigner, Coordinator Christopher Nemeth

Chapter Coordinators Subcommittee Lance Fung, Chair Enrique Herrera-Viedma Imre Rudas Adrian Stoica Maria Pia Fanti Karen Panetta Hideyuki Takagi Ching-Chih Tsai

Lance Fung Robert Kozma Roxanna Pakkar Saeid Nahavandi Okyay Kaynak Tadahiko Murata Ferial El-Hawary Paolo Fiorini Shun-Feng Su Virgil Adumitroaie Peng Shi Ashitey Trebi-Ollennu Hideyuki Takagi

Student Activities Subcommittee Roxanna Pakkar, Chair Bryan Lara Tovar Piril Nergis JuanJuan Li X. Wang

Standards Committee Loi Lei Lai, Chair (China) Chun Sing Lai, Vice Chair (UK) Wei-jen Lee (USA) Thomas Strasser (Austria) Dongxiao Wang (Australia) Chaochai Zhang (China) Haibin Zhu (Canada)

Young Professionals Subcommittee György Eigner, Chair Ronald Bock Sonia Sharma Xuan Chen Raul Roman Fernando Schramm

Nominations Committee Imre Rudas, Chair C.L. Philip Chen Vladimir Marik Ljiljana Trajkovic Awards Committee Dimitar Filev, Chair Edward Tunstel Laurence Hall Ljiljana Trajkovic Peng Shi Michael H. Smith Vladik Kreinovich

IEEE PERIODICALS MAGAZINES DEPARTMENT 445 Hoes Lane, Piscataway, NJ 08854 USA Peter Stavenick Journals Production Manager

Fellows Evaluation Committee Edward Tunstel, Chair Mengchu Zhou, Vice Chair Liping Fang Maria Pia Fanti Vladimir Marik Germano Lambert-Torres Karen Panetta Ching-Chih Tsai Electronic Communications Subcommittee Saeid Nahavandi, Chair Syed Salaken, Web Editor Darius Nahavandi, Social Media Mariagrazia Dotoli Patrick Chan Haibin Zhu Ying (Gina) Tang Ferat Sahin György Eigner

Katie Sullivan Senior Manager, Journals Production Janet Dudar Senior Art Director Gail A. Schnitzer Associate Art Director Theresa L. Smith Production Coordinator Mark David Director, Business Development— Media & Advertising Felicia Spagnoli Advertising Production Manager Peter M. Tuohy Production Director Kevin Lisankie Editorial Services Director Dawn M. Melley Staff Director, Publishing Operations

IEEE SYSTEMS, MAN, AND CYBERNETICS MAGAZINE (ISSN 2333-942X) is published quarterly by the Institute of Electrical and Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, Telephone: +1 212 419 7900. Responsibility for the content rests upon the authors and not upon the IEEE, the Society or its members. IEEE Service Center (for orders, subscriptions, address changes): 445 Hoes Lane, Piscataway, NJ 08855-1331 USA. Telephone: +1 732 981 0060. Subscription rates: Annual subscription rates included in IEEE Systems, Man, and Cybernetics Society member dues. Subscription rates available on request. Copyright and reprint permission: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of U.S. Copyright law for the private use of patrons 1) those post-1977 articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without a fee. For other copying, reprint, or republication permission, write Copyrights and Permissions Department, IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854. Copyright © 2022 by the Institute of Electrical and Electronics Engineers Inc. All rights reserved.

IEEE prohibits discrimination, harassment, and bullying. For more information, visit http://www. ieee.org/web/aboutus/whatis/policies/ p9-26.html.

Digital Object Identifier 10.1109/MSMC.2022.3207037



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

3

Editorial

by Haibin Zhu 

Blockchain and Artificial Intelligence Are Also Hot Topics in the IEEE Systems, Man, and Cybernetics Society

T

his issue presents three regular feature articles and a special section including three articles. I appreciate the effective efforts of the guest editors of this special section on “Explainable Artificial Intelligence for the Social Internet of Things: Analysis and Modeling Using Collaborative Technologies”: Prof. Gaurav Dhima n from Chandigarh University, India; Prof. Atulya Nagar from Liverpool Hope University, United Kingdom; and Prof. Seifedine Kadr y from Noroff University College, Norway. Their special issue attracted many high-quality submissions. We decided to publish the accepted papers as a special section in this issue. We believe that this special section is beneficial for all of the related parties, the editorial board of the magazine, the guest editors of the special issue, and the authors of the accepted papers. A similar special issue will be published in 2023. The first feature article presented in this issue, “On Blockchain: Design Principle, Building Blocks, Core Innovations, and Misconceptions,” is authored by Prof. Wenbing Zhao Digital Object Identifier 10.1109/MSMC.2022.3205493 Date of current version: 20 October 2022

4

Haibin Zhu

from Cleveland State University, OH, United States. He believes that blockchain has been one of the hottest research topics recently. This technology may lead to a new generation of decentralized organizational architectures and applications. However, there are many misunderstandings regarding blockchain. Most notably, blockchain has been used as a synonym for data immutability and trust. As a matter of fact, this is incorrect. In the article, Zhao provides a concise description of exactly what blockchain is, including its design principles, building blocks, core in-

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

novations, and benefits, followed by an analysis of data immutability. He shows that to create an insurmountable barrier against attacks on data immutability, both decentralization and system scale are necessary. Through this analysis, Zhao further demonstrates the benefits provided by private and consortium blockchains when decentralization is removed. He also shows that private and consortium blockchains cannot support data immutability and trust, as many authors have claimed or implied, if decentralization is not followed. Instead, the centralized version of blockchain technology provides an elegant solution to achieving fault tolerance and atomic contract execution, which could make private and consortium blockchains useful for enterprises that would like to provide high availability to their customers and their internal operations. The second article, “A Transfer Learning-Based Method to Detect Insulator Faults of High-Voltage Transmission Lines via Aerial Images,” is contributed by a research team led by Prof. Mengchu Zhou of the New Jersey Institute of Technology, Newark, NJ, United States. In the article, the authors propose a transfer learning framework based on a pretrained VGG-19 deep convolutional neural network (CNN) to detect the “missing faults” (broken insulators) in aerial images. In their proposed method, a large well-known imagery dataset called ImageNet is used to train VGG19, and then the knowledge of this deep CNN is transferred. By using a few layers of fine-tuning, a newly built deep CNN is capable of distinguishing the corrupted and intact insulators. Their method can diagnose the faults using aerial images taken from transmission lines in different environments. The dataset used

tion of Flooding Due to Heavy Rainfall is the Chinese Power Line Insulator in India Using Machine Learning AlDataset, which is an imbalanced datagorithms: Providing Advanced Warnset that includes only 3,808 insulator i ng.” T he aut hor images. They then claims that floods propose a random are one of the deadimage augmentation By using a few liest disasters in the procedure to generlayers of finecoastal areas of Inate a more suitable tuning, a newly dia. For the past two dataset with 16,720 built deep CNN decades, most of the images. This new coastal states of Indataset allows them is capable of dia have been affectto offer higher dedistinguishing the ed by heavy rainfall tection accuracy than corrupted and during the monsoon the original one since intact insulators. season from June it is a balanced datato September, espeset. The experiments cially in the propshow that the proerty and lives. The authors developed posed method has advantages over an effective flood prediction system various existing ones. using machine-learning algorithms Researchers R. Balamurugan, that can help in preventing the loss of Kshitiz Choudhary, and S.P. Raja from human lives and property. They used Vellore Institute of Technology, India, the K Nearest Neighbors (KNN), supoffered the third article titled “Predic-

port vector machine (SVM), Random Forest (RF), and Decision Tree (DT) methods in building their models. They also utilized a stacking classifier to resolve the issue of oversampling and low accuracy. The experimental results show that the proposed models effectively predict floods due to realtime rainfall in that area. The other articles are introduced by the guest editors in the following special section editorial. About the Author Haibin Zhu ([email protected]) is a full professor and the coordinator of the Computer Science Program, the founding director of the Collaborative Systems Laboratory, and a member of the Arts and Science Executive Committee, Nipissing University, North Bay, Ontario P1B 8L7, Canada. 

Share Your Preprint Research with the World! TechRxiv is a free preprint server for unpublished research in electrical engineering, computer science, and related technology. TechRxiv provides researchers the opportunity to share early results of their work ahead of formal peer review and publication. BENEFITS: • • • • •

Rapidly disseminate your research findings Gather feedback from fellow researchers Find potential collaborators in the scientific community Establish the precedence of a discovery Document research results in advance of publication

Upload your unpublished research today! Follow us @TechRxiv_org Learn more techrxiv.org

Powered by IEEE



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

5

On Blockchain Design Principle, Building Blocks, Core Innovations, and Misconceptions by Wenbing Zhao 

B

lockchain has become one of the hottest research areas in recent years. The technology could potentially lead to a new generation of decentralized applications and decentralized autonomous organizations. Unfortunately, there is simply too much misinformation regarding blockchain. Most notably, blockchain has been used as a buzzword synonymous with data immutability and trust. In fact, this is far from the truth. In this article, we provide a concise description of exactly what blockchain technology is, including its design principle, building blocks, core innovations, and benefits. This is followed by an analysis of data immutability. We show that to create an insurmountable barrier against attacks on data immutability, decentralization and system scale are both necessary. Based on this analysis, we further dissect what benefits private and consortium blockchain could actually offer when decentralization is removed. We show that private and consortium blockchain cannot offer data immutability and trust as many works in the literature have claimed or implied. Instead, the centralized version of blockchain technology provides an elegant solution to achieving fault tolerance and atomic contract execution, which could make private and consortium blockchain useful for enterprises that would like to provide high availability to their customers and for their internal operations.

Digital Object Identifier 10.1109/MSMC.2022.3192658 Date of current version: 20 October 2022

6

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Overview Blockchain technology was developed to power the first cryptocurrency, Bitcoin [1]. However, its impact goes far beyond the cryptocurrency domain. Indeed, the possibility of establishing decentralized applications and decentralized autonomous organizations that ensure censorship resistance, full transparency, and high level of trust is extremely desirable for civilization and highly relevant to the general public [2]. As such, blockchain has attracted great interest from virtually all sectors of society. The technology has been touted as the next big thing that could transform how society operates [2]. Unfortunately, there is simply too much misinformation regarding blockchain research and development. Most notably, blockchain has been used as a buzzword synonymous with data immutability and trust. In fact, this is far from the truth. In many new developments, blockchain has basically been interpreted as a data structure (i.e., a chain of blocks), perhaps in conjunction with peer-to-peer (P2P) network topology. We point out that these kinds of formality will not automatically give the users the benefits of blockchain technology. In this article, we define exactly what blockchain technology is, including its design principle, building blocks, and core innovations. We also describe its use in cryptocurrency. This is followed by a systematic view of the benefits of blockchain technology. Then, we engage in a discussion of data immutability. We lay out the mechanisms introduced in large blockchain systems, such as Bitcoin [1] and Ethereum 2333-942X/22©2022IEEE



blockchain. To accomplish this, blockchain technology introduced a set of clever mechanisms consisting of cryptography, the chaining of blocks, and decentralized consensus based on proof of work (PoW). However, the system must grow to be very large for the barrier to be high. To facilitate that growth, blockchain technology has a built-in incentive; i.e., it offers a block reward for the miner who successfully solves the puzzle at each block height. This mechanism attracts people to invest in and join the system. ◆◆ Finally, we dissect the value of private and consortium blockchain. We point out that the biggest difference between private/consortium blockchain and public blockchain is that the former is a centrally controlled system. The lack of decentralization removes many of the benefits of public blockchain, especially data immutability, because the owner of a system could arbitrarily change information. Related Work Due to the sheer number of review articles and books on blockchain, it is not practical to compare each of them. In Table 1, we highlight a subset of related works [2], [4]–[10] in comparison with this article. The comparison covers a number of categories, including whether or not and to what extent a publication elaborated on 1) the building blocks of blockchain technology, 2) the benefits of blockchain technology, 3) the applications of blockchain, 4) the challenges for future development in blockchain, 5) how data O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

7

IMAGE LICENSED BY INGRAM PUBLISHING

[3], to ensure data immutability. We demonstrate that to create an insurmountable barrier against attacks on data immutability, decentralization and system scale are both essential. Based on this discussion, we argue that private and consortium blockchain are centralized systems and violate the design principle of blockchain technology. Thus, they should be clearly differentiated from blockchain technology as we know it, and particularly, data immutability is not guaranteed at all in private and consortium blockchain. Before we conclude, we outline challenges and developments in blockchain research. More specifically, this article makes the following contributions: ◆◆ First, this article aims to define blockchain technology in an authoritative way by emphasizing its design principle and innovations. Blockchain technology is based on decentralization and needs network scale to achieve its purpose. It provides the first practical solution for reaching consensus in a large decentralized system. ◆◆ Second, this article describes the benefits of blockchain technology in a systematic manner. These benefits are largely determined by the design principle and innovations in blockchain, where decentralization plays an essential role. ◆◆ Third, we provide an in-depth discussion of how data immutability is achievable in blockchain technology. Being a decentralized, trustless system, the only way of achieving data immutability is to enact an insurmountable barrier to changing information on the

the IoT. The survey by Alshaikhli et al. [10] focused on a immutability is achieved in blockchain, and 6) the differreview of IoT technology and discussed blockchain as ences between permissioned (i.e., private and consortium) one of the enabling tools. Finally, quantitative and qualiblockchain and permissionless (i.e., public) blockchain. tative analyses of blockchain have been carried out by a The book by Swan [2] focused mostly on applications of number of works (such as [11]). blockchain technology in various Common to all these papers is sectors. The one by Antonopoulos a lack of in-depth discussion of [4] provided a deep technical To create an how data immutability is achieved description of Bitcoin, but it did not insurmountable in blockchain and the fundamenprovide a systematic discussion of t a l d i f fer ence s bet we en p erthe benefits of the currency. In addibarrier against m i s sioned and permissionless tion to providing a deep technical attacks on data blockchain. The primary goal of description of blockchain technolothis article is to elaborate on the gy and its applications, the work by immutability, issues that have been largely Zhao [5] covered research on the decentralization and unaddressed. One aspect contraditional distributed dependable cerns how to achieve data immusystem prior to the development of system scale are tability and why decentralization blockchain technology. This helps both necessary. and scale are essential. The secreaders to appreciate the innovaond involves differentiating pertions brought by blockchain techmissioned blockchain, such as nology. However, none of these private and consortium blockchain, from permissionless books explicitly discussed how data immutability is blockchain, including Bitcoin and Ethereum. achieved and the differences between permissioned and permissionless blockchain. Blockchain Technology The paper by Wang et al. [6] gave a concise description We define blockchain technology based on Bitcoin, the of blockchain technology and focused on various consenfirst cryptocurrency [1], and Ethereum [3], the second sus algorithms proposed for blockchain. The work by major blockchain platform. All other blockchain systems Alotaibi [7] focused on blockchain’s application in are derived from or inspired by Bitcoin and Ethereum. enhancing the security of the Internet of Things (IoT). Blockchain is composed of the following technical foundaHence, it is light on discussion of the building blocks of tions [4]: 1) an open P2P network, 2) cryptography, 3) the blockchain technology. The comprehensive survey by blockchain data structure, 4) decentralized consensus, Zhao et al. [8] covered important subjects, such as the and 5) smart contracts. We continue this section by providbuilding blocks, benefits, applications, and challenges of ing a brief description of how cryptocurrency works. blockchain technology. The paper by Dwivedi et al. [9] also focused on applications of blockchain technology in the IoT. It had a concise introduction to the building Open P2P Network blocks of the technology as well as an extensive discusBlockchain operates on an open P2P network where sion of the applications and challenges of blockchain in there is no centralized control or trusted entity deciding

Table 1. The comparison of this article and related work.

8

Work

Building Blocks

Benefits

Applications

Future Challenges

Data Immutability

Type

Swan [2]

Partial

Partial

Yes

Yes

No

No

Antonopoulos [4]

Yes

Partial

Yes

Yes

No

No

Zhao [5]

Yes

Yes

Yes

Yes

No

No

Wang et al. [6]

Yes

No

No

Yes

No

No

Alotaibi [7]

Partial

Partial

Yes

Yes

No

No

Zhao et al. [8]

Yes

Yes

Yes

Yes

No

No

Dwivedi et al. [9]

Yes

Partial

Yes

Yes

No

No

Alshaikhli et al. [10]

Yes

Partial

Yes

Yes

No

No

This work

Yes

Yes

No

Yes

Yes

Yes

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

transactions are totally ordered with no ambiguity. who can join [12]. We use the term open P2P network to Given an open P2P system, a highly robust consensus emphasize that we are not referring to the P2P network algorithm is needed to ensure that blockchain copies topology. Any individual or organization could set up a P2P maintained by all the mining nodes are identical so that network topology and retain full control over it. On the conthe ledger can be useful. trary, an open P2P network is one that anyone can join and leave without the permission of any individual or organization. There Decentralized Consensus are typically two types of nodes: 1) It is decentralized consensus that It is decentralized mining nodes (i.e., miners), which makes cryptocurrency possible, have a very high hashing rate and and indeed, this is the most sigconsensus that makes 2) regular user nodes, which create nificant innovation of blockchain cryptocurrency transactions and could run on technology. The first decentralpossible, and indeed, lightweight computing nodes, such ized consensus algorithm was the as smartphones. Miners aggregate PoW algorithm introduced as part this is the most transactions into new blocks and of Bitcoin. PoW takes an unprecesignificant innovation help secure the system by particidented approach to reaching conpating in the consensus competis en su s. I n s t ea d of rea ch i ng of blockchain tion in exchange for block rewards agreement via a collective protechnology. and transaction fees. Regular users cess, such as traditional distributwho generate transactions pay ed consensus [5], PoW converts the miners the transaction fees. the problem into a lottery competition, whose difficulty can be adjusted dynamically based on the system’s total hashCryptography ing power. Due to the finite search space for the solution, Cryptography is an indispensable building block for any it is guaranteed that some mining node will solve the secure system [13]. In blockchain, several cryptographic consensus puzzle. Since the competition is a stochastic primitives are used for various purposes. Cryptographic process [14], the winning probability is proportional to hashes are used to generate addresses as pseudoanonythe hashing power of the competing node. mous user identifiers from the public key, generate transThe design of PoW ensures that the puzzle is solvable action hashes so that transactions can be referenced, and within a finite amount of time (assuming that there is a create block hashes so that blocks are chained together. minimum hashing rate), but it does not guarantee that only Most important, cryptographic hashes are used in PoW as a single miner will solve the puzzle in each competition part of the puzzle design for reaching decentralized con[15]. If two or more miners happen to solve the puzzle sensus in the general form of H(P) < D, where P denotes almost at the same time at the same block height, a fork the materials to be hashed, H( ) is the cryptographic hash occurs, which leads to inconsistency among the miners. function, and D is the difficulty target. The cryptographic To resolve this, a miner should choose a chain that has the hash function makes it impossible for anyone to deduce P highest cumulative difficulty. when given D, and the only way to solve the puzzle is to try different values of P to see if H(P) is smaller than D. Therefore, the puzzle competition is a stochastic process, Smart Contract and it offers a fair playing ground to all mining nodes. The first Turing-complete smart contract was introduced Public key cryptographs are used to generate and verify in Ethereum, which made Ethereum the first large-scale, digital signatures. To prove ownership of coins received fault-tolerant, and secure computer powered by decentralin a transaction, a user must possess the corresponding ized technology [5]. The key innovation is the combination private key and use it to generate a digital signature. All of a virtual machine (VM), referred to as the Ethereum VM miners verify the signature and accept the ownership only (EVM), and a gas mechanism to prevent the halting probif the digital signature is valid (provided that other condilem. All smart contracts are run within the EVM. The EVM tions are met, as well). can abort a smart contract execution, and it ensures the atomicity of the smart contract execution. Any transaction that invokes a smart contract must provide some amount Blockchain Data Structure of gas, and every instruction during the execution of the The most essential requirement for a cryptocurrency is smart contract consumes some of the gas. If the gas is the prevention of double-spending attacks; i.e., the sysexhausted prior to the end of the smart contract executem needs to make sure no one can spend coins twice. tion, the process is terminated without causing any side To satisfy this, an important step is to record all transaceffect. The smart contract execution happens only when a tions in a secure ledger. Blockchain is an implementation block is successfully mined and the transaction has been of a transaction ledger in the form of a singly linked list included in the block by the miner. The smart contract is of blocks, starting with the genesis block, so that all

O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

9

specified in such a way that its execution is deterministic. The effect of the smart contract execution (the world state and transaction receipts) is stored with the blockchain and referenced by the block header [5]. Cryptocurrency The first killer application of blockchain technology is cryptocurrency. Although not a part of the key building blocks of blockchain, a wallet is an essential component for cryptocurrency because it facilitates the usability of the system. A user could create one or more wallets to manage private/public keys, corresponding addresses, and transaction histories as well as to create new transactions. There are two major types of wallets: hot and cold (also referred to as hot wallet storage and cold wallet storage). A hot wallet stores frequently used keys and is online all the time, while a cold wallet stores infrequently used keys and is kept offline. While a hot wallet makes it much easier for a user to engage in cryptocurrency transactions, it risks being hacked; e.g., the key could be stolen. A cold wallet is much more secure because it is inaccessible to hackers, but it is less convenient to use. A cryptocurrency, such as Bitcoin, works in the following ways. A user who wishes to engage in cryptocurrency transactions needs to download a distribution of wallet software and install it on his or her smartphone or computer. The wallet software consists of not only the wallet functionality but all the key blockchain software

Trust

components (usually, only the headers of the blocks and the software do not participate in the mining competition). Then, the user needs to purchase cryptocurrency, either through a friend who already has some or via an exchange. The cryptocurrency is deposited in a wallet. If the amount is small, the user can start spending it for services and products as soon as the deposit has been verified. If the amount is larger, the user has to wait until the deposit has been included in the next block. Benefits of Blockchain The benefits of blockchain are highlighted in Figure 1. They are enabled by the fundamental building blocks of blockchain technology, which are located at the center of the figure. They include privacy, security, transparency, data immutability, fault tolerance, censorship resistance, data provenance, atomic contract execution, and trust. We highlight trust at the outermost edge of the circle because it is a quality-of-service measure enabled by all the specific benefits. Privacy Because cryptocurrency is designed to mimic physical cash, blockchain adopts a zero-knowledge proof mechanism in the form of a digital signature on the proof of ownership of coins (i.e., the presence of the private key corresponding to the address of the coins). This mechanism not only greatly simplifies the burden of identity management but also offers pseudoanonymity because users can generate virtually unlimited private–public key pairs and associated addresses.

Data Immutability

Data Provenance

Cryptography Decentralization

Atomic Contract Execution

Consensus

Privacy

Blockchain (Smart Contract)

Security

Fault Tolerance

Transparency

Censorship Resistance

Figure 1. The benefits of blockchain technology. 10

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Security Security means that an asset is properly protected against various attacks [16]. In the context of blockchain, security means that the system has the following two properties: 1) only the owner of a coin can spend it and 2) the owner can spend the coin only once (i.e., no double spending). As we mentioned, whomever has the corresponding private key is able to generate a valid signature and, in turn, spend the coin. Hence, it is incumbent upon the owner to properly protect the private key (i.e., he or she cannot expose it to anyone else and/or lose it). To satisfy the second requirement, the history of all transactions must be protected against modification; i.e., blockchain must be immutable. Transparency In blockchain, the history of all transactions is available to every party to inspect and analyze. In conjunction with

data immutability, this ensures transparency; i.e., there is nothing to hide. Of course, this transparency could risk revealing trade secrets, in which case, a secure data store that only authorized users could access could be developed using smart contracts. Data Immutability Although security implies data immutability, we list data immutability separately because it is the most prominent property of blockchain technology. The blockchain data structure, decentralized consensus design, decentralization, and network scale create an insurmountable barrier to changing information. A more detailed discussion is provided in the “How to Achieve Data Immutability in Blockchain” section. Fault Tolerance Fault tolerance has been studied for several decades. Traditional fault-tolerant systems are highly complex, intrinsically not scalable, and quite often vulnerable to corner cases [5]. The issues with traditional fault tolerance stem from its fundamental approach of using membership and voting for collective decisions. In blockchain systems, a completely different approach is taken to reach decentralized consensus, which does not use membership, collective decision making, and unreliable failure detectors. The winner of the consensus puzzle competition gets the right to assemble a new block and execute the smart contract if it is supported. Fault tolerance is achieved in blockchain in a much more simple, elegant, and robust way. Censorship Resistance Only an open system with data immutability can ensure censorship resistance. Blockchain systems, such as Bitcoin and Ethereum, offer censorship resistance to a large extent. We caution that blockchain is not absolutely censorship resistant. Because blockchain runs on top of the Internet, which requires connectivity, governments could shut down even the largest blockchain systems by turning off Internet access. It is also possible for governments to enact laws and reinterpret existing laws to persecute those who participate in the operation of blockchain. P2P file sharing was shut down exactly this way. Data Provenance Data provenance depends on data immutability but goes beyond it. Data immutability typically refers to the fact that a particular item cannot be modified or removed from the record. Data provenance concerns the context of an item. More officially, it refers to the property that the chronology of the ownership, custody, and location of a data item can be fully documented [17]. It also means there is end-to-end traceability [7]. Data provenance is critical to meeting governmental regulations and during forensic investigations.

Atomic Contract Execution In traditional financial systems, a monetary transaction carries credit risk because it consists of two operations: a debit operation and a credit operation [18]. In blockchain, the two operations are combined into a single atomic action, thereby eliminating credit risk. This is the simplest form of atomic contract execution. With a smart contract, an arbitrary, single-threaded program can be executed atomically. In traditional fault tolerance systems, ensuring the atomicity of a group of operations is very difficult due to 1) replica nondeterminism [19] and 2) various possible failures and cyberattacks [20]. As we mentioned, blockchain cleverly solved the root cause of the problem by eliminating the necessity to run the code concurrently at multiple replicas [5]. Trust We define trust as the level of assurance a system provides its users [20], such as dependability, security, and privacy. Blockchain technology offers a great way to develop applications that provide a much stronger level of trust than if only traditional technologies are used. Trust is enabled by all the specific benefits we have identified. How to Achieve Data Immutability in Blockchain It is well known that cryptography can greatly help protect the integrity of data, including the detection of deletion and modification of existing information as well as the insertion of data in an unauthorized way. However, huge data breaches have been in the news so often that people are no longer surprised by them. One obvious reason is centralized control. A centralized system can be penetrated through its most vulnerable part (e.g., a human operator), and the barriers to launching attacks are low (e.g., executing phishing attacks, which costs hackers nothing). Decentralization helps avoid the presence of singlepoint failures. However, through decentralization alone, we still cannot solve the problem. Intuitively, we want to increase the barriers to successful attacks to a level so high that it is impractical for anyone to launch assaults. Bitcoin essentially provides such a solution via a set of mechanisms: 1) decentralized networking and computing, 2) the blockchain data structure, 3) cryptographic protection through secure hashes and digital signatures, 4) a massive degree of redundancy in data storage (i.e., many blockchain copies managed by a large number of nodes on an open P2P network), and 5) decentralized consensus. These mechanisms work together to establish a very high barrier to successful attacks against data integrity. Among them, the most innovative is decentralized consensus. The high electricity cost of PoW, in fact, constitutes the most essential barrier against attacks. Based on the design of PoW, the only way to double spend is to control more than half the total hashing power so that an attacker could win the consensus puzzle with O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

11

more than 50% probability. This is referred to as a 51% attack, double-spend attack, or fork attack [5]. We note that PoW provides a fair playing field in the consensus puzzlesolving competition, but it does not limit the hashing power of any particular mining node. The algorithm also does not include any mechanism to prevent collusion among miners. That is why mining pools are becoming more popular and threatening the decentralization of blockchain technology. To create an insurmountable barrier, the cost of investing in and maintaining more than half the hashing power must be very high, which requires blockchain to have a large number of mining nodes. To estimate the cost, let us consider Bitcoin as an example. As of this writing, the standard mining gear must have a hash rate of 140 terahashes (THs)/s to be competitive, where TH = 1012 hashes (based on the Bitcoin mining calculator at https://www. coinwarz.com/mining/bitcoin/calculator). Bitmain Antminer S19 XP is a popular mining machine that offers the 140 TH/s hash rate. The price of the machine fluctuates around US$10,000 for bulk orders [25]. Bitcoin had 14,815 reachable mining nodes as of 29 March 2022 [26]. To launch a 51% attack, an adversary must add at least 14,816 mining nodes with cutting-edge hash power (e.g., the Bitmain Antminer S19 XP). To have a comfortable margin to launch the attack, suppose the adversary decided to invest in 15,000 nodes. The initial cost would be US$10,000 ×15,000 = US$150 million for the mining gear alone. Even though this amount is out of reach for many, it is not a huge barrier for the wealthiest billionaires. In addition to the initial investment, the cost of sustaining a 51% attack is high. It includes electricity, Internet ser v ice, physical space and air conditioning, equipment maintenance, and facility security. The website https://www.crypto51.app/ provides an estimate of the cost for running a continuous 51% attack on a perhour basis. As of 31 March 2022, the highest per-hour cost was for Ethereum, at US$1,864,466, and the cost for Bitcoin was second, at US$1,596,505. The calculation was based on an estimate provided by NiceHash (https:// www.nicehash.com/), a PoW mining and trading platform. NiceHash is basically a special mining pool operator that attracts both miners and those who are interested in buying Bitcoin in the form of renting hash power from the platform instead of going to a cryptocurrency exchange. The business model requires NiceHash to have a fairly accurate estimate of the total hashing power of each PoW-based cryptocurrency. As can be seen, even for the largest blockchain system, data immutability is not bulletproof. For smaller cryptocurrencies, the barrier to attacks is a lot lower. It is not surprising that a successful 51% attack happened to Bitcoin Gold in May 2018 [27]. For reference, the cost of running a 1-h 51% attack on Bitcoin Gold is US$1,459, which is about 0.009% of the cost of attacking Bitcoin. Because Bitcoin Gold is a hard fork from Bitcoin that runs the same PoW algorithm, the only 12

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

major difference is the scale of the network. We note that data immutability is the most essential property of blockchain. Without data immutability, data provenance and censorship resistance cannot be guaranteed, and most important, such a system cannot be trustworthy at the level of large, public blockchains. Private and Consortium Blockchain: What Is Their Value? Bitcoin and Ethereum are large-scale, public blockchain systems that strictly follow the decentralized design principle. Anyone can join them without others’ permission. In recent years, two forms of permissioned blockchain, namely, private blockchain and consortium blockchain, have been proposed. A private blockchain is controlled by a single individual or organization. A consortium blockchain is controlled by a group formed by a number of institutions. The well-publicized Libra project proposed by Facebook is an example of a consortium blockchain, in which the Libra Association will be formed and up to 100 institutions will be allowed to join. Due to the centralized control by a single owner (i.e., an individual, an organization, or a consortium), the principle of decentralization and, therefore, openness and fairness are completely wiped out in permissioned blockchains. The owner of a permissioned blockchain can modify the data as it wishes, e.g., by suspending all but one mining node and instructing it to decide what transactions to include and exclude to build a new branch of the chain that will become the main blockchain. Therefore, data immutability and ultimately trust cannot be guaranteed in permissioned blockchains. With that being said, one may wonder: Does a permissioned blockchain have any value at all? Figure 2 conveys the results of an analysis in terms of the benefits identified in the “Benefits of Blockchain” section but in the context of centralized control instead of decentralization, summarized as follows: ◆◆ No data immutability: We have argued that data immutability is intrinsically not achievable in a permissioned blockchain because whomever controls the blockchain can arbitrarily change any information placed on it. In fact, data could be changed in a permissioned blockchain in a way much worse than that of a successful double-spend attack in a public blockchain. In a public blockchain, a hacker could insert his or her own transactions and delete transactions in a successful double-spend attack but could not modify transactions generated by other users. In a permissioned blockchain, however, whoever controls the blockchain could gain access to the private keys of all the miners, which would enable the owner to change the blockchain in an arbitrary way. Hence, in terms of data protection, a permissioned blockchain has no advantage over a traditional centralized system. ◆◆ No censorship resistance: This is obvious because censorship resistance relies on data immutability.

Future Research Directions Research on blockchain is highly active due to plentiful provenance relies on data immutability. opportunities. Here, we mention three directions of active ◆◆ No transparency: Transparency in a public blockchain research and development: 1) increasing the effective is established by making the entire blockchain accessithroughput of blockchain, 2) reducing the electricity cost of ble and guaranteeing that all records have not been achieving consensus in blockchain, and 3) increasing the tampered with and never will be. Even if a permisanonymity of blockchain. In the first direction, a general sioned blockchain is not encrypted and publicly accesapproach is to utilize off-chain and multichain channels. sible, the lack of data immutability destroys the The most well-known development is perhaps the payment foundation of transparency. channel proposed by the Lightening Network [6]. In the sec◆◆ No privacy: Because a user needs to be granted perond direction, proof of stake has been intensively studied, mission to join and stay in a permissioned blockchain, and various algorithms have been proposed [21], [22]. Trahis or her identity cannot be protected, at least not at ditional consensus algorithms, such as Practical Byzantine the same level offered by a public blockchain. Fault Tolerance, have also been proposed to address ener◆ No trust: Data immutability is one of the most fundagy consumption. However, we strongly oppose such an mental pillars for establishing trust. Without it, the integapproach because it is directly against the decentralization rity of a system’s operation will be called into question. principle of blockchain technology. In the third direction, ◆ No security: The inability to ensure data immutability there have been efforts to improve the pseudoanonymity means that the owner of a permissioned blockchain provided by blockchain technology such that transactions could allow favored users to double spend. According cannot be easily linked. The basic idea is to remove singleto our definition of security for blockchain operations, transaction-level linkability to mix a group of transactions this would be a serious breach. together, which is referred to as CoinJoin [23]. The scheme ◆  Fault tolerance and atomic contract execution: Blockcan be further improved via blind signatures [24]. chain technology created a completely new way of constructing robust fault-tolerant systems. In the conConclusion text of crash fault tolerance, the process of randomly In this article, we strived to give a precise description of selecting a single node to assemble the block, and in blockchain technology, including its design principle, the case of smart contracts, using the same single node to execute the smart contract code, still works well even if the nodes are centrally controlled by an individual Trust or organization. However, to tolerate malicious faults (i.e., Byzantine Data faults), decentralization is indispensImmutability able to fault tolerance mechanisms. Without decentralization, fault tolerAtomic ance mechanisms in a permissioned Data Cryptography Contract Provenance blockchain would suffer from the Execution Centralized Control same problem faced by all traditional Byzantine fault tolerance solutions. Consensus To be able to tolerate malicious faults, Blockchain we must assume that all replicas fail Privacy Security (Smart independently. However, such an assumpContract) tion is hardly valid in a permissioned blockchain because all the nodes are controlled by the owner. How can we be sure Fault the owner is not malicious? The whole Transparency Tolerance point of blockchain technology is decentralization so that we do not have to trust an individual or organization. That being Censorship said, all traditional Byzantine fault tolerResistance ance solutions are essentially permissioned systems because of the need to establish known membership. That is why we list fault tolerance and atomic contract execution as the remaining ben- Figure 2. The analysis of blockchain technology in the context of efits of permissioned blockchains. centralized control. ◆◆ No data provenance: This is also obvious since data



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

13

building blocks, key innovations, and benefits, and to discuss two important issues that have largely been ignored in the literature: 1) how data immutability is achieved in blockchain and 2) how different permissioned blockchain is from permissionless blockchain. To accommodate readers who are not familiar with blockchain technology and cryptocurrency, we provided a concise introduction to the building blocks of the technology, how cryptocurrency works, and the benefits of blockchain. The discussion of various topics was guided by the most important design principle of blockchain technology: decentralization. Due to space limitations, we cannot elaborate on some other very important issues. One such concern is the use of validators or some other form of committee to reach consensus. In our opinion, this is completely backward and would eliminate the most significant innovation brought by blockchain technology, i.e., decentralized consensus. Decentralized consensus, such as PoW, aims to reach consensus without explicit voting and membership. Using a committee to reach consensus is tantamount to circular logic because the formation of a universally agreed committee is itself a consensus problem.

[6] W. Wang et al., “A survey on consensus mechanisms and mining strategy management in blockchain networks,” IEEE Access, vol. 7, pp. 22,328–22,370, Jan. 2019, doi: 10.1109/ACCESS.2019.2896108. [7] B. Alotaibi, “Utilizing blockchain to overcome cyber security concerns in the internet of things: A review,” IEEE Sensors J., vol. 19, no. 23, pp. 10,953–10,971, 2019, doi: 10.1109/JSEN.2019.2935035. [8] W. Zhao, C. Jiang, H. Gao, S. Yang, and X. Luo, “Blockchain-enabled cyber–physical systems: A review,” IEEE Internet Things J., vol. 8, no. 6, pp. 4023–4034, 2020, doi: 10.1109/JIOT.2020.3014864. [9] A. Dhar Dwivedi, R. Singh, K. Kaushik, R. Rao Mukkamala, and W. S. Alnumay, “Blockchain and artificial intelligence for 5g-enabled internet of things: Challenges, opportunities, and solutions,” Trans. Emerg. Telecommun. Technol., p. e4329, Jul. 2021, doi: 10.1002/ett.4329. [10] M. Alshaikhli, T. Elfouly, O. Elharrouss, A. Mohamed, and N. Ottakath, “Evolution of Internet of Things from blockchain to IOTA: A survey,” IEEE Access, vol. 10, pp. 844–866, Jan. 2022, doi: 10.1109/ACCESS.2021.3138353. [11] M. Niranjanamurthy, B. Nithya, and S. Jagannatha, “Analysis of blockchain technology: Pros, cons and swot,” Cluster Comput., vol. 22, no. S6, pp. 14,743–14,757, 2019, doi: 10.1007/s10586-018-2387-5. [12] G. Fox, “Peer-to-peer networks,” Comput. Sci. Eng., vol. 3, no. 3, pp. 75–77, 2001, doi: 10.1109/5992.919270. [13] J. Katz and Y. Lindell, Introduction to Modern Cryptography. Boca Raton, FL, USA: CRC, 2020.

Acknowledgment This work was supported in part by the U.S. Department of Energy award DE-FE0031745.

[14] S. M. Ross et al., Stochastic Processes, vol. 2. New York, NY, USA: Wiley, 1996. [15] W. Zhao, S. Yang, and X. Luo, “On consensus in public blockchains,” in Proc. 2019 Int. Conf. Blockchain Technol., Honolulu, HI, USA: ACM, pp. 1–5, doi: 10.1145/3320154.3320162.

About the Author Wenbing Zhao ([email protected]) earned his Ph.D. degree in electrical and computer engineering from the University of California, Santa Barbara, in 2002. Currently, he is a full professor in the Department of Electrical Engineering and Computer Science, Cleveland State University, Cleveland, Ohio, 44115, USA. His research interests include dependable distributed systems and smart and connected health. He has more than 200 academic publications and won best paper awards at several international conferences. He has served on the organizing and technical committees of numerous IEEE conferences. He is an associate editor for IEEE Access, an associate editor for MDPI Computers, an academic editor for PeerJ Computer Science, and a member of the editorial board of several international journals. He is a Senior Member of IEEE.

[16] C. P. Pfleeger, Security in Computing. Englewood Cliffs, NJ, USA: Prentice-Hall, 1988. [17] Y. L. Simmhan, B. Plale, and D. Gannon, “A survey of data provenance in e-­science,” ACM Sigmod Rec., vol. 34, no. 3, pp. 31–36, 2005, doi: 10.1145/1084805.1084812. [18] Z. Xu and C. Zou, “What can blockchain do and cannot do?” China Econ. J., vol. 14, no. 1, pp. 4–25, 2021, doi: 10.1080/17538963.2020.1748968. [19] H. Zhang, W. Zhao, L. E. Moser, and P. M. Melliar-Smith, “Design and implementation of a byzantine fault tolerance framework for non-deterministic applications,” IET Softw., vol. 5, no. 3, pp. 342–356, 2011, doi: 10.1049/iet-sen.2010.0013. [20] H. Zhang, H. Chai, W. Zhao, P. M. Melliar-Smith, and L. E. Moser, “Trustworthy coordination of web services atomic transactions,” IEEE Trans. Parallel Distrib. Syst., vol. 23, no. 8, pp. 1551–1565, 2012, doi: 10.1109/TPDS.2011.292. [21] J. Chen and S. Micali, “Algorand: A secure and efficient distributed ledger,” Theor. Comput. Sci., vol. 777, pp. 155–183, Jul. 2019, doi: 10.1016/j.tcs.2019.02.001. [22] W. Zhao, S. Yang, X. Luo, and J. Zhou, “On peercoin proof of stake for blockchain consensus,” in Proc. 2021 3rd Int. Conf. Blockchain Technol., pp. 129–134, doi: 10.1145/3460537.3460547. [23] G. Maxwell, “CoinJoin: Bitcoin privacy for the real world,” Bitcoin Talk, 2013. [Online]. Available: https://bitcointalk.org/index.php?topic=279249.0

References

[24] E. Heilman, F. Baldimtsi, and S. Goldberg, “Blindly signed contracts: Anony-

[1] S. Nakamoto. “Bitcoin: A peer-to-peer electronic cash system.” Bitcoin. https://

mous on-blockchain and off-blockchain bitcoin transactions,” in Proc. Int. Conf.

bitcoin.org/bitcoin.pdf (Accessed: Mar. 1, 2022).

Financial Cryptography Data Security, pp. 43–60, doi: 10.1007/978-3-662

[2] M. Swan, Blockchain: Blueprint for a New Economy. Sebastopol, CA, USA:

-53357-4_4.

O’Reilly Media, 2015.

[25] “Products.” Bitmain. https://www.bitmain.com/ (Accessed: Jul. 25, 2022).

[3] V. Buterin, “Ethereum white paper,” Ethereum, Bern, Switzerland, White Paper,

[26] “Reachable Bitcoin nodes.” Bitnodes Project. https://bitnodes.io/ (Accessed:

2013. [Online]. Available: https://ethereum.org/en/whitepaper/

Jul. 25, 2022).

[4] A. M. Antonopoulos, Mastering Bitcoin: Programming the Open Blockchain.

[27] B. Dzhingarov. “Bitcoin Gold is now hit by a 51% attack—Cryptocurrency

Sebastopol, CA, USA: O’Reilly Media, 2017.

nightmare.” Cryptoext. https://cryptoext.com/bitcoin-gold-is-now-hit-by-a-51-attack-

[5] W. Zhao, From Traditional Fault Tolerance to Blockchain. Hoboken, NJ, USA:

cryptocurrency-nightmare/ (Accessed: Jul. 25, 2022).

Wiley, 2021.



14

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

©SHUTTERSTOCK.COM/KOTOVFOT

Distinguishing Intact and Broken Insulator Images

A Transfer Learning-Based Method to Detect Insulator Faults of High-Voltage Transmission Lines via Aerial Images by Fatemeh Mohammadi Shakiba  , S. Mohsen Azizi  , and Mengchu Zhou 

Digital Object Identifier 10.1109/MSMC.2022.3198027 Date of current version: 20 October 2022

2333-942X/22©2022IEEE



D

eep learning methods have shown great promise in high-voltage transmission lines’ (TLs’) intelligent inspections. The expansion of power systems, including TLs, has brought the problem of insulator fault detection into account more than before. In this article, a novel transfer learning framework based on a pretrained VGG-19 deep convolutional neural network (CNN) is proposed to detect “missing faults” (broken insulators) in aerial images. In this procedure, a well-known large imagery dataset called O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

15

intact operation of power grids, insulator fault detection ImageNet is used to train VGG-19, and then the knowledge and intelligent inspection have been considered salient of this deep CNN is transferred. By using a few layers for a tasks [1], [4]. fine-tuning purpose, the newly built deep CNN is capable Finding insulator faults using a traditional manual patrol of distinguishing the corrupted and intact insulators. This is inefficient, time consuming, and wastes human resourcmethod is able to diagnose these faults using the aerial es. Hence, it has been gradually replaced by an unmanned images taken from TLs in different environments. The aerial vehicle (UAV) patrol, as discussed in [5] and [6]. In this original dataset used in this article is the Chinese Power new process, workers do not require Line Insulator Dataset (CPLID), investigating TL insulator faults which is an imbalanced dataset using telescopes. However, the comand includes only 3,808 insulator plexity and variability of UAV appliimages. Therefore, a ra ndom Transfer learning cation scenarios have caused image-augmentation procedure is is generally used several new challenges in the autonproposed and applied to generate a when a new dataset omous detection of insulator faults more suitable dataset with 16,720 [1]. Artificial intelligence (AI)-based images. This new dataset allows us is smaller than the methodologies are among the most to offer higher detection accuracy primary dataset with important for addressing them. than the original one because it is There are many studies on insua balanced dataset. Training a which a base model lator fault identification using aerideep CNN by using it gives more is well trained. al images and image processing power to the system for detecting methodologies. In their first step, the corrupted insulators in differtraditional image processing algoent situations such as rotated, rithms classify insulator images dark, and blurry images with cominto classes with specific features such as texture, color, plex backgrounds. The comparison results of this study and shape. Then they take advantage of matching algoshow the advantages of the proposed method over various rithms to implement fault-detection procedures. Their existing ones. drawback is the fact that the features are designed manually and, therefore, they are inefficient in complicated power Introduction grids with a variety of image features. Moreover, correlaIncreasing demand for electrical energy results in expantion features of insulators in aerial images are not clear, sion of power systems, including TLs. Considering the and the accuracy of these algorithms is highly dependent vital role of insulators in TLs for mechanical support and on these features [7]–[10]. electrical insulation during power grid operations, and Recently, tremendous progress in AI has led to noticetheir proneness to faults and breakage, intelligent methable advancements in image processing methods based odologies have been proposed to enhance their safety on deep neural networks. CNNs are the most common and reliability [1]–[3]. Due to insulators’ exposure to outdeep neural networks that perform pattern recognition door environments for a long period of time, they face and object-detection tasks, are able to extract image fea“missing faults,” which are defined as broken insulators tures automatically, and learn under various environmen(see Figure 1). tal conditions [11]–[13] more than traditional ones. These Insulator faults are variant and random, and their CNN-based techniques overcome the limitations of tradioccurrences interrupt the safety and stability of the entire tional image processing approaches and conclude higher TL operation, thereby imposing tremendous economic performance and better accuracy in object-detection losses. To guarantee the safety and stability of TLs and tasks [14]–[16]. Although CNNs have completely dominated the image processing area in recent years and their results are much better than traditional methodologies, they have some drawbacks, such as requiring big datasets and considerable time for training. Therefore, an Broken Insulators advanced technique like transfer learning has been proposed, which makes deep CNNs capable of generating results faster and transferring knowledge from some already-trained CNNs to a new CNN that has common features with the base dataset for the purpose of saving time and resources [17]. A model gets trained and developed for one dataset first. It is then reused for a second related one, which is called transfer learning. In other words, a learned matter Figure 1. “Missing faults” (broken insulators) of TLs. 16

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

ImageNet for hierarchical convolutional feature extraction in one setting is exploited to facilitate another one for fast of visual object-tracking images. knowledge acquisition. Transfer learning is generally used In this article, we propose utilizing the transfer learning when a new dataset is smaller than the primary dataset method to develop a fault-detection system for distinguishwith which a base model is well trained. ing intact and broken insulator images for the first time. Several studies in the literature take advantage of The contributions of this work are two-fold: CNNs for fault diagnosis of TLs. Fahim et al. [18] propose a self-attention CNN framework and a time-series 1) implementing an image-augmentation procedure to genimage-based feature extraction model for fault diagnoerate a large, labeled dataset for insulator image classifisis of TLs with length of 100 (km) using a discrete cation based on the CPLID such that we have a balanced wavelet transform (DWT) for removing the noise of the dataset faulty voltage and current signals. The work [19] uses a 2) proposing a transfer learning technique for the balanced customized CNN for fault diagnosis of 50 (km) TLs intedataset using a VGG-19 CNN and an ImageNet dataset, grated with distributed generators. In [20], a machine which outperform the existing methodologies. learning-based CNN for TLs with a length of 280 (km) is used to execute fault diagnosis using a DWT for feature Preliminaries extraction. Shiddieqy et al. [21] propose a methodology that uses CNNs all TL fault features to generate various models for robust CNNs are capable of learning hierarchical features indefault detection. They perform various AI approaches, pendent from inputs, making them a common soluincluding CNN, to obtain 100% detection accuracy. In [22], tion for imagery dataset problems. The structure of a scheme to diagnose faults in power CNNs empowers them to have TLs with a length of 200 (km) using the fewest requirements for convolutional sparse auto-encoders is data preprocessing and handle proposed. This approach learns the data sets w ith nu merous Regional receptive extracted features from the voltage fe a tures more efficiently in fields, which are only and current signals for fault diagnosis. comparison to most of the artiMany researchers have applied ficial neural networks [36], a small, focused area transfer learning to various datasets in [37]. In general, CNNs consist of the input data, different domains [23]–[28]. In [29], the of three classes of layers: conauthors show how to transfer deep volution, pooling, and fully are used to connect CNN knowledge in real-time dataset connected. The convolutional to each node in a classification. The study in [30] proposand pooling layers build conconvolutional layer. es an intelligent approach using a deep volution blocks, which a re convolutional transfer learning netplaced one after another for work, which distinguishes the dynamic the purpose of feature extracsystem faults using an unlabeled datastion. Fully connected layers et. Shao et al. [31] present an intelligent fault diagnosis methplay the role of classifiers and the role of the output layer odology for a rotor-bearing system using an updated CNN as a fully connected one is to perform the classification with transfer learning. Li et al. [32] propose a deep adversarior regression tasks. al transfer learning network to investigate unknown emergCNN architectures are made of regional receptive ing faults in rotary machines. fields, shared weights, and the pooling operation. Regional In this study, the VGGNet model is used as a basic receptive fields, which are only a small, focused area of the model for training on a primary dataset (ImageNet) and input data, are used to connect to each node in a convoluthen reused to learn/transfer features for training on an tional layer. This feature is one of the main advantages of insulator imagery dataset. Taking advantage of the initial CNNs that causes a considerable reduction in the number training, transfer learning allows us to start with the of parameters in a CNN which, in turn, reduces its training learned features on the ImageNet dataset and then tune computational load [38]. the weights and structure of the base model to match the Numerous studies have been performed in image clasnew dataset/task instead of starting the learning process sification problems using CNNs to achieve better perforon the new data from scratch using random weight initialmances. CNNs are also used for protecting TLs in various ization [33]. There are also similar studies that perform aspects, such as their fault identification and distinguishtransfer learning on various datasets using VGGNet CNNs. ing intact waveforms from corrupted ones in waveform Huang et al. [34] used ImageNet to pretrain a VGG19 netimages. These studies include two various categories: the work for DenseBox initialization, which is defined as a approaches with a focus on imagery datasets recorded unified end-to-end full CNN framework for object detecfrom outdoor TLs [1], [15], [16], [39] and those that contion. Li et al. [35] adjust the VGG-19 pretrained on template time-series voltage and current waveforms

O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

17

stored from generators and fed to CNNs as blocks of data In the case of Ts ! Tt , a general subproblem of transfer points. The proposed method in this study is from the forlearning emerges, which is called domain adaptation. In mer category. this study, the relationship between the source and target In this article, a deep CNN introduced by Visual Geomedata is not the domain adaptation because the understudy try Group is used, called VGG-19. Two successful architecdatasets are both images, however, different ones [45]. tures of VGG are VGG-16 and VGG-19 and perform well on Transfer learning techniques help deep learning meththe ImageNet dataset. VGGNets are ods by distributing the learned feaan improved version of AlexNet tures across various learning [40], which takes advantage of applications. The transfer learning large kernel-sized filters in addifocus is on the training step to proTransfer learning tion to various small kernel-sized mote deep learning ability in featakes advantage of filters, and results in 13 and 16 conture extraction and adaptation volutional layers for VGG-16 and among a few comparable datasets. the pretrained layers VGG-19, respectively [41], [42]. Transfer learning takes advantage on a source task to of the pretrained layers on a source execute a target task. task to execute a target task. For Transfer Learning this aim, a pretrained neural netTransfer learning is a methodology work with its fully connected laythat provides the capability to ers removed is used, a nd its transfer learned knowledge from convolutional and pooling layers become frozen to carry one trained neural network to another one. The emerging out the feature extraction. For adaptation of the given preconcept of transfer learning, in addition to deep learning trained neural network to the target dataset, the fully constructures, shows remarkable improvements in the perfornected layers (classifier layers) need to update their mance of image classification and pattern recognition. weights to set up the desired model. Transfer learning includes two general steps: first, training Two distinct concepts exist in transfer learning: feature a base neural network on the basis of a source dataset, and extraction and fine-tuning. The first concept is when the second, transferring the learned features (weights) to fully connected layers are eliminated using the target dataanother neural network, which helps to train the target set, and the adjusted fully connected layers are combined dataset. The second step also can be achieved by adding with the frozen pretrained ones. In the fine-tuning process, only some extra layers for the purpose of adjustment and the structure of the fully connected layers from the pretuning to the base neural network. There are two main defitrained model is stored and their weights are updated. nitions for transfer learning, namely, domain and task, Fine-tuning is the process used in this study for the purwhich are described as follows: pose of adapting the ImageNet extracted features to the ◆◆ Definition 3 (domain [43], [44]): a doma i n insulator dataset under study. Figure 2 depicts the basic D = {X, P (X)} includes two elements, namely, a feaidea of transfer learning constituting two different datasets ture space X and a marginal probability distribution n that have resemblance. These datasets are the inputs of the P (X), where X = {x i} i = 1 ! X is a dataset in which source and target deep neural networks. The knowledge is every x i ! R D comes from this domain. transmitted to the target model to execute the target-train◆◆ Definition 4 (task [43], [44]): a task T = {Y, P (Y | X)} ing task rapidly. includes two elements, given a domain. The transfer learning technique is implemented in this ◆◆ D = {X, P (X)}, where Y stands for the label space, study for diagnosis of TL insulator problems. This method and P (Y | X) is the conditional probability distribufinds the rare occurrence of failures that are arduous or tion in which Y = {y i} in= 1 stands for the label vector of impossible to label. The fact that the transfer learningX with y i ! Y as the label of x i . based CNN has not been used before for TL insulator fault Transfer learning includes two domains, namely, a detection is noticeable, meaning that it is proposed and source domain D s = {X s, Ps (X s)} and a target domain studied in this article for the first time. X X D t = {X t, Pt (X t)}, where s and t present the feature spaces of the source and target domains, respectively, and ImageNet Ps (X s) and Pt (X t) show the marginal probability distribution of them. Based on Definition 3 and Definition 4, the ImageNet is a large-scale ontology of images built upon concept of transfer learning is presented as follows: the hierarchical structure provided by WordNet [46]. ImageNet aimed to collect the majority of the 80,000 synsets ◆◆ Definition 5 (transfer learning [44]): considering the of WordNet with an average of 500–1,000 clear and highsource domain D s, learning task Ts, target domain resolution images, and this resulted in tens of millions of Ts, and learning task Tt, the purpose of transfer annotated images categorized by the semantic hierarchy learning is to boost the performance of target predicof WordNet [46], [47]. ImageNet provides the most comtive function ft ( $ ) in D t by inducing the knowledge prehensive and diverse coverage of the image world. The from D s and Ts while D s ! D t or Ts ! Tt . 18

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

performances is a generally accepted notion [50], [51]. However, collecting large datasets can be a complicated task because of the need for manual efforts to collect and label data, especially in the field of image processing. The existing images from TL insulators are not an exception to this fact, and by searching Google Images it can be observed that there are not enough broken or defective insulator aerial images [50]. To the best of our knowledge, there is no standard dataset for TL insulator faults that consists of broken and intact insulators. Therefore, because the aerial images of insulator faults are rare and nearly impossible to collect, to obtain adequate insulator faults images, the simulated insulator faults samples are created on the basis of the CPLID [52]. In these images, which are created using the Photoshop software mentioned in [1], the normal insulator strings are erased and replaced by their nearby pixels. The process of using Photoshop to generate the desired dataset is time consuming and requires considerable effort. Therefore, only 248 faulty images are produced with this procedure while the number of images made including intact insulators is 3,560 [1]. Having this dataset from [1], image augmentation is the approach that helps generate a large, sufficient dataset, with 248 # 9 + 248 = 2, 480 faulty insulator images and 3, 560 # 3 + 3, 560 = 14, 240 intact insulator images in variant backgrounds. The image-augmentation process is done

current 12 subtrees consist of a total of 3.2 million cleanly annotated images spread over 5,247 classes. More than 600 images are collected for each synset of ImageNet. Today, ImageNet consists of more than 15 million annotated images. Some CNNs have shown great promise in classifying images in the ImageNet dataset into their corresponding categories, namely, AlexNet, VGGNet (used in this study), CaffeNet, ResNet, and so on [48]. Based on a review reported by Cheplygina et al. [49], ImageNet is the most commonly used dataset for transfer learning-based image analysis and classification methods [42], [47], [49]. In this article, ImageNet is used as a source dataset for the proposed transfer learning technique to perform feature extraction for insulator image fault classification. Insulator Dataset Generation Using Data Augmentation In classical discriminative examples such as the study of broken insulators versus intact ones discussed in this article, image recognition software has to overcome issues of viewpoint, lighting, occlusion, background, scale, and more. The task of data augmentation prepares these translational invariances for consideration of the dataset such that the resulting models can perform better despite the existing challenges. The concept of having a larger dataset results in better deep learning models, and

Source Model

Source Task

Fully Connected Layers

Convolutional Layer

Source Domain

(Classification) Pooling

Conv_n-2

Conv_n-1

Pooling

Conv_1-2

Input

Conv_1-1

(Feature Extraction)

Output Layer

Big, Labeled Dataset

Knowledge (Weights of Layers)

New Dataset (Unlabeled or Limited)

Target Model

Convolutional Layer

Target Domain

(Tuned Classification) Output Layer

Pooling

Conv_n-2

Conv_n-1

Pooling

Conv_1-2

Input

Conv_1-1

(Feature Extraction)

Target Task

Fully Connected Layers

Figure 2. The schematic of a typical transfer learning model.



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

19

randomly such that from each broken insulator image, nine augmented images, and from each intact insulator image, three augmented ones are generated with arbitrarily various features including brightness, angles, zooming level, and quality. The augmentation parameters are chosen randomly using Python’s Augmentor and preprocessing libraries. This library has features such as flip-left-right, black-and-white, rotate, skew, zoom, sample, and brightness, which can be added to a pipeline, providing random values to each

(a)

feature. This pipeline can generate the required number of augmented images that we need [50], [53]. Therefore, this augmented insulator image dataset includes 16,720 images. Figures 3 and 4 show some examples of these images. Proposed Method This study takes advantage of transfer learning techniques to improve the fault classification results for a TL insulators’ dataset. To perform this approach, a common deep CNN called VGG-19 is used for the purpose of

(b)

Figure 3. Images of an intact insulator. (a) An original image from the CPLID [59] and (b) three augmented

images generated from the original image.

(a)

(b) Figure 4. Images of a faulty (broken) insulator. (a) An original image from the CPLID [59] and (b) nine

augmented images generated from the original image.

20

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

fault diagnosis system. All the convolutional and maxfeature extraction. This CNN consists of 19 layers, out of pooling layers and the first fully connected layer perform which 13 are convolutional. as feature-extraction layers. Then, the weights of the two Although the benchmark CNNs for the ImageNet datasfully connected layers, which are located at the end of the et include numerous layers, the CNN models used in fault aforementioned feature-extraction layers, are free to diagnosis methods are relatively shallow. The depth of a become updated for the purpose of fine-tuning and adapCNN model for fault diagnosis is almost up to five hidden tation to the new dataset. The last layers, only because of the simple fully connected layer is a Softmax structure of the existing faults. classifier to adjust the pretrained This can underestimate the effecVGG-19 to the insulator dataset for tiveness of CNN models in fault The last fully fault identification. Two datasets diagnosis. Regardless of this difconnected layer is a a re used in this ex per iment, ference among the required CNN including ImageNet for the purmodels’ architectures, the volume Softmax classifier to pose of VGG-19 pretraining and of labeled samples is always limitadjust the pretrained feature extraction, and the auged in fault diagnosis applications, VGG-19 to the mented insulator dataset, comand it is complicated to train a prising 13,160 aerial images of deep CNN model without having a insulator dataset for insulators in two classes of defectremendous number of well-orgafault identification. tive (broken) and intact. Because nized datasets such as ImageNet. the size of the images in the ImaFeature transferring by taking geNet dataset is 224 × 224, we advantage of trained deep CNN resize all the images in the augmodels as a feature extractor has mented insulator dataset to this size using Python Numpy attracted a lot of attention recently and has become an and PIL libraries. Accordingly, we are able to feed the alternative solution to these sorts of limitations. By reusnew images into the same pretrained VGG-19. ing the knowledge (finalized weights) of pretrained netThe VGG-19 CNN is implemented using Keras, and the works as the feature extractor, deep CNNs can perform weights of convolutional, maxpooling, and first fully conwell on small datasets in other domains [54]. VGG-19 is nected layers are restored. Then, the two newly added one of the most common architectures used in transfer fully connected layers, including the Softmax classifier, learning-based methodologies for fault diagnosis in the litare randomly initialized. The weights of other layers of erature [34], [35], [55]. VGG-19 remain frozen and untrained during the new trainIn this study, a transfer learning technique is proing process, which is called fine-tuning. posed for TL insulator fault diagnosis by reusing the By using the aforementioned datasets, we could pretrained VGG-19 on the ImageNet dataset as the feadistinguish aerial images of broken and intact insulature extractor. The structure of VGG-19 is presented in tors with 99.93% accuracy, which is a reliable result. In Figure 5. The notion of “Conv3-64” implies that the filter the next section, we compare the proposed methodol(kernel) of the layer is 3 × 3, and its depth is 64. The ogy with existing benchmarks on these datasets. To pooling layers in this architecture are maxpooling, have a decent comparison between the outcome of the which store the maximum values to be fed into subseexisting CPLID dataset and the generated large dataquent layers. set, we trained the transferred CNN with these two To take advantage of the transfer learning technique, data sets a nd tested them with a common va r ia nt VGG-19 is implemented and transferred, and its layers dat a set. In the next section, we also generate an remain frozen during the training process of the insulator

224 × 224 × 64

Convolutional Layer

112 × 112 × 128

Maxpooling Layer Fully Connected Layer

56 × 56 × 256 28 × 28 × 512

2 × Conv3-64

2 × Conv3-128

4 × Conv3-256

4 × Conv3-512

14 × 14 × 512 7 × 7 × 512 4 × Conv3-512

2 × FC-4096 FC-1000

Figure 5. The architecture of a VGG-19 CNN.



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

21

imbalanced large dataset to prove the effect of making this dataset balanced. Simulation Results and Comparisons This section describes results of the proposed transfer learning-based method for classification of insulator images into two categories of broken and intact insulators. Simulations are run on a PC with Microsoft Windows 10. This PC uses an Intel Corei7-4710 MQ at a 2.50-GHz processor with 8 GB of random-access memory. The programming language used in this study is Python version 3.8. Keras 2.4 library [56] with TensorFlow 2.3 backend [57] is used to design VGG-19, and scikit-learn library [58] is used for classification modules. In this study, we compared our proposed methodology using transfer learning and image-augmentation techniques with five benchmarks from the literature and that were implemented for this study, which are offered in [1]. In [1], a modified model based on You Only Look Once (YOLO) is presented for insulator fault detection in aerial images with compound backgrounds. Initially, aerial images with few faults are collected in various scenes, and then a new dataset is established using Photoshop. To enhance feature reuse and propagation in low-resolution feature layers, a Cross Stage Partial Dense YOLO (CSPD-YOLO) model is presented based on YOLO-v3 and the Cross Stage Partial Network. Finally, the feature pyramid network and improved loss function are applied to

the CSPD-YOLO model to improve the accuracy of insulator fault detection. In this article, the CSPD-YOLO model and compared models in [1] are trained and tested on the generated dataset for comparison. We used these three networks to perform the first step of comparison in this study. Table 1 lists these comparison results. The purpose of this comparison was to show that the proposed method can perform well in half the time, which is needed for other methodologies. As indicated in Table 1, not only have the accuracy, precision, and F1 score been improved using our proposed transfer learning method, but this methodology also reduced the fault-detection time of insulator datasets. In this table, the running time shows the required time for one image classification between two classes of intact or broken insulator. The definitions of accuracy, precision, recall, and F1 score are as follows:

TP + TN Accuracy = TP + TN + FP + FN (1)



TP Precision = TP + FP (2)



TP Recall = TP + FN (3)



Recall # Precision F1 Score = 2 # Recall + Precision . (4)

In (1)–(3), TP is the number of correct predictions of minority samples, TN the number of correct predictions

Table 1. A comparison among existing CNNs and the proposed transfer learning and VGG-19 method by using the original CPLID. Neural Network

Number of Images

Accuracy (%)

Precision (%)

Recall (%)

F1 Score (%)

Test Time (s)

YOLO-v3 [59]

3,808

93.31

94

94

94

0.01

YOLO-v4 [59]

3,808

96.38

98

95

97

0.01

CSPD-YOLO [1]

3,808

98.18

99

98

99

0.011

Transferred VGG-19

3,808

99.28

99.62

99

99.3

0.006

Table 2. The proposed transfer learning and VGG-19 method: a comparison between the original and (balanced) augmented CPLID for training while using 20% of the augmented CPLID for testing. Accuracy (%)

Precision (%)

Recall (%)

F1 Score (%)

Test Time (s)

3,344 (20% of Augmented)

96.04

95.1

95.19

95.14

0.007

—

99.93

99.2

99.41

99.2

0.007

Neural Network

Train Dataset

Test Dataset

Transferred VGG-19

3,808 (Original)

Transferred VGG-19

13,376 (Balanced augmented)

22

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Table 3. The proposed transfer learning and VGG19 method: a comparison between the balanced and imbalanced augmented CPLID. Neural Network

Number of Images

Accuracy (%)

Precision (%)

Recall (%)

F1 Score (%)

Test Time (s)

Transferred VGG-19

17,800 (Imbalanced augmented)

93.22

92.9

92.11

92.5

0.007

Transferred VGG-19

16,720 (Balanced augmented)

99.93

99.2

99.41

of majority samples, FP the number of wrong predictions of minority samples, and FN the number of wrong predictions of majority samples. To compare the proposed transfer learning-based methodology with existing CNN benchmarks, namely, YOLO-v3, YOLO-v4 [59], and CSPDYOLO [1], the proposed method is examined on the original CPLID dataset as well. In the next step, to prove reliability of the proposed methodology for generating a larger dataset with image augmentation, we trained the proposed transferred CNN two times: once with the original CPLID dataset and another time with 80% of the large, generated dataset. For testing, the remaining 20% of the generated dataset is given to the transferred CNN trained with the two different datasets separately. Table 2 lists the results for this step. Table 2 shows that with the same testing datasets, the larger, inclusive dataset generates better results, as expected. In the third step, the effect of balancing the dataset under study is tested. For this purpose, we perform the image augmentation with an equal portion of broken insulator images and intact ones. Therefore, although the dataset becomes larger and more robust to feature variation of the images, it would be still an imbalanced dataset. To implement this step, four augmented images are generated from each image in the CPLID dataset. Hence, the overall number of images including the original and augmented ones becomes 17,800, which is close enough to the number of the unequally augmented large datasets with 16,720 insulator images. Table 3 shows the results of this step. For all the generated datasets mentioned in this section, 80% of the images are used for training and the remaining 20% are used for testing. As understood from Table 3, the results for the balanced dataset are much better than the case that the dataset is imbalanced. The results of these three steps show that the proposed method can detect each insulator image category within almost half of the time consumed by the other methods in the literature because of the used transfer learning technique. The fact that the specifications of the utilized computer in this study are lower than those used computers in the benchmarks, makes the result improvements in our

proposed method more noticeable. The comparison parameters show that the generated dataset outperforms the other benchmarks, and it is the most reliable approach up to now.

Conclusion In this article, a deep learning methodolog y wa s proposed ba sed on a tra nsfer lear ning technique to 99.2 0.007 improve the solution quality for an insulator image classification problem. One contribution of this study is that the original CPLID dataset has only 248 broken insulator images among 3,808 images, which puts this dataset in an imbalanced dataset category. Using a data-augmentation approach with different portions, a well-balanced dataset with 16,720 images was produced, which is a more suitable dataset compared to the original one with only 3,808 images. In this proposed transfer learning methodology, a VGG-19 CNN was implemented as a base model for transfer learning, which was trained using the ImageNet dataset. In the second step, the weights of VGG-19 layers, except the two fully connected final layers, were kept frozen to perform the feature-extraction task. Weights of the fully connected final layers were updated using the insulator image dataset for fine-tuning. This transferred VGG-19 CNN generated better accuracy results compared to benchmarks. The results showed that the proposed transfer learning technique was able to distinguish the intact and broken insulator images with more than 99.9% accuracy, and the required time for insulator image classification in the proposed technique was roughly half of the reported time required by the existing methods, thereby well demonstrating the importance of transfer learning in advancing the field of insulator image classification for TLs. Using some intelligent optimization methods [60]– [66] to select the best parameters given a user dataset is our next work. About the Authors Fatemeh Mohammadi Shakiba ([email protected]) earned her B.Sc. degree in computer engineering from Iran University of Science and Technology, Iran, in 2010, and her M.S. degree in electrical and computer engineering from Southern Illinois University, Illinois, USA, in 2018. She is currently working toward her Ph.D. degree in electrical and computer engineering at the New Jersey Institute of Technology, Newark, NJ 07172 USA. Her research interests include artificial neural networks, machine learning, deep learning, data analyzing and classification, and fault diagnosis. She is a Student Member of IEEE. S. Mohsen Azizi ([email protected]) earned his B.S. degree from Sharif University of Technology, Tehran, Iran in 2003, O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

23

and his M.S. and Ph.D. degrees from Concordia University, Montreal, Canada, in 2006 and 2011, respectively. He is with the Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102 USA, and the School of Applied Engineering and Technology, New Jersey Institute of Technology, Newark, NJ 07102 USA. His interests include cooperative and distributed control, networked estimation, and fault diagnosis and prognosis in microgrids and renewable energy systems. He is a Senior Member of IEEE. Mengchu Zhou ([email protected]) earned his B.S. degree from Nanjing University of Science and Technology, Nanjing, China in 1983, his M.S. from Beijing Institute of Technology, Beijing, China in 1986, and his Ph. D. degree from Rensselaer Polytechnic Institute, Troy, NY, USA, in 1990. He is with the Helen and John C. Hartmann Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102 USA. His research interests include intelligent automation, Petri nets, Internet of Things, edge/cloud computing, and big data analytics. He has more than 1,000 publications, including 12 books, more than 700 journal papers (more than 600 in IEEE transactions), 31 patents, and 30 book-chapters. He is Fellow of IEEE, IFAC, AAAS, CAA and NAI.

[10] W. Wang, Y. Wang, J. Han, and Y. Liu, “Recognition and drop-off detection of insulator based on aerial image,” in Proc. 9th IEEE Int. Symp. Comput. Intell. Design (ISCID), vol. 1, pp. 162–167, 2016, doi: 10.1109/ISCID.2016.1045. [11] A. Muzahid, W. Wan, F. Sohel, L. Wu, and L. Hou, “CurveNet: Curvature-based multitask learning deep networks for 3d object recognition,” IEEE/CAA J. Automat. Sinica, vol. 8, no. 6, pp. 1177–1187, 2020, doi: 10.1109/JAS.2020.1003324. [12] F. M. Shakiba and M. Zhou, “Novel analog implementation of a hyperbolic tangent neuron in artificial neural networks,” IEEE Trans. Ind. Electron., vol. 68, no. 11, pp. 10,856–10,867, Nov. 2020, doi: 10.1109/TIE.2020.3034856. [13] F. M. Shakiba, CMOS-Based Implementation of Hyperbolic Tangent Activation Function for Artificial Neural Network. Carbondale, IL, USA: Southern Illinois Univ. Carbondale, 2018. [14] Y. Wang, Z. Li, X. Yang, N. Luo, Y. Zhao, and G. Zhou, “Insulator defect recognition based on faster R-CNN,” in Proc. IEEE Int. Conf. Comput., Inf. Telecommun. Syst. (CITS), pp. 1–4, 2020, doi: 10.1109/CITS49457.2020.9232614. [15] X. Lei and Z. Sui, “Intelligent fault detection of high voltage line based on the faster R-CNN,” Measurement, vol. 138, pp. 379–385, May 2019, doi: 10.1016/j.measurement. 2019.01.072. [16] Y. Wang et al., “Image classification towards transmission line fault detection via learning deep quality-aware fine-grained categorization,” J. Vis. Commun. Image Represent., vol. 64, p. 102,647, Oct. 2019, doi: 10.1016/j.jvcir.2019.102647. [17] F. M. Shakiba, M. Shojaee, S. M. Azizi, and M. Zhou, “Generalized fault diagnosis method of transmission lines using transfer learning technique,” Neurocomputing, vol. 500, pp. 556–566, Aug. 2022, doi: 10.1016/j.neucom.2022.05.022. [18] S. R. Fahim et al., “Self attention convolutional neural network with time series imaging based feature extraction for transmission line fault detection and classification,” Elect. Power Syst. Res., vol. 187, p. 106,437, Oct. 2020, doi: 10.1016/j.

References

epsr.2020.106437.

[1] C. Liu, Y. Wu, J. Liu, Z. Sun, and H. Xu, “Insulator faults detection in aerial images

[19] P. Rai et al., “Fault classification in power system distribution network integrated

from high-voltage transmission lines based on deep learning model,” Appl. Sci., vol. 11,

with distributed generators using CNN,” Elect. Power Syst. Res., vol. 192, p. 106,914,

no. 10, p. 4647, 2021, doi: 10.3390/app11104647.

Mar. 2021, doi: 10.1016/j.epsr.2020.106914.

[2] Y. Liu, S. Pei, W. Fu, K. Zhang, X. Ji, and Z. Yin, “The discrimination method as

[20] S. Fuada et al., “A high-accuracy of transmission line faults (TLFs) classification

applied to a deteriorated porcelain insulator used in transmission lines on the basis of

based on convolutional neural network,” Int. J. Electron. Telecommun., vol. 66, no. 4,

a convolution neural network,” IEEE Trans. Dielectr. Electr. Insul., vol. 24, no. 6, pp.

pp. 655–664, 2020, doi: 10.24425/ijet.2020.134024.

3559–3566, Dec. 2017, doi: 10.1109/TDEI.2017.006840.

[21] H. A. Shiddieqy et al., “Power line transmission fault modeling and dataset gen-

[3] Z. Zhao, Z. Zhen, L. Zhang, Y. Qi, Y. Kong, and K. Zhang, “Insulator detection

eration for AI based automatic detection,” in Proc. 2019 Int. Symp. Electron. Smart

method in inspection image based on improved faster r-cnn,” Energies, vol. 12, no. 7, p.

Devices (ISESD), pp. 1–5, doi: 10.1109/ISESD.2019.8909594.

1204, 2019, doi: 10.3390/en12071204.

[22] K. Chen et al., “Detection and classification of transmission line faults based on

[4] Y. Zhai, D. Wang, M. Zhang, J. Wang, and F. Guo, “Fault detection of insulator based

unsupervised feature learning and convolutional sparse autoencoder,” IEEE Trans.

on saliency and adaptive morphology,” Multimedia Tools Appl., vol. 76, no. 9, pp.

Smart Grid, vol. 9, no. 3, pp. 1748–1758, May 2016, doi: 10.1109/TSG.2016.2598881.

12,051–12,064, 2017, doi: 10.1007/s11042-016-3981-2.

[23] E. F. Ohata et al., “Automatic detection of covid-19 infection using chest X-ray

[5] Y. Li, W. Zhang, P. Li, Y. Ning, and C. Suo, “A method for autonomous navigation and

images through transfer learning,” IEEE/CAA J. Automat. Sinica, vol. 8, no. 1,

positioning of UAV based on electric field array detection,” Sensors, vol. 21, no. 4, p.

pp. 239–248, Jan. 2020, doi: 10.1109/JAS.2020.1003393.

1146, 2021, doi: 10.3390/s21041146.

[24] Q. Kang, S. Yao, M. Zhou, K. Zhang, and A. Abusorrah, “Effective visual domain

[6] Y. Ma, Q. Li, L. Chu, Y. Zhou, and C. Xu, “Real-time detection and spatial localiza-

adaptation via generative adversarial distribution matching,” IEEE Trans. Neu-

tion of insulators for UAV inspection based on binocular stereo vision,” Remote Sens.,

ral Netw. Learn. Syst., vol. 32, no. 9, pp. 3919–3929, Sep. 2020, doi: 10.1109/

vol. 13, no. 2, p. 230, 2021, doi: 10.3390/rs13020230.

TNNLS.2020.3016180.

[7] H. Cheng, Y. Zhai, R. Chen, D. Wang, Z. Dong, and Y. Wang, “Self-shattering defect

[25] G. Cai, Y. Wang, L. He, and M. Zhou, “Unsupervised domain adaptation with adver-

detection of glass insulators based on spatial features,” Energies, vol. 12, no. 3, p. 543,

sarial residual transform networks,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31,

2019, doi: 10.3390/en12030543.

no. 8, pp. 3073–3086, Aug. 2019, doi: 10.1109/TNNLS.2019.2935384.

[8] Y. Zhai, R. Chen, Q. Yang, X. Li, and Z. Zhao, “Insulator fault detection based on

[26] G. Cai, L. He, M. Zhou, H. Alhumade, and D. Hu, “Learning smooth representation

spatial morphological features of aerial images,” IEEE Access, vol. 6, pp. 35,316–

for unsupervised domain adaptation,” IEEE Trans. Neural Netw. Learn. Syst., early

35,326, Jun. 2018, doi: 10.1109/ACCESS.2018.2846293.

access, 2021, doi: 10.1109/TNNLS.2021.3119889.

[9] S. Liao and J. An, “A robust insulator detection algorithm based on local features

[27] G. Wang, J. Qiao, J. Bi, W. Li, and M. Zhou, “Tl-GDBN: Growing deep belief net-

and spatial orders for aerial images,” IEEE Geosci. Remote Sens. Lett., vol. 12, no. 5,

work with transfer learning,” IEEE Trans. Autom. Sci. Eng., vol. 16, no. 2, pp. 874–885,

pp. 963–967, 2014, doi: 10.1109/LGRS.2014.2369525.

Apr. 2018, doi: 10.1109/TASE.2018.2865663.

24

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

[28] M. Zeng, M. Li, Z. Fei, Y. Yu, Y. Pan, and J. Wang, “Automatic ICD-9 coding via

[48] A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A survey of the recent archi-

deep transfer learning,” Neurocomputing, vol. 324, pp. 43–50, Jan. 2019, doi: 10.1016/j.

tectures of deep convolutional neural networks,” Artif. Intell. Rev., vol. 53, no. 8,

neucom.2018.04.081.

pp. 5455–5516, 2020, doi: 10.1007/s10462-020-09825-6.

[29] H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Transfer learn-

[49] V. Cheplygina, M. de Bruijne, and J. P. Pluim, “Not-so-supervised: A sur-

ing for time series classification,” in Proc. IEEE Int. Conf. Big Data (Big Data), 2018,

vey of semi-supervised, multi-instance, and transfer learning in medical image

pp. 1367–1376, doi: 10.1109/BigData.2018.8621990.

analysis,” Med. Image Anal., vol. 54, pp. 280–296, May 2019, doi: 10.1016/j.media.

[30] L. Guo et al., “Deep convolutional transfer learning network: A new method for

2019.03.009.

intelligent fault diagnosis of machines with unlabeled data,” IEEE Trans. Ind. Elec-

[50] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation

tron., vol. 66, no. 9, pp. 7316–7325, Sep. 2018, doi: 10.1109/TIE.2018.2877090.

for deep learning,” J. Big Data, vol. 6, no. 1, pp. 1–48, 2019, doi: 10.1186/s40537

[31] H. Shao et al., “Intelligent fault diagnosis of rotor-bearing system under varying working

-019-0197-0.

conditions with modified transfer convolutional neural network and thermal images,” IEEE

[51] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effective-

Trans. Ind. Informat., vol. 17, no. 5, pp. 3488–3496, May 2020, doi: 10.1109/TII.2020.3005965.

ness of data in deep learning era,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp.

[32] J. Li, R. Huang, G. He, S. Wang, G. Li, and W. Li, “A deep adversarial transfer

843–852, doi: 10.1109/ICCV.2017.97.

learning network for machinery emerging fault detection,” IEEE Sensors J., vol. 20, no.

[52] A. Raimundo, “Insulator data set - Chinese power line insulator dataset (CPLID),”

15, pp. 8413–8422, Aug. 2020, doi: 10.1109/JSEN.2020.2975286.

IEEE DataPort, 2020.

[33] M. Hussain, J. J. Bird, and D. R. Faria, “A study on CNN transfer learning for

[53] A. Mikołajczyk and M. Grochowski, “Data augmentation for improving deep learn-

image classification,” in U.K. Workshop on Computational Intelligence, A. Lotfi,

ing in image classification problem,” in Proc. Int. Interdisciplinary PhD Workshop

H. Bouchachia, A. Gegov, C. Langensiepen, and M. McGinnity, Eds. Cham: Springer

(IIPhDW), pp. 117–122, 2018, doi: 10.1109/IIPHDW.2018.8388338.

Nature Switzerland AG, 2018, pp. 191–202.

[54] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features

[34] L. Huang, Y. Yang, Y. Deng, and Y. Yu, “Densebox: Unifying landmark localization

in deep neural networks?” in Proc. Adv. Neural Inf. Process. Syst., 2014, vol. 27,

with end to end object detection,” 2015, arXiv:1509.04874.

pp. 3320–3328.

[35] Y. Li, Y. Zhang, Y. Xu, J. Wang, and Z. Miao, “Robust scale adaptive kernel cor-

[55] L. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neu-

relation filter tracker with hierarchical convolutional features,” IEEE Signal Process.

ral networks,” in Proc. Adv. Neural Inf. Process. Syst., 2015, pp. 262–270.

Lett., vol. 23, no. 8, pp. 1136–1140, Aug. 2016, doi: 10.1109/LSP.2016.2582783.

[56] F. Chollet et al., “Keras,” 2015.

[36] D. H. Hubel and T. N. Wiesel, “Receptive fields, binocular interaction and func-

[57] M. Abedi et al., TensorFlow: Large-Scale Machine Learning on Heterogeneous

tional architecture in the cat’s visual cortex,” J. Physiol., vol. 160, no. 1, pp. 106–154,

Systems. (2015). [Online]. Available: https://www.tensorflow.org/

1962, doi: 10.1113/jphysiol.1962.sp006837.

[58] F. Pedregosa et al., “Scikit-learn: Machine learning in python,” J. Mach. Learn.

[37] M. Kiruthika and S. Bindu, “Classification of electrical power system conditions

Res., vol. 12, pp. 2825–2830, Jan. 2011.

with convolutional neural networks,” Eng., Technol. Appl. Sci. Res., vol. 10, no. 3,

[59] L. Du et al., “Improved detection method for traffic signs in real scenes applied

pp. 5759–5768, 2020, doi: 10.48084/etasr.3512.

in intelligent and connected vehicles,” IET Intell. Transp. Syst., vol. 14, no. 12, pp.

[38] S. Shao et al., “Highly accurate machine fault diagnosis using deep transfer learn-

1555–1564, 2020, doi: 10.1049/iet-its.2019.0475.

ing,” IEEE Trans. Ind. Informat., vol. 15, no. 4, pp. 2446–2455, Apr. 2018, doi: 10.1109/

[60] J. Tang, G. Liu, and Q. Pan, “A review on representative swarm intelligence

TII.2018.2864759.

algorithms for solving optimization problems: Applications and trends,” IEEE/CAA

[39] D. Zhiyong et al., “Fast and accurate cable detection using CNN,” Appl. Intell., vol.

J. Automat. Sinica, vol. 8, no. 10, pp. 1627–1643, Oct. 2021, doi: 10.1109/JAS.2021.1004129.

50, no. 12, pp. 4688–4707, 2020, doi: 10.1007/s10489-020-01746-9.

[61] Y. Wang, S. Gao, M. Zhou, and Y. Yu, “A multi-layered gravitational search algo-

[40] C. Alippi, S. Disabato, and M. Roveri, “Moving convolutional neural networks to

rithm for function optimization and real-world problems,” IEEE/CAA J. Automat.

embedded systems: The AlexNet and VGG-16 case,” in Proc. 17th ACM/IEEE Int. Conf.

Sinica, vol. 8, no. 1, pp. 94–109, Jan. 2020, doi: 10.1109/JAS.2020.1003462.

Inf. Process. Sensor Netw. (IPSN), pp. 212–223, doi: 10.1109/IPSN.2018.00049.

[62] H. Yuan, M. Zhou, Q. Liu, and A. Abusorrah, “Fine-grained resource provision-

[41] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale

ing and task scheduling for heterogeneous applications in distributed green clouds,”

image recognition,” 2014, arXiv: 1409.1556.

IEEE/CAA J. Automat. Sinica, vol. 7, no. 5, pp. 1380–1393, 2020, doi: 10.1109/

[42] M. A. Morid, A. Borjali, and G. Del Fiol, “A scoping review of transfer learning

JAS.2020.1003177.

research on medical image analysis using ImageNet,” Comput. Biol. Med., vol. 128, p.

[63] G. Tian, Y. Ren, and M. Zhou, “Dual-objective scheduling of rescue vehicles to

104,115, Jan. 2021, doi: 10.1016/j.compbiomed.2020.104115.

distinguish forest fires via differential evolution and particle swarm optimization com-

[43] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data

bined algorithm,” IEEE Trans. Intell. Transp. Syst., vol. 17, no. 11, pp. 3009–3021, Nov.

Eng., vol. 22, no. 10, pp. 1345–1359, Oct. 2009, doi: 10.1109/TKDE.2009.191.

2016, doi: 10.1109/TITS.2015.2505323.

[44] H. Zheng et al., “Cross-domain fault diagnosis using knowledge transfer strat-

[64] Z. Zhao, M. Zhou, and S. Liu, “Iterated greedy algorithms for flow-shop schedul-

egy: A review,” IEEE Access, vol. 7, pp. 129,260–129,290, Sep. 2019, doi: 10.1109/

ing problems: A tutorial,” IEEE Trans. Autom. Sci. Eng., vol. 19, no. 3, pp. 1941–1959,

ACCESS.2019.2939876.

Jul. 2021, doi: 10.1109/TASE.2021.3062994.

[45] G. Csurka, “Domain adaptation for visual applications: A comprehensive survey,”

[65] A. C. Ammari, W. Labidi, F. Mnif, H. Yuan, M. Zhou, and M. Sarrab, “Firefly algo-

2017, arXiv:1702.05374.

rithm and learning-based geographical task scheduling for operational cost minimiza-

[46] C. Fellbaum, “Wordnet,” in Theory and Applications of Ontology: Computer

tion in distributed green data centers,” Neurocomputing, vol. 490, no. C, pp. 146–162,

Applications, R. Poli, M. Healy, and A. Kameas, Eds. New York, NY, USA: Springer Sci-

Jun. 2022, doi: 10.1016/j.neucom.2022.01.052.

ence & Business Media, 2010, pp. 231–243.

[66] Y. Fu, M. Zhou, X. Guo, and L. Qi, “Scheduling dual-objective stochastic hybrid

[47] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-

flow shop with deteriorating jobs via bi-population evolutionary algorithm,” IEEE

scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog-

Trans. Syst., Man, Cybern. Syst., vol. 50, no. 12, pp. 5037–5048, Dec. 2020, doi: 10.1109/

nit., pp. 248–255, 2000, doi: 10.1109/CVPR.2009.5206848.

TSMC.2019.2907575.



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

25

by R. Balamurugan , Kshitiz Choudhary, and S.P. Raja

F

loods are one of the deadliest disasters in the coastal areas of India. Consistently, flood, the most widely recognized catastrophe in India, has an enormous effect on the nation’s property and lives. Therefore, this article is focused on developing an effective flood-prediction system using machine learning (ML) algorithms that can help with preventing the loss of human lives and property. We will use k-nearest neighbors (KNNs), support vector machines (SVMs), random forests (RFs), and decision trees (DTs) to build our ML models. And to resolve the issue of oversampling and low accuracy, a stacking classifier will be used. For comparison among these models, we will use accuracy, f1-scores, recall, and precision. The results indicate that stacked models are best for predicting floods due to real-time rainfall in that area. It is

Prediction of Flooding Due to Heavy Rainfall in India Using Machine Learning Algorithms Digital Object Identifier 10.1109/MSMC.2022.3183806 Date of current version: 20 October 2022

26

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Providing Advanced Warning 2333-942X/22©2022IEEE

[2]. Out of the total geographical area of 329 million hectares (mha), it is seen that approximately 40 mha are flood prone. In the last 10 years, it is estimated that annual flood damage cost was roughly 4,745 crore rupees as compared to 1,805 crore rupees, which is an annual average of the previous 53 years. On average every year, ©SHUTTERSTOCK.COM/MTD_MYTRAVELDIARIES

approximately 75 mha of Indian land are affected, whereas 1,600 human lives are lost due to floods. This not only damages crops, houses, and public utilities but also brings panic to every human being affected by it. In the past few years, the frequency of major floods has increased by more than twice. It is therefore important to have a flood-prediction system in India that can provide an early noted that Andhra Pradesh achieves the highest accuracy

warning to all people residing in impacted areas so that they

of 97.91%, whereas Orissa achieves an accuracy of 92.36%,

evacuate as soon as possible [3], [4].

lowest among the eight coastal states. Literature Review Introduction

Kunverji et al. [5] aimed at building an operative, flood-deci-

Floods are one of the most common types of natural calam-

sive prototype. The authors used three supervised ML tech-

ities and occur when water overflows and submerges the

niques: DTs, RFs, and gradient boost algorithms for

land, which is typically waterless [1], [2], [6]. They are

developing a forecast model. The data set used in their work

habitually triggered by swift snowmelt, or by storm surges

was derived from the Indian Water Portal for Bihar and

from a heavy rainfall, a tropical cyclone, or a tsunami in the

Orissa. It was observed that DTs performed best out of all

coastline. Second to China, India

the algorithms used, with a 94.4%

has the most recorded floods every

accuracy. The ML algorithms [6]

year, an average of 17, and more

Being a peninsular country encircled by the Arabian Sea in the southwest, the Indian Ocean in the south, and the Bay of Bengal in the southeast, India is especially inclined to flooding.

than 345 million of its population has been impacted by them in the past 20 years. Heavy rainfall has always contributed to floods, known as flash floods. India is highly vulnerable to floods. Being a peninsular country encircled by the Arabian Sea in the southwest, the Indian Ocean in the south, and the Bay of Bengal in the southeast, India is especially inclined to flooding. According to the Geological Survey of India, the majority of the floodinclined areas of India covers practi-

used for predicting floods have employed temperature and rainfallintensity parameters. Deep learning models were compared to the supervised learning methods, namely, KNNs, naive Bayes, and SVMs. Thus, a recent data set is required to achieve better accuracy. The data set used in this research paper was derived from India Water Portal. It was observed that deep neural networks outperformed the other algorithms, achieving 91.18% accuracy. The analysts intended to take advantage of information accessible from the Geographical Information

cally a 12.5% region of the country



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

27

System (GIS) and mechanical progression in the cuttingmodels outperformed the RBF-FA and RBF models in preedge world in delivering a solid flood weakness and likelidicting river floods. The goal [12] was to discover the best supervised learning hood map [7]. The data set used in this research paper was derived from the GIS. The authors used a convolutional neumodel configuration. The most relevant prediction factors, according to the analysis, are water levels at both the surral network (CNN) and an SVM for their research and rounding stations and in the control found that the CNN was highly effective in spatial resolution station’s prehistory. The amount of rain that falls was proven to be a imagery, whereas the SVM has the poor predictor of flooding situaability to predict using linear data. Using social media tions. With little experimental data And also, when an SVM is comand unknown important variables, bined with a CNN, a robust flood data, emergency this article presented a challenge. map is obtained. The analysts veriwarning response Using social media data, emergency fied that a probabilistic way of management can warning response management can dealing with investigating the employ imperviousness maps to future gamble of seaside flooding employ imperviousness locate vital areas during pluvial would be successful to help decimaps to locate vital flooding disasters. The optimum sion production for a coordinated model was selected and 11 flood beachfront zone of the board [8]. areas during pluvial model combinations were used [13]. They went through different flooding disasters. The following conclusions can be machine techniques applied to the drawn from the findings: the LR flood data set in southern Korea to model has a higher prediction rate predict floods. The data set used [area under the receiver operating in this research paper was derived characteristic (AUROC of 86.8%)] than the other models. To from the official Korean Government website. The results investigate flash flood-prediction mapping, researchers used showed that KNNs outperformed the other algorithms, two supervised learning models [14]: extreme gradient achieving 94.6% accuracy. boosting (XGB) and KNNs. The XGB and KNN algorithms Rani et al. [9] aimed to build an effective method that achieved ROC area under the curve accuracies of 90.2 and can detect floods in the area and alert citizens. Logistic 80.7%, respectively, with XGBoost’s ability to provide more regression, linear regression (LR), SVMs, and neural netoutputs enabling greater accuracy. In future research, it is works are used to detect floods, and their effectiveness is hoped that using other optimization algorithms can increase determined using mean absolute error (MAE) and standard the model’s performance. Furthermore, distance from the deviation. The researchers used a data set from the Indian stream network, slope, topography, and topographic wetness Meteorological Department. In the output, it is observed index (TWI) conditioning factors were found to be the most that neural networks performed better than the other ML influential components in the modeling procedures. algorithms, achieving an MAE of 21.809, whereas the SVM Tehrany et al. [15] suggest that including conditionachieved 90.606, and logistic regression achieved 40.246. based factors in addition to the lidar-derived parameters In [10], a multilayer perceptron based on a faster regiondoes not always improve outcome precision, according to based CNN (R-CNN) is proposed to detect coastal garbage. their research. According to some academics, altitude is a It is hard to synchronize a variety of settings to obtain a crucial factor in flooding. This provides alternate options typically faster R-CNN performance. Fusing high-resolution features with high-dimensional ones from a low-resolution for improving and strengthening the flood monitoring image could be used to detect small objects. To solve posimodel in Malaysia. By merging the computational eletion offset, region of interest (RoI) align instead of RoI ments, the model was effectively constructed. If the water pooling could improve the performance of automated exceeds a specific level, the system can automatically iscoastal waste detection. The accuracy of a radial basis sue a warning message. Furthermore, the device can be function-firefly algorithm (RBF-FA) and support vector used to monitor flash floods as well. However, the system machine-firefly algorithm (SVM-FA) models for river-flood has yet to be tested in a real-life situation in a flood-affectprediction was also investigated [11]. By combining an FA ed area. To be more efficient, future work should take into with an RBF and an SVM, RBF-FA and SVM-FA models account the following factors: 1) data transmission (e.g., were created. There were two resolution levels used in this wireless communication), 2) multifunctional messages, study. These methods were tested using monthly river flow and 3) the Android app, the construction of which (for exdata from Silchar and Dholai. In comparison to regular ample, a flood-control system) is greatly anticipated, givSVM and RBF models that consider all nodes, statistical en Malaysia’s huge advancements in information and metrics used to examine the error of the methods exhibit communication technology. Mosavi et al. [16] proposed a lower root-mean-square error and greater R2. The outcomes method that will surpass the probability mapping system and generate f lood susceptibility. By using CNN’s of the assessment also demonstrated that SVM-FA and SVM 28

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

metamodels (called level 1 models) that consolidate forecasts of the fundamental model. Level 0 (base models) models are those that fit the preparation information and have the expectations accumulated. Level 1 (meta model) models are those that figure out how to join the expectations of the fundamental model ideally. The metamodel is prepared using the expectations made by the base model in view of the There are standard information outside the example. Proposed Method ML algorithms That is, information that isn’t utiThe climate of India is divided into that are used in lized for preparing the base model four parts. The first one is winter, is taken ca re of for the ba se which occurs from December to the research work, model, expectations are made, February; the second is summer or namely, SVMs, KNNs, and these forecasts give the inforpremonsoon, which lasts from mation and result sets of the prepMarch to May; the third is MonDTs, and RFs. aration data set, alongside the soon or rainy, lasting from June to normal result. The result of the September; and the four th is base model utilized as a contribuautumn or postmonsoon, which tion to the metamodel is a genuine number for relapse, occurs from October to November. India receives its heavian incentive for likelihood values, and a class name for est rainfall in June–September. This period causes floods characterization. The most widely recognized way to in certain areas due to a sudden increase in rainfall water deal with setting up a metamodel-preparing a data set is levels. Rainfall is the highest affecting factor in the cause the base-model k-overlay cross approval, which utilizes of floods in any area. out-of-overlap expectations regarding the reason for the We use two data sets. The first is obtained from the metamodel-preparing data set. The preparation informaIndian Meteorological Department, which is a statewise tion for the metamodel can likewise incorporate a contrirainfall data set, and the other data set is obtained from bution for the base model. Then enter the components of India Flood Inventory. The data set [17] used for the prethe preparation information. diction system is the first data set, whereas the other Once the metamodel-preparing data set is ready, the data set [18] is used for the exploratory analysis of floods metamodel can be prepared separately on that data set and their impact on India over the years. The data set and the base model can be arranged on the whole unique consists of nine coastal states and their respective rainpreparation data set. Stacking is reasonable when a few fall statistics from the years 1901 to 2018. The data set different artificial-intelligence models have usefulness was then split into a 67:33 ratio, wherein training data for a data set, however in various ways. Ultimately, the represented 67% of the total data. Four supervised algomodel foreca st or model ex pectation mistake is rithms are used to train the data set of the model. There are standard ML algorithms that are used in the research work, namely, SVMs, KNNs, DTs, and RFs. This was done to Problem Definition determine which of the algorithms works best with the data set. capacity to work with high-dimensional images, objects of interest result in having solid spatial structure and also help in findings of other studies, which gives an advantage. An SVM is also capable of predicting nonlinear data. The combination of these two dissimilar network a rchitectures will result in a flood map that is both effective and reliable.

Data Preprocessing

Data Set

Stacking Classifier Stacking is a gathering learning method for consolidating numerous order models by means of a metaclassifier. Individual arrangement models are prepared based on a total preparation set. The metaclassifier is then tuned in light of the resulting meta capacity of the whole individual arrangement model. It can be prepared utilizing either the anticipated class mark or the likelihood from the group. This engineering model incorporates a t le a s t t wo e s s ent i a l model s a nd

Train Test Split

Model Definition Best Model Selected

Metrics

Train Model No

High Bias

Yes

Figure 1. The architecture of the model.



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

29

less than the threshold rainfall uncorrelated or inadequately conlevel in June – September. The nected. Direct models, choice Stacking is threshold describes normal raintrees, SVMs, brain organizations, fall in the area, so, if supposedly and so on can also be used. Other a gathering the rainfall water level is higher troupe calculations, like RF, can learning method in that area at that time, it will likewise be utilized as the base send an alert to all the residents model. Base models use various for consolidating in that area. Every area has their models to make diverse suspinumerous order respective threshold water level, cions about the prescient assignand ba sed on that, the ta rget ment. Metamodels are many times models by means of class is determined. more straightforward and give a a metaclassifier. After defining all the inputs smooth translation of the base and output, one by one, all the model’s ex pectations. Conse supervised ML algorithms are testquently, direct models are freed on that. As all supervised ML quently utilized as metamodels. A algorithms are being applied, some of them show an straight relapse for relapse undertakings (anticipating overfitting issue and some of them score less accurately. numbers) and a strategic relapse for characterization So, to overcome this issue, GridSearchCV was applied. In errands (foreseeing class marks) is also utilized. This is the KNN algorithm, the param_grid ‘n_neighbors,’ normal, yet at the same time not needed. ‘weights,’ and ‘metric’ was taken. The value that is considered under ‘n_neighbors’ is in the form of a list ranging Architecture from 1 to 31. Similarly, for ‘weights,’ the value considered The architecture of the flood-prediction system is illusis ‘uniform’ and ‘distance.’ For ‘metric,’ the values are trated in Figure 1. The Indian Meteorological Depart‘euclidean’ and ‘manhattan.’ ment data set consists of data from nine states from In the SVM algorithm, param_grid contains ‘C,’ ‘gamma,’ 1901 to 2018. There are 18 total features in the data set. and ‘kernel.’ For ‘C,’ the values are [0.1, 1, 10, 100, 1,000], For our work, the attributes used in the model are whereas for ‘gamma,’ the values are [1, 0.1, 0.01, 0.001, June, July, August, and September. As this is a classifi0.0001], and for ‘kernel’ the values are ‘off’ and ‘sigmoid.’ In cation problem, the output is in the form of 0 and 1, the DT algorithm, param_grid contains ‘criterion’ and where “0” is when the average rainfall of monsoon sea‘max_depth.’ For ‘criterion,’ the values are ‘Gini’ and ‘entrosonal month (June– September) is greater than the py,’ whereas in ‘max_depth,’ the value ranges from 2 to 16. threshold rainfall level in June–September of the specific In the RF algorithm, param_grid contains four area (see Table 1). Similarly, “1” is when the average rainparameters: ‘n_estimators,’ ‘max_features,’ ‘max_depth,’ fall of monsoon seasonal month (June–September ) is

Table 1. The threshold rainfall water level.

Table 2. Evaluation metrics of the stacked model.

State Number

State

Threshold (Normal) Rainfall Water Level

1

Kerala

2,049.3

2

Coastal Karnataka

3

30

States

Recall

Precision

F1-Score

Accuracy

Kerala

92.51

92.06

92.22

92.3

Coastal Karnataka

94.42

94.42

94.42

94.87

3,095.1

94.84

94.84

94.87

751.2

Madhya Maharashtra

94.84

Madhya Maharashtra

94.42

94.42

94.87

Gangetic West Bengal

1,181.5

Gangetic West Bengal

94.42

4

Orissa

92.36

92.36

92.3

92.3

5

Orissa

1,155.3

Tamil Nadu

94.44

95.65

94.78

94.87

6

Tamil Nadu

336.1

Gujarat

92.52

91.84

92.12

92.3

7

Gujarat

922.9

94.7

94.7

94.7

94.87

8

Konkan and Goa

2,914.3

Konkan and Goa

9

Andhra Pradesh

586.9

Andhra Pradesh

97.91

96.87

97.32

97.43

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Results and Discussions The result of the analytical model is validated using an anaconda simulation tool. The analysis metrics utilized in this work are accuracy, exactness score, recall, and f1-score. These metrics will also be used for comparison among the models. The data set considered for our work is the Indian Meteorological department, which contains 19 features and 1,062 data points. The results we obtained from experiments for the stacking model and other ML approaches, RFs, DTs, KNNs, and SVMs, are analyzed with respect to various accuracy parameters. The data were preprocessed using standard normalization to scale values in the range of 0 to 1. After applying KNNs, SVMs,

DTs, and RFs, it was observed that DTs scored lower on exactness, accuracy, review, and f-measure as well as also facing the oversampling issue. To overcome oversampling and achieve higher accuracy, the GridSearchCV method was applied to every four algorithm. Under this method, every algorithm had a parameter grid that includes different parameters of every classifier algorithm and its list of respective values. Along with that, 10-fold cross validation was applied. At the end, a stacking classifier was applied where the estimator’s value is the list of all the modified models, and the final estimator or metaestimator is an RF classifier. Table 2 shows the

roc Plot 1 True-Positive Rate

and ‘criterion.’ For ‘n_estimators,’ the values are [5, 10, 50, 100, 200]. For ‘max_features,’ the values are ‘auto,’ ‘sqrt,’ and ‘log2,’ whereas for ‘criterion,’ the values are ‘Gini’ and ‘entropy,’ and the values for ‘max_depth’ values are [2] and [4]–[8]. After running GridSearchCV on every algorithm, the best parameters are determined, which will give maximum frequency for the data set as well as run cross validation on the data set, reducing the overfitting issue. A stacking classifier is then defined to combine all the classification models (KNN, SVM, DT, and RF) as level 0/ base models and a level 1/metamodel as an RF classifier. The accuracy is calculated on both the training and test data. In the evaluation phase, the accuracy, classification report, f1-score, recall, and precision are determined for their effectiveness with the problem statement and applicability. An ROC bend is a graphical portrayal of a paired classifier’s symptomatic limit used. A discrete classifier that mainly returns the anticipated class delivers a solitary ROC point.

0.8 0.6 0.4 0.2 0 0

0.2

1

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.841) SVM (AUROC = 1) DT (AUROC = 0.813) RF (AUROC = 0.884) Figure 3. An ROC plot of Tamil Nadu.

roc Plot

roc Plot

1

1

0.8

0.8

True-Positive Rate

True-Positive Rate

0.4 0.6 0.8 False-Positive Rate

0.6 0.4 0.2 0

0.6 0.4 0.2 0

0

0.2

0.4 0.6 0.8 False-Positive Rate

1

0

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.957) SVM (AUROC = 0.989) DT (AUROC = 0.721) RF (AUROC = 0.911) Figure 2. An ROC plot of West Bengal.

0.2

0.4 0.6 0.8 False-Positive Rate

1

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.939) SVM (AUROC = 0.997) DT (AUROC = 0.717) RF (AUROC = 0.934) Figure 4. An ROC plot of Orissa.



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

31

experimental outcome of the stacked technique of every state. From the plots in Figures 2–10, it is observed that a DT performs worst among the rest of the ML algorithms considered. The overall results showed that the stacked model performed better than the KNN, SVM, DT, and RF, achieving the highest exactness, accuracy, review, and f-measure. At the same time, the SVM performed best in every state flood prediction. Conclusion The world’s second-worst impacted country after Bangladesh is India, accounting for one-fifth of all flood-related deaths worldwide. The Indian rainy season lasts mostly

from June to September, which contributes to roughly 75% of the country’s annual rainfall. From the AUROC plot of different states, it is easily observed that the DT performed worse than the other three ML algorithms used in the research work. Using a stacking classifier model, the problem of oversampling as well as low accuracy was resolved, and also higher accuracy, f1-score, recall, and precision are achieved compared to all the other MLsupervised algorithms. It is hoped that this work will help government and nongovernment associations make a preventive move on the flood peculiarity that ordinarily happens because of weighty precipitation in beachfront places of India.

roc Plot 1

0.8

0.8

True-Positive Rate

True-Positive Rate

roc Plot 1

0.6 0.4 0.2 0

0.6 0.4 0.2 0

0

0.2

0.4 0.6 0.8 False-Positive Rate

1

0

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.979) SVM (AUROC = 0.997) DT (AUROC = 0.837) RF (AUROC = 0.968)

0.2

Figure 7. An ROC plot of Coastal Karnataka.

roc Plot

roc Plot 1

1

0.8

0.8

True-Positive Rate

True-Positive Rate

1

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.963) SVM (AUROC = 0.991) DT (AUROC = 0.741) RF (AUROC = 0.901)

Figure 5. An ROC plot of Madhya Maharashtra.

0.6 0.4 0.2

0.6 0.4 0.2 0

0 0

0.2

0.4 0.6 0.8 False-Positive Rate

1

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.93) SVM (AUROC = 0.984) DT (AUROC = 0.801) RF (AUROC = 0.925) Figure 6. An ROC plot of Kerala. 32

0.4 0.6 0.8 False-Positive Rate

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

0

0.2

0.4 0.6 0.8 False-Positive Rate

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.986) SVM (AUROC = 0.995) DT (AUROC = 0.788) RF (AUROC = 0.962) Figure 8. An ROC plot of Gujarat.

1

puter science and engineering at Vellore Institute of Technology, Vellore-632 014, Tamil Nadu, India.

roc Plot

True-Positive Rate

1

References

0.8

[1] M. Zarei, O. Bozorg-Haddad, S. Baghban, M. Delpasand, E. Goharian, and H. A. Loáiciga, “Machine-learning algorithms for forecast-informed reservoir operation

0.6

(FIRO) to reduce flood damages,” Sci. Rep., vol. 11, 2021, Art. no. 24295.

0.4

[2] “Using AI to predict floods and save lives.” INDIAai. https://indiaai.gov.in/casestudy/using-ai-to-predict-floods-and-save-lives (Accessed: 2022).

0.2

[3] A. O. Hashi, A. A. Abdirahman, M. A. Elmi, and S. Z. Mohd, “A real-time flood detection system based on machine learning algorithms with emphasis on deep learning,” Int. J. Eng.

0 0

0.2

0.4 0.6 0.8 False-Positive Rate

1

Trends Technol., vol. 69, no. 5, pp. 249–256, 2021, doi: 10.14445/22315381/IJETT-V69I5P232. [4] H. Abdul-Kader and M. Mohamed, “Hybrid machine learning model for rainfall forecasting,” J. Intell. Syst. Internet Things, vol. 1, no. 1, pp. 5–12, 2021, doi: 10.54216/

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.978) SVM (AUROC = 0.984) DT (AUROC = 0.788) RF (AUROC = 0.927)

JISIoT.010101. [5] K. Kunverji, K. Shah, and N. Shah, “A flood prediction system developed using various machine learning algorithms,” in Proc. 2021 4th Int. Conf. Adv. Sci. Technol. (ICAST). [6] S. Sankaranarayanan, M. Prabhakar, S. Satish, P. Jain, A. Ramprasad, and A. Krishnan, “Flood prediction based on weather parameters using deep learning,” J. Water

Figure 9. An ROC plot of Konkan and Goa.

Climate Change, vol. 11, no. 4, pp. 1–18, 2019, doi: 10.2166/wcc.2019.321. [7] J.M. Opella and A. Hernandez, “Developing a flood risk assessment using support vector machine and convolutional neural network: A conceptual framework,” in Proc. 2019

roc Plot

IEEE 15th Int. Colloq. Signal Process. Appl. (CSPA), pp. 260–265, doi: 10.1109/CSPA.2019.

True-Positive Rate

1

8695980.

0.8

[8] S. Park and D. Lee, “Prediction of coastal flooding risk under climate change

0.6

15, no. 9, p. 10, 2020, doi: 10.1088/1748-9326/aba5b3.

impacts in South Korea using machine learning algorithms,” Environ. Res. Lett., vol. [9] D.S. Rani, G. Jayalakshmi, and V.P. Baligar, “Low cost IoT based flood monitoring

0.4

system using machine learning and neural networks: Flood alerting and rainfall prediction,” in Proc. 2020 Int. Conf. Innov. Mech. Ind. Appl. (ICIMIA), pp. 261–267, doi:

0.2

10.1109/ICIMIA48430.2020.9074928.

0

[10] C. Ren, H. Jung, S. Lee, and D. Jeong, “Coastal waste detection based on deep

0

0.2

0.4 0.6 0.8 False-Positive Rate

1

Random Prediction (AUROC = 0.5) KNN (AUROC = 0.983) SVM (AUROC = 1) DT (AUROC = 0.842) RF (AUROC = 0.994) Figure 10. An ROC plot of Andhra Pradesh.

convolutional neural networks,” Sensors, vol. 21, no. 21, pp. 1–17, 2021, doi: 10.3390/ s21217269. [11] A. Sahoo, S. Samantaray, and D. K. Ghose, “Prediction of flood in Barak river using hybrid machine learning approaches: A case study,” J. Geol. Soc. India, vol. 97, no. 2, pp. 186–198, 2021, doi: 10.1007/s12594-021-1650-1. [12] N. Jeerana, N. O. Nikitin, and A. Kalyuzhnaya, “Urban pluvial flood forecasting using open data with machine learning techniques in Pattani basin,” Proc. Comput. Sci., vol. 119, pp. 288–297, Dec. 2017, doi: 10.1016/j.procs.2017.11.187. [13] M. Rahman, N. Chen, and A. Washakh, “Flood susceptibility assessment in Bangladesh using machine learning and multi-criteria decision analysis,” Earth Syst. Envi-

About the Authors R. Balamurugan ([email protected]) earned his Ph.D. degree in data mining from Anna University, Chennai, India, in 2016. He is an associate professor of computer science and engineering at Vellore Institute of Technology, Vellore-632 014, Tamil Nadu, India. Kshitiz Choudhary ([email protected]) earned his B.Tech. in computer science and engineering from Vellore Institute of Technology, Vellore-632 014, Tamil Nadu, India. He is affiliated with the same institution. S.P. Raja ([email protected]) earned his Ph.D. degree in image processing from Manonmaniam Sundaranar University, Tirunelveli. He is an associate professor of com

ron., vol. 3, no. 3, pp. 585–601, 2019, doi: 10.1007/s41748-019-00123-y. [14] S. A. A. El-Magd, B. Pradhan, and A. Alamri, “Machine learning algorithm for flash flood prediction mapping in Wadi El-Laqeita and surroundings, Central Eastern Desert,” Egypt Arabian J. Geosci., vol. 14, no. 4, p. 323, 2021, doi: 10.1007/s12517-021-06466-z. [15] M. S. Tehrany, S. Jones, and F. Shabani, “Identifying the essential flood conditioning factors for flood prone area mapping using machine learning techniques,” Catena, vol. 175, pp. 174–192, Apr. 2019, doi: 10.1016/j.catena.2018.12.011. [16] A. Mosavi, P. Ozturk, and K. Chau, “Flood prediction using machine learning models: Literature review,” Water, vol. 10, no. 11, pp. 1–41, 2018, doi: 10.3390/w10111536. [17] “Rainfall dataset,” India Meteorological Department, New Delhi, India. Accessed: 2022. [Online]. Available: https://mausam.imd.gov.in/ [18] “India flood inventory: HydroSense Lab, IIT Delhi.” GitHub. https://github.com/ hydrosenselab/India-Flood-Inventory (Accessed: 2022.)

 O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

33

Special Section Editorial

by Gaurav Dhiman , Atulya Nagar  , and Seifedine Kadry 

Explainable Artificial Intelligence for the Social Internet of Things Analysis and Modeling Using

N

ex t - gener a t ion ex pl a i n able artificial intelligence (XAI) is rapidly combining machine intelligence and human intelligence to generate social intelligence, which supports the rising interactions between social spaces. With transformative advances in science and society, a new challenge emerges when attempting to understand the basic mechanisms and principles of the evolving multidimensional reality. Next-generation wireless communication, such as the Internet of Things (IoT), will play a vital role in supporting complex wireless interconnectivity [1]. Social IoT (SIoT) systems promise to enable ubiquitous connectivity among users by merging human social behaviors with physical IoT devices. This vast variety and variability, combined with the pervasive expansion of the IoT, presents new difficulties for academia, business, and standards organizations to handle. Adding more semantics to smart objects, on the other hand, can enable them to undertake previously inconceivable tasks and activities [2]. The social interaction of SIoT objects contributes a large volume of data to be processed and used by various applications in the areas of smart cities, smart homes,

Digital Object Identifier 10.1109/MSMC.2022.3198845 Date of current version: 20 October 2022

34

Sugandh et al. provide a thorough analysis of the use of merging blockchain and the IoT in the creation of smart applications for smart agriculture. The study is broken into three parts. The first section contains a thorough survey and bibliometric study of blockchain in agriculture. Then, the authors present the issues that Indian farmers confront as well as the study Collaborative Technologies aims and research questions. Finally, the study develops a novel blockchain model that has the potential to smart grids, and smart factories to be used as a substantial solution to satisfy human needs, interests, and important issues in IoT-based smart objectives, such as social vehicular agricultural systems. The study also ad hoc networks, social connected assesses and extensively addresses health, SIoT-based recommendation the primary roles and strengths of the services, traffic services, policing, enmost prevalent blockchain platforms ergy management, and so on. used in managing various subsectors Such a dynamic landscape, with in smart agriculture, such as crops, billions of social communities of oblivestock grazing, and the food supply jects and devices, necessitates the dechain, among others. velopment of new models, theories, Alotaibi and Yadav and approaches to solve concerns by interaction and coldeveloping a social laboration, which Social IoT systems could be established network information promise to enable by drawing on the transmission model experience that peothat incorporates ubiquitous ple have gained in XAI and is compatible connectivity the social networkwith a normal comamong users by ing domain over the munication connecmerging human past few years [3]. tion. They propose a Given these oppormethod of informasocial behaviors tunities, this special tion transfer known with physical section invited novel as local greedy, which IoT devices. research and practiassists in the presercal papers that imvation of user privacy. prove SIoT systems Its influence serves as using XAI methods in terms of architeca buffer between the competing goals ture, technology, and applications. This of privacy protection and information special section on “Explainable Artifidissemination. To address the enumeracial Intelligence for the Social Internet tion problem of seed set selection, an of Things: Analysis and Modeling Using incremental strategy for creating seed Collaborative Technologies” of IEEE sets with minimal time overhead is Systems, Man, & Cybernetics Magazine described; a local influence subgraph attracted three outstanding research method for computing nodes is also proarticles discussing new research reposed to efficiently evaluate the influsults covering a wide range of the SIoT. ence of seed set propagation. The group

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

with link prediction show that the multisource-information, combinedk nowledge representation learning (XAI–convolutional neural network) model, which is based on XAI, can effectively use multisource SIoT information beyond triples and that others may outperform the baseline model.

H o p e University, Hope Park L16 9JD U.K. Seifedine Kadry (seifedine. [email protected]) is with the Department of Applied Data Science, Noroff University College, Mølleparken 4, 0459 Oslo Norway. References

About the Authors Gaurav Dhiman (gdhiman0001@ gmail.com) is with the Department of Computer Science, Government Bikram College of Commerce, Patiala 147001 India; University Centre for Research and Development, Department of Computer Science and Engineering, Chandigarh University, Gharuan, Mohali 140413 India; and Department of Computer Science and Engineering, Graphic Era (Deemed to be University), Dehradun 248002 India. Atulya Nagar (atulya.nagar@ hope.ac.uk) is with Liverpool

[1] K. Yadav, A. Jain, N. M. Osman Sid Ahmed, S. A. Saad Hamad, G. Dhiman, and S. D. Alotaibi, “Internet of Thing based Koch fractal curve fractal antennas for wireless applications,” IETE J. Res., pp. 1–10, Apr. 2022, doi: 10.1080/03772063.2022.2058631. [2] W. Viriyasitavat, L. D. Xu, A. Sapsomboon, G. Dhiman, and D. Hoonsopon, “Building trust of Blockchainbased Internet-of-Thing services using public key infrastructure,” Enterprise Inf. Syst., pp. 1–24, Jan. 2022, doi: 10.1080/17517575.2022.2037162. [3] K. Yadav, E. Kareri, S. D. Alotaibi, W. Viriyasitavat, G. Dhiman, and A. Kaur, “Privacy protection against attack scenario of federated learning using Internet of Things,” Enterprise Inf. Syst., p. 2,101,025, Jul. 2022, doi: 10.1080/17517575.2022.2101025. 

IMAGE LICENSED BY INGRAM PUBLISHING

satisfies the ­ privacy requirements. On the crawled Sina Weibo dataset, a solution is described for determining the upper bound on the chances of a nodeleaking state without resorting to the time-consuming Monte Carlo approach with XAI. The proposed approach is validated experimentally and by example analysis, and the results indicate its use. Alkwai proposes a unit vector and a unit description vector, which are then projected onto a particular relational space using hierarchicaltype information, limiting the semantic content. The graph attention technique is then utilized to fuse topological structure information from the graph to determine the effects of various nearby points on the entity. To address the data-sparsity issue, multihop relationship information between entities is generated concurrently. Finally, global information between dimensions is collected using a decoder. Experiments

IEEE connects you to a universe of information! As the world’s largest professional association dedicated to advancing technological innovation and excellence for the benefit of humanity, the IEEE and its Members inspire a global community through its highly cited publications, conferences, technology standards, and professional and educational activities. Visit www.ieee.org. Publications / IEEE Xplore ® / Standards / Membership / Conferences / Education



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

35

Blockchain Technology in Agriculture for Indian Farmers A Systematic Literature Review, Challenges, and Solutions by Urvashi Sugandh , Swati Nigam , and Manju Khari 

36

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

2333-942X/22©2022IEEE

B

©SHUTTERSTOCK.COM/MANOEJ PAATEEL

lockchain technology swiftly rose to prominence in a wide range of applications within the smart agriculture field. The necessity to construct smart peer-to-peer systems capable of validating, securing, monitoring, and analyzing agricultural data has prompted discussions regarding the development of blockchain-based Internet of Things (IoT) systems in smart agriculture. Blockchain technology plays a critical role in the transformation of traditional means of storing, sorting, and exchanging agricultural data into a more trustworthy, immutable, transparent, and decentralized method of sharing data. Smart farming will benefit from the integration of the IoT and the blockchain, which will take us from having merely smart farms to having an Internet of smart farms as well as provide better control over supply chain networks in general. As a consequence of this combination, smart agriculture will be managed more autonomously and intelligently, resulting in greater efficiency and optimization of ­operations. This study presents a detailed assessment on the relevance of combining both blockchain and the IoT in the development of smart applications in smart agriculture. The research is divided into three sections. In the first section, we provide a systematic survey and bibliometric analysis related to blockchain in agriculture. Then we provide challenges faced by Indian farmers and provide research objectives with research questions. Finally, the article develops a unique blockchain model that has the potential to be employed as a significant solution to critical difficulties in IoT-based smart agricultural systems. The research also evaluated and thoroughly addressed the major roles and strengths of the common blockchain platforms used in managing different subsectors in smart agriculture, such as crops, livestock grazing, and the food supply chain, among others. Blockchain Applications in Agriculture Blockchain technology has become a most popular technology within the last decade as many companies and institutions have shown interest in its fruitful characteristics and features. Blockchain technology gained more attention after the publication of a white paper titled “Bitcoin: A Peer-to-Peer Cash System,” authored by a person pseudonymously named “Satoshi Nakamoto” in 2008 [1]. In this paper, the author explained the trustworthy implementation of cryptocurrency (Bitcoin) Digital Object Identifier 10.1109/MSMC.2022.3197914 Date of current version: 20 October 2022



using blockchain technology. After the popularity of Bitcoin, several digital cryptocoins have been developed for trading. Due to the immutable property of blockchain technology, various different areas are going to be discovered day by day to find out the solution against the challenges of that field. Some of the discovered fields where blockchain technology has been implemented are in financial services, industrial products, telecommunication and manufacturing, energy and utilities, health care, government, and entertainment and media, among others. In the recent past, blockchain technology has also found application in the agriculture sector. When the word “agriculture” comes to mind, we are forced to think about the thousands of farmers who feed millions of people with the hard work that they do under the worst climatic conditions. Simultaneously, a thought also comes about the people who consume these products as well as the nutritional value of agro products. Before reaching the customers, these products are processed with various types of chemicals. Because the agro products are processed multiple times, the nutritional value of the products becomes degraded to a high percentage, and the products also contain synthetic chemicals used in agriculture in the form of pesticides or fertilizers [2]. For these reasons, the need for transparency and traceability is raised. It requires building an efficient and effective architecture to provide transparency and traceability for the development of the agriculture field in many ways. In this way, the quality of food may be maintained, and the scandals and fraud related to this sector also could be lessened. In recent times, blockchain technology is the latest method to be combined with the IoT to resolve the challenges of the field of agriculture in an effective and efficient manner. Blockchain technology is significant in a number of areas, for example, in smart farming, monitoring and tracking of supply chain management, weather observation, soil analysis, finance management, and so on. Technologies in Agriculture With the help of technologies like big data, the IoT, radiofrequency identification (RFID), and so on, many researchers have done a lot of work to achieve the transparency and traceability of goods and services. For traceability, precise data and a protected storage platform are required [3]. Sensor technology and the IoT provide secured and fast methods for the collection of data. These methods also are used for component scanning, item identification, storage, and shipment as well as for the intelligence gathered throughout the complete process [4]. The main purpose of transparency and traceability in a food supply chain-management system is to avoid fraud and scandals regarding farm products in terms of food quality, food distribution, and so on. To avoid losses, regular systems are now becoming automated systems through RFID, the IoT, and cloud computing. Sensor tags are considered to be the main element of any tracking system. These sensor O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

37

tags are used to identify the agro products [5]. Sensors located under the soil are used to determine the planting conditions by gathering information. Deep learning technology is used to identify the different fruits and their segmentation. Agricultural robotics plays a very vital role in the agriculture sector [6]. Robots are used for weed control, harvesting purposes, seed planting, weather monitoring, and soil observation.

Research Methodology and Literature Review For this review, we followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses declaration and the principles of literature review and systematic mapping research procedures. The purpose of this type of mapping study is to gain an overview of the subject field and to supplement it by assessing the status of evidence in particular issues. Mapping studies may help us find and map blockchain use cases in agriculture as well as assess how far blockchain-based applications have progressed in terms of those use cases. They might also point out areas in which further study is needed. We would be able to look at the current trends in technological techniques, processes, and ideas used to build blockchain-based agriculture applications as a result of the systematic study. What follows is a step-by-step breakdown of the mapping procedure, as illustrated in Figure 1. Definition of Research Questions Using our aim of unraveling the state of the art in research on the use of blockchain technology in agriculture as a starting point, we identified four research questions to be addressed as part of the first process stage in the systematic mapping study. ◆◆ What are some of the applications of blockchain technology in agriculture? ◆◆ W hat blockchain-based applications a re being developed in response to the use cases that have been identified? ◆◆ In what ways are blockchain-based applications hindered and limited? ◆◆ What steps are being taken to address these limitations and challenges at this time?

Process Steps

Conducting the Research The main articles for the systematic mapping research are selected in the second phase of the procedure by searching

Definition of Research Questions

Conduct Search

Conduct Search

Keywording Using Abstracts

Data Extraction and Mapping Process

Outcomes

Blockchain in Agriculture and the Food Supply Chain Perhaps the most talked-about technological advancement in today’s world is blockchain technology, which is what makes bitcoin possible. However, blockchain applications are not confined to the financial sector; in fact, it has had a significant impact on a wide range of sectors, including real estate, health care, banking, law, insurance, and supply chain management, among others. Also worthy of consideration is the potential of blockchain technology in agriculture, which we cannot afford to ignore at this time [7]. It is predicted that the value of blockchain technologies in the agricultural business would expand from an approximately US$41.2 million in 2017 to roughly US$430 million by 2023, reflecting a remarkable 47.8% compound annual growth rate (CAGR). There are several applications of blockchain technology in agriculture, which we shall examine in further detail in this article. Applications of blockchain technology in agriculture may be classified into the following types: ◆◆ Use case 1: Production of crops and foods ◆◆ Use case 2: Improved food safety and quality control ◆◆ Use case 3: Controlling the weather crisis ◆◆ Use case 4: Food supply chain ◆◆ Use case 5: The fairer payment of farmers. The rest of the article is structured as follows. The next section presents the research methodology and literature review. The section “Problems in Indian Agriculture Practices” describes problems in the Indian agriculture system. The section “The Proposed Methodology” describes the proposed architecture for improved food safety and quality control using blockchain technology, and the section “Expected Outcomes” describes the expected results. The

final section contains the conclusion of the article and recommendations for further research.

Review Scope

All Papers

Relevant Papers

Classification Scheme

Systematic Map

Figure 1. A step-by-step breakdown of the mapping procedure. 38

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

scientific databases using a search string or keywords in the first step. During our search, we used four scientific databases—the Association for Computing Machinery Digital Library, IEEE Xplore, Web of Science, and ­Scopus—among others. By selecting these databases, we hoped to limit our attention to peer-reviewed publications that had been published in credible journals, conferences, workshops, books, or symposiums, rather than general information. “blockchain” AND (“Agriculture*” OR “e-agriculture*” OR “smart agriculture*” OR “precision agriculture*” OR “food safety” OR “food security”) were the search terms we used to find relevant information in the databases. A pilot search in which we tested some commonly used agriculture-related terms and acronyms, such as food safety, food security, agro-food supply, food supply chain, agriculture supply chain, and so on, led to the selection of the search string used in this article. All of these agriculture-related phrases and derivatives as well as the acronyms were found to be caught in our search string since none of them, when coupled with “blockchain,” yielded any new results that were not already returned by our search string. We also discovered that searching for articles using simply “blockcha in” a nd “agr iculture*” did not retur n cer ta in publications that do not have “agriculture*”-related keywords in their metadata but may have used other agricu ltu re -related ph ra ses, such a s weather cr isis, environment, or transparency in their metadata. Alternatives to “blockchain,” such as “distributed ledger technology,” did not provide any additional results when searched for. It is also crucial to note that the literature search was undertaken without regard to time constraints since the issue is relatively new. As a result, all available material in this field was deemed relevant to our research, regardless of how long it took to find it. Problems in Indian Agriculture Practices Agriculture is one of the most important pillars of the Indian economy and plays a vital role in the economic growth. More than 50% of the population survives on agriculture to fulfill its nutritional requirements. But in the current scenario, the agriculture sector of India is facing a lot of issues that act as a barrier in its development. Some of the encountered issues are discussed in the following sections. Small-Scale and Divided Land Holdings It has been observed that the sown area of approximately 141.3 million hectares and the cropped area of 190.7 million hectares are decreasing day by day. In 1970–1971, an individual farmer held approximately 2.28 hectares of land for agriculture purposes, which has been reduced to less than 1.5 hectares. There are multiple reasons for the reduced holdings of land under individual farmers. The

first is the incrementing ratio of population and the disintegration of the joint family into a nuclear family. The inheritance laws are another reason for the small holdings of land. The land that belongs to the father is further distributed equally among his inheritors. Urbanization is also equally responsible for the overall decrementing rate of land used in agriculture [8]. Lack of Irrigation Facilities Irrigation is an important and crucial activity for effective crop growth. In India, a major part of the irrigation depends upon the rainfall, which is unreliable and uncertain. A water supply must be provided in required amounts for proper irrigation. Because of the lack or unavailability of water, irrigation is not possible [9]. Improper Storage Facilities In India, rural areas are not supplied with proper storage facilities, and they may even be totally unavailable. For this reason, farmers are pressured to sell their agriproducts immediately, just after harvesting, at the low prices in the markets regulated by the middlemen. Recently, there are some agencies that do what is necessary to store the products and buy them at an appropriate rate from the farmers. But there are still not enough of them compared to the amount of crops [10]. Absence of Markets In India, agricultural markets are in very bad shape. Because of the absence of good markets for the buying and selling of agro products, farmers have become dependent on local traders or middlemen, and they sell their products at very low prices. Involvement of Middlemen and Local Traders A middleman buys products from farmers at very low or even throwaway prices. The middleman then sells them to an organization at very high prices and earns a lot of profit. It seems that all of the farmers’ hard work then becomes fruitful for the middlemen, not for the farmers. The involvement of middlemen or local traders comes into play because of the lack of storage facilities and the absence of good trading markets. The government would have to become active with operational markets to avoid the middlemen and save the farmers. Use of Manure, Fertilizers, and Pesticides In India, crops have been growing for thousands of years while neglecting soil replenishment. This negligence has resulted in the reduction and depletion of the production of yields. This problem can be avoided with the use of manure and fertilizers. Inability to Adopt New Technologies India is a diverse country and the second largest country based on agriculture. Some of the states are developed, and some are developing, but some still have remote areas O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

39

where the latest facilities and technologies have not been introduced. Also, the people in developing or developed states are using old implements for the cultivation of fields as they do have not complete knowledge about the benefits of using machines. But one of the problems, besides the shortage of machines, is that the area for cultivation is too small. Because of these small units, permission is not granted for the use of such machines [11]. Inadequate Transport Facilities Transport facilities are one of the major problems why Indian agriculture suffers greatly. Currently, lakhs of villages still exist where the connectivity with main roads is inadequate. Bullock carts are used by farmers, and they cannot be used in the rainy season [12]. Because of the lack of proper connectivity, good roads, and nearby markets, farmers are unable to reach the main markets with their products and have to sell them to the middlemen or in the local markets.

The Proposed Methodology Distributed ledger and blockchain technologies in agriculture are reengineering several current farming processes, from tracing the origin of food, to tracking customer demand, to settling transactions, and even creating new marketplaces. Given the fast development of blockchain technology, it is essential to keep abreast of its breakthroughs and advancements in the agricultural industry if one wants to remain one step ahead of the competitors. Here, we have proposed a framework according to our different research objectives, as ex­­ plained in the last section.

Proposed New Model to Improve Food Safety and Quality Control Using Blockchain Technology Tracing the food supply chain is essential for determining the source of food. It makes certain that the food that is provided is safe to consume. However, the existing management of the food supply chain makes it difficult for food producers and merchants to verify the provenance of the food they sell to consumers. But it has become feasible, thanks to the development of the blockInadequate Monetary Support chain, to instill confidence and openness in the food supEvery start-up or business requires capital for the manageply chain network, thereby assuring the safety of food ment and growth of the business. In the same way, farming also requires an investment of money. But because they for all consumers. F ­ igure 2 shows the proposed architecare selling their products at throwaway prices, farmers are ture to Improved food safety and quality control using unable to invest. So monetary support is also a task for the blockchain technology. government to undertake. It could introduce some scheme How might the blockchain-based food supply chain help that could help the farmers in that way. to minimize the incidence of food fraud? The following methods may assist in decreasing food fraud in the food supply chain using the blockchain technology: ◆◆ Step 1: IoT sensors generate data, or Complete Information Consumer Application farmers store data, as appropriate. of Food Supply Chain Layer Activity ◆◆ Step 2: Harvested crops are transported to food-processing businesses. ◆◆ Step 3: Processed foods are distributed to retailers and wholesalers. ◆◆ Step 4: Consumers can track the supBlockchain ply chain from start to finish. Layer

Smart Contract

Digital Flow Layer

Physical Flow Layer Farmer

Processor

Distributor

Retailer Customer

Figure 2. A proposed architecture to improve food safety and quality

control using blockchain technology.

40

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Step 1: IoT sensors generate data, or farmers store data, as appropriate The use of sensors in smart farming, as stated in use case 2 (Improved food safety and quality control), allows for the generation of critical information about the crops that are planted in the fields. Assume that the farmers do not use technologically based approaches. Then, by using their mobile application, they may record critical information, such as crop quality, kind of seed, and the meteorological circumstances under which crops were seeded. A distributed storage platform, such as InterPlanetary File System, is used to

Soil Temperature Air Temperature Farmer IoT Nodes

Proposed New Model to Control the Weather Crisis Using the IoT and Blockchain Farmers are often confronted with huge surges in production and uncertain price increases as a result of unpredictable weather and a lack of oversight in the food chain system. By contrast, customers do not comprehend why crops were subjected to adverse weather conditions and

Base Station

IoT Gateway

Gateway

Step 4: Consumers Can Track the Supply Chain From Start to Finish Due to the fact that information about food products is digitally connected to them inside the blockchain, customers may investigate anything by tracing the supply chain back to the farm, including farm origination facts, transit details, batch numbers, processing of foodstuffs, factory data, expiration information, and storage temperature. The blockchain-based food supply chain may assist various stakeholders in gaining access to information on the quality of the food at each step of the process. The use of blockchain technology will make it simpler to determine when and how food has been tainted since it will increase openness across the food supply chain ecosystem.

Wireless Router

Different Types of Crop Growth Information

Step 3: Processed Foods Are Distributed to Wholesalers and Retailers Following the processing of the food items or crops, wholesalers and retailers may place bids on the goods they want using the bidding platform. Similar to how crops are transported to refineries, food products are likewise delivered to wholesalers and merchants with the use of IoT-enabled trucks. In addition to providing traceability, the blockchain supply chain assists food firms in conducting product recalls or investigations in a timely and seamless manner.

Database

Network

Step 2: Transportation of Harvested Crops to Food-Processing Businesses Once the crops are harvested, the food-processing firms begin bidding on the products using an online auction platform. The crops may be carried to the refineries using IoTenabled vehicles, which can monitor and record the temperature conditions under which they are stored and supplied throughout the process. A bid is approved by smart contracts, and then the crops are processed, with the information gathered at each stage of the process being stored in real time on the blockchain by the firms. The information acquired from the refineries may assist wholesalers and retailers in determining whether or not the food being provided is of high quality or not. Data stored on the blockchain may also be used to determine whether or not compliance has been satisfied at each stage of the food supply chain.

O c tob e r 2022

Figure 3. A proposed architecture to control the weather crisis using the IoT and blockchain. CIF: cost, insurance, and freight; DCT: digital control technology.

store the data collected either via the use of IoT sensors or manually by farmers. The addresses of the data are kept in the blockchain.

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

41

why prices have grown as a result of these circumstances. With its openness and transparency, blockchain in agriculture allows farmers and other parties to have a thorough grasp of the subject matter in question. What is the process through which this occurs? With smart agriculture, one may strategically deploy agricultural weather stations around fields to produce critical information about weather that can be used to influence the behavior of crops. Farmers may use this method to measure and keep track of some of the following information: ◆◆ temperature ◆◆ precipitation ◆◆ relative humidity ◆◆ wind direction ◆◆ soil temperature at various elevations ◆◆ atmospheric pressure ◆◆ leaf wetness ◆◆ wind speed ◆◆ dew-point temperature ◆◆ solar radiation ◆◆ other variables. This information is kept in a blockchain so that any authorized entity may access it in a transparent manner at any time. Consequently, producers may take necessary steps, such as crop insurance plans, depending on the preceding provided information. Furthermore, they will be able to get reimbursement in a simple and timely manner. Figure 3 shows the proposed architecture. Expected Outcomes Agriculture is a relatively unexplored area that has the potential to be fundamentally transformed by blockchain technology. Blockchain technology has the potential to benefit the agricultural industry in a variety of ways. In this research, we use blockchain technology in different types of agriculture use cases. We next discuss some expected outcomes of this research. Improved Quality Control and Food Safety The proposed framework can help all stakeholders in terms of increased openness across the supply chain, the elimination of inefficient procedures, and making certain that the best possible quality control conditions are in place. Increased Traceability in the Supply Chain There is an ever-increasing demand from consumers for high-quality food. Customers are more concerned about where their food originates, and this trend is expected to continue. This issue may be resolved by using blockchain technology, which would allow customers to know precisely where their food comes from, who grew it, and how fresh it is. Increased traceability in the supply chain will have a significant influence on the following areas: lowering the incidence of food fraud, removing erroneous labeling, removing middlemen from the process, ensuring that 42

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

producers are adequately compensated for their work, and allowing consumers to understand exactly what they are paying for. Increased Efficiency for Farmers Currently, most farmers capture and manage data using a variety of applications made by software development companies, spreadsheets, and notes in their daily work. Sending these data to other service vendors is not simple and needs a significant amount of effort. The use of blockchain technology would make it possible for farmers to keep all of their data in a single location, making it more accessible to those who need it and saving time and energy in the process. For example, they would be able to monitor things like objectives and strategies for achieving these goals. How many animals there are, their health conditions, what they eat, and how frequently they should be fed are just some of the things to consider, find out how many kinds of crops were planted, when and how well they’ve done, how many hours each employee has worked, how much each employee is entitled to be paid, and expenses and income information. As a result, the process becomes more efficient, and information is less likely to be lost if everything is recorded in one place. Fairer Payments for Farmers Farmers have several difficulties in being compensated for their goods. For example, typically, farmers don’t get the entire payment for their commodities for many weeks, and traditional payment methods, such as wire transfers, may eat up a considerable portion of their profits. In the case of smart contracts based on the blockchain, a certain prespecified condition is met, and payments are automatically triggered without incurring excessive transaction costs. Farms might conceivably get money for their commodities as soon as they’re delivered, without losing a large percentage of their revenue in the process. Conclusion A total of 118 publications on the food and agricultural business were reviewed in this research, and the findings were presented in this report. The articles were published between 2018 and 2021. The current investigation was carried out with the help of the VOSviewer program. From the data, it can be concluded that research on the idea of blockchain in agriculture began in 2018 and has continued to get increased interest from academics, notably in 2021. Furthermore, the findings revealed that India, China, and the United States are the top three nations in the world for publishing articles on blockchain technology in agriculture. The findings also revealed that traceability, supply chain, the IoT, smart contracts, and food security are the top terms that have appeared most often in this context. According to the results, IoT technology has been extensively used in the development of blockchain systems in agriculture. In addition to that, the authors have brought up the challenges

faced by Indian farmers and provided research objectives with research questions. Lastly, a unique blockchain model was also developed in the article, and it has the potential to be employed as a significant solution to critical difficulties in IoT-based smart agricultural systems. About the Authors Urvashi Sugandh ([email protected]) earned her master’s degree in technology from the Department of Information Technology of Banasthali Vidyapith, Rajasthan in 2014. She is an assistant professor in the Department of Computer Science Engineering, HMR Institute of Technology and Management, affiliated with Guru Gobind Singh Indraprastha University, Delhi, by the Government of the National Capital Territory of Delhi. Currently, she is pursuing a Ph.D. degree from the Department of Computer Science, Banasthali Vidyapith, Jaipur, Rajasthan 304022 India. Her research interests include blockchain, data mining, and software engineering. She has published two patents and four research papers in international/ national conferences. Swati Nigam ([email protected]) earned her Ph.D. degree in computer science from the Department of Electronics and Communication, University of Allahabad, India, in 2015. She is currently an assistant professor in the Department of Computer Science, Banasthali Vidyapith, Jaipur, Rajasthan 304022 India. She has been a postdoctoral fellow under the National Postdoctoral Fellowship scheme of Science and Engineering Research Board, Department of Science and Technology, Government of India. Earlier, she was awarded a Senior Research Fellowship by the Council of Scientific and Industrial Research, Government of India. Her research interests include object detection, object tracking, and human behavior analysis. She has authored more than 20 articles in peer-reviewed journals, book chapters, and conference proceedings. She has also published an authored book for Springer publications. She is a designated reviewer of several Science Citation Index journals, like IEEE Access, Computer Vision and Image Understanding, Journal of Electronic Imaging, and Multimedia Tools and Applications. She has been publication chair, publicity chair, technical program committee member, and reviewer of various respected conferences. She is a Member of IEEE and a member of the Association for Computing Machinery. Manju Khari ([email protected]) earned her Ph.D. degree in computer science and engineering from the National Institute of Technology Patna and her master’s degree in information security from the Ambedkar Institute of Advanced Communication Technologies and Research, affiliated with Guru Gobind Singh Indraprastha University, Delhi, India. She is an associate professor in the School of Computer and Systems Sciences, Jawaharlal Nehru University, New Delhi, Delhi 110067 India. Prior to this, she worked with Netaji Subhas University of Technology, East Campus, formerly Ambedkar Institute of Advanced Communication Technologies and Research, under the Government of the

National Capital Territory of Delhi. She has 80 published papers in refereed national/international journals and conferences (for IEEE, the Association for Computing Machinery, Springer, Inderscience, and Elsevier) and 10 book chapters for Springer, CRC Press, IGI Global, and Auerbach. She is also coauthor of two books published by NCERT for classes XI and XII and coeditor of 10 edited books. She has also organized five international conference sessions, three faculty development programs, one workshop, and one industrial meeting. She delivered an expert talk, was a guest lecturer in an international conference, and a member of reviewer/technical program committees in various international conferences. Besides this, she is associated with many international research organizations as an associate editor/guest editor of Springer, Wiley, and Elsevier books and a reviewer for various international journals. References [1] T. Shestakovska, “Socio-economic security of the agricultural sector in the context of the use of blockchain-technology,” Investytsiyi Prakt. ta Dosvid, vol. 23, p. 27, Dec. 2018, doi: 10.32702/2306-6814.2018.23.27. [2] K. Leng, Y. Bi, L. Jing, H. C. Fu, and I. Van Nieuwenhuyse, “Research on agricultural supply chain system with double chain architecture based on blockchain technology,” Future Gen. Comput. Syst., vol. 86, pp. 641–649, Sep. 2018, doi: 10.1016/j.future.2018.04.061. [3] M. C. Aldag, “The use of blockchain technology in agriculture,” Zesz. Nauk. Uniw. Ekon. w Krakowie, vol. 4, no. 4982, pp. 7–17, 2019, doi: 10.15678/znuek.2019.0982.0401. [4] A. Kamilaris, A. Fonts, and F. X. Prenafeta-Boldύ, “The rise of blockchain technology in agriculture and food supply chains,” Trends Food Sci. Technol., vol. 91, pp. 640–652, Sep. 2019, doi: 10.1016/j.tifs.2019.07.034. [5] T. Surasak, N. Wattanavichean, C. Preuksakarn, and S. C. H. Huang, “Thai agriculture products traceability system using blockchain and Internet of Things,” Int. J. Adv. Comput. Sci. Appl., vol. 10, no. 9, pp. 578–583, 2019, doi: 10.14569/ijacsa.2019.0100976. [6] H. Wang, Z. Liu, and Y. Liang, “Research on the three-in-one model of agricultural products E-commerce logistics under the combination of resource saving and blockchain technology,” IOP Conf. Ser. Mater. Sci. Eng., vol. 677, no. 3, p. 032111, Dec. 2019, doi: 10.1088/1757-899X/677/3/032111. [7] H. Zhan, X. Lv, and D. Xu, “Research on blockchain technology in promoting environmental protection development of agricultural products E-commerce model in Jilin province,” IOP Conf. Ser. Mater. Sci. Eng., vol. 612, no. 5, p. 052037, Oct. 2019, doi: 10.1088/1757-899X/612/5/052037. [8] T. Moroz, “Prospects for the use of blockchain technology in the agricultural sector of the economy,” Modern Econ., vol. 17, no. 1, pp. 153–157, Oct. 2019, doi: 10.31521/ modecon.v17(2019)-24. [9] U. Sengupta, S. Singh, and H. M. Kim, “Meeting changing customer requirements in food and agriculture through application of blockchain technology,” SSRN Electron. J., 2019. [Online]. Available: https://ssrn.com/abstract=3429200 [10] C. Xie and D. He, “Discussion on the development path of China’s agricultural products E-commerce based on blockchain,” DEStech Trans. Eng. Technol. Res., vol. 15, pp. 1–5, Jan. 2019, doi: 10.12783/dtetr/eeec2018/26859. [11] Y.-B. Son and Y.-H. Kim, “Design of management system for registering agricultural machine using blockchain,” J. Korea Contents Assoc., vol. 19, no. 12, pp. 18–27, 2019, doi: 10.5392/JKCA.2019.19.12.018. [12] K. Salah, N. Nizamuddin, R. Jayaraman, and M. Omar, “Blockchain-based soybean traceability in agricultural supply chain,” IEEE Access, vol. 7, pp. 73,295–73,305, May 2019, doi: 10.1109/ACCESS.2019.2918000. 

O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

43

Explainable ArtificialIntelligenceBased Privacy Preservation Approach for Information Dissemination on Social Networks An Incremental Technique by Shoayee Dlaim Alotaibi and Kusum Yadav 

T

his article aims to address the issues above by defining a social network information transmission model with the amalgamation of explainable artificial intelligence (XAI) compatible with the paranormal connection. It suggests a way of information transmission called local greedy that aids in the preservation of user privacy. Its impact acts as a buffer between the conflicting interests of privacy protection and information dissemination. Aiming at the enumeration problem of seed set selection, an incremental technique is presented for constructing seed sets to Digital Object Identifier 10.1109/MSMC.2022.3188406 Date of current version: 20 October 2022

44

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

minimize time overhead; a local influence subgraph method for computing nodes is also proposed to evaluate the influence of seed set propagation rapidly. The group meets privacy protection conditions. A strategy is presented to determine the upper bound on the likelihood of a node leaking state without resorting to the time-consuming Monte Carlo approach with XAI on the crawled Sina Weibo dataset. The suggested technique is validated experimentally and by example analysis, and the findings demonstrate its usefulness. Introduction The privacy of users in social networks includes personal information and communication information. To ensure 2333-942X/22©2022IEEE

©SHUTTERSTOCK.COM/RAWPIXEL.COM

that people trusted by users only access user information, it is necessary to implement information access control between user nodes for privacy protection. The authors in [1] designed a centralized social network model that supports high availability and real-time content dissemination and decides users’ privacy by learning their social behaviors. The authors in [2] proposed online social network architecture Lotus Net based on a distributed hash table which controls private resources through a flexible and fine-grained autonomous access control unit, making it easier for users to adjust their privacy settings. In practice, however, neither centralized nor distributed solutions can protect pervasive social network communications as expected. To this end, a kind of access control scheme based on 2D trust degree is proposed [3], [4], which calculates the 2D trust degree between nodes by using information node attributes and implements access control to user privacy according to 2D trust degree. To summarize, in the existing information dissemination prediction research, researchers have not considered the possibility of using paraphrasing as an information dissemination method, which will cause the omission of the dissemination path when tracing the source of disseminated information, as that in information dissemination modeling. Leaks of private information cannot be captured when a paraphrase occurs. In the research of influence maximization,

although various algorithms explore the best solution to maximize influence from different angles, they ignore the privacy protection problem in communication [5], [6], [7], [8], [9]. User privacy is the most common information carrier for social networks, and influence dissemination schemes lacking privacy protection measures are challenging to directly apply to the actual environment. Therefore, in this article, aiming at the reasons why the existing information dissemination models cannot well model the leakage of user privacy and the problem that related research is challenging to balance the needs of user privacy protection and information dissemination, this work researches social network information dissemination methods that support privacy protection [10], [11], [12]. It helps to maximize the dissemination of information influence while protecting user privacy. The main applications of this method are as follows (see Figure 1): 1) For individual users: Only the blocked users are specified when publishing information. Therefore, the algorithm can automatically give the push objects of the data so that the information will not leak to the blocked users. 2) For merchants: By setting non-target users as block lists, algorithms can find out which users in the social network are more valuable for advertising and push them accurately. Methods of Constructing Communication Networks to Support Retelling Behaviors For new information to be released by users, to maximize the influence of information dissemination while satisfying the preset privacy protection constraints, it is necessary to predict the information dissemination process based on the social network dissemination model. The social network propagation model relies on the probability distribution of user behavior, so it is necessary to build a propagation network of published information to train a user behavior classifier. Method Overview Problem Definition The difficulty of building a communication network is that there are only two types of user behaviors obtained from

1

4

2

5

7 0

3

6

Figure 1. The process of information communication

based on the model of social network communication.

O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

45

social networks: Custpost and Custforward. But, in fact, a part of all Cust post (original publishing behaviors) obtained should belong to Custmention (reporting behaviors), which is different from the communication network. Therefore, related and need to be distinguished. Method Steps The primary process of building a dissemination network is to associate the equivalent information and derivative information released by other users after information N is released. For example, for the forwarding behavior list Kf, the information in the list contains the relationship of its entire forwarding sequence, so it is easy to add to the propagation network. For the publishing behavior list K p , according to definition 4, part of the information is derived information from information N, corresponding to the behavior type Custmention, which needs to be added to the propagation network H N. The other part is the original information of other users, corresponding to the behavior type Custpost, and has nothing to do with the communication network HN. The basic idea of building a propagation network is to establish a relationship between the information in the current action and the existing nodes in the propagation network in sequence according to time. The basic steps are as follows: 1) establish the initial node of the propagation network HN, < y, Equaled > 2) merge the list Kf and the list Kp and sort to obtain the list K 3) traverse the list K, assuming that the current traversed behavior is < uid, custtype, tl , N l >, denote modeold as the node that publishes information N’ in the propagation network 4) if custtype = Custmention, create a new node modenew for the information N’, and update the information dissemination network HN = CustNode(HN, modeold, modenew, forwarded) 5) If custtype = Cust post, create a new node modenew for the information N’, and update the information dissemination network H N = Cust Node(HN, modeold, modenew, Mentioned).

Experiment and Result Evaluation Experimental Data To build a dissemination network, accurate social network data are needed. Here, we choose to use a crawler to crawl users’ public information in Sina Weibo as the experimental data. The crawled data can be divided into microblog data, personal information data, and social relationship data. In the micro blog data finally crawled, the number of users is 2,903,403, and the total number of micro blogs is 23,068,115, of which 7,006,235 are forwarded. The crawled Weibo content is all content posted by users during the 2018 National Day.

Result Analysis The existing inf luence ma ximization methods are mainly divided into three types: heuristic algorithm, Monte Carlo method, and community-based algorithm. Since the community-based algorithm is difficult to extend to user privacy protection requirements, the Monte Carlo-based method has certain theoretical guarantees. The node degree and the node’s distance in the network are the basis of standard heuristic algorithms. Therefore, three representative Monte Carlo methods are selected for the comparative experiments in this paper: simulate greedy algorithm, degree algorithm, and distance algorithm. The degree algorithm and the distance algorithm are both heuristic algorithms. The Monte Carlo simulation part of the algorithm is only used to judge whether the seed set satisfies the privacy protection constraint. 1) Incremental greedy algorithm: Construct the seed set incrementally according to the greedy rule and use Monte Carlo to estimate the size of the influence and whether the privacy protection constraints are met. 2) Degree-based algorithm: a degree heuristic algorithm based on the degree of the node. In the beginning, all nodes are sorted according to the degree of the node from large to small, and then try to join the seed set in sequence, which requires Monte Carlo simulation. 3)  Distance-based algorithm: a heuristic algorithm based on the average distance to all other nodes. In the beginning, all nodes are sorted from small to large, and then they are added to the seed Proposed Greedy Algorithm set in sequence, and Monte Carlo simula1,600 Incremental Greedy Algorithm tion is required. 1,400 Distance-Based Algorithm 4) Influence-based greedy algorithm (pro1,200 Degree-Based Algorithm posed): The algorithm Calculate Bound is 1,000 800 used to calculate the vq (y) of each node 600 initially and the algorithm Local Influ400 ence is used to estimate the influence of 200 the seed set. The algorithm also con0 structs the seed set incrementally but 10 20 40 60 80 requires Monte Carlo simulations. The simulate greedy algorithm uses the Figure 2. A comparison of algorithms running time based on Monte Carlo method to calculate the influence index. 46

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Table 1. A comparison of algorithms running time based on influence index. Dimension of Collection

Proposed Greedy Algorithm

Incremental Greedy Algorithm

Distance-Based Algorithm

Degree-Based Algorithm

10

50

210

200

200

20

80

500

400

400

40

100

800

600

600

60

150

1400

800

800

80

200

1600

1000

700

influence of the seed set, which has better theoretical guarantees when the number of simulation rounds is large. The degree of a node and the average distance from a node to other nodes are often used in influence computing research and as the basis for algorithm design in influence maximization research. Therefore, this paper uses the degree algorithm and the distance algorithm as a comparison algorithm. In addition to using J(R) and PH(R) as the basis for judging the algorithm’s effectiveness, the experiment also evaluates the algorithm’s efficiency according to its running time. Figure 2 and Table 1 draw the line graphs of the algorithm’s running time when the optional set size is taken as the abscissa. Additionally, as seen in the image, the incremental greedy method’s longest running time is somewhat faster than the two heuristic algorithms, and the heuristic approach based on node degree is faster than the heuristic strategy based on average distance. By combining the aforementioned algorithm effect comparisons, it is possible to infer that the algorithm described in this work is significantly faster than the mainstream method and has various benefits in terms of impact.

distributed during the dissemination process, and incorporate additional features into the process of dissemination network construction. About the Authors Shoayee Dlaim Alotaibi ([email protected]) is with the College of Computer Science and Engineering, University of Ha’il, Hail, 1475852, Saudi Arabia. Kusum Yadav ([email protected]) is with the College of Computer Science and Engineering, University of Ha’il, Hail, 1475852, Saudi Arabia. References [1] T. Ding, F. Hasan, W. K. Bickel, and S. Pan, “Interpreting social media-based substance use prediction models with knowledge distillation,” in Proc. 2018 IEEE 30th Int. Conf. Tools Artif. Intell. (ICTAI), pp. 623–630, doi: 10.1109/ICTAI.2018.00100. [2] F. K. Sufi and M. Alsulami, “Automated multidimensional analysis of global events with entity detection, sentiment analysis and anomaly detection,” IEEE Access, vol. 9, pp. 152,449–152,460, Nov. 2021, doi: 10.1109/ACCESS.2021.3127571. [3] C. Wiriyakun and W. Kurutach, “Feature selection for human trafficking detection models,” in Proc. 2021 IEEE/ACIS 20th Int. Fall Conf. Comput. Inf. Sci. (ICIS Fall), pp. 131–135, doi: 10.1109/ICISFall51598.2021.9627435. [4] B. Boenninghoff, S. Hessler, D. Kolossa, and R. M. Nickel, “Explainable authorship

Conclusion Aiming to resolve the conf lict between maximizing influence dissemination and protecting user privacy during the process of social network information dissemination, this paper proposes a social network information dissemination model and inference method that incorporates the parametric relationship, as well as the incremental-greedy seed set selection algorithm, the local influence calculation algorithm influence, and the node leakage probability algorithm calculate bound. The influence-based greedy algorithm is presented on this premise. The crawling Sina Weibo data collection is used for experimental verification and instance analysis. The results demonstrate that the influence-based greedy algorithm may increase dissemination influence while maintaining user privacy. Future research will expand on the characteristics of social network information dissemination and the reasons for privacy leakage, take into account changes in the amount of information



verification in social media via attention-based similarity learning,” in Proc. 2019 IEEE Int. Conf. Big Data (Big Data), pp. 36–45, doi: 10.1109/BigData47090.2019.9005650. [5] A. Malte and S. Sonawane, “Effective distributed representation of code-mixed text,” in Proc. 2019 IEEE 16th India Council Int. Conf. (INDICON), pp. 1–4, doi: 10.1109/INDICON47234.2019.9028960. [6] F. Tafannum, M. N. Sharear Shopnil, A. Salsabil, N. Ahmed, M. G. Rabiul Alam, and M. T Reza, “Demystifying black-box learning models of rumor detection from social media posts,” in Proc. 2021 IEEE 12th Annu. Ubiquitous Comput., Electron. Mobile Commun. Conf. (UEMCON), pp. 358–364, doi: 10.1109/UEMCON53757.2021.9666567. [7] G. Chen, W. Mao, Q. Kong, and H. Han, “Joint learning with keyword extraction for event detection in social media,” in Proc. 2018 IEEE Int. Conf. Intell. Secur. Informat. (ISI), pp. 214–219, doi: 10.1109/ISI.2018.8587340. [8] W. Guo, “Partially explainable big data driven deep reinforcement learning for green 5G UAV,” in Proc. 2020 IEEE Int. Conf. Commun. (ICC), pp. 1–7, doi: 10.1109/ ICC40277.2020.9149151. [9] C.-Y. Lin, Y.-F. Chiu, L.-C. Wang, and D. Niyato, “Modeling of multilayer multicontent latent tree and its applications,” IEEE Trans. Comput. Soc. Syst., vol. 8, no. 1, pp. 5–19, Feb. 2021, doi: 10.1109/TCSS.2020.3035202.



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

47

An Explainable ArtificialIntelligenceBased CNN Model for Knowledge Extraction From the Social Internet of Things Proposing a New Model by Lulwah M. Alkwai

R

IMAGE LICENSED BY INGRAM PUBLISHING

ich material is buried in the entity’s textual description information, its hierarchical-type information, and the graph’s topological structure information in the knowledge graph. As a result, these data can be a useful supplement to triple information in terms of improving performance. To appropriately exploit these social Internet of Things (IoT) data, entity details are first encoded using artificial-intelligence (AI)-based convolutional neural networks (CNNs). The unit vector and unit description vector are then projected into a given relational space using the hierarchical-type information, thus restricting its semantic content. The graph attention approach is then used to fuse the graph’s topological structure information to calculate the Digital Object Identifier 10.1109/MSMC.2022.3198023 Date of current version: 20 October 2022

48

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

2333-942X/22©2022IEEE

learning (XAI-CNN) model. XAI-CNN includes a TransE influence of various neighboring points on the entity. To model-based encoder that combines the structural triples deal with the data-sparse problem, the multihop relationin the knowledge graph with the textual description inforship information among entities is calculated at the same mation of the entity, the hierarchical-type information of time. Finally, a decoder is used to collect global informathe entity, and the structural information of the graph to tion among dimensions. Link prediction experiments learn the information described by show that the multisource inforcomplex relationships. The hierarmation combined knowledge repchical-type information of entities resentation learning (XAI-CNN) The encoder receives can assist individuals in automatimodel based on explainable AI the entity vector cally connecting distinct entities (XAI) can effectively use multiand constraining the semantic source social IoT information acquired from the properties of the entities through beyond triples and that other techencoder, which is the type of information to which niques may be better than the the entities belong. baseline model. used to continue The entity’s description informatraining, and the tion is a more complete explanation Knowledge Representation outcome is the of the entity’s relevant content, Learning Models which contains a lot of critical In recent years, a variety of knowltriplet’s final score. knowledge and is a valuable addiedge representation learning modtion to the triple structure. The els has been proposed. Inspired graph’s topological information by word2vec [2], TransE [1] regards explains the connections among distinct items and may the relationship in the triplet as the translation among accurately depict their spatial interactions. By combinthe head entity and the tail entity and obtains a large ing them with the original triples, the knowledge graph’s performance improvement with fewer parameters, but it hidden entity and relation properties may be better capis suitable only for dealing with one-to-one relationtured. XAI-CNN, on the other hand, employs the ConvKB ships, so many extended models are derived from it. model as a decoder to obtain the global features of tripTransH [1] introduces a specific relation hyperplane, let vectors in multiple dimensions while keeping the which uses the hyperplane normal vector and translamodel’s translation characteristics. The encoder receives tion vector to represent the relation vector, and maps the entity vector acquired from the encoder, which is entities to the hyperplane space of different relations. used to continue training, and the outcome is the triplet’s TransR [3] further assumes that relations should have final score. their own semantic space and that entities should be regarded as vectors in the semantic space. Therefore, it defines transition matrices for different relations, maphd td hs ts ping entities into the corresponding spaces. RESCAL [4] uses vectors to represent entities to capture the implicit semantic information and uses matrices to represent Hierarchical-Type Hierarchical-Type relationships to simulate the pairwise interaction Projection Projection among various elements. DistMult [5] greatly reduces the parameters in the RESCAL model by restricting the r relational matrix to be diagonal, but it is suitable only for simulating symmetric relations. Encoder HolE [6] combines the advantages of RESCAL and DistGraph Attention Graph Attention Mult. First, the head and tail entities in the triplet are comNetworks Networks bined through cyclic correlation operations and then semantically matched with the relation vector, which not only has the powerful representation effect of RESCAL but + also has DistMult simplicity. To fully simulate asymmetric relations, ComplEx [7] applied complex numbers to knowledge representation learning for the first time to solve the asymmetric problem in DistMult. The interpretability ConvKB Decoder problem is a technical term for the difficulty of explaining AI choices. The XAI-CNN model, which is a typical encoder– decoder structure, is proposed in this study as a multiFigure 1. The overall framework of XAI-CNN for the source information combined knowledge representation social IoT.

O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

49

The XAI-CNN Model Structure Figure 1 shows the overall framework of the model. First, the text description information of the entity is encoded into a vector through a 1D CNN, and then the entity vector (h s, t s) in the triple structure and the text description vector (h d, t d) of the entity are subjected to hierarchicaltype projection. The semantic information of the constraining entity in the corresponding vector space is filtered to filter out the noise interference of other irrelevant descriptions and semantics. These are then combined with the relation vector (r), respectively, and input into the graph attention network, and the graph attention mechanism is used to capture the features of the entity’s neighbors and find out the interactions among each entity and its neighbors. Finally, the two entity vectors and relation vectors are combined together for training through the gate mechanism. The results of the encoder training are further input into the decoder composed of ConvKB to obtain the final representation of the entity and relation vectors. Combining Entity Information To trade off the most valuable information among the two entities, this article adopts the joint model [8] to learn a combined representation of structural information and textual information. The combined head and tail entities are represented as h = g h 9 h s + ^1 - g hh 9 h d t = g t 9 t s + ^1 - g th 9 t d .

Here, g h and g t, respectively, represent the gates corresponding to the head and tail entities, where the elements are located in the interval [0, 1], and 9 represents the element-level multiplication, which means that the data in all dimensions in the two entities are weighted differently when combined. To constrain g h, g t ! [0, 1], real vector

u h, g u t are introduced into the model, and g h parameters g and g t are expressed as u h), g t = sigmoid ( g u t). g h = sigmoid ( g Encoder Training The scoring function of the encoder model is expressed as fe ^h, r, t h = h + r - t

L 1 /L 2

.

During training, the maximum interval method is used, and the loss function is defined as Le =

| |

^h, r, t h ! T ^hl , r, tlh ! T l

max ^c + fe ^h, r, t h - fe ^hl , r, tlh, 0 h.

Here, c 2 0 is the specified interval parameter, T l is the negative sample corresponding to the triplet in T after replacing the head and tail entities from the entity set E, and T l can be expressed as T l = "^hl , r, t h ; hl ! E , , "^h, r, tlh ; tl ! E ,. Decoder Training To capture the global features of triples, generalize the translation properties of the model, and improve the accuracy of knowledge representation, ConvKB [1] is used as the decoder to further train the vectors in the encoder. ConvKB is trained using a soft margin loss function, which can be expressed as m Ld =

|

^h, r, th ! T , T l

2 m ln ^1 + exp ^l^h,r,th $ fd ^h, r, t hhh + 2 w 2 .

1, ^h, r, t h ! T , fd represents the decoder In l^h,r,th = ( - 1, ^h, r, t h ! T l scoring function, w refers to the fully connected layer weight in fd, and m refers to the L 2 regularization

Table 1. The experimental results of link prediction. FB15K Dataset

FB15K-237 Dataset Hits

Hits

Model

MR

MRR

1

3

10

MR

MRR

1

3

10

Trans model [1]

118

0.425

0.297

0.593

0.796

331

0.247

0.146

0.346

0.414

Compels model [2]

100

0.464

0.378

0.579

0.762

594

0.279

0.198

0.294

0.495

TKRL model [1]

121

0.548

0.331

0.647

0.791

384

0.294

0.183

0.249

0.421

Jointly model [3]

71

0.634

0.496

0.683

0.732

293

0.212

0.209

0.349

0.464

ConvKB model [4]

69

0.646

0.579

0.741

0.816

248

0.298

0.194

0.314

0.456

Attention based [5]

43

0.893

0.769

0.889

0.941

278

0.471

0.377

0.495

0.578

XAI-based CNN

35

0.837

0.792

0.897

0.933

168

0.513

0.492

0.547

0.614

50

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1 MRR

3

10

1

Hits

MRR

FB-15K Dataset

3

10

Hits FB-15K-237 Dataset

Figure 2. The MRR and hit ratio for link prediction.

parameter. Obviously, the decoder turns the model into a binary classification problem to train. For positive samples in T, label the class as 1, and for negative samples in T l , label the class as −1 and keep training iteratively. Ideally, for any triplet, the model can distinguish between true and false. To verify the effect of the model, the knowledge graph link prediction and triplet classification experiments are carried out on FB15K and FB15K237, respectively. Experiment and Analysis For link prediction, three evaluation metrics are used 1) mean reciprocal rank (MRR), which represents the mean of the reciprocal ranks of the correct triples 2) mean rank (MR), which represents the correct triple and the average of the tuple rankings 3) Hits@N, which represents the proportion of correct triples in the top N (N = 1, 3, 10) prediction results. The link prediction results on the two datasets are shown in Table 1 and Figure 2. According to the experimental results, it can be found that the XAI-CNN model outperforms the other models on almost all metrics of the two datasets, which proves that the combined model in this article is very effective to indicate the text description of the entity. Descriptive information, hierarchical-type information, and graph structure information are all important supplements to the original triples and can improve the effect of knowledge representation learning.

results show that our method outperforms other baseline models on these two types of tasks, demonstrating that entity description information, entity level-type information, and graph structure information are useful additions to the original triple structure information of knowledge graphs and that the combined model presented in this article can significantly improve the effect of knowledge representation learning. About the Author Lulwah M. Alkwai ([email protected]) is with the School of Computer Science and Engineering, University of Ha’il, Ha’il, 152485, Saudi Arabia. References [1] A. Bordes, J. Weston, R. Collobert, and Y. Bengio, “Learning structured embeddings of knowledge bases,” in Proc. 25th Annu. Conf. Artif. Intell. (AAAI), 2011, pp. 1–6. [2] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS 26), 2013, pp. 1–9. [3] A. Bordes, X. Glorot, J. Weston, and Y. Bengio, “A semantic matching energy function for learning with multi-relational data,” Mach. Learn., vol. 94, no. 2, pp. 233–259, 2014, doi: 10.1007/s10994-013-5363-6. [4] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 2787–2795. [5] K.-W. Chang, W.-t. Yih, and C. Meek, “Multi relational latent semantic analysis,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2013, pp. 1602–1612. [6] R. Collobert and J. Weston, “A unified architecture for natural language processing:

Conclusion The XAI-CNN model proposed in this work combines multisource data. First, an encoder based on TransE is created, which merges structured triple data, entity description data, entity hierarchical-type data, and graph topology data. The ConvKB model is then utilized as a decoder to calculate global information in many dimensions while maintaining the model’s translation features. On two classic datasets, FB15K and FB15K-237, experiments on link prediction and triplet classification are conducted. The

Deep neural networks with multitask learning,” in Proc. 25th Annu. Int. Conf. Mach. Learn., 2008, pp. 160–167, doi: 10.1145/1390156.1390177. [7] M. Nickel, V. Tresp, and H.-P. Kriegel, “A three-way model for collective learning on multi-relational data,” in Proc. 28th Int. Conf. Mach. Learn., 2011, pp. 809–816, doi: 10.5555/3104482.3104584. [8] S. Riedel, L. Yao, and A. McCallum, “Modeling relations and their mentions without labeled text,” in Machine Learning and Knowledge Discovery in Databases, J. L. Balcázar, F. Bonchi, A. Gionis, and M. Sebag, Eds. Berlin, Germany: Springer-Verlag, 2010, pp. 148–163. 

O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

51

CALL FOR PAPERS IEEE Systems, Man, and Cybernetics Magazine Special Issue on Federated Learning for Cybersecurity Management in the era of AI I AIM AND SCOPE The explosion of IoT data has shown the cloud's limitations. In many circumstances, storing vast amounts of data in a central place is impracticable. These concerns prompted the development of novel Machine Learning (ML) approaches to aid in the resolution of privacy and enormous data volume difficulties. Experts have presented numerous methods to address these concerns; Federated Learning (FL) is the most prominent. FL is the new generation of Artificial Intelligence (AI) based on decentralized data and training, which puts learning to the edge or on the device. The main objective for the foundation of FL in AI is a lack of adequate data to be stored in a centralized server and data security via the use of local information stored through edge devices. FL makes it possible for areas with heterogeneity and sensitive data to exploit the benefits of artificial intelligence. FL may be used to test and train not just on smartphones and tablets but on any edge layer device that has the potential to make a significant contribution to the area of cybersecurity, regardless of the platform. FL enables autonomous cars to learn on decentralized driver behavior worldwide or hospitals to improve diagnostics without jeopardizing patient data confidentiality. FL mitigates cybersecurity risks by relocating computing to the periphery rather than relying on centralized data to execute the learning process. Cyberattacks are regularly occurring in a wide variety of applications implemented in real-world environments, causing most manufacturers to be hesitant to embrace the IoT. FL encapsulates the concepts of targeted data gathering and reduction and mitigates a significant portion of the systemic privacy concerns and costs associated with conventional, centralized ML and data Digital Object Identifier 10.1109/MSMC.2022.3211385

52

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

science and ML methodologies. Emerging technologies have focused on increasing FL security and overcoming statistical problems. Additional initiatives exist to create FL, a broad paradigm for any decentralized collaborative ML approaches that preserve anonymity. In the decade ahead, FL is projected to break boundaries across sectors and create a network where knowledge and data may be exchanged safely, and rewards are given evenly based on each participant's contribution. Hence, to build more resilient infrastructures to manage security threats and assaults, this new investigation, sometimes stated as a new dawn in AI, requires a more comprehensive study into its confirmation and applications for cybersecurity management in the era of AI.

II TOPICS We welcome researchers to share their latest research findings from both academia and industry, including but not limited to the following: • Advanced FL protocol built on the blockchain technology • Efficiently maintains and deploys deep learning models • Novel Anti-forensics-based deep learning model using FL • Data protection using AI-based information hiding in FL • FL-trained SMS model for spam prediction • Secure FL for EDGE • Enhancing MLOps and automation for FL in Cybersecurity Management • AI-based authentication in FL in cybersecurity management • Advanced FL protocols for trust, security, and privacy for cybersecurity management • FL and AI-based quantum computing advance cybersecurity management

III SUBMISSIONS Manuscripts should be prepared according to the “Information for Authors” section of the journal website: https://www.ieeesmc.org/publications/smc-magazine /information-for-authors, and submissions should be done through the journal submission website: https://mc.manuscriptcentral.com/smcmag, by selecting the Manuscript Type of “Federated Learning for Cybersecurity Management in the era of AI”. Submitted papers will be reviewed by at least two different reviewers. Submission of a manuscript implies that it is the authors’ original unpublished work and is not being submitted for possible publication elsewhere.

IV IMPORTANT DATES

Manuscript submissions due: 08 December 2022 First round of reviews completed: 10 March 2023 Revised manuscripts due: 09 May 2023

Second round of reviews completed: 12 July 2023 Final manuscripts due: 15 August 2023

V GUEST EDITORS Dr. Jia-Bao Liu Anhui Jianzhu University, China Email: [email protected] Dr. Muhammad Javaid University of Management and Technology, Pakistan Email: [email protected] Dr. Mohammad Reza Farahani, Iran University of Science and Technology, Iran Email: [email protected] Prof. H. Jafari University of South Africa, South Africa Email: [email protected] Prof. Shui Yu University of Technology Sydney, Australia Email: [email protected]

TAP. CONNECT. NETWORK. SHARE. Connect to IEEE–no matter where you are–with the IEEE App. Stay up-to-date with the latest news

Schedule, manage, or join meetups virtually

Get geo and interest-based recommendations

Read and download your IEEE magazines

Create a personalized experience

Locate IEEE members by location, interests, and affiliations



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

53

Meet Our Volunteers

by Haibin Zhu 

Getting to Know Our Volunteers

L

et us recognize new members in the IEEE Systems, Man, and Cybernetics Society (SMCS) governance.

Tadahiko Murata Vice President of Organization and Planning Tadahiko Murata is a professor at the Depa r tment of I n for m at ic s, K a n s a i Un iversit y, Japa n. He ea r ned a Ph.D. Tadahiko Murata in engineering in 1997 from Osaka Prefecture University (currently Osaka Metropolitan University). His research interests include computational intelligence and nature-inspired systems, such as fuzzy systems, neural networks, evolutionary computation, and reinforcement learning. He is developing real-scale social simulations using agent-based modeling to solve multiobjective and many-objective optimization problems for individuals, communities, and nations. He has been leading a research project of synthetic populations for analysis and simulations using supercomputers of the national institute and universities in Japan since 2019. He has provided his synthetic populations to the COVID-19 Artificial Intelligence Digital Object Identifier 10.1109/MSMC.2022.3205500 Date of current version: 20 October 2022

54

degree in electrical engineering from the University of Southern California, Los Angeles, CA, USA, in 2009. He is now a professor and deputy director of the Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. and Simulation Project managed by Prof. Wu’s research interests include the cabinet secretariat of the Governbrain–computer interfaces, machine ment of Japan since 2020. learning, computational intelligence, He served as a cochair of the TC on and affective computing. He has more Soft Computing in 2005–2008, the Jathan 190 publications (9,700+ Google pan Chapter chair of the SMCS in 2008– Scholar citations; h = 51). He received 2011, the founding the IEEE Computacochair of the TC on tional Intelligence Prof. Wu’s Awareness ComputSociety Outstanding ing from 2010 to the Ph.D. Dissertation research interests present, a member Award in 2012, the include brain– of the SMCS Board IEEE Transactions computer interfacof Governors for two on Fuzzy Systems es, machine learnterms in 2015–2020, Outstanding Paper and the associate Award in 2014, the ing, computational vice president for cyN or th Am eri can intelligence, bernetics in 2021. He Fuzzy Information and affective was appointed as the Processing Society computing. vice president for orEarly Career Award ganization and planin 2014, the SMCS ning in the SMCS in Early Career Award 2022. He is also serving as president in 2017, the Universal Scientific Eduof the Japanese Society for Evolutioncation and Research Network Prize ary Computation for 2020–2022. in Formal Sciences in 2020, the IEEE Transactions on Neural SysDongrui Wu tems and Rehabilitation Engineering Best Paper Award in 2021, and the Member-at-Large, Board of Chinese Association of Automation Governors (Term Ending in 2022) (CAA) Early Career Award in 2021. Dongrui Wu earned His team won the first prize of the his B.E. degree in China Brain–Computer Interface automatic control Competition in four successive from the Univeryears (2019–2022). sity of Science Prof. Wu is a Board of Governors and Technology member and associate vice president of China, Hefei, Dongrui Wu for human–machine systems of the China, in 2003; his SMCS. He will be the editor-in-chief M.E. degree in electrical and computer of IEEE Transactions on Fuzzy Sysengineering from the National Universitems in 2023. ty of Singapore in 2006; and his Ph.D.

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

ergy Systems TC. He received a Best Chun Sing Lai Paper Award from the IEEE InternaAssociate Vice President of Systems tional Smart Cities Conference in OcScience and Engineering tober 2020. Dr. Lai has contributed to Dr. Chun Sing Lai, four journal articles that appeared in a Senior Member Web of Science as highly cited papers, of IEEE, earned out of which he is the lead author for his B.Eng. (First three of them. He has coauthored two Class Hons.) de­­ books: Smart Grids and Big Data gree in electrical Analytics for Smart Cities (Springer, a nd ele c t r o n ic Chun Sing Lai 2021) and Smart Energy for Transengineering from portation and Health in a Smart City Br unel University London, Lon(Wiley, 2023). He is an IET member don, U.K., in 2013, and his D.Phil. and a Chartered Engineer. degree in engineering science from the University of Oxford, Oxford, Jiacun Wang U.K., in 2019. He is currently an honorary visiting fellow of the School Associate Vice President of Automation, Guangdong Univerof Finance sity of Technology, China, a nd a Ji a c u n Wa n g lecturer in circuits a nd dev ices earned his Ph.D. as well as the course director of deg ree i n comthe M.Sc. degree in electric vehiputer eng i neercle systems with the Department of ing from Nanjing Electronic and Electrical EngineerUniversity of Sciing, Brunel University London, U.K. ence a nd TechJiacun Wang From 2018 to 2020, he was a U.K. nology (NJUST), Engineering and Physical Sciences China, in 1991. He is currently a proResearch Council Research Fellow fessor of software engineering at with the School of Civil Engineering, Monmouth University (MU), West University of Leeds, Leeds, U.K. His Long Branch, NJ, USA. He was the MU Computer Scicurrent i­nterests are ence a nd Sof tin power system op­­ ware Engineering timization, energy Dr. Lai was the Department chair system modeling, publications from 2009 to 2015. data analytics, eleccochair for both He h a s b e e n t he tric vehicle systems, graduate program hybrid powertrain the 2020 and 2021 director of the optimization, and IEEE International depar tment since energy economics Smart Cities 2 016 . F r o m 2 0 01 for renewable energy Conferences. to 2004, he was a and storage systems. member of the sciDr. Lai was the entific staff with publications cochair Nor tel Networks in Richardson, for both the 2020 and 2021 IEEE InTX. Prior to joining Nortel, he was ternational Smart Cities Conferenca research associate with the School es. He is the vice chair of the IEEE of Computer Science, Florida InterSmart Cities Publications Commitnational University (FIU) at Miami. tee and associate editor for the InPrior to joining FIU, he was an assostitution of Engineering and Techciate professor at NJUST. nology (IET) Energy Conversion His research interests include and Economics. He is the working s of t w a r e e n g i ne e r i n g , m a c h i ne group chair for the IEEE P2814 and learning, formal methods, discrete P3166 sta ndards, associate v ice event systems, and real-time dispresident of systems science and ent r ibuted systems. He ha s pub gineering of the SMCS, and chair of lished four books, including Forthe SMCS Intelligent Power and En

O c tob e r 2022

mal Methods in Computer Science (C R C P r e s s , 2 019), R e a l - Ti m e Embedded Systems (Wiley, 2017), Finite State-Based Models and Applications (CRC Press, 2012), and Timed Petri Nets: Theory and A pplication (K luwer, 19 98) a nd published more than 130 research papers in peer-reviewed international journals and conferences. He is an associate editor of the IEEE/CAA Journal of Automatica Sinica. He was an associate editor of IEEE Transactions on Systems, Man, and Cybernetics, Part C and has served as general chair, program chair, program cochair, special sessions chair, or program committee member for many international conferences. Dr. Wang has been a Senior Member of IEEE since 2000. György Eigner Associate Vice President of Membership and Student Activities György Eigner, a Member of IEEE since 2017 and a S en ior Member since 2021, earned his B.Sc. degree in mechatronic engiGyörgy Eigner neering at Óbuda University, Bánki Donát Faculty of Mechanical and Safety Engineering, in 2011 and his M.Sc. degree in biomedical engineering at Budapest University of Technology and Economics in 2013. He earned his Ph.D. degree from Óbuda University in 2017. He is the acting dean of the John von Neumann Faculty of Informatics and the head of the Biomatics and Applied Artificial Intelligence Institution, where he is currently an associate professor. His main research focus is the application of advanced control methods in physiological relations, biomedical engineering, humanin-the-loop systems, and artificial intelligence-based cybermedical systems. Having published more than 120 scientific works on these topics, his h-index is seven. He is a member of the Board of Governors of the

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

55

SMCS, cochair of the Computational Cybernetics TC, and the professional leader of Zsámbék Future Industries Science Park of Óbuda University. Long Chen Associate Vice President of Publications Long Cheng earned his B.S. (Hons.) degree in control engineering from Nankai University, Tianjin, China, in 2004 and Long Chen his Ph.D. (Hons.) degree in control theory and control engi­­neering from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 2009. He is currently a full professor with the

the IEEE ComputaInstitute of Autotional Intelligence m a t ion , Ch i ne s e Dr. Cheng has Society, the Aharon Aca demy of Sciauthored and Katzir Young Invesences. He is also an coauthored more tigator Award from ad junct professor than 100 technical the Inter nationa l with the University Neural Networks of Chinese Acadpapers in peerS o c i e t y, a n d t h e emy of Sciences, refereed journals Young Researcher Beijing, China. He and prestigious Aw a r d f r o m t h e has authored and conference Asian Pacific Neural coauthored more Networks Society. than 100 technical proceedings. He is currently servpapers in peer-refing as the associate ereed journals and v ice president for publications prestigious conference proceedings. for the SMCS as well as the associHis research interests include rehaate editor of IEEE Transactions on bilitation robots, intelligent control, Cybernetics, IEEE Transactions on and neural networks. Automation Science and Engineering, Dr. Cheng was the recipient of the Science China Technological Sciences, IEEE Transactions on Neural Netand Acta Automatica Sinica. works Outstanding Paper Award from

Are You Moving? Don’t miss an issue of this magazine— update your contact information now! Update your information by: E-MAIL: [email protected] PHONE: +1 800 678 4333 in the United States or +1 732 981 0060 outside the United States If you require additional assistance regarding your IEEE mailings, visit the IEEE Support Center at supportcenter.ieee.org. IEEE publication labels are printed six to eight weeks in advance of the shipment date, so please allow sufficient time for your publications to arrive at your new address.

© ISTOCKPHOTO.COM/BRIANAJACKSON

56

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE O cto ber 2022

Conference Reports

by Celal Savur 

T

he 17th IEEE International System of Systems Engineering Conference (SoSE 2022) was held in Rochester, NY, USA, between 7 and 11 June 2022. This year, the theme of the conference was “Artificial Intelligence and Machine Learning in Systems of Systems.” The conference was organized in a hybrid mode, where authors presented their work in person or online. The conference was technically sponsored by the IEEE Systems, Man, and Cybernetics Society. The Multi-Agent Biorobotics Laboratory at the Rochester Institute of Technology and Autonomous Control Engineering Lab at the University of Texas, San Antonio (UTSA) were financial sponsors of the conference.

and Ferat Sahin

The 17th IEEE International Conference on Systems and Systems Engineering Brazil 1.3% Estonia 1.7% Israel 2.1% Australia 2.6% Sweden 4.7%

United States 46.3%

South Korea 7.7% China 8.1%

This year, we accepted 73 virtual and inperson papers to the conference from 15 countries worldwide.

Germany 9% France 12%

Figure 1. The statistics for SoSE 2022: the percentage of accepted

papers by country.

This year, we accepted 73 virtual and in-person papers to the conference from 15 countries worldwide (Figure 1). Most of the papers submitted were from the United States, France, Germany, China, and Korea. The technical program consisted of a panel session, three keynote speeches, a tutorial session, and a live demo of a human–robot collaboration experiment where a collaborative robot adapts its behavior using an estimated Digital Object Identifier 10.1109/MSMC.2022.3205492 Date of current version: 20 October 2022

Figure 2. The founder of SoSE, Mo Jamshidi, delivers his keynote speech.



O c tob e r 2022

IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE

57

human comfort index from physiological signals. The first keynote speech was presented by Mo Jamshidi (Figure 2),

founder of the SoSE. In his keynote speech, he talked about the history of the SoSE. It was an honor to have ­Jamshidi as the first keynote speaker.

Figure 3. The list of panelists and their short bios.

Figure 4. Said Navahandi delivering his keynote speech.

Figure 5. Darryl Nelson delivering his keynote speech.

In his keynote speech, Navahandi highlighted how a group of robots could work together to solve a series of complex tasks autonomously in a highly unstructured environment.

Jamshidi retired from UTSA in July 2022. The second event on the first day was the International Council on Systems Engineering (INCOSE) panel (Figure 3). In the panel, Judith Dahman, Garry Roedler, Kerry Lunney, and Alan Harding attended and discussed “A Systems of Systems Perspective on INCOSE Systems Engineering Vision 2035.” Dahman is a technical fellow from the MITRE Corporation, Roedler is a retired senior fellow from Lockheed Martin, Lunney is an engineering director at Thales Australia, and Harding is the head of and an systems engineer/engineering fellow at BAE Systems. The second keynote speaker was Said Navahandi (Figure 4). He is an Alfred Deakin Professor, vice chancellor, chair of engineering, and the founding and current director of the Institute for Intelligent Systems Research and Innovation at Deakin University. In his keynote speech, he highlighted how a group of robots could work together to solve a series of complex tasks autonomously in a highly unstructured environment. Recognition of human intentions, planning, and actions from multimodal signals in robotics and autonomous systems were explored. The progression of developed technologies from the bilateral control of a group of robots, initially with humanin-the-loop; to more advanced cases of on-the-loop; to, finally, a case of a

fully autonomous collection of robots performing a convoying operation independently. Several case studies were presented, and major challenges were discussed. Darryl Nelson is an engineering fellow at Raytheon Intelligence & Space, where he is the technology director of data and software engineer-

ing. Nelson gave the third keynote speech (Figure 5). In his speech, he elaborated on how countering and exploiting emergence, both negative and positive, respectively, at the intersection of artificial intelligence (AI) and system of systems is fundamental to navigating through the entanglement. In addition, he discussed the charac-

Figure 6. Dr. Haibin Zhu delivering his tutorial session.

1% 1% 3% 3% 3%

1% 1%

1% 1% 1% 1% 19%

3% 3% 12%

3% 3% 3% 4%

7% 4% 4%

7% 4%

5%

teristics of this intersection, effective strategies and tools, and predictions for the future. On the second day of the conference, Dr. Haibin Zhu gave a tutorial on E-CARGO and a system of systems (Figure 6). Dr. Zhu is a professor of computer science and mathematics at Nipissing University. At SoSE 2022, there were 25 tracks i n wh ich the author s subm it ted t hei r pa per s ( Figure 7). A I a nd M a chine Learning Application in System of Systems Engineering, SoSE Simulation; Modeling and Analysis Methods; and Robotic Systems, Unmanned Aerial Vehicles, ROVs, and Drones were the tracks that received most of the paper submissions. For more information on SoSE 2022, please visit the conference website at http://www.sosengineering. org/2022. SoSE 2023 is going to take place at the Polytechnic of Lille, Lille, France.

AI and Machine Learning Application in System of Systems Engineering SoSE Simulation, Modeling and Analysis Methods Robotic Systems, UAVs, ROVs, Drones, etc. SoSE Architecture, Design and V & V Methods AI and Machine Learning in System of Systems Engineering AI and Machine Learning in SoSE Simulation, Modeling, and Analysis Methods Management Models for System of Systems Engineering Manufacturing Transportation (Including Civil Air Traffic Control) Agriculture 5.0 Automotive SoSs Cyberphysical Systems and IoT: Engineering Issues Cybersecurity Decision Making Maritime Systems Engineering Software Intensive System-of-Systems Engineering SoSE Availability, Maintainability, Reliability, Resilience, Safety Cloud and Distributed Computing Control Defense, Space, National Security Energy, Smart Grid Technologies Engineering Education Environment Health Care Human-Centered Design

Figure 7. The SoSE 2022 conference track statistics. UAV: unmanned aerial vehicle.