Artificial Intelligence for Autonomous Vehicles: The Future of Driverless Technology (Advances in Data Engineering and Machine Learning) 9781119847465

Citation preview

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence for Autonomous Vehicles

Publishers at Scrivener Martin Scrivener ([email protected]) Phillip Carmical ([email protected])

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Scrivener Publishing 100 Cummings Center, Suite 541J Beverly, MA 01915-6106

Sathiyaraj Rajendran Munish Sabharwal Yu-Chen Hu Rajesh Kumar Dhanaraj

Balamurugan Balusamy

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence for Autonomous Vehicles

Edited by

and

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions. Wiley Global Headquarters 111 River Street, Hoboken, NJ 07030, USA For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com. Limit of Liability/Disclaimer of Warranty While the publisher and authors have used their best efforts in preparing this work, they make no rep­ resentations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchant-­ ability or fitness for a particular purpose. No warranty may be created or extended by sales representa­ tives, written sales materials, or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further informa­ tion does not mean that the publisher and authors endorse the information or services the organiza­ tion, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Library of Congress Cataloging-in-Publication Data ISBN 9781119847465 Front cover images supplied by Pixabay.com Cover design by Russell Richardson Set in size of 11pt and Minion Pro by Manila Typesetting Company, Makati, Philippines Printed in the USA 10 9 8 7 6 5 4 3 2 1

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

This edition first published 2024 by John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA and Scrivener Publishing LLC, 100 Cummings Center, Suite 541J, Beverly, MA 01915, USA © 2024 Scrivener Publishing LLC For more information about Scrivener publications please visit www.scrivenerpublishing.com.

Preface xi 1 Artificial Intelligence in Autonomous Vehicles—A Survey of Trends and Challenges Umamaheswari Rajasekaran, A. Malini and Mahalakshmi Murugan 1.1 Introduction 1.2 Research Trends of AI for AV 1.3 AV-Pipeline Activities 1.3.1 Vehicle Detection 1.3.2 Rear-End Collision Avoidance 1.3.3 Traffic Signal and Sign Recognition 1.3.4 Lane Detection and Tracking 1.3.5 Pedestrian Detection 1.4 Datasets in the Literature of Autonomous Vehicles 1.4.1 Stereo and 3D Reconstruction 1.4.2 Optical Flow 1.4.3 Recognition and Segmentation of Objects 1.4.4 Tracking Datasets 1.4.5 Datasets for Aerial Images 1.4.6 Sensor Synchronization Datasets 1.5 Current Industry Standards in AV 1.6 Challenges and Opportunities in AV 1.6.1 Cost 1.6.2 Security Concerns 1.6.3 Standards and Regulations 1.7 Conclusion References

1 2 4 6 7 7 8 9 12 15 16 16 16 17 17 17 17 19 19 19 19 20 21

v

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Contents

25 2 Age of Computational AI for Autonomous Vehicles Akash Mohanty, U. Rahamathunnisa, K. Sudhakar and R. Sathiyaraj 2.1 Introduction 26 2.1.1 Autonomous Vehicles 27 2.1.2 AI in Autonomous Vehicles 30 2.1.2.1 Functioning of AI in Autonomous Vehicles 30 2.2 Autonomy 31 2.2.1 Autonomy Phases 31 2.2.2 Learning Methodologies for Incessant Learning 33 in Real-Life Autonomy Systems 2.2.2.1 Supervised Learning 33 2.2.2.2 Unsupervised Learning 34 2.2.2.3 Reinforcement Learning 34 35 2.2.3 Advancements in Intelligent Vehicles 2.2.3.1 Integration of Technologies 37 2.2.3.2 Earlier Application of AI in Automated Driving 39 2.3 Classification of Technological Advances in Vehicle Technology 39 2.4 Vehicle Architecture Adaptation 42 2.5 Future Directions of Autonomous Driving 49 2.6 Conclusion 50 References 51 3 State of the Art of Artificial Intelligence Approaches Toward Driverless Technology Sriram G. K., A. Malini and Santhosh K.M.R. 3.1 Introduction 3.2 Role of AI in Driverless Cars 3.2.1 What is Artificial Intelligence? 3.2.2 What are Autonomous Vehicles? 3.2.3 History of Artificial Intelligence in Driverless Cars 3.2.4 Advancements Over the Years 3.2.5 Driverless Cars and the Technology they are Built Upon 3.2.6 Advancement of Algorithms 3.2.7 Case Study on Tesla 3.3 Conclusion References

55 56 58 58 59 60 62 66 69 70 73 73

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

vi  Contents

4 A Survey on Architecture of Autonomous Vehicles Ramyavarshini P., A. Malini and Mahalakshmi S. 4.1 Introduction 4.1.1 What is Artificial Intelligence? 4.1.2 What are Autonomous Vehicles? 4.2 A Study on Technologies Used in AV 4.2.1 Artificial Vision 4.2.2 Varying Light and Visibility Conditions 4.2.3 Scenes with a High Dynamic Range (HDR) 4.2.3.1 3 Dimensional Technology 4.2.3.2 Emerging Vision Technologies 4.2.4 Radar 4.2.4.1 Emerging Radar Technologies 4.2.5 LiDAR 4.2.5.1 Emerging LiDAR Technologies 4.3 Analysis on the Architecture of Autonomous Vehicles 4.3.1 Hardware Architecture 4.3.2 Software Architecture 4.4 Analysis on One of the Proposed Architectures 4.5 Functional Architecture of Autonomous Vehicles 4.6 Challenges in Building the Architecture of Autonomous Vehicles 4.6.1 Road Condition 4.6.2 Weather Condition 4.6.3 Traffic Condition 4.6.4 Accident Responsibility 4.6.5 Radar Interference 4.7 Advantages of Autonomous Vehicles 4.8 Use Cases for Autonomous Vehicle Technology 4.8.1 Five Use Cases 4.9 Future Aspects of Autonomous Vehicles 4.9.1 Levels of Vehicle Autonomy 4.9.2 Safer Mobility Technology 4.9.3 Industry Collaboration and Policy Matters 4.10 Summary References

94 95 95 95 95 96 96 97 98 99 99 100 100 101 102

5 Autonomous Car Driver Assistance System

105

R. Annamalai, S. Sudha Mercy, J. M. Mathana, N. Banupriya, Rajalakshmi S. and S. D. Lalitha

5.1 Introduction

75 76 76 77 78 78 78 78 79 80 80 81 81 82 82 83 83 90 92

106

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Contents  vii

5.1.1 Traffic Video Surveillance 5.1.2 Need for the Research Work 5.2 Related Work 5.3 Methodology 5.3.1 Intelligent Driver Assistance System 5.3.2 Traffic Police Hand Gesture Region Identification 5.3.3 Vehicle Brake and Indicator Light Identification 5.4 Results and Analysis 5.5 Conclusion References

108 110 111 113 113 115 119 123 127 128

6 AI-Powered Drones for Healthcare Applications 131 M. Nalini 132 6.1 Introduction 6.1.1 Role of Artificial Intelligence in Drone Technology 133 6.1.2 Unmanned Aerial Vehicle—Drone Technology 133 6.2 Kinds of Drones Used by Medical Professionals 135 136 6.2.1 Multirotor 6.2.2 Only One Rotor 136 6.2.3 Permanent-Wing Drones 136 6.2.4 Drones for Passenger Ambulances 136 6.3 Medical and Public Health Surveillance 136 137 6.3.1 Telemedicine 6.3.2 Drones as Medical Transportation Devices 138 6.3.3 Advanced System for First Aid for the Elderly People 139 6.4 Potential Benefits of Drones in the Healthcare Industry 140 6.4.1 Top Medical Drone Delivery Services 141 141 6.4.2 Limitations of Drones in Healthcare 143 6.4.3 The Influence of COVID on Drones 6.4.4 Limitations of Drone Technology in the Healthcare Industry 144 6.4.4.1 Privacy 144 6.4.4.2 Legal Concerns 144 6.4.4.3 Rapid Transit—One of the Biggest 144 Drawbacks of Drones is Time 145 6.4.4.4 Bugs in the Technology 6.4.4.5 Dependence on Weather 145 145 6.4.4.6 Hackable Drone Technology 6.5 Conclusion 145 References 146

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

viii  Contents

7 An Approach for Avoiding Collisions with Obstacles in Order to Enable Autonomous Cars to Travel Through Both Static and Moving Environments T. Sivadharshan, K. Kalaivani, N. Golden Stepha, Rajitha Jasmine R., A. Jasmine Gilda and S. Godfrey 7.1 Introduction 7.1.1 A Brief Overview of Driverless Cars 7.1.2 Objectives 7.1.3 Possible Uses for a Car Without a Driver 7.2 Related Works 7.3 Methodology of the Proposed Work 7.4 Experimental Results and Analysis 7.5 Results and Analysis 7.6 Conclusion References 8 Drivers’ Emotions’ Recognition Using Facial Expression from Live Video Clips in Autonomous Vehicles Tumaati Rameshtrh, Anusha Sanampudi, S. Srijayanthis, S. Vijayakumarsvk, Vijayabhaskar and S. Gomathigomathi 8.1 Introduction 8.2 Related Work 8.2.1 Face Detection 8.2.2 Facial Emotion Recognition 8.3 Proposed Method 8.3.1 Dataset 8.3.2 Preprocessing 8.3.3 Grayscale Equalization 8.4 Results and Analysis 8.5 Conclusions References 9 Models for the Driver Assistance System B. Shanthini, K. Cornelius, M. Charumathy, Lekshmy P., P. Kavitha and T. Sethukarasi 9.1 Introduction 9.2 Related Survey 9.3 Proposed Methodology 9.3.1 Proposed System 9.3.2 Data Acquisition 9.3.3 Noise Reduction 9.3.4 Feature Extraction

151 152 152 154 155 155 159 163 165 168 169 173 174 179 179 180 181 182 184 184 184 190 190 193 194 196 198 198 200 200 202

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Contents  ix

9.3.5 Histogram of Oriented Gradients 9.3.6 Local Binary Pattern 9.3.7 Feature Selection 9.3.8 Classification 9.4 Experimental Study 9.4.1 Quantitative Investigation on the NTHU Drowsy Driver Detection Dataset 9.5 Conclusion References

203 203 204 204 204

10 Control of Autonomous Underwater Vehicles M. P. Karthikeyan, S. Anitha Jebamani, P. Umaeswari, K. Chitti Babu, C. Geetha and Kirupavathi S. 10.1 Introduction 10.2 Literature Review 10.3 Control Problem in AUV Control System 10.4 Methodology 10.5 Results References

209

205 207 207

210 211 214 217 220 223

11 Security and Privacy Issues of AI in Autonomous Vehicles 229 K. Ramalakshmi, Sankar Ganesh and L. KrishnaKumari 230 11.1 Introduction 11.2 Development of Autonomous Cars with Existing Review 232 11.3 Automation Levels of Autonomous Vehicles 233 11.4 The Architecture of an Autonomous Vehicle 234 235 11.5 Threat Model 11.6 Autonomous Vehicles with AI in IoT-Enabled Environments 236 11.7 Physical Attacks Using AI Against Autonomous Vehicles 238 11.8 AI Cybersecurity Issues for Autonomous Vehicles 240 11.9 Cyberattack Defense Mechanisms 242 11.9.1 Identity-Based Approach 242 11.9.2 Key-Based Solution 243 244 11.9.3 Trust-Based Solution 11.9.4 Solution Based on Behavior Detection 244 244 11.10 Solution Based on Machine Learning 11.11 Conclusion 244 References 245 Index 247

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

x  Contents

With the advent of advanced technologies in AI, driverless vehicles have elevated curiosity among various segments of the society. The automotive industries are at the technological uprising with Autonomous vehicle concepts. Autonomous driving is one of the crucial application areas of Artificial Intelligence (AI). Autonomous vehicles are armed with sensors, radars and cameras. This made driverless technology possible in many parts of the world. In short, our traditional vehicle driving may swing to driverless technology. The goal of this book is to highlight the efforts as well as obstacles involved in bringing self-driving cars to fruition while offering solutions that are feasible. The book’s primary accomplishment is the collection of AI-based notions for driverless technology, which can give a risk-free, riskclean, and more convenient means of transportation. The book begins with a brief overview of autonomous vehicles, which serves as a survey of the latest developments and the need of AI-based solutions in the present and the future. The architecture of AI-based systems for monitoring and managing autonomous cars is covered in the next part, along with topics like AI drones, using sensors to detect obstructions, and identifying driver emotions to prevent collisions. The final portion covers a variety of AI-based models, autonomous underwater vehicle control, and security and privacy concerns related to these models. This book is perfect for professionals, academicians, researchers, data scientists, data analysts, engineers, marketers, robotic scientists, e-­commerce, and students. This book also outlines the future paths towards research into artificial intelligence on driverless technology, which has the potential to give a risk-free, risk-clean, and more convenient means of transportation.

xi

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Preface

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends and Challenges Umamaheswari Rajasekaran, A. Malini* and Mahalakshmi Murugan Thiagarajar College of Engineering, Madurai, India

Abstract

The potential for connected automated vehicles is multifaceted, and automated advancement deals with more of Internet of Things (IoTs) development enabling artificial intelligence (AI). Early advancements in engineering, electronics, and many other fields have inspired AI. There are several proposals of technologies used in automated vehicles. Automated vehicles contribute greatly toward traffic optimization and casualty reduction. In studying vehicle autonomy, there are two categories of development available: high-level system integrations like newenergy vehicles and intelligent transportation systems and the other involves backward subsystem advancement like sensor and information processing systems. The Advanced Driver Assistance System shows results that meet the expectations of real-world problems in vehicle autonomy. Situational intelligence that collects enormous amounts of data is considered for high-definition creation of city maps, land surveying, and quality checking of roads as well. The infotainment system of the transport covers the driver’s gesture recognition, language transaction, and perception of the surroundings with the assistance of a camera, Light Detection and Ranging (LiDAR), and Radio Detection And Ranging (RADAR) along with localization of the objects in the scene. This chapter discusses the history of autonomous vehicles (AV), trending research areas of artificial intelligence technology in AV, state-of-the-art datasets used for AV research, and several Machine Learning (ML)/Deep Learning (DL) algorithms constituting the functioning of AV as a system, concluding with the challenges and opportunities of AI in AV.

*Corresponding author: [email protected]; ORCID ID: 0000-0002-3324-5317 Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (1–24) © 2024 Scrivener Publishing LLC

1

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

1

Keywords:  Vehicle autonomy, artificial intelligence, situational intelligence, datasets, safety standards, network efficiency, algorithms

1.1 Introduction An autonomous vehicle in simple terms is that its movements are from the start to predecided stop in “autopilot” mode. Autonomous vehicle technology is developed to provide several pros in comparison with human-driven transport. Increased safety on the road is one such potential primacy—connected vehicles could drastically decrease the number of casualties every year. The automated driving software is the most comfortable transport system that highly supports the class of people who could not drive because of age and physical constraints. Autonomous vehicles help them to find a new smart ideas, and it is predicted that it could provide them with different opportunities to work in fields that require driving. Automated decisions are taken without any human intervention, and the necessary actions are implemented to ensure stability of the system. The smart connected vehicles are supported by an AI-based self-driving system that responds to external conditions through the ability to sense their surroundings using Adaptive Driver Control (ADC) that uses lasers and radars enabled with ML and DL technologies. To extract object information from noisy sensor data, these components frequently employ machine learning (ML)-based algorithms. An understandable representation of autonomous vehicles as a system is shown in Figure 1.1. The history of driverless cars took the spark from the year 1478 with Da Vinci creating the first driverless automobile prototype. In Da Vinci’s vehicle, there was a self-propelled robot that was driven by springs. It had programmable steering and could output predetermined routes. This brief history shows where the roots of artificial intelligence may be found in philosophy, literature, and the human imagination. Autonomous vehicles (AVs) were first conceptualized in the 1930s. It was the Houdina Radio Control project that demonstrated a radio-controlled “driverless” car. In the mid-20th century, General Motors took the initiative to develop a concept car called Firebird II. This was considered the basis for the first cruise control car named “Imperial” designed by the Chrysler company in 1958. Back in the 1990s, organizations slowly proceeded by considering safety measures including cruise and brake controls. After the turn of the century, blind-spot detection and electronic stability controls through sensors were made available in self-driven vehicles. One of the biggest achievements in the year 1995 was the VaMP-designed autonomous vehicle that

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

2  Artificial Intelligence for Autonomous Vehicles

LIDAR UNIT

GPS

MAIN COMPUTER IN TRUNK

RADAR SENSOR

ALGORITHMS FOR VEHICLE DETECTION, PATH PLANNING, TRAFFIC SCENE ANALYSIS, DECISION MAKING etc.

Figure 1.1  Representation of the AV system.

drives (almost) by itself for 2,000 km. Between 1995 and 1998, the National AHS Consortium was held for cars followed by the PATH organization that conducted the automated bus and truck demos. In the year 2009, the Google Self-Driving Car Project began; eventually, Tesla Multinational Automotive released the update in autopilot software. Self-driving cars designed by Google, having met with a test drive accident in the year 2016, were considered a major damage to the development of AVs. At the end of 2016, the Bolt and Super Cruise in Cadillac have autonomous controls. At the start of January 2022, Volvo unveiled Ride Pilot, a new Level 3 autonomous driving technology, at the CES consumer electronics expo. Without human input, the system navigates the road using LiDAR, radar, ultrasonic sensors, and a 360-degree camera setup. As of 2019, self-driven vehicles possess the following features: Free-hand handles the steering, but monitoring the system is also needed. With additional improvement in free-hand steering, the adaptive cruise control (ACC) keeps the vehicle at a predetermined displacement from the surrounding objects. When an interruption is encountered like the motorist crossing the lane, the car insists that the system slows down and changes the lane. One of the main problems with AV mobility is how to combine sensors and estimate circumstances to distinguish between risky scenarios and less dangerous ones. AV’s mobility is quite tedious, but continuous

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  3

accepted releases in this field give better mobility to the vehicles. AI gives a fantastic change in the technological revolution. The Department of Transportation (DOT) and NHTSA deal with safety in addition to automation protocols. Section 1.2 of this chapter presents a detailed survey of machine learning and deep learning algorithms used in the AV literature where the important AV-pipeline activities are analyzed individually. Section 1.3 presents a survey of the state-of-the-art datasets used in autonomous vehicles, and Section 1.4 discusses the industry standards, risks, challenges, and opportunities, with Section 1.5 summarizing the analysis.

1.2 Research Trends of AI for AV To provide readers a comprehensive picture of the status of the literature in AI on AV, the current research trends of ML/DL algorithms in the design of autonomous vehicles are presented. The systems rely on several technological advancements in AI technicalities. When looking into the automation ideology in vehicles, it comprises six different levels. Starting from level 0, the human driver operates with no self-control in the cars. We could say that there is no self-working at level 0. At the first level, the Advanced Driver Assistance System (ADAS) in vehicles supports the driver with either accelerating or steering controls. In addition to level 1, the ADAS provides brake controls to the system such as Automated Emergency Braking System (AEBS). However, the driver must pay complete attention to the surroundings—level 2. At level 3, almost all automation processes are performed by the system, but driver’s access is required in certain operations. Real-time high-definition maps must be accessible, complete, and detailed. These maps are necessary, and they are used to decide their path and trajectory. Further development of level 3 is that the vehicle’s advanced driving system (ADS) provides complete automation without a human even if a person does not respond to a request in some circumstances. A vehicle’s six degrees of freedom are indicated by the stance, which is monitored by AV pose sensors: x, y, z, ϕ, θ, and ψ; here, x, y, and z depicts actual positions of the system at level 4. Eventually, at level 5, virtual chauffer concept is introduced by ADS along with ACC and does all the driving in all circumstances. Autonomous vehicles can be successfully implemented in a range of use case situations combining the complete working of sensors, actuators, maps, detectors, etc. It is necessary to demonstrate the functionalities in real-life environments like urban and rural developments combining smart vehicles.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

4  Artificial Intelligence for Autonomous Vehicles

LiDAR technology ensures the functional and safe operation of an autonomous vehicle. LiDAR is more efficient in creating 3D images of the surroundings that is considered critical in urban settings [1]. A rotating roof-mounted LiDAR sensor creates and maintains a live 3D map of a 60-m range of surroundings and thus generates a specific route for travel to the destination specified by the driver. Radars are mounted to measure the distance between obstructions and are placed at the front and rear. To determine the car’s position in the lane concerning the 3D map, a sensor on the left rear wheel tracks and signals the system’s sideway movement. All of the sensors are connected to the AI program, which receives the dataset in accordance. Thirdly, collaborative mapping efforts gather massive amounts of raw data at once utilizing GPS technologies to recover information on traffic signals, landmarks, and construction projects that cause deviations. By acquiring more knowledge on the NVH behavior of the entire vehicle, the vehicle’s performance systems can be improved. Utilizing Simcenter NVH systems has additional advantages, such as AI-based strategies for improving the effectiveness of NVH testing, and NVH performance prediction without the need for intricate simulation models. After the development of the 3D models, the developers focused on substances in the surroundings. A revolutionary YOLOv2 vehicle architecture has been developed to address accuracy-based concerns, enhance sluggishness identification, and solve the lagging in classification criteria. An autonomous driving system was overlaid with augmented reality (AR), displayed on the windshield. AR vehicle display systems are necessary for navigation. Situational awareness has been considered as the command-line interface that was developed using AR technology. The INRIA dataset was used to identify vehicles and pedestrians as part of the demonstration of an AR-HUD-based driving safety instruction based on an article released by SAE International. SVM and HOG algorithms have been used to build the identification method, which detected partial obstacles with 72% and 74% accuracy in frames per second [2]. Since improvement in the above system has been demanded, technologists modeled a better version through the use of stereo cameras and augmented reality. The forward collision alert system detected vehicles and people and showed early warnings while applying an SVM classifier for apprehension that received 86.75% for identifying cars and 84.17% for identifying pedestrians [3]. Although it was challenging to establish the motion-detecting strategy, the model was able to control the car in confusion situations. This method established a rule base to handle illogical situations. The method might complete 45.6 m of designing a path in

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  5

50.2  s [4]. The study suggested the use of R-tree data structure and the continuous nearest neighbor search [5]. To simulate 3D rain, stereo pictures are used. The method can be used for already existing photographs, preventing the need to retake. The Recurrent Rolling Convolution (RRC) for object prediction and KITTI dataset were employed in the study’s tests at the Karlsruhe Institute of Technology. The results of testing 450 photographs reveal that the algorithm’s measured average precision decreases by 1.18% [6]. The most likely to happen 5G-enabled vehicle autonomy says that through C-RAN, autonomous and connected vehicles running on 5G will be able to access cloud-based ML/AI to make self-drive decisions. In high-mobility settings, edge computing and 5G can greatly speed up and improve data decryption. In these cases, edge computing-based data analysis and processing are automatically executed in the servers that are nearest to the cars. This might significantly lessen data traffic, remove Internet constraints for handling greater speed in data transfer, and lower the expense of transmission [7]. The researchers confirm that high-speed Internet access can improve data transaction, and eventually, autonomous driving is highly benefited by the same. The stochastic Markov decision process (MDP) has been used to simulate how an autonomous vehicle interacts with its surroundings and learns to drive like an expert driver. The MDP model considers the road layout to account for a wider range of driving behaviors. The autonomous vehicle’s desired expert-like driving behavior is achieved by choosing the best driving strategy for the autonomous car. A deep neural network (DNN) has been used to make an approximation of the expert driver’s unknown reward function. The maximum entropy principle (MEP) has been used to train DNN. The simulations show the ideal driving characteristics of an autonomous vehicle. The use of the MEP has been modeled to train a DNN’s reward function. Additionally, essential derivations for applying the MEP to know the applications and workings of a reward function have also been offered (certain functionalities) [8].

1.3 AV-Pipeline Activities Each domain was examined by looking at various approaches and techniques and comparing their benefits, drawbacks, results, and significance. To hasten the construction of a level 4 or 5 AVS, the examination of each domain is summarized as follows.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

6  Artificial Intelligence for Autonomous Vehicles

1.3.1 Vehicle Detection This section analyzes deep learning methods, which are used for quicker and more precise vehicle identification and recognition in a variety of unpredictable driving situations. This paper suggested an online network architecture for identifying and tracking vehicles. Using the KITTI dataset [9] and a trajectory approximation of LSTM, the framework changed 3D poses when instances were moved around in a global coordinate system, outperforming long-range LiDAR results. LiDAR obtained 350.50 false negatives in a 30-m range. LiDAR-based false-negative scores for 50-m and 10-m tests have been mentioned as 857.08 and 1,572.33 [10]. To improve recognition speed, and as a way of resolving it, a novel YOLOv2 vehicle architecture was proposed [11]. One of the main problems with the real-time traffic monitoring approach was the low resolution of the images, which may be attributed to factors like poor weather. Vehicles in low-resolution photographs and videos were examined for this issue in terms of CNN’s effectiveness. The architecture utilized by the neural network operated in two stages: first, it detected high-level features, and then it detected low-level attributes. It evaluated the model’s capability to recognize automobiles at various degrees of input resolution. Results state that CNN is surprisingly positive in the identification even with low resolution [12]. Liu et al. demonstrated the least complex approach for AVS by combining a lightweight YOLO network. The technique was applied to a dataset that was created by themselves and, within 44.5 ms, achieved 90.38% precision. This could be a turning point for AVS, a quicker and more precise solution for diverse fields of vision and implementation during the day or at night [13].

1.3.2 Rear-End Collision Avoidance Collision negatively influences road safety and the assurance for a safe ride. Wide and varied publications emphasize extensively for proposing methods to reduce collisions based on different approaches such as collision avoidance, collision warning, and collision risk assessment (VANET). These algorithms were implemented in vehicular ad hoc network. In this paper [14], the authors proposed a collision avoidance theory model. This model works by giving weights to the inputs. Certain characteristics like driver’s age, visual power, mental health, and physical health conditions are provided to the algorithm. The algorithm computes the weights for the characteristics, processes the input, and generates the output. Generally, the output is always a warning. The implementation of the algorithm was done

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  7

in MATLAB, and the other platform of implementation was Vissim. The above paper was completely based on human factors. Whereas the other model called collision warning that was discussed by the authors considers human factors and weather situations and timings. This algorithm highly focuses on the visual power of humans and puts forth a visuality-based warning system. The MATLAB software and PreScan commercial software were used for testing the visuality-based warning system. This system is accepted to provide better performance than the other warning systems. To prevent crashes, a Multi-Factor-Based Road Accident Prevention System (MFBRAPS) was suggested. The implementation made use of MATLAB and NetLogo. Based on a previous study, the authors are retrieving new significant elements that might aid collision avoidance algorithms in a more efficient manner in this work. They added these fresh parameters to MFBRAPS to improve the V2V collision warning system. The suggested algorithm activates the various functions of the driver assistance system to control the speed and apply the brake after estimating the likelihood of an accident. The alarm is produced as necessary by the driver assistance system [15].

1.3.3 Traffic Signal and Sign Recognition The literature on traffic sign recognition is very vast, and there are so many unsolved questions yet to be addressed by researchers from all over the world. Traffic sign detection is indispensable in the design of the autonomous vehicle, as the signs communicate commands to the driver in place to take appropriate actions. The results of traffic sign recognition using the GTSRB dataset is shown in Figure 1.2. A novel structure named Branch Convolutional Neural Network (B-CNN) is now changed into site visitors’ signal identification. A branch-output mechanism was introduced to the architecture and inserted between the pooling and convolutional layers to increase the recognition along with the machine’s speed and accuracy. The B-CNN-based technique performed exceptionally well in complicated visual settings [17]. The LeNet-5 CNN architecture was used to train the 16 varietal Korean traffic. The practice set consisted of true positives—25,000—and false positives—78,000 [18]. Additionally, a convolutional traffic light identification feature for AVS based on Quicker R-CNN is appropriate for traffic light identification as well as categorization. They applied their method to a sizable dataset called DriverU traffic light and got an average precision of 92%. However, restrictions on false positives will be depreciated by using a strategy or an RNN [19]. Traffic sign recognition based on the HSV color model by the LeNet-5 architecture with Adam

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

8  Artificial Intelligence for Autonomous Vehicles

Figure 1.2  Traffic signs from the GTSRB dataset [16].

optimizer was proposed and tested with the German technology-based traffic identification and yielded a median accuracy of 99.75 with 5.4 ms per frame [20]. DeepTLR traffic light recognition and classification system are a real-time vision-dependent, intricate, and very complex system that did not want positional information or temporal principles [14]. The GTSRB dataset served as the training set for majority of the deep learning techniques. In the LetNet-5-based CNN on a self-created dataset with spatial threshold segmentation using the HSV, the GTSRB dataset scored the best for traffic sign identification, despite a decline in performance in a complicated environment and the detection of separated signs. Despite a drop-off act in a composite setting and the apprehension of divided indications on account of the submitted extent, the GTSRB dataset outperformed the LetNet-5-based CNN on a designed dataset accompanying geographical opening separation adopting the HSV. Despite relying on prelabeled data, the YOLO method-based technique for traffic light detection with recognition of interior signs obtained the highest accuracy in the shortest amount of time from the above discussed [21].

1.3.4 Lane Detection and Tracking Lane detection and tracking have been considered as essential modules in a complete autonomous vehicle system. In various literatures, it has been considered as an important field of research in autonomous vehicles, and many algorithms have been proposed to solve the encountered problems of

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  9

lane detection and tracking. It is an important module that solves the perception problems of autonomous vehicles. In comparison with straight lane detection, some literatures introduce curved road lane detection tracking as challenging and important to avoid accidents. Figure 1.3 and Figure 1.4 shows straight lane line and curved lane line respectively. This paper categorized the lane detection methods into three as feature-based, model-based, and methods based on other trending technologies and discussed a novel curved road detection algorithm, where the road images were initially divided into ROI and the background. The ROI has been further divided into linear and curved portions. The proposed approach evolved as a series of mathematical equations and Hough transformation. It has been concluded that the proposed algorithm effectively identifies lane boundaries and to be effective enough to assist drivers, improving safety standards [22].

Figure 1.3  Straight lane line.

Figure 1.4  Curved lane line.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

10  Artificial Intelligence for Autonomous Vehicles

This paper presented a lane detection and tracking approach based on Gaussian sum particle filter. The proposed feature extraction approach in this paper was based on the linear propagation of boundary and lane points while zooming in. Three types of algorithms used for tracking, namely, SIR particle filter, Gaussian particle filter, and the used GSPF, are compared, and it has been concluded that GSPF outperforms the other two widely used tracking algorithms [23]. This paper presented a survey of trends and methods in road and lane detection, highlighting the research gaps. Seven different modalities, namely, monocular vision, LiDAR, stereo imaging, radar, vehicle dynamics, GPS, and GIS, have been utilized for perception of roads and lanes. The survey highlighted the gap in the literature of perception problems of multiple-lane roads and nonlinear lanes. Detection of lane split and merges has been proposed as an unnoticed perception problem to be solved in the near future for the development of fully autonomous vehicles. Lack of established benchmarks has also been discussed as an important issue in order to compare the currently efficient methods in different scenarios [24]. This paper compared the computer vision-based lane detection techniques with sensor-based algorithms. The traditional modules in conventional computer vision-based lane detection algorithms as highlighted in the survey are image preprocessing, feature extraction, fitting the lane, and the tracking module. Popular edge detection operators such as Canny, Sobel, Prewitt, Line detection operators, and thresholding techniques have been highlighted as the popularly used feature extraction techniques based on the assumptions in the structure, texture, and color of the road. The potential of deep learning for automatic feature extraction and to learn the environment with most variance conditions has been highlighted. The survey also criticized the lack of benchmarked datasets and standard evaluation metrics to test the efficiency of the algorithms [25]. In this paper [26], the authors presented a detailed survey of deep learning methods used in lane detection. The survey categorized the state-of-the art methods into two based on the number of modules as two-step and single-step methods based on feature extraction and postprocessing. Classification, object detection, and segmentation-based architectures have been detailed as three types of model architectures used for lane detection. The paper also discussed the evaluation metrics of different algorithms based on the TuSimple dataset, which has been regarded as the largest dataset for lane detection. Based on the analysis, the CNNLSTM(SegNet+) architecture has been reported to have the highest accuracy of 97.3%. The computational complexity of deep learning models like CNN and RNN has been emphasized as a disadvantage for implementation on low-memory devices. In this paper [27], the authors applied a new CNN

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  11

learning algorithm for lane detection, namely, Extreme Learning Machine (ELM), to reduce the training time, also by using a relatively smaller dataset for training. The training time of the ELCNN has been reported to be 800 times faster, and the experimental results established that incorporating ELM into CNN also increases the performance of the model based on accuracy. This paper emphasized an edge-cloud computing-based CNN architecture for lane detection, where the concepts of edge computing and cloud computing have been introduced to increase the data-processing efficiency for real-time lane detection. Their approach achieved fair accuracy than other state-of-the art algorithms in challenging scenarios of insufficient light, shadow occlusion, missing lane line, and curved lane line. The accuracy achieved on normal road conditions was 97.57, and all of the evaluation metrics have been derived based on the TuSimple dataset [28]. In this paper [29], the author proposed a multiple-frame-based deep learning lane detection architecture using CNN and RNN to avoid the ambiguities that arise due to detections based on a single frame in the presence of shadow, occlusion of vehicles, etc. The continuous scene-based architecture has CNN layers, where the multiple-frame input-outputs constitute time series data that are fed into the recurrent neural network. The CNN architecture acts as the encoder decoder circuitry for reducing the size of the input, and the LSTM architecture determines the next scene, thus detecting the lane. The work laid a strong foundation for using CNN-RNN architectures for detecting lanes in challenging scenarios. It established the idea that the situations that cannot be analyzed using a single-image input can be improved by using a multiple-frame input architecture. In this paper [30], the authors introduced YOLO-based obstacle and lane detection algorithms based on video inputs and TuSimple datasets. The inverse perspective transform has been used for constructing the bird’s-eye view of the lane. The deep learning models are criticized for their poor performance, and their YOLO-based architecture achieved an accuracy of 97.9% for lane detection within a time span of 0.0021 s. This increased speed of processing has been claimed advantageous for implementation in realtime autonomous vehicle scenarios.

1.3.5 Pedestrian Detection One of the main vision-based issues for AVS is accurately recognizing and localizing pedestrians on roads in a variety of settings. The study of vehicle autonomy aims at increasing the accuracy in identifying and localizing the pedestrians to prevent fatalities. Numerous studies have been successful in lowering accident rates and developing a more accurate and long-lasting

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

12  Artificial Intelligence for Autonomous Vehicles

approach to autonomous driving technologies. The approaches used for pedestrian detection can be broadly categorized into two as those that use man-in-the-loop feature extraction and deep learning-based methods. For pedestrian detection, a most sought deep learning model is the convolutional neural network, which has been used in many studies to detect pedestrians. In that way, a deep grid named large-field-of-view (LFOV) was imported to uniformly skim perplexing representations of walkers. The submitted structure was created to analyze and create categorization judgments across a broad extent of neighborhoods concurrently and efficiently. The LFOV network can implicitly reuse calculations since it analyzes enormous regions at higher speed that are substantially faster than those of standard deep networks. This pedestrian detection system demonstrated a convincing performance for practical application, taking 280 ms per image on GPU and having an average miss rate of 35.85% [31]. For passersby discovery, a new CNN structural version was proposed [32]. To annotate the positions of pedestrians in the model, they used synthetic photographs and transfer learning, together with a bounding box suggestion for an uncovered region. When crowded scenarios were taken into account, it got a 26% lost count in the CUHK08 dataset and a 14% missing proportion in the Caltech walker dataset. The major benefit was that it did not need a region proposal method and did not require explicit detection during training. In this paper [33], the authors presented a detailed survey of the state-ofthe-art pedestrian detection algorithms in the domain of automobiles and safety. The position of humans on the lane, pose, color, environment, weather, texture, etc., increase the complexity of the pedestrian detection problem, making the classical template matching approach to be inefficient for realtime detections. The feature-based detection approach requires man-in-theloop feature extraction methods, whereas deep learning models self-learn features during training. This has been highlighted as an advantage of deep learning-based detection methods where the models can achieve fair accuracy with a huge training set; however, both the approaches have been considered to be implemented on par with each other. CNN has been regarded as the most widely used algorithm for pedestrian detection. Bounding box, missing rate, TP, TN, FP, FN, False Positives per Window, AMR, and IoU are regarded as the most commonly used evaluation metrics. It has been concluded that accuracy and cost are still a trade-off to be balanced in pedestrian detection, as the robust solution to this complex problem is yet to be proposed more efficiently. In this paper [34], the authors exemplified the similarities between pedestrian detection pipeline and object detection pipeline and conducted experiments to optimize a CNN-based detection

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  13

pipeline for pedestrian detection. Sliding Window, Selective Search, and LDCF algorithms have been tested for best efficiency in the selection of candidate regions. Selective Search has been concluded not suitable, the Sliding Window approach achieves better recall, and the LDCF algorithm has been concluded as the best region proposal algorithm. Their approach has been evaluated based on the Caltech-Pedestrian dataset using error metrics such as MR and FPPI. The entire experiment has been tested on the NVIDIA Jetson TK1, the most widely used hardware in smart cars. In this paper [35], the authors proposed a hybrid approach by combining HOV, LUV, and CNN classifier, namely, the multilayer-channel feature framework. It has been discussed that by eliminating the overlapping windows, the cost of CNN has been reduced. The proposed approach has been evaluated in the Caltech, KITTI, INRIA, TUD-Brussels, and ETH datasets. The performance of MCF has been considered fair in comparison with the state-of-the-art methods. In this paper [36], the authors proposed a feature sharing-based deep neural network architecture for pedestrian detection, thereby reducing the computational cost of feature extraction during the model training phase. Feature sharing has been emphasized among the different ConvNet-based detectors that differ in their model window sizes. INRIA and Caltech have been used for the evaluation of their approach, and it has been reported using four ConvNet detector units; the time for detection decreased almost by 50%. In this paper [37], the authors criticized the slow detection speed of conventional deep learning algorithms and used YOLO architecture for pedestrian detection. The experiments have been trained using the INRIA dataset and a few selected images from PASCAL VOC. YOLOv2 has been concluded to have a higher detection speed than the presented approach in the paper. In this paper [38], the authors proposed a two-modular pedestrian detection architecture based on convolutional neural networks. In the region generation module, contrasting to image pyramids, feature pyramids are used to capture features of different resolutions. The Modified ResNet-50 architecture has been used as the background network of both modules. The deep supervision-based region prediction module thus achieved a precision score of 88.6% and an MR of 8.9%. It has been concluded that a fair trade-off balance has been achieved between real-time predictions and accuracy. In this paper [39], the authors proposes a convolutional neural network architecture for real-time pedestrian detection tasks. The approach signified a lightweight architecture for real-time detection and warning to avoid accidents. A ­regression-based problem-solving approach has been implemented in contrast to the conventional classification modules to increase the inference speed. Pruning and quantization techniques have been highlighted as the model compression techniques employed. The observed inference speed was

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

14  Artificial Intelligence for Autonomous Vehicles

35 frames per second with a maP of 87% in the specifically designed hardware for autonomous vehicles. Hazy weather pedestrian detection has been considered even more challenging than the clear-day pedestrian detection. In this paper [40], the authors proposed a hazy weather pedestrian detection framework based on a modified YOLO architecture, reducing cost, thereby improving accuracy. The contributions made in the research work also include a hazy weather pedestrian detection dataset. They present three YOLO-based detection models, namely, Simple-YOLO, VggPrioriboxesYOLO, and MNPrioriboxes-YOLO. The MNPrioriboxes-YOLO that uses depth-wise convolutions based on the MobileNetV2 architecture has the least number of parameters among all of the other methods. It has been concluded that the strategies employed reduced the number of parameters, thereby decreasing the running time of the algorithm. In this paper [41], the authors presented an approach to pedestrian detection in nighttime scenarios using multispectral images based on CNN. The authors studied different image fusion and CNN fusion methods. The pixel-level image fusion technique has been considered superior to CNN fusion methods. All of the experimental conclusions presented in the research work were based on the KAIST multispectral database. In this paper [42], the authors considered the multispectral approach to pedestrian detection as a solution to solve challenges encountered in cases of detection in poor illumination, occlusion, etc. The R-FCN-based detector has been used, and information on both the color and thermal images is fused using a network in the network model. Their approach evaluated based on the KAIST dataset revealed that the proposed approach has better performance than the other state-of-the-art methods implemented for multispectral images.

1.4 Datasets in the Literature of Autonomous Vehicles Datasets contain advanced and to a certain extent upgraded information in many scholarly studies by benefiting to solve legitimate-experience models of issues. They enable quantitative analysis of methods, revealing important information about their strengths and weaknesses. The performance of algorithms must be ensured on real-world datasets. Algorithms used, for instance, must deal with complicated objects and settings while contending with difficult environmental factors like direct illumination, shadows, and rain. This section surveys the current datasets used in the literature of autonomous vehicles.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  15

1.4.1 Stereo and 3D Reconstruction To evaluate the effectiveness of stereo matching algorithms, the Middlebury stereo was developed in 2002 that created the Middlebury Multi-View Stereo (MVS). Being only two scenes in size, the benchmark was crucial in the development of MVS methods. In comparison, 124 diverse scenarios were captured in a controlled laboratory environment. Light scans from each camera position are combined to create reference data, and the resulting scans are extremely dense, with an average of 13.4 million points per scan. The whole 360-degree model for 44 sceneries were created by rotating and scanning in contrast to the datasets so far. High-resolution data were captured. DSLR images and synced stereo videos in various interior and outdoor settings were also included. With the use of a reliable procedure, all images can be registered using a high-precision laser scanner. The evaluation of a thorough 3D reconstruction is made possible by high-­ resolution photographs [43].

1.4.2 Optical Flow The mobility concept in AI has a huge capability and upgrades the working efficiency by collecting enormous amount of data, and norms will direct the system to the output platform; thereby, it meets the goal of vehicle autonomy. A toothbrush is used to track concealed fluorescent presence on the objects in all non-fixed sequences to decide the base reality progress. The dataset is made up of eight distinct sequences with a frame count of 8 each. For each sequence, a pair receives the ground truth flow [44]. Contrary to other datasets, the Middlebury dataset offers extremely accurate and deep ground truth, which enables the evaluation of sub-pixel precision. Accurate reference data are generated by monitoring pixels over extensively sampled space-time volumes using high-speed video cameras. With this technique, optical flow ground truth may be automatically acquired in difficult everyday settings, and realistic elements like motion blur can be added to compare approaches under various circumstances.

1.4.3 Recognition and Segmentation of Objects The Microsoft COCO dataset was introduced in 2014 for object detection. They offer visual representations with intricate scenarios with typical items established in their normal surroundings. The count of situations by means of division for Microsoft COCO is indeed above in consideration of the PASCAL VOC target separation standard. The dataset contains 328k

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

16  Artificial Intelligence for Autonomous Vehicles

photographs, 2.5 million commented details, and 91 object classes. In a significant crowd, all objects have per-instance segmentations tagged on them. The intersection-over-union measure is employed for evaluation, much like PASCAL VOC [45].

1.4.4 Tracking Datasets Multi-object tracking benchmarks consists of 14 difficult video sequences shot in unrestricted contexts with both stationary and moving cameras. The trio groups were annotated: moving or stationary walkers, humans who are not standing up straight, and others. They assess the approaches using two well-known tracking standards, MOTA—Multiple Object Tracking Accuracy, and the other one is MOTP—Multiple Object Tracking Precision [46].

1.4.5 Datasets for Aerial Images Data from air sensors for the detection of urban objects as well as the reconstruction and segmentation of 3D buildings were indicated by the ISPRS benchmark. It is made up of the Vaihingen and Downtown Toronto databases. The scene in a road setup is the object classes taken into account in the object detection process. Three locations with different object classes and sizable road detection test methods are available in the Vaihingen dataset. There are two smaller sections for building reconstruction and object extraction, similar to Vaihingen.

1.4.6 Sensor Synchronization Datasets A camera’s sync connects different levels and angles of photographs. The exposure trigger time is represented by the time stamp of the image, and the full rotation of the current LiDAR frame is represented by the time stamp of the LiDAR scan. This method typically produces good data alignment since the camera’s sync is instantaneous [47].

1.5 Current Industry Standards in AV Innovative technologies like AVs come with risks and consequences that could reduce society’s acceptance. These risks include those related to the environment, market, society, organizations, politics, economy, technology, and turbulence. The examination of 86 documents produced by

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  17

29 leading AV technology firms summarizes the industry standards. The main topic of this essay is a technological risk that is defined as potential adverse social, economic, and physical effects linked to individuals’ worries about utilizing cutting-edge technologies. The consequence is that since no single framework can adequately account for the wide range of potential machine decisional applications, we should not try to create a generic “one size fits all” model of artificial intelligence application. AVs are related to five different categories of technological risk: security, privacy, cybersecurity, liability, and industry impact. Governments must implement new policies and laws to resolve the risks connected with AVs to guarantee the society with benefits as much as possible from the developing AV sector. According to estimates, human error is to blame for at least 90% of automobile collisions. By surpassing drivers (humans) in perception, decision-making, and execution, AV adoption potentially reduces or eliminates the main cause of auto accidents. According to authors Collingwood and Litman, very less drivers use seatbelts and pedestrians may become less cautious as a result of feeling safer. This means that a nonlinear relational model of classification notions must be constructed and that the technology must be accepted at face value to appropriately frame each unique choice situation. Additionally, the absence of human mistakes does not imply the absence of mechanical error. The possibility of scientific weaknesses risking vehicle security hikes in addition to the ramification of telecommunications. The catastrophic Tesla automatic steering system accident in 2016 emphasizes the obstructions of components’ skill in order to avoid accidents and discovered the vagueness of the developed arrangement understanding. Concerns are raised about how “crash algorithms” should be used to program AVs to react in the event of inevitable accidents. Rules to control AVs’ responses to moral standards are necessary because the harm produced by AVs in accidents cannot be subjectively assessed due to the “absence of responsibility.” The overall tone of the AV reports examined for this study is predominantly favorable. However, this finding must be taken in the context of the fact that these reports are prepared for a specific group of stakeholders, including investors, consumers, and regulatory bodies. The AV industry papers make numerous references to ethical dilemmas, even though deficient in the definiteness and insight of the descriptions in the experimental essay. It manifests that prevalent reliability, sustainability, comfort, human listening, control, and scrutinizing, as well as the ­science-policy relationship, are the ethical issues that were handled by most organizations, with safety and cybersecurity coming in second.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

18  Artificial Intelligence for Autonomous Vehicles

1.6 Challenges and Opportunities in AV Without a doubt, the driverless car has many benefits, such as providing a means of transportation for people who cannot drive and reducing the driver’s stress on roads. However, in addition to these positive outcomes, numerous obstacles were encountered and they have to addressed to implement a successful AV model. The subsequent are few of the predominant difficulties alongside driverless jeeps:

1.6.1 Cost Numerous automakers had to invest a significant sum of money in constructing these autonomous vehicles. One can use Google as an example, which pays about $80,000 for one of its AV models, making it completely out of reach for the average person or business. This price is expected to decrease by half in the future, which is still more affordable, according to projections. According to a recent JD Power survey, 37% of consumers say they will select an autonomous vehicle as their next vehicle in the future.

1.6.2 Security Concerns The largest problem with electronic systems is usually security and privacy. The AI system that autonomous vehicles are based on needs Internet connection to manage and transmit information, making it a vulnerable medium that hackers can exploit. The second main worry is the possibility of terrorist action, where the autonomous car platform could provide a convenient location for them to execute their self-murder duty. Moreover, the jeeps depend on GPS schemes; anybody can operate them for malicious purposes by gaining access to them.

1.6.3 Standards and Regulations There are some rules and regulations that must be established before autonomous vehicles can be used. Not only the owner of these vehicles but also the automakers using this technology must formally adopt and strictly adhere to these criteria. The United States has proposed the following laws and guidelines that include Nevada (NRS 482.A and NAC 482.A) and California (Cal Veh. Code Division 16.6).

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  19

1.7 Conclusion This chapter highlighted the research flows and issues related to artificial intelligence in autonomous cars, elaborating on the history of autonomous vehicles. SVM is identified as the ML algorithm that has been given significant importance for the development of classification models in AV literature. Other than SVM, mostly deep learning algorithms such as CNN, R-CNN, and YOLO architectures are utilized for all of the object recognition and classification jobs. The performance of the model highly depends upon the quality of data. Deep learning algorithms require huge datasets to achieve fair accuracy. KITTI, KAIST, COCO, PASCAL VOC, GTSRB, INRIA, and TuSimple are some of the benchmarked datasets observed to be widely used in the survey. In the AV pipeline activities, machine learning-based algorithms demand the man-in-the-loop feature extraction method; hence, deep learning has been suggested as the better alternative. However, due to the computational complexity of the deep learning models, they are not suitable for real-time implementation. The literature on AI in AV profoundly suggests the design of both less costly and more accurate design models solving the problems of AV pipeline activities. For object detection and recognition activities, YOLO replaced the traditional convolutional architectures. For edgebased implementations, in a few research papers, edge-compatible lightweight convolutional architectures are proposed using techniques such as depth-wise separable convolutions. Lack of proper and benchmarked datasets for the comparison of the state-of-the-art algorithms has been reported in a few literatures. Several studies also suggested that the performance of the algorithms may not be the same in all of the datasets; hence, without proper comparison based on datasets, the efficiency of the algorithms cannot be concluded. The modules described in the AI pipeline, for example, pedestrian detection, has been regarded not only in the literature of AV but also in other domains of research where there are necessities to identify humans. The literature of AI in autonomous vehicle can be concluded as the booming field of research with more and more focus laid on the development of self-driving cars, ADAS, etc. With the advancements of cloud computing, edge computing, and the swiftly refined designs of edge-compatible GPU, the research in AV promises advanced outcomes in the future.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

20  Artificial Intelligence for Autonomous Vehicles

References 1. Khayyam, H., Javadi, B., Jalili, M., Jazar, R.N., Artificial intelligence and internet of things for autonomous vehicles, in: Nonlinear Approaches in Engineering Applications, pp. 39–68, Springer, Cham, 2020. 2. You, C., Lu, J., Filev, D., Tsiotras, P., Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning. Robot. Auton Syst., 114, 1–18, 2019. 3. Park, H.S., Park, M.W., Won, K.H., Kim, K.H., Jung, S.K., In-vehicle AR-HUD system to provide driving-safety information. ETRI J., 35, 1038–1047, 2013. 4. Yoon, C., Kim, K., Park, H.S., Park, M.W., Jung, S.K., Development of augmented forward collision warning system for Head-up display, in: Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, China, pp. 2277–2279, 8–11 October 2014. 5. Li, X. and Choi, B.-J., Design of obstacle avoidance systems for mobile robots using fuzzy logic systems. Int. J. Smart Home, 7, 321–328, 2013. 6. Liu, Z., Jiang, H., Tan, H., Zhao, F., An overview of the latest progress and core challenge of autonomous vehicle technologies, in: MATEC Web of Conferences, vol. 308, p. 06002, EDP Sciences, 2020. 7. Elfatih, N.M., Hasan, M.K., Kamal, Z., Gupta, D., Saeed, R.A., Ali, E.S., Hosain, M.S., Internet of vehicle’s resource management in 5G networks using AI technologies: Current status and trends. IET Commun., 16, 5, 400– 420, 2022. 8. You, C., Lu, J., Filev, D., Tsiotras, P., Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning. Robot. Auton. Syst., 114, 1–18, 2019. 9. Geiger, A., Lenz, P., Stiller, C., Urtasun, R., Vision meets robotics: The kitti dataset. Int. J. Robot. Res., 32, 1231–1237, 2013. 10. Hu, H.-N., Cai, Q.-Z., Wang, D., Lin, J., Sun, M., Krahenbuhl, P., Darrell, T., Yu, F., Joint monocular 3D vehicle detection and tracking, in: Proceedings of the IEEE International Conference on Computer Vision, pp. 5390–5399, Seoul, Korea, 27 October–2 November 2019. 11. Sang, J., Wu, Z., Guo, P., Hu, H., Xiang, H., Zhang, Q., Cai, B., An improved YOLOv2 for vehicle detection. Sensors, 18, 4272, 2018. 12. Bautista, C.M., Dy, C.A., Mañalac, M., II, Orbe, R.A., Cordel, M., Convolutional neural network for vehicle detection in low resolution traffic videos, in: 2016 IEEE Region 10 Symposium (TENSYMP), pp. 277–281, IEEE, 2016, May. 13. Liu, J. and Zhang, R., Vehicle detection and ranging using two different focal length cameras. J. Sensors, 2020, 1–14, 2020. 14. Zhang, Y.J., Du, F., Wang, J., Ke, L.S., Wang, M., Hu, Y., Zhan, A.Y., A safety collision avoidance algorithm based on comprehensive characteristics. Complexity, 2020, 1–13, 2020.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  21

15. Razzaq, S., Dar, A.R., Shah, M.A., Khattak, H.A., Ahmed, E., El-Sherbeeny, A.M., Rauf, H.T., Multi-factor rear-end collision avoidance in connected autonomous vehicles. Appl. Sci., 12, 3, 1049, 2022. 16. https://www.kaggle.com/datasets/daniildeltsov/traffic-signs-gtsrb-plus162-custom-classes 17. Hu, W., Zhuo, Q., Zhang, C., Li, J., Fast branch convolutional neural network for traffic sign recognition. IEEE Intell. Transp. Syst. Mag., 9, 114–126, 2017. 18. Jung, S., Lee, U., Jung, J., Shim, D.H., Real-time traffic sign recognition system with deep convolutional neural network, in: Proceedings of the 2016 13th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), pp. 31–34, Xi’an, China, 19–22 August 2016. 19. Bach, M., Stumper, D., Dietmayer, K., Deep convolutional traffic light recognition for automated driving, in: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 851–858, IEEE, 2018, November. 20. Cao, J., Song, C., Peng, S., Xiao, F., Song, S., Improved traffic sign detection and recognition algorithm for intelligent vehicles. Sensors, 19, 18, 4021, 2019. 21. Pavel, M., II, Tan, S.Y., Abdullah, A., Vision-based autonomous vehicle systems based on deep learning: A systematic literature review. Appl. Sci., 12, 14, 6831, 2022. 22. Wang, H., Wang, Y., Zhao, X., Wang, G., Huang, H., Zhang, J., Lane detection of curving road for structural highway with straight-curve model on vision. IEEE Trans. Veh. Technol., 68, 6, 5321–5330, 2019. 23. Wang, Y., Dahnoun, N., Achim, A., A novel system for robust lane detection and tracking. Signal Process., 92, 2, 319–334, 2012. 24. Bar Hillel, A., Lerner, R., Levi, D., Raz, G., Recent progress in road and lane detection: A survey. Mach. Vision Appl., 25, 3, 727–745, 2014. 25. Zhou, H. and Wang, H., Vision-based lane detection and tracking for driver assistance systems: A survey, in: 2017 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), pp. 660–665, IEEE, 2017, November. 26. Tang, J., Li, S., Liu, P., A review of lane detection methods based on deep learning. Pattern Recognit., 111, 107623, 2021. 27. Kim, J., Kim, J., Jang, G. J., Lee, M., Fast learning method for convolutional neural networks using extreme learning machine and its application to lane detection. Neural Networks, 87, 109–121, 2017. 28. Wang, W., Lin, H., Wang, J., CNN based lane detection with instance segmentation in edge-cloud computing. J. Cloud Comput., 9, 1, 1–10, 2020. 29. Zou, Q., Jiang, H., Dai, Q., Yue, Y., Chen, L., Wang, Q., Robust lane detection from continuous driving scenes using deep neural networks. IEEE Trans. Veh. Technol., 69, 1, 41–54, 2019. 30. Huu, P.N., Pham Thi, Q., Tong Thi Quynh, P., Proposing lane and obstacle detection algorithm using YOLO to control self-driving cars on advanced networks. Adv. Multimed., 2022, 2022.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

22  Artificial Intelligence for Autonomous Vehicles

31. Angelova, A., Krizhevsky, A., Vanhoucke, V., Pedestrian detection with a large-field-of-view deep network, in: Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 704–711, 26–30 May 2015. 32. Ghosh, S., Amon, P., Hutter, A., Kaup, A., Reliable pedestrian detection using a deep neural network trained on pedestrian counts, in: Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), pp. 685–689, Beijing, China, 17–20 September 2017. 33. Ragesh, N.K. and Rajesh, R., Pedestrian detection in automotive safety: Understanding state-of-the-art. IEEE Access, 7, 47864–47890, 2019. 34. Tomè, D., Monti, F., Baroffio, L., Bondi, L., Tagliasacchi, M., Tubaro, S., Deep convolutional neural networks for pedestrian detection. Signal Process.: Image Commun., 47, 482–489, 2016. 35. Cao, J., Pang, Y., Li, X., Learning multilayer channel features for pedestrian detection. IEEE Trans. Image Process., 26, 7, 3210–3220, 2017. 36. Jiang, X., Pang, Y., Li, X., Pan, J., Speed up deep neural network based pedestrian detection by sharing features across multi-scale models. Neurocomputing, 185, 163–170, 2016. 37. Kuang, P., Ma, T., Li, F., Chen, Z., Real-time pedestrian detection using convolutional neural networks. Int. J. Pattern Recognit. Artif. Intell., 32, 11, 1856014, 2018. 38. Li, Z., Chen, Z., Wu, Q.M., Liu, C., Real-time pedestrian detection with deep supervision in the wild. Signal, Image Video Process., 13, 4, 761–769, 2019. 39. Ayachi, R., Said, Y., Ben Abdelaali, A., Pedestrian detection based on lightweighted separable convolution for advanced driver assistance systems. Neural Process. Lett., 52, 3, 2655–2668, 2020. 40. Li, G., Yang, Y., Qu, X., Deep learning approaches on pedestrian detection in hazy weather. IEEE Trans. Ind. Electron., 67, 10, 8889–8899, 2019. 41. Hou, Y.L., Song, Y., Hao, X., Shen, Y., Qian, M., Chen, H., Multispectral pedestrian detection based on deep convolutional neural networks. Infrared Phys. & Technol., 94, 69–77, 2018. 42. Ding, L., Wang, Y., Laganiere, R., Huang, D., Fu, S., Convolutional neural networks for multispectral pedestrian detection. Signal Process.: Image Commun., 82, 115764, 2020. 43. Badrinarayanan, V., Galasso, F., Cipolla, R., Label propagation in video sequences, in: Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2010. 44. Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R., A database and evaluation methodology for optical flow. Int. J. Comput. Vision (IJCV), 92, 1–31, 2011. 45. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, C.L., Microsoft coco: Common objects in context, in: Proc. of the European Conf. on Computer Vision (ECCV), 2014.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Artificial Intelligence in Autonomous Vehicles—A Survey of Trends  23

46. Milan, A., Schindler, K., Roth, S., Detection- and trajectory-level exclusion in multiple object tracking, in: Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2013. 47. Bertoni, L., Kreiss, S., Alahi, A., Monoloco: Monocular 3D pedestrian localization and uncertainty estimation, in: ICCV, 2019.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

24  Artificial Intelligence for Autonomous Vehicles

Age of Computational AI for Autonomous Vehicles Akash Mohanty1, U. Rahamathunnisa2, K. Sudhakar3 and R. Sathiyaraj4* School of Mechanical Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu, India 2 School of Information Technology and Engineering and Technology, Vellore Institute of Technology, Vellore, India 3 Department of Computer Science and Engineering, Madanapalle Institute of Technology, Madanapalle, Andhra Pradesh, India 4 Department of CSE, GITAM School of Technology, GITAM University, Bengaluru, Karnataka, India 1

Abstract

Autonomous vehicles have made a great impact on research and industrial growth over the past era. The automobile industry is now being revolutionized by self-driving (or driverless) technology owing to enhanced and advanced autonomous vehicles that make use of cutting-edge computational methods from the fields of machine intelligence and artificial intelligence (AI). Autonomous vehicles are now able to assess their surroundings with high accuracy, make sensible choices in real-time environments, and function legitimately without human intervention and technological advancements in the arena of computationally powerful AI algorithms. The development of autonomous vehicles relies heavily on cutting-edge computational technologies. The chapter aims to review the contemporary methods of computational models over time and presents the computational models in the arena of Machine Learning, its subset Deep Learning and Artificial Intelligence. The chapter initially discusses the role of AI, followed by its autonomy levels. The learning algorithms that perform continual learning are addressed along with advances in intelligent vehicles. We disparagingly evaluate the key issues with computational approaches for driverless complex applications. *Corresponding author: [email protected] Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (25–54) © 2024 Scrivener Publishing LLC

25

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

2

Integration of computational technologies is presented in brief, addressing how technologies can empower autonomous vehicles. Classification of technological advancements with future directions was given and concluded. Keywords:  Autonomous vehicles, computational methods, artificial intelligence, machine learning, deep learning, continual learning, classification, intelligent vehicles

2.1 Introduction Autonomous vehicles are vehicles that can operate independently of a human driver, a scientific development that aims to revolutionize transportation and is currently one of the biggest trends. Although the development of autonomous vehicles has gained popularity over the past 20 years, it is true that it first began in the 1990s. Francis Houdina, an electrical engineer from New York, was the first to put the idea of an autonomous vehicle into practice in 1925; however, his vehicle required remote control. A car crashed with the prototype while it was on display to the public in Manhattan, delaying its path for nearly 19 km between Broadway and Fifth Avenue. Chandler, Houdina’s car, was nevertheless created between 1926 and 1930. A Mercedes-Benz van was later transformed into an autonomous vehicle in the 1980s by the German Ernst Dickmanns, who is regarded as the inventor of the modern autonomous vehicle. This vehicle was driven by an integrated computer. The vehicle was able to travel 63 km per hour through streets with no traffic in 1987. Similar maneuvers were made in 1994 with a car that covered more than 1,000 km through congested Paris. A Mercedes-Benz made an autonomous trip from Munich to Copenhagen in 1995. These initiatives were funded by the European Commission as part of Project Eureka, which gave Dickmanns over 800 million euros to conduct this kind of vehicle research. Driverless cars are indeed a major topic in the realm of vehicular technology, which now has rapidly grown in prominence [1]. Over the past few decades, numerous businesses and scholars have become interested in the burgeoning field of autonomous driving, which includes self-driving automobiles, drones, trains, and agricultural equipment in addition to military uses [2]. The first radio-controlled Chandler, American Wonder, which introduced the idea of autonomous driving, was displayed in New York City in the 1920s [3]. After several years, Carnegie Mellon University built the first self-driving car, which was given in 1986; it is merely a van that can drive itself.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

26  Artificial Intelligence for Autonomous Vehicles

In 2005, a different team from Carnegie Mellon University developed a self-driving car that was successful over an 8-mile range and won the DARPA Grand Challenge [3]. In 2010, Google researchers created an autonomous vehicle that has already driven over 140,000 miles between San Francisco and Los Angeles. Self-driving hardware was implemented into the products of many automakers in 2016 [3], comprising MercedesBenz, BMW, Tesla and Uber is used by various transit companies. Several categories of automated vehicles, such as drones or trains, may evolve more rapidly while their habitats have been less influenced by human activities than automated vehicles. The central concept behind self-governing vehicles is to employ artificial intelligence (AI) to intelligently guide the vehicles depending on their surroundings [4]. These disciplines include (a) machine learning-supervised and unsupervised approaches for ­decision-making; (b) to analyze the information from video, deep learning and computer vision technologies were used; and (c) to ensure vehicle safety and control, sensor-based processing techniques were applied [4, 5]. The progress of autonomous vehicles [1] also strongly depends on improvements in distributed hardware devices, namely, big data technologies and network connectivity, and utilizes graphical processing units (GPUs) for quick processing [6, 7].

2.1.1 Autonomous Vehicles Automated vehicle users are enthusiastic about its advent to the public sector. A self-driving vehicle is one that can run unsupervised and without human interference. Modern autonomous vehicles, according to Campbell et al., [9] can assess their immediate surroundings, classify various items they come across, and interpret sensory data to choose the best routes to take while adhering to traffic laws. Remarkable encroachments been formed in generous a relevant reaction to unforeseen circumstances wherever there may be a payback in the drive systems or where an outward medium might not perform as anticipated by underlying prototypes. To successfully execute self-governing navigation in those kinds of situations, it is crucial to integrate a variety of technologies from various fields, notably electronics engineering, mechanical engineering, computer science, control engineering, and electrical engineering, among many others [8]. The first radio-controlled car, known as the “Linriccan Wonder,” marks the beginning of the autonomous vehicle era in 1926. The emphasis has been mostly upon vision-guided structures consuming GPS, LiDAR, computer vision, and radar from the launch of the vision-guided vehicle Mercedes-Benz, a robotic van in the year 1980, and there have been

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  27

considerable developments in technology for autonomous vehicles since then. As a way, steer assist, adaptive cruise control, and lane parking became autonomous technologies seen in modern cars. We will eventually live in a world with fully autonomous vehicles, as per official projections given by auto manufacturers. Traffic calamities are some of the most prevalent reasons of death globally. The world may avert 5 million civilian mortalities and 50 million major harms by 2020 by putting newer creative ideas into practice and improving road safety at all scales—from the small to the global. It is critical, in the opinion of the Commission for Global Road Safety, to put an end to this horrifying and unnecessary rise in traffic accidents and begin year over year declines [9]. According to a paper [8], over 50% of the 3,000 people who pass away each day as a consequence of traffic disasters are not passengers in a car. Deshpande et al. [8] have also stated that if urgent action is not taken, the number of transportation-related injuries is expected to increase to 2.4 million annually and rank as the fifth greatest cause of mortality worldwide. The amount of traffic collisions will consequently drastically decline as a result of an automated system’s greater durability and speedier response time than humans. Autonomous vehicles would also reduce safety gaps and improve traffic flow management, which would increase roadway volume and lessen traffic congestion. Figure 2.1 presents a summary of fatalities due to traffic accidents. With the introduction of autonomous vehicles, parking shortages will be a thing of the past because vehicles may pick up passengers, drop them off, park in any available spot, and then pick them up again. There would be less parking available as a result. Physical road signs will become less important as autonomous vehicles obtain all the information they require via a network. Percentage of road accident deaths 35% 30% 25% 20% 15% 10% 5% 0%

29% 20%

Jeep

Three Wheeler

6% 5% 4%

SUV/Station…

Other…

Bus

8% 8% 8%

Non-…

Car

Truck Lorry

Two Wheeler

12%

Figure 2.1  Summary of fatalities due to traffic accidents.

Type

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

28  Artificial Intelligence for Autonomous Vehicles

There would be less demand for traffic officers. Autonomous vehicles can therefore cut back on government spending on things like traffic enforcement. Along with a decline in automobile theft, there will also be a reduction in the demand for auto insurance. The elimination of unnecessary passengers altogether will allow for the establishment of efficient systems for the transportation of goods and shared vehicles. Not everyone is appropriate. Therefore, autonomous vehicles relieve the need to drive and navigate. Additionally, commuting times will be shortened because autonomous vehicles can move more quickly with little danger of mistake. The ride will be more comfortable for the car’s passengers than it would be in a non-autonomous vehicle. While autonomous vehicles have many advantages, there are also some drawbacks. Although the idea has been disproven, it is still believed that the introduction of autonomous vehicles would result in a decline in occupations involving driving. Another significant difficulty is when drivers lose control of their vehicles because of inexperience, for example. Many individuals enjoy driving, so giving up control of their vehicles would be tough for them. Additionally challenging is the interaction of human-driven and autonomous cars within the same path. Another concern with self-driving cars is who should be held accountable for damage: the automaker, the car’s users or proprietor, or the government. Determining a legal mechanism and government regulations for automated automobiles is thus a crucial delinquent. Another significant issue is software reliability. Additionally, there is a chance that the computer or communication system in a car could be hacked. There is a chance that terrorist and criminal activity may increase. For instance, terrorist groups and criminals might pack cars with explosives. They might also be utilized in numerous other crimes and as getaway cars. Autonomous vehicles thus have benefits and drawbacks. This article explores the history of autonomous vehicles in a sequential fashion, starting with historical precedents and moving on to modern developments and projections for the future. The development of autonomous vehicle technology has advanced significantly during the last couple of decades. The vehicle’s adaptive safe and energy-saving capabilities are significantly enhanced, which establishes it as a distinct area of study for automotive engineering, a trend for the automotive industry, and a force for economic progress [10]. The integration of so many advanced and cutting-edge technologies into autonomous driving presents both obstacles and opportunities in plenty. Critical questions include what the autonomous vehicle’s purpose is, what the main technical obstacles are, and how to overcome them so that the objective may be achieved. The computer science of science is acknowledged as the greatest method for keeping track of research activities and trends [11]

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  29

and may be clustered to learn about the hottest subjects in a certain field [12]. It may offer useful details regarding the state of autonomous driving research. This study analyzes the research hot spots that have changed over the last 20 years, pointing out the path for additional research and tracing the evolution of autonomous driving technology. It also discusses recent accomplishments based on bibliometric analysis and literature evaluation.

2.1.2 AI in Autonomous Vehicles John McCarthy, a computer scientist, coined the word “artificial intelligence” in the year 1955. Artificial intelligence is the capability of a computer program or machine to reason, learn, and make decisions. In most cases, the phrase describes a machine that mimics human intelligence. Through AI, we can instruct computers and other gadgets to carry out tasks that a human being would do. Such algorithms and devices were pumped enormous volumes of data, which are then evaluated and processed in order to think logically and perform human actions. The primary components of driverless technology were given in Figure 2.2. AI is being used in automated driving and diagnostic imaging tools with the goal of preserving life, and automating repetitious human labor is simply the idea of an AI iceberg.

2.1.2.1 Functioning of AI in Autonomous Vehicles Although AI has recently gained popularity, how will it work in driverless vehicles? Let us think about how a person may drive a car when using his LIDAR UNIT CAMERAS RADAR SENSORS

ADDITIONAL LIDAR UNITS

Figure 2.2  Primary components in driverless technology.

MAIN COMPUTER IN TRUNK

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

30  Artificial Intelligence for Autonomous Vehicles

or her sight and hearing to maintain an eye on both the road and other cars. We swiftly decide whether it should stop at a red light or wait for someone to cross the road by using our memory. Decades of driving practice teach us to always look for the little things like a greater speed bump or a more direct route to work. While we are developing autonomous vehicles, we want them to operate similarly to human drivers. Because of this, it is essential to enable these robots access to the sensorial, intellectual, and executive processes that individuals hold when driving a car. The automobile sector has undergone steady change during the last few years in order to achieve this. According to Gartner, multiple V2X (vehicle-to-everything) techniques will connect 250 million vehicles to each other and the associated functions by 2020. Since this quantity of information entered into telematics systems or in-vehicle infotainment (IVI) units increases, vehicles would be able to not only capture and broadcast underlying system status and location information but also alter in their environment in real-time basis. By the addition of sensors, cameras, and communication systems, autonomous cars will be capable to produce massive volumes of facts that, combined via AI, will allow them to see, hear, think, and act in the same ways as human drivers.

2.2 Autonomy With no human input, an autonomous vehicle is one that can perceive its environment and operate on its own. A human rider is not required to control the vehicle at any time. Figure 2.3 portrays a compacted architecture block diagram for autonomous vehicles.

2.2.1 Autonomy Phases The term “autonomous” have multiple definitions, and this vagueness led specialists in the field of autonomous cars to categorize autonomy into Perception

• Sensor Modules • Recognition Modules

Decision Making and Planning

• Decision Module • Trajectory Module • Planning Module • Interface Module

Control

• Control Module • Microprocessor unit

Figure 2.3  Schematic block diagram for autonomous vehicles.

Undercarriage

• Pedal

acceleration

• Brake • Steering wheel • Gear shifting

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  31

different levels, with the following stages denoting the degree of an automated system control and the involvement of drivers in a vehicle [13]. Autonomous levels are summarized in Figure 2.4. Level 0: Only optional and necessary hazardous alarms are offered as autonomous and zero-­ control driving capabilities at this level. Level 1: Drivers still have complete control of the vehicle, but automated controlling systems share that power with them. These cars frequently come equipped with cutting-edge driving aids. Level 2: The automated control system provides the opportunity to take complete control of the car, but the driver is still entirely in charge of operating it. Level 3: The autonomous car can drive completely independently of the passengers, but drivers must take over if the alarm systems call for any physical intervention. Level 4: With no driver’s engagement or observation, the automated mechanism takes complete governance of the vehicle, except in unforeseen events where the driver regains control. Level 5: No consideration is given to human interference in this final level. Artificial intelligence, which also has fundamentally changed the automotive sector, has hastened the advancement of Level 4 and Level 5 autonomous vehicles. An autonomous vehicle (AV) is a means of transportation that is proficient in handling a range of conditions and drive itself throughout public highways with minimal to no direct human interaction [14]. Intelligent and connected vehicles (ICVs) are a broader and looser LEVEL 0 – Manual Driving

LEVEL 1 – Advanced Driver Assistance System (ADAS)

LEVEL 2 – ADAS that can steer and either brake or accelerate

LEVEL 3 – Automated Driving System (ADS)

LEVEL 4 – ADS can perform all driving tasks and monitor the driving environment

LEVEL 5 – The ADS of the vehicle handles all of the driving in any scenario, serving as a virtual driver

Figure 2.4  Autonomous levels.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

32  Artificial Intelligence for Autonomous Vehicles

Motion

Interaction

Localisation

Mapping Vehicle Navigation

Figure 2.5  Fundamental navigation functions.

concept, and their evolution can be broken down into four stages: high/ fully automated driving, connected assistance, cooperative automation, and advanced driver assistance systems (ADAS) [15]. According to this standard, ICV’s AV stage is its conclusion. Figure 2.5 represents the components required for navigation functionality. The accurate position estimations, the accuracy of the digital plots, then the map-matching techniques all play a part in determining the position of a means of transportation for autonomous navigation. Mapping: Establishing the spatiotemporal link between the vehicle and its surroundings is necessary for vehicle guiding. The same can be viewed by means of modeling and comprehending the realm from the perspective of the chauffeur or computer control. Motion: This function can be described as a set of activities that allows the platform to move safely and effectively. These activities include path arrangement to get to the target, collision avoidance, and vehicle control. High levels of interaction exist between the localization, mapping, and actuation tasks. At the vehicle levels, they adhere to a complicated reliant system that is developing in a wildly unpredictable environment.

2.2.2 Learning Methodologies for Incessant Learning in Real-Life Autonomy Systems 2.2.2.1 Supervised Learning It is the technique of learning using training information that has been labeled and has comprehensive information of the system’s input and output [16]. It is possible to map facts to labels throughout the whole series

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  33

using supervised incessant learning, which permits learning from sequential streaming data. The complexity of the learning method was decreased by the usage of continuous learning in supervised learning.

2.2.2.2 Unsupervised Learning Algorithms that do not need labels include those used in unsupervised learning. The most typical application is the training of generative models that replicate the distribution of input data. When the distributional shift happens in the continuous learning scenario, the generative model is modified, causing in the final yield being engendered from the complete input dissemination. The generative replay continuous learning approach has been seen to make the most frequent use of these models. In autonomous systems, unsupervised continuous learning may be helpful for building progressively more potent illustrations over interval, which can subsequently be fine-tuned via an outward feedback indication from the environments. Unsupervised learning’s major objective is to create adequate self-supervised learning signals that can serve as a suitable substitute in order to build adaptable and robust representations.

2.2.2.3 Reinforcement Learning This trains an agent to carry out a series of actions in a particular environment by using a prize function as a tag. It is possible to think of reinforcement learning in complicated environments as a continuous learning situation because they do not afford users admittance to all data at formerly. Reinforcement learning utilizes numerous crucial continuous learning model elements, including the capacity to learn several agents concurrently and the usage of a replay memory, in order to assume data distribution. Additionally, the TRPO algorithm, a well-liked stable reinforcement learning technique, enhances learning in a manner comparable to the continuous learning strategy because of the Fisher Matrix used in it. The strong restrictions on reinforcement learning algorithms are the problems of continuous learning. Therefore, enhancing continuous learning will also enhance the effectiveness of reinforcement learning. Unpretentious electromechanical systems gave way toward sophisticated computer-controlled interacted electromechanical systems as ground transportation systems developed. Initially intended for recreation, automobiles are now a crucial component of our daily life. Highly practical modes of transportation. Road infrastructure has been created as a result over the Vehicle costs have decreased throughout the years and as a result of mass production and the general public’s ability to afford them. However, these modifications have

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

34  Artificial Intelligence for Autonomous Vehicles

brought up a variety of issues, including fatal or serious injuries, traffic accidents, gridlock, pollution, and aggressive drivers.

2.2.3 Advancements in Intelligent Vehicles The way that society views transportation networks has altered during the last few years. Automobiles were formerly seen as a source of social status and convenience that gave industry a great deal of flexibility, despite the high accident rate, environmental constraints, high fuel costs, etc. Today’s modern transport structures are a basis of significant apprehension. To solve the aforementioned concerns, governments, businesses, and society at large are shifting to what are known as sustainable forms of transportation. Automobile manufacturers are forced by societal expectations for safety, pollution reduction, and networked mobility to continuously adapt their product lines and explore new ways to innovate their offers. Information and communications technology (ICT) advancements created prospects for the addition of innovative features to existing vehicles. These vehicles already include a variety of proprioceptive sensors and are gradually adding exteroceptive sensors, which are cameras, radars, and other devices. Additionally, several propulsion methods are employed, and the majority of automobiles utilize their identifiable internal computer networks where diverse nodes handle dissimilar functions. Vehicles can also link to one another and the infrastructure that provides them with constant communication by forming networks. As a result, the car is gradually evolving from a single, mostly mechanical entity into a network of cutting-edge intelligent platforms [17]. Genetic algorithm-based approach for traffic congestion reduction and avoidance with smart transportation focusing on intelligent vehicles is presented [18]. Region identification, extraction of features, and classifications are the three broad classifications of object recognition [19]. The process of employing sensors to find pedestrians in or near an AV’s route is known as pedestrian detection. Segmentation, Extraction of Features, Segment Classification, and Track Classification are its four constituent parts [20]. A related study looked at how people assign blame for a good or bad event underneath the Expectancy Violations Theory to humans or artificial intelligence agents [21]. The various advanced predictive models were presented [22]. This can be applied for smart traffic management and sustainable environment. An agent-based system for traffic light monitoring and controlling is illustrated [23]. Modern AI algorithms are used by autonomous cars to localize themselves in both known and unknown settings [24]. Regarding short-term prediction, a pattern-based technique utilizing local weighted learning is suggested [25]. A summary

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  35

of the most recent advancements in autonomous vehicle architecture is made in the research work [26]. Despite the fact that some individuals may seem to be suspicious about the practical application of autonomous driving (AD) as a kind of replacement for current cars, the abundance of research and trials being carried out indicates the contrary. Automated vehicle researchers and organizations are developing effective tools and strategies. An extensive investigation of the design methodologies for smarter frameworks and tools for AI and IoT-based automated driving was performed in this study [27]. Additionally, its implications encompass the functionality of automated electric vehicles. In this article, several nations’ and organizations’ actual driverless truck, car, bus, shuttle, rover, helicopter, and subterranean vehicle implementations are explored. While academic robotics research shifts from using internal mobile systems by means of full-scale means of transportation with high stages of automation, the automobile industry is undergoing a shift at the same time. The motor systems of modern vehicles have changed, making them easier to handle by computers. They also now have various sensors for navigational purposes, and they have nodes in vast communication networks. The most recent advancements in intelligent vehicles as they develop autonomous navigational abilities were discussed. A typical intuition of a self-driving car is given in Figure 2.6. The driving forces behind this ongoing modern vehicle revolution are discussed in the context of application, security, and external concerns like pollution and fossil fuel restrictions.

Obstacle Road Edge Stone

Laser

Environment Perception Road Sign

Radar

Vision

Figure 2.6  An insight of a self-driving car.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

36  Artificial Intelligence for Autonomous Vehicles

Navigation: In order to concentrate on the device intellect and ­ decision-making methodologies that exist being established and implemented to renovate contemporary automobiles into modular components through cutting-edge advanced features, the province analysis is organized in terms of the vehicle’s infotainment functionalities. It discourses concerns with driving requirements, interaction, and autonomous challenges. By structuring the onboard cognition of the vehicles as a navigational challenge, it establishes the functional criteria for automobiles to demonstrate autonomous navigation ability. Three main views have been used to categorize recent developments in the related vehicle technologies: (1) driver-­centric systems aim to improve drivers’ situational awareness by offering a number of driving assisting technologies that let the driver make decisions; (2) network-centric systems emphasize the practice of wireless communications to enable sharing facts among infrastructure and cars, enhancing awareness of the operators and machineries beyond the capabilities of stand-alone vehicle systems; (3) vehicle-centric systems focus on the goal of making cars totally autonomous, with the driver out of the control loop. We will describe these various viewpoints and discuss recent advancements in research and business. A viewpoint on potential advances in the future and in what way these techniques might be implemented while pleasing into interpretation societal, legitimate, and financial limits were presented. Autonomous vehicles are among the primary uses of AI. Autonomous vehicles were using a variety of sensors to assist them in better understanding their environment and charting their routes, including certain camera systems, radar systems, and LiDAR. Huge volumes of data are produced by these sensors. To extract meaning from the information produced by these sensors, AVs require processing rates that are comparable to those of supercomputers. AV system developers heavily rely on AI to train and assess their self-driving techniques. Companies are turning to deep and machine learning, in particular, to manage the enormous amount of data quickly.

2.2.3.1 Integration of Technologies Despite the frequent interchangeability of the concepts artificial intelligence, machine learning, and deep learning, these concepts are not necessarily related. AI, as it is commonly known, is a branch of computer science that examines all elements of making computers intelligent. Therefore, when a device accomplishes activities in accordance with appropriate rules that address a specific issue, this behavior can be regarded as intelligent

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  37

behavior or AI. Deep learning and machine learning approaches can be used to develop AI. The study of structured information then the methods a device utilizes to carry out a certain activity without precise instructions are referred to as machine learning. AI is used in machine learning, which enables computers to get better by the mistakes they make. Deep learning is a component of machine learning; otherwise, the progression of machine learning. Deep learning draws its inspiration from how the mammalian brain deals with information. It uses sophisticated neural networks that continuously learn from and assess their input data to extract more precise information. Unsupervised learning uses less structured training sources, while supervised learning uses labeled training data. Apparently, supervised as well as unsupervised deep learning is acceptable. Businesses creating AV technology primarily rely on deep learning or machine learning or both. Machine learning and deep learning differ greatly in that machine learning needs that features be explicitly labeled with less flexible rule sets, while in unsupervised tasks, deep learning might inevitably determine the feature to be utilized for categorization. In contrast to machine learning, deep learning necessitates a large quantity of computing power with training facts in order to create outcomes that are more precise. AVs are aimed to decrease vehicle accidents, improve traffic flow and maneuverability, use less fuel, eliminate the need for driving, and ease commercial operations and transit [28, 29]. Despite the enormous potential benefits, there are numerous unresolved technological, social, ethical, legal, and regulatory challenges [30–32]. Deep learning has assisted businesses in accelerating AV development efforts in recent years. These businesses are depending more and more on deep neural networks (DNNs) to further effectively interpret sensor information. DNNs permit AVs to acquire how to navigate the world on their specific by means of sensor information rather than taking to be physically programmed with a set of instructions like “halt if you perceive red.” Since these procedures were modeled after the human brain, it is likely that they learn via experience. A deep learning expert named NVIDIA claims that uncertainty a DNN is exposed to photographs of a halt sign in various settings, it can acquire to recognize halt signs on its peculiar. To provide safe autonomous driving, however, businesses creating AVs must create a whole set of DNNs, each focused on a different task. The number of DNNs needed for autonomous driving has no predetermined upper bound and is actually increasing as new capabilities are developed. The indications produced by each distinct DNN requisite are managed in real time by powerful computer platforms in order to actually operate the car.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

38  Artificial Intelligence for Autonomous Vehicles

2.2.3.2 Earlier Application of AI in Automated Driving In the Second Defense Advanced Research Projects Agency (DARPA) Intelligent Vehicle Challenge, the autonomous robotic car named “Stanley” from the Stanford University Racing Team in 2005 marks the beginning of the use of AI for autonomous driving. Multiple sensors and specialized software, including machine learning techniques, supported Stanley’s ability to recognize impediments and avoid them while maintaining its course. Thurn then administered Google’s “Self-Driving Car Project,” which in 2016 evolved into Waymo. Waymo greatly utilized AI to bring completely autonomous driving to the masses. In partnership with the Google Brain team, the company’s engineers developed a pedestrian detection approach that integrates DNN. Employing a deep learning model, the engineers have been able to reduce the error rate for walker identification by half. Waymo’s CTO and vice president of engineering, Dmitri Dolgov, deliberated how AI as well as machine learning abetted the business in emerging an AV scheme in a blog post on Medium last year.

2.3 Classification of Technological Advances in Vehicle Technology Recent developments in artificial intelligence and deep learning have made autonomous vehicles more sophisticated. Most modern parts of self-­ driving cars employ current AI algorithms [33]. Autonomous vehicles are sophisticated systems for transporting people or goods. Similar to establishing AI-powered automated driving on public highways, doing so on public roads comes with a number of difficulties. Controlling the movable platform as it moves safely from its starting point to its goal while navigating an infrastructure is referred to as vehicle navigation, constructed for human driving, the most recent developments in sensing and computing. If a mobile platform were to make decisions today, it could traverse the majority of road networks. The main challenge is utilizing the infrastructure together with additional entities, such as several mobile platforms with varied power sources, at-risk drivers, etc. Despite having laws and driving norms, their behavior is unpredictable. Despite enforcement measures, driver errors still happen and result in a lot of traffic accidents. The greatest obstacle to the adoption of driverless automobiles was indeed identifying solutions that enable utilization of the identical workspace with other parties. The development of autonomous vehicles is influenced by network-centric, vehicle-centric, and driver-centric advances, which is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  39

Steering wheel angle as recorded

Instructed steering command Adjust for shift and rotation

Network-based steering instruction Camera

Random shift and rotation

CNN

Error Back Propagation Weight adjustment

Figure 2.7  Training the model using deep learning techniques.

why it is important to examine vehicle navigation technology used in passenger vehicles. The illustration of training the model with deep learning is shown in Figure 2.7. Intelligent vehicles are now being developed from a variety of angles by the transportation sector, academic institutions, and government R&D facilities. For instance, the automotive industry is implementing vehicle onboard technologies to improve driver convenience and safety. That is to say, some activities are performed by the vehicle’s assigned computer control systems that use sensors. This tactic is defined by two key concerns, and responsibility. The way people view cars has altered as a result of the late 1970s, are no longer considered status symbols or arouse emotion; instead, today’s convenience. The primary determinants of a vehicle’s acquisition are cost and usefulness. Exteroceptive sensors, including laser scanners as well as infrared cameras, have been shown to have the ability to identify obstacles in regular traffic, but their application is currently restricted to high-end vehicles owing to the associated expenses. The intention is that the costs are too high for mid-priced cars, where radars and video cameras are still chosen despite their drawbacks. From this vantage point, improvements in modern off-the-rack. The first definition of an intelligent vehicle is one that puts the driver first. Driver-centric methods make it possible to comprehend the circumstances that self-governing vehicles will face when they are utilized in actual traffic conditions. This proves that the methods used by human-controlled structures to improve protection are also applicable to autonomous vehicles. Additionally, it enables the transfer of technologies from advanced experimental platforms to mass market vehicles and vice versa, enhancing

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

40  Artificial Intelligence for Autonomous Vehicles

the engineering expertise of the vehicle OEMs. Autonomous vehicles’ longitudinal and lateral control, for instance, can be affected by the interface, actuation, and safety mechanisms provided by vehicles equipped with automatic parking assistance systems, which consume the organization to undertake sensor-based computer-manageable motions. In network-centric, vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) data distribution is made possible by the inclusion of communication technologies in passenger vehicles. Communications between V2V and vehicle-to-roadside unit (V2R) are depicted in Figure 2.8. This has steered to the development of various vehicle types known as cooperative vehicles, whose operation depends on their amalgamation onto a transportation network that supports wireless V2V and V2I connectivity. These kinds of vehicles are categorized as network-centric from a functional perspective. All players in road networks can now share information—thanks to network-centric technologies. Data can then be gathered and analyzed by the network before being disseminated to other networks. This is a crucial contribution for driverless vehicles because it implies that future autonomous vehicles will not necessarily need to be stand-alone systems. They must be mobile nodes that cooperate by means of further mobile nodules to move. Vehicle-Centric: A diverse strategy involves automating the car navigation features as much as is practical. The driver’s involvement is diminished until it is prolonged

Figure 2.8  Communications between V2V and V2R.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  41

to a part of the vehicle governor loop, at which point the vehicle is put under computer control and becomes autonomous. The system design is concentrated on the vehicle utilities since the architectures duplicate the functionality required to drive an autonomous car. These vehicle designs could be categorized as vehicle-centric. Vehicle-centric cars are autonomous vehicle systems that take into account the most notable experimental platforms that have been created so far and the related technology. A summary of the technology presence created for self-governing vehicles is given by this complete panorama.

2.4 Vehicle Architecture Adaptation Since the standpoint of automobile controllability, if the driver has complete control, there will not be a navigation system in the car. Today’s applications tend to be driver-centric, focusing initially on providing information before allowing the driver control the car. There are not many applications that need direct machine management, but those that do are being added one at a time, such as Lane Keeping Support. Computers are rapidly taking over the controller of the automobile from the driver. From a diverse angle, computer-driven vehicles are becoming more autonomous as they progress from basic tasks. Recent vehicle OEM prototypes have shown that it is possible to use wireless communications for safety-related energetic functions, since they allow information to be shared and so increase driver awareness. These technologies ought to be usable in a system that is centered around automobiles, since communication networks help create intelligent environments that make it easier to use autonomous vehicles. Network-centric designs are those that result from the utilization of communication systems. In order to completely commercialize the technology, autonomous vehicle dealings have capitalized a momentous amount of currency in its research. There are several issues that hinder this goal. These difficulties comprise legal, nonlegal, and technical complications [34]. More than ever, we are very close to achieving the long-desired goal of vehicle automation. Additionally, major automakers are spending billions of dollars to create automated vehicles. Among other advantages, this innovative technique has the prospective to increase passenger safety, reduce traffic congestion, streamline traffic, save fuel, reduce pollution, and improve travel experiences [35]. Similarly, AVs can examine other cutting-edge technologies like blockchain and quantum [36]. In order to communicate information, autonomous cars use wireless sensor networks [37]. The architecture for

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

42  Artificial Intelligence for Autonomous Vehicles

effective knowledge distillation using transformers is proposed in Liu et al. [38] for the semantic segmentation of driving incidents on roads. For example, segmentation is proposed to use a convolutional neural networks technologies is a leading multiscale attention [39]. A strong cooperative positioning (RCP) [40] approach that adds an ultrawide band to GPS has indeed been presented to get accurate position (UWB). To estimate grid maps, a multitask recurrent neural network is suggested [41]. Sematic data, occupancy estimates, velocity forecasts, and drivable area are all provided via grid maps. The motion control of autonomous US vehicles now places a significant emphasis on deep learning-based techniques [41, 42]. But integrated optimization of sensory, decision-making, and motion control can also be achieved by utilizing the power of DNNs [43]. In the driver-centric technology category, the driver still maintains overall control of the vehicle. Various features are created to improve the driver’s situational awareness and safety via enhancing environmental sensing or preserving stability and controllability of the automobile. Every perception function aims to provide information to the driver regarding what takes place in its immediate surroundings. The merging of is what is highlighted, data from proprioceptive and exteroceptive sensors that enable model construction of the spatiotemporal interactions among the vehicle and its immediate surroundings and its current condition. Then, situational information is inferred using this model. Decisions are constantly made in all facets of driving. As previously stated, mistakes made by

Figure 2.9  Automated control system.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  43

the driver that result in poor decisions are the main cause of accidents. The focus is on data collection and association to promote awareness and give the best options for improving the driver’s situational awareness and, as a result, aid in making decisions. This typically entails observing what happens within the near area in front of the car or to gauge how it will react. A common illustration is the detection of pedestrians using vision systems or laser scanners either to alert drivers or lower the vehicle’s rapidity. Here stays a feature that is now being incorporated into new car generations. Due to the wide range of situations that could occur, the design of the sensors and their capabilities, and deployment costs, the perception systems face the greatest challenges. There are numerous programs available to help with driving. Many of these can be found on current automobiles or are built into upcoming products. Others have been demonstrated and are being tested in real-world settings. Today, drivers being informed is preferred. The justification is tied to liability; specifically, by observing the driver in the circlet during vehicle control, the driver is held accountable for any vehicle maneuvers. The automated communication and control of the vehicle to the road sign indications were illustrated in Figure 2.9. Its operation is based on the integration of a priori data from cardinal navigational maps and the combined usage of onboard sensors. The three groups of functions include enhanced perception, longitudinal and lateral control, and route guidance (navigation). Applications that employ digital road maps are included in the first column, applications that have some degree of control over how the car moves are in the second column, and applications that improve the driver’s perception are in the third column. A vehicle might be operated tenuously or under the control of a principal server, as is the case with the usage of automated guided vehicles (AGVs), and the former role would be ancillary driving—thanks to driver-­centric architectures. Tele-operation of vehicles: It suggests that the safety of a driver operating a vehicle from a distance can be maintained while exchanging enough information. The military prefers this application domain for managing land vehicles in hazardous conditions. For this aim, video cameras are employed to examine the environment. Laser scanner returns are utilized to give the impression of depth, and they are mixed with video images to create depth-colored images. This kind of technology needs a way to move the vehicle controller near to the vehicle and its surroundings. The defense vehicles use driving strategies where the driver solely interacts with the external environment and the vehicle’s instructions via electrooptic mechanisms, i.e., a mix of cameras and other perceptive sensors are utilized to show the driver the surroundings. The driver is using his or her

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

44  Artificial Intelligence for Autonomous Vehicles

eyes instead of evaluating and engaging with the outside world via monitors that show live camera system images and other sensor information. Thus, it is feasible to identify the In the event of a military car, the driver will be positioned wherever inside the placed in a safe area, resulting in a substantial decrease in risk and expense. This gets around the challenges of needing to find the driver on a helpful surveillance point while remaining safe and capable to respond to the vehicle directions. The ability to create accurate 3D representations of the area around a vehicle is made possible by the availability of sophisticated perception technologies, such as scanning lasers then video cameras, along with GPS besides Inertial Measurement units. Consequently, it is conceivable to give the driver a comprehensive panoramic interpretation of the area around the vehicle, aiding situational cognizance and, ultimately, vehicle operation in difficult circumstances. Wireless connections between vehicles (V2V) and between vehicles and infrastructure (V2I) are becoming more prevalent because developments in computer and communications technology are deployed in network-centric. V2X refers to both varieties of communication lines. These connections are altering how vehicles are created. It permits sharing and information gathering that results in a wider awareness horizon for the driver outside what the onboard car sensors will detect. From a safety standpoint, it permits the ability of drivers or machines to anticipate and respond to interactions with other mobile entities as well as to modifications in the environment. The concept of network-centric cars is based on three essential components: a digital map, wireless communications, and localization. Information on the spots and accelerations of surrounding vehicles and their extrapolation onto a digital map that reflects the road network and associated attributes make it possible to determine the spatiotemporal scenario of certain other cars with regard to the subject vehicle (SV) as well as their communication with the transport network. This information is then utilized to determine whether or not risk scenarios exist, allowing a ­computer-controlled vehicle or the operator to respond far earlier than what the onboard sensors of the vehicle would otherwise be able to do due to the physical constraints on them. If the SV only has a forward-looking sensor when it approaches a junction, certainly a radar, camera, or scanning laser, the driver or computer controller may only have access to partial information due to the sensors’ constrained field of view (FOV). Here remains no knowledge, for instance, that the scenario is further problematic because the SV will annoy a road juncture or that the car is approaching quickly and might not be able to stop in time as the traffic signal on the road connected to its trajectory suggests. In this situation, it is possible to

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  45

boost awareness by putting the information from the automobile’s onboard sensor into setting by combining facts from the digital plot. Vehicle OEMs and governments are currently very interested in the global deployment of cooperative vehicles. According to statistics, despite the fact that accident reduction is slowing down despite improvements in road safety. Studies suggest that the usage of cooperative vehicle safety is the next stage in lowering safety. It claims that deploying passive and active safety systems alone will not be enough to continue reducing accident rates; rather, safety V2X applications must be introduced in order to reduce accidents in ways that current methods cannot. Cost is one of the reasons for this; using maps and onboard communication equipment in a vehicle will be far less expensive than using sophisticated sensors like radar and cameras. When all of the structural nodes are linked together, distributed computations could be run on the network to control how all of the mobility nodes move about. Autonomous vehicles develop ad hoc networks as they move over road networks in network-centric systems. Modern automobiles include electromechanical motion control systems that turn them into controllable nodes. Autonomous maneuvers could be carried out by integrating their onboard intellect and the network’s insight. The context collection and handling tasks are dispersed through sensor networks and wireless transmission lines in order to lower the amount of onboard intellectual ability and perception systems needed onboard vehicles to work safely beneath partial driving supervision or control software. A centralized model like that one might gather information from a variety of sources, including moving cars, traffic lights, surveillance equipment, people using smartphone, and road signs. These data are then utilized to construct an extensive model of the environment on top of an appropriate prediction of the road network. Information on all the entities may then be derived by downloading strategic decisions (such as the prioritizing of a signalized intersections) as orders to the appropriate vehicles. When used, autonomous vehicles will undoubtedly make use of some of the technology. When wireless technology and increasing levels of automation are developed for cooperative cars, communications turn into a common good. Economically speaking, it might be more appealing to put communication equipment in place of the onboard sensors. This also can be applied to assist ambulance vehicles. The emergency vehicles shall be operated with sensors and can assist in navigating with their location. The location of ambulance vehicles is exemplified in Figure 2.10. In case the lane is highly congested, then the vehicle can be suggested to take an alternate route to save the life of humans. The alternate

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

46  Artificial Intelligence for Autonomous Vehicles

Figure 2.10  Identification of ambulance locations.

routes connecting ambulances are given in Figure 2.11. This can further be connected to driverless technology for enhanced safety and reliability. Vehicles move autonomously when control is placed in the onboard processing systems; hence, vehicle-centric developments result. Every perception function operates in a way that a computer must be able to comprehend. In other words, as the vehicle travels to its goal, onboard intelligence analyzes the vehicle’s environment and selects the best maneuver. This entire ecosystem design addresses a group of automated systems having capacities for perception, understanding, decision-making, and actuating. It is no longer an issue, even though it was primarily designed for defense or space purposes. Department of Defense (DOD) and Department of Transportation (DOT) researchers came together to discuss potential automated vehicle research at a joint conference after the association between these applications and those in the auto sector was earlier found (IVVT 2011). Whereas the latter was devoted to improving the efficiency, security, and safety of passenger vehicles, the initial effort focused on using robotics techniques for military applications. The automated motion of transportation applications stretches back to the earliest robots. A mobile robot that the volume of a card table called the Stanford Cart that was equipped with a television camera and transmitters is one of the first instances. Using only

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  47

Figure 2.11  Alternate routes to reach the target.

the visuals projected by the onboard TV system, a computer was trained to steer the cart through crowded interior and outside areas. The cart moved in bursts of 1 m, with breaks of 10–15 min for image analysis and route planning. Observing the occurrence of impediments along the targeted path is necessary for determining whether or not the vehicle can navigate the nearby surroundings. The main focus is on spotting obstructions and comprehending how they relate to the SV. The ease of getting sensors to researchers and businesses is probably what slowed down growth as automated mobile platforms moved from lab settings to well-organized workplaces like warehouses to off-road navigating and finally to urban settings. These advancements were linked to the accessibility of many sensors. The main deciding factor for vehicle manufacturers is cost, and many sensors that are being incorporated as a component of driver assistance systems are nearly identical to those employed in vehicle-centric systems. Since the early 1980s, robotic platforms with various perception systems have been used, first with considerable use of ultrasonic sensors and the associated collection and finding accuracy restrictions. Infrared sensors remain a typical component in the majority of newly made passenger cars and were

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

48  Artificial Intelligence for Autonomous Vehicles

used not merely to help with movements in confined spaces but also to steer autonomous cars when performing parking maneuvers, such as the INRIA completely automatic parking system. The usage of cinematic cameras in the past was made, then this necessitated specialized computational hardware, which prevented their extensive usage, with examples. Stereo vision has been used to retrieve detailed information for robotic automation applications in the past. Video cameras are employed in detecting systems today. A supervisory controller system is used to oversee various systems and make sure they operate safely or fail-safe. It contains procedures and monitoring mechanisms that are activated when one or more of the systems perform poorly or malfunction. The majority of functions are based on the vehicle’s condition, specifically its position. During the planning stage, these data are used to increase the clarity of the machine situation as it relates to the vehicle global model. The estimation of the vehicle status is what this is known as and includes tools for estimating the location of the vehicle comparative to the world and its dynamics both in a local way and with relation to a frame.

2.5 Future Directions of Autonomous Driving According to their assessment, driverless cars will either progress in backward segments or combine with higher-level systems later on. This claim is supported by TRIZ theory as well as hot spot research issues and recent triumphs in three important systems. One advantage of the resulting intricacy is the capacity to distribute the functional load among some of the constituent parts, as the principle of nonuniform development of subsystems says that the least stable subsystems are required to accelerate the adaptation changes [44]. Adapt to intelligent transportation systems (ITSs): ITSs can play a significant role in the future, since it strives to boost energy efficiency, lessen threats to the environment associated with transportation, improve vehicle safety, and increase transportation efficiency. A driverless car has an advantage of adjusting to ITS requirements, since it can communicate with other cars and drive without error or randomness. Additionally, the system can receive components of the functioning from each vehicle. The need for each vehicle’s ambient perception system and decision-making system can be reduced, for instance, if information regarding the distance between two cars can be acquired from both of them, calculated once, and sent to both at the same time. Sync up with

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  49

sophisticated eco-friendly vehicles (i-EFV). By enhancing traffic flow, ITS can address environmental and energy issues, while the idea of an i-EFV encourages more thought about the pairing with new-energy vehicles. In addition to being safer and using less energy, autonomous vehicles make it simpler to integrate the energy and transportation systems. As a result, the flow of energy channel is shortened, information loss is decreased, and they are able to work together efficiently. Advancement in sensor technology: Due in significant part to the inadequate environmental awareness system, the majority of modern autonomous vehicles can only operate in favorable weather and on structured roads. Studies on compact, less affordable, stable LiDARs or even other suitable alternatives, as indicated above, and investigation on sensor fusion systems with quick adequate operational speeds and reasonable hardware demands are the two main ways to improve it. Specifically, meeting the competing objectives of real time and being inexpensive. Three among the most impressive AI methodological approaches in use are deep learning, (deep) inverse reinforcement learning, and (deep) reinforcement learning. These techniques have proven to be effective for decision-making and object recognition, and they have developed automated driving technology in several fascinating ways. However, there may be a few challenges. For instance, the necessity for a large quantity of training samples, a lot of computer resources, or a lack of theory to invert its progression or feedback mistake. The qualities of a more sophisticated method include creating training data independently, generalizing performance to unevidenced states, and satisfying engineering requirements.

2.6 Conclusion This chapter has provided a thorough analysis of the need for autonomous vehicles and the current state underlying computing models, which can help with the advancement of driverless technology. Modern artificial intelligence and machine learning technologies have been emphasized in the discussions. There is also an interpretation of how driverless vehicles impacted computational models. The varying levels of autonomy and ­cutting-edge technologies in intelligent vehicles were thoroughly discussed. The chapter represents insight into the importance of continuous learning in training the model and how a contemporary automated system can benefit from it. The major concerns with computational methods for sophisticated driverless applications were also discussed, where researchers

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

50  Artificial Intelligence for Autonomous Vehicles

ought to enhance the capabilities of autonomous vehicles. Future directions and a classification of technological advancements were provided. In the future, we anticipate developing an automated model using transfer learning approaches that will advance current autonomous systems. Furthermore, security and various driving styles will indeed be assessed with the aid of agents.

References 1. Parekh, D., Poddar, N., Rajpurkar, A., Chahal, M., Kumar, N., Joshi, G.P., Cho, W., A review on autonomous vehicles: Progress, methods and challenges. Electronics, 11, 14, 2162, 2022. 2. Wang, J., Liu, J., Kato, N., Networking and communications in autonomous driving: A survey. IEEE Commun. Surv. & Tutor., 21, 2, 1243–1274, 2018. 3. Sarraf, S., Current stage of autonomous driving through a quick survey for novice. Am. Acad. Sci. Res. J. Eng. Technol. Sci., 73, 1, 1–7, 2020. 4. Grigorescu, S., Trasnea, B., Cocias, T., Macesanu, G., A survey of deep learning techniques for autonomous driving. J. Field Robot., 37, 3, 362–386, 2020. 5. Sarraf, S., Desouza, D.D., Anderson, J.A.E., Saverino, C., MCADNNet: Recognizing stages of cognitive impairment through efficient convolutional fMRI and MRI neural network topology models. IEEE Access, 7, 155584– 155600, 2019. 6. Sarraf, S. and Ostadhashem, M., Big data application in functional magnetic resonance imaging using apache spark, in: 2016 Future Technologies Conference (FTC), IEEE, pp. 281–284, 2016. 7. Sarraf, S., 5G emerging technology and affected industries: Quick survey. Am. Acad. Sci. Res. J. Eng. Technol. Sci., 55, 1, 75–82, 2019. 8. Deshpande, P., Road safety and accident prevention in India: A review. Int. J. Adv. Eng. Technol., 5, 2, 64–68, 2014. 9. Campbell, M., Egerstedt, M., How, J.P., Murray, R.M., Autonomous driving in urban environments: Approaches, lessons and challenges. Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci., 368, 1928, 4649–4672, 2010. 10. Ziegler, J., Bender, P., Schreiber, M., Lategahn, H., Strauss, T., Stiller, C., Dang, T. et al., Making bertha drive—An autonomous journey on a historic route. IEEE Intell. Transp. Syst. Mag., 6, 2, 8–20, 2014. 11. Moed, H., De Bruin, R., Van Leeuwen, T.H., New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications. Scientometrics, 33, 3, 381–422, 1995. 12. Ginn, L.K., Citation analysis of authored articles in library & information science research, 2001-2002. Mississipi Libraries, 67, 106–109, 2003. 13. Helgath, J., Braun, P., Pritschet, A., Schubert, M., Böhm, P., Isemann, D., Investigating the effect of different autonomy levels on user acceptance

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  51

and user experience in self-driving cars with a VR driving simulator, in: International Conference of Design, User Experience, and Usability, pp. 247– 256, Springer, Cham, 2018. 14. Fagnant, D.J. and Kockelman, K., Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations. Transp. Res. Part A: Policy Pract., 77, 167–181, 2015. 15. Keqiang, L., Yifan, D., Shengbo, L., Mingyuan, B., State-of-the-art and technical trends of intelligent and connected vehicles. J. Automotive Saf. Energy, 8, 01, 1, 2017. 16. Russell S. J., Artificial intelligence/Stuart J. Russell, Peter Norvig. A modern approach (Third ed.), Prentice Hall, 2010. 17. Borroni-Bird, C. E., Burns, L.D., Mitchell, W.J., Reinventing the automobile: Personal urban mobility for the 21st century. MIT Press, Cambridge, MA, USA, 2010. 18. Sathiyaraj, R., Bharathi, A., Khan, S., Kiren, T., Khan, I.U., Fayaz, M., A genetic predictive model approach for smart traffic prediction and congestion avoidance for urban transportation. Wirel. Commun. Mob. Comput., 2022, 2022. 19. Gupta, A., Anpalagan, A., Guan, L., Khwaja, A.S., Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues. Array, 10, 100057, 2021. 20. Bachute, M.R. and Subhedar, J.M., Autonomous driving architectures: insights of machine learning and deep learning algorithms. Mach. Learn. Appl., 6, 100164, 2021. 21. Hong, J.W., Cruz, I., Williams, D., AI, you can drive my car: How we evaluate human drivers vs. self-driving cars. Comput. Hum. Behav., 125, 106944, 2021. 22. Sathiyaraj, R., Bharathi, A., Balusamy, B., Advanced intelligent predictive models for urban transportation, Chapman and Hall/CRC, 2022. 23. Sathiyaraj, R. and Bharathi, A., An efficient intelligent traffic light control and deviation system for traffic congestion avoidance using multi-agent system. Transport, 35, 3, 327–335, 2020. 24. Naz, N., Ehsan, M.K., Amirzada, M.R., Ali, Md Y, Qureshi, M.A., Intelligence of autonomous vehicles: A concise revisit. J. Sensors, 2022, 2022. 25. Rajendran, S. and Ayyasamy, B., Short-term traffic prediction model for urban transportation using structure pattern and regression: An Indian context. SN Appl. Sci., 2, 7, 1–11, 2020. 26. Saha, D. and De, S., Practical self-driving cars: Survey of the state-ofthe-art, Preprints, 2022, 2022020123, https://doi.org/10.20944/preprints 202202.0123.v1. 27. Bathla, G., Bhadane, K., Singh, R.K., Kumar, R., Aluvalu, R., Krishnamurthi, R., Kumar, A., Thakur, R.N., Basheer, S., Autonomous vehicles and intelligent automation: Applications, challenges, and opportunities. Mobile Inf. Syst., 2022, 2022.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

52  Artificial Intelligence for Autonomous Vehicles

28. Clements, L.M. and Kockelman, K.M., Economic effects of automated vehicles. Transp. Res. Record, 2606, 1, 106–114, 2017. 29. Speranza, M.G., Trends in transportation and logistics. Eur. J. Oper. Res., 264, 3, 830–836, 2018. 30. Lindqvist, U. and Neumann, P.G., The future of the internet of things. Commun. ACM, 60, 2, 26–30, 2017. 31. Parkinson, S., Ward, P., Wilson, K., Miller, J., Cyber threats facing autonomous and connected vehicles: Future challenges. IEEE Trans. Intell. Transp. Syst., 18, 11, 2898–2915, 2017. 32. Złotowski, J., Yogeeswaran, K., Bartneck, C., Can we control it? Autonomous robots threaten human identity, uniqueness, safety, and resources. Int. J. Human-Computer Stud., 100, 48–54, 2017. 33. Ning, H., Yin, R., Ullah, A., Shi, F., A survey on hybrid human-artificial intelligence for autonomous driving. IEEE Trans. Intell. Transp. Syst., 23, 7, 6011–6026, 2021. 34. Singh, S. and Saini, B.S., Autonomous cars: Recent developments, challenges, and possible solutions, in: IOP Conference Series: Materials Science and Engineering, vol. 1022, IOP Publishing, p. 012028, 2021. 35. Hataba, M., Sherif, A., Mahmoud, M., Abdallah, M., Alasmary, W., Security and privacy issues in autonomous vehicles: A layer-based survey. IEEE Open J. Commun. Soc., 3, 811–829, 2022. 36. Shuaib, M., Hassan, N.H., Usman, S., Alam, S., Bhatia, S., Mashat, A., Kumar, A., Kumar, M., Self-sovereign identity solution for blockchain-based land registry system: A comparison. Mobile Inf. Syst., 2022, 2022. 37. Kumar, A., de Jesus Pacheco, D.A., Kaushik, K., Rodrigues, J.J.P.C., Futuristic view of the internet of quantum drones: Review, challenges and research agenda. Veh. Commun., 36, 100487, 2022. 38. Liu, R., Yang, K., Roitberg, A., Zhang, J., Peng, K., Liu, H., Stiefelhagen, R., TransKD: Transformer knowledge distillation for efficient semantic segmentation. arXiv preprint arXiv, 2202, 13393 2022. 39. Gaihua, W., Jinheng, L., Lei, C., Yingying, D., Tianlun, Z., Instance segmentation convolutional neural network based on multi-scale attention mechanism. PLoS One, 17, 1, e0263134, 2022. 40. Gao, Y., Jing, H., Dianati, M., Hancock, C.M., Meng, X., Performance analysis of robust cooperative positioning based on GPS/UWB integration for connected autonomous vehicles. IEEE Trans. Intell. Veh., 8, 1, 790–802, 2022. 41. Schreiber, M., Belagiannis, V., Gläser, C., Dietmayer, K., A multi-task recurrent neural network for end-to-end dynamic occupancy grid mapping, in: IEEE Intelligent Vehicles Symposium (IV), pp. 315–322, IEEE, 2022. 42. Du, Y., Chen, J., Zhao, C., Liu, C., Liao, F., Chan, C.Y., Comfortable and energy-efficient speed control of autonomous vehicles on rough pavements using deep reinforcement learning. Transp. Res. Part C: Emerg. Technol., 134, 103489, 2022.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Age of Computational AI for Autonomous Vehicles  53

43. Chen, L., He, Y., Wang, Q., Pan, W., Ming, Z., Joint optimization of sensing, decision-making and motion-controlling for autonomous vehicles: A deep reinforcement learning approach. IEEE Trans. Veh. Technol., 71, 5, 4642– 4654, 2022. 44. Fey, V. and Rivin, E., Innovation on demand: New product development using TRIZ, Cambridge University Press, 2005.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

54  Artificial Intelligence for Autonomous Vehicles

State of the Art of Artificial Intelligence Approaches Toward Driverless Technology Sriram G. K., A. Malini* and Santhosh K.M.R. Department of Computer Science and Business Systems, Thiagarajar College of Engineering, Madurai, Tamil Nadu, India

Abstract

Throughout the last decades, the number of vehicles on the road has steadily increased due to the rising demand for urban mobility and contemporary logistics. Two of the many detrimental effects of more vehicles on the road, which also impede economic development, are increased traffic congestion and traffic accidents. The issues mentioned above can be significantly resolved by making vehicles smarter by reducing their reliance on humans. Over the past century, various nations have conducted extensive research that has fueled the automation of road vehicles. The development of autonomous vehicle (AV) technologies is currently being pursued by all significant motor manufacturers worldwide. Undoubtedly, the widespread use of autonomous cars is more imminent than we realize given the development of artificial intelligence (AI). In order for AVs to perceive their surroundings and make the right decisions in real time, AI has emerged as a crucial component. This development of AI is being driven by the growth of big data from numerous sensing devices and cutting-edge computing resources. We must first examine AI’s development and history in order to comprehend its functions in AV systems. Keywords:  Artificial intelligence, autonomous driving system, LiDAR, ADAS, center prediction neural network, CNN, AV

*Corresponding author: [email protected]@tce.edu; ORCID: 0000-0002-3324-5317 Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (55–74) © 2024 Scrivener Publishing LLC

55

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

3

3.1 Introduction The development of communication and robotics has had a significant impact on our daily lives, especially transportation. The advancement of autonomous vehicle (AV) technology has been made possible by these breakthroughs. As sensory-based beings, humans use object recognition as a part of daily life. The future of cars is computer-based object recognition. The transition from manual object recognition to automated object recognition is a significant one. Thousands of automobile traffic and collisions in contemporary cities mostly as a result of human error cause unending annoyance as well as significant loss of life, property, and productivity. Automobiles can be fully automated so that no human involvement is necessary as a way to mitigate this. Fuel efficiency, comfort, and convenience are additional benefits of autonomous vehicles. The greatest benefit to a self-driving society is the elimination of driver error. Driver error, which accounts for over 90% of all crashes, is the leading cause of the more than 31,000 fatal auto accidents that occur each year in the United States alone. According to the Eno Center for Transportation, there would be a reduction in collisions of over 4 million, a saving of over a hundred billion dollars, and a saving of 21,000 deaths annually if 90% of the vehicles on American roads were self-driving. Recent significant advances in artificial intelligence (AI), along with cutting-edge data collection and processing technologies, are what are propelling AV research to new heights. AVs have received a lot of attention recently due to their rapid development. The magnitude of success of AVs depends on the better obstacle-detecting sensors and AI that paves the way for their incorporation. A computer uses artificial intelligence in a similar way to how a human would. AIs in AV development have been greatly influenced by the success of AI in many complex applications such as predictive analytics to validate the need for surgery. In particular, the development of deep learning (DL) has made it possible for numerous studies to address complexities in the field of autonomous vehicles, which are precisely identifying and locating obstacles on roads, making the right decisions at the right time like controlling the speed and direction of movement of the vehicle, etc. Fuzzy logic, the computing method where the bases of decisions are based on degrees of truth rather than the standard true or false Boolean logic and artificial neural networks (ANNs), and systems of computation modeled after the biological neural networks that make up animal brains are two of the many AI techniques used. Nowadays, a variety of passive and active sensors, including those that record the subsequent echo similar to that of

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

56  Artificial Intelligence for Autonomous Vehicles

a LiDAR, RADAR, Global Navigation Satellite System (GNSS), cameras, and GPS sensors, can be integrated extensively in autonomous vehicles. Laser light is used by LiDAR sensors to illuminate its surroundings. The time delay or interval is measured after this light is reflected. This aids in building 3-dimensional images of its surroundings. The altitude, latitude, and longitude information provided by the position sensors like GNSS and GPS allows the car to be precisely located on a map for navigational purposes. Microwaves or radio waves are used by radar systems to measure an obstacle’s speed, direction, and distance from the autonomous vehicle. The reflected wave is picked up by the transmitter after the waves are reflected or bounced off. At the site of the transmitter, the reflected wave is picked up after the waves are reflected or bounced off. Using information from the aforementioned cameras, sensors, geolocation sensors, maps, navigation programming, and systems for connecting with other autonomous vehicles, software combined with AI process the amassed data and orchestrate the mechanical operations of the automobile. These techniques mimic the extraordinarily challenging effort that human drivers undertake when they have to monitor the road, the vehicle, and themselves in order to drive. The idea of driverless automobiles has been in existence for a while, but their prohibitive costs have prevented their widespread production. The AV concept has advanced to previously unheard-of levels—thanks to rapid research and development activities over the past 10 years. This is due to the fact that a wide range of stakeholders, such as transit agencies, IT oligopolies, transportation networking firms, automakers, chip and semiconductor producers, and others, have made large investments in and pushed these improvements. The necessity to tend to the geriatric population in developed countries and the rapid advancement of communication technology may have made AVs essential to business operations [1]. By 2040, these are anticipated to overtake an overall half the amount of the total vehicle sales and 40% of the travels, according to a forecast based on automatic transmission technologies or hybrid vehicle deployment and adoption of prior smart vehicles [2]. AV is associated with many beneficial societal effects, including safety during transport, reduced travel costs, and some mobility for people with limited mobility and those living in low-income families. Recently, it was predicted that, by 2025, about 1.9 trillion dollars of annual direct societal value will be produced [3]. These advantageous effects are what propelled the development of AV technology and will continue to do so in the long run. Autonomous vehicles should be taken into account when planning transportation in the future though, as they are likely to have a big impact on how people travel and how the road system functions. In this chapter,

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  57

we will discuss the past and current state of affairs with regard to autonomous vehicles and their potential effects on transportation in the future.

3.2 Role of AI in Driverless Cars 3.2.1 What is Artificial Intelligence? The term “artificial intelligence” was first used in 1955 by computer scientist John McCarthy. The ability for cognition, learning, and decision-making in a computer program or machine is known as artificial intelligence (AI). The phrase generally refers to a machine that mimics human intelligence. AI allows us to program machines and computer programs to perform humanlike tasks. These programs and machines receive a tremendous amount of data from us, which are then evaluated and processed so that they can act like humans and think logically. Automating commonplace human jobs is just the tip of AI’s iceberg; potential uses include driverless vehicles and life-saving diagnostic medical tools. Reactive machines, limited memory, theory of mind, and self-awareness are the four different categories of AI. Reactive machines, the most basic type of AI, neither have the capacity for memory nor are able to recall the past or draw inferences from it in order to render judgments today. IBM’s Deep Blue supercomputer that can play chess can be considered one of the best examples of this kind of variety of AI system, which back in early 1997 beat the grandmaster of chess. This type of cognition involves the computer actively studying the outside environment and reacting in accordance with its observations that is not influenced by any personal ideology. Rodney Brooks, an Australian AI expert, argued that, in contrast to what is often assumed in the domain of AI, we are not particularly adept in modeling realistic computer simulations. The second type is capable of gazing backward. AVs have already incorporated some of this. For instance, they keep an eye on the pace and direction of other cars on the road. It takes time and persistence to attain it; it requires identifying specific things and persistent monitoring of them. This is then incorporated to the previously preprogrammed world models the AVs possess, which also contain lane markings, traffic signals, and a few other important aspects, including bends in the road. To avert a breach in driving etiquette or being hit by passing cars or vehicles, they are taken into account while determining when to switch lanes by the vehicle. However, these basic tidbits of historical knowledge are not permanent. The automobile cannot learn from them, since they are not saved in the same way that years of driving experience help human drivers become better drivers.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

58  Artificial Intelligence for Autonomous Vehicles

The third type could be chosen as the defining distinction between the machines of current generation and those that will be developed in the upcoming years. Increased clarity is preferred when discussing the types of representations that machines must produce and the subjects of such depictions. The following more complex category of machines generates depictions of not only the environment but also other agents or entities that inhabit it. The “theory of mind” is a psychological phrase used to support the idea that humans, animals, and other beings can all have the ability to think and feelings that affect one’s behavior in either a direct or an indirect manner. Self-awareness is the final stage in the AI development phase. Finally, AI researchers will have to develop intelligent machines that are potentially capable of understanding the abovementioned environment in which it is being deployed. This is sort of a continuation of the “theory of mind” that the AIs in this level possess. The term “self-awareness” is another name for consciousness. Conscious beings are aware of who they are, are aware of how they are feeling internally, and can anticipate how others are feeling. We naturally believe that everyone who honks at us in traffic is furious or irritated because we have a similar feeling when we honk at others. Without a theory of mind, such conclusions would not be conceivable.

3.2.2 What are Autonomous Vehicles? The main feature of AI in interpreting its surroundings enables or gives way to a type of independent functional vehicle that is automated, the autonomous vehicle, to drive itself and carry out necessary actions without human interaction or assistance. The fully automated driving system allows the AV to make decisions in complicated situations as a human driver would. The independent operational driverless cars are the most automated ones standing in the level of at most 6; hence, it can be said that as the degree increases, so does the sophistication and its operations. The basic-level car/vehicle is completely human-driven and has zero operational control. The movement of the car is directed by the driver. For instance, one must manually control the vehicle’s speed, use the brakes, and judge when to halt. This holds true for cars that include features like traditional cruise control systems that have to be manually set or adjusted in order to work. Advanced driver assistance system (ADAS) in the AV can assist the driver with Level 1 basic tasks such as braking or steering or acceleration. One or more technologies for assisted driving, such as parking sensors or adaptive cruise control, are present in Level 1 automobiles. Like

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  59

traditional driving control systems, adaptive driving control systems can change the speed of the vehicle and maintain safe stopping distances in addition to maintaining a predetermined speed. Despite the fact that this feature might slow down a car, a driver must remain vigilant and manage brake pressure in accordance with the information at hand. Parking sensors sound a warning when they identify an object in the driver’s parking route. Although this is intended to help drivers, the automobile must still be physically moved when parking. A large percentage of the vehicles we encounter on the road today have Level 1 driving automation. The second-level advanced driver assistance systems can only control limited tasks such as braking, steering, and acceleration, so they must interact with the driver to make decisions based on road conditions. Vehicles at the second level have two or more simultaneous assisted driving technologies. For instance, a Level 2 car can adjust its acceleration and steering on the highway according to the speed of the traffic in front of it. These cars can only drive themselves in specific situations. At the third level, it is capable of handling driving activities most of the time, but a handler is still necessary to be able to take control of the vehicle when required. Thus, the exceptional scenarios are handled by the human driver. Audi’s AI traffic jam pilot is an illustration of conditional automation. When a traffic jam is encountered, the system has the ability to step in and move slowly through it. When the road is clear, the vehicle signals the driver to take control again because the conditions for autonomous operation are no longer met. Under specific circumstances, the car merely keeps an eye on its surroundings; in all other situations, the driver is responsible. At the fourth level, the ADS of the vehicle is open to handle all the driving tasks by itself in circumstances where the attention and interaction of humans are not necessary. Complete automation is incorporated in Level 5, where the ADS of the vehicles is capable of handling every task in all circumstances without the need for human driver assistance. The 5G technology will take AV to the next level by enabling full automation by allowing AVs to communicate not only with fellow vehicles but also with traffic lights, roads, and signage; hence, complete automation will be made possible.

3.2.3 History of Artificial Intelligence in Driverless Cars The first production car, dubbed the “Benz Patent-Motorwagen,” was created in 1885 by Karl Benz in Mannheim, Germany, and it included a gasoline-powered internal combustion engine. Ever since the inception of the Benz Patent-Motorwagen, people wanted cars to go autonomous.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

60  Artificial Intelligence for Autonomous Vehicles

In 1925, Francis Houdina, the inventor, operated a remote-controlled unmanned vehicle on the streets of Manhattan. The radio had the ability to start the engine, blow the horn, and shift gears. There was a display of the first self-driving vehicle created by General Motors. The metal spikes on the road assisted in the functioning of the electric vehicle that was being steered by radio-controlled EM fields. In the 1960s, proponents of AI started to imagine autonomous cars that could drive through regular streets. Reverse-engineering the necessary systems similar to those found in a moving animal posed enormous problems that included sensing, processing, and reacting. With the technology available to them, the first and the last steps were possible. Processing, the machine intelligence required in between them, was the unknown component. It can be very deadly when a vehicle’s AI confuses a human in the road with the puddle reflection alongside. Several teams were competing in 2004 for $1 million by the US DARPA for attaining the dream in by 2015, the US military vehicles one-third will be automated. The initial batch of 15 competitors failed badly, making it only a short distance from the planned 227-km race before colliding. That was not the case the next time around. The Mojave Desert in California was traversed by an odd armada of driverless cars and trucks without a scratch the next year in which the autonomous robotic automobile “Stanley” from the Stanford University Racing Team took First Place. This team was mentored by the director of Stanford AI Laboratory and associate professor of CS department, Sebastian Thrun, and was a victory in the use of ML. Stanley was well versed in machine learning algorithm integration with sensors that helped it recognize impediments and avoid them while maintaining its course. By 2007, the Urban Challenge had expanded those accomplishments to a fictitious urban setting. The foundation for self-driving had been created by European experts, but the United States was now a strong competitor. The disparity was caused by several things that include better radar and laser sensors, as well as upgraded software for road following and collision prevention. Also helpful was good mapping. Animals are better at analyzing their environments than robots are, but a car that is always aware of its surroundings can concentrate on several variables. After leading the Stanford University racing team to win the DARPA Grand Challenge in 2005, Thrun led Google’s self-driving car project and became Waymo in 2016. To bring complete autonomous driving to the masses, Waymo made great benefit of AI. Engineers from the firm and the Google Brain team worked together to integrate deep neural network (DNN) into their pedestrian detection system. Proposed in the 1960s, DNNs, also known as deep

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  61

neural learning, is part of a broader area of artificial intelligence. DNNs are more complex and abstract than smaller neural networks, allowing them to learn high-level features [4]. With the help of DNN, engineers were able to cut pedestrian identification errors in half using deep learning techniques. More than 10 million kilometers of road training and hundreds of millions of interactions involving pedestrians, cyclists, and cars have been integrated into Waymo’s deep learning modular training program. It is recorded by Waymo that it has traveled in an autonomous mode more than 10 billion miles in simulation as part of the company’s deep learning training.

3.2.4 Advancements Over the Years Like any previous automotive breakthrough, it took a while for this technology to develop to the point where it is now. The autonomous vehicle concept is the result of intense work by a number of educational institutions and research scholars, as well as big giant automobile companies. It is similar to how cruise control, ABS, and airbags, as well as today’s advanced emergency braking (AEB), were all developed. A new system for the AV test vehicle of Continental was developed by experts from the industry like Siemens and technical universities in Darmstadt and Munich. To standardize testing, the Continental created an autonomous vehicle in 1969. To the shock of the onlookers, Continental’s first electronically controlled driverless vehicle made its debut on the Lüneburg Heath Contidrom test track on 11 September 1968. This car was directed by the conductor wire in the lane of the road’s surface. The vehicle’s sensors in the electronics system allowed it to determine whether it was still on course and automatically changed the steering as necessary. Hans-Jürgen Meyer said, “In the end, it was a wire-driven automobile.” The engineers fitted several cutting-edge pieces of technology in the Mercedes-Benz 250 automatic with electromechanical steering, throttle valve controller, and radio system for transmitting measurements. There were numerous antennas in the bumper, and the control electronics and electro-pneumatic braking system were in the trunk. The control station next to the test track delivered instructions to the car via wires telling it to accelerate or honk its horns or brake. At the Tsukuba Mechanical Engineering Laboratory in Japan, S. Tsugawa and his colleagues created the first truly autonomous vehicle in 1977. This car was capable of processing images of the road in front of it and was self-driving. It had two cameras and processed signals using analog

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

62  Artificial Intelligence for Autonomous Vehicles

computer technology. It was capable of recognizing white street markings when traveling at a speed of 33 km per hour. The concept of self-driving automobiles really got its start with this advancement in the realm of vehicular automation. Then, many major automakers began making an effort to develop autonomous vehicles [5]. University of Munich researcher Ernst Dickmanns first began working independently on AVs in the 1980s. He purchased a Mercedes van and filled it to the brim with electronics. The VaMoRs, or “Versuchsfahrzeug für autonomous mobilität und Rechnerseh,” was the name of this vehicle that translates to “test car for self-driving and computer vision.” It was outfitted with 16-bit Intel logic chips, as well as a number of additional sensing devices and software that help it to identify anything on the road. While being tested on the German Autobahn, it went about 20 miles at a speed of over 90 km per hour, with 96 km per hour top speed. The main invention of Dickmanns was the aptly named “dynamic vision,” which allowed the imaging system to exclude irrelevant noise and concentrate exclusively on pertinent items. The VaMoRs trial piqued Daimler’s interest in Dickmanns’ work. He was given the opportunity to work with €800 million that is over a billion dollars when adjusted for inflation as part of the EUREKA Prometheus Project to create the VaMP, a Mercedes-Benz W140 500 SEL with the VaMoRs’ autonomous driving technologies installed. The VITA-2 was the identical twin of the VaMP. The largest research and development initiative ever undertaken in the realm of autonomous cars was the Eureka PROMETHEUS. Here, €749 million in funding was received from EUREKA Member States. This pan-European effort included participation from numerous colleges and automakers. VaMP, developed after 7 years of VaMoRs, used two cameras to process 320 × 240-pixel images at a distance of 100 m to discern lane markers, their correlative positioning in the lane, and the prevalence of other automobiles. During a trial run close to Paris, VaMP managed to navigate traffic and speed to 130 km per hour while detecting the right time to switch lanes. The following year, Dickmanns’ team traveled 1,600 km in a Mercedes S-Class between Munich and Denmark speeding up to 180 km per hour with “approximately 95% of the distance...traveled totally automatically,” according to Dickmanns. The DARPA Grand Challenge that was held for the first time was held in the United States in the Mojave Desert and announced on 30 July 2002 was approved by the US Congress and offered a $1 million prize with the goal of reducing ground warfare by one-third. Vehicles were used by the military until 2015. The Carnegie Mellon University Red Team covered the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  63

farthest distance, covering 11.9 km out of the planned 227 km, although none of the cars managed to cover the entire distance. As a result, no team could win because they could only cover 5% of the total distance. In June of that same year, the next grand challenge with a little over 200 km offroad course and a $2 million reward was announced by DARPA in the year 2004, which was twice as much as the first. The 23 final contestants performed in October 2005 with the benefit of lessons learned and upgraded cars. The tough course included three tunnels, a steep pass with precipitous drop-offs, and over 100 twists and turns. With a winning time of 6 h 53 min, the Stanford Racing Team took home the $2 million prize, followed by the Carnegie Mellon Red Team. Five teams in all finished the contest. As the goal of the competition was to create a vehicle that could detect oncoming obstacles and track GPS waypoints, it was announced as one of the cornerstones of implementing autonomous driving. These developments sparked curiosity and creativity, and the results were pleasing to herald the next challenge. In May 2006, the Defense Department revealed the Urban Challenge, the third in the sequence of challenges. It occurred on 3 November 2007 at the California location formerly known as George AFB Victorville. The goal of this competition, which built on the success of the Grand Challenges in 2004 and 2005, was to create a vehicle that could operate in traffic without a human driver and handle challenging circumstances including parking, passing, and navigating intersections. As the first instance in which autonomous cars and human-driven cars have interacted in traffic in an urban setting, this occurrence was both singular and absolutely revolutionary. With a 97-km urban circuit, the competition was more difficult this time. With an average speed of 22.5 km per hour over the course of 6 h in a difficult metropolitan region, Tartan Racing’s “Boss” from Carnegie Mellon University triumphed. According to Mr. Tony Tether, the director of DARPA, autonomous vehicle technology is reliable and will inevitably protect lives both on and off the road. Teams of participants were sponsored by major manufacturers and technology companies like GM, VW, Caterpillar, Continental AG, Intel, Google, and others. The first transcontinental driving test route was undertaken by the VisLab at the University of Parma. It began on 20 July 2010, in Parma, and ended on 28 October 2010, in Shanghai, encompassing 15,926 km through nine different nations. The scientific findings of the VisLab Intercontinental Autonomous Challenge (VIAC), which involved driving photovoltaic electric vehicles across more than 13,000-km distance, will serve as the first examples of autonomous driving. These exacting tests offer a comprehensive evaluation of the technology being developed. The system is tested in

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

64  Artificial Intelligence for Autonomous Vehicles

various settings to see how it performs under various circumstances. At the conclusion of the excursion, the data will be used for further lab analysis. This expedition consists of four autonomous vehicles, which includes two traveling vehicles and two backup vehicles plus a non-autonomous support vehicle [6]. As described in Figure 3.1, when the leader is visible, the cars take a leader-follower method, where the follower precisely follows the leader’s trajectory while fine-tuning its road lane position is taken care by sensing locally using various technologies including image selection and KLT tracking. GPS data are used to define the route when the leader is concealed or too far away for the following to see it. The car also has a radio for inter-vehicle communication, GPS, and an IMU. Additionally, the vehicle has a solar panel that can provide enough electricity to properly all of the systems in the vehicle. Each vehicle has three personal computers (PCs), two of which handle sensor data and one of which has a world model that receives messages from the other two PCs. The optimal maneuver, and consequently the best trajectory, is chosen from the synthetic model in order to complete the chosen mission [7]. The BRAiVE experience-based sensor is used as a suit, with the exception of an extra laser scanner that is used particularly for off-road driving and is positioned to frame the ground. Additionally, the vehicle has V2V, GPS, and IMU communication systems. The car’s roof is an inertial as well as a GPS-like gadget powered by TopCon AGI3. The Egnos/WAAS adjustment is used by the GPS to attain an accuracy of roughly 1 m. IMU comprises gyroscopes, accelerometers, and magnetometers of axes 3 and 2, respectively. The VIAC until this date is the most tested self-driving project, traveling 13,000 km across several Data collection

Image selection

Processing of vision features

Candidate creation with LiDAR

Kanade Lucas Tomasi Tracking

Pre Processing

Validation of potential candidates

Processing Primary suggestion of vehicle

Figure 3.1  System architecture for selecting the leader car.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  65

nations, i.e., from the land of Italy to far China, which came to an end on 28 October 2010 [8]. The primary distinction of the usual AV tests and VIAC tests, demonstrations, or agility testing, such as the VIAC and DARPA, was able to cross usual roads under common circumstances and that regarding traffic, road conditions, or adherence to traffic laws is not considered and assumed [7]. According to a Google report from September 2016, since the project’s 2009 launch, it has traveled more than 3.38 million kilometers [9]. Back in 2005, Stanford won its $2 million prize in DARPA challenge with the help of Sebastian Thrun, the former director of Stanford AI lab and a coinventor of Google Street View and who served as the principal investigator of the projects. Sebastian Thrun, who eventually ended his tenure in the company with an idea to launch his own businesses, was succeeded by Chris Urmson, who later oversaw the team. In August 2016, Mr. Urmson also departed the project [10].

3.2.5 Driverless Cars and the Technology they are Built Upon Serial number 1

Name

Year

The Continental Contidrom

1969

Testing platform

Technology used

MercedesBenz 250 Automatic

Electromechanical steering Electromechanical throttle control Radio system for reporting measurements Electro-pneumatic braking system

2

Tsukuba Mechanical Engineering

1977

3

VaMoRs

1986

Analog computer technology Mercedes-Benz van

Saccadic Vision Kalman Filters Parallel Computers 16-bit Intel Microprocessor (Continued)

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

66  Artificial Intelligence for Autonomous Vehicles

(Continued) Serial number 4

Name

Year

The EUREKA 1987–1995 PROMETHEUS project

Testing platform

Technology used

Mercedes-Benz W140 500 SEL

PIMM1 integrated processors Mathematical morphology ASIC developed by the CMM Temporal dynamic morphological filter (TDF) PROLAB2 EMS-Vision autonomy system

5

Team Stanford Racing from DARPA Grand Challenge

2005

Modified Volkswagen Touareg R5 named Stanley

Pentium M computers 6DOF measurement unit for inertia Range finding with the help of laser 24GHz Radio Detection And Ranging system A pair of stereo camera Single eye type vision system Global Positioning System (Continued)

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  67

(Continued) Serial number 6

Name

Year

2010 VisLab Intercontinental Autonomous Challenge

Testing platform

Technology used

Piaggio Porter Electric Power

BRAiVE platform X-BY-WIRE SYSTEM Laser Scanner V2V communication System IMU Global Positioning System Panoramic camera system LiDAR GOLD framework Felisa and Zani road position detection system VisLab designed Calibration tools and procedures

7

Waymo LLC

2009 Chrysler - Present Pacifica Hybrid minivan, Jaguar I-Pace and Class 8 truck Daimler’s Freightliner Cascadia, Modified Toyota Priuses, Lexus SUVs, Custombuilt prototype vehicle named Firefly,

TensorFlow TPUs SurfelGAN LiDARs Sensor fusion technology Deep Neural Networks (DNNs) Center Prediction Neural Network (CPNN) Object Property Neural Network (OPNN) AutoML NAS CNNs

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

68  Artificial Intelligence for Autonomous Vehicles

3.2.6 Advancement of Algorithms Using an image processing technique in 1979, an experimental Stanford Cart that used the Cart’s Vision Algorithm was able to move around on its own in a small environment. This algorithm, which was modeled after the Blocks World planning technique, was shown to be ineffective when used outdoors even though it effectively downgraded images as edges set, where there were numerous complex forms and colors [11]. However, one of the most well-known techniques in artificial intelligence planning [12] and environment recognition is the 1960 Blocks World method. The first autonomous car capable of traveling a predetermined area was the experimental VaMP vehicle in 1995, covering more than a thousand kilometers without human aid. The prototype of the EMS-Vision autonomous system could drive through traffic and past cars—thanks to data collected from bifocal camera systems mounted on biaxial platforms. Maps of the road network, waypoints on the map that were static objects, and statistics were all employed by EMS-Vision. The EMS-Vision is a sophisticated system that can continually configure itself throughout the operation in response to the current situation and also has specialized modules for road recognition, attention control, navigation, and vehicle control [13]. The EMS-Vision system discussed above excludes advancements made after 1995; however, it does show the complexity and volume of algorithms used, which are meant as specified actions necessary to complete certain tasks in AV systems [11]. The culmination of parallel lines at a single position in three-dimensional space gives crucial information for recognizing roadways, much as the parallel lines that make up a lane meet at a single point in a two-dimensional picture. Identifying a road lane involves looking for lines that confluence at a specific location. By detecting the predominant picture section orientations, C. Rasmussen proposed the Vanishing Point method in the first place (640 × 480-pixel images are divided into 72 segments), predicting the location of the intersection point and then succeeding picture frames after that point. Today, this method, developed to locate the road in a desert area, is essential for identifying lanes of the road in a picture and forms the basis for more complex algorithms. Machine learning can be used for labeling specific regions of an image according to what’s being shown, and an example of it is the full-resolution residual networks (FRRNs). This is hardly used in practice because of the high computing cost and the numerous artifacts produced [14]. With the introduction of specialized equipment and new quicker techniques for semantic segmentation, this strategy might alter. Machine vision is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  69

used to identify particular picture properties that were determined earlier, including moving objects, traffic lanes, barriers, and distance estimation. Road signs, traffic lights, and other road markers can all be recognized by specialized Advanced Driver Assistance Systems (ADAS) [15]. To reliably detect road signs and other visual features, three types of algorithms are used: general machine learning algorithms, artificial neural networks, and deep machine learning. These methods arose from multiple studies and were reported in scientific journals. The SVM approaches are the most widely used and thoroughly studied techniques for identifying road signs. AdaBoost is known for its quick execution times, while neural network techniques are the slowest. Deep machine learning techniques might be challenging to implement in ADAS systems due to their high-technology requirements. R-CNN that runs more quickly might be an alternative for semantic image segmentation [16]. The initial choice made by an autonomous vehicle is the route (path) to the destination, which is typically dependent on variables like distance, travel time, or estimated fuel consumption. The Bellman-Ford algorithm, which is constrained by the requirement to define non-negative weighted edges, constraint, the Dijkstra algorithm, which can be used if the topology of roads is known, and the 1968 A* algorithm, along with its modifications, are examples of heuristic algorithms that take into account the criterion of travel [17].

3.2.7 Case Study on Tesla The year 2003 saw the founding of the American electric car manufacturer Tesla by Elon Musk, Marc Tarpenning, JB Straubel, Ian Wright, and Martin Eberhard. The production of EVs with lithium-ion batteries is Tesla’s area of expertise. Tesla, a forerunner in the utilization of renewable energy for generating sustainable clean energy for vehicle propulsion, is establishing a firm platform for the expansion of the EV market. Unlike traditional vehicles that are primarily reliant on using fossil fuels to power their engines, this approach is more ecologically responsible. From its very founding in 2003, Tesla has had tremendous market growth mostly as a result of its leadership in the EV sector development of cutting-edge technology innovation. Tesla not long ago became a member of the Trillion Dollar Club, which has only been attained by a select few businesses. The rise of the electric vehicle sector is being aided by government policies and efforts, which has a positive impact on the worldwide market. The internal combustion of gas, which occurs in conventional fossil fuel-powered vehicles, is what contributes to the production of greenhouse gases that harm the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

70  Artificial Intelligence for Autonomous Vehicles

environment. However, an EV uses a motor that is powered by electricity from a source that keeps the current at a specified value regardless of the load condition, and this source is named the current source, thus helping in the reduction of overall pollution. Countries around the world including the United States and China have imposed various regulations and laws to protect their environment. The use of EVs is greatly supported by initiatives taken by various organizations like the California Air Resources Board (CARB). The aim to increase the adoption of electrical vehicles is supported globally by three primary pillars: the government’s initial fleet purchase, purchasing EVs at a lower selling price as a result of regional and nationwide incentives, and the law that forces automobile manufacturers to begin selling a certain number of EVs from the year 2019. As a result of the synchronic alignment with the environmental goal of the government, Tesla has substantial backing from the government. EVs are generally known to increase the energy security and quality of air as the carbon dioxide emissions are very well lower than that of conventional vehicles. Under the Bush administration’s advanced technology vehicle manufacturing program, Tesla originally obtained a loan, which it later promised to repay with no interest. Tesla has received governmental backing in the form of many state subsidies in addition to a federal loan. For instance, additional income tax credits are now being provided by several states for each Tesla purchase. These incentives eventually became a crucial part of Tesla’s business model and its EV advertising campaigns. Tesla’s stock price increased by over $12 billion USD, demonstrating that the company’s financial performance is closely tied to government-granted privilege and that political preferences are given more weight than delivering value to customers [18]. Automobile makers are now focusing more on developing electric automobiles than traditional ones, which encourages the development of EV in the automotive sector. In the automobile business, EVs provide a significant amount of advantages over internal combustion vehicles. For example, energy can be employed to naturally powered EVs. Alternatively, conventional vehicles that are propelled by the burning of gasoline use up oil, an exhaustible natural resource. Additionally, gas is more costly than electricity. In order to supply the energy needed for the car to run while it is moving, electrical vehicles use regenerative braking. As a result, they are substantially less expensive than gas-powered automobiles, often costing only one-third as much [18]. A form of rechargeable battery called a lithium-ion battery is used to power electric vehicles and portable electronics. Akira Yoshino originally created a prototype Li-ion battery in 1985, and Sony and Asahi later

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  71

improved on it to produce a commercial Li-ion battery. In the past, the usage of lithium-ion batteries was widespread in the EV market [19]. Even though researchers are developing replacement technologies for Li-ion batteries, the International Energy Agency (IEA) still predicts that the Li-ion battery will remain to be the industry standard for the next few years in the EV market. In the meantime, Tesla is the only firm able to use battery cells in the shape of a cylinder in its battery packs to lower the price of a cylinder battery cell greater than 158 USD per kWh [20]. Tesla employs sophisticated engineering techniques along with the creation of more gigafactories to make it a reality. Thus, it is safe to say that Tesla has therefore transformed the battery sectors by producing battery cells in the shape of a cylinder to make it more cost-efficient to be used in battery packs. Due to Tesla’s enviable position as the undisputed leader in the worldwide EV industry, other automakers claim they would invest huge amounts of money to overtake Tesla. Established firms like Toyota and Panasonic have begun to merge horizontally along with the recognized EV sector in order to create and build more potent battery packs to propel the EVs. Furthermore, in an effort to increase the number of battery operations in Ohio, General Motors and South Korea’s LG Chem have committed millions and millions of dollars to attempt to stay up with Tesla’s production of powerful batteries. The price-to-earnings (P/E) ratio signifies the correlation between the change in stock price and earnings. The company’s future gains and growth can be greatly known by analyzing this metric. Tesla outperforms its competitors in the market with a significantly higher P/E ratio. A study on this supremacy inferred that there is a higher chance of Tesla stocks being overvalued, as, in the market, it is almost 12 times greater and constantly growing besides having a staggeringly low overall electric vehicle sales. In the year 2020, Tesla was the dominant force in the sector of automobiles, as it showed significant growth in P/E from the lowest to the highest ratio. The P/E ratio of Tesla greatly exceeds those of the other automotive sector P/E ratios; hence, making it stable enough to continuously outperform its competitors. To conclude, Tesla keeps expanding abroad to hasten the uptake of sustainable energy production and transportation worldwide. Political, economic, social and technological (PEST) perspectives claim that Tesla has received a lot of support in order to expand its impact globally. Due to its monopolistic market domination and technological advancements, Tesla has an advantage over its rivals in terms of how it views market rivalry and crucial technology. These characteristics increase the likelihood that shareholders will be upbeat about Tesla’s future growth, which might increase

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

72  Artificial Intelligence for Autonomous Vehicles

Tesla’s real market value. The results of the numerous valuation techniques employed in this study clearly demonstrated that the true market rate for Tesla is still exorbitant, proving that the share price is excessive.

3.3 Conclusion In conclusion, even if autonomous vehicles appear to be a distant concern for present road users, global tests of vehicles in motion indicate that public cars may soon be available. Another revolution similar to those of the DARPA Grand Challenge will undoubtedly result in changes in transportation, so it is critical to inform the public about how to handle autonomous vehicles. This instruction should cover proper conduct on the roadways where the vehicles would be able to move. However, it is crucial to educate the general public with the applied nomenclature and the division of vehicles owing to the degree of autonomous driving before cars are fully integrated in the existing transportation systems. This will make it easier for society to learn about a new mode of transportation that, although it may currently appear abstract, represents the transportation of the future.

References 1. Hong, D., Kimmel, S., Boehling, R., Camoriano, N., Cardwell, W., Jannaman, G., Purcell, A., Ross, D., Russel, E., Development of a semi-autonomous vehicle operable by the visually-impaired, in: IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI, pp. 539–544, 20082008. 2. Litman, T., Autonomous vehicle implementation predictions, Victoria Transport Policy Institute 28, Victoria, British Columbia, Canada, 2015. 3. Manyika, J., Chui, M., Bughin, J., Dobbs, R., Bisson, P., Marrs, A., Disruptive technologies: Advances that will transform life, business, and the global economy, McKinsey Global Institute, New York, 2013. 4. Sze, V., Chen, Y.-H., Yang, T.-J., Emer, J.S., Efficient processing of deep neural networks: A tutorial and survey, in: Proceedings of the IEEE, vol. 105, pp. 2295–2329, Dec. 2017, doi: 10.1109/JPROC.2017.2761740. 5. Bhat, A., Autonomous vehicles: A perspective of past and future trends, 2017. 6. Broggi, A., Cerri, P., Felisa, M., Laghi, M., Mazzei, L., Porta, P.P., The VisLab intercontinental autonomous challenge: An extensive test for a platoon of intelligent vehicles. Int. J. Veh. Auton. Syst., 4, 1185, 2012, 10. 10.1504/ IJVAS.2012.051250.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Advancements of AI in Driverless Technology  73

7. Bertozzi, M., Bombini, L., Broggi, A., Buzzoni, M., Cardarelli, E., Cattani, S., Cerri, P., Debattisti, S., Fedriga, R., Felisa, M., Gatti, L., Giacomazzo, A., Grisleri, P., Laghi, M., Mazzei, L., Medici, P., Panciroli, M., Porta, P.P., Zani, P., The VisLab intercontinental autonomous challenge: 13,000 km, 3 months, no driver, 2010. 8. Broggi, A., Bombini, L., Cattani, S., Cerri, P., Fedriga, R., II, Sensing requirements for a 13,000 km intercontinental autonomous drive. 2010 IEEE Intelligent Vehicles Symposium, pp. 500–505, 2010, doi: 10.1109/ IVS.2010.5548026. 9. Hongyu, H., Chi, Z., Yuhuan, S., Bin, Z., Fei, G., An improved artificial potential field model considering vehicle velocity for autonomous driving. IFAC-PapersOnLine, 51, 31, 1, 2018. 10. Maddodi, S. and Prasad, K., Recent advances in technological innovations in IT, management, education & social sciences. An Evol. Auton. Vehicle-A Case Study Waymo, Oct. 2019. 11. Bugała, M., Algorithms applied in autonomous vehicle systems, 2018. 12. Gupta, N. and Dana, S.N., On the complexity of blocks-world planning. Artif. Intell., 56, Elsevier, 2, 1992. 13. Gregor, R., Lutzeler, M., Pellkofer, M., Siedersberger, K.-H., Dickmanns, E.D., EMS-Vision: A perceptual system for autonomous vehicles. IEEE Trans. Intell. Transp. Syst., 3, 1, 48–59, March 2002. 14. Rasmussen, C., Grouping dominant orientations or Ill-structured road following. International Conference on Computer Vision and Pattern Recognition, IEEE, 2004. 15. Li, L., Wen, D., Zheng, N.-N., Shen, L.-C., Cognitive cars: A new frontier for ADAS research. IEEE Trans. Intell. Transp. Syst., 13, 1, 395–407, March 2012, doi: 10.1109/TITS.2011.2159493. 16. Mukhometzianov, R. and Wang, Y., Review. Machine learning techniques for traffic sign detection, 4, 2017. 17. Ibarra, O.H. and Kim, C.E., Heuristic algorithms for scheduling independent tasks on nonidentical processors. J. ACM, 24, 2, 280–289, 1977. 18. Liu, S., Competition and valuation: A case study of tesla motors. IOP Conference Series: Earth and Environmental Science, vol. 692, p. 022103, 2021, 10.1088/1755-1315/692/2/022103. 19. Blomgren, G.E., The development and future of lithium ion batteries. J. Electrochem. Soc., 164, A5019, 2017. 20. Drexhagen, P., Tesla: A tech company selling cars-a story-driven valuation? Diss., 2021, https://doi.org/10362/122665.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

74  Artificial Intelligence for Autonomous Vehicles

A Survey on Architecture of Autonomous Vehicles Ramyavarshini P.*, A. Malini and Mahalakshmi S. Department of Computer Science and Business Systems, Thiagarajar College of Engineering, Madurai, Tamil Nadu, India

Abstract

Artificial intelligence is now a necessary component for both production and service systems in recent years, as technology has become a vital aspect of daily life. Automated driving vehicles operate autonomously, also known as driverless cars that can operate without a human driver. Research on autonomous vehicles has substantially advanced in recent years. Artificially intelligent autonomous vehicles are the current need of the society. Although some people might be apprehensive to give a computer control of their vehicle, automated driving technologies have the potential to make roads safer. Self-driving automobiles can address environmental issues as well as safety-related ones. Unlike humans, computers do not really have difficulty keeping attention when driving. Additionally, by responding appropriately, an automated car can prevent accidents to potentially dangerous events on the road. Self-driving technology has many advantages, one of which will make more easily accessible means of transport to people who are unable to drive. For a variety of reasons, such as inexperience, incapacity, or age, many people are unable to operate a vehicle. These individuals can travel considerably more safely and independently. Therefore, we will explore the architectures of both software and hardware of autonomous cars in this chapter, as well as their parts, benefits, and future developments. Keywords:  Autonomous vehicles, artificial intelligence, radar, LiDAR, GPS, CAN bus, GNSS

*Corresponding author: [email protected]; ORCID: 0000-0002-3324-5317 Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (75–104) © 2024 Scrivener Publishing LLC

75

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

4

4.1 Introduction 4.1.1 What is Artificial Intelligence? Unlike the intelligence possessed naturally by both humans and animals, the knowledge demonstrated by machines is called artificial intelligence (AI). An investigation of intelligent agents that perceive their environments that react in a manner that maximizes their success chances might be characterized as AI research [1]. Before, AI was called to be mimicking and exhibiting human cognitive capabilities that are linked to the human brain. This was rejected by many AI researchers, and all are now reexpressing artificial intelligence in a way of acting rationally [2]. Innovative Web search engines like Google, engines recommending as YouTube, Amazon, and Netflix, speech-recognition mechanisms like Siri, cars like Tesla that can drive on its own, decisions that are automated and overpowering the top games that use strategies. Artificial intelligence impact is an event where the capability of tools is more, moves that were once thought to need intelligence are now frequently excluded as more competent according to the concept of AI. Despite being a prevalent technology, optical recognition systems are commonly excluded from the list of things that are considered to be AI [3]. While talking about AI, supercomputers are the things that we get remembered of. A supercomputer has huge processing power, adaptive actions (via sensors), and a variety of many abilities that help it to incorporate human’s understanding as well as functional capacity, which actually helps in enhancing the communication of the supercomputer with humans. In fact, a variety of films had been produced in order to demonstrate the capabilities of AI, which includes smart buildings, like the ability of it to control temperatures, quality of air, and music depending on the detected feel of the residents of the place. Apart from the formal understanding of AI as well as a supercomputer to encompass integrated computer systems, there has been a growth in the use of AI within the education sector [4]. The ability of reasoning as well as taking actions that have the best likelihood to reach a certain objective is the ideal quality of AI. Machine learning (ML), being a subtype of AI, is a way in which computer programs automate learning from previous data and adapt themselves to new data, and these are done without human intervention. Deep learning (DL) is again a subtype of ML and these algorithms help in autonomous learning by feeding mass quantities of unstructured data, like text, photos, and video. AI was developed with the goal of explaining human intelligence in a way that would make it simple for a machine to replicate it and perform challenging tasks. AI’s goal is to mimic human cognitive

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

76  Artificial Intelligence for Autonomous Vehicles

functions. When it comes to specifically characterizing various processes like learning, thinking, and perception, academics have made great strides in this area. Few ideas are emerging that new systems will be created that are much better than learning and understanding of humans. But still, few are standing on the point that cognitive processes involve value judgment and these come from human experience. Others, however, continue to hold this view, since value judgments are a part of all cognitive processes and are influenced by human experience [5].

4.1.2 What are Autonomous Vehicles? In a self-driving car, a variety of in-car technologies like sensors, GPS, anti-lock braking system, and radars are used [6]. The sensing capability of its surroundings enables an autonomous vehicle (AV) to drive by itself and carry out analytical tasks without the intervention of a human. To respond to the environment like a human driver, AVs use a complete automated driving system [7]. Siegfried Marcus created the first vehicle in 1870. That was just a wagon that had an engine and did not have any steering wheel or brakes. The legs of the driver were used to control it. To convert conventional vehicles into autonomous ones, it nearly took more than one step. In 1898, the first initiative was taken. Operating the vehicle using a remote control was the prime idea behind it (Nikola, 1898). From this stage and because computers became sophisticated and powerful, contemporary automobile functions were transformed to automatic ones that did not even require remote control. The vehicles that were able to change gear without the driver’s assistance were named automatic automobiles (Anthony, 1908), but today, there are vehicles that can travel completely on their own, despite the fact that in many places of the world, they are still not allowed to drive on public roadways. These automobiles are referred to as “driverless cars” or “autonomous vehicles” [8]. A new type of infrastructure is being developed for autonomous vehicles. This technology interests manufacturers of automobiles, electronics, and IT services, and academic research has greatly influenced the development of their prototype systems. For example, Carnegie Mellon University published one important work. 1. AVs are not systematically arranged despite this tendency. 2. Commercial vehicles shield their in-vehicle system interface from users, making it difficult for third parties to test new autonomous vehicle components.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   77

3. Additionally, no two sensors are the same. Some cars might just use cameras, while others might mix cameras, milliwave radars, laser scanners, and GPS receivers [9].

4.2 A Study on Technologies Used in AV All forms of autonomous cars need sensors because they can offer the information needed to comprehend the environment and, as a result, support decision-making. Sensors are essential parts of autonomous vehicles [24]. IoT is crucial to these cars. For example, connected cars are focusing on automation of internal functions of automobiles. These cars work by identifying the voice.

4.2.1 Artificial Vision Popular technology known as artificial vision has been employed for years in fields like surveillance, industrial inspection, and mobile robotics. This technology has intriguing factors like affordable sensors for the most common types and a variety of information types like spatial, dynamic, and semantic (information about the meaning of words) (shape analysis). The industry offers a broad variety of camera configurations, including sensor size, frame rate, resolution (from less than 0.25 to more than 40 Mpx), and optical specifications.

4.2.2 Varying Light and Visibility Conditions People can drive throughout the day or night. The application of dependable artificial visible algorithms is hampered by dark areas, shadows, glares, reflections, and other factors. Some of these issues can be resolved by expanding the capturing spectrum. Cameras with a far-infrared (FIR) range of 900–1,400 nm are efficient for detecting animals and people in the dark, through smoke and dust. The visible spectrum is complemented by near infrared (NIR), which has a stronger contrast in settings with a high dynamic range (HDR) and better nighttime visibility (750–900 nm) [10].

4.2.3 Scenes with a High Dynamic Range (HDR) In scenarios with a High Dynamic Range (HDR), such as when entering or exiting a tunnel, both dimly lit and brightly lighted areas can be visible in the same frame. Most sensor systems have single shot dynamic ranges

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

78  Artificial Intelligence for Autonomous Vehicles

between 60 and 75 dB, which causes information loss at the extremes (underor over exposure). In 2017, Sony released a 120dB automotive sensor with 2K resolution in 2017. Analyses are performed using an HDR and NIR capable automotive grade sensor. When photographing sport scenes [11], a sensor with a 130/170dB range (global/rolling shutter configurations) is given, which offers a more thorough investigation of camera and sensor problems.

4.2.3.1 3 Dimensional Technology Although most vision sensors used in conventional cameras are 2 Dimensional, some of them are capable of sensing depth information. While not usually aimed at the automotive market, this section highlights the three main types of currently available commercial devices. • Stereo Vision: The evident movement of perceptible attributes of images taken by the two precisely calibrated monocular cameras pointed toward a similar direction and spaced apart results in the calculation of depth (known as baseline). The capacity of stereo vision systems to produce rich depth maps rather than sparse sensors is one of their major advantages (e.g., LiDARs). Low-textured patterns, such as solid colors, might make it harder to establish correspondences between frames, which is one of the limitations of stereo vision. The functioning of a single monocular camera shifts an artificial baseline between subsequent frames, which is employed in monocular SLAM (simultaneous location and mapping) algorithms to evaluate depth and camera motion. Numerous works present a good substitute for stereo sensors for positioning and mapping. • Structured light: Monocular cameras attached to a system that casts a familiar pattern of infrared light over the scene. The camera picks an evident distortion of the light pattern caused due to irregular surfaces and is converted into a depth map. Due to the fact that they are less expensive to compute and do not rely on textured surfaces, structured light devices are able to circumvent some of the drawbacks of stereoscopic systems. However, they both require the same extremely accurate calibration as well as their operative range (often under 20 m) is constrained due to the strength of the emitter and the brightness of the surrounding light. Its performance can be impacted by reflections.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   79

• Time-of-flight: It is a form of active sensing that operates on the identical round-trip time theory that an infrared LED emitter floods the image with changed light, which, after being reflected by numerous things in the surroundings, is then captured by the sensor. The shift of phase of the approaching light that is converted to a distance can be used to determine the duration of each pixel’s round-trip. In contrast to the lower divergence laser emitter used in LiDAR, using a non-directed light source has benefits such as capacity to produce high refresh rate (above 50 Hz) and detailed depth maps. For automobile applications, meanwhile, its operative range is limited (10–20 m), and it struggles to operate in bright environments. Avalanche photodiodes, pulsed light time-of-flight, and indirect time-of-flight are some study areas that could extend operating in 50–250 m in range.

4.2.3.2 Emerging Vision Technologies When the sensor elements (pixels) detect a change in light intensity, they are triggered asynchronously and independently in event-based vision. In order to create a picture that resembles a frame, the sensor generates a stream of events that can be divided into time frames. The sensor’s dynamic range is increased to 120 dB by its independent sensor parts, enabling highspeed applications in low light. Despite the sensor’s ability to operate at sub-microsecond timescales, it illustrates tracking at 1,000 FPS under typical indoor lighting conditions. Events could be used as an input for apps for visual odometry as well as SLAM, freeing CPU from laborious operations on raw photographs. Light polarization sensors, which consistently function under challenging weather circumstances and provide unusual sorts of information, are the subject of active study.

4.2.4 Radar The round-trip time principle, refers to the amount of time that it takes for a wave to reach an item, bounce off of it, and then return to the sensor, is used by radar technology to calculate the distance to objects. The majority of contemporary car radars employ digital beamforming to regulate the direction of the radiated wave and are based on frequency-­modulated continuous wave (FMCW) technology. A well-known stable signal with

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

80  Artificial Intelligence for Autonomous Vehicles

another continuous signal that modulates it with an up-and-down frequency variation is what makes up FMCW (typically using a triangular shape). The difference in frequency between the emitted and reflected signals is used to calculate distance. Radars also use the Doppler effect to directly observe the target’s relative speed in relation to the sensor. The independence of radar sensing from light and weather conditions is one of the most compelling justifications for its inclusion in driverless vehicles. It operates in the dark and makes virtually equally accurate snow, rain, fog, or dust detections. In extremely challenging situations, where no other sensor can function, long-range radars will see 250 m.

4.2.4.1 Emerging Radar Technologies High-resolution radar imaging for autos is one of the most active research fields. A greater resolution can obtain more semantic information and enable other applications like target categorization as well as environment mapping in addition to gains in target tracking and object separation [12]. A rotating radar of 90 GHz is installed in the car roof and is used as an illustration for mapping the surrounding area, which includes cars, stationary objects, and the ground. The purpose of this study is to demonstrate the practicality of radars operating between 100 and 300 GHz by analyzing the air absorption and reflectance of materials frequently seen in driving environments. High resolution radar imaging is made possible by meta-material based antennas for efficient synthetic aperture radars. Based on the technology, certain producers, like Metawave, are starting to offer goods geared toward the automotive sector.

4.2.5 LiDAR The active ranging technology known as LiDAR (light detection and ranging), uses the laser light pulse’s round-trip time to compute distances to objects. Low-power, non-visible, and safe for the eyes NIR lasers (900–1,050 nm) are used in robotic and automated applications’ sensors. Due to their low divergence, laser beams can estimate distances of up to 200 m in direct sunlight by decreasing power degradation with distance. The laser pulse is typically changed direction using a spinning mirror to provide 360-degree horizontal coverage. Commercial methods create numerous vertical levels using a variety of emitters (between 4 and 128). A 3D point cloud that represents the surroundings is created as a result. Because of their high accuracy in detecting distances, LiDAR sensors are a feasible alternative for creating precise digital maps. Inaccuracy with these sensors typically ranges from a few millimeters to 0.1–0.5 m in the worst circumstances.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   81

4.2.5.1 Emerging LiDAR Technologies FMCW LiDAR [13] continually produces light to measure an object’s speed using the Doppler effect. Few research prototypes appropriate for the automobile market have begun to surface in recent years. Speed observation can aid to increase activity recognition and behavior prediction in addition to target-tracking abilities, for example by detecting the different speeds of a cyclist’s and a pedestrian’s body parts. A solid-state oscillating micromirror and optical phased array are two of the technologies that fall under the general term “LiDAR,” which covers several more optical phased array (OPA). The first method uses tiny mirrors that can revolve around two axes to focus laser beams. Devices based on this technology are offered for sale by the manufacturer LeddarTech. Similar to the technique used in phased array radars, optical phased arrays provide rapid and very accurate beam direction control. One of the few companies commercializing products based on this concept is Quanergy. Over the entire FoV, OPA technology can use ­random-access scan patterns (field of view). This enables dynamically changing beam density (resolution) and monitoring only particular locations of interest. Combining these qualities enables quick inspection of the entire field of view at low resolution, followed by following objects of interest at greater resolution for improved shape recognition even at great distances [14].

4.3 Analysis on the Architecture of Autonomous Vehicles To improve autonomous vehicles day by day, engineers are exploring and employing more advanced sensors and technology in both software and hardware. The traditional idea for autonomous vehicles has up until now made use of cameras, radar, ultrasonic sensors, and LiDAR. Each subsystem no longer performs its individual tasks independently of the others. The results of one task must be used as relevant information for the next; as a result, autonomous vehicles must be capable of carrying out a wide range of activities in various scenarios. We achieve the best results when hardware and software collaborate and use each other’s finest capabilities to complete a task. An autonomous system has a number of components and its architectural design that offers an abstract perspective of how the system is structured and operated. A layered design allows for the sharing of information between components via public well-defined communication interfaces. By adjusting a few components to maintain the same communication interface, this characteristic makes it possible to define a similar architecture for all tracks in

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

82  Artificial Intelligence for Autonomous Vehicles

this challenge. This method shortens the time required for agent development and makes it possible to compare the effectiveness of autonomous navigation using several algorithms for a given job. The most prevalent hardware and software architectural structures will be covered in this section.

4.3.1 Hardware Architecture Hardware components are essential for enhancing the safety and redundancy of self-driving systems. These hardware elements are used in combination to conduct automated driving operations. Driving assistance systems are designed to increase the driver’s comfort and the safety of the passengers and the surroundings. • Environmental detector configuration: Self-driving vehicles cannot depend on the singular type of detector; they must include a redundant sensor for safety. The most common sensors for a self-driving automobile include cameras, radars, LiDARs, and GPS. Another common sensor setup for self-driving cars comprises a millimeter-wave radar, a GPS unit, a 16-line laser radar, and other similar sensors. • Vehicle-mounted computing platform: Two of the Industrial Personal Computers (IPCs) compose the computing environment for autonomous cars. To meet automotive gauge specifications, the IPC must be able to operate for an extended period of time in conditions of high temperature and vibration. The hot standby IPC is used instead of the primary control IPC. They both perform online in real time. To ensure safety, the standby IPC output command is switched as soon as the primary control IPC fails. • Actuators and Sensors: One of the most crucial parts of self-driving automobiles is the actuator, followed by the sensor. They are essential to preventing the need for human interaction while traveling. To gather information from the environment, a lot of sensors will be required. The numerous actuators will be turned on by sensors, which will then produce the instruction to turn on the last component [15].

4.3.2 Software Architecture The software architecture of the intelligent vehicle includes public service support and user interfaces for computers. The most crucial aspect of self-driving cars is the communication between components.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   83

• Human–machine interaction: This covers remote control intervention, driver interface, and programmer debugging. Emergency response is one of the most important applications for autonomous vehicles. Autonomous automobiles are capable of lane keeping, lane changing, overtaking, and avoiding emergency accidents. The intelligent vehicle’s hardware and software configurations are used to evaluate the automated driving capacity of the intelligent vehicle in various modes. • Public Service Support: This includes services for logging, process observation, and virtual switching. The virtual switch should logically be made up of many virtual buses. Each and every bus has several software modules connected to it. Communication across modules on the same bus is accomplished via the subscription and release mechanism. The virtual switch eliminates the communication function of the module, allowing it to concentrate on its functionality. The statement on the digital bus is logged by the log and sent to the database in accordance with the time sequence. The messages may also be delivered back to the application layer module that needs debugging, bringing back the test's observed driving state. This depends on the time. Data about heartbeats are gathered and analyzed through process monitoring. When exceptions are found, the system immediately takes corrective action and notifies the debugging team so they can examine the system. The System Architecture for Autonomous Vehicles involves five stages. They are discussed below. • Sensing: The sensing layer consists of a collection of exteroceptive and proprioceptive sensors, which measure information about the outside world and the vehicle itself, respectively. In order for the system to reliably reflect the status of the vehicle and its surroundings, the perception layer must understand and transform the raw data from the sensors that the sensing layer components collect. A layer for sensor management is a valuable tool for architecture design, since it increases flexibility and scalability, allows for the replacement of individual layers or even their entirety, and ensures that the architecture will continue to function as long as its communication interface does. If the set of sensors is altered, there is little to no change in the architecture design.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

84  Artificial Intelligence for Autonomous Vehicles

1. GPS In a geodesic coordinate frame, which is defined as: 2.

3.

4. 5.

i. pgeo k = (lat k,lon k,alt k),

where lat and lon stand for latitude and longitude, alt stands for altitude, and k is the time stamp of the frame. This sensor delivers the position of the vehicle at a frequency of 10 Hz. Camera Six cameras are arranged into three stereo pairs, positioned at a height of 1.8 m. The stereo right and stereo left cameras’ picture sizes are 600 × 320 (width × height), while the center camera’s image size is 1,080 × 540 (width × height). LiDAR This sensor produces about 500,000 points per second, or a point cloud, with a 360-degree horizontal field of view at a frequency of 20 Hz. It uses 32 laser-simulated channels with 45 degrees of vertical field of view (15 degrees of upper FoV and 30 degrees of lower FoV). Each point is described by position (x, y, z) with relation to the location of the sensor in a Cartesian coordinate system. CAN Bus This proprioceptive pseudo sensor offers data on the interior status of the car, including speed, steering angle, and shape dimensions. Object Finder This fictitious sensor provides data on the location and orientation of all moving and stationary objects such as pedestrians and vehicles as well as information on the shape and status of traffic lights that are a part of the simulated environment.

• Perception: During this step, the automated vehicles sense their environment by a lot of sensors and determine its location in correspondence to those surroundings. It also includes LiDAR, radar, real-time kinetics, cameras, and others, processing the data as soon as data are supplied to the recognition modules from sensors. A control system, Vehicle Positioning and Localization (VPL), Unknown Obstacles Recognition (UOR) modules, etc., are the typical components of an autonomous vehicle. The components of the perception layer use methods and theoretical definitions from several academic domains, including computer

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   85

•

•

•

•

vision, linear algebra, probability, and machine learning to transform sensor input into meaningful information. These data permit the representation of the vehicle state and surrounding environment in order to feed the navigation layer components in the architectural design. The most important tasks for automated vehicles are detection of road borders, detection of obstacles, identification of pedestrians and vehicles, detection of traffic signs, detection of lanes, detection of traffic lights, and estimation of vehicle body state. For identifying obstacles, the support vector machine method is frequently employed. The main applications of convolutional neural network techniques in the detection of pedestrians and vehicles and classification and identification of that rapid convolution that occurs quickly Traffic signs are recognised using neural network technology. Obstacle Detection: CaRINA Agency uses two vision systems: a stereo camera-based system and a LiDAR system with perception capabilities for Track 1, Track 2, and Track 3. They execute the obstacle identification task and provide three-dimensional point clouds in a Cartesian coordinate system (x, y, z). LiDAR-Based Obstacle Detection: Contrary to commercial LiDAR, the simulated sensor provides neither intensity nor ring data because it is a ray-casting approach. Therefore, strategies relying on ring compression and virtual scanning are not usable during competition. Stereo-Based Obstacle Detection: Driving in urban environments necessitates precise and accurate 3D perception of the surroundings. LiDAR was a useful sensor for this work on Tracks 1 and 3. Track 2 is instructed to travel to a specific location using only cameras. Beyond LiDAR, there are no sensors in the challenge that can produce RGB-D images or depth data. We performed scene reconstruction and obstacle recognition using a stereo system. The stereo method was chosen because it allows for 3D reconstruction using just two monocular cameras equipped with the right calibration matrix. Hazardous Obstacle Monitor: Risk assessment is yet another crucial element of an autonomous system for ensuring safe driving, in addition to obstacle detection. It analyzes potential risks to accidents, such as collisions with other traffic participants. Collisions with stationary impediments,

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

86  Artificial Intelligence for Autonomous Vehicles

•

•

•

•

•

moving obstacles, and unforeseen obstacles are the three basic categories into which collisions fall (which may occur due to occlusions). Therefore, both binary collision prediction and quantitative risk appraisal can direct decision-making systems toward the proper course of action for collision avoidance. Decision and Planning: This stage makes decisions, plans, and directs the movement of the autonomous vehicles using the information received during the perception process. This stage, which the brain would represent, is where choices are made on things like obstacle avoidance, action prediction, etc. The decision is based on current and historical information, including real-time map data,traffic contains details and patterns, user data, and so on. There could be a data logging module which keeps track of mistakes and data for later use. The primary responsibility of the driving mission choice is to determine the intelligent vehicle’s driving style, switching lanes, catching up, making a left turn at a crossroads and a right turn at a junction,from starting and stopping at junctions. Local route planning’s primary goal is to create a suitable path for each selected driving style, which is typically between 10 and 50 meters long. In order to produce the steering wheel angle and move laterally along the specified track, lateral motion control is primarily responsible for these tasks. Control: This module controls the autonomous vehicles physically by performing tasks such as steering, stopping, and accelerating after receiving information from the planning and decision module. Lateral Control: The model-based predictive control (MPC), which manages the lateral control that provides the engineering signal, optimizes a cost function over a preset time horizon H, producing a series of actions, one for each time step ∆t. The immediate action is carried out, and the process is continued in the following time step, resulting in a narrowing of the horizon. Longitudinal Control: The agent must have a speed rate change as the Markov decision processes (MDP) problem’s solution. A new agent’s velocity can be computed using the current agent’s velocity. PI controls, or proportional-­ integral, perform this tracking. Chassis: Last step is about making an interaction with the mechanical parts that are connected to the chassis, including

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   87

the gear system, brake motor, steering wheel motor, and accelerator and brake pedal motors. The control module signals manage these. • Sensors: We discuss the design, functionality, and utilization of some key sensors after discussing the overall communication and detector architecture of an automated vehicle. 1. Ultrasonic sensor: These sensors operate at frequencies ranging from 20 to 40 kHz. A magneto-resistive tissue used to determine the distance between two objects produces these waves. By comparing the emitted wave’s time-of-flight (ToF) to the echoing signal, the distance is determined. The range of ultrasonic sensors is often less than 3 m, which is quite short. Every 20 ms, the sensor output is refreshed, which prevents it from adhering to the ITS’s rigorous QoS requirements. These sensors have a very limited beam detecting range and are directional. Therefore, to obtain a full-field vision, numerous sensors are required. Multiple sensors, however, will impact one another and can result in significant ranging errors. The conventional approach will provide a unique signature or identifying code that will be required to eliminate the echoes of other detectors working in close proximity. • Radio Detection and Ranging: RADARs are used in AVs to scan the environment for cars and other objects. RADARs are frequently used for both military and civilian purposes, such as airports or meteorological systems, and they function in the millimeter-wave (mm-Wave) frequency range.Various frequency bands, consisting of 24, 60, 77, and 79 GHz, are used in contemporary automobiles and have a measurement range of 5 to 200 m.By figuring out how long it took from the transmitter and the obtained echo, the distance between the AV and the object is determined. To increase range resolution and the ability to detect numerous targets. RADARs in AVs employ collections of micro-antennas that produce a series of lobes. Because mm-Wave RADAR can directly measure short-range aims in any direction by utilizing Doppler shift variation,it has greater penetrability and greater bandwidth. Since mm-Wave radars have a longer wavelength, they feature anti-pollution and anti-blocking capabilities that enable them to operate in fog, rain, snow and low light.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

88  Artificial Intelligence for Autonomous Vehicles

• LiDAR: Light Detection and Ranging: The spectra at 905 and 1,550 nm are used in LiDAR. Modern LiDAR operates in the 1550 nm wavelength range to minimize retinal degeneration due to the 905 nm spectrum that may harm the human retina. LiDAR can operate at a distance of up to 200 m. LiDAR is divided into 3 categories: 2D, 3D, and solid-state models. A single laser beam is dispersed over a fast-rotating mirror in a 2D LiDAR. Several lasers can be placed on the pod so that a 3D LiDAR can produce a 3D image of its surroundings. • Camera: Depending on the wavelength of a device, shutter in autonomous vehicles can be categorized as infrared light or visible light. The camera’s image sensors are constructed using two technologies: charge-coupled devices and complementary metal oxide semiconductors. Depending on lens quality, a camera can capture images out to a maximum range of about 250 m. The visible cameras have red, green and blue bands (RGB) and work within the same range of wavelength as the human eye, or of 400–780 nm. There are two VIS cameras coupled with fixed focal lengths that produce a new channel with depth D data, mandatory for producing stereoscopic vision. Using this function, cameras (RGB) can get a 3-dimensional view of the environment of the vehicle. Passive sensors ranging from 780 nm to 1 mm in wavelength are being used by the infrared (IR) camera. The vision control is provided by IR sensors in autonomous vehicles in peak illumination. This camera assists autonomous vehicles in their works like detecting objects, side view control, recording of accidents, and blind spot detection (BSD). Nevertheless, adverse weather circumstances like snow and fog and variations in the quality of the light affect how well the camera performs [16]. • Global Navigation Satellite System (GNSS): The most popular technology for obtaining precise position data on the Earth’s surface is GNSS. The GPS is a US-owning utility that offers customers with aligning, precise site, and time frame services and is the most well-known GNSS system. GPS is a vital component of the global information architecture, which pervades all aspects of today’s society because of its free, open, and reliable nature. The US DoD (Department of Defense) created the GPS in the 1970s and has three sections:

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   89

space, control, and user section. The US Air Force created, maintained, and ran control components and GPS system’s space. Operational satellite count for the space segment is 31, at least 24 of which are typically available. While in medium Earth orbit (MEO) at the height of 20,200 km, each of these satellites completes two daily orbits of the planet. Any receiver placed on the surface of the planet can receive signals from 6 to 12 satellites in the S–band and L–band fluctuations range because of its configuration. The method for separating into separate groups based on shared features is referred to as user segmentation [17]. It is important to note that GNSS signals have a number of flaws that reduce the system’s accuracy, including the following: (1) timing inaccuracies resulting from discrepancies between the receiver’s quartz clock and the satellite’s atomic clock, (2) signal lags brought on by signal propagation through the troposphere and ionosphere, (3) multipath impact, and (4) uncertainties in the orbit of a satellite. Inertial Measuring Units (IMU), radars, cameras, and LiDARs, among other sensors, are coupled with satellite data to create a more comprehensive picture to produce trustworthy location data in order to enhance the precision of today’s car positioning systems.

4.4 Analysis on One of the Proposed Architectures A software architecture is suggested for the simulation of in a traffic scenario, an autonomous vehicle. As a result of the requirement that each simulator utilize all available resources in Figure 4.1 [18]. Figure 4.1 design has a distributed nature. There are four main modules: Microscopic Traffic Simulator, Robotics Simulator, Coherent Network Data and Autonomous Vehicle interface and control. • Microscopic Traffic Simulator It simulates nearly every type of vehicle in a manner that is similar to the macroscopic modeling of actual traffic flows. Additionally, it keeps up with all construction systems, such as induction loops and traffic signal schedules. Because of the simulator’s adaptability, a pretty high level of statistical template could be linked to examine the patterns of traffic behavior of autonomous vehicle strategies for individuals and groups.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

90  Artificial Intelligence for Autonomous Vehicles

Autonomous driver agent

Autonomous driver agent

Robotics framework

Robotics framework

Robotics Simulator Game engine

Traffic network data

Microscopic Traffic Simulator

Figure 4.1  An autonomous vehicle simulator in a traffic environment is proposed.

• Robotics Simulator It simulates every autonomous vehicle in the surroundings, including every one of its sensors, actuators, and reflectors of everything in the immediate vicinity. Additionally, it has a game engine with strong physics and visuals modules for immersive 3D animation. • Coherent Network Data It represents the topology of the traffic network and the data for its accurate 3D surroundings. • Autonomous Vehicle Interface and Control When using an agent-based methodology, an external software programme motive force is frequently used to carry out high-level tasks for self-sufficient cars. For smooth real/ digital worldwide development and sensor/actuator permutation, a hardware abstraction layer (HAL) is used. • Virtual Server for Data Transmission Data transfer is essential. As a result, the data transmission module needs to be reliable, secure, effective, and bug-free. Each module involved is seen as a node connected, with no impact with other nodes, when the related node is idle. The other modules will not submit any data to module A if A just requires data from module B. Efficiency should be given the most consideration because a transmission delay is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   91

detrimental for automated driving. Though the ROS (Robot Operating System) is a famous system utilized for robotics, testing has revealed that due to its cross-platform capability and real-time performance, this system is not particularly ideal for self-­driving vehicles when the related node is idle. Additionally, decoupled nodes in their IPC (“Cocktail”) increase system availability in the event of partial failure. However, a destination-based model is employed rather than a channel-based approach. The virtual server that connects to all the nodes and relays all the messages carries out the connection checking. So, a data transmission module for self-driving vehicles called “Project Cocktail” was created as a result to address these issues. A crucial mechanism for data transmission and sharing, Project Cocktail may receive and send messages to and from network peers (using user datagram protocol). Project Cocktail, albeit created for this particular application, is a general-purpose intermediate that treats all flow of data as simple binary creeks and is not dependent on any specific data format used by network peers. Because of the way it is built, it is very resilient in the sense that it functions well when a large diversity of data types is created by a large number of network neighbors. Testing on this single server design demonstrates that it is less scalable than the distributed SimpleComms design. Compared to the “singleton of data transferred, the ‘all to all’ mode single,” controlling for the net amount has a higher latency, indicating that “it is best to avoid listening to the same module twice.” SimpleComms incorporates crowding avoidance for each individual module by distributing the message-relaying workload across the local servers.

4.5 Functional Architecture of Autonomous Vehicles In this section, the functional infrastructure for self-driving vehicles that has resulted in our research is summarized here. It combines all of the functional elements already mentioned and distributes them among the vehicle platform and cognitive driving intelligence components. Additionally, this adheres to our suggestion of achieving a reasonably clear separation between the two. Some elements for technical and practical considerations,

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

92  Artificial Intelligence for Autonomous Vehicles

both the vehicular platform and the driving intellectual ability, are assigned power management and diagnostic testing. However, the duties and area of operation are slightly varied for each allotment. Although conceptually united, the capable sensing and global model aspects are divided within those that deal with the vehicle’s exterior surroundings and those that deal with the ego vehicle platform. When the functional infrastructure is refined to become a technical infrastructure via ISO 26262 method, the separation makes it easier to establish different technical implementations, if necessary. As demonstrated, depending on how much operation and fusion is necessary, the sensing components’ outputs are sent either directly or indirectly to remaining perception, decision, and control components. Creating a data connection between localization and sensor fusion is beneficial. At set points along predetermined routes, some sensors may show repetitive trends such as an increase in false positives or dropouts. An intriguing area of research involves altering a sensor’s degree of confidence based on geographic location, and the design should not be a stumbling block. The relationship between the sensor components and the semantic understanding component is another intriguing data link. There are three situations where this is helpful. First, so-called focused attention techniques are advantageous in some specific autonomous driving circumstances. Focused attention entails delving deeper into a certain area of the surroundings. This might call for the sensors to physically move or for configuration adjustments (such as altering the zoom on a lens or moving or adjusting a camera’s field of view). Most autonomous cars’ sensors are now physically bound to a fixed attitude with regard to the vehicle coordinate system. However, it is relatively typical in the area of mobile and intellectual robotics. To have, for example, a pan-tilt-zoom image sensor to help the robot in a search experiment. Second, sensor adjustment during runtime adjustments may be required (for example, changing levels of exposure based on the time of day, provoking recalibration if physical adjustment changes are suspected). Third, the semantic understanding component can use communication transceivers as a type of sensor or actuator to react to arriving communication requests, post ego vehicle data, and request asynchronous details. Such communication requirements are frequently a critical component of cooperative driving scenarios, in which a vehicle is constantly in communication with the surrounding buildings and other neighboring automobiles. Energy management from the viewpoints of mission accomplishment and total vehicle energy demands is one of the decision and control components (internal and external lights, HVAC). In comparison, the vehicle platform’s component of energy management controls regenerative braking, the integration of hybrid propulsion systems,

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   93

and in a portion of electric and hybrid vehicles, charging management and cell load balancing. Since our perception technical implementation decision and control system have been slow to respond to unexpected events as aspect of the deliberate control, the reactive command in this design is dedicated to the vehicle platform. Additionally, having reactive command inside the car, it is easier to ensure a minimal amount of self-protection for the car’s platform if the perceptual driving system fails or becomes completely incapacitated. The vehicle platform also includes the control components for passive safety features like airbags, seat belt pretensioners, and other features that are tightly tied to and it is unlikely that it will be easily repurposed in other vehicle platforms. Due to the limited space in this work, the connections between the functional parts have not been depicted on the automobile platform. Platform stability, responsive control, and motion control actuator abstraction are all features seen in more recent vehicles. As a result, the vehicle platform’s uniqueness is less than the driving intelligence. However, it is crucial to emphasize that the “Propulsion/ Steering/Braking” component abstracts the actual actuation systems [19].

4.6 Challenges in Building the Architecture of Autonomous Vehicles Automated vehicles are now a reality after more than 50 years of continuous development and research. Several challenges remain in the development of a completely autonomous method for driverless vehicles. We go over the problems that are frequently raised in research on AI in AV in this part. The many benefits that AV offers are highlighted by research on the subject. No real-world testing is done in pedestrian detection to see how well the suggested approaches identify things in real-time. Heavy impediments can sometimes cause a pedestrian’s orientation to differ from another’s image mask, which can lead to inaccurate orientation assessment. Therefore, no algorithm is entirely precise or quick. When spotting pedestrians in the dark, accuracy and quickness must be traded off. Behavioral forecasting of pedestrians is frequently disregarded. The majority of research articles on trajectory identification in trajectory planning relied solely on simulation or presented challenges to solutions utilizing deep learning algorithms, with little to no real-world demonstration of their technique. The papers that did use real-world methods are now out of date. The Model Predictive Control method is the primary algorithm used for lateral motion control in motion control. It nevertheless has a limited

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

94  Artificial Intelligence for Autonomous Vehicles

ability to detect faults, and uncertainties that do not fit the specified conditions are not completely eliminated. The literature search was constrained because there are few studies on transparency in self-driving cars and no real-world implementation for nonfunctional criteria. The cognitive bias increases, as it is more likely that people will be impacted by fatal crashes that include various types of driverless cars rather than the new novel experiences that automated vehicles can provide due to the rapidly changing nature of technology.

4.6.1 Road Condition The condition of roads could be little surprising and may shift over time. In several places, there exist massive, seamless, well-marked roads. In further cases, there are no markings on roads and the road is gravely decayed. Roads are poorly drafted, there are pits, highlands, and underpass paths, and equivalent conditions prevail on par to external direction signs.

4.6.2 Weather Condition The weather is another troublemaker. Weather can be cloudless, or it can be wet, windy, and rainy. Autonomous cars should be capable of driving in all meteorological states. No room for mistakes or shutdowns.

4.6.3 Traffic Condition Self-automated vehicles should adapt to different traffic scenarios on public roads. They would have to navigate a large number of pedestrians while sharing the roadway with some other driverless vehicles. Most emotions are present everywhere humans are involved. It is possible to have traffic that is heavily controlled and autoregulated. However, there are many times when anyone might be overriding the law quite often. Unexpected things may result in the exploration of an object. When there is heavy traffic, movement of even a few centimeters each minute counts. A traffic jam may form if a larger number of such vehicles are on the roadway hoping for traffic jams to move.

4.6.4 Accident Responsibility The crucial feature of autonomous vehicles is their accident liability. Who is to be blamed for crashes involving autonomous vehicles? While talking about self-driving automobiles, the operating system will be the key element

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   95

that controls the car and assists in making all significant decisions. In contrast to the original prototypes, which depicted a person actually seated behind the wheel, Google’s most recent concepts omit a steering wheel and dashboard. How does a person inside the vehicle intend to manage the car in the sudden situation when the car does not appear to have any controls, such as a brake pedal, steering wheel, or an accelerator pedal? The passengers and drivers of AVs will often be relaxed and may not pay close attention to traffic problems due to the nature of these vehicles. If they need to focus on something, by the time they have to answer, it might be too late to stop it.

4.6.5 Radar Interference Radar and laser beams are used by automated cars to navigate. While the lasers are mounted on the roof, the sensors are mounted on the body of the vehicle. Radar works by identifying radio wave reflections from nearby moving objects. While driving, a car continuously emits radio frequency waves that are reflected off of other adjacent objects and by other cars. To determine the distance between the car and the object, the reflection’s time is noted. On the basis of the radar data, the appropriate action is then conducted. By identifying radio wave reflections from nearby objects, radar operates. On the basis of the radar data, the appropriate actions are then done. Radar operates by spotting radio wave refractions from nearby moving objects [20].

4.7 Advantages of Autonomous Vehicles Self-driving automobiles, which are highly automated technology, provide a number of potential advantages. • Improved road safety Automated systems could minimize the number of crashes in the roadways. As per government data, 94% of accidents are caused by the actions or missteps of drivers. AVs can help minimize errors by drivers. Drivers with greater autonomy may be less likely to be involved in dangerous and hazardous driving practices. The best hope is that distracted driving, accelerating, intoxicated driving, unbuckled car occupants, and stoned driving will be lowered.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

96  Artificial Intelligence for Autonomous Vehicles

• Enhanced Independence Personal freedom is increased with full automation. People who are blind are self-sufficient, and heavily automated cars can help them live the lives needed by them. Seniors’ freedom could be improved by these cars. Highly autonomous vehicle (HAV) ride-sharing can make personal transportation less expensive and increase traversal. • Saving cash Automated driving methods may have a number of financial implications for us. HAVs can help to reduce collapse-­ related expenditures such as medical costs, productivity loss, and car repairs. Less accidents may result in lower insurance prices. • More Efficiency The widespread use of HAVs might enable time travel for drivers. In the upcoming years, HAVs may make it easier to drop off travelers at their desired location, whether it's an airport or a shopping complex, while the vehicle itself parks. In a totally automated car, all passengers could safely take part in more useful or enjoyable tasks such as responding to emails or watching a film. • Less Congestion AVs could help with a variety of traffic congestion problems. When there are fewer collisions or fender benders, road backups are lowered. By maintaining a safe and consistent spacing between vehicles, HAVs help to reduce the frequency of stop-and-go waves that cause road congestion. • Gains for the environment AVs have the ability to reduce carbon emissions and consumption of fuel. Fuel is saved by lowered traffic, and HAVs reduce emissions of greenhouse gasses. Automation and carpooling may increase supply for all types of electric vehicles. The cost-effectiveness of electric vehicles is increased when the vehicle is used for longer hours each day [21].

4.8 Use Cases for Autonomous Vehicle Technology In a world where vehicle automation technology is widely used, there may be less traffic, safer roads, and linked cars that let drivers relax and enjoy the journey. The market for AV technology is expanding swiftly and is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   97

predicted, by 2026, to be $556.67 billion USD. The sector still has a way to go though. Technology for autonomous vehicles needs to cooperate with various fields in order to perform successfully. The following list includes the top 5.

4.8.1 Five Use Cases • 5G It is anticipated that an autonomous car will produce 2 Petabytes of data annually. If the greatest Wi-Fi was available, it would take months to transfer that much data. 5G speeds are 10 times quicker than 4G speeds and are practically real time. The future of self-driving vehicles is already achievable. • Latency Reduced response time is yet another benefit of the 5G feature, which is advantageous for autonomous vehicles. When it comes to passenger safety, 4G’s current time delay of 50 ms is considered as a significant delay. • Smart cities and IoTs A self-driving vehicle needs knowledge of its surroundings in order to make wise decisions. That is possible in IoTready smart cities. A self-driving automobile can go more shrewdly and easily about town if the city can report on traffic, signals, etc. • Data Management It takes time to examine all of the data that a self-driving car generates. Edge computing can speed up this research by looking at information closer to the source because there could be nearly 10 million new cars on the road. • V2X Data from automated vehicle sensor systems and other source materials can be transmitted over rising, elevated, and moderate channels—thanks to vehicle-to-everything (V2X) technology. Because of the ecosystem it creates, cars can communicate with one another and with infrastructure such as parking spaces and traffic lights. This not only improves car safety but also alerts passengers and drivers to impending roads, allowing them to react appropriately. An automated car would be capable of deciding for itself when paired with artificial intelligence (AI) [22].

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

98  Artificial Intelligence for Autonomous Vehicles

4.9 Future Aspects of Autonomous Vehicles During the last 10 years, the public’s fascination with autonomous cars has fueled collaboration between makers and the IT pioneers. How far we are from integrating automatic vehicles autonomous vehicle (AVs) to the systems for transportation? By 2030, 1 in 10 vehicles will be fully automated globally, according to projections, but until significant obstacles can be overcome, the industry can only make educated guesses. Before driverless vehicles were a common sight on the highways, however, several parts of a very complex puzzle needed to fit together. The buzz surrounding AVs is still being fueled by real-world testing and exciting vehicle projects, but many automakers now recognize that developing technology is much more difficult than they initially believed. Nevertheless, confidence for 5G-enabled AI-powered automated driving technology is on the rise as advances in partially automated vehicles provide a realistic view of what the following decade may involve. What variables will contribute to the development of a self-driving future that have the ability to revolutionize everything from our transportation habits to the future design of smart cities?

4.9.1 Levels of Vehicle Autonomy As a set of standards that serve as a benchmark for capabilities in AV, the Society of Automotive Engineers (SAE) has developed five phases of autonomy of the vehicle. They begin at Level 0 (Manual Balance) and progress to Level 5 (fully independent). With Tesla’s automatic pilot system, which is divided as second level, the driver has to be prepared to take control while the car handles tasks such as steering and acceleration. For several years, Google’s automated driving project Waymo has been escorting passengers around Phoenix using Level 4 autonomy. Automakers such as Ford are conducting a test to see how far autonomous technology can go in a fictitious city with the size of 24 football pitches in the University of Michigan’s Mcity Test. Researchers are reading critical lessons about how

0

1

No Automa tion

Driver Assista nce

2

3

4

5

Partial Automa tion

Conditi onal Automa tion

High Automa tion

Full Automa tion

Figure 4.2  Levels of vehicle autonomy.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   99

autonomous and networked vehicles can be functioned in the controlled testing environment.

4.9.2 Safer Mobility Technology One of the primary goals of the autonomous vehicle industry is to provide a safe and secure journey for travelers, car owners, and bicyclists. Nine out of 10 collisions, as per the National Highway Traffic Safety Administration in the United States, are the output of human mistake. AVs do have the ability to replace humans in the bulk of traffic incidents if the technique is capable of living up to its words; nevertheless, they must initially depend on automated driving vehicles that can perceive better than the greatest human driver on the roadway. Data will obviously be essential for maximizing the potential of AVs. The ongoing progression of safety systems sets the stage for future development of intelligent autonomous systems capable of navigating major roads with minimal or no living creature intervention. ADAS, commonly known as advanced driver assistance systems, which is common in modern automobiles, utilizes sensing devices including such sensor and laser scanners to identify an object and becomes advanced in the new generation. Deployment of AVs will always be successful to the extent that the 5G technologies enable artificial intelligence and observational abilities in self-driving vehicles.

4.9.3 Industry Collaboration and Policy Matters It might take long-term cooperation among government agencies, automakers, tech innovators, telecoms, and others to innovate cars for the future. As competition is advancing in the scene, the disabilities are very costly and complex for any one side to manage on its own. Progression is made in China where there is a very high level of private-public support for autonomous car technologies. By 2030, a journey service in China, Didi, intends to apply more than a hundred thousand robotaxis across its launch pad, and a surge in the commercialization of autonomous vehicles is being fueled by new regulations governing AV development. Consumers are ready for driverless automobiles despite the considerable challenges that lie ahead. When industry participants and experts concur that AVs do have the ability to alter mobility, making predictions of what should happen within commercialization gets harder. If the automotive industry is to advance over the next 10 years, problems with many layers will be overcome in upcoming generations of technological advancements [23].

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

100  Artificial Intelligence for Autonomous Vehicles

In order to usher in an autonomous future, vehicles and other transportation systems are now being researched and developed. A technologically advanced driverless society is on the horizon. Further technological advances will result from these developments in AVs. On the basis of Internet of Vehicle (IoV) and VANETs, numerous applications have already been suggested. Powerful next-generation infrastructures may now be built—thanks to developments in IoV, acting as an interface to link various items to the Internet, including actuators, sensors, and complete vehicles. As smart cities based on IoT are being developed, research is being done on technologies like BCG to disseminate content through cloud support. An IoV-based technology is called VANETs. Its main responsibility is to provide constant connectivity to resources like the Internet. The number of users on the Internet has expanded due to its quick development, and VANETs can be utilized to meet their demands by enabling mobile access to resources and Internet connectivity. It also had potential applications in other industries like intelligent transportation systems, military systems, and health and safety, to mention a few. In order to maintain an adaptable strategy in monitoring traffic and pollutant density, VANET enables us to cluster cars according to routing, mobility, and driving habits. The AVs will be able to choose routes more appropriately with the aid of this knowledge. Once government agencies have legalized vehicle-to-vehicle communication and all vehicles on the road have adapted to this technology, the benefits of using VANET for tracking, navigating, routing, and communication will become more obvious. According to the study, since there are effective cellular or LTE communication channels between roadside devices and the cloud platform, radio frequency identification should be used by moving cars and roadside units for vehicle-to-infrastructure communication. According to the study, it will have a huge impact on the healthcare industry and can also be utilized for vehicle-to-vehicle communication. The paper proposes a novel routing algorithm based on collaborative learning for information delivery to the target, throughput maximization, and delay minimization. In the event that there were more cars on the road and there was network route congestion, this technology would aid vehicular sensor networks (VSNs). The learning automata swiftly decide on a route based on past experience and the nearby access points.

4.10 Summary In conclusion, even if autonomous vehicles appear to be a distant concern for present road users, global tests of vehicles in motion indicate that

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   101

public cars may soon be available. Another revolution will undoubtedly result from these changes in transportation, so it is critical to inform the public about how to handle autonomous vehicles. This instruction should cover proper conduct on the roadways where the vehicles would be able to move. However, it is crucial to educate the general public with the applied nomenclature and the division of vehicles owing to the degree of autonomous driving before cars are fully integrated into the existing transportation systems. Fully autonomous vehicles such as buses, lorries, and cars that can travel across large expanses of territory without the involvement of drivers would revolutionize ground transportation. Accidents and fatalities could be reduced significantly. Humans may utilize the time they spend trapped in traffic to accomplish assigned work or for recreation. The environment might reshape, requiring so little car park while increasing productivity and safety for everyone. Robotic vehicles would seamlessly transport people and products throughout the world on demand, resulting in the emergence of new models of business for distribution commodities and services—the “Physical Internet.” We might also observe human drivers freed from the demands on their attention that come with driving, free to travel far larger distances and jam up the roads and pollute the air. This issue is even more urgent given the negative effects on employment caused by the COVID-19 epidemic. COVID-19 has worsened the disparities in mobility and employment that already exist in cities and has hurt ride-sharing and public transportation. Due to the rise in e-commerce, robotic package delivery is becoming more popular, and more people are now working from home. Safe and effective public transit will continue to be essential for our communities as commuter, educational, and shopping habits shift toward a new normal. More than ever, investments in workforce training are required to guarantee that COVID-19-affected workers can participate in automated transportation systems in the future, however long that future takes to materialize.

References 1. https://en.wikipedia.org/wiki/Artificial_intelligence 2. Hauser, L., Artificial intelligence, Internet Encyclopedia of Philosophy. 3. Thomason, R., Logic and artificial intelligence, E.N. Zalta (Ed.), Stanford Encyclopedia of Philosophy, 2018. 4. Chen, L., Chen, P., Lin, Z., Artificial intelligence in education: A review. IEEE Access, 8, 75264–75278, 2020. 5. https://www.investopedia.com/terms/a/artificial-intelligence-ai.asp

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

102  Artificial Intelligence for Autonomous Vehicles

6. https://www.gartner.com/en/information-technology/glossary/autonomousvehicles 7. https://www.twi-global.com/technical-knowledge/faqs/what-is-anautonomous-vehicle 8. Wiseman, Y., Autonomous vehicles, in: Research Anthology on Cross-Disciplinary Designs and Applications of Automation, pp. 878–889, IGI Global, 2022. 9. Kato, S., Takeuchi, E., Ishiguro, Y., Ninomiya, Y., Takeda, K., Hamada, T., An open approach to autonomous vehicles. IEEE Micro., 35, 6, 60–68, 2015. 10. Pinchon, N., Cassignol, O., Nicolas, A., Bernardin, F., Leduc, P., Tarel, J.-P., Brémond, R., Bercier, E., Brunet, J., All-weather vision for automotive safety: which spectral band?, in: Advanced Microsystems for Automotive Applications 2018, pp. 3–15, Springer, Cham, sep 2018. 11. Pueo, B., High speed cameras for motion analysis in sports science. J. Hum. Sport Exercise, 11, 1, 53–73, Dec 2016. 12. Reina, G., Johnson, D., Underwood, J., Radar sensing for intelligent vehicles in urban environments. Sensors (Switzerland), 15, 6, 14661–14678, June 2015. 13. Nordin, D., Optical frequency modulated continuous wave (FMCW) range and velocity measurements. Thesis, Optical Fiber Communication Conference, 110, 2004. 14. Marti, E., De Miguel, M.A., Garcia, F., Perez, J., A review of sensor technologies for perception in automated driving. IEEE Intell. Transp. Syst. Mag., 11, 4, 94–108, 2019. 15. https://www.aionlinecourse.com/tutorial/self-driving-cars/hardware-andsoftware-architecture-of-self- driving-cars 16. https://encyclopedia.pub/entry/8473 17. Vargas, J., Alsweiss, S., Toker, O., Razdan, R., Santos, J., An overview of autonomous vehicles sensors and their vulnerability to weather conditions. Sensors, 21, 16, 5397, 2021. 18. Pereira, J.L. and Rossetti, R.J., An integrated architecture for autonomous vehicles simulation, in: Proceedings of the 27th Annual ACM Symposium on Applied Computing, pp. 286–292, 2012, March. 19. Behere, S. and Törngren, M., A functional architecture for autonomous driving, in: Proceedings of the First International Workshop on Automotive Software Architecture, pp. 3–10, 2015, May. 20. https://www.iiot-world.com/artificial-intelligence-ml/artificial-intelligence/ five-challenges-in-designin g-a-fully-autonomous-system-for-driverless-cars/ 21. https://coalitionforfuturemobility.com/benefits-of-self-driving-vehicles/ 22. https://innovationatwork.ieee.org/use-cases-for-autonomous-vehicle-­ technology/ 23. https://www.cubictelecom.com/blog/self-driving-cars-future-of-autonomous-vehicles-automotive-vehi cles-2030/ 24. Dhiviya, S., Malathy, S., Kumar, D.R., Internet of Things (IoT) elements, trends and applications. J. Comput. Theor. Nanosci., 15, 5, 1639–1643, 2018.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

A Survey on Architecture of Autonomous Vehicles   103

Autonomous Car Driver Assistance System R. Annamalai1, S. Sudha Mercy2, J. M. Mathana3, N. Banupriya4*, Rajalakshmi S.4 and S. D. Lalitha4 Department of CSE, R.M.K. Engineering College, Jeppiar Institute of Technology, Tamil Nadu, India 2 Jeppiar Institute of Technology, Tamil Nadu, India 3 Hindustan Institute of Technology and Science, Tamil Nadu, India 4 RMK Engineering College, Tamil Nadu, India

1

Abstract

For the larger part of the last few decades, researchers have been actively pursuing their goals of developing automobiles that can operate without human intervention. Numerous studies have been carried out on the issue of using a camera that is positioned on the front of a vehicle for the purposes of localization and navigation of the vehicle, environment mapping, and obstacle avoidance. These are all goals that the camera is intended to accomplish. Every single algorithm for recognizing traffic signs has four primary objectives that it is striving toward accomplishing. These objectives are listed below. The algorithm includes a list of these objectives. The first and most important thing for us to do is to guarantee that the algorithm will provide reliable results. The fundamental concept of accuracy has to be adhered to throughout the whole process of assessing it, beginning with the methodology all the way up to the measurement that is employed. There is a good chance that the activation of driver assistance features will need nothing more than a high degree of accuracy when the settings are left at their defaults. On the other hand, accuracy in the worst-case situation has to be addressed in other contexts, such as the context of fully autonomous automobiles, and it needs to be tested adequately. This is something that needs to be done. This is an activity that absolutely must be carried out. An algorithm that can recognize traffic signs by their color and form has been created. This method is based on the detection of the signs’ colors and shapes. The creation of this algorithm has already taken place. The photographs used in the software were captured using a camera with a poor resolution that was *Corresponding author: [email protected] Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (105–130) © 2024 Scrivener Publishing LLC

105

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

5

attached to the windscreen of a moving automobile. These photographs were then used in the program. Then, these photographs were sent into the algorithm as its input. The capacity of two different forms of traffic indicators, namely, red stop signs and yellow warning signs, to offer an early warning to vehicles is evaluated, and the findings are collected. As a direct result of this, the technique of formbased detection is sensitive to the complexity of the backdrop, while the colorbased detection approach is sensitive to the lighting environment. Keywords:  Vehicle, environment mapping, and obstacle avoidance, accuracy, color based detection approach

5.1 Introduction Over the course of the last few decades, one of the areas of study that has seen the most activity in the field of computer vision is that of automated driving assistance. This area of study has been one of the topics that has seen the highest activity. In recent years, there has been a growth in both the number of different modes of transportation and the overall amount of traffic. This has resulted in a significant rise in the complexity of coordinating logistics and the stress that is placed on society as a whole. It is anticipated that the introduction of driverless cars will result in a wide range of beneficial changes for society. One of these changes will be a reduction in the number of people who are killed or injured as a result of traffic accidents, or even the elimination of these occurrences entirely. Sixty percent of the time, drivers are to blame for accidents and injuries that are the direct result of road traffic [1]. Passengers are to blame for 20% of these incidents, and pedestrians are to blame for 20% of these incidents as well. Because of this, the driver assistance system for autonomous cars has to conform scrupulously to the rules that have been stipulated by the authorities in charge of traffic. These rules include those that apply to traffic signals, vehicle indicators, lane markings, vehicle speed, and traffic signs. Other regulations that fall under this category include those that govern the speed of vehicles. As needed components, an automated driving assistance system has to be able to recognize hand gestures used by traffic police, recognize indications, and detect lane markings. Additionally, the system should be able to locate lane markings. The exchange of information and ideas that takes place between individuals is heavily reliant on the use of various hand gestures. The recognition of the gestures used by law enforcement officers in various traffic circumstances is the most important piece of information that can be gathered for autonomous driving in urban areas. This may be accomplished by watching police officers interact with various traffic

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

106  Artificial Intelligence for Autonomous Vehicles

scenarios. It is essential to keep an eye on the movement of other cars and to have an awareness of how the vehicles interact with one another in the many traffic conditions that may occur in order to drive in a manner that is both safe and efficient. Markings on the road, which are sometimes referred to as road marking instructions on occasion, present motorists with a variety of signals that may aid them in driving in a way that is safer. The word “video,” which refers to still photographs that move, derives from the Latin word “videre,” which means “I see.” The term “video” refers to still photographs that move. Simply “I see” is the literal translation of the phrase “I see.” Video is a kind of recorded media that is distinguished by the transmission of a series of still pictures at a frame rate that is sufficiently quick to offer the illusion that the images are moving. This gives the impression that the images are progressing in time. Analog video tape formats include things like VHS and Betamax, while digital video formats include things like Blu-ray Disc, DVD, QuickTime, and MPEG-4. Blu-ray Disc, DVD, QuickTime, and MPEG-4 are examples of formats that may be used for digital video. Additionally, video may be created and conveyed via the use of the three primary video standards. Each of these standards determines the maximum resolution of the display as well as the color palette that can be used. Any one of these standards may be used for the production and transmission of video. These three acronyms stand for three different technologies: Phase Alternating Line (PAL), Sequential Color with Memory (SECAM), and National Television System Committee (NTSC). SECAM was the first color television standard in Europe, and its data were stored on magnetic tape at the time. The PAL and NTSC television systems were both developed by the National Television System Committee. It is not conceivable for any one of them to coexist in the same location as the other one. It is not possible to play back video that was captured in one format using a different format. This is not something that is possible. As a consequence of the debut of the Sony D-1 format in 1986, the year that marked the beginning of the commercial distribution of digital video, digital video has become more widespread. This digital movie is made up of a sequence of digital photographs that are shown one after the other in quick succession at a consistent rate. The tempo remains the same throughout the whole movie. The movie is rendered in digital format for the presentation. The individual still pictures that make up a video are referred to as “frames,” and the word “frame” is used when discussing a video. The number of photographs that are created in a single second is used as a yardstick to determine how often these images are updated. One second equals 33 frames (fps). Because of this, the number of still pictures that are formed throughout the course of a single second of video is referred to as

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  107

the “frame rate,” and this is the meaning of the phrase “frame rate.” It might be as low as six or eight frames per second for older mechanical cameras, or it could be as high as more than 120 frames per second for more modern specialized cameras. Either way, the rate can vary greatly. When a series of still pictures or digital images are strung together to create a moving picture or movie, each individual still photograph or digital image is referred to as a frame. A raster of pixels is utilized to construct each individual frame of a digital movie, and each frame is its own self-­ contained digital picture. A digital movie is composed of several distinct digital movies. If it has a width of w pixels and a height of h pixels, then the frame size, also known as the image size, is equal to w pixels times h pixels. In other words, the picture size is the same as the frame size. This is often referred to as the size of the image. The user’s needs will determine whether the digital picture is captured in color or in black-and-white mode, since both of these options are available to them. The value of each pixel in a grayscale picture is digitally represented using 8 bits, and the total number of levels in the image is 256. The total number of levels in an image is 256. It is possible for there to be anything between 0 and 255 levels in the picture. A color model is often used whenever the representation of an image that incorporates color is being done. The RGB values that are encoded using 24 bits per pixel are created by utilizing three unsigned integers of 8 bits each, ranging in value from 0 to 255. These values are then used to construct the pixel. The relative intensities of the colors red, green, and blue are represented by these three numbers (blue). Simply by combining the three main colors of red, green, and blue, a person is able to create any one of approximately 16 million distinct hues using only these three primary colors. The video of gray levels, denoted by the symbol “I,” can be described as “I = I1, I2,..., IM,” where “I1, I2,..., IM” refers to the sequence of gray-level photographs and “M” refers to the total number of frames that are included in the movie. The symbol “M” also denotes the total number of gray-level photographs. The letter “I” serves as a sign for the video of the gray level.

5.1.1 Traffic Video Surveillance Over the course of the last several decades, the image processing community has shown a considerable level of interest in the subject of traffic video surveillance. Within the realm of computer vision, one of the most important research areas to focus on is the traffic video surveillance system. Its principal functions are to recognize objects, identify them, and follow their movements over a sequence of images. In addition to this, it makes an effort to grasp and make sense of the behavior of the things that are being observed. Analysis of the road’s state for the purpose of traffic video surveillance and

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

108  Artificial Intelligence for Autonomous Vehicles

safety warning is a task that involves expertise from a range of disciplines. These fields include, among others, computer science, robotic engineering, and psychology. A traffic video surveillance system is able to provide efficient and effective applications, some of which include commercial and public safety, visual surveillance and congestion analysis, human identification and the detection of anomalous behavior, and the monitoring of vehicles both inside and outside of cities, highways, bridges, and tunnels, among other places. One of the applications that a traffic video surveillance system is able to provide is the ability to provide efficient and effective applications. Despite the significant progress that has been made in this area, the essential goal of developing a system that is both effective and reliable and that is also able to function normally under demanding real-time situations has not yet been achieved. This is an essential objective that must be achieved in order for the system to be considered successful. As a consequence of this, the objective of this research is to develop an intelligent driver assistance system in order to forestall the occurrence of accidents. There are two separate variants of the system that monitors and records a video of traffic, and these variants are as follows: 1. A security system that is composed of cameras that are permanently installed 2. A surveillance system that employs cameras that can pan and tilt The primary goal of the surveillance system that makes use of stationary cameras is to investigate the capabilities of the system to watch and organize urban traffic in order to track traffic occurrences that might lead to accidents. This is done in order to track traffic events that might lead to accidents. The static images captured on the recorded movies are of a very good quality, and they might provide the security and police departments with important information if they are analyzed. These indications include things like the license plate of the automobile, the time it passed, the movement path it took, the driver’s face, and a variety of other details. The moving surveillance camera is permanently fixed in the vehicle, and it gives the driver access to a high-quality video feed of the surrounding traffic in addition to providing improved performance on the road. In the course of this research endeavor, a stationary camera has been used as the tool of choice in order to pinpoint the precise position of various hand gestures employed by police officers. Methods using moving cameras are used for the purpose of identifying road lane markings and the indicating signals seen on the back of motor vehicles.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  109

For a very long time, those who study computer science and robotics have harbored the dream of developing automobiles that can navigate their own routes without human intervention. Autonomous vehicles have the potential to bring a variety of benefits to the general public, including the removal of the need for drivers to pay attention to the road and the promotion of calm and safe driving. Currently, drivers are required to pay attention to the road at all times. It is the job of a driver assistance system to provide a driver with support and, as a result, the necessary information automatically to a person who is driving for the sake of the driver’s own safety, the safety of the vehicle he is operating, and the safety of the environment in which he is operating it. The average person in today’s society has a heightened understanding about the significance of the issue of road safety, which is a positive development. One of the most important aspects that contributes to safer driving is compliance with the many rules and regulations that govern traffic in India. It is very necessary to have a comprehensive awareness of the Indian traffic rules in order to reduce the likelihood of being involved in an accident. The following are the components of Indian traffic legislation that need to be taken into account: • The hand signals used by law enforcement personnel • The traffic light signals and signage along the route • Road signs and lane markings strategically placed along the roadway • The hand signal is used by drivers of motor vehicles. • Limits on the maximum speed that are legally allowed • Lights on the vehicle that show its state (taillight, brake light, indicator lights, and beam light)

5.1.2 Need for the Research Work Traffic regulations have been enacted in every part of the world in order to regulate the ever-increasing number of automobiles on the road and to promote living and driving conditions that are safer. Each geographical location has its own set of rules and regulations regarding the laws that control transportation. People who have to drive for lengthy periods of time often feel distractions, exhaustion, and boredom in their driving. As a consequence of these elements, accidents happen rather often, and depending on the circumstances, they might even be fatal. As a direct consequence of this, the development of driverless automobiles is now the main focus of the majority of research being conducted today. The provision of automated driving assistance is the major role that an autonomous vehicle is expected to play. This body of research is being conducted with the intention of formulating

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

110  Artificial Intelligence for Autonomous Vehicles

a plan for an automated driver assistance system. This plan will serve as the basis for the development of completely autonomous vehicles that are capable of functioning in the Indian traffic environment. It focuses on the hand gestures employed by those who enforce traffic laws.

5.2 Related Work Since that point in time up to the current day, there have been a great many different research groups researching on the topic. Using their own distinct approach, each of these organizations has looked for a way to resolve this problem in recent years. Despite the fact that, at first glance, the fundamental stages toward a solution seem to be extremely well outlined and uncomplicated, the specifics of the ways that have been employed reveal that there are a variety of possibilities and a great deal of concepts. At this moment, no one solution approach has distinguished itself as the undeniable leader, and it is quite evident that it will be quite some time before solutions begin to arrive on the market. The process of detecting the identification of a road sign involves a number of processes, the most important of which are the steps of detection and recognition. During the stage that is referred to as “detection,” the various research groups are split up into three main categories. The first set of researchers came to the conclusion that the colors of traffic signs are key bits of information that may be utilized to identify and categorize different traffic signals. The second school of thinking claims that it is feasible to recognize traffic signs based just on their shapes, but the third school of thought maintains that the combination of color and form is what creates what comprises the important component of any road sign identification system. As a result, there are three basic approaches to recognizing traffic signs: recognition based on the information supplied by color, recognition based on the information provided by shape, and recognition based on the information provided by both color and shape. The pictures that were used in all of the articles that were examined were ones that were taken from real-life traffic scenarios, and they were pretty similar to the photographs that were gathered for this research. When it comes to carrying out the task of traffic sign detection, the approaches that are used might vary greatly depending on the author. It is possible to find a solution to this issue by using any one of a wide variety of various strategies. Baklouti et al. [6] used thresholding to separate pixels in a digital image into those that make up objects and those that make up the background. The approach consists of determining the distance in RGB space between the color of a pixel and a reference color in order to achieve the desired

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  111

effect. This distance is measured relative to the color of the reference color. If the color of the unidentified pixel is sufficiently close to the color of the reference pixel, then the unidentified pixel will be considered an object pixel. Nickel and Stiefelhagen [7] suggested the creation of an algorithm that would be able to recognize the traffic warning signs for Stop, Yield, and Do Not Enter. This would be a useful tool. It is made up of a total of six distinct modules, which include the following: color segmentation, edge localization, RGB differencing, edge detection, histogram extraction, and classification. The use of color segmentation is confined to the localization of red edge areas, the segmentation process itself is carried out in a sparse way, and the interpixel segmentation distance is determined. Wu and Huang [8] came up with a method for discovering signs that may be of use to those who are blind or have some other kind of visual impairment. The author made the premise that signs are composed of two distinct colors—one for the sign itself and another for the text—and that sign limitations are specified in advance (rectangle, hexagonal). In order to locate regions relevant to a hypothesis, an algorithm for expanding regions is utilized, and inside this algorithm is a series of tests that pick seeds to serve as its starting point. Corradini [9] proposed a method for determining the road sign’s identity that included making use of the sign’s color distribution while doing so within the framework of the XYZ color space. They created a color similarity map by using the color distribution, which was then included into the picture function of an active net model. The road sign [10] may still be freed from the muddle even if it becomes entangled in a functional net throughout the process. Yoon and Kuijper [11] came up with a way to identify traffic signs by including the HSV and YUV color spaces into their algorithm. The system is brought online in two distinct stages one after the other. The first step of the process involves translating the RGB image into the YUV color space and then equalizing the histogram of the Y channel. The second stage involves creating a new RGB. This concludes the first step of the process. In the second stage, color segmentation is achieved by first converting the RGB picture that was produced in the first stage into the HSV and YUV color spaces and then applying a sufficient amount of threshold to the H and UV values. This sequence of steps is repeated until the desired results are obtained. This puts an end to the procedure. Following this, an AND operation is carried out, which combines the two results obtained before. Stenger [12] developed a computer vision system that is capable of recognizing traffic signs and can be placed in a car. Stenger’s [12] technology

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

112  Artificial Intelligence for Autonomous Vehicles

can be used to help drivers stay safe on the road. Several tests, each employing a different kind of road sign, were carried out in order to explore the stability of colors when exposed to a wide range of lighting conditions. The goal of these investigations was to determine how colors changed over time. Segmentation is made possible by the use of the RGB color space. It has been shown that there are substantial disparities between the red, green, and blue components, and that these differences, when paired with an appropriate threshold, have the potential to be used for segmentation of the data.

5.3 Methodology The vehicle-mounted cameras that are put on smart vehicles are the ones that record the pictures of the road traffic, and the goal of the traffic sign detection is to adequately extract the interested traffic sign areas from the photographs of the road traffic that are now available. However, the quality of the captured pictures might vary depending on the surrounding environment. In order to properly identify these qualities, it is necessary to follow the intrinsic properties of traffic signs, such as their color and form. Within this area, it is primarily composed of two components: traffic sign segmentation based on the color space and traffic sign recognition based on form attributes. i. Road lane mark detection as shown in Figure 5.1.

5.3.1 Intelligent Driver Assistance System The term “Intelligent Driver Assistance Systems” (IDAS) refers to systems that provide assistance to drivers of motor vehicles in order to increase both comfort and safety. This is achieved by presenting a display that is ergonomically appropriate for the environment in which the vehicle is operated and by sending a warning indicator in the event that any potentially dangerous situations are present in the traffic scenes that are located around the vehicle. The Driver Assistance System, also known as DAS, is intended to be of assistance to drivers of motor vehicles. It does this by transmitting signals, such as emergency braking systems and detecting indication lights, with the intention of alerting drivers to moving vehicles, the road, or any hidden threat. It is in the best interest of drivers to keep their attention on other moving vehicles, since doing so will reduce the amount of accidents that involve moving cars.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  113

traffic sign

autonomous

brake and indicator

lane mark detection

Figure 5.1  Aim of the proposed research.

Over 1.2 million people are murdered and an additional 50 million people are injured in road traffic accidents each year, as reported by the World Health Organization (WHO) [2]. As a result, one of the key priorities of governments and residents all over the world is to ensure that people who are traveling on roads do so in a safe manner. India has the highest number of road accidents of any nation in the world. This is mostly because of the ever-increasing number of automobiles that are driven on the country’s roads as well as the ever-increasing congestion levels. Advanced driver assistance systems are designed to make everyone in the vehicle, including the driver, safer by communicating information on the vehicle to the driver and the passengers in the vehicle. Intelligent driver assistance systems incorporated a variety of technologies that enhanced safety, such as electric lighting, which made its debut in 1898 on the Columbia Electric car, and turn signal lights, which made their debut in 1907. Both of these innovations were integrated in the system. These two advancements came about as a result of efforts to make driving a safer activity. In the 1920s, medical experts first started campaigning for seat belts to be installed in cars as a method of protecting passengers from injury. In the 1960s, a group of researchers working together created the prototype for what would later become known as the airbag system. On the other hand, it took the United States and Europe an additional 20 and

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

114  Artificial Intelligence for Autonomous Vehicles

30 years, respectively, to make it the industry standard. In 1978, Bosch and Mercedes-Benz were the first companies to make their antilock braking system (ABS) technology accessible to the general public on commercial cars. Over the course of the last 15 years, researchers have made significant strides in developing an intelligent system that is able to anticipate potentially dangerous conditions and to make accurate predictions about accidents. They referred to it as an Advanced Driver Assistance System (ADAS), which means that it assists the driver in the sense that it provides warnings, aids in forming judgements, and even conducts autonomous evasive maneuvers when required in extreme situations. It differs from earlier generations of safety technology in that in addition to receiving mechanical or physical signals from the vehicle it is installed in, it is also capable, albeit to a lesser degree, of comprehending the environment outside the vehicle in which it is installed. The ADAS has a wide variety of applications, which include human machine interfaces, object identification, vehicle detection, tracking, and alert assistance. Other applications include human machine interfaces, object identification, and vehicle detection. The year 1986 saw the development of an autonomous highway driving system by the group led by E. Dickmanns [3, 4], which marked the beginning of the field of ADAS. They exhibited a system that was capable of driving through obstructed streets at speeds of up to 59 miles per hour (96 km per hour) using cameras, simple image processors, and Kalman filtering. This system was shown to us. These days, a range of advanced driver assistance technologies have made their way into the commercial market and may be found in computer vision technology. Some of these technologies include automatic parking and lane departure warning systems. The use of this technology will make it possible to construct advanced driver assistance systems. Autonomous Driving System (IDAS) that makes use of traffic video surveillance to identify traffic police gestures, detect vehicle indication signals, and provide lane mark warning in traffic scenes system.

5.3.2 Traffic Police Hand Gesture Region Identification The management of traffic on roadways is a challenging task that is receiving an increasing amount of assistance from automated technology. These kinds of systems are starting to appear more often. The formation of rules and regulations for traffic is done with the purpose of allowing the safe and orderly movement of motor vehicles along highways. This is the goal of the rules and regulations that are established. In addition, the rules and laws that govern traffic are not just meant for motorists who are driving vehicles

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  115

on the road; they are also created for walkers, cyclists, motorcyclists, and any other people who utilize the road. The correct knowledge of traffic rules has the ability to reduce the number of accidents that take place and, as a consequence, to make it easier to establish a transportation system that is both secure and well-organized over the whole of our country [5]. In an environment where human body gestures are used for the purpose of controlling traffic, the drivers are obligated to follow the instructions that are given to them by the traffic police officer. In an effort to make driving a more secure endeavor for motorists, researchers are focusing their efforts on developing methods that can automatically recognize the hand signals used for traffic management. When this technology is utilized in combination with a traffic police control system, a human traffic controller is able to analyze the flow of traffic within a visual range around the traffic intersection. This range includes the area surrounding the intersection. The instructions for controlling traffic can be broken down into three distinct categories, which include “stop all cars on every road direction,” “stop all vehicles in front of and behind the traffic police officer,” and “stop all vehicles on the right and behind the traffic police officer.” Each of these categories contains specific instructions for controlling traffic. Each and every one of these hand signals for traffic is created by combining the arms pointing in a variety of different directions. There are a total of 12 distinct Indian traffic hand signals that may be constructed using the various control command types. The following table provides an explanation of each of the 12 hand signals that are used by the police during traffic stops. The most significant obstacle that must be overcome in computer vision technology for human and traffic video surveillance is the development of a system for detecting and identifying gesture regions. This technology is able to recognize and pick up on a variety of different hand movements. The primary goal is to create hand gesture recognition that may be utilized in recordings of traffic police employees by making use of established ways for enforcing traffic police laws. These recordings might be used to improve traffic safety. In recent years, it has become more vital for members of the police force to be able to recognize hand signals. The increasing volume of traffic that may be seen on metropolitan streets is primarily responsible for this phenomenon. The purpose of this is to create a traffic surveillance system that makes use of computer vision techniques to recognize human beings, hand gesture regions, and hand gestures themselves in films that have been gathered specifically for the purpose of recognizing those hand gestures. This system will be built with the intention of recognizing those hand gestures. Recognizing such hand motions is the objective of this effort in the long run. Table 5.1 shows the attributes of the Indian traffic

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

116  Artificial Intelligence for Autonomous Vehicles

Table 5.1  Indian traffic police hand gestures. A1, A2, A12

Hand gesture action meaning

A1

To start one side of vehicles

A2

To stop vehicles coming from front

A3

To stop vehicles approaching from back

A4

To stop vehicles approaching simultaneously from front and back

A5

To stop vehicles approaching simultaneously from right and left

A6

To start vehicles approaching from left

A7

To start vehicles coming from right

A8

To change sign

A9

To start one side of vehicles

A10

To start vehicles on T-point

A11

To give VIP salute

A12

To manage vehicles on T-point

police hand gestures. As a direct result of this, the primary purpose of this effort is to: • Determine the identities of the individuals shown in the video sequences of the traffic police force that take place along the road traffic scenes. Naturalistic and intuitive human hand gesture has been a great motivating factor for researchers to put their efforts in research and develop the most promising means of interaction between humans and computers. Researchers have put their efforts in research and developed the most promising means of interaction between humans and computers. The most promising techniques of interaction between people and computers have been established—thanks to the efforts of researchers who invested their time and energy into study. Through the investigation and creation of novel approaches, researchers have been concentrating their efforts on the establishment of the most productive ways for humans and computers to connect with one another. To be more precise, the many various kinds of gestures are of immense aid in reflecting the many different communication signals that are necessary to complete particular jobs. This is due to the fact that different tasks require different communication signals. As a result of this, the various types of gestures are an especially valuable tool for communicating with one another. As a direct result of this, the second objective of this body of work is to make use of computer vision technology

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  117

that has been created from dynamic gesture movement in order to partition the hand gesture area of traffic police laws. Despite the fact that there may be a number of different components present, the motion of gestures will be considered to comprise all of the relevant local information relative to the location of the gesture. The ability to comprehend the motion of the gesture is the single most important ability that one must possess in order to recognize the hand laws that are utilized by traffic police. Therefore, the end goal of the approach is to extract from the area of the human gesture that has been subdivided into the components that will allow the machine to comprehend the significance of that motion. This procedure takes place during the evaluation of the gesture identification. The investigation that has been carried out on the subject of pattern classification has resulted in the production of a variety of distinct instances of typical classifiers. Support Vector Machines (SVM), Decision Trees, Random Forests, and Naive Bayes are some examples of these types of models. As a direct result of this, the ultimate goal for the gesture recognition system is to choose the classification model and then include the recovered features into the classifier that is used by the system. The recognition of human hand gestures is the most difficult job involved in the processing of videos. This is due to the fact that it enables the computer to detect, recognize, and comprehend the hand gestures of the user in order to communicate with a diverse selection of human machine interfaces. This is the primary reason why it is of such great value. The gesture action of traffic police is a vital communication tool for ensuring that drivers are able to operate their cars in a safe manner while dealing with difficulties with traffic. The goal of gesture recognition is to categorize the myriad of action rules that are associated with gestures, then assign those categories names that have some kind of connotation associated with them. The following is a summary of the primary issues that are encountered by the gesture recognition systems that are used by the traffic police within the setting of the environment of the traffic: Acquiring Information With a Camera: In an urban setting, a fixed camera is positioned at a busy intersection in order to collect videos. These videos are collected utilizing the camera. All of the action that takes place there is recorded by this camera at all times. The computer vision technology is able to separate out particular personnel of the law enforcement agency who are present in a traffic video scenario. This is possible even when the backdrop is always changing. This is done in a myriad of various settings and circumstances.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

118  Artificial Intelligence for Autonomous Vehicles

traffic police

hand gesture detection

image capture

pre-processing

segmentation

Figure 5.2  Challenges for gesture rule recognition.

Transformation of Gesture: A gesture may be articulated within a human’s hands, arms, or body; it can also be the movement of a person’s head, face, or eyes, such as winking, nodding, or rolling the eyes. Gestures can also be expressed within a human’s hands, arms, or body. It is also possible for a gesture to be articulated within the hands, arms, or body of a human being; a gesture can also be articulated within the hands, arms, or body of a human being. Animals are also capable of expressing motions on the inside of their bodies. When it comes to situations involving traffic, it is exceedingly challenging to discern the movement of the human body from the movement of the gesture due to the dynamic gesture activity that occurs in these kinds of conditions. Recognition Is Done Through Gesture in Indian Culture: The rules that traffic police employ to communicate through gestures have a profound impact on the manner in which drivers are expected to interact with one another. The traffic regulations themselves are varied, as can be shown in Figure 5.2, despite the fact that the rotation of the hand signals used by traffic police is identical to one another. The inability to recognize traffic gesture norms that are reasonably consistent across a number of traffic conditions is the primary obstacle that has to be overcome. There are many different traffic scenarios.

5.3.3 Vehicle Brake and Indicator Light Identification Research into autonomous vehicle systems has been one of the sectors with the greatest rate of recent increase in recent years. This may be attributed to

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  119

the vital part it plays in maintaining the safety of drivers. It is vital to keep an eye on the behavior of other vehicles and to have a strong grasp of how they interact with one another in typical traffic scenarios in order to drive in a way that is both safe and successful. During the course of this project, videos will be shot inside the automobile with the assistance of a camera that will be left inside permanently. In situations involving road traffic, the characters in these movies get important information about the driving car in front of them. The brake light, the indicator on the left side of the vehicle, and the indicator on the right side of the car are the three kinds of automotive lights that drivers use the most often. In the framework of the traffic environment, the vehicle that is following behind is expected to heed the instruction of the front moving vehicle lights. This is the case whether or not the lights are activated. Numerous important particulars, such as the license plate number, the vehicle symbol, and the vehicle’s lights, are often shown on the trunk or trunk lid of a car. As can be seen in the figure, the logo of a vehicle and the license plate of the car both have the potential to supply information that may be utilized for confirming and identifying the vehicle. Red is the color that is used for each of the indicator lights that are seen on a car, and each one has its own particular look. For instance, the left and right indicators on a vehicle that turns have a different design from the brake light that is seen on a vehicle that stops. The lights of a vehicle are positioned in such a way that they are centered on the left and right corners of the vehicle, as can be seen in Figure 5.3. The detection of automobile brakes and indicators is essential for maintaining a safe driving input image

Figure 5.3  Vehicle features.

visualization of HOG features

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

120  Artificial Intelligence for Autonomous Vehicles

environment, and the work that is being presented here demonstrates a novel approach to completing the job at hand. It is of the highest significance to reliably and accurately recognize front moving vehicle signs such as turn signals and brake lights, especially for applications that depend on vision for traffic surveillance. The increasing number of vehicles adds to a range of challenges, including those relating to traffic control and management, issues with parking, accident rates, and a variety of other concerns. This is done so that computer methods can be used to construct a traffic surveillance system that is able to recognize vehicles and vehicle signals, as well as perform tracking and recognition in a video sequence that has been captured. The reason for this is so that computer methods can be used to construct a traffic surveillance system. As a consequence of this, the major goal of the work that is being proposed is to recognize the vehicles that are existing in the road environment throughout the whole of the traffic video sequences. It is quite important for a vehicle to be equipped with a driver assistance system that can read out light indicators while the vehicle is in motion. Within the realm of computer vision technique, color segmentation is a frequent strategy that is used. As a consequence of this, the second objective of this body of work is to segment the vehicle lights using a color segmentation technique and locate the vehicle light zones within the car. A video sequence displaying an automobile offers an explanation of a distinct indicator at each and every frame of the series. These indicators may be observed under a variety of different traffic circumstances. When it comes to autonomous driving, correct identification and continuous transmission of the light information from the car are necessary. As a consequence of this, the third objective is to develop an algorithm for monitoring automobiles that is able to get data on brake and turn signal functionality. It has been shown by a large number of academics that the area of the development of automated techniques for the extraction of video characteristics is a subject that is booming. Therefore, the fourth objective is to extract novel and useful elements from each snapshot of a vehicle so that a representation of how the car’s lights are may be produced. In conclusion, identification algorithms are very dependent on the plethora of light properties that the vehicle has. A large range of conventional classifiers may be found in the research that has been done on pattern classification. As a consequence of this, the ultimate objective of the work that has been described is to figure out which classifier model will be able to recognize the vehicle’s brake and indicator lights. When you are behind the wheel of a car on the highway, it can be a challenging task to keep an eye on the brake and indicator lights. This is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  121

because there are a number of factors that can affect how you see them, including changes in the background, the amount of lighting, and the location of the car. It is of the highest significance to monitor and categorize the numerous indicator lights that occur in the traffic video sequences. These lights serve as a warning signal to assist in the prevention of future collisions, thus monitoring and categorizing these lights are essential. The summary that follows provides an overview of a few of the difficulties that are associated with the vehicle light identification system. • Camera Acquisition: This portion of the training consists of a moving camera that is stationed at a highway traffic scene. During this part of the class, records are obtained by the camera. In the video that was taken, the automobile would seem to be going very rapidly, which would explain why the picture from the camera is hazy.

Figure 5.4  Challenges for vehicle light detection.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

122  Artificial Intelligence for Autonomous Vehicles

• Vehicle Position: The movies that were acquired were obtained when the cars were positioned in highway traffic situations. This is shown by the fact that the movies were captured. When driving on the highway, you may encounter a large number of automobiles moving in a variety of directions, as well as other objects and people that may pass you by. In light of this fact, it may be challenging to recognize a particular vehicle in a scenario such as the one that we are in right now. • It is possible that certain video sequences will include moving cars and other objects in the background. These compiled video sequences could include moving automobiles or other things in the backdrop at varied distances, which adds an extra level of difficulty. • Variations in Lighting: It is possible that the backdrop model will not be able to adjust to the subtle variations in lighting that occur throughout some of the traffic scenarios. Vehicle Color: If the vehicle is red in color, then determining the vehicle brake and indicator light is difficult as shown in Figure 5.4.

5.4 Results and Analysis The navigable zone of a road is often found in the space in between the lanes of a road. This is especially true in the context of highway settings; hence, road lane-marking and direction are essential components of autonomous navigation. The phrase “road network” may refer to either the actual infrastructure of roads or the many control systems that are in place for motor vehicle travel. Accidents that are caused by traffic difficulties are becoming more common in today’s world as a result of the rising number of cars on the road and drivers who disregard the regulations that govern safe driving practices. In the past, accidents that were caused by traffic difficulties were much less common. The lane lines on the road may be the major or only indication that enables a driver to go safely in situations such as dense fog, mist, or when the warning lights of an approaching car cause the driver to lose his or her vision. The basic objectives of the driver assistance system are the lane departure warning system (LDWS), road lane-mark recognition, the lane change assistance system (LCAS), and the Lane Maintaining System (LMS). Road markings and instructions on road markers may have a range of different connotations based on where they are situated on the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  123

road. Figure 5.5 provides an illustration of the key meanings that lie behind the different road signs. If you want to drive in a manner that is not only successful but also safe, it is vital to pay attention to the lane lines on the road and understand how they interact with one another. When it comes to the creation of a driver assistance system, the classification of lane markings is one of the most important aspects that must be taken into consideration. Because the primary goal of the proposed method is to classify road lane markings, such as road area extraction and lane mark region recognition, the characteristics need to be more resistant to the variety of the road environment. This is because the method’s major objective is to classify road lane markings. When a vehicle is traveling in front of another vehicle, the region of the road that the vehicle is traveling on is always positioned in front of the vehicle and is not too far away from the car at that moment [9]. It really should not come as a surprise that the information available at the roadside sites is more extensive. As a consequence of this, the primary objective of this body of work is to make use of the region of interest (ROI) technique in order to discover and extract road area from the road environment. Car Image

0

Car HOG Image

0

10

10

20

20

30

30

40

40

50

50 60

60 0

10

20

30

40

50

60

Non-Car Image

0

0

10

20

20

30

30

40

40

50

50

60

20

30

40

50

60

Non-Car HOG Image

0

10

10

60 0

10

20

30

40

50

60

Figure 5.5  Road markings along the road.

0

10

20

30

40

50

60

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

124  Artificial Intelligence for Autonomous Vehicles

It is vital to have an accurate portrayal, identification, and classification of the lane marking information before continuing with the analysis. This is required in order to go forward with the process. As a consequence of this, the second objective is to identify the lane marking and identify it from the other areas of the road. A road feature may be used to record the lane marking in traffic video sequences, which is a useful source of information [10]. This can be done with the help of a camera. According to this school of thought, a feature may be seen as an expressive component that is derived from the lane markers. As a consequence of this, the third objective is to extract lane marking elements from each photograph of a road area so that a representation of lane markers may be created. Numerous academics have shown that the creation of an automated system for assigning a category to a video sequence is a vibrant topic of research, and they have done it in a variety of ways. Many of the conventional classifiers, such as SVM and hidden Markov model (HMM), have been developed as a result of the research that has been carried out on video classification (HMM). As a consequence of this, the final objective is to choose the classification model and determine the kind of lane markings and directions to use. One of the fundamental building blocks of autonomous vehicle navigation is the categorization of the numerous lane markers that appear on the road. In a perfect world, the lines delineating the lanes of traffic would be white, and the pavement would be black. When identifying lanes that have markings, the following factors need to be thought about and taken into account. • A state of variable lighting may be present, in which the level of illumination of the surrounding road environment varies depending on factors such as the time of day [11], the weather, and the presence or absence of shadows. This type of lighting can be dangerous for drivers because it makes it more difficult to judge distances on the road. It is possible that this will have an impact on the quality of the video sequences that are captured [12]. Figure 5.6 illustrates the impact that shadow has on the legibility of road lane markers. Extremely low levels of lighting: The video sequences that were captured had extremely low levels of illumination, and the winding road surface is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  125

0

100

200

300

400

500

600

700

0

200

400

600

800

Figure 5.6  Road lane marks affected by shadow.

Figure 5.7  Road lane marks affected by low illumination.

1000

1200

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

126  Artificial Intelligence for Autonomous Vehicles

especially difficult to control. Figure 5.7 illustrates how improper illumination might affect the visibility of road lane markers.

5.5 Conclusion The goal of the work that is now being done in this research project is to create an autonomous vehicle system with the intention of achieving effective detection and identification of traffic video sequences. This is the aim of the work that is currently being done. By using this approach, we were able to resolve the issues listed below. Recognition of Hand Gestures for Law Enforcement Use in Traffic: The purpose of the research that is going to be given is to find a solution to the issue of gesture recognition for traffic rules that are being shown in a video sequence. It may be difficult to recognize a human person [13] in a video sequence due to the intricacy of the human body [13]. This study makes use of information related to gesture areas in order to construct a system that is capable of recognizing gesture movements. This is done because of the significant part that gestures play in the functioning of the human body. When there is a lot of traffic, for instance, a driver will watch a traffic police officer carefully since the information that is sent by the officer’s gestures is the most crucial. As a result of this, the work that has been done up to this point to create a gesture rules recognition system for use by traffic police contains the following steps: • Identification of human beings • Division of the gesture area used by the police in traffic situations • The extraction of cumulative block intensity characteristics from the portion of the gesture area that is traveling in a certain direction • Familiarity with the hand signals used by the Indian police in traffic situations Vehicle Detection and Vehicle Indicator Recognition: The work that is being recommended addresses the challenges of vehicle detection and vehicle light tracking for autonomous cars by analyzing traffic video sequences. This is done in order to improve safety and efficiency. It is a challenging challenge to correctly categorize moving cars when the environment around the traffic scene is so complex. In addition to this, the split of the vehicle’s lights is still another significant characteristic that sticks out

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  127

among the other components of the automobiles. The automated vehicle light tracking system is a very engaging system because of the fact that moving vehicles exhibit [14] a range of braking and indicator lights. As a consequence of this, the work that is being given offers a method for recognizing the indicator and brake lights that are seen on automobiles. The Different Types of Road Lane Markings and Their Classification [15]: The topic of investigation for the research that has been proposed is the identification and classification of road lane markers for use in intelligent transportation systems. The removal of noise from the road environment is an extremely significant step due to the presence of noise in the road environment. Depending on the condition of the road, it may be difficult to recognize the lane markers. As a consequence of this, the classification of lane marking and lane marking direction in a video sequence works toward the following aims in order to accomplish them: • Road area extraction • The delineation of road lanes for identification purposes • Extracting attributes from the road’s lane markings in order to improve driving safety • Developing a method for the organization of road lane markings and the directions of lane marker

References 1. Hussain, T., Shu, L., Sosorburan, T., Adji, A.S., Khan, A.H., Raja, A.F., Road traffic accidents: An observational and analytical study exploring the hidden truths in pakistan and south east asian countries. Healthline, 2, 1, 52–7, 2011. 2. Peden, M. et al., World report on road traffic injury prevention, WHO and UNICEF, London, 2004. 3. Dickmanns, E.D. and Zapp, A., A curvature-based scheme for improving road vehicle guidance by computer vision, in: Cambridge Symposium Intelligent Robotics Systems, International Society for Optics and Photonics, pp. 161–168, 1987. 4. Vlacic, L., Parent, M., Harashima, F., Intelligent vehicle technologies: Theory and applications, Butterworth-Heinemann, Australia, 2001. 5. Fang, G., Gao, W., Zhao, D., Large-vocabulary continuous sign language recognition based on transition-movement models. Syst. Man Cybern. Part A: Syst. Humans, IEEE Trans., 37, 1, 1–9, 2007. 6. Baklouti, M., Monacelli, E., Guitteny, V., Couvet, S., Intelligent assistive exoskeleton with vision based interface, in: Smart Homes and Health Telematics, pp. 123–135, Springer, 2008.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

128  Artificial Intelligence for Autonomous Vehicles

7. Nickel, K. and Stiefelhagen, R., Visual recognition of pointing gestures for human–robot interaction. Image Vis. Comput., 25, 12, 1875–1884, 2007. 8. Wu, Y. and Huang, T.S., Hand modeling, analysis and recognition. Signal Process. Mag., 18, 3, 51–60, 2001. 9. Corradini, A., Real-time gesture recognition by means of hybrid recognizers, in: Gesture and Sign Language in Human-Computer Interaction, pp. 34–47, Springer, Paris, 2001. 10. Le, Q.K., Pham, C.H., Le, T.H., Road traffic control gesture recognition using depth images. IEIE Trans. Smart Process. & Comput., 1, 1, 1–7, 2012. 135. 11. Yoon, S.M. and Kuijper, A., Human action recognition using segmented skeletal features, in: Pattern Recognition (ICPR), 2010 20th International Conference, pp. 3740–3743, IEEE, 2010. 12. Stenger, B., Template-based hand pose recognition using multiple cues, in: Computer Vision–ACCV 2006, pp. 551–560, Springer, London, 2006. 13. Dong, G., Yan, Y., Xie, M., Vision-based hand gesture recognition for human-vehicle interaction. Proc. Int. Conf. Control, Autom. Comput. Vision, 1, 12, 151–155, 1998. 14. Zabulis, X., Baltzakis, H., Argyros, A., Vision-based hand gesture recognition for human-computer interaction, in: The Universal Access Handbook, pp. 34–1, LEA, USA, 2009. 15. Kim, C.-H. and Yi, J.-H., An optimal chrominance plane in the rgb color space for skin color segmentation. Int. J. Inf. Technol., 12, 7, 73–81, 2006.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Autonomous Car Driver Assistance System  129

AI-Powered Drones for Healthcare Applications M. Nalini

*

Electronics and Instrumentation Engineering Department, Sri Sairam Engineering College, Chennai, Tamil Nadu, India

Abstract

Modern technology is becoming more prevalent in all procedures to enhance the standard of people’s lives. As they advance in the healthcare industry, advanced technologies have paved the path for all industries over time. The use of technology in healthcare is growing involvement in nearly all activities from practically showing up on the screen for lab testing. Many individuals live in isolated places, those are lacking access to basic healthcare, making it accessible through pharmaceuticals, vaccinations, blood, and sample collection for analysis. Drones are unmanned aerial vehicles (UAVs). Common uses for this approach include situations where deploying a human-piloted aircraft poses a significant danger or when it is not practicable to use human planes. A drone can quickly bring medicines, supplies, and vaccinations to the location where they are needed, ending the outsourcing of deadly infectious illnesses. A drone may be employed for both disaster relief and administration. A live video feed of the impacted areas can be provided. There are a few spots in India that are difficult to get to owing to hazardous pathways and others that have dense traffic or subpar transportation infrastructure, but for some operations, a drone is seen to be enough. Consumers and the healthcare system are benefiting from a new perspective brought about by drones. The primary drivers of drone adoption will be advances in technology, rising investment, and assistance from the government. In a similar vein, these variables are also anticipated to increase the distribution of healthcare goods and services in rural regions. It has several potential advantages and can act as a pillar of support for the current healthcare system. The COVID-19 epidemic has propelled medical drone development and utilization over the last 2 years. Email: [email protected]; ORCID https://orcid.org/0000-0001-6457-7990

*

Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (131–150) © 2024 Scrivener Publishing LLC

131

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

6

Keywords:  Drones, unmanned aerial vehicles, healthcare drone, artificial intelligence, surveillance, telemedicine, drug delivery

6.1 Introduction Today, a medical professional may virtually appear on a computer, make a sound diagnosis, and provide sound counsel after evaluating the data. Distance no longer has any value. Actually, geography has morphed into history! Making medications, vaccinations, blood, or even biological samples for research available in genuinely distant places is a huge challenge. In order to reach the unreachable and make isolation relative rather than absolute, drones are being employed more and more in healthcare today. The term “inaccessible” itself will eventually lose its meaning. The reader is introduced to yet another instance of possible creative disruption in the healthcare industry in this overview. The author is certain that this will soon occur in several rural areas of India. We are currently experiencing a technological revolution. Healthcare professionals throughout the world are notoriously conservative though. In a recent analysis, Goldman Sachs predicted that over the next 5 years, the world will spend over $100 billion on drones. The military has made substantial use of drones in battle. The majority of drone uses include employing an onboard camera for surveillance. The different use cases include surveillance in agriculture; crop spraying; surveillance of sharks at beaches; wildlife conservation monitoring; fire monitoring; riot monitoring by police and governments; media coverage of events, sports, and entertainment and entertainment news; emergency services disaster responses for humanitarian aid; and scientific research and exploration [1, 2]. Unmanned aerial vehicles (UAVs) is another name for drones. The phrase “autonomous, or remotely operated, multi-use aerial vehicles” was originally used in the 1980s. UAV technology’s speed and adaptability provide essentially limitless potential for delivering aid and medical supplies to people in isolated or dangerous locations. They can even more quickly connect patients with doctors. Bystanders who are given remote instructions on how to deliver treatment to victims of emergencies or natural disasters may be able to save lives [3]. Drones may cease to operate due to bad weather or variations in the environment’s temperature. Drone construction and maintenance costs need to be thoroughly examined in order to do a cost–benefit analysis.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

132  Artificial Intelligence for Autonomous Vehicles

6.1.1 Role of Artificial Intelligence in Drone Technology Unmanned aerial vehicles called drones are used for a range of different tasks. These types of gadgets were first controlled manually and remotely. But nowadays, artificial intelligence is frequently included into drones, automating some or all tasks. Drone suppliers can gather and exploit visual type of data and environmental data by using data from sensors attached to the drone in combination with AI [4, 5]. By enabling autonomous or aided flying, these data increase accessibility and facilitate operation. Drones are now commercially available to companies and individuals as a part of the smart mobility services. Drones powered by artificial intelligence (AI) mostly rely on computer vision. Drones can now identify items while in the air and analyze and record data on the ground with the use of AI technology. High-performance onboard image processing using a neural network is how computer vision functions. A layered architecture known as a neural network is what machine learning algorithms are implemented using. Drones can realize, classify, and trace objects with the help of neural networks [6]. Drones can find and track objects while avoiding collisions because of the real-time combination of these data. Researchers must first teach the machine learning techniques to detect and precisely classify things in a range of scenarios before using neural networks in drones. This is done by giving the algorithm specifically labeled photographs. These pictures show the neural network what characteristics distinct item classes have and how to recognize one object type from another. Advanced neural networks operate autonomously and keep learning while in use, getting better at detection and processing [7]. Programming a drone to create it is different from programming it to carry out certain tasks. In order to have a drone fly steadily, sensors, actuators, and some sort of CPU must be interfaced. However, all that is required to create an application is a computer that can run algorithms and a drone that is easily accessible. It depends on the gear used to make the drone if the inquiry is about building one. If the Arduino-based controllers are utilized as an illustration, more embedded systems are involved. In order for the controller to interpret sensor data and deliver control signals to actuators in order to create a steady flight, communication between the microcontroller and all other sensors must be established [8, 9].

6.1.2 Unmanned Aerial Vehicle—Drone Technology Due to their low cost and widespread use, UAVs, sometimes known as drones, are now a part of our daily lives. In addition to legal and ethical

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  133

concerns, drone technology is still developing, and many industries are looking into using drones as a replacement to improve difficult operations and save operational costs. Agriculture, gas detection and mapping, delivering medical supplies, search and rescue operations, surveillance, humanitarian aid, and pest and disease control, to name a few, are just a few of the many applications and services that have been proposed. Although drones are a complex assemblage of hardware devices, each one is controlled by a microprocessor with auxiliary sensors that may be integrated into or external to the microprocessor board. Each drone has its own specifications, technical details, and firmware. The technical word for this piece of hardware, flight controller (FC), will be used to refer to these technical features going forward because they are outside the focus of this essay. Each of the necessary flying components that make up the drone system will be essential to its safe functioning. Figure 6.1 shows the architecture’s composition; the sections that follow provide further information on each component [10, 11]. • The drone’s primary communication bus is supported by the drone broker. Additionally, the system contains a relay mechanism that is in charge of directly relaying messages back and forth with the ground systems. • To process and examine the drone’s activity, the flight analyzer connects to flight telemetry data. This system may

Networking Drone Mapper

Physical Connection Drone Broker

Flight Analyser

Fail Safe Systems

Drone Controller

Drone Logger

Figure 6.1  Architecture of a complete drone.

Flight Controller

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

134  Artificial Intelligence for Autonomous Vehicles

•

•

•

•

•

identify abnormalities to actively alert the ground system and attempt to resolve the problem using pattern identification and behavior profiling. It can activate fail-safe systems as a final resort. When activated, the fail-safe system is a device that works to lessen the effects of a failure. These can be quite simple, such as stopping takeoff or requiring a safe landing when a low battery level is detected [12]. The drone logger connects to the necessary broker channels or topics to make a local copy of all drone-related events for registry and debugging reasons. The drone controller, represented by a code file, serves as an adapter design pattern, converting broker command messages into messages that are readable by the FC and distancing the platform from the technical requirements necessary to effectively communicate with the FC. The drone controller, which is represented by a code file, serves as an adapter design pattern, converting broker command messages into communications that are readable by the FC and shielding the platforms from the technical requirements needed to effectively communicate with the FC. In order to accurately represent the command end-point and the inner drones’ remote monitoring (altitude, temperature levels, pressure sensor, motion sensor, velocities, voltage and battery voltages, GPS) various data generator, the FC does have the duty of monitoring the flying drone procedures and correcting its behavior. The drone mapper is an expansion of the drones’ real geofencing capabilities. It offers dynamic map loading based on recent GPS data and a specified radius in direct contact with its second half, the ground mapper. Additionally, it is able to offer more detailed information, such as constraints on the minimum and maximum height that are typical of metropolitan environments.

6.2 Kinds of Drones Used by Medical Professionals Contrary to what its name would imply, a medical drone is no different from any other commercial drone. In other words, these are the same kinds of crafts you would find in the transportation and utility sectors [13, 14].

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  135

Typically, medical IT teams add cameras, sensors, and lights to drones to meet specialized navigation, data gathering, and communication requirements. In the end, there are a variety of drone kinds, each of which is better suited for particular use scenarios [15, 16].

6.2.1 Multirotor Multiple lift-producing rotors are found in multirotor drones. They are suitable for modest medical deliveries, since they are affordable and typically have a carrying capacity of between 50 and 100 lb. These drones are also excellent for data collecting and aerial photography. For these reasons, multirotor drones are frequently used by emergency response teams to assist in rescue missions.

6.2.2 Only One Rotor Single-rotor drones, as their name indicates, contain just one rotor, making them look like miniature helicopters. They can fly for longer stretches of time and are more effective than multirotor drones in terms of operation. But they are frequently also much heavier, more intricate, and more costly.

6.2.3 Permanent-Wing Drones Fixed-wing drones, which resemble airplanes, and fixed-wing hybrids, which combine wings and rotors, are increasingly being used by certain medical professionals. These drones are better suited for long-distance flight.

6.2.4 Drones for Passenger Ambulances A number of businesses are now testing passenger ambulance drones, which can fly patients and medical personnel from one location to another. Undoubtedly, it will take some time before these sorts of aircraft are economically viable for commercial use. But in the future, they may change how emergency medical services are provided.

6.3 Medical and Public Health Surveillance Drones are utilized for disease monitoring, disaster site surveillance, and places with biological and chemical risks. In high-risk situations, it has been demonstrated that drones can gather data on how many patients are in

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

136  Artificial Intelligence for Autonomous Vehicles

need of treatment and prioritize. Drones were deployed in the Philippines during Typhoon Haiyan in 2013 to offer overhead surveillance, assess initial storm damage, and prioritize relief operations. The National Health Service (NHS) in England has looked into the utilization of drones to evaluate injuries connected to biological, chemical, and radioactive hazards in an effort to increase the effectiveness of response teams [19, 23]. Drone technology has been used to identify radiation, aerosols, and heavy metals as potential health risks. Drones outfitted with high-­ resolution photogrammetry tools were utilized in a research from southern Italy to precisely assess and forecast the cancer risk associated with high levels of copper content in agricultural regions which has the capacity to precisely detect aerosol and trace gas concentrations in challenging terrain using a quadrotor drone with an integrated sampling platform. This technology can stop the spread of pathogen-caused health risks through early identification. Along the same lines, drone technology has also been used to map radiation from uranium mines and detect radionuclides that are characteristic in nuclear accidents [25, 26]. Drones are also a promising option for epidemiological research because of their affordability and real-time high-precision temporal and geographical data acquisition capabilities. Tracking of deforestation, increased agricultural production, and many other activities that affect biological populations and natural ecosystems are examples of such an application. In Malaysia, altering land use and deforestation patterns that affect the zoonotic transmission of malarial parasites were characterized using drones by Fornace et al. [29]. In a different example study, Barasona et al. [28] tracked the geographic spread of big animals in southern Spain that carried TB using drones. Staphylococcus aureus and the Ebola virus have recently been found by researchers using drones equipped with nucleic acid analysis modules.

6.3.1 Telemedicine Drone applications in the developing field of telemedicine, which involves treating patients remotely via communications technology, are among the most promising [35]. Telecommunications is the important term in the concept of telemedicine. Unfortunately, commercial networks cannot provide the essential connectivity for telemedicine operation in distant ­disaster-relief and war areas. The senior author (JCR) spoke about the concept of creating instant telecommunication infrastructure (ITI) utilizing drones in 1998 in Athens, Greece, at the Yale/NASA Commercial Space Center Telemedicine Program [27, 28].

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  137

Start

Read GPS For delivery point

Stop

Take off

Fly to reach Delivery point

Return to Base

Take off

Recipient Image Capture

Deliver Drugs

Figure 6.2  Drug delivery flow diagram.

Figure 6.2 displays the drone’s medicine distribution process in a region affected by an epidemic. A purchase order is placed for the region’s urgently required medical supplies. The delivery point’s GPS coordinates are made publicly. The necessary medicine is carried by the drone. A single pill of a malaria medication is fastened to the drone for display reasons. The operator’s primary duty is to control the drone’s navigation; hence, there is no programming code used in this process. The drone was piloted in a wide open area with enough room to avoid hitting any people or buildings. The drone was lowered to the ground after it arrived at the delivery location [29]. A drone platform that specializes in delivering communications for preoperative and postoperative patient assessments and telementoring of some important surgical and clinical procedures in remote locations was carried out with the use of medical drones. With new procedures utilizing computers and telecommunications, telementoring means providing remote assistance by a highly experienced surgeon or a special proceduralist to a less experienced coworker. With the onboard camera, the receiver was captured in a photograph. The medicine was taken by the receiver. The drone was then returned to the base in flight [36].

6.3.2 Drones as Medical Transportation Devices Drones are an appealing medical delivery tool due to their quick reaction times and ability to cross otherwise impenetrable terrain. The findings revealed that drones can aid in medical decision-making by providing quick diagnoses [37]. Drones or UAVs have been introduced globally as a result of the rapid growth of technology to promote mobility over difficult geographic boundaries, reduce carbon emissions, and maybe even raise the cost-effectiveness of healthcare delivery. The time consumption is 25% less when compared to road transport [30].

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

138  Artificial Intelligence for Autonomous Vehicles

Given the inherent requirement for a quick reaction to improve patient outcomes, the use of drones in emergency care is particularly intriguing. Drone use is especially well suited for quick distribution of medical supplies and medications to patients for self-administering [24]. Aerial camera use can also help with rapid evaluation and improve prompt emergency responses. Drones, for instance, can speed up patient assessment and treatment initiation compared to ambulance response alone. Patients in locations with historically lengthy Emergency Medical Services (EMS) response times might particularly benefit from this application [17]. In the case of a medical emergency, drones can offer a first reaction by immediately bringing the best tools on-site, mainly when every second counts [14]. Drones have also been used in emergency care to carry automated external defibrillators (AEDs) to people assisting victims in cardiac arrest. According to a simulation-based study conducted in Salt Lake County, Utah, correctly operated drones may reach 96% of the population in less than 1 min [31]. Conventional ambulance response times, on the other hand, accomplished this goal in just 4.3% of cases. Unluckily, present systems continue to be plagued by issues like high accident rates, airspace rules, and injury control [20]. As a result, further research is needed to improve their overall efficiency and overall performance [18]. The inquiry on the use of drones for patient transportation has been spearheaded by the military. The US Army Medical Research division gave a demonstration of how injured soldiers may be extracted using VTOL drones while avoiding airspace conflicts. Their paper does note, however, that there are still no clear criteria or norms covering the physiological effects of flying on patient health and safety. Unmanned air and ground vehicles were successfully employed in 2015 to react to a simulated distress call reporting a fatality in the field [32]. One individual used an Android tablet to control both unmanned devices [34].

6.3.3 Advanced System for First Aid for the Elderly People There are two key components to the AFAS. The prototype fall detection device (FDD) in the first component is intended to monitor heart rate and find falls. It has a microprocessor, two biosensors for heart rate and acceleration (HB and ACC), a GPS system to track position, and a GSM system to send a message of notice to the smartphones of caretakers at a call emergency center (CEC). The second component involves the CEC giving the patient first help, which includes giving the patient a first aid kit, a smartphone, and a UAV

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  139

Transmitter Node

Receiver Node

GSM Network

Smart Phone

Caregivers

Drone

First Aid Kit

Elderly People

Figure 6.3  Block diagram of first aid delivery system.

to deliver the kit. The smart phone or mobile phone at the CEC has two functions: it is used to plan the UAV’s flight route and to receive communications from the FDD. The FDD, which is linked to the patient’s upper arm and uses the FDB-HRT algorithm to execute monitoring and decision-making tasks, is the central component of the proposed AFAS. The suggested FDD would send signals to the CEC with the patient’s information once it notices a body fall and an irregular HR measurement (ID, health status, and location). This entire process is shown in block diagram (Figure 6.3). Messages are sent to the caregivers in the CEC for display on the smartphone’s LCD. As a result, the first aid kit will be created in accordance with the patient’s condition and delivered to the patient by UAV based on the message’s coordinates. Thus, it was determined that the proposed design significantly cuts down on delivery time as compared to administering first aid by ambulance. In terms of heart rate measurement precision, fall detection, information messages, and UAV arrival time, the suggested enhanced first aid system performed better than earlier systems [21].

6.4 Potential Benefits of Drones in the Healthcare Industry One of the common uses of drones in healthcare is medicine and prescription delivery. These drones can be used to transfer biological specimens such as blood, plasma, and other tissues in some advanced circumstances. In rural and distant areas, accessing healthcare is extremely difficult. To address this problem, drones shall be used to transport pharmaceuticals

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

140  Artificial Intelligence for Autonomous Vehicles

that are unreachable by bus, truck, or train. However, the region must be fully-equipped with a good communication infrastructure. Aside from delivering pharmaceuticals, drones may deliver various biological materials. Drones are one of the greatest options for delivering critical medical equipment and medications in the event of a medical emergency. They can greatly reduce transit time and save the life of critically ill patients at the right time. One of the finest instances of emergency service is organ transfer. Likewise, in the case of cardiopulmonary arrest emergencies, drones are a better alternative than conventional modes of transportation for delivering necessary supplies [36]. Drones can also be utilized for public safety purposes, such as fighting viral illnesses. Drones can also be used to clean regions such as parks, stations, and various public places to combat illness. It may spray sanitizer in an area more accurately and efficiently without the user’s contact participation. Drones’ scope and applicability for cleaning public locations may expand in the future. Drones can also be used for contactless delivery of hazardous materials, monitoring of remote regions, and the supply of other medical supplies. After considering these advantages and applications, drones may be beneficially used in the healthcare delivery system to improve the entire patient experience.

6.4.1 Top Medical Drone Delivery Services Several firms across the world are running trial run programs and planning to commercially use drones for healthcare delivery. Zipline, Volocopter GmbH, Volansi Inc., Novant Health, Matternet, Vayu, SZ DJI Technology, Embention, Manna Drone Delivery Inc., Flirtey, Ehang, Tu Delft, Sky Sports Ltd., Project Wing, HiRO (Healthcare Integrated Rescue Operations), and others are among the market leaders. Given the market potential, numerous new firms with more creative and advanced applications are likely to enter the market in future years.

6.4.2 Limitations of Drones in Healthcare The drone has several restrictions when it comes to the delivery of healthcare, despite its advantageous uses. The drone’s flight may be adversely affected by technological factors such as short battery life, speed, vibration, poor load capacity, g-force, and accuracy. The utilization of drones and their performance are influenced by the frequency and availability of drone landing sites. To fly the drone, a ground pilot must maintain a

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  141

constant communication channel with the vehicle. Additionally, while providing essential medicinal goods, environmental conditions like humidity, abrupt changes in pressure, and temperature must be taken into account. The quality of the medicinal supply might be impacted by unfavorable weather conditions. Similar to this, widespread deployment of drones depends on social approval. The broad deployment may have an impact on the job market and result in unemployment. The consumer’s personal information is necessary for the delivery of the medical supply. Companies must adhere rigorously to data security and compliance to prevent security breaches. People occasionally may believe that their privacy and private property are being invaded by drones, since they are fitted with cameras. Another problem is the carriage’s high economic expense. The development and acceptance of drone delivery systems are being hampered on the legal front by rigorous laws and regulations. There is no standardized procedure for permission or use of drones, and each country has its own set of regulations. Similar to how drones pose a serious hazard to planes, organizations using them must obtain the required authorization from aviation authorities and must adhere to certain strict regulations. Delivery via drones is still in its early stages. With a high success rate, businesses are undertaking test flights across several locations. Zipline and Rwanda’s Health Ministry have worked together to introduce the drone delivery system there. Additionally, Zipline has used drones to carry personal protection equipment to medical facilities in North Carolina. The business is also looking at the possibility in other nations. Merck and Volansi, a drone company, have teamed up to send “cold-chain” medications to isolated areas of North Carolina. In the fight against COVID-19, a UK-based firm called Skyports is helping the NHS and carrying out ­hospital-to-hospital medical supplies. Along with these, a number of other businesses are looking into the utilization of drones in the area of healthcare delivery [22, 33]. The delivery of healthcare might undergo a change in the next few years as a result of the use of drones and cutting-edge technology like telehealth. However, the government must set clear criteria while making sure that medical items are handled properly in order to promote and inspire businesses and other stakeholders. The market for drone deliveries is anticipated to expand in the upcoming year as a result of increased investment, partnerships and collaboration, and technical advances. There is no

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

142  Artificial Intelligence for Autonomous Vehicles

denying that drones have countless applications in the healthcare system, but in order to ensure their widespread adoption in the system, issues with their high cost, laws and regulations, social acceptance, and technological problems must be resolved.

6.4.3 The Influence of COVID on Drones What can we infer about COVID-19 and drones? The drone business does not seem to have suffered financially greatly from the outbreak. The fact that industry executives entered 2020 worried about the consumer market for flying drones for fun had perhaps even aided it. That might not be the case anymore. In a January 2021 study, market analysts at NPD said that sales of consumer drones “soared” during the pandemic, with revenues more than tripling from March to November of 2020 in comparison to 2019. This was likely due to individuals who were imprisoned looking for socially distant hobbies. According to a research on the drone’s hardware market published in April 2021, sales of drones used for agriculture spraying increased by a staggering 135%, with Asia accounting for the majority of the gain. According to DroneDeploy, a significant drone map-making software provider, business in April 2020 was up 130% over April 2019 according to Fast Company. Many drone delivery businesses, like Zipline, seem to have profited greatly from the surge in interest in their services. According to certain estimates, the worldwide drone industry will reach $6.15 billion by 2023, a significant increase from the $3.64 billion revenue recorded in 2020. However, COVID-19 has also made it more difficult for the drone sector to market products that are popularly seen as “normal” and unremarkable due to associations with authoritarian authority. Over the past 10 years, camera-equipped drones have proven to be a crucial tool for a variety of civil endeavors, including ecological research, planning for transportation, disaster management, winemaking, and other activities. Nevertheless, most of the people still have a mistrust for drones and the people who use them. With a land size of 3.28 million square kilometers, cleaning and disinfecting created a significant task and threat for public sanitation personnel using a manual spraying procedure. To reduce the possibility of manual spraying employees and their families becoming infected with the virus, automated sanitation employing drones was implemented.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  143

Small drones are not more fundamentally military than an iPhone, a typical digital camera, or an SUV, all of which are frequently utilized by police, but that is not how most people see them. Unfortunately, the government’s adoption of drone technology for crowd control during the COVID era may have increased the perception of drones as an instrument of oppression rather than making them seem friendly. If the epidemic makes it more difficult to use drones for benign purposes, it would be unfortunate.

6.4.4 Limitations of Drone Technology in the Healthcare Industry Businesses and society will gain a lot from drone technology. However, a variety of considerations, from user privacy and safety to legal concerns and unfair use, add to the drawbacks of drones.

6.4.4.1 Privacy Drone technology has a number of drawbacks even though its advantages are limitless. UAVs are easily exploited and have the potential to invade users’ privacy. Although many people want to use drones to maintain safety, doing so may infringe on many people’s rights in the name of public safety.

6.4.4.2 Legal Concerns Although there are many uses for UAVs or drones, there are also worries regarding misuse and abuse. Many state law authorities still have regulations governing the usage of drones because of these worries. There are currently no laws protecting property against airborne trespassing. UAV technology thus operates in a legal limbo. Government rules and any state or local legislation that regulate airspace property rights are at odds with one another. Many individuals might not enjoy the concept of being observed by the unknown as a drone pilot can fly wherever he or she chooses.

6.4.4.3 Rapid Transit—One of the Biggest Drawbacks of Drones is Time Drones run on rechargeable batteries; hence, they are unable to go great distances. The trip time is further lowered in cases of heavy winds because a drone may fly for up to 30 min. One of the main issues for consumers is the short flying time, which also ranks as one of the largest drawbacks of

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

144  Artificial Intelligence for Autonomous Vehicles

drones. Drones may go up to 5 km in that period, but they must return to their starting point, which takes time.

6.4.4.4 Bugs in the Technology A drone cannot handle lithium-ion batteries, sensors, weight, or air pressure. Additionally, a novice pilot may be more risky due to the need to avoid potential hazards while in the air. The major drawback of drones that carry ammunition is that software or hardware failures might result in many deaths. Drones are continually being developed to reduce incidents that might jeopardize the safety of human beings.

6.4.4.5 Dependence on Weather As long as the weather is sunny and clear, drones are trustworthy. If it is windy or stormy outside, using UAVs to transport goods will not be a suitable alternative. Drone flight is impossible on rainy days because lithium polymer batteries are extremely sensitive to moisture. For eCommerce enterprises, this issue poses a significant obstacle, and they may want to think about utilizing delivery vehicles. Additionally, the fog obscures the field of view, making it challenging to fly a drone. Fog makes it difficult to see and contains small water droplets, both of which might cause the drone’s batteries to malfunction while it is in flight.

6.4.4.6 Hackable Drone Technology One of the main drawbacks of drones, whether used for military operations or commercial purposes, is their vulnerability to hackers. Drones operate using preprogrammed commands and algorithms. Only the Internet system makes it possible to navigate and operate the aircraft. Attacking a drone’s central control system makes it simple for hackers to take control of the drone. Without the original controller’s awareness, they may take complete control of UAVs. Additionally, hackers have the ability to extract confidential information, destroy or damage files, or even expose data to uninvited parties.

6.5 Conclusion The growth of medical drone applications has advanced more slowly than other domains, despite the rapidly growing maturity of applications

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  145

in different sectors like media, transportation, agriculture, infrastructure inspection, and other areas. It should also be remembered that medical applications are more difficult due to the urgency of clinical conditions, which frequently precludes control over date, time, and place. Research on airspace integration, safety, reaction times, participation expansion, and privacy best practices must be accelerated, in order to encourage the secure integration and creative use of drones. The demands of users should be researched and converted into functional requirements and design standards when building technologies that benefit both medical professionals and patients. Important medications are now being delivered to isolated communities and rural populations all over the world with the help of UAVs. When a disease threatens or obstructs road transit, they are frequently the greatest means to supply a product in the shortest amount of time. Delivering patient samples to central laboratories for testing more swiftly is essential during a pandemic like COVID-19 because it enables clinicians to decide on treatments earlier and advances patient results.

References 1. Aabid, A., Parveez, B., Parveen, N., Khan, S.A., Zayan, J.M., Shabbir, O., Reviews on design and development of unmanned aerial vehicle (Drone) for different applications. J. Mech. Eng. Res. Dev., 45, 2, 53–69, 2022. 2. Aabid, A., Parveez, B., Parveen, N., Khan, S.A., Shabbir, O., A case study of unmanned aerial vehicle (Drone) technology and its applications in the Covid-19 pandemic. J. Mech. Eng. Res. Dev., 45, 2, 70–77, 2022. 3. Flemons, K., Baylis, B., Khan, A.Z., Kirkpatrick, A.W., Whitehead, K., Moeini, S., Schreiber, A. et al., The use of drones for the delivery of diagnostic test kits and medical supplies to remote first nations communities during COVID-19. Am. J. Infect. Control, 50, 8, 849–856, 2022. 4. K., H., Khanra, S., Rodriguez, R.V., Jaramillo, J. (Eds.), Machine learning for business analytics: Real-time data analysis for decision-making (1st ed.). Productivity Press, 2022, https://doi.org/10.4324/9781003206316. 5. Mora, P. and Araujo, C.A.S., Delivering blood components through drones: A lean approach to the blood supply chain. Supply Chain Forum: An Int. J., 23, 2, 113–123, Taylor & Francis, 2022. 6. Rajabi, M.S., Beigi, P., Aghakhani, S., Drone delivery systems and energy management: A review and future trends. ArXiv preprint arXiv:2206.10765, 2022.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

146  Artificial Intelligence for Autonomous Vehicles

7. Sanz-Martos, S., López-Franco, M.D., Álvarez-García, C., Granero-Moya, N., López-Hens, J.M., Cámara-Anguita, S., Pancorbo-Hidalgo, P.L., CominoSanz, I.M., Drone applications for emergency and urgent care: A systematic review. Prehosp. Disaster Med., 37, 1–7, 2022. 8. Shi, Y., Lin, Y., Li, B., Li, R.Y.M., A bi-objective optimization model for the medical supplies’ simultaneous pickup and delivery with drones. Comput. & Ind. Eng., 171, 108389, 2022. 9. Munawar, H.S., Inam, H., Ullah, F., Qayyum, S., Kouzani, A.Z., Mahmud, M.A.P., Towards smart healthcare: Uav-based optimized path planning for delivering COVID-19 self-testing kits using cutting edge technologies. Sustainability, 13, 18, 10426, 2021. 10. Vaishnavi, P., Agnishwar, J., Padmanathan, K., Umashankar, S., Preethika, T., Annapoorani, S., Subash, M., Aruloli, K., Artificial intelligence and drones to combat COVID-19. Preprints.org, 2020. 11. Brock, J.K.U. and Von Wangenheim, F., Demystifying AI: What digital transformation leaders can teach you about realistic artificial intelligence. California Manag. Rev., 61, 4, 110–134, 2019. 12. Zaoui, F., Assoul, S., Souissi, N., What are the main dimensions of digital transformation? Case of an industry. Int. J. Recent Technol. Eng. (IJRTE), 8, 4, 9962–9970, 2019. 13. Herrmann, M., Boehme, P., Mondritzki, T., Ehlers, J.P., Kavadias, S., Truebel, H., Digital transformation and disruption of the health care sector: Internetbased observational study. J. Med. Internet Res., 20, 3, e9498, 2018. 14. Rosser, J.C. Jr., Vignesh, V., Terwilliger, B.A., Parker, B.C., Surgical and medical applications of drones: A comprehensive review. JSLS, 22, 3, e2018.00018, 2018 Jul-Sep, doi: 10.4293/JSLS.2018.00018. 15. Yamin, M., IT applications in healthcare management: A survey. Int. J. Inf. Technol., 10, 4, 503–509, 2018. 16. Brady, J.M., Stokes, M.D., Bonnardel, J., Bertram, T.H., Characterization of a quadrotor unmanned aircraft system for aerosol-particle-concentration measurements. Environ. Sci. & Technol., 50, 3, 1376–1383, 2016. 17. Cousins, S., Condoms by drone: A new way to get birth control to remote areas, National Public Radio, Washington, DC, May 19, 2016. 18. Lippi, G. and Mattiuzzi, C., Biological samples transportation by drones: ready for prime time? Ann. Transl. Med., 4, 5, 92, 2016 Mar., doi: 10.21037/ atm.2016.02.03. 19. Priye, A., Wong, S., Bi, Y., Carpio, M., Chang, J., Coen, M., Cope, D. et al., Lab-on-a-drone: Toward pinpoint deployment of smartphone-enabled nucleic acid-based diagnostics for mobile health care. Anal. Chem., 88, 9, 4651–4660, 2016. 20. Pulver, A., Wei, R., Mann, C., Locating AED enabled medical drones to enhance cardiac arrest response times. Prehosp. Emerg. Care, 20, 3, 378–389, 2016.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  147

21. Tang, X.-B., Meng, J., Wang, P., Cao, Y., Huang, X., Wen, L.-S., Chen, D., Efficiency calibration and minimum detectable activity concentration of a real-time UAV airborne sensor system with two gamma spectrometers. Appl. Radiat. Isotopes, 110, 100–108, 2016. 22. Amukele, T.K., Sokoll, L.J., Pepper, D., Howard, D.P., Street, J., Can unmanned aerial systems (drones) be used for the routine transport of chemistry, hematology, and coagulation laboratory specimens? PLoS One, 10, 7, e0134020, 2015. 23. Capolupo, A., Pindozzi, S., Okello, C., Fiorentino, N., Boccia, L., Photogrammetry for environmental monitoring: The use of drones and hydrological models for detection of soil contaminated by copper. Sci. Total Environ., 514, 298–306, 2015. 24. Scott, J.E. and Scott, C.H., Drone delivery models for medical emergencies, in: Delivering Superior Health and Wellness Management with IoT and Analytics, N., Wickramasinghe, F. Bodendorf (eds.), Healthcare Delivery in the Information Age, Springer, Cham, 2020, https://doi. org/10.1007/978-3-030-17347-0_3. 25. Choi-Fitzpatrick, A., Chavarria, D., Cychosz, E., Dingens, J. P., Duffey, M., Koebel, K., Siriphanh, S. et al., Up in the air: A global estimate of non-violent drone use 2009-2015. 2016. 26. Martin, P.G., Payton, O.D., Fardoulis, J.S., Richards, D.A., Scott, T.B., The use of unmanned aerial systems for the mapping of legacy uranium mines. J. Environ. Radioact., 143, 135–140, 2015. 27. Thiels, C.A., Aho, J.M., Zietlow, S.P., Jenkins, D.H., Use of unmanned aerial vehicles for medical product transport. Air Med. J., 34, 2, 104–108, 2015. 28. Barasona, J.A., Mulero-Pázmány, M., Acevedo, P., Negro, J.J., Torres, M.J., Gortázar, C., Vicente, J., Unmanned aircraft systems for studying spatial abundance of ungulates: Relevance to spatial epidemiology. PLoS One, 9, 12, e115608, 2014. 29. Fornace, K.M., Drakeley, C.J., William, T., Espino, F., Cox, J., Mapping infectious disease landscapes: Unmanned aerial vehicles and epidemiology. Trends Parasitol., 30, 11, 514–519, 2014. 30. Wright, V., Dalwai, M., Smith, R. V., Jemmy, J. P., Médecins Sans Frontières' Clinical guidance mobile application: Analysis of a new electronic health tool. Public Health Action, 5, 4, 205–208, 2015. 31. Zailani, M. A. H., Sabudin, R. Z. A. R., Rahman, R. A., Saiboon, I. M., Ismail, A., Mahdy, Z. A., Drone for medical products transportation in maternal healthcare: A systematic review and framework for future research. Medicine, 99, 36, 2020. 32. Beebe, M. and Ret, C., Unmanned aircraft systems for casualty evacuation: What needs to be done, in: Proceedings of the NATO STO-MP-HFM-231 Symposium, Beyond Time and Space, 2013.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

148  Artificial Intelligence for Autonomous Vehicles

33. Agarwal, R., Gao, G., DesRoches, C., Jha, A.K., Research commentary—The digital transformation of healthcare: Current status and the road ahead. Inf. Syst. Res., 21, 4, 796–809, 2010. 34. Beebe, M. K. and Gilbert, G. R., Robotics and unmanned systems–‘Game changers’ for combat medical missions. Proc. NATO RTO-HFM, 182, 2010. 35. Breen, G.-M. and Matusitz, J., An evolutionary examination of telemedicine: A health and computer-mediated communication perspective. Soc. Work Public Health, 25, 1, 59–71, 2010. 36. Harnett, B.M., Doarn, C.R., Rosen, J., Hannaford, B., Broderick, T.J., Evaluation of unmanned airborne vehicles and mobile robotic telesurgery in an extreme environment. Telemed. e-Health, 14, 6, 539–544, 2008. 37. Mendelow, B., Muir, P., Boshielo, B.T., Robertson, J., Development of e-Juba, a preliminary proof of concept unmanned aerial vehicle designed to facilitate the transportation of microbiological test samples from remote rural clinics to national health laboratory service laboratories. South Afr. Med. J., 97, 11, 1215–1218, 2007.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI-Powered Drones for Healthcare Applications  149

An Approach for Avoiding Collisions with Obstacles in Order to Enable Autonomous Cars to Travel Through Both Static and Moving Environments T. Sivadharshan1*, K. Kalaivani1, N. Golden Stepha2, Rajitha Jasmine R.2, A. Jasmine Gilda2 and S. Godfrey3 Vels Institute of Science, Technology & Advanced Studies, Tamil Nadu, India 2 RMK Engineering College, Tamil Nadu, India 3 SRM Institute of Science and Technology, Tamil Nadu, India

1

Abstract

Because it is aware of its surroundings and can, as a result, see them, an intelligent automobile is able to recognize any potential road hazards that it may encounter. In point of fact, an intelligent automobile has to be able to recognize both other automobiles and any possible obstructions in its route, such as pedestrians or bicycles. This is necessary in order for the vehicle to function properly. These next-­generation driver assistance systems are designed to comprehend the circumstances in order to make the highway a safer place for everyone. It has been determined to be of the highest significance for intelligent autos to be able to identify obstacles that are located in the immediate proximity of a host vehicle and provide accurate forecasts of the locations and speeds of such obstacles. Within this framework, a vast number of systems have been designed to deal with the detection of obstacles in a variety of diverse contexts. These systems may be found in a wide range of settings. The locations in which one may find these systems are somewhat varied. On streets that have been organized, many forms of technology, like multisensory fusion and stereovision, are used. When it comes to mapping specific patterns, there are a few different ways to choose from, but they all depend on the identification of possible obstacles (features such as shape, symmetry, or edges). *Corresponding author: [email protected] Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (151–172) © 2024 Scrivener Publishing LLC

151

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

7

The process of stereo matching has many different uses, some of which include the detection of obstructions, the reconstruction of three-dimensional models, the development of autonomous vehicles, and the enhancement of real-­world environments. Keywords:  Multisensory fusion, stereovision, autonomous vehicles, augmented reality

7.1 Introduction 7.1.1 A Brief Overview of Driverless Cars An intelligent car is able to recognize possible road dangers, since it is aware of its surroundings and can therefore perceive them. In point of fact, an intelligent automobile has to be able to recognize both other cars and any potential barriers in its path, such as pedestrians or cyclists. These next-generation driver assistance systems are meant to grasp the situations that are occurring around the vehicle, with the intention of enhancing the safety of the roadway. It has been found to be of the utmost importance for intelligent automobiles to be able to recognize impediments in the near vicinity of a host vehicle and offer precise predictions of their positions and speeds. Case in point: Within this framework, a large number of systems have been built to deal with the detection of barriers in a range of different environments. These systems may be found in a variety of places [1]. Technologies such as multisensory fusion and stereovision, such as those pioneered by Batkovic et al., [2], are used on streets that have been organized. There are a few different approaches to the mapping of particular patterns, and they all rely on the detection of potential impediments (features such as shape, symmetry, or edges). The process of stereo matching is used in a wide range of applications in Belongie et al., [3] such as the detection of obstacles, the reconstruction of three-­dimensional models, autonomous vehicles, and augmented reality. A condensed analysis of the most recent advancements in the area of vision-based obstacle detection is provided in the article titled “Visionbased obstacle detection for outdoor situations.” When it comes to vision-based obstacle detection for an environment’s surroundings, monocular and multicamera techniques are both valid solutions that may be considered. For the objective of identifying obstacles faced by robots, a variety of techniques, including optical flow, were used in both [4] and appearance-based method. Methods that relied only on

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

152  Artificial Intelligence for Autonomous Vehicles

monocular vision were still another option. Bounini et al., [5] judged the obstructions only on the basis of their appearance or the color characteristic they had. Recent times have seen some study being done to detect impediments using three-dimensional reconstruction from a single still photograph. This research was carried out in recent times. Carnegie Mellon [6] featured the publication of “Robust Algorithms and Evaluation” written by YW Xu. However, these algorithms fall short when it comes to estimating an obstacle’s location, velocity, and posture; for a considerable amount of time, this has been regarded as one of the most difficult challenges in the field of computer vision. Computer vision researchers have been working on this problem for a long time. As things stand right now, a number of countries situated in Europe and Asia, in addition to the United States of America, are making significant contributions to the discussion around this topic. Even though these countries are currently at varied stages of acceptability with respect to connected and autonomous automobiles, there is still further work that has to be done before these technologies can be effectively adopted [7] on a large scale. Research in Asia continues to focus a large emphasis on adapting and upgrading the present autonomous driving technologies to the distinctive traffic patterns and special situations that are characteristic of the region. This is something that has been going on for quite some time. The gaps in knowledge that need to be filled in as a result of study are as follows: 1. The fact that the solution that is already in place takes a significant amount of time to implement is the key barrier. 2. The lack of a real-time detection system: As of current moment, there is no solution that is capable of finding individuals and doing detection on the road by using live video. This is a significant limitation. Figure 7.1 shows the driverless car.

gps system

Lidar with camera

ultrasonic sensor radar sensor

Figure 7.1  Driverless car.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  153

3. This is not true for all facets of the technology that is used to operate autonomous vehicles. There is no answer that can deal with the bulk of the problems that have been presented. The vast majority of the currently available solutions are tailored to address specific problems, such as person detection or local area network (LAN) detection. 4. There is no solution that can simultaneously do justice to the management of time and quality, which is the primary challenge faced by those who are tasked with addressing both of these aspects. 5. Accuracy: The bulk of the previously developed procedures has an existing methodology that cannot achieve higher level of accuracy. 6. Quality: There is an issue with the strategy that is currently being implemented, and there is no alternative that can produce a level of quality that is considered to be sufficient. This is due to the extra time and complexity that are needed.

7.1.2 Objectives According to the results of the earlier investigation, there are a great number of gaps in the research that need to be filled; as a result, in this work, the following objectives will serve as our goals, and we will make every effort to complete them: 1. Real-time system: During the course of this project, we are going to construct a system that is capable of providing an intelligent system that is able to make decisions about autonomous automobiles and analyze the real-time analysis by making use of the real-time data collection. In addition to the real-time system, we are also going to construct a system that is capable of providing a system that is capable of providing an intelligent. 2. Raise the overall quality by increasing the overall quality component in the equation. 3. Accurate management of both time and quality while making an effort to find a middle ground between the two aspects of the project. 4. To implement a strategy that has a high degree of both accuracy and precision in its operations.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

154  Artificial Intelligence for Autonomous Vehicles

5. Carry out an examination that compares and contrasts the ways that were explored in the past with the cutting-edge nonintrusive strategy of the portable device that was recommended. 6. The purpose of the ongoing inquiry and the proposed work is based on two separate parameters: the first is a parameter that refers to a static object, such as LAN detection, and the second is a parameter that refers to a dynamic object, such as person detection.

7.1.3 Possible Uses for a Car Without a Driver The completion of a fully functional autonomous vehicle is the result of a variety of interdependent components coming together to make the whole. It is fitted with video cameras that are able to recognize traffic signals, interpret road signs, monitor other vehicles and objects on the road, and keep track of all of them simultaneously. It is equipped with radar sensors so that it is able to monitor the location of the vehicle and determine the state in which it is now operating. LiDAR sensors have the ability to determine with a high degree of precision both the lane lines and the margins of roadways. The car’s wheels have ultrasonic sensors that are able to detect the presence of other cars and are thus equipped with the feature. In conclusion, a centralized control system is a control system that analyzes the information that is obtained from a broad range of sensors in order to manage the directional stability, rate of acceleration, and stopping power of the vehicle. The speed of technological advancement is quickening at a pace that is difficult to keep up with. The amount of money that a nation invests into the study and creation of new technologies directly correlates to the magnitude of the long-term advantages that the nation reaps from doing so. These incentives consist of a wide variety of things, such as upgrades to the underlying infrastructure as well as increases in job possibilities. One of the key motivations for the creation of new technologies in today’s society is the desire to lessen the amount of physical labor that is necessary for performing daily tasks. Machine learning, artificial intelligence, and computer vision are the most essential components for the success of this endeavor.

7.2 Related Works The V-disparity and G-disparity pictures were created by Crowe [8] with the intention of automatically finding obstacles by an evaluation of the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  155

disparity between the ground plane and the image. Since the turn of the century, technologies that allow autonomous driving have been created, and these technologies have the potential to make driving much safer and more economically viable. Although it is reasonable to anticipate that the autonomous driving technologies described in Dalal et al., [9] will initially be implemented in structured environments such as highway driving and low-speed parking, it is arguable that other scenarios, such as urban driving, present a greater challenge because of the presence of nonautonomous road users such as pedestrians, cyclists, and other vehicles. For example, it is reasonable to anticipate that the technologies described in Dalal et al., [9] will initially be implemented in structured environments such as highway driving and low-speed parking. As a consequence of this, the focus of research has to be turned toward driving models that not only have the ability to anticipate the erratic behavior of other human road users but also have the ability to avoid collisions with such persons. The fall in cost of integrated micro-electro-mechanical systems (MEMS) sensors over the course of the last couple of decades has been one of the key developments in enabling technologies that have greatly contributed to the expansion of advanced driver assistance systems (ADAS). Additionally, the development of ADAS [10] and intelligent vehicles is aided by essential computing resources and memory that are not unreasonably costly [11]. These technologies make it feasible to enhance road safety and give answers to some of the issues that occur with traffic. They also make it possible to supply remedies. There are a number of challenges that must be overcome before fully autonomous vehicles can become a reality. One of the most important of these challenges is tied to issues with navigation in situations that are unpredictably dynamic or not static. Despite this, artificial intelligence and computer vision provide prospects for future solutions to challenges such as the navigation of autonomous automobiles in an unstructured environment, the analysis and classification of scenes, and other activities of a similar kind. One of the solutions that are presented in Forsyth et al., [11] is a system vision that could be based on either one monocular camera with morphological image processing [12], fusing road geometry and B-Snakes [13], or several cameras for advanced processing such as interobject distance estimation and 3D object reconstruction [12]. Fusing road geometry and B-Snakes [13] is another one of the solutions that are presented. Both of these choices are viable ones. Nevertheless, one of the most difficult challenges that driverless automobiles face is the task of recognizing the lanes of a road. The overwhelming

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

156  Artificial Intelligence for Autonomous Vehicles

majority of this may be attributed to the enormous amount of data that have to be processed in real time. The algorithms should be able to recover the current condition of the road in addition to any uncertainties that may be present. Some examples of these uncertainties are shadows, vehicle shaking, sensor noise, Dollar et al., [14], and other similar factors. In order to get over these restrictions, new approaches need to be coupled with traditional image processing algorithms in order to improve the performance of the latter, better target the region of interest, and minimize the amount of computation that is necessary for real-time applications. The progression of new technology throughout a wide range of scientific fields has made the process of living a human life much less difficult, more convenient, and less demanding. The growth of embedded technology [15] inside the automotive industry helps the enhancement of the quality of human existence by making it safer and more comfortable for people to live. Recent research [15] indicates that there are over 1.3 million people who lose their lives in India due to accidents involving motor vehicles on an annual basis. The task that involves the most moving parts and presents the greatest challenge in terms of maintaining the safety of passengers and cars on the road is the detection of impediments in real time. The very first obstacle detection system was developed by Delco System Operations in Goleta, California, in the year 1988. This business is considered a pioneer in the field of obstacle detection. The primary function of this system was to serve as a safety device that monitors the path ahead of the vehicle and alerts the driver to any possible dangers that it detects. In addition to that, this system was able to determine whether or not there were any moving objects on the lane to the right of it. After the completion of the installation of this system, the automotive sector is anticipated to adopt a cutting-edge approach to the detection of objects that will include the use of infrared sensors, radar sensors, and ultrasonic sensors. The ever-increasing need for integrated technology in the automotive industry has led to the creation of a better and more reliable safety feature that protects not only the driver but also the passengers. The use of a number of different obstacle detection systems leads to the supply of safety measures as well as an improvement in the overall efficiency of the transportation sector [16]. Autonomous vehicle technology, which consists of a variety of sensors to identify impediments in front of, to the side of, and behind the vehicle, is currently standard in the vast majority of vehicles currently operating on public roadways. This technology is installed in the majority of vehicles currently on the road. The fundamental purpose of the work that is described in this thesis is to make a contribution to the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  157

identification of obstacles in a vehicle’s lateral blind spot and in front of the vehicle. This contribution will be made via the use of a camera system. The driver will get a warning from the system, which will give them the opportunity to use the brakes or turn the wheels in order to avoid getting into an accident [17]. Because ultrasonic sensors can detect an object that is extremely close to the vehicle, have a quick reaction, and create an exact distance between the obstacle and the vehicle, it is proposed that this technology implants ultrasonic sensors for the purpose of detection. This is because ultrasonic sensors can detect an item that is extremely close to the vehicle. In the earlier attempts made at obstacle detection, infrared sensors were used. These proximity sensors were used widely in a variety of applications with the primary goal of preventing collisions with various obstructions. The error in the distance that was measured is due to the fact that infrared sensors, which are also known as IR sensors [18], have a behavior that is comparable to nonlinear behavior, and the fundamental concept relies on the reflection from objects in the surrounding area. This led to the error in the distance that was measured. Because of this, one could not rely on the readings that these sensors provided as being accurate. As a result, the closest distance that can be accurately measured with these sensors is 25 cm at the earliest. Image processing and computer vision technologies are also being used in the improvement of pedestrian safety [19] as well as pedestrian detection. This is the most challenging task, since rapid processing is necessary in order to warn the driver as soon as it is practically possible to do so. It is essential that the pedestrian be identified in each and every single frame in which they are visible. The system that relies on image and vision technology does, in fact, have a few shortcomings that need to be addressed. In areas with unfavorable weather circumstances, such as thick fog, severe weather, and significant precipitation, the approach cannot be relied upon. There is a possibility of the algorithm making mistakes when it is trying to differentiate between shadows and pedestrians. The system requires cameras with a high resolution, and the installation of such a system is a tough endeavor, since it results in inaccuracy due to the damping and vibrations that are created by the automobiles. The system, however, requires high-resolution cameras in order to function properly. Image processing is one of the key driving causes behind the growth of automation, security, and safety-related applications in the electronic sector [20]. The vast majority of approaches to image processing require the completion of a series of phases, such as “treating the picture as a two-­ dimensional signal and applying traditional signal processing techniques

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

158  Artificial Intelligence for Autonomous Vehicles

to it” [12]. Images are also processed as three-dimensional signals, with either the time axis or the z-axis acting as the third dimension in these processing operations. If embedded systems and image processing are combined in the process of developing an application, it is possible that the resulting solutions will be highly effective, need a little amount of memory, and be dependable. This will bring out the benefits of both of these technologies. One of the companies worth a billion dollars, Google is one of the companies that has unveiled its own autonomous vehicle. The design of this car does away with all of the conventional controls, such as the steering wheel, and also highlights a number of other astounding technical developments. One of the most important technologies that Google has included in their autonomous vehicle is called LiDAR, which is an abbreviation that stands for “light detection and ranging” [21]. Google has included a large number of other cutting-edge technologies in their car, in addition to image processing, which is an impressive development. Lasers are sent into the atmosphere by a device that may take the shape of a cone or a puck and sends them out. By having these lasers reflect off of the many things found in the environment, a high-resolution map of the environment may be constructed in real time.

7.3 Methodology of the Proposed Work In order to organize the ground focuses and simulate the slope of the road in various situations, a continuous gourd discovery system that is reliant on a single LiDAR sensor has been presented [22]. In the process of information preparation for applications involving autonomous vehicles, dependable and skilled ground division plays an important role. This is because it has the potential to reduce the amount of information that needs to be handled and, as a result, the overall amount of time spent performing computations. In order to deal with these characteristics, a unique technique that is based on the probability inhabitance matrix guide has been developed in order to achieve high exactness in addition to high proficiency. The structure is composed of three different submodules, which are as follows: (1) knowledge procurement and inclining, (2) probability inhabitance lattice guide showing, and (3) weighted direct relapse. The piece of information that each edge is responsible for either preparing or processing is the component that serves as the framework’s focal point. The continuing analyses have been directed in such a way so as to validate the appropriateness and efficacy of our suggested framework. The recognition system for autonomous cars often requires a variety of sensors in order to

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  159

get information. These sensors may include LiDAR, radar, cameras, and other similar devices. An outstanding challenge for ongoing handling is to be able to interpret the enormous amount of information that has been acquired from all of these many sensors [23]. At the moment, ground division is an important project that is being undertaken in the center of the road in order to remove repeated information and to reduce the complex nature of calculations. The task of ground division is achieved by dividing the available information into several groups that will be used for different purposes, such as following, anticipation, or a driving assistance system. Over-division, under-division, or moderate division are the three primary problems that arise with division frameworks [24]. The presentation of the recognition framework is impacted as a result of these concerns [25]. At the conclusion of the day, having a reliable, proficient, and commotion-free ground division calculation is of an amazing importance for the steadfast quality and computationally diverse nature of the framework. It is suggested and carried out that a continuous approach for ground division be used in order to efficiently and strongly measure the ground position and tilt. This is accomplished by taking points of interest of the geometry of the rooftop-mounted LiDAR sensor. In order to achieve high levels of accuracy in the findings as well as productive computation, the suggested continuing framework relies on the probability inhabitance lattice map and weighted straight relapse for ground division. The raw LiDAR [26] point cloud is separated into a low twisting informative index and a high twisting informational collection because of the fact that focuses from closer proximity have, on average, less twisting. The information obtained from the low bending is applied in the process of building up the ground estimate model. A ground model is responsible for a piece of the high contortion informational index’s calculation [27]. The ground estimate model is produced by registering crude back assessments from an inhabitance matrix map. This solves the problem by providing maps that talk to the earth based on noisy and questionable sensor estimation information. An informative collection with a low contortion is used in conjunction with a recursively weighted direct relapse to determine the slope of the roadway. A significant amount of the raw information that was collected from LiDAR by way of UDP bundles was converted from circular directions to Cartesian directions as stated. This was done in order to portion the ground focuses. A naive strategy may get rid of all of the foci that are below a

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

160  Artificial Intelligence for Autonomous Vehicles

specific preset stature. Using the random sample consensus (RANSAC) for fitting models in the vicinity of multiple information anomalies is a technique that provides a slightly improved method for fitting an even plane. This method is performed via the usage of the RANSAC. The RANSAC computation is quite easy to understand, and it consists of the following four steps: (1) Choose non-standard instances that do not match the model. (2) Construct the model using the data from the test set. (3) Using the comprehensive informative index, figure out how the inliers to this model should be arranged. (4) Repeat stages 1 through 3 until a model is discovered that has the most outliers relative to the total instances. Figure 7.2 shows the flow diagram of ground surface detection. The decoupled equation for forward motion can be written as follows:



(m Xa )u X uur

X| | ur | ur

1



where t1 is the control force due to propeller in surge direction, and the linear surge model at the given operating speed u0 is given as follows:



(m Xi )uo

data acquisition

occupancy grid

X|| | u0 | u0

1



point projection

yes

detect grid features

X uu

point inside map

no

belong to ground no

Figure 7.2  Flow diagram of ground surface detection.

linear regression

segmentation linear model

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  161

The thrust force is calculated as a function of propeller rpm and vessel speed by performing systematic runs in the towing tank with free running self-propelled forward speed tests. The decoupled motion in sway and yaw linearized about the constant speed can be represented as follows:

Mv + N (uo) vr = bδr Where m Yi mx g N i

M



m Yi

0

0

1 0

mx g

Ni

mx g Yr , N (uo ) Iz Ni mx g Yr 0 Iz Nr

Yr ( Xu Yi )ua N v

(m Xv )uo Yr and b (mx g Yi )uo N r

v 

Yv 0

0 0

m Xu ua Yr 1

v

Ys 0

r

Nv

0

mx g Yr u0 N r

r

Ns

Yd , Ns

(3.15)

r



Ys = Yas + Yρs, Ns = Nss + Nps is the combined force and moment due to the twin rudder system and δ is the rudder deflection. The prime system is mostly used in ship maneuvering, whereas the BIS can be used for zero speed as in the case of dynamic positioning. The difference between the two is illustrated below: In the prime system: Linear velocity is U In the BIS: Linear velocity is Lg The nondimensional expression for the time constants and gain constants using prime system are given as follows:



K

L K ,T1 U

U T1 ,T2 L

U T2 ,T3 L

U T3 L

where U denotes the instantaneous speed and L is the length of the ship between fore and aft perpendiculars. The first-order ship dynamics expressed in nondimensional form is given by the following:

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

162  Artificial Intelligence for Autonomous Vehicles



L T r r U

U K L



This representation is useful since the nondimensional gain and time constants will typically be in the range 0.5 K N 2 and 0.5 < T < 2 for most ships. The parameter KN and T are obtained by performing well-defined maneuvers such as the turning circle test and the zig-zag test using the self-propelled model.

7.4 Experimental Results and Analysis This is a generic method for image processing and signal processing, and the homomorphic filter is an essential picture enhancement approach that is based on the frequency domain, involving the use of nonlinear mapping to transfer data to a different domain, followed by the use of linear filtering methods and a subsequent mapping back to the initial domain. A homomorphic filter will simultaneously boost the contrast of a picture while also leveling the brightness of the image. Gaussian Noise The normal distribution, also known as the Gaussian distribution, is referred to as Gaussian noise. Gaussian noise is a kind of statistical noise that has a probability density function that is equal to the normal distribution. The most significant contributor to the presence of Gaussian noise in digital photographs is the process of image capture itself. The Gaussian noise may be decreased in digital picture processing by making use of spatial and frequency filters; however, this may have unintended consequences, such as the blurring of finely sealed image edges, and correlates to high frequencies. The probability density function name p of a Gaussian random variable z is given by the equation p G (Z) = 1/(2) e((z-)2)/(22); this equation may be found by using the notation. Where Z represents the amount of gray, a represents the mean value, and s stands for the standard deviation. When used for the homomorphic filtering approach, the process is referred to as either Gaussian homomorphic filtering or Gaussian noise-based homomorphic filtering.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  163

In this specific setting, the process of calculating the logarithm of the picture intensity makes the filtering of light and reflectance into an additive operation. Therefore, the components of the picture may be expanded, and it is presumed that the low-frequency components will primarily represent the reflectance in the image. It is generally accepted that the low-frequency components of a picture comprise the majority of the image’s lighting. It is possible that the role of the homomorphic filters will be to reduce the low frequency while simultaneously raising the high frequency. Again, with regard to the application at hand, the homomorphic filtering provides the lower gray-level enhancement that is necessary to increase the quality. In the log intensity domain, the high-pass filter is used so as to reduce the low frequency while simultaneously amplifying the high frequency. I(x,y) = I(x,y)+n(x,y), where n is the noise, as determined by the results of the additive noise method that is currently being used by researchers. However, there is still a significant amount of research being conducted, such as the multiplicative models I(x,y) = I(x,y)n(x,y). The nonuniform lighting shifts seen in photographs may be easily fixed with this homomorphic approach, which is why it sees widespread use. The illumination reflectance of the image creation is composed of the intensity at each given pixel, which is the quantity of light that is reflected by a dot on an item in an image. It is the result of the lighting of the scene being combined with the reflectance of the item that is present in the scene.

I (x, y) = L (x, y) R(x, y) The car maintains a high rate of speed as it travels down the lane in step 1. Step 2: The north sensor block locates an obstruction in front of the vehicle using its sensors. Determine the distance that separates the vehicle and the barrier that is in the front of the vehicle. If the distance is within the range of what is considered to be a safe distance, then you should gradually slow down by lightly using the brakes. Step 3: Using the RF module to communicate with the vehicle that is acting as an obstruction, determine the speed that it is traveling at. Keep an eye on how fast the car in front of you is going during the process of avoiding a collision. Step 4: Determine whether or if there are opportunities to change lanes. First, keep an eye on the lane on the left. In the fifth step of the process, the left sensor block, the northwest sensor block, and the southwest sensor block search the left lane for any potential obstructions. Step 6: If there is no impediment found in the left lane, the automobile will be moved proportionally to the left lane

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

164  Artificial Intelligence for Autonomous Vehicles

if it is controlled proportionately. Step 7: If there is an obstruction found in the left lane, then it is necessary to examine for opportunities to switch lanes to the right. In the eighth step, the northeast sensor block, the east sensor block, and the southeast sensor block search the right lane for any potential obstructions. Step 9: If there are no obstacles found in the right lane, the angle of the automobile will be adjusted proportionally so that it may go into the right lane. Step 10: If an obstruction is also found in the right lane, go to the next step and check the distance between the vehicle and the obstruction in front of the vehicle once again. Step 11: If the distance is once again within the range of acceptable distances, proceed to repeat steps 4 through 10.

7.5 Results and Analysis Forty megahertz is the clock frequency of the CPU that is being utilized. The time period, often known as the clock cycle, is thus 25 ns. Because of this, the update is performed about once per few tens of nanoseconds. As a result, an approximation of the distance is created and maintained about every few tens of nanoseconds. If the number of updates that take place each second is minimal, then the distance estimates could be off, and the system would not be able to prevent collisions, which would lead to accidents and financial losses. Because it performs an update on the 264 register every few tens of nanoseconds, the 40-MHz CPU is sufficient for avoiding collisions. Since the delay was not there in the system, there was not a collision because of this reason. With the passage of time, technological advancements continue, and new CPUs with lower clock cycles and higher frequencies are introduced into the market. Within the microcontroller that is being used, the software code is carried out in a sequential fashion. As a consequence of this, the execution of each instruction requires a certain quantity of clock cycles. In this instance, Image 1

0

Image 2

0 100

100

200

200

200

300

300

300

400

400

400

500

500

500

600

600

700

0

200

400

600

800

1000

1200

700

Image 3

0

100

600

0

200

400

600

Figure 7.3  Input and the obstacle detection.

800

1000

1200

700

0

200

400

600

800

1000

1200

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  165

Image 1

0

Image 2

0

100

100

200

200

300

300

400

400

500

500

600

600

700

700 0

200

400

600

800

1200

1000

0

200

400

600

800

1000

1200

Figure 7.4  Image processing stage 2.

just a few clock cycles have passed, and each clock cycle has lasted for 25 ns. The value from the sensor is read at the beginning of each iteration of the code, but it is not read again until after all of the instructions that came before it have been carried out in the same cycle. Figure 7.3 and Figure 7.4 shows the image processing stage 1 and stage 2. A study was done to determine whether or not it would be possible to use IVC for CAS, and the results of that study were successfully verified using the LabVIEW simulation environment. After that, distance was measured by using a received signal strength indicator (RSSI), which turned out to be nonlinear. Because of this, it was necessary to use roadside infrastructure for distance measurement, identification of vehicles, and timing of crossing of a particular location, as well as initiating actions to avoid collisions. It was proven that the testing of RF Modules for IVC in a CA environment was successful. This topic was covered. It was claimed that CA will use an infrastructure-based methodology. A successful prototype demonstration has been carried out utilizing 8051 Development Boards and personal PCs. The use of IVC for CA was at long last put into action making use of prototype Freescale Smart automobiles. It has been hypothesized, following research into a number of different collision avoidance systems, that the intervehicular communication could be a workable solution to the issue of collision avoidance in futuristic vehicles. Figure 7.5 shows the image processing stage 3 results. This is primarily Car ch 2

0

Car ch 2 features

0

Not Car ch 2

0

10

5

10

5

20

10

20

10

30

15

30

15

40

20

40

20

50

25

50

25

60

30

60

0

10

20

30

40

50

60

0

5

10

15

20

Figure 7.5  Image processing stage 3.

25

30

Not Car ch 2 features

0

30 0

10

20

30

40

50

60

0

5

10

15

20

25

30

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

166  Artificial Intelligence for Autonomous Vehicles

due to the fact that it is inexpensive, has universal applications, and can be easily integrated with ITS. Figure 7.6 to Figure 7.8 shows the final output results of proposed work. In addition, IVC serves as the central support structure for intelligent transportation systems. It is conceivable that many issues pertaining to ITS may be resolved by using this essential component of ITS as a device for the prevention of collisions. It is anticipated that ITS would have an exponential growth throughout the course of the next years. As a result, an extremely efficient solution for the prevention of collisions would be a system that is included in ITS [28], has the potential to be cost-effective, and is capable of effectively avoiding collisions. Compared to radar systems, the CAS that is based on wireless communication is easier to implement and has a lower level of complexity. However, this is very contingent on the availability of all other automobiles (market penetration).

Original

0

Result

0

100

100

200

200

300

300

400

400

500

500

600

600

700

700 0

200

400

600

800

1000

1200

0

200

400

600

800

Figure 7.6  Image with the car detection.

0 100 200 300 400 500 600 700 0

200

400

600

Figure 7.7  Image processing background reduction.

800

1000

1200

1000

1200

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  167

0 100 200 300 400 500 600 700 0

200

400

600

800

1000

1200

Figure 7.8  Final result analysis.

This can be achieved by using the hardware and software of ITS. The IVC-based CAS covers a wide range comparable to radar. Therefore, IVC that is an inherent and integral backbone element of ITS has been considered to reduce the hardware complexity [29] and cost and increase the speed of sensing and control, thereby the cars become economical and eligible to be a part of ITS.

7.6 Conclusion This study was conducted with the intention of developing an automated parallel parking system that would be capable of parking a car in reaction to the dynamics of its surrounding environment and would be adaptive enough to take into account the dimensions of the vehicle being parked. A parking controller that makes use of fuzzy logic is the primary element that makes up our system. Its core design consisted of classical route planning, and its parking arrangement was designed to follow a fifth-order polynomial path. Its name comes from the fact that the path it followed was a polynomial of the fifth order. The configuration of its parking lots was planned. It is possible to construct smooth trajectories that are constrained in a non-holonomic way. The design of this controller was quite stable, and it was able to park the car in any orientation and starting place. The ultrasonic sensor system that has been built so far is capable of providing accurate information in all of the horizontal directions surrounding the vehicle. When compared to other sensing technologies, one way to generate heterogeneity is by employing a high number of sensors, while one way to achieve homogeneity is by utilizing sensor groups. To the greatest extent that is humanly feasible, the orientation and location of the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

168  Artificial Intelligence for Autonomous Vehicles

sensors are kept in optimal condition. A sensing method simply required to do arithmetic operations in order to find the data that it had felt, as a consequence, it was computationally efficient and rapid. The incorporation of traditional knowledge of mobile robot navigation inference along with sensor information into already existing parallel parking controllers has been determined to significantly improve the usability of those controllers in dynamic and uncertain environmental conditions. This conclusion was reached on the basis of qualitative analysis and simulation results provided for various proposed approaches. Following an investigation into, as well as a simulation of, the outcomes of each of the many different ways that were suggested, this conclusion was reached. The suggested architectures that we have built make use of a hybrid strategy that blends fuzzy logic-based parking controllers with fuzzy logic-based navigation design concepts. This approach was taken in order to maximize efficiency and effectiveness. This method outperforms any and all other parking algorithms that have ever been developed to a level that is considered to be adequate. Autonomous parking systems that are already in use could reap the benefits of a novel obstacle avoidance architecture. This architecture, which integrates a number of controllers to provide humanlike intelligence when working in the presence of a stationary or moving obstacle, could be of use to existing parking systems. In addition, thanks to this construction, there is no possibility of a collision taking place while the parking operation is being carried out. The definitions of the system’s parameters may be derived from the margin of safety, which is a generalization of the system’s parameters themselves. Thanks to the decision fuzzy controller that is included into our suggested architecture, it is feasible to conduct fuzzy behaviors such as target steering, obstacle avoidance, and wall following as and when they are necessary for the execution of CLMR parking.

References 1. Aurelien, G., Hands-on machine learning with scikit-learn and tensorflow, 1st ed., O’Reilly Media, Sebastopol, California, 2017, https://oreilly.com/catalog/ errata.csp?isbn=9781491962299. 2. Batkovic, I. et al., Real-time constrained trajectory planning and vehicle control for proactive autonomous driving with road users. Proc. Eur. Control Conf. (ECC), 2019. 3. Belongie, J.M.S. and Puzicha, J., Shape matching object recongition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 24, 24, 509– 522, 2002.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  169

4. Boden, M.A., Mind as machine: A history of cognitive science, p. 781, Clarendon Press, India, ISBN 978-0-19-954316-8, 2006. 5. Bounini, F. et al., Autonomous vehicle and real time road lanes detection and tracking. 2015 IEEE Vehicle Power and Propulsion Conference (VPPC), IEEE, 2015. 6. Carnegie Mellon. Navlab, The carnegie mellon university navigation laboratory. Robotics Institute, 15, 759–762, 2014, Retrieved 20 December 2014. 7. Colquitt, J. and Dowsett, D., Driverless cars: How innovation paves the road to investment opportunity, 4, 894–898, 2017. 8. Crowe, S., Back to the future: Autonomous driving in 1995 - robotics trends. roboticstrends.com, 2015. Retrieved 2 March 2017. 9. Dalal, N. and Triggs, B., Histograms of oriented gradients for human detection. Conference on Computer Vision and Pattern Recognition (CVPR), 2005. 10. Dalal, N. and Triggs, B., Histograms of oriented gradients for human detection. IEEE Proc. CVPR, pp. 886–893, 2005. 11. Forsyth, D.A. and Ponce, J., Computer vision, A modern approach, Prentice Hall, Italy, ISBN 978-0-13-085198-7, 2003. 12. Ballard, D.H. and Brown, C.M., Computer vision, Prentice Hall, Germany, ISBN 978-0-13-165316-0, 1982. 13. Daily, M. et al., Self-driving cars. Computer, 50, 12, 18–23, 2017. 14. Dollar, P., Wojek, C., Schiele, B., Perona, P., Pedestrian detection: An evaluation of the state of the art. IEEE Trans. PAMI., 34, 4, 743–761, 2012. 15. Duyoung, H., Lee, E., Byoung, C.K., Pedestrian detection at night using deep neural networks and saliency maps. J. Imaging Sci. Technol. R, 61, 6, 97–101, 2017. 060403-1_060403-9, 2017. Society for Imaging Science and Technology. 16. European roadmap smart systems for automated driving (PDF). EPoSS, 2015. 2015. Archived from the original (PDF) on 12 February 2015. 17. Felzenszwalb, P. and Huttenlocher, D., Pictorial structures for object recognition. Int. J. Comput. Vis. (IJCV), 61, 1, 55–79, 2005. 18. Gavrila, D.M., The visual analysis of human movement: A survey. J. Comput. Vis. Image Underst. (CVIU), 73, 1, 82–98, 1999. 19. Gavrila, D.M. and Philomin, V., Real-time object detection for smart vehicles. Conference on Computer Vision and Pattern Recognition (CVPR), 1999. 20. Garethiya, S., Ujjainiya, L., Dudhwadkar, V., Predictive vehicle collision avoidance system using raspberry-pi. ARPN J. Eng. Appl. Sci., 10, 8, 2015. 21. Gansbeke, W.V., Brabandere, B.D., Neven, D., Proesmans, M., Gool, L.V., End-to-end lane detection through differentiable least-squares fitting, in: ICCV Workshop, 2019. 22. Gehrig, S.K. and Stein, F.J., Dead reckoning and cartography using stereo vision for an automated car. IEEE/RSJ International Conference on Intelligent Robots and Systems, Kyongju, vol. 3, pp. 1507– 1512, ISBN 0-7803-5184-3, 1999, doi:10.1109/IROS.1999.811692.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

170  Artificial Intelligence for Autonomous Vehicles

23. Hang, X., Wang, S., Cai, X., Zhang, W., Liang, X., Li, Z., CurveLaneNAS: Unifying lanesensitive architecture search and adaptive point blending, in: European Conference on Computer Vision (ECCV), 2020. 24. Huang, T. and Vandoni, Carlo E., Computer vision: Evolution and promise (PDF). 19th CERN School of Computing, Geneva, CERN, pp. 21– 25, ISBN 978-9290830955, 2019, doi:10.5170/CERN-1996-008.21. 25. Iménez, F. et al., Advanced driver assistance system for road environments to improve safety and efficiency. Transp. Res. Proc., 14, 2245–2254, 2016. 26. Jähne, B. and Haußecker, H., Computer vision and applications, a guide for students and practitioners, Academic Press, ISBN 978-0-13-085198-7, 2000. 27. Kanade, T., Three-dimensional machine vision, Springer Science & Business Media, USA, ISBN 978-1-4613-1981-8, 2012. 28. Kanade, T., Autonomous land vehicle project at CMU. Proceedings of the 1986 ACM Fourteenth Annual Conference on 5 Computer Science - CSC ‘86. CSC ‘86 Proceedings of the 1986 ACM Fourteenth Annual Conference on Computer Science. CSC ‘86, pp. 71–80, 1986. 29. Klette, R., Concise computer vision, Springer, California, ISBN 978-1-44716320-6, 2014.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Colloison Avoidance in Autonomous Vehicles  171

Drivers’ Emotions’ Recognition Using Facial Expression from Live Video Clips in Autonomous Vehicles Tumaati Rameshtrh1*, Anusha Sanampudi1, S. Srijayanthis1, S. Vijayakumarsvk1, Vijayabhaskar1 and S. Gomathigomathi2 RMK Engineering College, Tamil Nadu, India Sairam Engineering College, Tamil Nadu, India 1

2

Abstract

The goal of this research is to catalog the myriad of emotions that are experienced by people and to determine a person’s state of mind by observing his or her behavior and drawing conclusions from those observations. Anger, sadness, fear, pleasure, disgust, surprise, and neutral are some of the facial emotion classes that may be recognized by utilizing facial expression recognition, often known as FER. Finding the database in the emotion by applying the Fluorescence-activated cell sorting (FACS) action unit (AU), such as impression investigation, diagnosing depression and behavioral disorders, lying detection, and (hidden) emotion identification, among other things. A deep convolutional neural network (CNN) approach was used in order to realize the goal of recognizing different facial expressions. The technique that was recommended puts a significant focus on achieving a high degree of accuracy when determining the feelings that are communicated in live video footage. Keywords:  Image processing, facial expression, emotion recognition, deep learning, autonomous vehicle

*Corresponding author: [email protected] Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (173–192) © 2024 Scrivener Publishing LLC

173

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

8

8.1 Introduction With the use of FACS, we are able to determine a person’s mental condition and provide appropriate recommendations. This analysis of different facial expressions provides a number of methods for evaluating current emotions in the here and now (facial electromyography [FEMG] is another option). Research that was carried out by Paul Ekman in the early 1970s led to the discovery of six different facial expressions that are common to all human cultures. Disgust, wrath, happiness, grief, surprise, and fear are some of the feelings [1] that fall under this category. These behaviors could be discovered after doing a thorough investigation of the profile indicator. For example, bringing the inner corners of one’s eyes together while simultaneously elevating the corners of one’s lips might be seen as a sign of pleasure. Facial expression recognition (FER), which is one of the most practical methods to integrate nonverbal information with various types of analytics due to the fact that it may provide some insight into a person’s emotional state as well as his or her objectives, is one of the methods to integrate nonverbal information. The only assignment that can be finished using our suggested technique, which is based on convolutional neural network (CNN)-based attribute styles [2], is finding out the emotions that each person in the video clip is experiencing throughout the different frames. This is the only task that can be done using our suggested strategy. We are currently gathering the data from the video clip in order to figure out the feelings experienced by each participant. The number of heads that are necessary to have specific knowledge is dropping and so is the number of variables that need to be skillfully worked out. In addition, the number of heads that are required to have certain information is falling. It is possible that the algorithm may have been enhanced if it had been carried out in a different manner. This article presents a method for identifying the FER that is centered on CNN [3], and it is presented by us to the reader. As the input, the video or movie clip is utilized, and then CNN is used to observe human activities and their behavior. The Virtual Focal Reality (VFR) technology is a kind of computer vision technology that collects considerable data about the video sequences that are included inside a single frame. This information may then be analyzed. When the VFR system is operational, a variety of businesses, including RFID [4], will be interested in the field of system recognition and computer vision due to the widespread application of RFID technology in the fields of (radio frequency identification) cards, smart cards, surveillance systems, pay systems, and access control. When the VFR system [5] is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

174  Artificial Intelligence for Autonomous Vehicles

operational, RFID will also be interested in the field of system recognition and computer vision. The development of autonomous vehicles (AVs) [6] has taken place in phases over the course of the last couple of decades using a wide range of technology. These may include proximity warning systems, monitoring of the vehicle’s operation, autonomous cruise control, detection and avoidance of impediments on the road, management of the brakes, aid with driving and parking, monitoring of the vehicle’s operation, and more. In this chapter, the concept of “automated vehicle parking” is dissected into its component elements in order to better understand it. The ability of a vehicle to park itself without any assistance from a human from its starting location all the way to its destination position while maintaining the correct orientation is the definition of autonomous vehicle parking. This capability must be maintained while the vehicle is traveling from its starting location to its destination position. The many ways that automobiles may be parked are outlined in the following list: • Garage Parking •  Parallel Parking • Diagonal Parking. When a car is parked parallel to the road or the curb, or when it is situated in between two other parked cars, we say that the automobile is parked parallel. One of the several kinds of parking is called parallel parking [7]. In the majority of situations, you will be able to locate this kind of parking on streets, highways, crowded areas, and other comparable sites. Other such places include the following: When it comes to parking, parallel parking is often considered to be one of the most difficult kinds of parking, even for experienced human drivers. Parking in forward parallel or parking in reverse parallel are the two approaches that may be used to complete this assignment. From Figure 8.1 researchers are having a difficult time developing an

camera

face detection

happy

lbp feature extraction

sad PCA disgust

database

Figure 8.1  Flowchart for the testing phase.

anger

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  175

automatic parallel parking system since the available parking space for this kind of parking is either very limited or entirely unknown. When the area that the vehicle is working in is very dynamic [8], it makes it a lot more challenging; as a consequence, the vehicle has to be furnished with an adequate feedback system. According to the most current trends, the automobile industry seems to be heading toward the use of much more automated procedures. Currently, a number of different automakers are actively toiling away at the creation of an extensive range of various types of driverless cars. It is hoped that an autonomous vehicle would behave in a responsible manner and be able to negotiate difficulties with just the slightest amount of help from a human driver. It is reasonable to anticipate that an automobile with autonomous capabilities will be able to park itself. Both the infrastructure for such a system and the building of a prototype [9] for an autonomous parking system are now the subject of a substantial amount of research that is currently being carried out. It will be very challenging for a self-driving vehicle to achieve a level of expertise and capability that is equivalent to that of a human driver who has years of experience. Every algorithm for machine learning works toward the ultimate aim of reaching a point where human intelligence and machine intelligence are on an equal footing. In order to properly fulfill the task of parking an autonomous vehicle, an intelligent onboard technology is also required. In addition to this, learning how to park the automobile may be difficult for novice drivers, and human error can be the root of a range of driving-related problems. The convenience and peace of mind that autonomous parking technology provides to human drivers are well worth the investment. In addition to this, it cuts down on the amount of time that is wasted sitting in traffic, which in turn decreases the quantity of fuel that is used. However, the system must be designed with a sufficient amount of care; otherwise, it runs the danger of failing, putting the lives of the passengers at hazard. When it comes to parking, the majority of the solutions that are now available exclusively concentrate on the static environment, and they approach the problem of automobile parking as a distinct obstacle. The work that is being recommended extends upon car parking challenges by adding the navigation module in order to produce solutions that are robust and more acceptable in settings of dynamic complexity. This is done in order to generate more workable alternatives. • Our algorithms, which are based on fuzzy logic [10] and neural networks, take into consideration the dynamic character of the environment, which is caused by the existence or arrival of any live or nonliving object or hindrance. This may be an item or an impediment of any kind. • The study that we have recommended seeks to further integrate the dimensionality of the vehicle,

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

176  Artificial Intelligence for Autonomous Vehicles

or the capacity to park automobiles of varied sizes, and it does so by using flexible and behavior-based reasoning [11]. • A fuzzy-based parallel parking controller was constructed initially and was based on the fifth-order polynomial. This controller was designed to allow an automobile to be parked in a parallel position in a predetermined parking slot in either the forward or the reverse direction, depending on the starting and ending states of the vehicle. • After successfully implementing the parking algorithm based on traditional path planning, we extend our work to incorporate dynamicity in the environment in order to avoid collisions while still keeping track of the destination. This was accomplished by incorporating dynamicity in the environment in order to achieve this goal. The incorporation of dynamic elements [12] into the environment helped achieve this goal by reducing the likelihood of collisions occurring. 1. Having taken into consideration the following: We developed a navigation system that is based on fuzzy logic and may redirect the vehicle based on the information that is received from the various sensors. This is useful in situations in which an item restricts the parking path. • A one-of-a-kind method for calculating ultrasonic range is developed, and it is put to use in order to provide the navigation controller with information (sensing) on the surroundings all around it. This is done in order for the navigation controller to understand where it is in relation to its surroundings. The approach that is now under discussion is capable of simulating actual environmental sensing and calculates the distance between sensors depending on the form of the vehicle. • Unlike previous algorithms, we addressed the problem of navigation as an extension of the parallel parking task. This allowed us to solve both problems simultaneously. It has been made possible for the driver to steer the vehicle to the parking area from any starting position—thanks to a system that has been devised. After the vehicle has completed its navigation and has arrived at its parking spot, a parallel parking controller will assume control of the vehicle. After that, it parks the car inside the predetermined space while ensuring that it is facing the right direction. In addition to a controller designed for parallel parking, this system made use of a controller designed specifically for navigation. • As a result of this, a hybrid model that is composed of a navigation controller and a controller for parallel parking is provided as a solution for handling the scenarios that include a large number of parking places [13]. When it is believed that the vehicle has arrived at the first available parking place and occupied that position, the process is considered to have begun. Initially, it has no awareness of the available parking places and is

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  177

completely clueless about them. The process starts with parallel parking, and ultrasonic sensors are utilized to assess whether or not a parking place is occupied at that particular location. If the automobile determines that there is already another vehicle using the parking spot, the parking procedure is canceled, and the navigation controller [14] is used in its stead. The navigation controller will drive the vehicle to the next parking location that is available, and this process will continue until there is a parking spot that can be used. When the vehicle finds a location in which it may park without being blocked by another vehicle, parking is regarded to be complete. • As a solution to our first problem, which was the ability to park the vehicle in a dynamic environment, we proposed a novel parallel parking architecture in a dynamic environment with the integration of an obstacle avoidance fuzzy controller. This was done in order to meet the requirements of our first challenge. Both the forward and the reverse parking strategies are effective when used in this design, which works well for N number of parking places. The obstacle avoidance controller and the fifth-order polynomial-based parking controller are brought together and combined as part of this novel solution. We made use of a third controller that was known as the “decision” controller. The purpose of this controller was to direct the workflow of the other two controllers. During the course of the parking process, establishing this system with the intention of identifying any automobile or obstacle that was not anticipated was the primary goal. The path that the car takes is intelligently changed when the sensors of the vehicle identify an obstruction, but the destination of the vehicle is not forgotten in the process. The vehicle makes its route selection throughout this process of rerouting by taking into consideration the amount of space that is available between the congested areas. In order to determine whether or not an intelligent option from a system may effectively avert a collision, this architecture is put through its paces with both fixed and moving impediments in a range of different circumstances. In addition, we investigated multistage procedures, and as a consequence of our research, we provided a neuro-fuzzy architecture for parking in dynamic situations. This was accomplished as a result of the findings of our study. It has been found that making improvements to the fuzzy system [15] by adding neural networks as a prefix can result in enhanced performance while parking autos. The findings of the simulation indicate that it is effective in overcoming the limitation of ambiguity that is inherent in a fuzzy-based system with a single stage due to the fact that it avoids collisions. This is shown by the fact that it was able to do so. As a response to our second obstacle, which was the capacity of parking cars of varied sizes, we came up with a revolutionary

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

178  Artificial Intelligence for Autonomous Vehicles

fuzzy-based adaptive dimension parking method. This technique was suggested by us as a solution. This algorithm is able to adjust its threshold for the behavior controller and carry out its ideal route while taking into consideration the dimensions of the vehicle. Scaling factors are determined and calculated depending on the dimensions of the vehicle that are supplied in the description. After that, the fuzzy [16] variable thresholds that are utilized in the obstacle avoidance module are adjusted based on these scaling factors that are employed. In addition, the application of the artificial potential field theory is used during the course of the runtime in order to make these thresholds malleable and adjustable. This algorithm was developed because a parking controller designed for one set of fixed-sized automobiles, such as hatchbacks, sedans, or SUVs, was unable to work efficiently with another set of fixed-sized vehicles, such as convertibles. This prompted the development of this algorithm. This approach has been examined using the measurements of a Range Rover and a Hyundai i20.

8.2 Related Work 8.2.1 Face Detection CNN used a range of drop sizes in its investigation of the effectiveness of models for facial recognition, which may be found in this article. The subsequent system, which was utilized in our case studies, carried out an analysis of the structure by using the following criteria: The M convolutional layers, also known as the spatial volume normalization (SBN) layers, are what are meant to be referred to when using the phrase “beginning area” in the early stages of the phase. In addition to dropout and maximum pooling, these surfaces are guaranteed to be there at all times. The system always results in combined layers, which are always found after the M convolution layers that it has previously analyzed [17]. In their communications, they encounter nonlinearity and offline functioning, as well as volume normalization (BN) and dropout. The offline layer, which comes in at number 3 on our list, is the one that is in charge of generating the scores and the smooth-maximum loss function after the network. The advanced model is concerned with the volume normalization and the number of complete and connected layers per user, and it also gives the capability to assess whether or not there is a drop and how many layers may pool to their maximum. All of these factors contribute to the normalization of the volume. In addition to methods for dropout and volume normalization, our process included L2 modulation as one of its steps. In addition, the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  179

DEMO IMAGE fear

happy

sad

surprise

Figure 8.2  Different types of facial emotions.

user has the possibility to specify the number of filters, upgrades, and zerofill [18]; however, in the case that these values are not specified, default values will be taken into consideration as shown in Figure 8.2.

8.2.2 Facial Emotion Recognition The three basic processes that make up the process of facial recognition are known as face detection, CNN feature extraction, and facial modeling [19] (Figure 8.3). In order to clarify, the functioning of the system may be broken down as follows: It has the ability to obtain input n clips from the databases that correspond to those clips. It is possible for it to park itself automatically in a parallel manner with a given initial and final position; however, if there is any uncertainty or unanticipated event, this control design will cause the vehicle to collide if it is used on its own and if it is used as a stand-alone. This is provided if all environmental conditions are clear enough. In order for a modern parking system to be considerate of its environment, it is necessary for it to have a sensing mechanism that is incorporated into the parking controller. When it comes to parking, a human driver with a lot of experience can perform the necessary steering maneuvers to park a car by quickly evaluating the vehicle’s [20] end position in addition to its orientation in the ideal parking condition. This is possible because the human driver is able

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

180  Artificial Intelligence for Autonomous Vehicles

to visualize both the ideal parking condition and the end position of the vehicle. On the other hand, in order for autonomous robots to recreate the same piece of artwork, a sensory system that is adequate is necessary. It is possible that a vision system is an excellent duplicate of the human eye. Despite this, it is a fact that such sensing provides only a vast field of view, and it is possible for them to completely overlook a very limited range of obstacles such as corners, the face of other cars, or the walls themselves. One option to lessen the impact of this problem is to make use of active sensors, such as ultrasonic sensors, infrared sensors, laser sensors, and so on, among other types of active sensors. In this section, we are going to talk about a one-of-a-kind geometric method that will allow us to get a better sense of the distance information from the boundary of the Contrastive Learning of Musical Representations (CLMR). This information that has been perceived is really simulated data, but they may be contrasted with real-time sensor data that were acquired by any active sensors [21]. The devices known as ultrasonic sensors are advantageous in that they are not only affordable but also provide a direct range of data regardless of the environment in which they are located. Furthermore, in contrast to infrared sensors, the accuracy of the detection performed by ultrasonic sensors is not affected by the color of the object [22] that is being detected. This is an advantage that ultrasonic sensors have over infrared sensors. Because of this, the sensing technique will only be explained in relation to ultrasonic sensors throughout this whole article. The work that we have done involves combining the data obtained from ultrasonic sensors with those of multilevel fuzzy controllers in order to complete the multifunctional task. The next step is to find the face in the photograph and then delete it from the file. After that, the facial images are sent as a capture to a neural network [23] that may be updated in order to extract the compatibility qualities of the faces in the photographs. The real work will eventually be between modeling that is carried out by an algorithm for machine learning at some point in the future.

8.3 Proposed Method In order to assess human emotions, this research used the following features: SVM, Gabor Feature, Histogram of Oriented Gradients, Mask-CNN, and Local Binary Pattern. For the purpose of putting the theory to the test, it is necessary to isolate the human body and mind from the movie. In the process of assessing human mood, well-known machine learning

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  181

algorithms such as SVM, VFR, RF, and K-NN [24–27] have all been used at various points in the investigation. Four different CNN models, namely, Alex Net, VGG-19, ResNet-50, and Inception-ResNet-v2, are selected to be used in the process of determining the temperature of the inside of the body. These CNN models have been trained on ImageNet2 to differentiate between pictures of people and pictures of nature with an exceptional level of accuracy, and these models are the ones that are utilized. LBP and SVM have provided the most accurate spectrum of emotions that may be used in real time. Both of these methods have been successful. The use of neural networks was necessary in order to attain this goal. The picture from the input video was extracted and preprocessed at the beginning of the procedure. After that, the CNN approach was used to extract the features. After removing all of the characteristics, the data were categorized using the Softmax classifier model.

8.3.1 Dataset The Kaggle website, which provides more than 37,000 well-trained grayscale snapshots of profile images at a resolution of 48 × 48 pixels, was scoured for datasets to compile this article. The images have been meticulously curated and nearly give off the impression of being positioned in the  same spot. From Figures 8.3 to 8.5, each one of the distinct profiles has provided the exact same amount of scope in every clip. Every single photograph in the training programmed has to be assigned to one of seven different states of mind, each of which is denoted by a different look on the subject’s face. These facial expressions of emotion may be categorized according to a number of different states, including anger, hatred, fear, pleasure, grief, surprise, and neutrality. More than 29,000 images have already been produced, 4,000 photographs are to be used as evidence, and 4,000 snapshots are used for instructional reasons. Following further analysis of the raw pixel data, the exact and tested images were chosen A B C D

Candidate images

Face detection

CNN-Feature Extraction

Modeling

Figure 8.3  Segmentation of the Facial Expression Recognizer (FER) method. Respective sections are sketched in detail in its comparable section.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

182  Artificial Intelligence for Autonomous Vehicles

PreProcessing

Feature Extraction With CNN

Result

Softmax classifier

Figure 8.4  Preprocessing and extraction/classifier of frame using CNN.

Static Facial Expression Image Database

Static Human Face Image Database

Facial expression Feature Extraction

AdaBoost, Optical Flow, Gabor Feature 1

Classifier Learning and Test

Automatic Facial Expression Recognition

Real Emotion Recognition base on BiModality Fusion under Natural Scene

PSO-SVMs Feature Selection and Parameters Optimizations

Early Warning For Intelligent surveillance System

1 Image Sequence Analysis

Real Image Capture by Webcam/Movie clip

Classifier Learning and test

Face Detection

Human Motion Recognition

N

Human Motion Feature Extraction

Figure 8.5  System architecture for real emotion recognition using movie clips.

from each and every trained image by first excluding the training photographs and then selecting the images from each and every trained image. During the process of developing the data augmentation, similar images were patched together and used for the purpose of putting attention on the horizontal train set. Characteristics make use of the fundamental raw pixel data that were used to update the features that were generated by layers. These features have been put to use in the process of forming expressions. Learning models were built in order to make it easier to conduct more research. These models were constructed out of HOG features, which were then changed to connect the layers together before being combined into fully integrated input feature (FC) layers.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  183

8.3.2 Preprocessing Because the size of the picture has already been decided, the original photograph has to be reduced in size before it can be used as an input. This is because the size of the image has already been determined. It was point (x, y) in the very first snapshot that was given to us, and it is this point that has to be expressed and plotted to become point (x’, y’). Where SX indicates the computing ratio of the picture when viewed in the direction of the x-axis, and see represents the scaling ratio of the image when seen in the  direction of the y-axis. A, B, C, and D are the letters that stand in for the picture’s individual pixels (x, y). As it moves through the state, the provided clip’s dimensions are gradually shrunk until they equal 128 × 128.

8.3.3 Grayscale Equalization When snapped together, they expose a seemingly haphazard distribution of light and shade, making the information difficult to decipher. It is not at all difficult to locate the brilliant spots, trails, and other elements that were produced by the activity in the actual image that was recorded. It is essential to normalize the gray scale of the image in order to get the maximum amount of contrast in the picture that is physically achievable. In order to proceed with the snapping, the Histogram Equalization (HE) model was used in the data in this circumstance. The primary objective here is to convert the map from the original map into a distribution that is uniform all the way through. If the gray level of the film is L, then its magnitude will be M > N, and the number of bytes that make up its gray level will be E. If the gray level of the film is L, then these values will all be true. The decay of the snap has the highest magnitude, and the contrast of the snap has the largest magnitude, when the comic histogram of the image is precisely the same. Therefore, gray-level uniformity acknowledges that the even dispersion of the snap histogram is good for obtaining the facial features. This is because it enhances the contrast of the snap and offers clear clarity. In order to get the particulars of the snap edge, the Kirsch edge speculation method is used.

8.4 Results and Analysis The development of a model is going to be the key focus of the system that has been suggested to be used. The system had three distinct surfaces: two

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

184  Artificial Intelligence for Autonomous Vehicles

of which were curved and one that was flat and round. The transition layer is located on the initial surface, and it is constructed from the components listed below in the order in which they are listed: 35 4 × 4 puzzles with a stem size of 1, including both state and foreigner groups; nonetheless, the maximum was only obtained by not using any pooling tactics at any point in the process. From Figure 8.6 grayscale images transition layer is a jigsaw that has a dimension of 68 square inches and has a stem size of 1, drop groups, normalization, and normalization groups in addition to normalization groups. For the purpose of making the biggest pool possible, a filter with the dimensions of 2 × 2 is utilized. In the context of this investigation, the modified linear unit will serve the purpose of representing a functional unit. Before beginning the process of training our model, we carried out a series of balancing tests in order to verify that the functions of the network are operating as intended. This was completed before we began teaching our model new skills. During the preliminary investigation of the system’s dependability, the first loss is found and documented. We have devised a system of categorization that is composed of a total of seven distinct classes, and we anticipate that the value range will be somewhere in the 1.96 region. To begin, we focused on the second balance and used just a little portion of the practice set in an attempt to track down the fit. After that, we will go on to the next step of the process, which is to train our model using the findings that we have accumulated up to this point. In order to make the process of training

0 50 100 150 200 250 300 350 400

0

100

200

Figure 8.6  Collections of grayscale images.

300

400

500

600

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  185

Extracting Features

Classification

d ne tte Fla

Input

Convolution

Pooling

Output

Fully connected

Figure 8.7  CNN architecture.

the model move more quickly, we made advantage of GPU-accelerated in-depth learning capabilities. Using these capabilities allowed us to complete the treatment in a timelier manner shown in Figure 8.7. All of the pictures that make up the sample, in addition to the hyperparameters that control the learning rate of the sample and the number of neurons that are hidden, were put through a series of cross-checks that used a wide range of different values. This was done in order to ensure that the results were accurate. A series of tests was performed on the package in order to ascertain the level of craftsmanship that went into the creation of the copy. These examinations included validating the samples at each consecutive phase of manufacturing as well as evaluating the samples at each level. Additionally, the samples were tested at each stage. The leading superficial sample, which was assessed as part of the verification procedure and found to have an accuracy of 65%, provided 64% of the information for the package. The hyperparameters that have been tested several times for the shallow model are summarized in Table 8.1, which offers an overview Table 8.1  Shallow version of emotion accuracy. Guidelines

Avails

Acquirements Rate

0.001

Regulation

1e-6

Invisible Neurons

256,512,1024

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

186  Artificial Intelligence for Autonomous Vehicles

of the subject matter. This was done to guarantee that the information is correct. A deep convolutional neural network that was developed with four conversion layers and two FC layers was utilized for the training process in order to analyze the impact that transformer layers attached to the network have and to investigate the function that FC layers play. There are a total of 512 3 × 3 filters across the primary, secondary, tertiary, and final replacement layers. The primary replacement layer has 64 3 × 3 filters, the secondary replacement layer has 128 filters, the tertiary replacement layer has 512 3 × 3 filters, and the final replacement layer also has 512 filters. The size 1 volume normalization will be used in each transition layer as an activation function to initiate the construction of the layer. This will take place at the beginning of the creation process. The dropout system, the max-pooling system, and the ReLu system have all seen some degree of development as a result of these developments. There are 256 neurons in the layer that cannot be seen in the main FC layers, and there were 512 neurons in the secondary FC layers as well. The layer that cannot be seen in the primary FC levels is located below the primary FC layers. We would reel in either the FC layers or the transition layers, which is where the volume normalization was situated, when we were in a rush. In case you were curious about it, the kind of loss function that we used was a Softmax. Prior to instructing students on how to use the system, initial loss checks, smaller groups of the training package, and investigations into the capabilities of better fitting the network were carried out in a manner that was comparable to that of the basic model. This was done before students were instructed on how to use the system. The results of these balance checks, which were employed in the process of activating the network, were evaluated, and it was found that they were accurate. Following that, we went on to train the network using each and every one of the images that were shown in the tutorial, making use of a total of 35 epochs and 128 volume scales throughout the process. In addition, we performed a second check on the superfluous variable in order to determine which sample had the maximum degree of accuracy. This was done in order to choose the best one. At this juncture, 65% of the verification package has been finished, and during testing on the package, an accuracy of 64% has been achieved. The representation of this model that is contained in Table 8.2, which has a table of the values that may be entered for each hyperparameter, is the most exact one that can be found anywhere else. Even though this did not result in an increase in the accuracy of classification, networks were trained with five and six interchangeable layers in order to explore more intricate CNNs. This was done despite the fact that this did not result in

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  187

Table 8.2  Deep version of emotion accuracy. Expression

Shallow model

Deep model

Anger

45%

58%

Disgust

35%

80%

Fear

64%

56%

Happy

80%

86%

Sad

38%

69%

Surprise

67.5%

62.5%

Natural

40%

52.6%

an improvement. This was carried out so that more complicated CNNs might be investigated. As a consequence of this, it was concluded that this network, which consists of two FC layers and four database levels, is the most effective choice for our database. The source pixel data are the most important components of our classification work, and this is true regardless of the complexity of the model that we use to analyze the data. This is the case regardless of whether or not we use a model with a greater level of detail. The only criteria that were taken into consideration were those that had been generated by the layers themselves. As a result of the fact that HOG properties are sensitive to boundary conditions, it is common practice to rely on them while trying to recognize facial expressions of emotion. To employ HOG features with source pixels in our network to investigate if there was any manner and to observe the performance of the model when it has a mixture of two distinct features, a new learning model was constructed using two neural networks: one for feature extraction and the other for feature combination. In the first one, some of the layers were interchangeable with others, but in the second one, all of the levels were entirely linked to one another. In order to accomplish this goal, a ­cutting-edge learning model was developed by combining the capabilities of two neural networks. The first neural network featured layers that could be switched out for others, but the second neural network only comprised levels that were fully connected to one another. The characteristics that are established by the first network are combined with the HOG characteristics, and the hybrid characteristics that are formed as a consequence are made available across the second network.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

188  Artificial Intelligence for Autonomous Vehicles

A superficial network and a deep network were both taught to have the same features as part of the preliminary investigation into the superficial and deep networks. This was done in order to evaluate how well the system worked when it was put to use with characteristics that were analogous to those being contrasted. At this point in the process, the correctness of the shallow model will only be evaluated with reference to the source pixels. This estimate was pretty near the accuracy that had been acquired from the shallow sample, which indicated that it was a reliable one. It should be brought to everyone’s attention that the precision of the model was the same as that of the raw pixels. The model has been trained and validated using data obtained from the Kaggle repository. This dataset contains approximately 3,500 grayscale front-view photos of 300 human faces ranging in age from 18 to 65 years. In this particular piece of research, the model was trained using close to 480 different human faces. While the initial video clip in each and every frame sequence depicts a face that is expressionless, each subsequent clip in the series captures a different feeling and places it at the conclusion of the expression seen before. The variation in the conversation distance between all of the frames in the database is no more than 30%, which means that the minimum distance was less than 80 pixels and the maximum distance was less than 100. Therefore, as shown in Figure 8.8, the interocular distance that was determined using the input picture is significantly less than 80 pixels or significantly more than 120 pixels.

Adam Optimizer 0.8

Training Loss Validation Loss

2.0

Training Accuracy Validation Accuracy

0.7

1.8 1.6

Accuracy

0.6

1.4 1.2

0.5

0.4

1.0 0.8

0.3

0.6 0

10

20

30

40

0.2

0

10

20

Figure 8.8  Matrix waveform for the sets of six global expressions.

30

40

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  189

8.5 Conclusions The face expressions of the characters in this piece of literature serve to depict the emotional state of each individual character. A deep learning convolutional neural network that uses GPU technology was constructed in the system that is suggested for the purpose of collecting facial characteristics from individuals and measuring their core body temperature. It is suggested that the GPU system be used as the method to evaluate and deliver the precise range of human emotions and to extract the facial expression from the video clip. In addition, the body temperature of the individual may also be obtained. The proposed system should be able to collect data from a novel facial expression recognition dataset, which has two requirements: it must be able to recognize static video sequences and it must be able to extract face features. The proposed system should also be able to collect data from an existing facial expression recognition dataset. In addition, the core focus of this study report was on the temperature that is maintained inside an individual’s body.

References 1. Mehmood, R.M., Du, R., Lee, H.J., Optimal feature selection and deep learning ensembles method for emotion recognition from human brain EEG sensors. IEEE Access, 5, 14797–14806, 2017. 2. Song, T., Zheng, W., Lu, C., Zong, Y., Zhang, X., Cui, Z., MPED: A multimodal physiological emotion database for discrete emotion recognition. IEEE Access, 7, 12177–12191, 2019. 3. Batbaatar, E., Li, M., Ryu, K.H., Semantic-emotion neural network for emotion recognition from text. IEEE Access, 7, 111866–111878, 2019. 4. Zhang, Y., Yan, L., Xie, B., Li, X., Zhu, J., Pupil localization algorithm combining convex area voting and model constraint. Pattern Recognit. Image Anal., 27, 4, 846–854, 2017. 5. Meng, H., Bianchi-Berthouze, N., Deng, Y., Cheng, J., Cosmas, J.P., Timedelay neural network for continuous emotional dimension prediction from facial expression sequences. IEEE Trans. Cybern., 46, 4, 916–929, Apr. 2016. 6. Feng, X.U. and Zhang, J.-P., Facial micro expression recognition: A survey. Acta Automatica Sinica, 43, 3, 333–348, 2017. 7. Özerdem, M.S. and Polat, H., Emotion recognition based on EEG features in movie clips with channel selection. Brain Inf., 15, 44, 241–252, 2017. 8. Escalera, S., Baró, X., Guyon, I., Escalante, H.J., Tzimiropoulos, G., Valstar, M., Pantic, M., Cohn, J., Kanade, T., Guest editorial: The computational face. IEEE Trans. Pattern Anal. Mach. Intell., 40, 11, 2541–2545, Nov. 2018.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

190  Artificial Intelligence for Autonomous Vehicles

9. Yu, X., Zhang, S., Yan, Z., Yang, F., Huang, J., Dunbar, N.E., Jensen, M.L., Burgoon, J.K., Metaxas, D.N., Is interactional dissynchrony a clueto deception? Insights from automated analysis of nonverbal visual cues. IEEE Trans. Cybern., 45, 3, 492–506, Mar. 2015. 10. Vella, F., Infantino, I., Scardino, G., Person identification through entropy oriented mean shift clustering of human gaze patterns. Multimed. Tools Appl., 76, 2, 2289–2313, Jan. 2017. 11. Lee, S.H., Plataniotis, K.N.K., Ro, Y.M., Intra-class variation reduction using training expression images for sparse representation basedfacial expression recognition. IEEE Trans. Affect. Comput., 5, 3, 340–351, Jul./Sep. 2014. 12. Ghimire, D., Jeong, S., Lee, J., Park, S.H., Facial expression recognition based on local region specific features and support vector machines. Multimed. Tools Appl., 76, 6, 7803–7821, Mar. 2017. 13. Wang, L., Behavioral biometrics for human identification: Intelligent applications: Intelligent applications, IGI Global, Hershey, PA, USA, 2009. 14. Yan, W.Q., Biometrics for surveillance, in: Introduction to Intelligent Surveillance, pp. 107–130, Springer, New York, NY, USA, 2017. 15. Hess, U., Banse, R., Kappas, A., The intensity of facial expression is determined by underlying affective state and social situation. J. Personal. Soc Psychol., 69, 280, 1995. 16. Dhall, A., Goecke, R., Joshi, J., Hoey, J., Gedeon, T., Emotiw2016: Video and group-level emotion recognition challenges, in: Proceedingsof the 18th ACM International Conference on Multimodal Interaction, pp. 427–432, ACM, 2016. 17. Benitez-Quiroz, C.F., Srinivasan, R., Feng, Q., Wang, Y., Martinez, A.M., Emotionet challenge: Recognition of facial expressions of emotion in the wild. arXiv preprint arXiv:1703.01210, 14, 227–229, 2017. 18. Fasel, B. and Luettin, J., Automatic facial expression analysis: A survey. Pattern Recognit., 36, 1, 259–275, 2003. 19. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9, 2015. 20. Hasani, B. and Mahoor, M.H., Spatio-temporal facial expression recognition using convolutional neural networks and conditional randomfields. 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 790– 795, 2017. 21. Siddiqi, M.H., Ali, R., Khan, A.M., Kim, E.S., Kim, G.J., Lee, S., Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimed. Syst., 21, 6, 541–555, 2014. 22. Mlakar, U.  and Potočnik, B., Automated facial expression recognition based on histograms of oriented gradient feature vector differences. Signal Image Video Process., 9, 1, 245–253, 2015.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Facial Expression from Live Video Clips in AVs  191

23. Jeni, L.A., Takacs, D., Lorincz, A., High quality facial expression recognition in video streams using shape related information only, in: Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pp. 2168–2174, 2011. 24. Erhan, D., Szegedy, C., Toshev, A., Anguelov, D., Scalable object detection using deep neural networks, in: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2155–2162, 2014. 25. Rychetsky, M., Algorithms and architectures for machine learning based on regularized neural networks and support vector approaches, Shaker Verlag GmbH, Germany, ISBN 3826596404, 2001. 26. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J., Coding facial expressions with gabor wavelets, in: Proceedings of the 3rd International Conference on Face & Gesture Recognition; FG ‘98, Washington, DC, USA, IEEE Computer Society, p. 200–, ISBN 0-8186-8344-9, 1998. 27. Kahou, S.E., Froumenty, P., Pal, C., Facial expression analysis based on high dimensional binary features, pp. 135–147, Springer International Publishing, Cham, ISBN 978-3-319-16181-5, 2015.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

192  Artificial Intelligence for Autonomous Vehicles

Models for the Driver Assistance System B. Shanthini1*, K. Cornelius1, M. Charumathy1, Lekshmy P.2, P. Kavitha3 and T. Sethukarasi3 St. Peter’s Institute of Higher Education and Research, Tamil Nadu, India 2 LBS Institute of Technology for Women, Tamil Nadu, India 3 RMK Engineering College, Tamil Nadu, India

1

Abstract

Road traffic collisions are one of the leading causes of death and injury in our society; as a result, they are a significant contributor to the loss of both human lives and financial goods. The vast majority of accidents are brought on by human mistake, such as a lack of attention, being interrupted, being sleepy, having insufficient preparation, and so on; these errors lead to catastrophic bodily injuries, fatalities, and large financial losses. Driver assistance systems (DASs) have the potential to reduce the number of mistakes that are made by humans by monitoring the conditions of the road and alerting the driver of an impending danger by issuing a number of recommendations, suggestions, and alerts. Because drivers being distracted is one of the leading causes of traffic collisions, this is something that has to be done. Sluggishness location is the goal of this research, which will accomplish this goal with the assistance of data on the eye state, head posture, and mouth state of the driver. This will be done in order to improve the quality of the information that was gathered in the first place, which was received from the public drowsy driver data bank. Following the use of the camera response model (CRM), this will be done in order to achieve this goal. In order to separate the highlights from the differentiated eye district, step-by-step application of feature extraction strategies was required. These strategies included the histogram of oriented gradients (HOG) and the local binary pattern (LBP). When paired with the most accurate estimate of the mouth area and the head’s present location, the features of the eye region that had previously been segregated were revealed. Following the extraction of the component vectors, an infinite procedure was utilized in order to

*Corresponding author: [email protected] Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (193–208) © 2024 Scrivener Publishing LLC

193

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

9

choose the element vectors that were pertinent to the situation. At the conclusion of the day, the phases of the languor were organized by picking which highlights to group together and then used a technology called support vector machine (SVM). The findings of the reconstruction suggested that using the suggested structure would result in an increase in the degree of exactness of the grouping to 5.52%. Keywords:  Driver assistance system, support vector machine, camera response model, histogram of oriented gradients (HOG), local binary pattern (LBP)

9.1 Introduction Drowsy driving has emerged as one of the most significant threats to public health in the transportation sector over the last several decades. An onboard driver sleepiness detection framework in cars is essential in order to prevent accidents from occurring on the road or when driving over hazardous terrain [1]. The research on fatigue takes a wide range of measurements, including things like behavioral highlights, visual highlights, physiological highlights, and so on. Because these techniques are noncontact in nature, the approaches that rely on visual highlights revealed a feasible execution in the diagnosis of driver tiredness [2]. Methods that are based on the driver’s visual highlights have recently emerged as a potentially fruitful area of research for identifying driver languor. The tactics that are focused on yawning are unable to anticipate the development of sluggishness, since this factor does not speak about sleepiness [3]. The goal of this study is to create a pattern for a drowsiness identification system that will be helpful in reducing the number of car accidents that occur on the road. The technology that is being presented has created three distinct methods to determine the level of weariness shown by the driver. The first and second techniques are carried out with the assistance of MATLAB, while the third approach is carried out with the assistance of Keil Compiler with Embedded C. In the first way, it focuses mostly on the eyes, and the Viola–Jones algorithm is what is utilized to discover the face. The Haar-like properties are used in order to locate the eyes. If the eyes are open, the iris is detected using Canny edge detection, and then circles are drawn on the iris using the Circle Hough Transform algorithm. If the driver is alert, then two circles appear around the eyes. If the driver is not alert, then a warning message and an alarm are displayed to the driver.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

194  Artificial Intelligence for Autonomous Vehicles

In the second way, the outcome is determined by the driver’s mouth, eyes, and the angle of his or her head. The Viola–Jones algorithm is responsible for face identification, and Haar-like [4] characteristics are what are employed to locate the eyes and mouth region of the face. The method of binarization may be used to successfully transform photographs into binary format. In order to separate skin pixels from non-skin pixels, the skin segmentation technique may be used to transform pictures from RGB to Y Cb Cr, and subsequently from Y Cb Cr to HSV. In the event that the eyes are closed, a warning message and an alert will be created. If the driver’s mouth is open, yawning is assumed to be occurring, and a warning message is sent to the driver at that time. If the driver’s head is turned even slightly to the side or tilted, the driver will get a warning in the form of a message with an alarm. Finding an accurate representation of the condition of the eyes and eyelids is the primary focus of the development of the third approach. The pattern of the driver’s eyes and eyelids shutting and opening is first captured by this device when the driver is awake and aware [5]. The same information is collected at an average rate and continually captures the pattern of the driver’s eyes opening and shutting. This system analyzes the differences in the samples it has collected by comparing them, and then it alerts the driver to any potential danger based on its findings. An advanced embedded system is capable of carrying out a certain function on its own. The engineers who build the system as a devoted system to a certain job may minimize both the cost and the size of the product via the process of optimization, which reduces the size of the product while also reducing the cost. The term “embedded system” refers to a system that is both hardwareand software-based. This kind of system is a highly fast-evolving piece of technology that is used in many different sectors, including the automotive, aerospace, and home appliance industries, among others. Assembly language [6] programming or embedded C is used to carry out the work at hand, and this technology may use a personal computer or a control unit to do so. Programming may be accomplished by one of these methods. People are facing a significant challenge in the form of an alarming rise in the number of accidents that occur on the roads as a consequence of the drivers’ inability to concentrate properly because of insufficient rest or dim lighting. According to the data and statistics, between 10% and 20% of car accidents are caused by drivers who were not paying attention to the road. When compared to other sorts of collisions, the one caused by the driver’s lack of concentration stands out as the most terrible. It is possible

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Models for the Driver Assistance System  195

that a distracted motorist will not have time to avoid an accident before it happens. The findings of the most current study on road safety indicate that driver weariness is a significant risk factor. In India, roughly 20% of drivers admit that they have fallen asleep behind the wheel or been caught driving while distracted at some point throughout the course of a single year. Before this poll, the most significant problem was drivers becoming too tired and falling asleep behind the wheel, which led to a large number of accidents on the roads both during the day and at night. According to some recent data, over 2,000 people lose their lives and 1,000,000 people are injured as a result of collisions that are caused by weariness each and every year all over the globe. According to the most recent statistics, road accidents claim the lives of approximately 13 lakh people per year throughout the globe, and the number of individuals who sustain injuries that do not result in death is estimated to be between 20 and 50 million [7].

9.2 Related Survey In order to keep an eye on the number of car accidents, weariness is a crucial element [8]. It is estimated that driver weariness is the cause of approximately 20% of all accidents that occur on the roads. The performance of the driver when driving while drowsy deteriorates, which leads to collisions with other vehicles [9]. Accidents of varying severity may be attributed to drowsy driving by motorists behind the wheel [10]. It is unquestionably helpful to the driver to identify signs of driving weariness and provide timely warnings so that accidents may be avoided. This will ultimately help save lives. Driver weariness is the primary contributor to accidents that occur on the road. When it comes to preventing accidents with the help of modern technology, one of the most difficult challenges is to identify and prevent driver weariness. Due to risky driving, accidents arise. The metrics that were described before need to be used as a weapon against drowsy driving and dangerous driving in order to lessen the impact of accidents and the aftermath they leave behind. The vision-based technique was presented [11, 12] as the way for identifying tired drivers and distracted drivers by monitoring the system. This innovative system uses an eye-detecting method that combines adaptive template matching, adaptive boosting, and blob detection with eye

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

196  Artificial Intelligence for Autonomous Vehicles

validation. The algorithm is one of many that fall under the category of blob detection. It cuts down on the amount of time needed for processing greatly. Support vector machine, often known as SVM, is used in order to improve eye recognition. In order to determine the condition of the eye, principal component analysis (PCA) and logical analysis of data (LAD) were used, along with statistical characteristics including sparseness and kurtosis [13]. Eye blinking rate and eye closure length, face identification [14], and skin color segmentation based on neural network approximation of RGB skin color are utilized to identify the driver’s tiredness. By detecting the resting potential of the retina or the electrooculography (EOG) signal, the system provides a design and construction of a low-cost blinking detection system. The EOG signals are sent to a personal computer so that they may be analyzed and processed by the computer. This allows for the development of algorithms that can identify and make sense of eye blinks. In Ghosh et al. [15], there is a system that discovers shifts in the eye locations by making use of the horizontal symmetry characteristics of the eyes. Additionally, there is a system that monitors shifts in the eye blink length, and there is also a system that does eye tracking. The PERCLOS algorithm is used to determine the amount of weariness. It has been suggested that an EEG-based brain–computer interface (BCI) system may be used to detect sleepiness. This technology would be practical for use in real-time automobile applications. In order to determine whether or not a driver is sleepy, photoplethysmography is used to analyze changes in the waveform of signals during the physiological phase of the test, and image processing evaluates the pattern of the eyes to determine whether or not the subject is sleepy. Combining the two approaches into a single profile is the best way to identify sleepiness. It was hypothesized that a combination of vibrations from the steering wheel, auditory stimulation, and physiological signals from a driving simulator may be used to identify sleepy driving. It is more effective and reduces the amount that the lanes deviate from the center. In order to determine whether or not a driver is dozing off behind the wheel, one of the physiological features, such as eye movement, can be observed through the use of electrooculography. In addition to this, an adaptive detection approach is utilized in order to measure driving-related eye movements, such as saccades and microslip events.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Models for the Driver Assistance System  197

Using a warning system before an accident is something that the image processing technology known as ROI does. Face tracking and other image-processing methods are employed in the Viola–Jones algorithm along with other algorithms used to compute the distance to the front car. Drowsiness may be detected by movement in the autonomous nervous system (ANS). This is a statistic that is used to ingest information from heart rate variability (HRV). It is possible to identify and avoid a traffic collision by using a strategy that is predicated on the behavior of spontaneous pupillary fluctuation. Calculating the pupil diameter variability is one way to test one’s level of awareness. Drowsiness may be determined by the use of an ECG-measuring technique that combines HRV, a physiological signal. A system that monitors the driver’s subsidiary behavior indicates that the driver has entered a sleepy state if the system detects an increase in the driver’s subsidiary behavior and a decrease in the driver’s arousal level. Subsidiary behavior includes things like yawning and making hand motions, among other things. The problem of wearing glasses may be circumvented by developing a mechanism that uses both the mouth and the condition of the driver’s eyes to determine whether or not the driver is fatigued. The goal of this project is to design a system that includes a wireless intelligence sensor network to identify driver weariness. To determine whether or not a driver is fatigued, one strategy relies on hybrid observers. A system that analyzes visual information and uses artificial intelligence (AI) to detect and follow drivers’ faces and eyes in order to determine whether or not they are drowsy. This system is used to determine whether or not a motorist is fatigued.

9.3 Proposed Methodology Drowsy driving has emerged as one of the most significant threats to public health in the transportation sector over the last several decades.

9.3.1 Proposed System The proposed system is composed of a total of six stages, which are as follows: the collecting of data, the preprocessing stage, the object

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

198  Artificial Intelligence for Autonomous Vehicles

recognition and tracking stage, the extraction of features stage, the selection of optimal features stage, and the categorization of driver tiredness stage. Figure 9.1 provides a flowchart representation of the proposed system; for more details, please refer to the explanation that is provided below.

Data collection NTHU drowsy driver detection

Pre-processing Camera response model

Object detection and tracking Viola-jones and Kanade-lucas-tomasi

Face region

Eye region

Mouth region

Head pose angle

Local binary pattern and histogram of oriented gradients

Online region based active contour model

Feature level fusion Feature selection Infinite feature selection algorithm Classification Support vector machine

Non-drowsy

Figure 9.1  Input flowchart of the proposed work.

Drowsy

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Models for the Driver Assistance System  199

9.3.2 Data Acquisition The National Tsing Hua University (NTHU) fatigued driver identification dataset, which was collected from the ACCV 2016 competition, was where the initial information was obtained and used to compile the list. In this particular case, the video captured a number of different people participating in a range of behaviors, including speaking with one another, remaining still, yawning, and other similar activities. In addition, this dataset includes two sets, which are respectively known as the approval set and the train set. These sets are both referred to simply as the sets. The length of the movies that make up the package that has been authorized might range anywhere from 1 to 10 min in length. The films of the different preliminary sets have a runtime that ranges from 1 min to 1.5 min. Each video is exactly 1 min long. In addition, the information that was gathered includes four separate sorts of ground truth: ground truth about the eyes, ground truth about the lips, ground truth regarding the head, and ground truth regarding sluggishness. The examination of ocular ground truth helps to evaluate whether or not the habit is starting to wear on the eyes of the individuals. If a person is yawning, chatting, keeping his or her lips closed, or just standing still, the ground truth of his or her mouth will tell you what he or she is doing. The head ground truth is speaking to the activity that is taking place in the person’s mind (looking aside, gesturing, or quietness). The languor ground truth is an indicator of whether or not the individual is weary, which brings us to our last point. Figure 9.2 shows some of the typical pictures taken from the NTHU dataset that were used to identify drivers who seemed to be fatigued.

9.3.3 Noise Reduction In the phase that comes after the collection of information, which is called the preplanning phase, the Customer Relationship Management (CRM)

Figure 9.2  Input image.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

200  Artificial Intelligence for Autonomous Vehicles

software is used in order to bring the quantity of chaos to an acceptable level. This phase is followed by the planning phase. The bulk of the time, the manufacturer of the camera will include a few nonlinear features within the camera, such as demosaicing and white balancing, in order to enhance the overall esthetic appeal of the photographs that are taken with the camera. The Brightness Transform Function, more often referred to as BTF, and the Camera Response Function are both necessary components of the Customer Relationship Management system (CRF). The problem of resolving the limits of the CRF may be addressed by using the camera alone, while the BTF can be managed by utilizing the camera in conjunction with the introduction proportion at the same time. In the beginning, the BTF is determined by determining the weight that is given to the perception of two distinct presentation images and basing the calculation on that. These photographs are stored in a separate location from the others. When you reach this stage, you will need to disassemble it in order to find the CRF that matches to the comparative metric condition so that you may continue. If equals 1, the CRF transforms into a power function, whereas the BTF reverts to being a straightforward linear function. The reason for this is because some camera makers build f to be a gamma curve, which works very well with their products. The CRF will become a two-parameter function, and the BTF will become a nonlinear function if G 1 is less than 1. The Figure 9.3 displays several examples of the preprocessed photographs included in the NTHU sleepy driver detection dataset.

Figure 9.3  Sample preprocessed images of NTHU drowsy driver detection dataset.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Models for the Driver Assistance System  201

9.3.4 Feature Extraction Following the successful recognition of the eye areas, feature extraction is carried out on the identified eye regions. In this work, high-level texture characteristics (HOG and LBP) were used to extract features from the observed eye areas as shown in Figure 9.4 and Figure 9.5. These features were used to help identify eye regions. The following is a condensed explanation of HOG and LBP.

(a)

(b)

(c)

(d)

Figure 9.4  (a) Preprocessed image, and panels (b), (c), and (d) are the extracted face, eye, and mouth regions.

(a)

(b)

(c)

(d)

Figure 9.5  (a) Face image, (b) eye blink, (c) yawning, and (d) head bending.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

202  Artificial Intelligence for Autonomous Vehicles

9.3.5 Histogram of Oriented Gradients When performing an evaluation of the gradient value present in the input pictures, the HOG descriptor makes use of a gradient operation. In addition, the points on the gradients of the video frames are designated by the letter G, and the video frames (pictures) that are being entered are identified by the letter I. The gradient point of the picture may be found by utilizing Equation 9.4 to do the calculations.

Gx = O × I (x, y) and Gx = OT × I (x, y)

(9.1)

Next, the windows that make up the input pictures are divided up into a number of distinct spatial areas that are referred to as cells. The HOG feature descriptor takes into account the direction of the edge when determining the gradient magnitude of a pixel. The equation used to determine the gradient magnitude of the pixel with coordinates (x, y). Additionally, the edge orientation of the pixel is taken into consideration.

9.3.6 Local Binary Pattern Through the use of LBP, the photographs are converted into labels based on the brightness value of the pixels in the image. As a consequence of this, the invariance of gray scale is a fundamental part of LBP, which is built around the ideas of texture and local patterns. The position of a pixel is indicated by the coordinates x and y in each framef. These coordinates are derived by making the value of the pixel that is located in the frame’s center (xc) the threshold value for determining the value of the pixel that is located next to it (m). The binary value of the pixel is first increased to the power of 2 in order to generate a weighting, which is then added in order to make a decimal number that can be placed in the center pixelxc location according to the equation. This process is repeated in order to produce a weighting as shown in Figure 9.6.

Figure 9.6  Output image of histogram of oriented gradients (HOG).

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Models for the Driver Assistance System  203

9.3.7 Feature Selection In this inquiry, a never-ending algorithm is utilized to choose the features that provide the most potential benefits. Each element aij of A, where 1 is less than I and j is more than n, indicates a pairwise energy term when G is represented as an adjacency matrix and A is the matrix that defines the nature of the weighted edges. When G is represented as an adjacency matrix, A is the matrix that defines the nature of the weighted edges.

9.3.8 Classification After the ideal highlights have been selected, the following step in the process of characterization is to increase it via the use of SVM in the process of rating the awake and asleep drivers. By imposing a restriction on the casual categorization mistake, the SVM classifier lowers the probability that there will be a double issue. In a similar vein, the testing and preparation procedures may be sped up with the use of SVM classifiers, which save a substantial amount of arrangement accuracy. The support vector machine, often known as SVM, is a discriminative organizing method that communicates its meaning via the usage of a certain hyperplane. The SVM characterization approach has grown in popularity over the last few decades because of the fact that it is able to work with high-dimensional information. As a result, it is presently used in a broad number of applications. Signal processing, bioinformatics, and computer vision are just a few of the industries that might benefit from these applications. This is the case despite the fact that the SVM classifier is a support vector machine. The expression w.x+b = 0 is the formula that may be used to figure out the amount of work that is done by the direct discriminant. An ideal hyperplane is used in the SVM classifier between the two classes (sluggishness and sleepiness) in order to differentiate the data without causing any disruption.

9.4 Experimental Study In order to carry out the experimental investigation, a computer equipped with Windows 10 and an i5 Intel core CPU operating at 3.2 GHz was employed in conjunction with MATLAB (environment 2018a). Fusion of features and deep neural networks. The proposed infinite method with

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

204  Artificial Intelligence for Autonomous Vehicles

SVM classifier was tested on the NTHU drowsy driver detection dataset that was used for the ACCV 2016 competition.

9.4.1 Quantitative Investigation on the NTHU Drowsy Driver Detection Dataset When compared to the mean classification accuracy supplied by SVM (90.37%), the mean classification accuracy delivered by other classification approaches, such as random forest, K-Nearest Neighbor (KNN), and Neural Network (NN), is 66.55%, 82.311%, and 77.9%, respectively. In addition to this, the SVM has a mean sensitivity of 91.70%, while the other classifiers have a mean sensitivity of 76.27%, 85.29%, and 76.59% correspondingly. SVM has a mean specificity of 89.80%, while the other classifiers have mean specificities of 81.25%, 83.83%, and 79.04% correspondingly. Table 9.1 shows input parameter. The SVM has the highest mean f-score among these

Table 9.1  Input parameters with results. Classifier Random forest

Subjects ID

Sensitivity (%)

Specificity F-score (%) (%)

Accuracy (%)

004

76

80

76.67

67.89

022

78.90

80.45

79.09

76.92

026

71.20

80.12

734

60.80

030

79

84.44

75.55

60.60

89.98

82.22

85.39

76.51

80.80

83.86

85

80.98

85.4

84.43

83.09

83.98

K Neural 004 Network 022 (KNN) 026

85

84.84

83

87.77

Neural 004 Network 022 (NN) 026

030

78.15

81.78

82.54

80.22

72.45

73.22

80.14

73.14

79.32

81.47

84.23

81.24

030

76.45

79.69

83.78

77.12

90

89.62

87.87

89.10

92.56

90.59

90.44

88.13

92.81

87.42

92.23

92.58

91.45

91.57

91.90

91.67

Support 004 Vector 022 Machine 026 (SVM) 030

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Models for the Driver Assistance System  205

current classifiers. A performance evaluation of the proposed system was carried out, and the results of that evaluation are graphically shown in Figure 9.7. Figure 9.8 illustrates the findings of a visual study that was carried out on the recommended system using a number of different classifiers. When paired with the infinite approach, the SVM had a somewhat favorable influence on the grouping accuracy in driver sluggishness detection. In fact, when compared to when it was employed without the infinite technique, it improved the grouping accuracy by as much as 2.86%. Figure 9.7 shows the graph analysis in proposed work. In this particular research, the HOG and LBP not only have the ability to properly identify the straight

100

Sensitivity Specificity F-score Accuracy

Percentage (%)

80 60 40 20 0

004

030

022 026 Subjects ID

Percentage (%)

Figure 9.7  Graphical representation of the proposed system performance. 100 90 80 70 60 50 40 30 20 10 0

Sensitivity

Specificity F-score Performance measures

Random forest

KNN

NN

Accuracy SVM

Figure 9.8  Graphical comparison of the proposed system with dissimilar classifiers.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

206  Artificial Intelligence for Autonomous Vehicles

and non-straight features of the driver’s face, mouth, and eye regions but also fundamentally safeguard the relationship between lower- and higherlevel highlights. Run-off road accidents, both major and minor, may be avoided with the help of the model that is now being described, which is one of the advantages offered by it. During the course of this experiment, the infinite algorithm, when used in conjunction with the SVM classifier, was able to attain an average specificity level of 89.80% and a sensitivity level of 91.70%.

9.5 Conclusion In the present investigation, a novel conceptual framework is presented for the purpose of categorizing the several varieties of driver languor. The goal of this inquiry is not only to organize the steps of fatigue detection, rather it is to come up with a method that can determine component parts in a more accurate and efficient manner (tired or non-sluggish). The development of the unending algorithm, which may be found here, is the first step in the process of selecting the significant component vectors. In order to provide a description of the optimal component vectors that were chosen, the SVM classifier is used. The proposed framework used f-score, specificity, sensitivity, and accuracy to obtain a better execution in the detection of driver sleepiness in comparison to the hybrid CNN-LSTM model. This was done by achieving a higher level of accuracy. On the NTHU tired driver detection dataset, the proposed framework was able to reach an accuracy of 90.37% and a speed of 12 casings per second, as shown by the results of the simulation. The hybrid CNN-LSTM model, on the other hand, achieved an accuracy of 84.85%.

References 1. Ess, A., Schindler, K., Leibe, B., Van Gool, L., Object detection and tracking for autonomous navigation in dynamic environments. Int. J. Robot. Res., 29, 14, 1707–1725, 2010. 2. Khan, M.Q. and Lee, S., A comprehensive survey of driving monitoring and assistance systems. Sensors, 19, 2574, 2019, doi:10.3390/s19112574. 3. Stutts, J.C., Reinfurt, D.W., Staplin, L., Rodgman, E., The role of driver distraction in traffic crashes, U.S. Department of Transportation, Washington, DC, USA, 2001.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Models for the Driver Assistance System  207

4. Kukkala, V.K., Tunnell, J., Pasricha, S., Bradley, T., Advanced driverassistance systems: A path toward autonomous vehicles. IEEE Consumer Electronics Magazine, 7, 5, 18–25, 2018. 5. World Health Organization, Global status report on road safety, WHO, Geneva, Switzerland, 2015, [Online]. Available: http://www.who.int/ violence_injury_prevention/road_safety_status/2015/en/. 6. Association for Safe International Road Travel, Annual global road crash statistics, ASIRT. Potomac, Maryland, 2018, [Online]. Available: http://asirt. org/initiatives/in-forming-road-users/road-safety-facts/roadcrashstatistics. 7. Seki, A. and Okutomi, M., Robust obstacle detection in general road environment based on road extraction and pose estimation. Electronics and Communications in Japan (Part II: Electronics), 90, 12–22, 2007. 8. Jazayeri, A., Cai, H., Zheng, J.Y., Tuceryan, M., Vehicle detection and tracking in car video based on motion model. Intell. Transp. Syst. IEEE Trans. on, 12, 2, 583– 595, 2011. 9. Xue, F., Mittal, S., Prasad, T., Saurabh, S., Shin, H., Pedestrian detection and tracking using deformable part models and kalman filtering. J. Commun. Comput., 10, 960–966, 2013. 10. Perrone, D. et al., Real-time stereo vision obstacle detection for automotive safety application. IFAC Proceedings Volumes, 43, 240–245, 2010, doi:10.3182/20100906-3-it-2019.00043. 11. Chisty, and Gill, J., A review: Driver drowsiness detection system. IJCST, 3, 4, 243–252, Jul-Aug 2015. ISSN: 2347-8578. 12. Singh, K. and Kaur, R., Physical and physiological drowsiness detection methods. IJIEASR, 2, 35–43, 2013. 13. Picot, A. and Charbonnier, S., On-line detection of drowsiness using brain and visual information. IEEE Trans. Syst. Man Cybern. Part A: Syst. Humans, 42, 3, 45–48, 2012. 14. McDonald, A.D., Schwarz, C., Lee, J.D., Brown, T.L., Real-time detection of drowsiness related lane departures using steering wheel angle. Proc. Hum. Factors Ergon. Soc. Annu. Meet., 56, 1, 2201–2205, 2012. 15. Ghosh, S., Nandy, T., Manna, N., Real time eye detection and tracking method for driver assistance system. Adv. Med. Electron., 02, 45–48, 2015.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

208  Artificial Intelligence for Autonomous Vehicles

Control of Autonomous Underwater Vehicles M. P. Karthikeyan1, S. Anitha Jebamani2, P. Umaeswari1, K. Chitti Babu1*, C. Geetha1 and Kirupavathi S.1 RMK Engineering College, Tamil Nadu, India Sairam Engineering College, Tamil Nadu, India 1

2

Abstract

In recent years, increasing the level of autonomy that underwater vehicles are capable of operating on their own has become a crucial component in order to cope with the highly dangerous and alien environment of the ocean. This is because the ocean is a totally different environment from land. This is due to the fact that the environment of the ocean is unfamiliar as well as lethal. Nowadays, autonomous underwater vehicles, also known as AUVs, are largely chosen over remotely operated vehicles, also known as ROVs, for the majority of the jobs that need engagement with the underwater environment. This is done to prevent the operators from becoming exhausted and to increase the likelihood that they will remain safe. Research on ocean resources, oceanographic mapping, inspections of deep-sea pipelines, and other activities of a similar kind made up the majority of the work linked with underwater intervention. In these kinds of applications, having precise control over an AUV’s location of the greatest importance if one wishes to gather data of the best possible quality. This is because the precision with which the AUV maintains its station and records its location is a crucial role in influencing the quality of the data that are acquired. This is the reason why this is the case. On the other hand, achieving precise trajectory tracking control of an AUV is an extremely challenging task. This is owing to the unstructured nature of the undersea environment as well as the fact that vehicle dynamics is extremely nonlinear, coupled, and time-varying. Additionally, this is a result of the fact that the vehicle is moving. In addition to these, changes in the hydrodynamic coefficients, which are induced by changing operating conditions and vehicle, may be vulnerable to unknown variables such as ocean *Corresponding author: [email protected] Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (209–228) © 2024 Scrivener Publishing LLC

209

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

10

currents, which make the design of the trajectory tracking control much more difficult. Changes in the hydrodynamic coefficients are induced by changing operating conditions and vehicle. Because of this, it is of utmost importance to have a tracking control system for an AUV that is built to accommodate the unpredictability of the marine environment. In order to carry out nontrivial autonomous operations in the deep sea, specifically those that are in regions that are inaccessible to humans, it is necessary to have a manipulator arm that is affixed to the underwater vehicle. The issue is made more complex to tackle as a result of these elements working together. As a direct result of the connected manipulator arm, the auxiliary undersea vehicle maintenance system (AUVMS) transforms into a structurally redundant example of kinematic redundancy. As a direct result of this, it is necessary to put into action various strategies for the resolution of redundancy. The task space-based control scheme design ideas have been offered as a possible remedy to the issue of redundancy resolution as part of the scope of this research project. Keywords:  Auxiliary undersea vehicle maintenance system (AUVMS), hydrodynamic coefficient, ocean currents, redundancy

10.1 Introduction Due to India’s strategic location and the abundant resources that can be found in the Indian Ocean Region (IOR) [1], it just so happens that the IOR is distinguished by littoral waters that are rather shallow in addition to weather patterns that are typical of tropical regions. Under these conditions, any fleet that tried to operate would have a tough time making an early discovery of the adversary, which would result in a delay in the deployment of operational troops [2]. It is estimated that 37% of the population of the globe lives within 100 km of the shore. We have a tendency to ignore the ocean in favor of paying attention to problems that occur on land and in the atmosphere; as a result, we have not been able to explore the ocean’s entire depths and discover all of the living and non-living resources that it contains. This is because we tend to ignore the ocean in favor of paying attention to problems that occur on land. It is vitally crucial to make use of underwater robots in order to get a greater understanding of marine and other environmental concerns, as well as to protect ocean resources from the consequences of pollution. Robots that are able to function submerged in water are referred to as autonomous underwater vehicles or AUVs for short. AUVs are a crucial part of India’s robotic warfare doctrine and play a major role in the country’s Navy’s preparations for early operational

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

210  Artificial Intelligence for Autonomous Vehicles

deployment. In addition, AUVs are an important part of India’s overall defense industry. The term “unmanned underwater vehicles,” of which AUVs are a subset, is used to refer to a broader category of underwater systems. It is possible to preprogram it so that it can carry out responsibilities of precise navigation, control, and direction. This is something that can be done. During the 1970s, the Massachusetts Institute of Technology (MIT) was the driving force behind the creation of a number of the first AUVs. One of these may be seen on display in the Hart Nautical Gallery at the MIT. This is something that has been happening for the purpose of investigating the ocean’s resources. The value of these underwater vehicles has been shown by their ability to effectively do challenging underwater activities, in particular those that are inaccessible to human beings. The surveying and tracking of deep-sea pipelines, the creation of oceanographic maps, the investigation and monitoring of shipwrecks, and other similar activities are some examples of these duties [3]. For the aim of conducting autonomous underwater manipulation activities, these days, AUVs are often preferred over remotely operated vehicles, also known as ROVs [4]. This is because the data that are acquired will be used to inform future decisions. This is due to the fact that the efficacy of the station-keeping and position-tracking done by the AUV is a major factor in determining the quality of the data that are obtained [3, 5]. Nevertheless, accurate trajectory tracking control of underwater robots is a very difficult task to accomplish due to the unstructured nature of the underwater environment and highly nonlinear coupled vehicle dynamics that are subject to time-varying behavior [6]. This makes the task an extremely challenging one to complete. As a result, completing the assignment will be an extremely challenging endeavor. As a consequence of this, it is highly suggested that an AUV-tracking control system be used that is outfitted with the adaptability necessary to accommodate the unpredictability of the aquatic environment. Over the course of a considerable amount of time, a number of different researchers have put up the idea of developing a variety of cutting-edge control systems for AUVs.

10.2 Literature Review This section offers a condensed analysis of the research that has been conducted on the use of motion control in AUVs for a wide range of underwater manipulation tasks. The literature review is not intended to be thorough; nonetheless, it does include the bulk of the sources that inspired or directly

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  211

link to the innovative results that are offered in this research. The objective of the literature review is not to be exhaustive. Because of the study that has been conducted on the difficulties associated with AUV control, a variety of alternative control strategies have been published throughout the course of some time. These strategies have been proposed at various points. It is also one of the most effective control techniques. To begin, Jalving et al. [7–14] designed the sliding mode control (SMC) in order to provide reliable trajectory tracking control for underwater vehicles [15]. This was done so that they could conduct research. This was accomplished with the help of the SMC. Later in the year 1991, Yoerger et al. [16] investigate the possibility of using adaptive SMC to the experimental underwater vehicle in some capacity. They accomplish this within the framework of the year 1991. In addition to this, Cristi et al. [17] described an adaptable SMC that may be used with AUVs when they are in the dive plane. Rodrigues et al. [18–20] describe the development of higher-order sliding modes for diving control of a torpedo AUV. The goal of this work was to improve the functionality of classical SMC. This was done with the intention of improving the performance of standard SMC. Extensive study on the control and coordination of a large number of autonomous vehicles operating in the horizontal plane can be found in Liu et al. [21], which makes use of the finite-time optimal formation control technique as the major control strategy. Despite this, most of the aforementioned controllers need some amount of expertise with hydrodynamics and disturbances in order to effectively create a control scheme. This familiarity might be comprehensive or partial, depending on the situation. In addition, owing to the manner in which it is implemented, the SMC-based control method is susceptible to the chattering problem. This problem might be fixed by replacing the discontinuous term sgn() with an approximation that uses a continuous function, such as the hyperbolic tangent function or one of the many other examples that could be used [22]. Increasing the order of the sliding surface may also be helpful in preventing chattering, as shown in Joe et al. [23], which describes an experiment in which this tactic was put to the test and found to be successful. In addition, the use of a time-delay controller, which has been successfully applied to the tracking control of AUV and is advocated in Kim et al. and Kumar et al. [3, 24], may significantly cut down on the amount of chattering that occurs. Intelligent control strategies, including fuzzy logic and neural network control, have seen widespread use in the operation and management of robotic systems [25, 26]. [Note: These strategies provide a different solution to the problem of overcoming the limitations of the robust control methods that were covered earlier in the article. These intelligent control strategies have shown to be useful when applied to the management of

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

212  Artificial Intelligence for Autonomous Vehicles

underwater vehicles [27–31]. Yuh [32] was the first person to propose the use of a neural network (NN) as a control mechanism for an AUV. There are several iterations of the NN, including the one-layer NN [28], the feedforward NN [33], and the adaptive NN [34–36]. Fuzzy control of AUVs is another topic that has received a significant amount of research [37–39]. Choi et al. [40] came up with the concept for a fuzzy controller that was computationally more efficient and that only took in one input variable at a time. Ishaque et al. [41] came up with a single-input fuzzy controller that has a piecewise linear control surface in 2011. This controller was designed for the purpose of managing the depth of an AUV [41]. However, research into the robustness and stability of these intelligent control systems has demonstrated to be exceedingly arduous and difficult to evaluate [42]. The integration of soft-computing tactics such as fuzzy logic computing (FLC) and support vector machine control has emerged as one of the most promising control approaches in recent years. This is as a result of the fact that it guarantees stability and has a high degree of resilience in the face of perturbations in the parameters [42, 43]. The following is a list of the key strengths that the fuzzy sliding mode control (FSMC) possesses: (1) Because the fuzzy inference system offers a qualitative interpretation, having an in-depth knowledge of the system model is not required to use it. This is due to the fact that the system gives a qualitative interpretation. (2) The fuzzy control rules [44] may be developed in a straightforward manner by using the principles of sliding mode, which not only ensures the system’s stability and resilience but also makes the process of developing the rules reasonably straightforward. As a consequence of this, FSMC has experienced a significant uptick in use over the last several years in terms of the control of underwater robots. In the beginning, [45] recommended employing sliding mode fuzzy controllers for the heading and pitch control of an AUV in order to obtain time optimum control performance. This was done in order to acquire the best possible control performance [46–53]. On the other hand, the bulk of the controllers that were covered before use decoupled control approaches and a lower-order mathematical model of an AUV, but they ignore the cross-coupling features of the system. An AUV, on the other hand, has to be controlled with a respectable level of accuracy across all six degrees of freedom if it is to be capable of engaging in interactive activities while submerged in water. The attempts that are made in [3, 54–58] are few and far between. Over the course of some time, a variety of distinct algorithms have been proposed as possible answers to the problem of redundancy resolution [59–66]. These algorithms have been put up as possibilities. An additional possibility is that the control rule might be developed inside the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  213

body-fixed reference frame in order to accomplish the same goals as the control action. In addition, there is a diverse amount of work that must be completed by auxiliary undersea vehicle maintenance system (AUVMS) in order to achieve more than one objective. Consider the fact that the AUVMS must perform end-effector position tracking as its main aim and obstacle avoidance as its secondary purpose, as an example. It is necessary for it to do both of these jobs. Control strategies that are based on configuration space or on a body-fixed reference frame are not acceptable in these sorts of conditions [67]. In contrast to this, the control that is based on the task space does not call for any inverse kinematic solutions. Because of this, it is possible that it is the option that is most suitable to be chosen. In addition, it is desirable to manage the task space when it is essential to make frequent online modifications to the space of the end-job effector [67]. The use of a feedback loop that instantly lowers task space mistakes without the need for any explicit calculation of inverse kinematics is the key advantage that can be gained by using a method that is based on the concept of task space. Because of this, carrying out such calculations is no longer necessary. There have not been a lot of effort put into going in this specific route [68, 69].

10.3 Control Problem in AUV Control System The motion control of AUVs has proven to be such a difficult endeavor for a variety of reasons, some of which are stated below; yet, this has not stopped researchers from trying. Higher-order and redundant structure when the manipulator is connected; • environmental disturbances caused by ocean currents, winddriven waves, and other factors of a nature similar to those mentioned previously. In the event that the system of an autonomous underwater robot is unable to work as planned, the ocean’s unstructured and potentially dangerous environment poses a number of difficult barriers to overcome. Therefore, it is of utmost importance to have a dependable and specialized vehicle control system that is outfitted with the capability of auto-tuning in the event that the control performance suffers while the robot is in operation as a result of shifts in the dynamics of both the robot and the environment. This is because shifts in the dynamics of both the robot and the environment can cause the control performance to suffer. This is due to the fact that changes in these dynamics might result in a decrease in the control performance.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

214  Artificial Intelligence for Autonomous Vehicles

One of the most promising and efficient nonlinear robust control systems is the classical sliding mode control, which is also known as CSMC. The ease with which it can be developed, as well as the fact that it is invariant to external disturbances and uncertainties, makes the CSMC one of the most effective nonlinear control systems. On the other hand, in order to construct a control system making use of SMC, one has to possess precise information about the system that is being controlled. However, in order to construct the control system, this sort of information must be gathered even if it is not truly available in practice. In a scenario such as this one, having access to an intelligent control method like FLC would be a more helpful tool to have at one’s disposal than any other option. It is probable that the capacity of FLC to be applied to plants that are not so clearly characterized mathematically is the single most important and defining element of FLC. On the other hand, verifying the stability of a control system that is based on fuzzy logic is a job that is difficult and demanding. Integrating SMC and FLC might very well be the solution to this problem, since it offers the opportunity of resolving it. Because it overcomes the weaknesses of one technique in favor of another, this fusion approach is fast becoming the most promising kind of control because it enables the development of certain stability and powerful control performance. In addition, this method is rapidly evolving into the form of control that has the most potential. Despite this, there are a few obstacles, issues, and constraints that need to be taken into consideration in order to be fully understood. The following is a rundown of the order in which they will be delivered to you: 1. The discontinuous control component of the SMC is the root of an annoying chattering that may be heard in the control torques. Because of this chattering, there is an increased possibility that the actuators of an AUV may wear out before their time. 2. The sliding mode cannot be put into practice unless it is first feasible to determine the limits of the uncertainty and disruptions that are now present. This is an impossibility. On the other hand, it is not actually feasible to discover it in the vast majority of instances; therefore, finding it is not something that can be done. 3. An asymptotic representation of the convergence of the position tracking errors may be obtained via the use of the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  215

4.

5.

6.

7.

generic form of the position tracking errors. As a consequence of this, the target of establishing stability in a certain length of time is one that cannot realistically be attained. It is not possible to obtain accurate values for these hydrodynamic coefficients, since the values of these coefficients fluctuate depending on the operating circumstances. This makes it impossible to calculate accurate values for these hydrodynamic coefficients. Because of this, it is not possible to collect correct information on the dynamics of an AUV for the purpose of building the control system for the vehicle. The traditional FLC includes components such as a rulebase storage area, an inference mechanism, fuzzification, and defuzzification procedures in its arsenal of tools. However, due to the fact that there are a range of technical challenges that need to be addressed, it is possible that this is not acceptable for use in applications of AUVs that are put into practical use. Some of the problems that need to be addressed include the lack of real-time response, inadequate communication bandwidth, inadequate processing capability, and inadequate onboard battery life. There is not currently a standardized approach that can be used in real-world applications of autonomous underwater vehicles to find fuzzy control rules that are appropriate for the design of an FLC (AUVs). The design of a disturbance observer, which is the most typical need of observer design in the case of underwater vehicles, frequently requires high-precision accelerators, but these accelerators are not always accessible. This is a challenge because the design of a disturbance observer is the most typical need of observer design in the case of underwater vehicles. This is a problem due to the fact that the design of a disturbance observer is a necessity for observer design that occurs most often in the context of underwater vehicles.

The AUVMS has become a kinematically redundant structure ever since it was outfitted with an attached manipulator arm. This has been the case since the attachment of the arm. In order to find a solution to this issue, it is required to put into practice resolutions for issues with duplication. The primary objective of this thesis is to develop stable and intelligent control algorithms for the purpose of maneuvering control of an AUV in the presence of an unstructured oceanic environment. This will be

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

216  Artificial Intelligence for Autonomous Vehicles

accomplished in order to fulfill the purpose of the thesis. In order to do this and have the thesis serve its intended function, this will be done. Ocean currents and waves are the unidentified external factors that are responsible for creating this environment’s turbulent conditions. In addition, the hydrodynamic coefficients themselves come with their own set of inherent uncertainties: (1) the invention of a trustworthy technique of control that is based on SMC and all of its variations, which is necessary for the deployment of this astute and dependable method of control; (2) the execution of this trustworthy method of control. Create a method for SMC control that is based on an uncertainty estimate; Develop effective control strategies by using FLC and forwardthinking control algorithms; Develop a dependable higher-order SMC that can achieve high levels of tracking precision; Conceive a method of intelligent control with increased robustness, one that combines SMC and FLC. Develop a sturdy higher-order SMC that is capable of providing accurate tracking control. The following technique will be used for the whole research in order to successfully satisfy the control goals that were outlined earlier. The mathematical pillars that the control rule is built upon to support its operation. The Lyapunov approach was used in the creation of the stability research as well as the confirmation of the suggested control plan. Investigating the simulation environment in great detail and making careful assessments of the results. Conducting an analysis of the proposed procedures in light of the standard control standards in order to ascertain whether or not these procedures are reliable and productive. In the following lines, you will find a condensed version of a few of the most significant contributions that were produced in the course of this study.

10.4 Methodology An SMC-based robust control method for position trajectory tracking control of a 6-DOF nonlinear AUV model is created and utilized in this study. SMC with PID sliding surface-based control design is able to overcome the steady state error that is often caused by traditional SMC, which is based on the boundary layer.

c( )



Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  217

Since this SMC design makes use of linear sliding hyperplanes, it is able to guarantee the asymptotic stability of the system even while it is functioning in sliding mode. This is made possible by the fact that linear sliding hyperplanes are used. Because of this, the tracking errors will finally reach the equilibrium point after an unspecified period of time has elapsed. This is a direct result of the situation. One may achieve limited time convergence, also known as terminal sliding mode (TSM), by including a nonlinear element into a conventionally linear sliding surface. This allows for the possibility of achieving TSM. The decoupled equation for forward motion can be written as follows:

(m Xa )u X uur



X| | ur | ur



1

where t1 is the control force due to propeller in surge direction, and the linear surge model at a given operating speed u0 is given as follows:

(m Xi )uo



X u

X|| | u0 | u0

1



The thrust force is calculated as a function of propeller rpm and vessel speed by performing systematic runs in the towing tank with free running self-propelled forward speed tests. The decoupled motion in sway and yaw linearized about the constant speed can be represented as follows:

Mv + N (uo) vr = bδr where M



m Yi mx g N i

mx g Yr , N (uo ) Iz Ni

m Yi 0 mx g N i

1 0

0

mx g Yr 0 Iz Nr Ys

Yr ( Xu Yi )ua N v

v 

Yv 0

0 0

m Xu ua Yr 1

v

Ys

r

Nv

0

mx g Yr u0 N r

r

0 Ns

Yas Y s , N s

N ss

Yd , Ns

(m Xv )uo Yr and b (mx g Yi )uo N r

r

N ps

On the other hand, there is a worry over singularities in the TSM design techniques. As a direct consequence of this, a non-singular TSM control, also referred to as NTSMC, has been proposed in order to achieve control devoid of singularity. A complete nonlinear 6-DOF AUV model has been positioned using this control, which has also been utilized for other

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

218  Artificial Intelligence for Autonomous Vehicles

controls. The NTSMC strategy that has been presented is evaluated for its usefulness by using a flat-fish AUV in a numerical simulation and putting it through a number of uncertain operating scenarios. This helps determine how effective the strategy is. In addition, a disturbance estimator based on non-singular fast TSMC (NFTSMC) is proposed for the purpose of position trajectory tracking of an AUVMS in task space coordinates. The use of a disturbance estimator to get an online estimate of lumped disturbances while the estimator is working on an AUVMS is a contributor to the greater robustness given by the approach that was recommended. In addition, the technique enables high-precision control in addition to quick error convergence without the need for any prior knowledge on the uncertainty bounds. Taking into account a planar AUVMS for the purpose of carrying out sophisticated underwater operations helps to demonstrate mathematically that the proposed robust NFTSMC has the potential to be implemented successfully.

Ok

1 , net k 1 exp( net k )

WjkO j j



Because it is a technique that does not need a model, the FLC is one of the most powerful tools because it can be used to design a controller with just a partial knowledge of the system. This makes it one of the most practical tools. An easier method to build a standard fuzzy logic controller (FLC) is developed, and it is given the name resilient PD-like Fuzzy Controller (PD-FZ). This method is developed independently for an unmanned aerial vehicle’s diving and steering subsystems. Then, this system is put to use for combined activities including directing, diving, and controlling the pace. In order to produce fuzzy control rules for a controlled process, the behavior of a controlled process that displays the PD type control characteristics is analyzed. This is done in order to regulate the behavior of the controlled process.



(s )

1 1 e

s



Waypoint acquisition, which is driven by a line of sight, is the procedure that allows for path tracking to be completed successfully. The simulation of the Naval Postgraduate Schools’ (NPS) AUV II operating in a number

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  219

of different types of environmental disturbances is currently being run in order to test the effectiveness of the control method that has been provided. This test is being carried out in order to ensure that the simulation is accurate. In addition to this, it has been proposed that a robust task space control technique may be used. This strategy would be based on a nonlinear PID-like FLC approach. The bulk of the recommended control strategy is composed of a feed forward term, in addition to a disturbance estimator and a PID-like FLC law. The suggested control strategy is essentially split up into two different components. The recommended approach provides the answer to the problem of task space position trajectory tracking control that an AUVMS must deal with in the context of underwater manipulation task applications.

E

1 2

Q

k 1

y kd

yk

2



In order to guarantee the successful application of a nonlinear PID-like FLC strategy, a simplified version that is referred to as single-input FLC (SIFLC) is offered. This version is based on the concept of signed distance method. This particular version of FLC is what is known as a simplified version of the program. In the task space coordinate system, a robust SIFLC, sometimes referred to as an RSIFLC, is designed and implemented in order to accommodate planar AUVMS. This is done in order to make the system more resilient so that it can be enhanced. Because of its simple control structure and design approach, the RSIFLC that was shown is suitable for real-time implementation using a microprocessor that can be purchased at a lower cost.

10.5 Results Not only does the combination of FSMC and SMC gives great resistance against changes in the values of hydrodynamic parameter, but it also ensures the stability of the system. The acronym FSMC is derived from the combining of SMC and FLC from Figures 10.1 to 10.5 represents the output images of proposed work. In addition, the “chattering” in the control input that is created by conventional SMC has been completely eradicated as a result of the use of the fuzzy approach. This has occurred because of the traditional SMC. Making

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

220  Artificial Intelligence for Autonomous Vehicles

original image

Figure 10.2  Log Gabor filtered image.

Figure 10.3  Autonomous underwater vehicle.

Figure 10.4  Detected number plate.

log gabor filtered image original image

rgb to gray converted image

Figure 10.1  Input image.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  221

Figure 10.5  Recognized number.

use of the sliding mode concept makes it feasible to create the fuzzy control rules in a simple way [45]. An adaptive variant of sliding mode control, also known as AFSMC for its acronym in full, is recommended for use in the maneuvering control of an autonomous aerial vehicle (AUV) [25] that is capable of motion in six different degrees of freedom (DOF). This adjustable mechanism increases the closed-loop system’s stability by allowing for the fuzzy consequent parameters of the FLC to be modified. When using the FSMC technique, making use of a terminal sliding surface allows one to achieve convergence in a limited amount of time and in a short amount of time. As a consequence of this, a non-singular fast fuzzy TSMC (NFFTSMC) in conjunction with a disturbance estimator is provided for the purpose of use in the position tracking control of an AUV. An FLC tool is employed whenever there is a need to reduce the amount of chattering that occurs in the control inputs. Dynamic simulation studies have been used to test the viability of the proposed AFSMC and NFFTSMC systems. These studies have been conducted in conditions that are unanticipated, which demonstrates that the viability of these systems can be verified. It is advised to utilize a chatter-free second-order TSMC for the purpose of position monitoring an AUV that is aligned to a falling aircraft. This is done in order to get the desired results. A sliding surface that is incorporated into the design of the controller is used so that there will be no reaching phase when the controller is used. On the other hand, due to the fact that it serves a sign function, it is plagued by the chattering effect. A linear integral manifold and a non-singular terminal sliding manifold are coupled together in order to carry out the steps necessary to construct the sliding manifold controller (SMC) of the second order. This is done to lessen the amount of chattering that occurs in the control input.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

222  Artificial Intelligence for Autonomous Vehicles

The use of TSM inside the proposed method makes it possible to guarantee the finite-time stability of the closed-loop system. It has been determined that the simulation goals for the depth position control of the DSRV AUV have been met. In addition to this, a nonlinear disturbance observer-based Second-order terminal sliding mode (SONTSMC) (NDO-SONTSMC) is proposed to monitor a predefined task space trajectory of an AUVMS. This is an extension of what was discussed before. This is done as an extension to what was discussed before. Effectively estimating the aggregated degree of uncertainty is the suggested NDO phrase that does not include any measurements of acceleration. Both robustness and tracking accuracy have seen significant improvements as a direct result of the integration of NDO with SONTSMC. The effectiveness of the controller is proven with the aid of numerical simulations carried out on a planar variety of AUVMS undergoing motion in the horizontal plane.

References 1. Healey, A. and Lienard, D., Multivariable sliding mode control for autonomous diving and steering of unmanned underwater vehicles. IEEE J. Oceanic Eng., 18, 3, 327–339, Jul 1993. 2. Das, A., Naval operations analysis in the Indian ocean region: A review. J. Defence Stud., 7, 1, 49–78, 2013. 3. Kim, J., Joe, H., Yu, S.C., Lee, J.S., Kim, M., Time-delay controller design for position control of autonomous underwater vehicle under disturbances. IEEE Trans. Ind. Electron., 63, 2, 1052–1061, Feb 2016. 4. Jacobi, M., Autonomous inspection of underwater structures. Rob. Auton. Syst., 67, 80–86, 2015. 5. Sarhadi, P., Noei, A.R., Khosravi, A., Model reference adaptive PID control with anti-windup compensator for an autonomous underwater vehicle. Rob. Auton. Syst., 83, 87–93, 2016. 6. Ataei, M. and Yousefi-Koma, A., Three-dimensional optimal path planning for waypoint guidance of an autonomous underwater vehicle. Rob. Auton. Syst., 67, 23–32, 2015. 7. Jalving, B., The NDRE-AUV flight control system. IEEE J. Oceanic Eng., 19, 4, 497–501, 1994. 8. Herman, P., Decoupled PD set-point controller for underwater vehicles. Ocean Eng., 36, 6–7, 529–534, May 2009. 208. 9. Miyamaoto, S., Aoki, T., Maeda, T., Hirokawa, K., Ichikawa, T., Saitou, T., Kobayashi, H., Kobayashi, E., Iwasaki, S., Maneuvering control system design

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  223

for autonomous underwater vehicle, in: MTS/IEEE Oceans 2001. An Ocean Odyssey. Conference Proceedings (IEEE Cat. No.01CH37295), pp. 482–489, Honolulu, HI, USA, 5-8 Nov 2001. 10. Kim, M., Joe, H., Pyo, J., Kim, J., Kim, H., Yu, S.C., Variable-structure PID controller with anti-windup for autonomous underwater vehicle, in: 2013 OCEANS - San Diego, San Diego, CA, USA, pp. 1–5, 23–27 Sept 2013. 11. Feng, Y., Yu, X., Man, Z., Non-singular terminal sliding mode control of rigid manipulators. Automatica, 38, 12, 2159–2167, Dec 2002. 12. Roy, S., Nandy, S., Shome, S.N., Ray, R., Robust position control of an autonomous underwater vehicle: A comparative study, in: 2013 IEEE International Conference on Automation Science and Engineering (CASE), pp. 1002–1007, Madison, WI, USA, 17–20 Aug 2013. 13. Utkin, V., Variable structure systems with sliding modes. IEEE Trans. Automat. Contr., 22, 2, 212–222, Apr 1977. 14. Young, K.D., Utkin, V., II, Ozguner, U., A control engineer’s guide to sliding mode control. IEEE Trans. Control Syst. Technol., 7, 3, 328–342, May 1999. 15. Yoerger, D. and Slotine, J., Robust trajectory control of underwater vehicles. IEEE J. Oceanic Eng., 10, 4, 462–470, Oct 1985. 16. Yoerger, D.R. and Slotine, J.J.E., Adaptive sliding control of an experimental underwater vehicle, in: Proceedings. IEEE International Conference on Robotics and Automation, pp. 2746–2751, Sacramento, CA, USA, 9–11 Apr 1991. 17. Cristi, R., Papoulias, F., Healey, A., Adaptive sliding mode control of autonomous underwater vehicles in the dive plane. IEEE J. Oceanic Eng., 15, 3, 152–160, Jul 1990. 18. Rodrigues, L., Tavares, P., Prado, M., Sliding mode control of an AUV in the diving and steering planes, in: OCEANS ‘96. MTS/IEEE. Prospects for the 21st Century, Conference Proceedings, vol. 2, pp. 576–583, Fort Lauderdale, FL, USA, 23–26 Sep 1996. 19. Lee, P.-M., Hong, S.-W., Lim, Y.-K., Lee, C.-M., Jeon, B.-H., Park, J.-W., Discretetime quasi-sliding mode control of an autonomous underwater vehicle. IEEE J. Oceanic Eng., 24, 3, 388–395, Jul 1999. 20. Salgado-Jimenez, T. and Jouvencel, B., Using a high order sliding modes for diving control a torpedo autonomous underwater vehicle, in: Oceans 2003. Celebrating the Past .. Teaming Toward the Future (IEEE Cat. No.03CH37492), vol. 2, pp. 934–939, San Diego, CA, USA, 22–26 Sept 2003. 21. Liu, Y. and Geng, Z., Finite-time optimal formation tracking control of vehicles in horizontal plane. Nonlinear Dyn., 76, 1, 481–495, Nov 2013. 22. Elmokadem, T., Zribi, M., Youcef-Toumi, K., Trajectory tracking sliding mode control of underactuated AUVs. Nonlinear Dyn., 84, 2, 1079–1091, Dec 2015. 23. Joe, H., Kim, M., Yu, S.C., Second-order sliding-mode controller for autonomous underwater vehicle in the presence of unknown disturbances. Nonlinear Dyn., 78, 1, 183–196, May 2014.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

224  Artificial Intelligence for Autonomous Vehicles

24. Kumar, R.P., Dasgupta, A., Kumar, C., Robust trajectory control of underwater vehicles using time delay control law. Ocean Eng., 34, 5–6, 842–849, Apr 2007. 25. Chen, M., Shi, P., Lim, C.-C., Adaptive neural fault-tolerant control of a 3-DOF model helicopter system. IEEE Trans. Syst. Man. Cybern.: Syst., 46, 2, 260–270, Feb 2016. 26. He, W., Dong, Y., Sun, C., Adaptive neural impedance control of a robotic manipulator with input saturation. IEEE Trans. Syst. Man. Cybern.: Syst., 46, 3, 334–344, Mar 2016. 27. Venugopal, K.P., Sudhakar, R., Pandya, A.S., On-line learning control of autonomous underwater vehicles using feedforward neural networks. IEEE J. Oceanic Eng., 17, 4, 308–319, Oct 1992. 210. 28. Jagannathan, S. and Galan, G., One-layer neural-network controller with preprocessed inputs for autonomous underwater vehicles. IEEE Trans. Veh. Technol., 52, 5, 1342–1355, Sept 2003. 29. Ranganathan, N., Patel, M., Sathyamurthy, R., An intelligent system for failure detection and control in an autonomous underwater vehicle. IEEE Trans. Syst. Man. Cybern.- Part A: Syst. Humans, 31, 6, 762–767, 2001. 30. Smith, S., Rae, G., Anderson, D., Shein, A., Fuzzy logic control of an autonomous underwater vehicle. Control Eng. Pract., 2, 2, 321–331, Apr 1994. 31. Zilouchian, A. and Jamshidi, M., Eds., Intelligent Control Systems Using Soft Computing Methodologies, CRC Press, France, Mar 2001, [Online]. Available: https://doi.org/10.1201%2F9781420058147. 32. Yuh, J., A neural net controller for underwater robotic vehicles. IEEE J. Oceanic Eng., 15, 3, 161–166, 1990. 33. Venugopal, K.P., Sudhakar, R., Pandya, A., On-line learning control of autonomous underwater vehicles using feedforward neural networks. IEEE J. Oceanic Eng., 17, 4, 308–319, 1992. 34. Shi, Y., Qian, W., Yan, W., Jun, L., Adaptive depth control for autonomous underwater vehicles based on feed forward neural networks. Int. J. Comput. Sci. Appl., 4, 3, 107–118, 2007. 35. Li, J.-H., Lee, P.-M., Hong, S.W., Lee, S.J., Stable nonlinear adaptive controller for an autonomous underwater vehicle using neural networks. Int. J. Syst. Sci., 38, 4, 327337, 2007. 36. van de Ven, P.W., Flanagan, C., Toal, D., Neural network control of underwater vehicles. Eng. Appl. Artif. Intell., 18, 5, 533–547, 2005. 37. Gavish, G., Zaslavsky, R., Kandel, A., Longitudinal fuzzy control of a submerged vehicle. Fuzzy Sets Syst., 115, 2, 305–319, 2000. 211. 38. Akkizidis, I., Roberts, G., Ridao, P., Batlle, J., Designing a fuzzy-like PD controller for an underwater robot. Control Eng. Pract., 11, 4, 471–480, 2003. 39. Kanakakis, V., Valavanis, K., Tsourveloudis, N., Fuzzy-logic based navigation of underwater vehicles. J. Intell. Robot. Syst., 40, 1, 45–88, 2004. 40. Choi, B.J., Kwak, S., Kim, B.K., Design and stability analysis of single-input fuzzy logic controller. IEEE Trans. Syst. Man, Cybern. Part B: Cybern., 30, 2, 303–309, Apr 2000.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  225

41. Ishaque, K., Abdullah, S., Ayob, S., Salam, Z., A simplified approach to design fuzzy logic controller for an underwater vehicle. J. Ocean Eng., Elsevier, 38, 1, 271–284, Jan 2011. 42. Patre, B.M., Londhe, P.S., Nagarale, R.M., Fuzzy sliding mode control for spatial control of large nuclear reactor. IEEE Trans. Nuclear. Sci., 62, 5, 2255– 2265, Oct 2015. 43. Kaynak, O., Erbatur, K., Ertugnrl, M., The fusion of computationally intelligent methodologies and sliding-mode control-A survey. IEEE Trans. Ind. Electron., 48, 1, 4–17, 2001. 44. Shahraz, A. and Boozarjomehry, R.B., A fuzzy sliding mode control approach for nonlinear chemical processes. Control Eng. Pract., 17, 5, 541–550, May 2009. 45. Song, F. and Smith, S.M., Design of sliding mode fuzzy controllers for an autonomous underwater vehicle without system model, in: OCEANS 2000 MTS/IEEE Conference and Exhibition. Conference Proceedings (Cat. No.00CH37158), vol. 2, pp. 835–840, Providence, RI, USA, 11–14 Sept 2000. 46. Chiu, F.C., Guo, J., Huang, C.C., Tsai, W.C., Application of the sliding mode fuzzy controller to the guidance and control of an autonomous underwater vehicle, in: Proceedings of the 2000 International Symposium on Underwater Technology (Cat. No.00EX418), pp. 181–186, 212, Tokyo, Japan, 26–26 May 2000. 47. Guo, J., Chiu, F.-C., Huang, C.-C., Design of a sliding mode fuzzy controller for the guidance and control of an autonomous underwater vehicle. Ocean Eng., 30, 16, 2137–2155, Nov 2003. 48. Balasuriya, A. and Cong, L., Adaptive fuzzy sliding mode controller for underwater vehicles, in: 2003 4th International Conference on Control and Automation Proceedings, pp. 917–921, Montreal, Canada, 12 June 2003. 49. Kim, H.-S. and Shin, Y.-K., Expanded adaptive fuzzy sliding mode controller using expert knowledge and fuzzy basis function expansion for UFV depth control. Ocean Eng., 34, 8–9, 1080–1088, Jun 2007. 50. Bessa, W.M., Dutra, M.S., Kreuzer, E., Depth control of remotely operated underwater vehicles using an adaptive fuzzy sliding mode controller. Robot. Auton. Syst., 56, 8, 670–677, Aug 2008. 51. An adaptive fuzzy sliding mode controller for remotely operated underwater vehicles. Robot. Auton. Syst., 58, 1, 16–26, Jan 2010. 52. Xin, S. and Zaojian, Z., A fuzzy sliding mode controller with adaptive disturbance approximation for underwater robot, in: 2010 2nd International Asia Conference on Informatics in Control, Automation and Robotics (CAR 2010), vol. 2, pp. 50–53, Wuhan, China, 6–7 Mar 2010. 53. Lakhekar, G.V., Waghmare, L.M., Londhe, P.S., Enhanced dynamic fuzzy sliding mode controller for autonomous underwater vehicles, in: 2015 IEEE Underwater Technology (UT), Chennai, India, 23–25 Feb 2015, pp. 1–7. 54. Antonelli, G., Caccavale, F., Chiaverini, S., Fusco, G., A novel adaptive control law for underwater vehicles. IEEE Trans. Control Syst. Technol., 11, 2, 221–232, Mar 2003.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

226  Artificial Intelligence for Autonomous Vehicles

55. Antonelli, G., On the use of adaptive/integral actions for six-degrees-offreedom control of autonomous underwater vehicles. IEEE J. Oceanic Eng., 32, 2, 300–312, Apr 2007. 213. 56. Zhang, S., Yu, J., Zhang, A., Discrete-time quasi-sliding mode control of underwater vehicles, in: 2010 8th World Congress on Intelligent Control and Automation, USA, pp. 6686–6690, 7–9 Jul 2010. 57. Kim, M., Joe, H., Kim, J., Yu, S.C., Integral sliding mode controller for precise manoeuvring of autonomous underwater vehicle in the presence of unknown environmental disturbances. Int. J. Control, 88, 10, 2055–2065, Jul 2015. 58. Gao, J., Wu, P., Li, T., Proctor, A., Optimization-based model reference adaptive control for dynamic positioning of a fully actuated underwater vehicle. Nonlinear Dyn., 87, 2611–2623, Dec 2016. 59. Sarkar, N. and Podder, T., Coordinated motion planning and control of autonomous underwater vehicle-manipulator systems subject to drag optimization. IEEE J. Oceanic Eng., 26, 2, 228–239, Apr 2001. 60. Soylu, S., Buckham, B.J., Podhorodeski, R.P., Redundancy resolution for underwater mobile manipulators. Ocean Eng., 37, 23, 325–343, 2010. 61. Antonelli, G. and Chiaverini, S., Fuzzy redundancy resolution and motion coordination for underwater vehicle-manipulator systems. IEEE Trans. Fuzzy Syst., 11, 1, 109–120, Feb 2003. 62. Task-priority redundancy resolution for underwater vehicle-manipulator systems, in: Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146), vol. 1, pp. 768–773, Leuven, Belgium, 20 May 1998. 63. Chiaverini, S., Singularity-robust task-priority redundancy resolution for realtime kinematic control of robot manipulators. IEEE Trans. Robot. Autom., 13, 3, 398–410, Jun 1997. 214. 64. Podder, T.K. and Sarkar, N., A unified dynamics-based motion planning algorithm for autonomous underwater vehicle-manipulator systems (UVMS). Robotica, 22, 117–128, 2004. 65. Santos, C.H.F., Guenther, R., Martins, D., Pieri, E., Virtual kinematic chains to solve the underwater vehicle-manipulator systems redundancy. J. Braz. Soc. Mech. Sci. Eng., 28, 354–361, 2006. 66. Han, J. and Chung, W.K., Redundancy resolution for underwater vehicle manipulator systems with minimizing restoring moments, in: Intelligent Robots and Systems, IEEE/RSJ International Conference on, pp. 3522–3527, San Diego, CA, USA, 29 Oct–2 Nov 2007. 67. Ismail, Z.H., Task-space dynamic control of underwater robots, Ph.D. dissertation. Heriot-Watt University, France, 2011. 68. Kim, Y., Mohan, S., Kim, J., Task space-based control of an underwater robotic system for position keeping in ocean currents. Adv. Robot., 28, 16, 1109–1119, 2014. 69. Mohan, S. and Kim, J., Coordinated motion control in task space of an autonomous underwater vehiclemanipulator system. Ocean Eng., 104, 155–167, 2015.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Control of Autonomous Underwater Vehicles  227

Security and Privacy Issues of AI in Autonomous Vehicles K. Ramalakshmi1*, Sankar Ganesh1 and L. KrishnaKumari2 Department of ECE, P.S.R. Engineering College, Sivakasi, Tamil Nadu, India 2 Department of ECE, Ramco Institute of Technology, Rajapalayam, Tamil Nadu, India

1

Abstract

Artificial intelligence is one of the emerging technologies that simulate human intelligence in machines by programming it to think like human beings and mimic their actions. An autonomous vehicle can function itself and carry out necessary functions without any human involvement. This innovative technology may provide increased passenger safety, less congested roads, congestion reduction, optimum traffic, lower fuel consumption, less pollution, and better travel experiences. Autonomous vehicles play a vital role in industry, agriculture, transportation, and military applications. The autonomous vehicle’s activities are supported by sensor data and a few artificial intelligence systems. Artificial intelligence is the collection of data, path planning, and execution in autonomous vehicles that require some machine learning techniques that are a part of artificial intelligence. But this comes with some privacy issues and security concerns. Security is an important concern for autonomous vehicles. The issues of cybersecurity while incorporating artificial intelligence in autonomous vehicles will be covered in this article, along with the growing technology of self-driving automobiles. Keywords:  Artificial intelligence, optimized traffic, autonomous vehicles, cybersecurity

*Corresponding author: [email protected] Sathiyaraj Rajendran, Munish Sabharwal, Yu-Chen Hu, Rajesh Kumar Dhanaraj, and Balamurugan Balusamy (eds.) Artificial Intelligence for Autonomous Vehicles, (229–246) © 2024 Scrivener Publishing LLC

229

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

11

11.1 Introduction Artificial intelligence systems, which use machine learning techniques to collect, analyze, and transfer data, are used in autonomous vehicles (AVs) to make the same judgments that humans make in typical vehicles. These systems are susceptible to threats that could impair the vehicle’s proper operation, just like all other IT systems. Autonomous vehicles such as self-driving cars and drones have already been created and are always getting better. However, because they can wirelessly link to everything around them, autonomous systems are susceptible to hacking by hackers. For instance, a hacked unmanned aerial vehicle (UAV) in the wrong hands could pose serious threats. Hackers may render a drone inoperable, cause it to crash into a structure, an object, a person, or a car, or fly away and take it, depending on the degree of control they were able to obtain. However, despite the dangers, computer systems are gradually growing in autonomy. Companies that create systems for autonomous vehicles and industrial robotics will require expertise to safeguard their goods against hackers shortly. Here, the challenges in security and issues related to the privacy of artificial intelligence in the autonomous vehicle are examined. Figure 11.1 shows the layered architecture of the autonomous vehicle. An autonomous car is a smart system that communicates wirelessly with infrastructure, other vehicles, and the cloud. However, an autonomous car is a group of intelligent gadgets that can sense, process, and decide in real time. Figure 11.2 depicts the electronic control units of automated vehicles. Automated vehicles consist of several electronic control units (ECUs), each of which has a specific function. Examples of these functions include an engine control system, advanced driver aid system, and navigation system.

Sensors (GPS, IMU, Vehicle Network, Camera, Direction Scanner)

Perception and scene understanding (Positioning, Vehicle state estimation, Vision, moving object and barrier detection

Behavior and Motion Planning

System Management

(Path planning, Motion Planning)

Vehicle Control and Actuation (Steering, Brake, Acceleration)

Figure 11.1  The layered architecture of the autonomous vehicle.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

230  Artificial Intelligence for Autonomous Vehicles

Camera Wireless Communication

Long Range Radar

Park assistance surround view

Traffic Sign Recognition

LIDAR

Automated Vehicle Ultrasound

Short Range Radar Surround view

Adaptive Cruise Control Blind spot detection

Rear Collision warning

Figure 11.2  Automated vehicle.

An electronic control unit is a combination of hardware and software that would contain programming code to carry out certain tasks. These ECUs can be interconnected through a variety of interfaces, including controller area networks (CANs) (King and Yu, 2017). Karnouskos and Kerschbaum (2018) discussed an intelligent vehicle featuring cameras, a microphone, and sensors, which can be used to gather vast amounts of information about its occupants. Better decision-making capabilities for driving are provided by autonomous vehicles, which eliminate intoxication, distraction, exhaustion, and the inability to make quick decisions. All of these elements contribute to the technologies’ capacity to outperform human decision-making abilities when it comes to driving (Cunneen et al., 2019). Therefore, real-time responses and error avoidance represent key hurdles for AI-integrated autonomous cars. The significance of autonomous vehicle safety and performance measures has been covered in numerous research studies. These measurements need to take into account sensor errors, programming errors, unforeseen events and entities, likelihoods of cyberattacks and threats, and hardware failures. It will be critical to develop such metrics and conduct real-time analyses of them in the future. Cyberattacks can

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  231

target the following systems like control systems, components of driving systems, communications over vehicle-to-everything networks, risk assessment, and survey systems, among other targets. The very basic threat types that require investigation and analysis are sensor attacks, mobile application-based attacks on a vehicle information system, attacks on IoT infrastructure, physical attacks, and side-channel attacks. Artificial intelligence is also used in cybersecurity to identify attacks. Architecture autonomy is a further interesting characteristic. The important subsystems to pay attention to while designing autonomous vehicles are incorporating sensors, actuators, and the corresponding control variables, speed, control mechanisms, visibility, environment of the vehicle for external observing, and object recognition. The cost of communication will rise as the number of autonomous vehicles rises. This results in packet delays or losses, which subsequently reduce performance or raise communication errors. The remaining contents of the chapter are structured as follows: Section 11.2 provides the development of autonomous cars with existing reviews. Section 11.3 illustrates the automation levels of autonomous vehicles. The architecture of an autonomous vehicle is discussed in Section 11.4. Threat model is discussed in 11.5. Furthermore, autonomous vehicles with AI in IoT-enabled environments are discussed in Section 11.6. Physical attacks using AI against autonomous vehicles are elaborated in Section 11.7 with a detailed explanation. Furthermore, section 11.8 interprets the AI cybersecurity issues for autonomous vehicles. Then, various cyberattack defense mechanisms are included in Section 11.9. Last, Section 11.10 gives the conclusion for this work.

11.2 Development of Autonomous Cars with Existing Review The first attempt at a driverless car originates back to the middle of the 1920s (Juan Rosenzweig, 2015) and gained traction in the 1980s when experts won in the production of automated interstate frameworks. This prepared semiautonomous and autonomous systems for integration (Juan Rosenzweig, 2015) of automated cars into the thruway infrastructure. Between 1980 and 2000, the majority of the original autonomous vehicle pilots were developed in the United States and Germany (Asif Iqbal, 2019). In the US DARPA, the Defense Advanced Research Projects Agency (Juan Rosenzweig, 2015), AVs are extremely used to follow the directions of the full investigation. The autonomous Google AV was loaded with massive advertisements and drew in a reservoir of power from several controls.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

232  Artificial Intelligence for Autonomous Vehicles

As recently as July 2015, Google’s autonomous armadas had traveled more than 1,000,000 km, but just 14 minor auto accidents on public streets were reported. However, not every incident involved an AV; occasionally, it was driven physically or another motorist was at fault (Asif Iqbal, 2019). In any case, the primary collision for which the autonomous vehicle was judged at fault occurred when it struck the side of a public transportation vehicle in Mountain View (Juan Rosenzweig, 2015).

11.3 Automation Levels of Autonomous Vehicles Due to significant recent advancements in the Internet of Things (IoT), Programmable Logic Controllers (PLCs), and also in related computer fields, AVs are becoming a reality. Autonomous vehicles are developed with different levels of automation. Figure 11.3 represents the six levels of automation in autonomous cars. The following are the different levels of automation (Amit Kumar Tyagi, 2021). Level 0: This is well-known as a classic automobile. There are no automated functions, and the driver is in complete charge of the vehicle. When it comes to Level 1, the car is still under the driver’s control, but

Level 0 No Automation - There are no autonomous features Level 1 Driver Assistance - These cars can handle one task at a time Level 2 Partial Automation with cruise control and lane centering Level 3 Conditional Automation - dynamic driving activities Level 4 High Automation - It can operate securely on its own Level 5 Fully Automation - It is completely self-sufficient and doesn’t require a driver

Figure 11.3  Automation levels of autonomous cars.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  233

there are some automated functions. For instance, a vehicle with automated brakes would be considered at Level 1. Level 2 vehicles are required to have two autonomous features. This almost always appears in the form of cruise control and lane centering. Level 3 denotes that the vehicle can perform dynamic driving activities like lane changes, braking, and turning. But if the automobile signals that the driver is needed, the driver must engage. Everything from lane changes, braking, steering, and turning on a turn signal can be managed by the vehicle. It can operate securely on its own. A Level 5 vehicle would not require a steering wheel or brake because a human driver is not required. It is completely self-sufficient and does not require a driver.

11.4 The Architecture of an Autonomous Vehicle The electrical and electronic architecture is the foundation of an autonomous vehicle, and an ECU is its central component. This architecture is composed of a microcontroller, a microprocessor, sensors, and actuators to carry out certain tasks. The quantity and sophistication of ECUs are growing along with the functioning of autonomous vehicles. Consequently, a variety of E/E structures (Zerfowski and Lock, 2019) are suggested and expected to be realized in the future. Each task is discretely incorporated in ECUs in the traditional distributed E/E architecture. Mody et al. (2018) depicted another type of E/E architecture called domain-based, in which communication between ECUs in the same domain is handled by a domain gateway and across domains is handled by a central gateway. For fully autonomous vehicles, a server-based architecture is suggested in which highly computational duties are moved from microcontrollers to microprocessors. Utilizing the concept of virtualization also creates a central processing unit where resources are enlisted and arranged in accordance with system needs. In contrast to a distributed design, the majority of ECUs are not used continuously; therefore, the resources are effectively used centrally. Without a significant change in the autonomous vehicle’s software development cycle, this transformation in hardware architecture cannot be implemented. Therefore, the software for autonomous vehicles is developed using the automotive open system architecture (AUTOSAR). This standard comes in two categories: classic and adaptive. When creating software for a distributed E/E architecture, an ECU has full functionality built in, and AUTOSAR classic is used. Future autonomous cars changing architecture has led to the proposal of AUTOSAR adaptable (AUTOSAR, 2019). OTA software updates are one of the remote and distributed services

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

234  Artificial Intelligence for Autonomous Vehicles

that the adaptive AUTOSAR makes possible. AUTOSAR classic developed over conventional software architecture is known as the OSEK standard, whereas AUTOSAR adaptable adheres to the interface of a portable operating system. The development of software for the ECUs in autonomous vehicles of the future will use both traditional and adaptive AUTOSAR standards.

11.5 Threat Model Figure 11.4 shows the threat model; here, the goal of the intruders is classified as below (Anam Qureshi et al., 2022). • Read updates: The intrusive party is interested in disassembling an electronic control unit’s software. • Refuse updates: The attacker prevents the vehicle from updating the software. • Deny functionality: The attacker attempts to prevent a proper ECU operation. • Control: The attacker seeks to change the vehicle’s operation. An intruder could launch a man-in-the-middle assault to intercept communication within or outside the vehicle to accomplish the aforementioned objectives. The ECUs of a self-directed car requires cryptographic Compromise ECUs in a vehicle

Compromise cryptographic keys

Rollback Attack Arbitrary software attack

Outside Vehicle Man-in-theMiddle attacks

Attacker Capabilities

Inside Vehicle

Threat Model

Compromise Resilience

Defender Goals

Control

Endless data attack

Deny Functionality

Mixed-Bundle attack Mixed-and-Match attack

Attacker Goals Deny Updates

Partial-bundle installation attack

Read Updates Freeze attack

Malicious Update Detection

Eavesdrop attack

Figure 11.4  Threat model (Karthik et al., 2016).

Drop-request attack

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  235

keys to indicate the software updates that are used to organize these keys. An intruder can acquire the software updates by carrying out an eavesdropping attack. Updates can be prevented by an attacker using a drop request attack, which blocks both interior and outside traffic on the vehicle. The freeze attack, which transfers the most recent update to the ECU even when new updates are available, is another option. By dropping some traffic, an attacker can prevent some ECUs from installing the most recent upgrades through a partial-bundle installation attack. The features may be disabled through incorporating unlimited data, rollback, mix-and-match assaults, and mixed bundles. Rollback assaults are used to install older versions of software. An attacker can conduct an infinite data attack by delivering massive amounts of data that are greater than what the ECU can store. The primary targets of this attack type are secondary ECUs. In case of a mixed bundle attack, ECU is instructed to mount mismatched forms of software, which causes interoperability problems. However, in this instance, a hacker signs incompatible software bundles for installation using the cryptographic keys. The mix-and-match assault still causes interoperability problems. The most severe attack, in which the attacker attempts to seize control of the vehicle, can be launched through overwriting an ECU’s software with malicious software.

11.6 Autonomous Vehicles with AI in IoT-Enabled Environments The Industry 4.0 revolution is significantly influenced by the Internet of Things. This is because smart autonomous devices are used to provide effective communication to improve the value chain. The goal of Industry 4.0 is to streamline company operations. In Industry 4.0, IoT is crucial to corporate operations. Researchers and practitioners will be able to reach Level 5 autonomy entirely due to the integration of AI and IoT. IoT gathers data, which AI then analyzes to create useful knowledge for decision-­ making. AI synergy makes IoT smarter. Data creation, processing, and communication are necessary for AVs. Furthermore, Mobile Information Systems are frequently supplied data about path planning and traffic congestion. IoT enables the automobile to send and receive data, as objects do not require physical intervention. Figure 11.5 depicts the Internet of Things infrastructure for autonomous vehicles using AI. A shared networked infrastructure that can exchange information instantly is necessary. For instance, device information

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

236  Artificial Intelligence for Autonomous Vehicles

Sensors

Vehicles

Network Infrastructure

Roadside Devices

IoT Ecosystem

GPS

Autonomous Vehicle

Figure 11.5  IoT infrastructure for autonomous vehicles using AI.

should be received instantly and processed quickly to allow for decision-­ making. The benefit is effective communication between devices and AVs. Additionally, different AV components are connected to a hub, which transmits and receives data. This will enable effective autonomous vehicle operation. Platforms for IoT-based autonomous vehicles consist of the following four elements (Khayyam, 2020): (1) Hardware elements such as sensors transmit and receive data from the base station or the vehicle. (2) A communication network can be used for sending and receiving data. Big Data is an accumulation of data that consists of volume, velocity, and variety. (3) Big Data technologies are required to process huge amounts of data. (4) The cloud is where the data will be stored and then distributed to different devices. Data communication between IoT devices happens at many levels. Data transmission between automobiles and between vehicles and other devices is also possible. Autonomous vehicles use inputs from a variety of channels to make decisions (Schwarting, 2018). Autonomous vehicles cannot evaluate the critical data provided and received by connected IoT devices unless judgments are based on programmed artificial intelligence systems, including rule-based systems or neural networks.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  237

Based on the data available from many sources, the predictive model outputs for object recognition, trajectory planning, and line maintenance. Artificial intelligence-based autonomous vehicles can be connected to the ecosystem in smart cities to improve path planning. Cameras, LiDAR, and GPS are used to provide network data to the IoT cloud by autonomous vehicles. Various gadgets, base stations, and network infrastructure all receive information. The IoT cloud makes it feasible to use real-time data for improved decision-making. For AI-based AVs, IoT can be crucial throughout the following stages: Data gathering: To be trained, AVs powered by artificial intelligence need a lot of data. Real-time and pertinent data are required. This can be provided via ecosystem-based IoT devices. Path planning: Trajectory planning is the foundation for path planning, which uses path and movement planning to move from one state to another that is used for high-level decisions. IoT plays a vital role in this planning strategy to give data in real time for efficient pathfinding. Act: Object identification and the response associated toward the weather are performed during this stage. This phase will be processed properly if IoT device data gathering is increased and path planning is effective.

11.7 Physical Attacks Using AI Against Autonomous Vehicles Higher levels of computational capacity and connectivity are progressively necessary for the development of increasingly autonomous and connected vehicles, which increase the attack surface and increase the likelihood of physical and cyberattacks. The security of passengers, pedestrians, other cars, and connected infrastructures may be directly impacted by some vulnerabilities in autonomous driving vehicles. Investigating potential risks brought about by the use of AI is essential. Artificial intelligence has significantly aided autonomous vehicles in a variety of ways, including object identification and course planning. However, building AI models typically involves a lot of computing work and calls for a lot of sensitive training data. Attackers are motivated to launch numerous attacks by the monetary worth of such models. Model extraction attacks can be launched by cyberattackers for financial benefit or as a stepping stone for further attacks like model avoidance. A lot of studies are being put into identifying the security flaws and vulnerabilities in AI for AVs, suggesting potential solutions, and

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

238  Artificial Intelligence for Autonomous Vehicles

outlining the possible repercussions on the vehicle and associated infrastructures. The many sensors, controls, and connecting techniques have been linked to several hazards. Security and machine learning difficulties with antivirus software take advantage of flaws in modern system software. To be more specific, a few of these security flaws and vulnerabilities frequently highlighted are: • Sensor jamming/spoofing: An AI model may be given inaccurate data, which causes the system to perform incorrectly. • Denial-of-service attacks: the channels of communication employed by AVs. It is possible to temporarily deactivate operations needed for autonomous driving. Manipulating vehicle communications: When the communication channels used by autonomous vehicles are hijacked and manipulated, it is referred to as manipulating vehicle communications. This can result, for instance, in an incorrect perception of the road infrastructure. Information sharing: The systems employed in AVs hold sensitive information about the composition of AI components as well as personal information. Such systems are now a target for data breach attacks. For AVs to develop autonomous capabilities, data and AI models are crucial. These components have a dynamic character and can alter their behavior over time as a result of learning from new data, manufacturer updates, unexpected or purposefully altered data, or other factors. This necessitates systematic testing of an AI component’s security and resilience over its entire life cycle, rather than only at a certain moment in time during development. To guarantee that the vehicle will behave appropriately when faced with either unforeseen circumstances or malicious acts like attacks based on the manipulation of inputs, including poisoning and evasion attacks, both AI models and data must be thoroughly validated. This includes establishing and enforcing rigorous continuous processes to ensure that models are free of vulnerabilities that could be exploited and that data utilized throughout the development and production stages have not been altered with malicious intent. It also means that to overcome the difficulties involved in putting this systematic validation into practice, relationships between research institutions and industrial actors need to be strengthened.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  239

11.8 AI Cybersecurity Issues for Autonomous Vehicles In a culture overrun by social networks, cloud computing, digital transactions, and automated operations, technology is advancing quickly. Although as technology advances, so do cyber threats, leading to the development of the new type of attacks, tools, and methodologies that let attackers penetrate controlled systems more closely and produce more damage, and also even avoid detection of attacks in the worst case. In a society that is continuously driven by multiples of the latest technologies such as social networks, big data, and digital exchanges, the relevant data saved or information safety and information confidentiality are constantly under significant danger to automated processes that are carried out by IT systems and administered over the Internet. As a consequence of the development of new techniques, attacks are happening more frequently, and the damage done to victims of cybercrime is getting severe. AI systems should be protected from other autonomous vehicle services and components. It is best to follow conventional cybersecurity standards. Given that an AV is a multidimensional environment, successful attacks on it may have terrible impacts. As a result, affording cybersecurity in AVs needs an exclusive holistic approach that considers all factors, the diversity of AI systems, and their interactions. AI systems in autonomous vehicles are always attempting to recognize traffic signs and road markings, find vehicles, gauge their speed, and plan the path. These systems are susceptible to deliberate attacks that try to interfere with the AI system and interfere with safety-critical operations, in addition to accidental dangers such as sudden breakdowns. The research makes several recommendations to strengthen the security of AI in autonomous vehicles, one of which is to conduct security evaluations of AI components regularly during their life span. To ensure that the vehicle constantly reacts effectively to unforeseen scenarios or hostile actions, it is essential to consistently test AI models and data. The automotive industry should embrace security by design strategy, where cybersecurity is a crucial element of digital design from the inception to the development and implementation of AI features. To deal with emerging cybersecurity concerns associated with AI, the automotive industry must expand its capacity for rapid response and increase its degree of readiness. Security attacks against AVs include processing and assaults based on scheme. Unlike scheme-based attacks, which are dedicated to an objective or element to target, processing-based assaults have an impact on the

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

240  Artificial Intelligence for Autonomous Vehicles

AV system’s availability, originality, and integrity. The smart transportation system (STS), which has already been tested by several companies in the automotive sector, including Uber, Google, and Toshiba, all over the world, is one application that includes vehicle automation. The passengers’ safety, privacy, and security concerns have been raised about the use of these trials, including impersonation, denial-of-service attacks, timing challenges, and wormhole invasions. The increase in cyberattacks against AVs, such as Sybil, DoS, eavesdropping, replay, data alteration, and vehicle hacking, as well as data leakage, traffic jams, and spoofing (Torre et al., 2018). These dangers have the potential to seriously harm AVs, people, and property (Doss et al., 2018), which could have the following effects on AVs: • Failure in driving operations: loss of brake control and engine and steering components may also impair driving operations. • Vehicle system failure: This is brought on by the inability of the door locks, lights, and passenger safety system to function properly, as well as by incorrect diagnostics. • Theft of a car is a possibility because of GPS tracking, hijacking of the vehicle, and the sale of the vehicle for ransom. • Data theft: AVs may contain Relative Strength Index data, vehicle records, such as GPS and vehicle speed, as well as personal details about the passenger and the driver can all be utilized as a target by hackers to launch cyberattacks. • An AV incident may result in injuries or fatalities to both drivers and pedestrians, raising liability concerns. • Commercial loss: Cyberattacks on AVs have the potential to cause a significant financial loss by lowering consumer confidence and loss in revenue. To launch security attacks, attackers may be enticed by the real-time data that AVs can store and access concerning driving decisions and obstacle finding via wireless connection from the central third-party server. Some issues in the design of centralized AV are the following: (i) If there is a single point of failure, the whole network may become unusable. (ii) an insufficient level of data control (iii) weak accountability and detection (iv) constrained adaptability

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  241

The abovementioned difficulties and security threats connected with conventional AVs can be reduced using blockchain. A distributed database system called blockchain keeps track of transactions in a series of blocks. In this, each participating member node has a copy of the records, but there is no such central authority or server to keep track of the complete chain. The immutability attribute of a blockchain prevents any participating member from changing a block’s record once it has been created. By applying Proof-of-Work as well as Proof-of-Stake, which was initially created and employed by Bitcoin (the digital currency) (Ali et  al., 2018), the drivers and passengers may both ride safely and securely. All of the stakeholders in the car manufacturing industry have become more aware that security and privacy protection turn into more essential criteria in the design of autonomous vehicles. This will necessitate the addition of security elements and privacy protection features from basic components to electronic systems and the complete vehicle architecture. Security attacks are a massive threat to autonomous vehicles. Technical fault from cyberattacks could result in accidents and the loss of lives. Furthermore, cruel attackers may intentionally focus on a meticulous vehicle and interrupt its regular attempt to take it and cause some damage. In addition, privacy is an important consideration in autonomous vehicles. The constant interaction between the car and its surroundings puts the user’s personal information at danger. These potential risks include information leakage, information theft, tracking, and abuse.

11.9 Cyberattack Defense Mechanisms This is the strategy, procedure, or system to lessen AV-related attacks. To guard against spyware, malware, and wormhole attacks, there are several remedies available depending on the type of assault. These include public key cryptography, firewall systems, and encryption standards (Rajesh Gupta et al., 2020). Different mechanisms against cyberattacks are depicted in Figure 11.6.

11.9.1 Identity-Based Approach The authenticity and integrity of the vehicular ad hoc network (VANET) are maintained by its use. Utilizing identity qualities for attack resistance, such as Autonomous Vehicle ID, Autonomous Vehicle IP address, and (Autonomous Vehicle) AV registration number, is the fundamental aim

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

242  Artificial Intelligence for Autonomous Vehicles

Trust based

Key based Attacks on Autonomous Vehicles

Identity based

Misbehaviour detection Machine learning based

Figure 11.6  Cyberattack defense mechanisms.

of ID-based methods. The requirement for producing digital certificates is eliminated, thereby reducing the cryptographic difficulty of an autonomous vehicle. An identity-based system’s public key records must be stored on a central server. With only one point of failure, a centralized server, all communication is susceptible to disruption. Blockchain offers a practical remedy for the problem with this ID-based system, as it is dispersed and do not have one point of failure. The likelihood of failure is reduced since each autonomous vehicle linked to the blockchain may store a replica of all of the information or public keys.

11.9.2 Key-Based Solution Asymmetric key cryptography or symmetric key cryptography are two examples of the many cryptographic techniques that can be used to create key-based trustworthy solutions for autonomous vehicles. In order to make autonomous vehicles safer from a security standpoint, hash functions could be implemented. Key-based cryptographic methods such as elliptic curve cryptography and Modern Encryption Standards provide great security while using minimal resources. The cryptographic method is limited by key validation and exchange as well as trustworthiness between automated vehicles. This can be successfully handled by blockchain technology, which is distributed, interoperable, and unchangeable by nature according to consensus procedures.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  243

11.9.3 Trust-Based Solution Trust is an additional factor to protect autonomous vehicles from numerous security and privacy concerns. It represents the AV’s “degree of faith.” Depending on how recently communications have taken place, different AVs have different levels of trust. The behavior of neighboring AVs has an impact on how trustworthy the VANET is. Any effort to alter the value puts the entire blockchain that comes after the current block in jeopardy. This activity decreases the level of trust among the other participating nodes.

11.9.4 Solution Based on Behavior Detection Malicious behavior is defined as any AV’s incorrect behavior in the VANET. The nearby AV’s trust value, which is calculated using a number of findings, mathematical computations, and previous user input on data models, may be used to pinpoint its location. Antivirus software that is malicious may alter its trust importance to indicate that it is a genuine node or alter the trust significance of another credible AV to indicate that it is hateful. A realistic alternative to issues with the possessions of immutability is a public blockchain. As a result, once the trust value of an antivirus application has been added to a blockchain block, it cannot be changed or updated.

11.10 Solution Based on Machine Learning The aforementioned defense scan only recognizes assaults that are in their database. The aforementioned techniques will not be able to recognize any attacks performed against AV that is not specifically listed as attacks. Machine learning methods like clustering, classification, and Bayesian networks can be employed in these circumstances to understand and predict threats based on existing attack patterns. It is the most recent technique used by AVs to find cyberattacks. Machine learning techniques employ a centralized database system to store data and establish patterns in order to detect the assaults. The problem with such is a one point of failure, which can be resolved by utilizing blockchain technology.

11.11 Conclusion The scientific community is beginning to observe autonomous driving and vehicles as practicable alternatives to move forward in artificial

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

244  Artificial Intelligence for Autonomous Vehicles

intelligence. Artificial intelligence-based decisions made by self-directed vehicles and driver systems could herald an innovative era of industry growth. The significant limitations of artificial intelligence continue to obstruct the development of autonomous driving. In this study, a comprehensive assessment of the application of artificial intelligence in automated vehicles is presented. The endurance of safety attacks against self-driving cars and their traditional cutting-edge remedies is examined methodically and generally in this article.

References 1. Karnouskos, S. and Kerschbaum, F., Privacy and integrity considerations in hyperconnected autonomous vehicles. Proc. IEEE, 106, 1, 160–170, Jan. 2018. 2. King, Z. and Yu, S., Investigating and securing communications in the controller area network (CAN), in: 2017 International Conference on Computing, Networking and Communications (ICNC), pp. 814–818, IEEE, 2017. 3. Karthik, T., Brown, A., Awwad, S., McCoy, D., Bielawski, R., Mott, C., Lauzon, S., Weimerskirch, A., Cappos, J., Uptane: Securing software updates for automobiles, in: International Conference on Embedded Security in Car, pp. 1– 11, 2016. 4. Qureshi, A., Marvi, M., Shamsi, J.A., Aijaz, A., eUF: A framework for detecting over-the-air malicious updates in autonomous vehicles. J. King Saud Univ. –Comput. Inf. Sci., 34, 5456–5467, 2022. 5. Zerfowski, D. and Lock, A., Functional architecture and E/E-architecture–A challenge for the automotive industry, in: 19 Internationales Stuttgarter Symposium, pp. 909–920, Springer, 2019. 6. Mody, M., Jones, J., Chitnis, K., Sagar, R., Shurtz, G., Dutt, Y., Koul, M., Biju, M., Dubey, A., Understanding vehicle E/E architecture topologies for automated driving: System partitioning and tradeoff parameters. Electron. Imaging, 2018, 17, 358-1-358-5, 2018. 7. AUTOSAR, Explanation of firmware over-the-air, 2019, URL: https://www. autosar.org/fileadmin/user_upload/standards/classic/1911/AUTOSAR_ EXP_Firmw areOverTheAir.pdf. 8. Tyagi, A.K. and Aswathy, S.U., Autonomous intelligent vehicles (AIV): Research statements, open issues, challenges and road for future. International Journal of Intelligent Networks (IJIN), 2, 83–102, 2021. 9. Rosenzweig, J. and Bartl, M., A review and analysis of literature on autonomous driving. E-Journal: Making-of Innovation, 1–13, Oct 2015. 10. Faisal, A.I.M., Yignitcanlar, T., Kmaruzzaman, Md, Currie, G., Understanding autonomous vehicles: A systematic literature review on capability, impact, planning, policy. J. Transport Land Use, 12, 1, 45–72, Jan 2019.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Security and Privacy Issues of AI in Autonomous Vehicles  245

11. Cunneen, M., Mullins, M., Murphy, F., Autonomous vehicles and embedded artificial intelligence: The challenges of framing machine driving decisions. Appl. Artif. Intell., 33, 8, 706–731, 2019. 12. Khayyam, H., Javadi, B., Jalili, M., Jazar, R.N., Artificial intelligence and internet of things for autonomous vehicles. Nonlinear Approaches Eng. Appl., 1, 39–68, 2020. 13. Schwarting, W., Alonso-Mora, J., Rus, D., Planning and decision-making for autonomous vehicles. Annu. Rev. Control, Robot. Auton. Syst., 1, 187–210, 2018. 14. Ali, M.S., Vecchio, M., Pincheira, M., Dolui, K., Antonelli, F., Rehmani, M.H., Applications of blockchains in the internet of things: A comprehensive survey. IEEE Commun. Surv. Tutor, 21, 1676–717, 2018. 15. Torre, G.D.L., Rad, P., Choo, K.-K.R., Driverless vehicle security: Challenges and future research opportunities. Future Gener. Comput. Syst., 1–20, 2018. 16. Doss, S., Nayyar, A., Suseendran, G., Tanwar, S., Khanna, A., Hoang Son, L. et al., APD-JFAD: Accurate prevention and detection of jellyfish attack in manet. IEEE Access, 6, 56954–65, 2018. 17. Gupta, R., Tanwar, S., Kumar, N., Tyagi, S., Blockchain-based security attack resilience schemes for autonomous vehicles in industry 4.0: A systematic review. Comput. Electr. Eng., 86, 106717, 2020.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

246  Artificial Intelligence for Autonomous Vehicles

Actuators and sensors, 83 Adaptive cruise control (ACC), 3 Adaptive driver control (ADC), 2 Advanced driver assistance system (ADAS), 1, 4, 59, 70, 156 Advanced emergency braking (AEB), 62 Artificial intelligence (AI), 58, 76, 133, 229 Artificial neural networks (ANNs), 56 Augmented reality (AR), 5 Automated emergency braking system (AEBS), 4 Automated external defibrillators (AEDs), 139 Automation levels, 233 Automotive open system architecture (AUTOSAR), 234 Automotive sensor, 79 Autonomous aerial vehicle (AUV), 221 Autonomous nervous system (ANS), 198 Autonomous vehicle (AV), 2, 4, 56, 59, 77 Autonomous vehicle interface and control, 91 Autonomous vehicles (AVs), 26–27, 175, 230 Autonomy, 31 autonomous levels, 32 Auxiliary undersea vehicle maintenance system (AUVMS), 210, 214

Brain–computer interface, 197 Branch convolutional neural network (B-CNN), 8 Camera acquisition, 122 Circle hough transform algorithm, 194 CNN, 7, 8, 12 Contrastive learning of musical representations (CLMR), 181 Controller area networks (CANs), 231 Convolutional neural network (CNN), 173, 174, 182 COVID-19, 142, 143 Customer relationship management system, 201 Cyberattacks, 231 DARPA, 61, 66–67 Deep learning (DL), 56, 76 deep neural network (DNN), 61 Deep learning, 38 deep neural networks (DNNs), 38 Deep neural network (DNN), 6 Defense advanced research projects agency (DARPA), 39 Denial-of-service attacks, 239 Disturbance observer, 216 Driver assistance systems (DASs), 193 Driverless car, 153 Drone technology, 134, 145

247

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Index

Ebola virus, 137 Electronic control units (ECUs), 230 Emergency medical services (EMS), 139

Intelligent vehicles, 40 Internet of Things (IoT), 233 Internet of Vehicle (IoV), 101 IoT infrastructure, 237

Facial expression recognition (FER), 174 Facial expression recognizer, 182 Fall detection device (FDD), 139 Flight controller (FC), 134 Fluorescence-activated cell sorting (FACS), 173 Frame rate, 107 Frequency-modulated continuous wave (FMCW) technology, 80 Fuzzification, and defuzzification, 218 Fuzzy logic computing (FLC), 213 Fuzzy logic, 168, 176, 177 Fuzzy sliding mode control (FSMC), 213

Lane change assistance system (LCAS), 123 Lane departure warning system (LDWS), 123 Lane maintaining system (LMS), 123 Large-field-of-view (LFOV), 13 Layered architecture, 230 LiDAR sensors, 155, 160 LiDAR, 5, 7, 11, 81, 89, 159–161 LiDAR, RADAR, Global Navigation Satellite System (GNSS), 57 Local binary pattern (LBP), 193, 203 Log Gabor filter, 221 Logical analysis of data (LAD), 197

Gaussian noise, 163 Gaussian sum particle filter, 11 Global navigation satellite system (GNSS), 89 GPS sensors, 57 Hackable drone technology, 145 Heart rate variability (HRV), 198 Hidden Markov model (HMM), 125 High dynamic range (HDR), 78 Histogram equalization model, 184 Histogram of oriented gradients (HOG), 193, 203 Industrial personal computers (IPCs), 83 Information and communications technology (ICT), 35 Intelligent car, 152 Intelligent driver assistance systems, 113–115 Intelligent transportation systems (ITSs), 49

Machine learning (ML), 2, 76 extreme learning machine (ELM), 12 SVM, 5 Markov decision process (MDP), 6 MATLAB, 8 Maximum entropy principle (MEP), 6 Micro-electro-mechanical systems (MEMS), 156 Microscopic traffic simulator, 90 Middlebury multi-view stereo (MVS), 16 MOTA—Multiple Object Tracking Accuracy, 17 MOTP—Multiple Object Tracking Precision, 17 Multi-factor-based road accident prevention system (MFBRAPS), 8 Multirotor, 136 National television system committee (NTSC), 107 Navigation system, 177

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

248  Index

Neural network (NN), 213 Neuro-fuzzy architecture, 178 PERCLOS algorithm, 197 Phase alternating line (PAL), 107 Principal component analysis (PCA), 197 Programmable logic controllers (PLCs), 233 Radio detection and ranging, 88 Random sample consensus (RANSAC), 161 Recurrent rolling convolution (RRC), 6 Region of interest (ROI), 124 Reinforcement learning, 34 RNN, 8 Round-trip time principle, 80 Sensor jamming, 239 Sensors, 88 ultrasonic sensor, 88 Sequential color with memory (SECAM), 107 Single-input FLC (SIFLC), 220 Sliding manifold controller (SMC), 222 Sliding mode control (SMC), 212 Smart cities and IoTs, 98

Smart transportation system (STS), 241 Spatial volume normalization, 179 Staphylococcus aureus, 137 Supervised learning, 33 Surveillance system, 109 SVM classifier, 204 Terminal sliding mode (TSM), 218 Threat model, 235 Traffic signs, 111 Trajectory tracking, 219 Ultrasonic sensors, 181 Unknown obstacles recognition (UOR), 85 Unmanned aerial vehicles (UAVs), 131, 132 Unsupervised learning, 34 Vehicle positioning and localization (VPL), 85 Vehicle-to-roadside unit (V2R), 41, 45 Vehicle-to-vehicle (V2V), 41, 45 Virtual focal reality (VFR), 174 Weighted edges, 204 YOLOv2, 5, 7

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Index  249

Check out these other related titles from Scrivener Publishing AUTONOMOUS VEHICLES VOLUME 1: Using Machine Intelligence, Edited by Romil Rawat, A. Mary Sowjanya, Syed Imran Patel, Varshali Jaiswal, Imran Khan, and Allam Balaram. ISBN: 9781119871958. Addressing the current challenges, approaches and applications relating to autonomous vehicles, this groundbreaking new volume presents the research and techniques in this growing area, using Internet of Things, Machine Learning, Deep Learning, and Artificial Intelligence. AUTONOMOUS VEHICLES VOLUME 2: Smart Vehicles for Communication, Edited by Romil Rawat, Purvee Bhardwaj, Upinder Kaur, Shrikant Telang, Mukesh Chouhan, and K. Sakthidasan Sankaran, ISBN: 9781394152254. The companion to Autonomous Vehicles Volume 1: Using Machine Intelligence, this second volume in the two-volume set covers intelligent techniques utilized for designing, controlling and managing vehicular systems based on advanced algorithms of computing like machine learning, artificial Intelligence, data analytics, and Internet of Things with prediction approaches to avoid accidental damages, security threats, and theft. CONVERSATIONAL ARTIFICIAL INTELLIGENCE, edited by Romil Rawat, Rajesh Kumar Chakrawarti, Sanjaya Kumar Sarangi, Piyush Vyas, Mary Sowjanya Alamanda, Kotagiri Srividya, and K. Sakthidasan Sankaran. ISBN: 9781394200566. This book presents the need for, design, and application of conversational artificial intelligence (AI). Expert knowledge is shared on leading innovations in natural language processing (NLP) and machine learning (ML) techniques that are frequently combined with more traditional, static kinds of interactive technology, such as chatbots, to create conversational AI.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

Also of Interest

ROBOTIC PROCESS AUTOMATION, Edited by Romil Rawat, Rajesh Kumar Chakrawarti, Sanjaya Kumar Sarangi, Rahul Choudhary, Anand Singh Gadwal, and Vivek Bhardwaj. ISBN: 9781394166183. Presenting the latest technologies and practices in this ever-changing field, this groundbreaking new volume covers the theoretical challenges and practical solutions for using robotics across a variety of industries, encompassing many disciplines, including mathematics, computer science, electrical engineering, information technology, mechatronics, electronics, bioengineering, and command and software engineering. CONVERGENCE OF CLOUD WITH AI FOR BIG DATA ANALYTICS: Foundations and Innovation, Edited by Danda B. Rawat, Lalit K Awasthi, Valentina Emilia Balas, Mohit Kumar and Jitendra Kumar Samriya, ISBN: 9781119904885. This book covers the foundations and applications of cloud computing, AI, and Big Data and analyses their convergence for improved development and services. SWARM INTELLIGENCE: An Approach from Natural to Artificial, By Kuldeep Singh Kaswan, Jagjit Singh Dhatterwal and Avadhesh Kumar, ISBN: 9781119865063. This important authored book presents valuable new insights by exploring the boundaries shared by cognitive science, social psychology, artificial life, artificial intelligence, and evolutionary computation by applying these insights to solving complex engineering problems. FACTORIES OF THE FUTURE: Technological Advances in the Manufacturing Industry, Edited by Chandan Deep Singh and Harleen Kaur, ISBN: 9781119864943. The book provides insight into various technologies adopted and to be adopted in the future by industries and measures the impact of these technologies on manufacturing performance and their sustainability.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

QUANTUM COMPUTING IN CYBERSECURITY, Edited by Romil Rawat, Rajesh Kumar Chakrawarti, Sanjaya Kumar Sarangi, Jaideep Patel, and Vivek Bhardwaj, ISBN: 9781394166336. This cutting-edge new volume provides a comprehensive exploration of emerging technologies and trends in quantum computing and how it is used in cybersecurity, covering everything from artificial intelligence to how quantum computing can be used to secure networks and prevent cyber crime.

SMART GRIDS AND INTERNET OF THINGS, Edited by Sanjeevikumar Padmanaban, Jens Bo Holm-Nielsen, Rajesh Kumar Dhanaraj, Malathy Sathyamoorthy, and Balamurugan Balusamy, ISBN: 9781119812449. Written and edited by a team of international professionals, this groundbreaking new volume covers the latest technologies in automation, tracking, energy distribution and consumption of Internet of Things (IoT) devices with smart grids.

Downloaded from https://onlinelibrary.wiley.com/doi/ by University Of Wisconsin-Stout, Wiley Online Library on [27/02/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

AI AND IOT-BASED INTELLIGENT AUTOMATION IN ROBOTICS, Edited by Ashutosh Kumar Dubey, Abhishek Kumar, S. Rakesh Kumar, N. Gayathri, Prasenjit Das, ISBN: 9781119711209. The 24 chapters in this book provide a deep overview of robotics and the application of AI and IoT in robotics across several industries such as healthcare, defense. education, etc.