Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms (Studies in Systems, Decision and Control, 272) 3030378403, 9783030378400

Citation preview

Studies in Systems, Decision and Control 272

Alla G. Kravets   Editor

Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms

Studies in Systems, Decision and Control Volume 272

Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

The series “Studies in Systems, Decision and Control” (SSDC) covers both new developments and advances, as well as the state of the art, in the various areas of broadly perceived systems, decision making and control–quickly, up to date and with a high quality. The intent is to cover the theory, applications, and perspectives on the state of the art and future developments relevant to systems, decision making, control, complex processes and related areas, as embedded in the fields of engineering, computer science, physics, economics, social and life sciences, as well as the paradigms and methodologies behind them. The series contains monographs, textbooks, lecture notes and edited volumes in systems, decision making and control spanning the areas of Cyber-Physical Systems, Autonomous Systems, Sensor Networks, Control Systems, Energy Systems, Automotive Systems, Biological Systems, Vehicular Networking and Connected Vehicles, Aerospace Systems, Automation, Manufacturing, Smart Grids, Nonlinear Systems, Power Systems, Robotics, Social Systems, Economic Systems and other. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution and exposure which enable both a wide and rapid dissemination of research output. ** Indexing: The books of this series are submitted to ISI, SCOPUS, DBLP, Ulrichs, MathSciNet, Current Mathematical Publications, Mathematical Reviews, Zentralblatt Math: MetaPress and Springerlink.

More information about this series at http://www.springer.com/series/13304

Alla G. Kravets Editor

Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms

123

Editor Alla G. Kravets Volgograd State Technical University Volgograd, Russia

ISSN 2198-4182 ISSN 2198-4190 (electronic) Studies in Systems, Decision and Control ISBN 978-3-030-37840-0 ISBN 978-3-030-37841-7 (eBook) https://doi.org/10.1007/978-3-030-37841-7 © Springer Nature Switzerland AG 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

This monograph devoted to the analysis of the accumulated experience of modeling and design of robotic systems solutions in Russian and international practice. The authors perform integration of deep learning methods and mathematical modeling in the successful experience of the establishment of robotic systems in the different countries and analyze causal connections of the robotic modeling and design from the positions of new industrial challenges. The book also describes intelligent control methods for robotic groups on the base of multi-agent theory and describes their control architecture. The authors study the multi-agent control methods’ implementation in conditions of uncertainty and dynamically changing environment. The book is devoted to studying robotics through the prism of human–robot collaboration. The authors determine the methodological foundations of the collaborative robotic concept as a breakthrough in modern industrial technologies. The book also analyzes the performance and ergonomic problems of the collaborative robotic systems. The authors substantiate the scientific and methodological approaches to study the modern mechanisms of space robotic development and control. Breakthrough achievements of the last decade defined novel design imperative for space robotics. The authors determine the main features of space robotic design and the key control technologies. Finally, this book dwells with theoretical foundations of industrial robotics control, formulates its concept within the frameworks of intelligent control methods and theories, and views the peculiarities of control technologies as a specific sphere of modern industrial technologies, which distinguished in the Industry 4.0. The authors study the genesis of mathematical and control methods transformation from directly human-controlled machines to knowledge-based complex robotic systems in the tewenty-first century.

v

vi

Preface

Edition of the book is dedicated to the 105th anniversary of the founder of the Scientific and Educational Center of Bauman Moscow State Technical University Academician E. P. Popov and technically supported by the Engineering Center “Automation and Robotics” of Bauman Moscow State Technical University and the Project Laboratory of Cyber-Physical Systems of Volgograd State Technical University. Volgograd, Russia November 2019

Alla G. Kravets

About This Book

Industry 4.0 requires new approaches in a secure connection, control, and maintenance of robotic systems as well as enhancing its interaction with a human. This book is devoted to open issues of new intelligent control paradigms and its usage. The book contains new results on industrial robotics, robots’ design, and modeling for various domains. We determine the methodological foundations of the collaborative robotic concept as a breakthrough in modern industrial technologies. Implemented, multi-agent models, programs, and methods could be used in future processes for control, condition assessment, diagnostics, prognostication, and proactive maintenance. Also, the issue of ensuring the space robotics systems is discussed and new reliable solutions are proposed. The authors perform integration of deep learning methods and mathematical modeling in the successful experience of the establishment of robotic systems in the different countries and analyze causal connections of the robotic modeling and design from the positions of new industrial challenges. The book’s target audience includes practitioners, enterprises representatives, scientists, students, Ph.D. and master students who conduct scientific research on the topic of cyber-physical systems development and implementation in various domains.

vii

Contents

Robotics Modeling and Design Object Recognition of the Robotic System with Using a Parallel Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shuai Yin and A. S. Yuschenko

3

Mathematical Model of a Swarm Robotic System with Wireless Bi-directional Energy Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Konstantin Krestovnikov, Ekaterina Cherskikh and Andrey Ronzhin

13

Pareto Optimal Solutions and Their Application in Designing Robots and Robotic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Irina Romanova

25

Collaborative Robotics Control and Ergonomic Problems of Collaborative Robotics . . . . . . . . . Arkady S. Yuschenko

43

Human-Robot Interaction Efficiency and Human-Robot Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rinat R. Galin and Roman V. Meshcheryakov

55

Human-Robot Cooperation in Technological Wall Climbing Robot System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V. Gradetsky, M. Knyazkov, E. Semenov and A. Sukhanov

65

Features of Human-Exoskeleton Interaction . . . . . . . . . . . . . . . . . . . . . . V. Gradetsky, I. Ermolov, M. Knyazkov, E. Semenov and A. Sukhanov Analysis of Dynamics in Human—Exoskeleton Collaborative System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Matokhina, Alla G. Kravets, Daria Volodina, Stanislav Dragunov and Vladislav Shashkov

77

89

ix

x

Contents

Multi-agent Robotic Systems Formation Control of Ground Multi-agent System Using Quadcopter with Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Stanislav L. Zenkevich, Anaid V. Nazarova and Jianwen Huo Dependence of Dynamics of Multi-robot System on Control Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Vladimir Serebrenny and Madin Shereuzhev The Application of Multi-agent Robotic Systems for Earthquake Rescue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Anaid V. Nazarova and Meixin Zhai About Controlling the Shape of the Sprinkler . . . . . . . . . . . . . . . . . . . . 147 Eugene Briskin, Yaroslav Kalinin, Konstantin Lepetukhin and Alexander Maloletov Space Robotics Modern Space Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Pavel P. Belonozhko Moving-Base Space Robots—Applying Eigen-Dynamics of a Reduced System to Synthesize Controls . . . . . . . . . . . . . . . . . . . . . 171 Pavel P. Belonozhko The Concept of Failure- and Fault-Tolerance on Base of the Dynamic Redundancy for Distributed Control Systems of Spacecraft Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Irina V. Asharina Industrial Robotics Industrial Robotics Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 Ivan Ermolov Development of an Information Control System for a Remotely Operated Vehicle with Hybrid Propulsion System . . . . . . . . . . . . . . . . . 205 Olga I. Gladkova, Vadim V. Veltishev and Sergey A. Egorov Control System of a Starting-Landing Platform with Parallel Kinematics for Pilotless Flying Machines in the Conditions of Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 I. N. Egorov Intuitive Industrial Robot Programming Interface with Augmented Reality Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Alexander Schwandt and Arkady Yuschenko

Robotics Modeling and Design

This part is devoted to the analysis of the accumulated experience of modeling and design of robotic systems solutions in Russian and international practice. The authors perform integration of deep learning methods and mathematical modeling in the successful experience of the establishment of robotic systems in the different countries and analyze causal connections of the robotic modeling and design from the positions of new industrial challenges. 1. Shuai Yin, A. S. Yushchenko—Object Recognition by Robotic System Using Parallel Convolutional Neural Networks. 2. K. D. Krestovnikov, E. J. Cherskih, A. I. Saveliev, A. L. Ronzhin—Mathematical Model of a Swarm Robotic System with Wireless Bi-directional Energy Transfer. 3. I. K. Romanova—Pareto Optimal Solutions and Their Application in the Design of Robots and Robotic Systems.

Object Recognition of the Robotic System with Using a Parallel Convolutional Neural Network Shuai Yin and A. S. Yuschenko

Abstract The work relates to research in the field of creating a collaborative robotic system—an assistant surgeon. At the request of the surgeon calling the required tool, the robot must find this tool and pass it onto the surgeon. This article covers only the part of the task related to the search by the robot of the desired tool. The task is performed using a manipulation robot, in the grip of which the camera is located. Information processing is carried out using a convolutional neural network, which includes two parallel convolutional neural networks. One of them highlights the areas of the desktop, on which the individual tools lie, highlights the individual areas characteristic of each tool, and determines the position and orientation of the tool necessary for its capture by the robot. The second one determines the type and name of the tool. The procedure of training a neural network is considered, after which the system is able to solve the “coordinate change” problem, etc. by the name of the tool, immediately find it on the desktop and determine its position and orientation. The results of experiments confirming the high efficiency of the proposed method are given. Keywords Convolutional neural network · Coordinate change · Collaborative robotic · Training · Robotic system

1 Introduction In the robotic system, the complete object detection process includes the detection of recognition areas, the recognition of objects, the determination of object orientations and the transformation of object coordinates. The task of detecting recognition areas will be a search for areas where objects can exist. For example, methods of separating an object from the environment by the properties of objects [1–3], object segmentation using parametric minimum sections [4], etc. The task of object recognition is the classification of the area of object recognition using key points SIFT [5], Hog S. Yin (B) · A. S. Yuschenko Bauman Moscow State Technical University, Moscow, Russian Federation e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_1

3

4

S. Yin and A. S. Yuschenko

Fig. 1 Diagram of the complete network operation process

[6] and dual classifier SVMs [7], etc. The task of determining object orientations is to analyze in detail the spatial orientation of recognizable objects using 3D models [8] and geometric methods [9] in the case of its known forms. The task of converting object coordinates is to convert object coordinates relative to the camera to the object coordinate relative to the manipulator. If the camera coordinate relative to the manipulator does not change, the homogeneous transformation matrix also does not change. If the camera’s coordinate relative to the manipulator changes, then each time an object coordinate transformation is performed, it is necessary to perform another calculation of the homogeneous transformation matrix. The work uses a parallel mixed convolutional neural network for object recognition. In the case of small training data, this network can be more quickly, more accurately determine the position and orientation of objects, compared to R-CNN [10], SVM. The complete process of networking is shown in Fig. 1. In the first part of the network, recognition areas are presented where the desired objects are possible. The upper convolutional neural network should classify the recognition areas of each medical tool. It also divides the recognition areas into three groups: the head of the tool, the tail of the tool and the rest of the tool. The inferior convolutional neural network classifies the recognition regions and determines the names of medical tools. Additional algorithm No. 1 one calculates the position and orientation of objects on the plane of the table, as well as the characteristic features of objects. Additional algorithm No. 2 determines the names of objects by their signs.

2 Object Recognition Areas We divide the related work into three groups: exhaustive search, segmentation, and other sample groups that do not apply to any group. In the image, the tool can be located at any position and scale, it is required to search everywhere. However, the visual search space is huge, which makes an exhaustive search computationally expensive. For an exhaustive search, it is necessary to impose restrictions on evaluation for each location and the number of search places. Therefore, most of these sliding window methods use a coarse search grid and fixed proportions. This method is often used as a preliminary selection step in a cascade of classifiers [11, 12]. The second group proposes to create a set of classes of independent hypotheses of objects using segmentation. This gives lower estimated resources since we do not need to

Object Recognition of the Robotic System with Using a Parallel …

5

invest in one better segmentation strategy. For example, the use of an excellent, but expensive contour detector [13]. The third group is developed at the request of the project. For example, use a bouncing window [14], which defines the relationship between the individual visual devices and the location of the object to predict the location of the object in new images. In our work, given that medical tools are placed on the desktop, green is used as the working area. Using binarization, one can distinguish green and other colors from the results of edge detection and determine the boundaries of the identification area shown in green. After this, identification areas and objects are separated. Cut the identification areas and consider them as input for the lower convolutional neural network. To determine the orientation of tools, it is necessary to create recognition areas (size 56 * 56) for each tool separately by using recognition points. Recognition points are generated by changes in the curvature of the edges of each tool (see Fig. 2). The recognition areas are the input to the upper convolutional neural network.

Fig. 2 The process of generating object recognition areas

6

S. Yin and A. S. Yuschenko

3 Scheme of a Parallel Convolutional Neural Network A parallel mixed convolutional neural network contains two convolutional neural networks and two additional algorithms. The task of two convolutional neural networks is to solve the problem of image classification. The task of the two additional algorithms is to calculate the coordinates and rotation angle of medical tools (see Fig. 3). The upper convolutional neural network classifies the recognition areas of each medical tool. It will divide the recognition areas into three groups: the head of the tool, the tail of the tool and the other part of the tool. The upper convolutional neural network consists of an input matrix, three convolutional layers, two fully connected layers, and it forms three output signals. The size of the input matrix is 56 * 56 * 3. The first convolution uses 3 * 64 filters (size of filters 5 * 5) and max-pool (size of filters 2 * 2 and stride 2). The second and third convolutions have 64 * 64 filters (size of filters size 5 * 5) and max-pool (size of filters 2 * 2 and stride 2). The number of input signals of the first fully connected layer with 256 outputs is determined by the output of the third convolution with the active function ReLu. In the second fully connected layer, there are 256 inputs and three outputs without an active function. The lower convolutional neural network classifies the full image recognition areas to determine the name of the medical tool. Due to the fact that the size of the object recognition areas differs from each other, it is necessary to change the object recognition areas to the size (224 * 224 * 3). For this problem, the SPP algorithm can be used [15]. The lower convolutional neural network consists of an input matrix, three convolutional layers, two fully connected layers, and sixth outputs. The size of the input matrix is 224 * 224 * 3. The first convolution uses 3 * 64 filters (size of filters

Fig. 3 Scheme of parallel convolutional neural networks. The upper and lower network consists of 6 convolutional layers and 4 fully connected layers

Object Recognition of the Robotic System with Using a Parallel …

7

5 * 5) and max-pool (size of filters 2 * 2 and stride 2). The second and third convolutions have 64 * 64 filters (size of filters size 5 * 5) and max-pool (size of filters 2 * 2 and stride 2). The number of input signals of the first fully connected layer with 256 outputs is determined by the output of the convolution with the active function ReLu. In the second fully connected layer, there are 256 inputs and 6 outputs without active function. The additional algorithm. 1 calculates the coordinates and the angle of rotation of medical tools by using the output signals of the upper convolutional neural network. To reduce object recognition errors, we calculate the average of all coordinates of the head and tail parts. With using the obtained average value, the angle of rotation of medical tool is calculated according to the mathematical formula: δ = tan−1



y1 − y2 x1 − x2

 (1)

where x1 , y1 —the average value of object head coordinates; x2 , y2 —the average value of object tail coordinates. The additional algorithm 2 determines the name of medical tools according to the mathematical formula: a = softmax( pi ); i = 1, . . . , 6

(2)

where a—serial number of the name of the medical tool; pi —the probability of outputs of the lower convolutional neural network.

4 Training of Parallel Mixed Convolutional Neural Network The main purpose of neural networks’ training, which is designed to solve various recognition problems, is to minimize the error function after several iterations. The training set contains P images. Denote by z p the p-th image of the training set, and by d p the desired p-th output of the system. Then, the error function of the network is determined by the form: E(W) = −d P · log y P

(3)

where y p —the output probability of a convolutional neural network. The loss function is determined by the formula: n Loss =

p=1

p

p

−dn · log yn n

where y p —the output probability of a convolutional neural network.

(4)

8

S. Yin and A. S. Yuschenko

To minimize the error function, it is necessary to calculate its gradient of the network error function, which requires knowledge of the partial derivatives with respect to the main parameters of the network. In the work, separate training of two convolutional neural networks and the method of backpropagation of error are used. This provides a quick and detailed analysis of the learning process and reduces learning time. There is a problem that the larger the size of a recognizable image, the higher the recognition accuracy. In this case, the learning speed drops. Therefore, the choice of the size of the recognizable image is important. For the recognition of complete medical tools, when the image size is 110 * 110 * 3, we got low recognition accuracy. When the image size is more than 300 * 300 * 3, we have a slow learning speed. To ensure faster learning speed and high recognition accuracy, the image size is 224 * 224 * 3. For recognition of the head and tail parts of the tools, the size is 56 * 56 * 3. The number of learning iterations greatly affects the accuracy of recognition. If the number of iterations is large, retraining may occur. If the number of iterations is small, then low recognition accuracy appears. According to the results of 10 experiments (CPU i7-8550U laptop), 2000 iterations (9.8 h) are needed to recognize complete tools and 13,000 iterations (6 h) are needed to recognize the head and tail parts of the tools. The accuracy of recognition directly depends on the number and variety of images of training. The greater the variety and number of learning images, the higher the recognition accuracy. At the same time, image acquisition is more difficult. The study collected 1200 samples of training for each tool and 420 test samples containing 5 medical tools and 1 other possible case for the recognition of complete medical tools. For the recognition of the head and tail of the tools used 4050 samples of training, containing 1 head, 1 tail and 1 other part of the objects.

5 Determination of Object Coordinates Relative to the Robot We use a smartphone camera (Huawei Nova2, size of photo 3968 * 2976), located on the last link of the robotic arm. Since the selected medical tool is symmetrical, its center of gravity is on the symmetry line, and the symmetry line passes through the head and tail of the medical tool. The coordinates of the center of gravity on the picture can be determined according to the formula: 

xc = k · x h + (1 − k) · xt yc = k · yh + (1 − k) · yt

(5)

where (xc , yc )—the coordinate of the center of gravity; (x h , yh )—the coordinate of the head; (xt , yt )—the coordinate of the tail; k—coefficient of object centroid.

Object Recognition of the Robotic System with Using a Parallel …

9

After obtaining the coordinate of object centroid on the images, it is necessary to perform the coordinate transformation relative to the robot by using a homogeneous transformation. The task of coordinate transformation is the transformation of the coordinates of objects relative to the camera to the coordinate relative to the manipulator. It can be determined by the formula: ⎤ ⎡ ⎤ x0 x1 ⎣ y1 ⎦ = T · ⎣ y0 ⎦ z1 z0 ⎡

(6)



 R P where T = —a homogeneous transformation; 000 1 R—3 × 3 rotation matrix; P—3 × 1 displacement matrix; ⎡ ⎤ x0 ⎣ y0 ⎦—coordinates of objects relative to the camera; z0 ⎡ ⎤ x1 ⎣ y1 ⎦—coordinates of objects relative to the manipulator. z1 The coordinate of the objects relative to the manipulator is obtained after the transformation of the movement and rotation of the coordinates of the objects relative to the camera. The coefficients of the homogeneous transformation are constant, and the transformation is linear. The coordinates of the objects with respect to the manipulator can be more easily calculated using a mathematical formula: 

x1 = a1 · x2 + b1 · y2 + c1 y1 = a2 · x2 + b2 · y2 + c2

(7)

where x1 —coordinates of objects relative to the manipulator along the x-axis; x2 — the coordinate of objects relative to the camera along the x-axis;y1 —coordinates of objects relative to the manipulator along the y-axis; y2 —the coordinate of the objects relative to the camera along the y-axis. To determine the coefficients a1 , a2 , b1 , b2 , c 1 , c2 , it is necessary

to measure three points of objects relative to the camera x11 y11 , x 12 y12 , x 13 y13 and

three points of identical objects relative to the manipulator x21 y21 , x22 y22 , x23 y23 . Now we have 6 equations and 6 unknowns change a1 , a2 , b1 , b2 .

6 Experiments The experiment mainly studied the relationship between the number of iterations and recognition accuracy, the relationship between the number of convolutions and the

10

S. Yin and A. S. Yuschenko

processing speed of a single image, the implementation of the algorithm in robotics. In this work program Tensorflow, OpenCV, IP camera are used. Figure 4 shows the output of the complete network. In Fig. 4. The red rectangle is the recognition area; Yellow rectangle—the minimum area of objects; The pink dot is the head of the objects; The green dot is the tail of the objects; The blue dot is another part of the objects; The big blue dot is the center of gravity of objects. Tables 1 and 2 show the results of the study for the lower convolutional neural network. Special attention should be paid to the identification accuracy in the table only for training data. For testing data, our model recognition accuracy can reach 98% using 2000 iterations (9.8 h for training). For overfitting, we take random discarding some neurons. From the tables, you can find out when 3 convolutions and 2000 iterations are selected, the recognition accuracy is significantly improved. As the number of convolutions increases, the size of the model decreases. This is due to the storage format of the model.

Fig. 4 The result of parallel mixed convolutional neural networks

Table 1 The relationship between the number of iterations and recognition accuracy, learning time, losses, when using three convolutions Number of iterations

10

50

100

500

1000

2000

3000

Accuracy (%)

81.2

90.6

100

100

100

100

100

Loss

0.4515

0.3470

0.1280

0.0038

3.1e−4

1.5e−5

1.1e−5

Training time

3m

13 m

26 m

2.17 h

4.37 h

9.25 h

15 h

Object Recognition of the Robotic System with Using a Parallel …

11

Table 2 The relationship between the number of convolutions and the processing speed of a single image Number of convolutions

1

2

3

4

6

8

Single image processing time (3968 * 2976) (s)

1.48

1.27

1.27

1.10

1.31

1.02

Training time (h)

1.87

4.16

4.37

4.5

5.66

7.27

Loss

0.0017

5.1e−4

3.1e−4

0.0014

0.0011

0.0018

Model size

2.29G

589M

149M

40.3M

8.94M

8.47M

Accuracy (%)

100

100

100

100

100

100

7 Conclusion The developed software allows solving the tasks set, including recognizing certain surgical tools, determining their coordinates and orientation relative to the manipulation robot, which ensures the automatic capture of the tools and their transportation to the surgeon. Next step, we plan to use the depth camera (Kinect) to identify and calculate the coordinates of the surgeon’s hand.

References 1. Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2189–2202 (2012) 2. Uijlings, J., van de Sande, K., Gevers, T., Smeulders, A.: Selective search for object recognition. Int. J. Comput. Vis. 104, 154–171 (2013) 3. Endres, I., Hoiem, D.: Category independent object proposals. In: ECCV (2010) 4. Carreira, J., Sminchisescu, C.: Automatic object segmentation using constrained parametric min-cuts. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1312–1328 (2012) 5. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004) 6. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005) 7. Cortes, C., Vapnik, V.: Support vector machine. Mach. Learn. 20(3), 273–297 (1995) 8. Zhou, H., Huang, T.S.: Tracking articulated hand motion with Eigen dynamics analysis. In: Proceedings of the International Conference on Computer vision, Nice, France, 14–16 Oct 2003, Vol. 2, pp. 1102–1109. IEEE Computer Society, Washington DC (2003) 9. Yoruk, E., Konukoglu, E., Sankur, B., Darbon, J.: Shape-based hand recognition. IEEE Trans. Image Process. 15, 1803–1815 (2006) 10. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014) 11. Harzallah, H., Jurie, F., Schmid, C.: Combining efficient object localization and image classification. In: ICCV (2009) 12. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57, 137–154 (2004) 13. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 898–916 (2011) 14. Vedaldi, A., Gulshan, V., Varma, M., Zisserman, A.: Multiple kernels for object detection. In: ICCV (2009) 15. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. In: ECCV (2014)

Mathematical Model of a Swarm Robotic System with Wireless Bi-directional Energy Transfer Konstantin Krestovnikov , Ekaterina Cherskikh

and Andrey Ronzhin

Abstract Currently, the problem of charging of autonomous swarm agents and energy transfer among them is an important research topic. Research projects, where each agent can transfer and accept energy, are insufficiently developed. Present mathematical models don’t account for peculiarities of wireless energy transfer among swarm agents, acting inside the swarm, as well separately from it. Further, we consider the existing mathematical model and current research in this domain. Keywords Wireless charging system · Mathematical model · Swarm model · Bi-directional energy transfer · Energy transfer in the swarm

1 Introduction In paper [1] authors present a macroscopic analytical model of collaboration in a robotic system. The model consists of a series of linked differential equations, describing the dynamics of group behavior. Having presented the conceptual model, the authors further analyze a specific collaboration case. There is no explicit connection and coordination among the robots. The presented mathematical approach is a generalized one and may be applied well to other multi-agent systems. In [2] the authors describe an approach, enabling swarm robots to maintain standalone and collaborative power self-regulation. Onboard charging electronics and smart dockstations provide for autonomous charging of the robots. Here the authors propose an

K. Krestovnikov (B) · A. Ronzhin Laboratory of Autonomous Robotic Systems, St. Petersburg Institute for Informatics and Automation of Russian Academy of Science SPIIRAS, 14-th Linia VI, No. 39, St. Petersburg 199178, Russian Federation e-mail: [email protected] E. Cherskikh Laboratory of Big Data Technologies of Social Cyberphysical Systems, St. Petersburg Institute for Informatics and Automation of Russian Academy of Science SPIIRAS, 14-th Linia VI, No. 39, St. Petersburg 199178, Russian Federation © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_2

13

14

K. Krestovnikov et al.

approach for ensuring energy homeostasis in a robot swarm. Particularly, the aggregate energy content of the swarm may be used to prioritize charging demand for different agent, or for global self-regulation of the swarm, tuning individual workflows of the robots. Such regulation should ensure efficient usage of docks (which number is limited) and prevent critical energy depletion by any single agent. In [3] authors propose an inductive wireless robot charging system, consisting of several charging surfaces. To determine, how successfully the whole system can maintain robust swarm performance, we developed a model, accounting for swarm size, robot behavior and charging zone configuration. Based on this model, we developed a prototype with 12 charging surfaces and a small mobile robot Mona. Regular distribution of the charging devices and their proximity to each other ensure, that the robots frequently cross-dock stations. The robot charges every 15–30 s, depending on its velocity. In [5] authors present a charging system for E-bikes with supply voltage 46 V, output voltage 36 V, transmitted power 96 W, the efficiency of the system was 79%. Such systems, whose main operational principle depends on inductive power transfer, are also presented in [4, 6–9, 16]. In [10] a dynamic wireless charging system is considered, the design of the power transmission device and the necessary positioning algorithm. Here we study a system based on centralized energy capture mechanism (one receiving coil on the robot and one transmitting coil of the charger) and decentralized energy capture (two receiving coils, mounted on the robot, one transmitting coil on the charger). In this paper we present the model and performance algorithm for the charging system, that may be applied with swarm robots. In [11, 20] authors present a macroscopic probabilistic model for adaptive foraging and further conclude, that such an approach may be used in any adaptive swarm system. The swarm adapts the ratio of foraging units to the resting ones, to increase the food volume, available for the whole swarm. The probabilistic finite state machine (PFSM) and a number of differential equations have been developed to describe the collective foraging process at the macroscopic level. In [12] similar methods are used to describe the high-level swarm behavior. In [13, 14] mathematical description of a system is given, where some groups are formed in a multi-agent robotic system for more efficient problem-solving. The obtained model can be mathematically expressed as a series of differential equations, describing, how the number and distribution of groups change in time. In this research authors employ macroscopic modeling, allowing easily manage agent behavior. In [15] authors present energy search by swarm robots. This paper presents an algorithm, describing the behavior of each agent in dock search mode as well in operating mode. The agent may decide, whether continuing the work or starting to search the dock, where it can charge up to an acceptable battery charge level. Some robots are specialized in only a few charging activities, whereas others ca behave more versatile. The developed model ensures the good energy efficiency of the swarm. In [16] the authors study, how the integral of linear of birth and death processes may be used in terms of swarm system analysis. They show, that when robot swarm can be modeled as a linear birth-death process, well-established results can be used to

Mathematical Model of a Swarm Robotic System …

15

compute the expected value and/or the distribution of important swarm performance measures such as the swarm activity time or the swarming energy consumption. In [17] authors present a mathematical model of object foraging in a homogeneous system, including several robots. Two scenarios of object foraging have been considered: simplified foraging task, where robots only forage objects; advanced one, where they locate objects and deliver them to the previously defined place. The algorithm finds the optimal group size, maximizing the performance of this group. For larger groups performance decreases. Further authors examine models and compare their performance with modeled outcomes. The object foraging model, given in this paper, features a quantitative analytical approach of collective behavior in multi-agent systems. In [18] wireless dock station is presented, intended for charging of socially interactive robot m3Pi. The operating principle of this system is based on magnetically coupled resonant circuits. Energy transfer is achieved via magnetic induction. Robots are situated on a 17.7 mm thick tabletop whereas the dock station is under the table. The robot must reach the point above the dock to begin the energy consumption process. Energy is transferred via a high-frequency magnetic field. The receiving part of the system consists of a rectifier and a step-up converter for power regulation during battery charging. The reference transmitted power is 7 W, reference charge current—0.4 A. The maximum system efficiency was 43%. In [19] authors present a system, which can simultaneously ensure wireless supply and bi-directional connection among several mobile robots. The first prototype is a 60 × 60 cm surface, ensuring power supply and bi-directional connection in an initial group of five robots, consuming 200 mW each. Energy transfer is performed via magnetically coupled resonant circuits with a high goodness of the transmitting circuit, which is situated under the working surface. The system exhibits the power of 4.1 mW/cm2 at average. The system ensures the connection between the charging surface and all the robots via amplitude modulation of the magnetic field. It also ensures connections among individual robots. In [21–24] authors consider some issues of positioning and connection of robots in an autonomous reconfigurable system. They describe how to connect robots with active and passive gripper parts. Authors also consider a robotic array, where individual agents are connected to collectively interact with the environment. Similar tasks are to be solved when transmitting power among agents. Presented papers consider some types of mathematical models for swarm robotic systems, which have a common shortcoming: they don’t allow energy transfer among swarm agents. Mathematical models also don’t account for a possible number of agents, required to complete the task and don’t detail the relation of the power features of the agents with the swarm parameters as a whole.

16

K. Krestovnikov et al.

2 Development of a Mathematical Model Here we present a generalized mathematical model, describing the energy potential of an individual swarm agent and of a swarm as a whole. When developing this model, we used some elements of set theory [25–27]. First, suppose a number of swarm agents (Fig. 1): ni ∈ N = {n1 . . . nm }, where ni —functional swarm agent, N—swarm agent set, which ni belongs to. The functional swarm agent (Fig. 2) has some parameters. Number of possible energy connections of an individual swarm agent can be expressed as mk , ni = ni (m1 . . . mk ). The  functional agent has an actual energy reserve, specifically epi ∈ Ep = ep1 . . . epm , belonging to the actual energy reserve of the whole swarm—Ep . The numerical value of the actual energy reserve is calculated as: 

Ep =

m 

epi

(1)

i=1

Fig. 1 Swarm agents

Fig. 2 Functional swarm agent

N, Ep, Ed, Emax

Mathematical Model of a Swarm Robotic System …

17

δe2

Fig. 3 Energy transfer between swarm agents

Swarm agent has a parameter of desired energy reserve—edi , belonging to the desired energy reserve of the whole swarm—Ed , edi ∈ Ed = {ed1 . . . edm }. In such case the numerical value of the desired energy reserve can be calculated as follows: 

Ed =

m 

edi .

i=1

The agent has a parameter emaxi —maximum possible energy reserve. Energy reserve of an individual agent belongs to the maximum energy reserve of the whole swarm—Emax , emaxi ∈ Emax = {emax1 . . . emaxm }. In such case the numerical value of the maximum possible energy reserve can be calculated as follows: 

Emax =

m 

emaxi .

i=1

In Fig. 3 we show two swarm agents during energy transfer. Each energy transmission channel is characterized by a parameter of transmitted power—Wchm . The maximum possible power, transmitted by a swarm agent—Wtxi , depends on the number of energy connections of an individual swarm agent: Wtxi = (k · Wchm ). Power, get by a swarm agent, depends on the number of energy connections, the capacity of transmission channels and efficiency of energy transfer among swarm agents: Wrxi = (k · Wchm · η)

(2)

If a swarm agent to solve a problem needs a certain amount of energy, greater than its actual energy potential, it must restore the energy reserve up to the desired level— edi . The difference between the actual and the desired energy reserve is determined by a parameter ∂ei = edi − epi . When the task is fulfilled by the entire swarm, and the required amount of energy exceeds its actual  energy potential, the energy reserve should be restored up to the desired level— Ed . The difference between the actual and the desired energy reserve is determined by a parameter ∂E. Figure 4 illustrates the swarming energy restoration process.

18

K. Krestovnikov et al.

δE δe1 δe2 δe3 δe4 δe5 δe6

δe7

Fig. 4 Swarm energy reserve replenishment process



Ed −



Ep = ∂E

The agent needs some time to achieve the desired energy level, namely te = To achieve the desired energy reserve level—∂E, the swarm needs time tE . tE =

∂E nch · Wrxi

(3) ∂ei . Wrxi

(4)

where nch —the number of channels, transferring energy to the swarm. Suppose, the swarm has been passed a task with the following parameters: time—ttask , energy— Etask , required to complete the task. One more parameter of this task is how many agents can perform it simultaneously. To complete the task, a single robot from the swarm may be required or multiple ones, depending on the task in question. Maximum energy reserve of an individual agent—emaxi , may exceed the energy level, required to complete the task. If emaxi > Etask , then an individual agent can complete the task, Etask := edi . Then: edi − epi = ∂ei

(5)

Mathematical Model of a Swarm Robotic System …

19

Time to transfer energy to the agent: te =

∂ei Wrxi

(6)

Total time required to complete the task: τtask = ttask + te

(7)

If the maximum energy reserve of an individual agent—emaxi , is less or equal to agent the energy level, required to complete the task emaxi ≤ Etask , a single swarm  will be insufficient to complete the task and a group of robots Ntask ⊃ n1 . . . n j should be dedicated to doing it. The number of swarm agents, required to complete the task, is calculated as follows: |Ntask | =

Etask emaxn

(8)

If the actual energy reserve of the group is less than required to complete the task, it should be replenished. In this case calculation of the parameters of the group Ntask is the same as for the entire swarm, then Etask := Ed , ∂E calculated as (3), and the time for energy transfer to the group before task execution—as (4). If the task may be performed by several robots simultaneously, then the execution time is: τtask = ttask + tE

(9)

If only a single agent can perform the task at any given time, we should account for power losses, arising by energy transfer among agents during task execution. In this case, the energy to be transferred to the group of agents before task execution—∂Eo , includes the energy of power losses by energy transfer during Ntask . Parameter ∂E is calculated in the same manner, as when several robots perform the task simultaneously. ∂Eo = (Etask − emaxi ) · (1 − η) + ∂E

(10)

In this case parameter tE is calculated as in (11), and total task execution time τtask as in (9) tE =

∂Eo nch · Wrxi

(11)

20

K. Krestovnikov et al.

3 Limitations of the Developed Model The number of energy transfer channels to the swarm is limited by the following condition: 1 ≤ nch ≤ nchmax . Where nchmax —maximum possible number of energy transfer channels for this swarm. Parameter nchmax = m · k depends on the number of swarm agents and on the number of possible energy connections. The minimum value tE can be achieved, if we establish the maximum possible number of energy transfer channels to the swarm. Then tEmin = nchmax∂E·Wrxi . Based on the condition, given above, we get the maximum value tE − tEmax when using the minimum number of energy transfer channels, i.e. a single channel: ∂E . tEmax = 1·W rxi If several swarm agents are required to complete the task, but only one robot can work on it at a time, we calculate the energy to be transferred to the group of agents to complete the task with (10). Having calculated the parameter ∂Eo , accounting for power losses among agents of the group in question, we need to check, if the following condition is true: Ntask · emaxi − Ep ≥ ∂Eo

(12)

If the inequation isn’t true, the number of the agents in the group should be increased, then we redo calculations Ep , ∂E and ∂Eo . We should also note, that when edi ≤ epi , no time for energy transfer to the agent prior to the task is required, and τtask = ttask . The analogous situation arises with the parameters of the swarm (Ed , Ep ) or its subset (groupNtask ), dedicated to complete the task. When calculating ∂Eo with (10) parameter emaxi is used. Using emaxi means, that the agent at a task will have the maximum energy potential level before to approach this task, which allows reducing power losses during energy transfer among the agents from the Ntask subset.

4 Use Case of the Developed Model Currently, our laboratory has developed homogeneous robot models, shown in Fig. 5. Each robot is equipped with Li-Ion 3s batteries with 1700 mA h capacity. Because of their geometrical features, these models can have two energy transfer channels each. Data on the developed and tested wireless energy transfer systems show, that their efficiency ranges from 60 to 80%. Currently, we have a prototype WPTS-3 (wireless energy transfer system, release 3), developed for a mobile multipurpose robotic platform. The efficiency of WPTS-3 by transmitted power of 100 W and transfer distance 15 mm reaches 80%. Taking into account possible robot positioning inaccuracy and potential coil misalignment, for theoretical purposes we took energy transfer efficiency of 65%.

Mathematical Model of a Swarm Robotic System …

21

Fig. 5 MARS homogeneous robot models

Agent number in the swarm—N = 3. The number of possible energy connections of an individual swarm agent k = 2. Number of energy transfer channels from agent to the swarm—nch = 3. Maximum energy reserve of the agents: emaxi = 18.36 W h. Let the first agent has an actual energy reserve 70%: ep1 = 18.36 · 0.7 = 12.85 W h; the second—50%: ep2 = 18.36 · 0.5 = 9.18 W h; the third—65%: ep3 = 18.36 · 0.65 = 11.93 W h. Taking into account the proposed energy transfer efficiency η = 65%, maximum power, accepted by a swarm agent via an individual channel, will be calculated as (2) and equals to 6.5 W. Suppose, we need Etask = 45 W h energy to complete the task and only one robot at a time can work on it. Then the minimum number of swarm agents, required to complete the task, will be calculated as (8) and equals to 3 units. The actual energy potential of agents of the group Ntask will be calculated as (1)—Ep = 33.96 W. Because Ep < Etask , Etask := Ed , energy to be transferred to the agent group Ntask to complete the task, as follows from (3), will be 11.04 W. As only one agent can perform the task at the time, we need to account for power losses by energy transfer among swarm agents: ∂Eo = (Etask − emaxi ) · (1 − η) + ∂E = 20.36 W. Further, we check, that the energy potential of Ntask agents is sufficient, based on the calculated value ∂Eo . In our case the in Eq. (12) is true—21.12 W ≥ 20.36 W, and there’s no need to add an agent to Ntask . Time, required to transfer energy to the swarm via 3 channels is calculated as (11) and equals 3744 s. Let’s calculate the total time, required to complete the task, as in (9). If the time to complete ttask = 600s, then the total time for this task will be 4359 s.

22

K. Krestovnikov et al.

The given theoretical calculations with predefined parameters of swarm robots in terms of the presented task allowed to calculate, how much time is required to prepare to the task, specifically, to transfer energy to the agents. Having a task with presented parameters at hand (this task has been delivered from the control system), we calculated the total time to complete the task and energy input for a group of agents.

5 Conclusion The mathematical model, presented in this paper, describes a swarm robotic system and interactions within it in terms of its power features. The model accounts for possible energy transfer among swarm agents, as well as for other energy parameters of individual agents and swarms as a whole. Tasks being delivered to the swarm from the control system, are described in terms of time and energy, required to complete these tasks. The proposed mathematical model accounts for a possible number of agents, that can simultaneously perform it. The developed model has been approbation on a homogeneous swarm robotic system, consisting of three agents. We theoretically calculated energy parameters of a group of swarm agents, that perform a given task. We accounted for power losses during energy transfer among agents, performing the task, as the task assumed, that only one agent at a time will work on it. We calculated the total time, needed to perform the task by a group of swarm agents. Further research will be aimed to design swarm management algorithms, which will implement the presented mathematical model and approbation it in a virtual. Acknowledgements This research is supported by the RFBR Project No. 19-08-01215_A.

References 1. Lerman, K., Galstyan, A., Martinoli, A., Ijspeert, A.: A macroscopic analytical model of collaboration in distributed robotic systems. Artif. Life 7(4), 375–393 (2001) 2. Kernbach, S., Kernbach, O.: Collective energy homeostasis in a large-scale microrobotic swarm. Robot. Auton. Syst. 59(12), 1090–1101 (2011) 3. Arvin, F., Watson, S., Turgut, A.E., Espinosa, J., Krajník, T., Lennox, B.: Perpetual robot swarm: long-term autonomy of mobile robots using on-the-fly inductive charging. J. Intell. Rob. Syst. 92(3–4), 395–412 (2018) 4. Pellitteri, F., Boscaino, V., Di Tommaso, A.O., Miceli, R., Capponi, G.: Experimental test on a Contactless Power Transfer system. In: Ninth International Conference on Ecological Vehicles and Renewable Energies (EVER), pp. 1–6 (2014) 5. Dubal, P.: Rezence: wireless charging standard based on magnetic resonance. Int. J. Adv. Res. Comput. Commun. Eng. 4(12), 198–200 (2015)

Mathematical Model of a Swarm Robotic System …

23

6. Itoh, J.I., Noguchi, K., Orikawa, K.: System design of electric assisted bicycle using EDLCs and wireless charger. In: 2014 International Power Electronics Conference (IPEC-Hiroshima 2014-ECCE ASIA), pp. 2277–2284. IEEE (2014) 7. Low, Z.N., Chinga, R.A., Tseng, R., Lin, J.: Design and test of a high-power high-efficiency loosely coupled planar wireless power transfer system. IEEE Trans. Ind. Electron. 56(5), 1801– 1812 (2009) 8. Krestovnikov, K.D., Soleniy, S.V.: Wireless charging system for mobile robots. In: Zavalishin’s Readings ER(ZR)-2017, Youth Forum, pp. 71–76 (2017) 9. Saveliev, A.I., Krestovnikov, K.D., Soleniy, S.V.: Development of a wireless charger for a mobile robotic platform. Intellectual energy systems. In: Works of the V International Youth Forum, pp. 197–201 (2017) 10. Liu, H., Huang, X., Tan, L., Guo, J., Wang, W., Yan, C., Xu, C.: Dynamic wireless charging for inspection robots based on decentralized energy pickup structure. IEEE Trans. Ind. Inf. 14(4), 1786–1797 (2018) 11. Liu, W., Winfield, A.F.T.: Modeling and optimization of adaptive foraging in swarm robotic systems. Int. J. Robot. Res. 29(14), 1743–1760 (2010) 12. Winfield, A.F.T., Liu, W., Nembrini, J., Martinoli, A.: Modelling a wireless connected swarm of mobile robots. Swarm Intell. 2(2–4), 241–266 (2008) 13. Lerman, K.: Design and mathematical analysis of agent-based systems. In: FAABS 2000, LNAI 1871, pp. 222–234 (2001) 14. Gazi, V., Fidan, B.: Coordination and control of multi-agent dynamic systems: models and approaches. In: International Workshop on Swarm Robotic, pp. 71–102. Springer, Berlin (2006) 15. Kernbach, S., Nepomnyashchikh, V.A., Kancheva, T., Kernbach, O.: Specialization and generalization of robot behaviour in swarm energy foraging. Math. Comput. Model. Dyn. Syst. 18(1), 131–152 (2012) 16. Khaluf, Y., Dorigo, M.: Modeling robot swarms using integrals of birth-death processes. ACM Trans. Auton. Adapt. Syst. (TAAS) 11(2), 8 (2016) 17. Lerman, K., Galstyan, A.: Mathematical model of foraging in a group of robots: effect of interference. Auton. Robots 13(2), 127–141 (2002) 18. Hasan, N., Cocar, I., Amely, T., Wang, H., Zane, R., Pantic, Z., Bodine, C.: A practical implementation of wireless power transfer systems for socially interactive robots. In: IEEE Energy Conversion Congress and Exposition (ECCE) (2015) 19. Deyle, T., Reynolds, M.: Surface based wireless power transmission and bidirectional communication for autonomous robot swarms. In: 2008 IEEE International Conference on Robotics and Automation, pp. 1036–1041 (2008) 20. Liu, W., Winfield, A.F., Sa, J., Chen, J., Dou, L.: Towards energy optimization: emergent task allocation in a swarm of foraging robots. Adapt. Behav. 15(3), 289–305 (2007) 21. Pavliuk, N.A., Krestovnikov, K.D., Pykhov, D.E, Budkov, V.: Design and operation principles of the magnetomechanical connector of the module of the mobile autonomous reconfigurable system. In: International Conference on Interactive Collaborative Robotics, pp. 202–212. Springer, Cham (2018) 22. Pavliuk, N.A., Krestovnikov, K.D., Pykhov, D.E.: Mobile autonomous reconfigurable system. Problemele Energeticii Regionale 1, 125–135 (2018) 23. Andreev, V.P., Pletenev, P.F.: Method of information interaction for distributed control systems of robots with modular architecture. Trudy SPIIRAN 2, 134–160 (2018) 24. Pshihopov, V.K., Medvedev, M.Y.: Group control of the movement of mobile robots in an uncertain environment using unstable modes. Proc. SPIIRAN 5(60), 39–63 (2018) 25. Aleksandrov, P.S.: Introduction to set theory and general topology. “Science”, The Main Edition of the Physical and Mathematical Literature, Moscow (1977) 26. Hausdorf, F.: Set Theory. Ripol Classic (2014) 27. Cantor, G., Kolmogorov, A.N., Medvedev, F.A., Yushkevich, A.P.: Works on Set Theory. “Science”, Moscow (1985)

Pareto Optimal Solutions and Their Application in Designing Robots and Robotic Systems Irina Romanova

Abstract New solutions in the field of modern robotic systems are accompanied by increased requirements for the quality of their work. Often, the individual quality criteria are mutually contradictory, which in turn requires the involvement of methods and technologies for solving problems in the framework of multi-criteria optimization (MCO) or multi-criteria decision-making tasks. The use of these methods allows you to get a whole set of solutions that allows you to both adjust the system of preferences, and to take a fundamental decision on changing the structure of the robotic system. The analysis of multicriteria optimization methods for the tasks of robotics has been carried out. The most important areas of analysis and synthesis of robotic systems in which it is advisable to use the construction of Pareto-optimal solutions are identified. The tasks of the MCO are considered in relation to the design of UAV of the MCO in relation to the design of UAV control systems. Obtaining analytical solutions allows identifying the most important factors that determine the shape of the Pareto front. The problem of building the Pareto front in the conditions of insufficient experimental data, as well as fuzzy preferences, is formulated. The use of the Pareto front to select the coefficients of the regulator is shown. Receiving a set of compromise solutions that can be presented to a decision maker allows you to make the final choice more reasonable, even in the conditions of insufficient data or ambiguity of preferences, and thereby improve the quality of the designed robotic systems. Keywords Multi-criteria optimization · Pareto front · Optimum robotic system

I. Romanova (B) Bauman Moscow State Technical University, Moscow, Russian Federation e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_3

25

26

I. Romanova

1 Introduction 1.1 Formulation of the Problem Multi-criteria optimization (ICE) is one of the main tasks that are solved in the framework of studies of technical, economic, social, ecological and other systems and processes. Although the progress in the development and application of ICE technologies is very impressive in economic, social, information processes, technical applications, including robotics and mechatronics, are currently not without problemsolving within the framework of the ICE. The mathematical formulation of the decision-making problem under several criteria is as follows: let a set of m decision selection criteria be a set of functions h1 , h2 , …, hm defined on space X (or maybe some part of it, including, however, a set of admissible solutions x ∈ X). The set of criterion functions h = {h1 , h2 ,…, hm } defines a mapping acting from X to h. A set Y = h(X) = {y|y = h(x), x ∈ X} is called a set of permissible values of criteria. Obtaining Pareto—the best options due to the inconsistency of particular criteria does not mean the achievement of a final solution. A set of these options is only offered to the designer (DM). Pareto—Optimal analysis allows you to adjust the system of preferences. Finally, if Pareto—the best options do not satisfy the system requirements, a fundamental decision is made to change the structure of the system, for example, to introduce a new type of correction. For the representation of the Pareto—the best options, the concept of the Pareto frontier, which satisfies the conditions     P(X ) = x ∈ X : x 1 ∈ X : x 1 ≤ x, x 1 = x = 0

(1)

2 Literature Review 2.1 Classification of Applications Pareto-Optimal Solutions The approach to the ICE called the search of the Pareto-optimal solution, as the review of modern publications shows, is actively used in the research and design of robotic systems. As applied to the tasks of robotics, a brief overview of the methods of ICE is given in [1]. Typical robotics trade-offs between travel speed and energy efficiency are noted. Examples are also given such as a combination of maximum stability of a camera mounted on the head of a robot snake and its speed of movement, maximizing the speed and stability of a four-legged robot and minimizing energy consumption and torque variation in a humanoid robot. It is noted [2] that if the user does not have

Pareto Optimal Solutions and Their Application in Designing …

27

a clear preference among the goals, the best option would be to find the entire set from which the Pareto compromise is formed. MCO funds are currently implemented in application packages, including in the framework of CAD. The analysis of modern literature allowed us to concretize the directions and practical implementations in robotics, connected in one way or another with the use of Pareto-optimal solutions. Collaborative robotics. In [3], systems with general autonomy are considered: shared autonomy system, which is designed to bridge the gap between completely autonomous and fully human-controlled systems. Within the framework of a common task, a class of systems is distinguished that differ in switching control between a human operator and an autonomous controller for the joint achievement of a certain control goal. As part of reaching the Pareto-optimal solution, the algorithm switches control between the operator and the autonomous controller. Path planning. This is one of the most important tasks of mobile, including autonomous robotics. In [3] it is noted that possessing a priori knowledge of the environment, global planning of the path provides a collision-free route through the workspace. Optimizing path planning is usually reduced to minimizing the distance traveled from the beginning to the goal, but many applications for mobile robots require additional path planning goals, which is a problem of multicriteria optimization (MCO). The algorithm approaches the planning of an MCO path using Pareto fronts or finding non-dominated solutions according to the criteria of cost, distance, energy and time. In [4], the Markov–Monte Carlo (MCM) chains method is presented for solving multi-purpose motion planning problems. The problem of finding Pareto optimal trajectories is formulated as a problem of sampling trajectories from the Pareto optimal set. Coordination. In work [5], a class of coordination tasks for several robots is considered, when a team of robots seeks to reach target areas for a minimum time and avoid collisions with obstacles and other robots. A new numerical algorithm is proposed for determining Pareto optimal solutions when no robot can unilaterally shorten the time of its journey without increasing the time of others. The task of coordinating the work of several robots is considered in [6], and it is noted that it remains the main and complex research problem. Using the example of Autonomous Guided Vehicles (AGV), the task of coordinating their movement is considered in such a way as to avoid collisions between AGVs and at the same time maximize performance. At the same time, the scalarization of the cost vector functions is not used, but the Pareto optimality approach is tried on. A similar problem is considered in [7, 8], namely, the coordination of movements of several robots with predetermined trajectories, ensuring traffic safety and simultaneous minimization of travel time. A method for searching by the PRM probabilistic roadmap is proposed for obtaining the Pareto-optimal coordination solution for several robots.

28

I. Romanova

Group management. Coordination of movements is part of the more general task of group management. Models and control algorithms for a group of mobile robots are considered. The fair compromise method was used to search for a Pareto-optimal set of strategies. In [9], a group of robots in a common environment is considered, each robot is forced to move along a road map in its configuration space. For programming optimal collisionless motion, it is necessary to select the appropriate optimality criteria in the framework of obtaining a Pareto solution. Features of autonomous mobile robots. In [10], it was noted that the design of mobile autonomous robots is a complex task due to limited onboard resources, such as computing power and energy. It creates a Pareto front, which contains all the optimal compromises that are achievable. Robots—humanoids. The problems of controlling humanoid robots within the framework of the MKO concept are devoted to such articles as [11]. In [12], an optimal state feedback controller was introduced to control a two-legged robot. Three points are chosen from non-dominated solutions of the obtained Pareto front on the basis of two conflicting objective functions, that is, the normalized summation of angular errors and normalized summation of the controlling force. In [13], an adaptive gait generation method for a two-legged robot is proposed. The gait synthesis during walking is analyzed with minimum energy consumption (MCE) and minimum torque variation (MTC). The unification of dynamic elements on each plane is carried out using the optimal Pareto solution. In [14], the MKO problem is solved within the framework of the task of controlling a four-legged robot with 18 degrees of freedom. Snake-like robots. In the article [15], Pareto fronts or compromise curves are constructed for both ground-based and floating snake-robots with a different number of links according to the criteria of minimizing energy consumption and maximizing the speed of movement. In [15], the problem of improving the stability of the head of a serpentine robot during the gait, while at the same time increasing the speed of movement, is considered. Underwater robots, snakes, are considered and a multi-purpose optimization problem is formulated. Uses a weighted sum method to combine power consumption issues and speed-optimization. Economic aspects of the use of robotics. The successes of applying Pareto-optimum economic strategies are also reflected in the development of the economic aspects of robotics. In [16, 17] it is noted that the success of robotization has created a new economic problem, which can also be solved within the framework of the Paretocompromise solutions.

3 Analytical Studies of the Pareto Optimum 3.1 Assertions and Research Algorithm Consider the tasks of the ICE in relation to the design of UAV control systems.

Pareto Optimal Solutions and Their Application in Designing …

29

Despite the successes in the construction of numerical methods, it is equally important to have a qualitative and analytical assessment of the possibilities and methods for achieving Pareto Optimum. Obtaining analytical solutions allows you to select the most important factors that determine the shape of the Pareto front. An important role in analytical studies is played by the properties of comonotonicity and contra monotonicity of the functions studied.   It is known that if for all points of X ∈ X the derivatives ∂h i (x) ∂ x j  X  of all partial criteria hi for a component x j have the same sign, then the family of functions h is comonotone in X  ∈ X . If everyone x j assumes the fulfillment of comonotonic conditions, then this means that the of functions is strictly comonotone in   family X  . If for some x j not all ∂h i (x) ∂ x j  X  have the same sign, then the family of functions h is contra-monotonous for x j in X  . For mutually contradictory criteria, in the framework of finding the Pareto optimum, the following definition is used: if in X  for all x j h are contra-monotone, then they are completely (strictly) counter-monotone in X  . Taking into account the above concepts, we will concretely define the problem of determining Pareto areas: find an area X p consisting of all points X  where the functions h are strictly counter-monotone, i.e. points optimal (or equilibrium) for the Pareto something. By changing any variable in X p , we will improve the position of a part of the elements, but worsen it—in another part of them. Despite the differences in the formulation of the problem, in fact, an indispensable condition for the application of all approaches is to touch the level lines, namely, it is argued that the distribution is Pareto optimal if and only when the indifference curves (level lines) touch each other. The problem is that indifference curves may not have touchpoints. And under these conditions, a compromise curve can be constructed, i.e. offer the designer a set of effective solutions. The following statement is proved: if the criteria are monotone, i.e. the sign of the derivative of criteria for the parameters being optimized does not change, the Pareto line in the parameter space is the boundaries of the parameter area drawn in accordance with the derivative sign. It also contains several statements that make it possible to cover all possible cases of the topology of the two-criterion problem and, for monotone functions, to avoid direct solving systems of equations and, moreover, numerical simulation by probing the parameter space, which creates great difficulties. All cases of the mutual position of anti-gradients are shown in Fig. 1. The calculation algorithm is as follows: 1. 2. 3. 4.

The region of permissible parameter values is determined. The properties of comonotonicity and contra monotonicity are investigated. Build lines of an equal level. The type of contact will be determined: one-sided (the task is not strictly multicriterial) or two-sided (the task refers to Pareto). 5. For Pareto—problems in the presence of tangency, a system of algebraic equations is formed, whose solution is given by the equation of the Pareto line in the parameter space. The mapping to the criterion space gives the desired Pareto border.

30

I. Romanova

Fig. 1 Mutual location of anti-gradients and Pareto lines

6. In the absence of a touch, the monotony property is analyzed. The analysis of the derivatives allows one to determine two boundaries in the parameter space (along the abscissa axis and the ordinate axis or along the boundaries parallel to them).

3.2 Criteria in the Task of the MCO • Direct quality indicators calculated by the type of transition process in a mathematical model; • Integral, including quadratic, quality indicators calculated by the mathematical model; • Quality indicators obtained experimentally as a result of flight and other types of tests.

Pareto Optimal Solutions and Their Application in Designing …

31

Direct quality indicators. Analytical formulas are used for the overshoot, the time of the transition process, the rise time in combination with the analysis of areas of stability. σ = 100e−ζ π

√

1−ζ 2

Ts =

4 4T = ζ ωn ζ

tr =

 n

1

ω 1 − ζ2



π − arcsin



1 − ζ2



Integral criteria. The second problem solved by the mathematical model is an alternative approach to the synthesis of the UAV control system, which is based on modifying the quadratic quality criterion by dividing it into several components, in particular for the considered example, the integral of the phase coordinates J1 and the integral of the cost of control J2 . For the final choice of a compromise solution, dependencies between these integrals are constructed. In this case, the usual synthesis problem is solved according to the combined quality criterion, the solution of which is not affected by the division of the general criterion into two. Since the coefficient σ plays the role of a relative relationship between management costs and deviations, modified integrals should be considered for selection: 1 J1 (u) = 2

tk



x12 + F2 x22 + · · · + Fn xn2 dt;

t0

1 J2 (u) = 2

tk

 2 u 1 + U2 u 22 + · · · + Um u 2m dt,

t0

that reflect the real picture of transients in the system.

3.3 Pareto Construction Over a Limited Set of Data and in the Case of Fuzzy Preferences The third task is to build Pareto on a limited set of data. An example of such tasks in robotics, and in particular, the study of UAVs, is the processing of test results for the strength characteristics of corpora made from lightweight materials, such as composites. Another example is the processing of flight test results. To solve the problem of an MKO with a limited set of data on the criteria of more than two, obtained as a result of very expensive experiments, often accompanied by the destruction of samples, it is possible to use the following approaches. First approach: Estimate by average. We calculate the average values for each indicator and assume that objects with parameters not less than this average can be left for further comparisons. The graph shows the average and allowable values (values above/below the average).

32

I. Romanova

Second approach: Convolution of criteria. According to experimental data, approximating Pareto front curve is formed. To obtain a single solution, a randomized strategy is used based on the relative importance of the criteria. According to the linear convolution formula for a vector criterion h = (h 1 , h 2 ) with weight coefficients {αi }, a generalized criterion was considered J (x, α) = h(x), α =

m

i=1

αi h i ;

m

αi = 1,

i=1

where α = (α1 , . . . , αm ) is the vector of non-negative weights. Next, the weighted sum is minimized. The third approach: Reduction of a multicriteria task to a two-criterion. The multidirectional of the criteria does not mean an infinite number of directions, so after appropriate scaling, it is possible to divide all the criteria into two groups and carry out two separate convolutions for each group of mutually contradictory directions. For example, for three criteria, two of which are 2 and 3 unidirectional,  it is sufficient 2

2

to perform the convolution of these criteria using the formula h 2 = h 2 + h 3 , after which the usual two-critical problem is solved within the framework of the Pareto optimum. Fourth approach: Fuzzy preferences of the designer within the framework of the MCO task. The formulation of the MKO problem in a fuzzy formulation is used: 1. The preliminary stage is data processing in a normalized representation. 2. Fuzzy logic methods are used to defuse data according to non-clear criteria. 3. Based on the results of applying the Fuzzy Logic methods, a graph is constructed in terms of maximization. 4. Next comes the integration of information with other criteria. We leave only the given points of linear density.

3.4 Visualization of Two or More Criteria Requirements for visualization methods are formulated: 1. It is necessary to reflect the features of the investigated vector quality criterion in the presence of more than two criteria. 2. It is necessary to simultaneously show the effect of several parameters at once. 3. It is necessary to present information in a convenient form for qualitative analysis. 4. It is necessary to present information in a convenient form for quantitative conclusions. 5. The most important information is the data on the mutual influence of concessions and their quantitative values.

Pareto Optimal Solutions and Their Application in Designing …

33

Traditional methods. The following methods are known: 1. Application of Andrews curves, allowing to depict multidimensional data on the plane; 2. Using parallel coordinates; 3. Matrix graphs; 4. Construction of the simplest figures of different colors and sizes. New approaches to the visualization of multi-criteria functions. It is proposed along with the visualization of the objective functions to focus on the visualization of gradients. • Representation of areas of change of anti-gradients using a color scheme. • Images of areas of anti-gradients (areas of the earliest reduction of the function) in accordance with the color scheme; • Images of areas of anti-gradients (directions of the earliest reduction of the function) in accordance with the division into sectors of 45°; • Representation of anti-gradient vectors of the four criteria for the entire range of variable parameters; • Distribution of angles and moduli of vectors of anti-gradient criterion; • Joint assessment of the distribution of angles and anti-gradient modules for several criteria; • Estimation of losses in the choice of compromise solutions; • Dynamics of change of vectors of anti-gradients with variations of parameters; • The cycle of changing the position of the vectors of anti-gradients when the values of parameters 1 change for a fixed value of parameter 2; • Pairs of normalized criteria when combining graphs on which pairs of all four criteria are consistently combined.

4 Results and Discussions 4.1 Object of Study The system of differential equations for stabilized object UAV

34

I. Romanova

The system of equations uses dynamic coefficients a11, …; adjustable gain factors of angular velocity sensors K DUS and linear accelerations K DLU . Quality criteria: transient time, rise time, overshoot, the integral criterion of the weighted module of the error of the output signal, integral energy criterion of management costs. One of the criteria may be fuzzy. The research results are presented in Figs. 2, 3, 4, 5, 6, 7 and 8. From the analysis of Fig. 2, it follows that lines of equal level do not touch each other. Then we consider the properties of monotony. The analysis showed that in the range of permissible parameters, the overshoot function σ increases monotonically, i.e. the lower value of K DUS gives the best result. The analysis performed allows us to construct areas of contra monotonicity for the problem under consideration, namely, the functions are strictly counter-monotonous in the whole range of permissible parameter values. Determine the angular position of the gradients of the dependencies for derivatives. Analysis of the angles in the gradient field shows that the angle ϕ is turned to the lower-left corner, therefore the border is a vertical straight line with the coordinate (K DUS )min and the horizontal straight line (K DLU )min . The mapping of Pareto—lines in the parameter space gives the Pareto—the boundary in the space of criteria.

Fig. 2 Lines of equal level and direction of anti-gradients

Pareto Optimal Solutions and Their Application in Designing …

35

Fig. 3 The interdependence of the criteria for rising time t r and overshoot σ

Fig. 4 The dependence of quadratic criteria J 1 (u) and J2 (u). The arrow shows the direction of increase in the coefficient σ

36

I. Romanova

Fig. 5 Dependence of the control signal on time for a closed-loop system with optimal control: σpo —coefficient calculated from the Pareto optimum; σmax —the maximum allowable value

Fig. 6 The solution to the problem of choice by 5 criteria after the defuzzification of fuzzy criterion

Pareto Optimal Solutions and Their Application in Designing …

37

Fig. 7 Representation of anti-gradient vectors of the four criteria for the entire range of variable parameters K DUS and K DLU

Fig. 8 Pairs of normalized criteria when combining graphs for each K DLU

38

I. Romanova

5 Conclusion Obtaining analytical solutions allows identifying the most important factors that determine the shape of the Pareto front. Receiving a set of compromise solutions that can be presented to the decision maker allows you to make the final choice more reasonable, even in the conditions of insufficient data or ambiguity of preferences and thereby improve the quality of the designed robotic systems. For MCOs with a number of criteria greater than 2, the difficulties with imaging can be overcome by a new approach to the use of visualization tools.

References 1. Ariizumi, R., Tesch, M., Kato, K., Choset, H., Matsuno, F.: Multiobjective optimization based on expensive robotic experiments under heteroscedastic noise. IEEE Trans. Robot. 33(2), 468– 483 (2017) 2. Lavin, A.: A pareto front-based multiobjective path planning algorithm (2015). CoRR, abs/1505.05947 3. Lee, J., Yi, D., Srinivasa, S.S.: Sampling of pareto-optimal trajectories using progressive objective evaluation in multi-objective motion planning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–9 (2018) 4. Zhao, G., Zhu, M.: Pareto optimal multi-robot motion planning (2018) 5. Ghrist, R., Lavalle, S.M.: Nonpositive curvature and pareto optimal coordination of robots. SIAM J. Control Optim. 45(5), 1697–1713 (2006). https://doi.org/10.1137/040609860 6. Cui, R., Gao, B., Guo, J.: Pareto-optimal coordination of multiple robots with safety guarantees. Auton. Robots 32(3), 189–205 (2012). https://doi.org/10.1007/s10514-011-9265-9 7. Jae, B.J.: Pareto optimal multi-robot coordination with acceleration constraints University of Illinois at Urbana-Champaign Robert Ghrist University of Pennsylvania, University of Pennsylvania Scholarly Commons Lab Papers (GRASP) General Robotics, Automation, Sensing and Perception Laboratory. https://repository.upenn.edu/cgi/viewcontent.cgi 8. Lahijanian, M., Svorenova, M., Morye, A.A., Yeomans, B., Rao, D., Posner, I., Newman, P., Kress-Gazit, H., Kwiatkowska, M.: Resource-performance trade-off analysis for mobile robot design (2019). https://www.researchgate.net/publication/323199618_Resource-Performance_ Trade-off_Analysis_for_Mobile_Robots 9. Capi, G., Yokota, M., Mitobe, K.: A new humanoid robot gait generation based on multiobjective optimization. In: Proceedings of the IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Monterey, CA, USA, pp. 450–454 (2005) 10. Zhao, M., Zhang, J., Dong, H., Liu, Y., Li, L., Su, X., Iocchi, L., et al.: Humanoid robot gait generation based on limit cycle stability. In: RoboCup 2008, LNAI 5399, pp. 403–413 (2009) 11. Mahmoodabadi, M.J., Taherkhorsandi, M., Bagheri, A.: Pareto design of state feedback tracking control of a biped robot via multiobjective PSO in comparison with Sigma method and genetic algorithms: modified NSGAII and MATLAB’s Toolbox. Sci. World J. 2014, Article ID 303101, 8 p. (2014). https://doi.org/10.1155/2014/303101 (Hindawi Publishing Corporation) 12. Ha, S., Han, Y., Hahn, H.: Adaptive gait pattern generation of biped robot based on human’s gait pattern analysis. Digital Open Science Index. Int. J. Electr. Comput. Energ. Electron. Commun. Eng. 1(10) (2007) 13. Oliveira, M., Costa, L., Rocha, A., Santos, C., Ferreira, M.: Multiobjective optimization of a quadruped robot locomotion using a genetic algorithm. In: Soft Computing in Industrial Applications, vol. 96, pp. 427–436. Springer, Berlin (2011)

Pareto Optimal Solutions and Their Application in Designing …

39

14. Kelasidi, E., Jesmani, M., Pettersen, K., Gravdahl, J.: Locomotion efficiency optimization of biologically inspired snake robots. Appl. Sci. 8(1), 80 (2018). https://www.mdpi.com/20763417/8/1/80. https://doi.org/10.3390/app8010080 15. Tesch, M., Schneider, J., Choset, H.: Expensive multiobjective optimization for robotics. In: Proceedings of the IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, pp. 973–980 (2013). http://biorobotics.ri.cmu.edu/index.php 16. Costinot, A., Werning, I.: Robots, Trade, and Luddism. June 2018—Preliminary and Incomplete (2018). http://economics.mit.edu/, http://papers.nber.org/papers/w25103.pdf 17. Thuemmel, U.: Optimal Taxation of Robots. Erasmus University Rotterdam, VU Amsterdam and Tinbergen Institute (2018). http://www.cesifo-group.de/DocDL/cesifo1_wp7317.pdf

Collaborative Robotics

This part is devoted to studying robotics through the prism of human–robot collaboration. The authors determine the methodological foundations of the collaborative robotic concept as a breakthrough in the modern industrial technologies. The part also analyzes the performance and ergonomic problems of collaborative robotic systems. 1. A. S. Yuschenko—Control and Ergonomic Problems of Collaborative Robotics. 2. R. R. Galin, R. V. Meshcheryakov—The Performance Criterion of the Collaborative Robots Work. 3. V. Gradetsky, M. Knyazkov, E. Semenov, A. Sukhanov—Human–Robot Cooperation in Technological Wall Climbing Robot System. 4. V. Gradetsky, I. Ermolov, M. Knyazkov, E. Semenov, A. Sukhanov— Features of Human–Exoskeleton Interaction. 5. A. V. Matokhina, S. E. Dragunov, V. Shashkov, D. A. Volodina, Alla. G. Kravets—Analysis of Dynamics in Human–Exoskeleton Collaborative System.

Control and Ergonomic Problems of Collaborative Robotics Arkady S. Yuschenko

Abstract Together with the development of information technology and artificial intelligence robotics intended to enlighten the more and more area of human life and work. It results in a change of the method of communication between human and “intelligence robot” capable of autonomous behavior—from control to collaboration. The new branch of robotics has been formed nowadays as collaborative robotics. The new ways of application of robotics create the new problems of communication of human and “intelligent” robot which are not only technical but also of a psychological kind because human-operator now becomes a part of the human-robot system. The new problems demand the development of new methods in the field of cognitive ergonomics. Some of the problems are under discussion below. Keywords Collaborative robotics · Human-operator · Artificial intelligence · Speech dialogue · Cognitive ergonomics · Multimodal interface · Information perception · Multi-agent system

1 Introduction Modern robotics penetrates into many areas of human activity including not only industrial production but also agriculture, transport, medicine, individual human life. That is the reason for such term appearance as “Collaborative Robotics”. In accordance with International Standard ISO 8373:2012-03 “Robots and robotic devices” (2012) collaborative operations suppose simultaneous participation humans and robots in technological operations in the same working space. Nowadays the term acquires more wide sense. Collaborative Robots (Co-Robots) maybe not only industrial manipulators but also mobile robots capable of autonomous navigation, image detection, behavior planning. Now it is accepted that Collaborative Robotics is a

A. S. Yuschenko (B) Bauman Moscow State Technical University, 5, 2-ya Baumanskaya str., Moscow 105005, Russia e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_4

43

44

A. S. Yuschenko

Fig. 1 Co-robots as a part of industrial conveyer robotic complex

necessary part of the new stage of development of world industry which to-day is determined as Industry 4.0. The new role of robotics in human society demands to solve some new scientific problems both technical and psychological type. From 2016 three International Scientific Conferences ICR (Interactive Collaborative Robotics) took place, which has determined the main classes of Co-Robots and the main problems accompanied their development. Among them is one of the most complicated is the problem of human-robot interaction in the collaborative system. The first in time and the most traditional class of Co-Robots is the class of industrial manipulation robots-assistants of human-workers (Fig. 1). Here the main aim of the robot is the help of a human worker by auxiliary operations fulfilling. The main requirement for the robot is the safety of humans. As for human here we practically have not any special requirements. More complicated class than industrial robots-assistants is presented with the “Da-Winchi” surgical robotic complex (Fig. 2). The surgeon is working at the computer monitor using special joysticks. Now the control movements of the surgeon are quite different from the hand’s movements during the traditional operation. So the previous surgeon’s experience may be ineffective. Another difficulty is the necessity for the surgeon to accept the whole image of the operation area with the uncompleted screen image. Therefore it is necessary for the surgeon-operator to form some necessary experience beforehand. The surgical robot under consideration may be classified as a remote-controlled robot-assistant. Such robots may fulfill automatically many auxiliary operations under control of surgeon with accuracy often unattainable for a human hand.

Control and Ergonomic Problems of Collaborative Robotics

45

Fig. 2 “Da-Winchi” surgical robotic complex

Another class of Co-Robots forms the mobile robots-partners of humans. Such robots are working under control of human-operator in the area containing people and other moving objects. The area of application of such robots is wide. It may be Co-Robots—assistant of firefighters, guard- and patrol-robots. Such Co-Robots may be applied also as assistants for blind and disabled persons. The task of control of such robots is quite different from traditional task of remote mobile robot control. Co-Robot can realize many auxiliary functions autonomously because the system of autonomous control includes computer vision, the means of navigation and the intelligence human-robot interface [1]. One of the tasks to realize by the robot control system is the avoidance of moving obstacles including peoples [2]. Another function to be realized by the Co-Robot control through 3D virtual reality environment system is return autonomously to master-operator in case of control channel loss [3]. The most convenient for a human mode of such robot control is the bilateral speech dialogue. It seems that the task of control of the mobile robot-partner via speech dialogue is the easiest task for the unprepared operator. In difference, with remote mode control of robot movements, now operator solves much more complicated task of Co-Robot movements in a previously undetermined environment changing every minute both with the robot movement and with appearance another moving object. It needs from mobile Co-Robot operator not only some experience but also necessary psychological qualities.

2 Speech Dialogue Mode of Co-Robot Control Co-Robots are oriented on direct human-robot interaction, whereas users are not specially trained for that interaction. The development of service robotics gave some results in natural (or close to natural) speech interfaces. The control task for humanoperator transforms into a verbal dialogue between operator and robot. The latter may inform the operator of the current situation or asks to define more precisely his commands.

46

A. S. Yuschenko

Fig. 3 The diagram of speech interface

The speech interface includes speech recognition and speech understanding (linguistic) modules (Fig. 3). The latter realizes the semantic interpretation of information. The dialogue control module controls the dialogue process including response generation. Usually, speech recognition is based on the comparison of pronounced words with their patterns from a knowledge base. The most effective is the DTW (Dynamic Time Warping) and HMM (Hidden Markov Models). The speech base for Co-Robot operators often may be individual. The dialogue may be presented as a sequence of messages. There are some typical dialogue scenarios. Usually messages are used: (a) to inform the operator about the automatically compiled plan of operation; (b) to give the robot some tasks by operator; (c) to correct the plan of operation or conditions of its fulfillment; (d) to inform operator by robot of environment evaluation; (e) to inform the operator of operations results; (f) to inform human of own robot condition. The scenery itself usually is prescribed as a sequence of frames [4]. Management of Co-Robots interface requires “natural”, from the human point of view, space- and time-relations which greatly facilitate the task of human-robot collaboration. The description of a robot’s outer world includes both the description of objects relevant for the performance of the set task and the spatial relations between the objects of the world, including the robot itself. Extensional and intentional fuzzy relations are used to describe the spatial relations between the objects of the working scene [5]. The first includes the relationship between the position and orientation of objects. For instance, “object a1 is far away, ahead and to the right of object a2”. Intentional relations include such relations as “to contact”; “to be inside”; “to be out”; “to be in the center”, etc. It is possible to obtain other relations encountered in practice from elementary spatial binary relations using formal logic rules of conjunction and disjunction. The current situation embracing M objects, including a controlled robot, is described by a binary frame system (, , ), m, n = 1, 2, …, M. If the fuzzy binary relations between all objects that can be observed by robot in the course of movement are set in advance, then we will get a fuzzy semantic network or a fuzzy chart. Such a chart allows the robot

Control and Ergonomic Problems of Collaborative Robotics

47

to determine its own position in relation to the benchmarks. It is possible, in particular, to navigate the robot along with the observable benchmarks, i.e. the objects whose position is known in advance [6]. The image of the current situation may include other fuzzy features besides the spatial ones, for example, the environment temperature or radiation level. It is important that the objects of the internal environment also may be recognized through fuzzy logic technology. The choice of the fuzzy features for object classification as well as the production rules and membership functions depend on the task under consideration. Analysis of the typical situation for indoor work proved to determine the types of objects such as Wall, Door, Threshold, Block, etc. The typical obstacles are presented in the database beforehand by vectors of fuzzy features obtained from the computer vision system. Among them are the coordinates and dimension of the object, the mutual disposition of the objects, their geometrical characteristics [7]. The fuzzy features allow identifying the object also in the case when the object description is incomplete for obstacles or shadows. In such cases, the cognitive behavior of the robot may be planned to seek the information necessary for the object classification. The “natural” for human perception determination of space relations and terms makes the dialogue close to the dialogue between human-master and human-assistant. The important peculiarity of mobile Co-Robot control system is in the fact that the scale of the image is continuously changing due to the robot movement. That is why the membership functions for the mobile Co-Robot sensing system depend on the distance. In this case the 3D-membership functions had been introduced [7]. Since the outer world is continuously changing both due to the movement of the objects under surveillance and due to the movement of the robot itself, the description of the situation will also vary with time. This circumstance requires taking into general consideration not only spatial but also temporary fuzzy relations, such as “to coincide”, “to take place earlier”, or “to follow”. Such relations provide automatic accompaniment of mobile objects movement, and collision avoidance. Having described the current situation in the language of linguistic variables and fuzzy relations, one can specify the behavior of an autonomous robot in an external, not completely defined environment using fuzzy rules of the production type. These behavioral stereotypes have the form of production rules: “if the situation is Si , then the tactics are Ti ”. Typical situations can be input in the robot’s fuzzy knowledge base in advance, using the experience of the human operator. The tactics themselves can be included in the robot’s knowledge base by way of teaching a neural-fuzzy network, which is serving as the base of the robot controller. Now the operator can only inform the robot of the final goal of its movement in a space with a partially known structure. In this case, there arises the problem of autonomous traffic planning, which requires special consideration [6].

48

A. S. Yuschenko

3 Other Modes of Human—Co-Robot Interaction Often the speech dialogue is not sufficient for “mutual understanding” between humans and Co-Robot. In such cases may be applied systems that use gestures, which is important for people with hearing and speech dysfunctions. Lately, new results were acquired in dialogue with human-like Co-Robots with skin face imitation. Such robots allow reproducing the mimic of a speaking person [8, 9]. With mimics, the robot-companion can express its attitude to perceived messages, own current state, and state of the environment. Whereas, unlike speech, the mimic facial state does not need to be translated. Mimics represent a common image of evaluation in accordance with behavioral rules and goals, provided with the robot’s knowledge base. Here we have two interrelated problems: the first is to recognize the sense and emotional accompaniment of the operator’s speech message by analysis of his facial state by a robot. The more complicated task is to form the artificial mimic state of the robot “face” in connection with the sense of the information to present to the operator in the dialogue process. Formalization of the determined emotional state may be realized with the identification of 6 basic emotional states (BES), that can be recognized by people independent of their nationality and cultural level. P. Ekman proposed 6 BES based on his psychological works [10]. They are happiness, sadness, anger, fear, surprise, and disgust. They should be expanded with 7-th BES— the neutral state. So, the problem is lowered down to imitate the 7 mimic states. Note that BES identification is also a fuzzy logic task and it may be solved with the technology mentioned above. For facial expressions of human face describing a facial expression coding system was developed (FACS) [11]. This technology declared a facial state in terms of typical action units (AU) which describe the movement of distinctive facial points of a separate facial muscle. Now the mimic expression may be depicted at the humanlike robot “face” or at the screen of the user’s controller. Figure 4a, b shows the results of human operator face analysis and Fig. 4c depicted the artificial face of human-like robot Alex with expression corresponding the emotional state of “surprise” [12].

Fig. 4 Results of face analysis and reproduction: a determination of AU, b analysis of the operator emotional state, c reproduction of emotion state with FACS

Control and Ergonomic Problems of Collaborative Robotics

49

The emotional support of speech message may be most important in an extreme situation. For example, negative emotions may correspond to the message of a dangerous situation (for robot or for the operator). The same is for the message of the operator’s error during control or planning. Also for a deficit of time for task solution. The positive emotions usually connected with successful operation fulfillment, optimal planning, etc. As all the factors were formalized in fuzzy terms it seems possible using the fuzzy procedure to determine the modality and the intensity of emotion. The interpretation of the corresponding emotion on the “face” of Co-Robot makes a mutual understanding in human-robot dialogue easier. The possibilities of bilateral human-robot dialogue may be expanded with the recognition of gestures [13]. Some new results were achieved lately. In [14] robot learned to recognize figures by the fingers positions. The base of the recognition system was convolution neuro networks. Kinect v2 was applied for recognition of the space position of the operator’s hand. The accuracy of recognition reached 86%. Another way of gesture dialogue is the application of fingers positions to symbols of the alphabet for deaf [15]. Suppose that it is possible to develop the special gestureoriented language for collaborative robotics including the basic operations such as determination of the direction of movement. The traditional gesture may be supplemented with their “fuzzy” analogs such as “forward”, “a little to the left” “to the right”, “stop” etc. Mimic and gesture accompaniment of speech enlighten the human—Co-Robot interaction and understanding. One of the most radical approaches of human—Co-Robot interactive interface realization is the virtual and augmented reality. Virtual reality (VR) is an interactive interface that provides the user with visually realistic images created by computer graphics (Fig. 5). The technique is one of the modern technologies that considered a tool to enhance the user experience. Graphical simulation proves to generate robot command with no risk for humans and robots. Most of these systems are Robotics Teleportation Systems widely used today in industry, medicine, education, and military applications. The virtual reality application to control and simulate a manipulator Fig. 5 A virtual model of KUKA

50

A. S. Yuschenko

Fig. 6 Extracting Palm’s position from KR 10 in Unity-3D Raw Leapmotion’s

is considered as a helpful tool to improve robot usage by enabling interaction with the robot and workspace features [16]. Virtual Reality technology also was used to improve the human-robot interface to perform teaching tasks and real task execution using a data glove attached to a sensor tracker (Fig. 6). The right image in Fig. 6 is the operator’s palm from one camera of the Leap motion sensor and the left image represents the extracted data from the sensor. It allows reading the gestures of the operator’s hand and extracts position and orientation data. Now we get enough parameters as the target position and orientation of the end effector of the manipulator. The VR headsets and hand tracking hardware can be used for manipulation robots to perform complex tasks. The technology was applied also for a teleported mobile robot control using virtual reality to assist the human operator to achieve the most cost-effective way [17]. Simulation and manipulation of the teleportation system for remote control of the mobile robot using the VR allowed the operator to control and supervise a mobile robot through a 3D virtual reality environment [18]. It is necessary to mention that the approach has not yet sufficiently studied from the psychological point of view. In some investigations, it was discovered that immersion the human-operator may be the cause of high-stress level influenced the psychological state of operator and stability za the control process. The level of such influence depends on the psychological type of the person, of his motivation level and of imaginative capabilities [19].

4 Problems of Multi-agent Co-robotic System Control The experience of the practical application of Co-Robots in various fields of human activity suggests that it would be expedient to manage a group of robots jointly solving some common tasks, for instance performing surveillance, rescue and liquidation operations in case of accidents at various chemical and radiation hazardous facilities. Two related problems arise here. The first is the need to provide for well-coordinated

Control and Ergonomic Problems of Collaborative Robotics

51

functioning of the individual robots that interact with each other and form the actual multi-agent robotic system (MARS). The second is the use of MARS in interaction with the person who manages the system, bearing in mind that the system agents might find themselves in an area where other people are also present. That said, management should remain simple enough for the operator, and the functioning of the robotic agents should be completely safe for the people and the technical equipment located in the MARS envelope area. We should point out that the application range of such systems is far beyond the examples mentioned above, as it is not limited solely to use in extreme environments. These can be health care systems in hospitals, involving the use of groups of robots for various purposes, and robots designed for training in schools and universities. It is apparent that the use of groups of robots puts forward new requirements both towards the structure of the navigation and control system and towards the organization of interaction with the operator, as well as towards the very methods of controlling the individual robotic agents. In this case, we can refer to a new class of robotic systems that is the Collaborative Multi-Agent Robotic Systems (Co-MARS). One of the possible ways to control such a system is to apply a decentralized mode of the system organization. Robots including in system and using the information of the aim of operation stated by the operator and taking into conclusion other robot positions are autonomous planning their movements changing with information with neighbors. The “auction” technology may be applied [20]. At every stage of the procedure, the movements are chosen which provide the highest “price” for the operation aim in the whole. The operator’s task now is to state the task and to control the operation fulfillment with the group of robots. If necessary, the operator has the opportunity to intervene in the procedure of robot planning and fulfillment of the task in changing the situation or the aim of the operation. It seems that the operator’s task now becomes most enlighten and does not suppose any special demands for the operator’s possibilities. But it is not so because of the task change essentially. From the task of tactics of a single robot movement, it changes to a strategy of groups of robots in a partially undetermined environment. It needs special requirements for the psychological possibilities of human-operator. It is necessary now to retain in random access memory information of all the participants—both controllable robots and uncontrollable environment. The results of investigations show that human control becomes ineffective if the number of objects under control is more than 4–5 [19]. The conclusion is that the robots have to form themselves the new association’s number of which is no more. Such associations also may be organized using the mentioned above “auction” technology. The task becomes more complicated in the case of a heterogeneous group of robots. It seems one interesting aspect of the problem the real-time design of “cognitive network” with a group of robots. The network is necessary for robot movement planning and for solving the SLAM (Simultaneous Localization and Mapping) task for a group of robots [20, 21].

52

A. S. Yuschenko

5 Conclusion The roles of humans and robots in a collaborative robotic system are radical changes. From the role of “slave”, the robot attains the role of a human partner in the realization of complex operations in the working environment. That is the reason for changing all the interface structure and the modes of control. From the control with special devices, joysticks, etc., the control change to direct speech dialogue accompanied with hand gestures and face mimic expression. The same control procedure seems more and more simple to the procedure of human interaction. It is necessary to understand that the problems arise not only in the technical sphere but also in ergonomics, psychology, and ethics. For the first time in human history, mankind collides with the problems of artificial creation capable of autonomous behavior, to dialogue and to make autonomous decisions. It is a challenge of technical revolution which demands to change the traditional view on technical systems and their role in our lives and the ways of collaboration with artificial intelligence. Some new branches in science are to appear such as cognitive ergonomics, collaborative robotics, etc. It seems necessary to solve the appeared problems together with engineers, ergonomists, psychologists as the new scientific problems everywhere need a new methodology. Acknowledgements This study was supported by the Russian Fund for Basic Research (RFBR), project No. 18-07-01313.

References 1. Konyshev, D.V., Yuschenko, A.S.: Collaborative mobile robotics as a new stage of service robotics. In: Transaction of the Conference Extreme Robotics, S-Pb. (2014) 2. Mikhailov, B.B., Nazarova, A.V., Yuschenko, A.S.: Autonomous mobile robots—navigation and control. Izvest. SFedU. Eng. Sci. 2, 48–67 (2016) 3. Deveterikov, E.A., Mikhailov, B.B.: Visual odometer. Vestnik, BMSTU Priborostroenie 6, 68–82 (2012) 4. Yuschenko, A.S.: Dialogue mode of robot control on the base of fuzzy logic. In: Transaction of the Conference Extreme Robotics, S-Pb (2015) 5. Kandrashina, E.Yu., Litvinceva L.V., Pospelov D.A.: Knowledge representation of time and space in intelligence systems. Nauka (1989) 6. Yuschenko, A.S.: Intelligence task planning of robot work. Mechatron. Autom. Control 3, 5–18 (2018) 7. Yuschenko, A.S., Volodin, Yu.S., Michailov, B.B.: Autonomous robot behavior in 3-D world. Mechatron. Autom. Control 11, 11–15 (2014) 8. Mazzei, D., Lazzeri, N., Hanson, D., De Rossi, D.: HEFES: an hybrid engine for facial expressions synthesis to like androids and avatars. In: 4-th IEEE International Conference on Biomedical Robotics and Biomechatronics (2012) 9. Ishiguro, H.: Design of human-likeness in HRI: from uncanny valley to minimal design. In: 8-th ACM|IEEE International Conference on Human-Robot Interaction (HRI) (2013) 10. Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993) 11. Ekman, P., Friesen, W., Hager, J.: Facial action coding system. A technique for the measurements of facial movement. Consulting Psychologist Press (2012)

Control and Ergonomic Problems of Collaborative Robotics

53

12. Konyshev, D.V., Vorotnikov, S.A., Yuschenko, A.S., Zhonin, A.A.: Mimic recognition and reproduction in bilateral human-robot speech communication. In: Interactive Collaborative Robotics. First International Conference ICR, Budapest, Hungary (24–26 Aug. 2016) 13. Rogalla, O., Ehrenmann, M., Zöllner, R., Becher, R., Dillmann, R.: Using gesture and speech control for commanding a robot assistant. In: Proceedings of the 11th IEEE International Workshop on Robot and Human Interactive Communication, Berlin (Sept. 2002) 14. Gruber, I., Ryumin, D., Hruz, M., Karpov, A.: Sign language numerical gestures recognition using convolutional neural network. In: Third International Conference, ICR (2018) 15. Kanis, J., Krnoul, Z., Ryumin, D.: Improvements in 3D hand pose estimation using synthetic data. In: Third International Conference, ICR (2018) 16. Majewski, M., Kacalak, W.: Speech-based interfaces with augmented reality and interactive systems for controlling mobile cranes. In: First international Conference, ICR (2016) 17. Kot, T., Novak, P.: Application of virtual reality in teleoperation of the military mobile robotic system TARO. Int. J. Adv. Robotic Syst. (2018) 18. Baklouti, E., Jallouli, M, Amouri, L., Ben Amor, N.: Remote control of mobile robot through 3D virtual reality environment. In: International Conference on Individual and Collective Behaviors in Robotics (ICBR), Sousse (2013) 19. Velichkovskiy, B.B.: Psychological problems of cognitive ergonomics. Mir Psyhologyi 4, 102– 114 (2018) 20. Vorotnikov, S., Ermishin, K., Nazarova, A., Yuschenko, A.: Multi-agent robotic systems in collaborative robotics. In: Third International Conference, ICR (2018) 21. Aitygulov, E., Kiselev, G., Panov, A.: Task and spatial planning by the cognitive agent with human-like knowledge representation. In: Third International Conference, ICR (2018)

Human-Robot Interaction Efficiency and Human-Robot Collaboration Rinat R. Galin

and Roman V. Meshcheryakov

Abstract This paper presents the question about the interaction between human and collaborative robots, their joint actions in the safety-shared workspace. Presented difference between collaborative robots and other robotic solutions. Collaborative robot technical specification reviewed and the efficiency of human-robot interaction and the HRC Index. In safety case, it is important to identify sources of potential harm, to determine which of the persons in the robot’s vicinity may be in peril and to assess the type of injuries the robot may cause to this person. One of the main tasks that we reviewed is to organize effective interaction between a human and a collaborative robot, taking into account their capabilities. Keywords Human-robot interaction · Collaboration · Efficiency criterion · Human-robot collaboration

1 Introduction Human-robot interaction is one of the most discussed topics in modern robotic solutions. The human-robot interaction (HRI) could be presented from different points of view. The first point, it shows interaction in a shared workspace without direct human-robot contact and synchronization of their tasks. The next point is a joint process of performing tasks, taking into account the adjustment of the robot’s movements in accordance with the human movement in real-time. This is one of the central problems of HRI, which is expressed by the correct distribution of functions between human and robot. It is necessary to determine which functions we can to transfer to a robot possessing elements of artificial intelligence, and which to a person possessing cognitive R. R. Galin (B) · R. V. Meshcheryakov V.A. Trapeznikov Institute of Control Sciences of Russian Academy of Sciences, Moscow 117997, Russian Federation e-mail: [email protected] R. V. Meshcheryakov e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_5

55

56

R. R. Galin and R. V. Meshcheryakov

properties in order to obtain maximum efficiency from their interaction at the stage of analysis and design of the erratic robotic intelligent system. This issue has been considered with the problem of distribution of functions in the system “man-machine”, the solution is engaged in ergonomics [1]. An interesting fact is the observance of laws that in 1942 proposed by science fiction writer Asimov [2]. The three “Laws of Robotics” applicable to human-robot interaction are followed: 1. First Law—A robot may not injure a human being or, through inaction, allow a human being to come to harm. 2. Second Law—A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. 3. Third Law—A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws. Based on these laws, we can conclude that these laws can guarantee the safety of human-robot interaction. In practice, the issues of safety interaction are much broader and more complex. Robotics is constantly evolving. Been a long time since Isaac Asimov first wrote his laws for robots, their ever-expanding role in our lives requires a radical new set of rules, legal and artificial intelligent (AI). Robots should complement rather than a substitute for professionals. Rather than having a robot doctor, you should hope that you have a doctor who really understands how AI works and gets good advice from AI, but ultimately it’s a doctor’s decision to decide what to do and what not to do [3]. The collaborative process between a human and a robot is called Human-robot collaboration (HRC). And adapted for this class of robots called—collaborative robots (cobots) [4, 5]. The trend of the development of robotics today is the introduction of robotic solutions in the field of human activities. Collaborative robots provide a safe interaction with a human, taking into account the presence of a person in a shared workspace with a robot. This robot is adaptive to working conditions, takes into account human behavior and has a certain degree of autonomy. The purpose of creating collaborative robots is to work well and effectively not only with each other but also with humans. In addition, we need to stop the arms race. Unfortunately, many real research practices take start from military requirements and after successful tests, technologies in the field of robotics can be developed for civilian purposes.

2 The Relevance of the Research to Increase the Efficiency of HRI The field of research on improving the efficiency of HRI remains opened. There is no exactly a certain approach to increase efficiency to develop intelligent multi-agent robotic systems at the human-robot level. This is due to the complexity of ensuring the coordinated interaction of technical and biological parts of the erratic robotic intelligent system [6].

Human-Robot Interaction Efficiency and Human-Robot Collaboration

57

Thus, today remain relevant tasks such as follows: development of adaptive control systems for the robot, development of intelligent human-robot interface, development of sensor systems and information processing algorithms, development of elements base of robotic and mechatronic systems, development of effective ways of robot interaction with the external environment, bionic and biomedical technologies in robotic solutions, control of multi-agent robotic systems with secured reliable and secure communication. If we consider the current state of researcher’s works of the above problems for the systems “human-robot”, it becomes clear to understand which solutions to the problems actively conducted research. The analysis of directions of research presented in Table 1. The purpose of the research is to determine the efficiency criterion of the interaction between humans and collaborative robots in a shared workspace.

3 Safety Issues in Human-Robot Interaction Safety issues in human-robot interaction regulated in standards [4, 7]. The purpose of the standards is to increase the interoperability of robots and their components. Also, it helps to reduce the cost of their development and maintenance through standardization and harmonization of processes, interfaces, and parameters. Classification of general safety standards for robotics solutions in HRI classifies into three categories. It explained the basic legal framework of safety during the collaboration of robot-human system. There are three types of sub-classes: • Type A. General principles for design and risk assessment. This type presents general requirements of the functional safety of electrical, electronic, programmable electronic safety-related systems. • Type B. Safety-related parts of control systems and specific safety aspects. Principles for a design for emergency stop function. • Type C. Specifies safety requirements for collaborative industrial robot systems and the work environment and supplements the requirements and guidance on collaborative industrial robot operation given in ISO 10218-1 and ISO 10218-2. This classification is not a complete description of all safety standards. There are more than 700 different standards and regulations [8]. The main operation approaches in HRI during collaboration that we can consider through the classification of general safety standards are follows in four collaborative directions [9, 10]: • “Safety-rated monitored stop”—robot stops when human-operator enters the collaborative workspace and continues when workspace free (allows for direct operator interaction with the robot system under specific circumstances); • “Hand guiding”—robot movements are controlled by human-operator (operator uses a hand-operated device to transmit motion commands);

58

R. R. Galin and R. V. Meshcheryakov

Table 1 Analysis of directions of research in intelligent technologies of robotic and mechatronic systems Actual challenge

Description

Development of adaptive control systems for the robot

In contrast to the rigidly structured working space of the laboratory, production room (site, workshop) or test site, the real environment in which the robots will operate is characterized by a high level of dynamic uncertainty

Development of intelligent human-robot interface

Robots interact with humans, and in a number of applications, such as social and rehabilitation robotics, this interaction is extremely important. The purposes of using the human-robot interface can be different: programming the desired behavior of the robot, collaboration in the joint execution of complex tasks, etc.

Sensor systems and information processing algorithms

For the control system to work in a dynamic environment, reliable feedback is extremely important, giving complete and meaningful information about the state of the environment and the robotic system itself. This requires both physical quantity meters themselves, working on different principles, and algorithms for processing sensory information, which can be used on-Board computing systems with limited computational characteristics

Element base of robotic and mechatronic systems

Many tasks solved by robots and a variety of conditions in which robotic systems can operate require the use of special structural materials and components

Development of effective ways of robot interaction with the external environment

Robotic systems should be able to move in space and carry out manipulations and effectively affect external objects

Bionic and bio-medical technologies in robotics

Recently, the principles of biological systems for technical devices of a wide profile (bionics) are widely used

Management of multi-agent robotic systems with secure communication

Modern robotic systems are often distributed decentralized systems. To ensure their functionality, it is necessary to provide communication between the elements of the system (including wireless)

• “Speed and separation monitoring”—operator and robot system may move concurrently in the collaborative workspace; • “Power and force limiting”—contact forces between human-operator and moving robot are technically limited to a safe level (physical contact between the robot system and a human-operator can occur either intentionally or unintentionally).

Human-Robot Interaction Efficiency and Human-Robot Collaboration

59

Fig. 1 Human-robot interaction in four collaborative operation modes [11]

These approaches relate to the most difficult to understand aspects of human-robot cooperation. In each case, the robot and the human share a common workspace. There is information about approaches in [4, 7] that presents features of all four collaborative operation modes in a shared workspace. But the most interesting for us is the data on the robot’s work, which requires certain restrictions on the power of effort and the moment of force. Visually collaborative operation modes for human-robot collaboration presented in Fig. 1. These approaches are considered as elements of one scenario, not individual aspects of human-robot collaboration modes. According to Robert Nelson Shea, the idea to use collaborative robots in the industry had been met with skepticism, as the available solutions were to avoid direct contact between man and robot. Over time, the trend has changed: the robot and the human can achieve greater efficiency together, but at the same time observing safety standards to prevent the risk of hazard injuries [12]. The following information will help you to understand the differences between cobots and other robotic solutions [7, 9, 13]: • Ability to interact safely with a human. The cobots designed specifically to work with people. When using them, no protective barriers are required. Cobots help to solve complex tasks that could not be automated at full. • Risk reduction in the implementation of hazardous tasks. Cobots perform operations that pose a risk to humans. These risks include safety transportation of sharp and sharpened or hot items, tightening of bolts, etc. • Flexibility and learning. Programming collaborative robots are easier than industrial robots (for example). Some cobots are capable of self-learning.

60

R. R. Galin and R. V. Meshcheryakov

• Possibility of wide use and quick adjustment. Collaborative robots are not only easy to reprogram, but also relatively easy to move and use at other points in the production chain. Many models of robots can be installed on any surface— horizontal, vertical and even on the ceiling. Today, the use of collaborative robots has become perceived as a means of production and to estimate the feasibility of their application in certain areas of the process. The results of the use of cobots have become so effective that enterprises believe that collaborative robotic solutions are the future.

4 The HRI Efficiency of the Collaborative Robot’s Work The concept of the term efficiency is universal and is used in all spheres of human activities. Efficiency is associated, first, with the effectiveness of the work or action, and secondly, with the economy, that is the minimum amount of cost to perform this work or action. In the long term, the effectiveness of robots will only increase. There are three main factors that contribute to the onset of such a turning point moment [14]. The first— improving the economic efficiency of robots in relation to human labor. Modern collaboration robotic solutions meet the criterion of intelligence, i.e. have a number of functions that make their implementation simple and effective. The second— Introduction of such technological advantages in robotics solutions which will allow to effectively implement robots in key industries and economy. The third—the rapid development of the robotics market. Creation of mobile, easy-to-use and affordable robotic solutions. No matter how far advanced the complexity of the robotic system, no matter how revolutionary its design, the only purpose for which the robot is created is to perform the work more economically than before. Of course, the release of people from unattractive and dangerous work can be considered as the most important justification for the use of cobots, but if it will meet the conditions either not to increase or at least not to reduce the profit of production. There are many factors of using collaborative robotic solutions, some of the following: • The cost of the robotic components. The cost of the components of the assembled system can be very significant. The adaptation of the robot’s components depends on the task. Engineering costs are usually 30–50% of the total cost of the installed system. • The cost of installation. Even if installation costs are sometimes directly added to the cost of the robot itself, in many cases it is necessary to change the factory layout in one way or another when changing products. • The cost of training. When redetermining the cost of a robotic system, it is often forgotten that many workers need to be trained in various aspects of working with a robot. Installation and training can usually be 10–15% of the total cost of the system.

Human-Robot Interaction Efficiency and Human-Robot Collaboration

61

• The cost of maintenance. Depending on the design of the robot and the conditions of its work will require varying degrees of preventive maintenance, regular overhaul, and minor repairs. Maintenance costs can reach 10% per year of the original purchase price. • The costs of the operation modes. The main operating costs usually consist of the cost of energy required to operate the robot and the cost of the control device and additional equipment. • The cost of programming work. Before the robot can perform the specified work, it needs to be programmed, and this is an additional cost. Robots should be maintained and depreciated over their entire life. The problem of evaluating the effectiveness of robotic solutions arose with the advent of an automated control system. In the scope framework of the research, which is presented in [14], an additional classification feature was proposed. Robotic solutions were evaluated by the degree of costs for their implementation. With the aim of identifying the factors of effectiveness of the implementation of robotic solutions have been selected enterprises that have introduced robotics on their production FANUC robots. Conducted an assessment by the method of expert assessments. The result of the expert survey revealed that taking into account the experience and understanding of the robotics market, the following factors were formed in descending order of importance: 1. 2. 3. 4. 5.

Productivity increase; Quality improvement; Reduction of labor costs; The elimination of dangerous operations; Increase the flexibility of production.

The results of the expert survey are not indicative and the only true. In the modern tendency of the implementation of collaborative robotics solutions in a shared workplace and in everyday human activities is focused on the person and the effectiveness of their direct contact.

5 Human-Robot Collaboration Index Developing collaborative robots that can productively operate out of isolation and work safely with the human environment is critically important for advancing the field of robotics. The development of such systems, those that handle the dynamics of human environments and the complexities of human interaction, is a strong focus within HRI. Especially in domains where modern robots are ineffective, we wish to leverage human-robot teaming to improve the efficiency, ability, and safety of human workers. We need to create collaborative robots that can provide assistance when useful, remove dull or undesirable responsibilities when possible, and provide instruction or guidance when necessary for our community.

62

R. R. Galin and R. V. Meshcheryakov OperaƟng space

The indicators of the collaboraƟve robot

The indicators of the human CollaboraƟve workspace

The implementaƟon Ɵme of aa task task

EmoƟonal and cogniƟve components components of of behavior behavior

Energy Energy expended expended by by the the robot robot

Visual percepƟon of a robot by by aa human human

The The path-trail path-trail of of the the robot robot

CollaboraƟve robot

Speech Speech and and physical physical contact cont act

Fig. 2 Human-robot collaboration indicators

The authors propose to consider the index of human-collaborative robot interaction. This index will help to consider the effectiveness of human-robot interaction in combination with economic efficiency and factors affecting the effectiveness of the implementation of robotic collaborative solutions. Figure 2 shows the HRC Index in the form of the robot and human indicators on the example of a robotic arm. To consider the index of interaction between robot and human, six indicators were selected, which are divided into two groups: • The indicators of a collaborative robot: – The implementation time of a task. This indicator is responsible for the execution time of the task since the beginning of the collaboration. – Energy expended by the robot. The energy which is spent by the robot on the performance of a task together with the person is considered in a shared workspace. – The path-trail of the robot. Passed the path-trail of the robot during the execution of the task. • The indicators of the human: – Emotional and cognitive components of behavior. Clarification of models of human social behavior. – Visual perception of a robot by a human. Considering the effect of human-robot visual contact. – Speech and physical contact. Speech and physical human impact on the robot to perform the task. The HRC Index uses standard statistical techniques to aggregate the data into a single indicator.

Human-Robot Interaction Efficiency and Human-Robot Collaboration

63

6 Conclusion Today, robots and artificial intelligence systems are powerful tools that can not only facilitate human labor but also completely free him from some types of physical or mental work. The organization of effective interaction between humans and robots will achieve high economic performance and improve the quality of the production process and products and or services [14]. In further work, it is planned to pay attention to the models and methods of information processing in the interaction of humans and robots on the example of collaborative robotics. Acknowledgements The reported study was partially funded by RFBR according to the research project No. 19-08-00331.

References 1. Timofeev, A.V.: Robots and artificial intelligence [in Russian—Roboti i iskusstvenniy intellekt]. Publishing—Moscow “Science”, 192 p. (1978) (in Russian) 2. Asimov, I.: Runaround. In: Astounding Science Fiction (1942) 3. Pasquale, F.: New laws of robotics needed to tackle AI. Expert opinion, AFP 2019, Homepage, https://www.malaymail.com/news/tech-gadgets, last accessed 2019/05/17 4. ISO. ISO/TS 15066:2016-02 (e) Robots and robotic devices—Collaborative robots (2016) 5. Ermishin, K.V., Yuschenko, A.S.: Collaborative mobile robots—a new stage of development of service robotics. J. Robot. Tech. Cybern. (Sankt-Petersburg: RTC) 3(12), 3–9 (2016) (in Russian) 6. Yuschenko, A.S.: Human-robot: compatibility and cooperation. J. Robot. Tech. Cybern. 1(2), 4–9 (2014). (in Russian) 7. ISO 10218-2:2011 Robots and robotic devices—Safety requirements for industrial robots— Part 1, 2: Robot systems and integration Geneva (2011) 8. Villani, V., Pini, F., Leali, F., Secchi, C.: Survey on human–robot collaboration in industrial settings: safety, intuitive interfaces and applications. Mechatronics 55, 248–266 (2018). https:// doi.org/10.1016/j.mechatronics.2018.02.009 9. Khalid, A., Kirisci, P., Ghrairi, Z., Thoben, K-D., Pannek, J.: Towards implementing safety and security concepts for human-robot collaboration in the context of Industry 4.0. In: 39th International MATADOR Conference on Advanced Manufacturing (Manchester, UK), pp 0–7 (2017) 10. Lazarte, M.: Robots and humans can work together with new ISO guidance. Homepage, https:// www.iso.org/news/2016/03/Ref2057.html, last accessed 2018/04/13 11. Yanco, H., Drury, J.: Classifying human-robot interaction: an updated taxonomy. In: Proceedings of the IEEE International Conference Systems, Man and Cybernetics (SMC), vol. 3., pp. 2841–2846. IEEE (2004) 12. Anandan, T.: Robots and humans: safety collaboration. Control Eng. 6(72), 46–49 (2017) 13. Robla-Gomez, S., et al.: Working together: a review on safe human-robot collaboration in industrial environments. IEEE Access 5, 26754–26773 (2017) 14. Chueshev, A., Melekhova, O., Meshcheryakov, R.: Cloud robotic platform on basis of fog computing approach. In: International Conference on Interactive Collaborative Robotics, 34–43 15. Ershova, L., et al.: The factors of the effectiveness of introducing robotic complex at Russian enterprises News of the Ural State Mining University 2(50), 130–134 (2018)

Human-Robot Cooperation in Technological Wall Climbing Robot System V. Gradetsky, M. Knyazkov, E. Semenov and A. Sukhanov

Abstract Many R&D is related to man-machine interaction that characterized as general background results as original results depending on the technical structure, control, and application of every original machine. The goals of this paper are to take attention to the particularities of man-operator participation in multilevel control system of underwater wall-climbing robots (UWCR) occupied with vacuum control with vacuum contact devices (VCD), to analyze man-machine interface and to create control algorithms for UWCR with man-operator participation for the cases when some technologies necessary to produce in extreme conditions. Prescribed tasks for UWCR system are the following: satisfy control motion long underwater surfaces, produce inspection of the surfaces of the pool in nuclear power station and prepare some technological processes like cutting or welding along such surfaces. For the execution of this work, it is necessary to fix a robot on the surface and then perform the robot’s motion along the surface under control of the man-machine operator. To provide effective operator participation in the design control system it was satisfy predicted performance of UWCR robot motion. It was provided analyze of rational automation of UWCR motion and technologies performance. It was clear that real reveille processes may rational fulfilled only with operator participation in critical points. The structure of a multilevel control system for UWCR with VCD was designed and simulated which includes a man-machine interface. The control system takes into attention the peculiarities of operator participation. It was created several algorithms. Regular algorithms satisfy automation of main system motion of UWCR V. Gradetsky (B) · M. Knyazkov · E. Semenov · A. Sukhanov A. Ishlinsky Institute for Problems in Mechanics of the Russian Academy of Sciences, Pr. Vernadskogo 101-1, Moscow 119526, Russian Federation e-mail: [email protected] M. Knyazkov e-mail: [email protected] E. Semenov e-mail: [email protected] A. Sukhanov e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_6

65

66

V. Gradetsky et al.

to aim point. The algorithm provides intervention of an operator if obstacle avoidance operation and simple decision making not perform. Keywords Wall climbing robot · Man-robot cooperation · Collaborative robotics · Extreme conditions · Control system

1 Introduction Collaborative robotics becomes the advanced tendency of researches so long as combine rational facilities of man and robot to solve complex problems of automation including actions in extreme conditions [1–7]. Such types of robots are safety and flexible in cooperation with people, easy for programming and control [8–10]. Man-robot cooperation can be considered as a branch of collaborative robotics in the part of the control of technological wall climbing underwater robot system. They are a lot of collaborative robots of various robot’s type manipulators, industrial robots, mobile robots, with voice dialogue, various sensor, application, and others [11–16] but as applied to wall climbing machine such consideration was not be analyzed. Early published some papers were connected with the design and application of wall climbing robots for various applications in nuclear power stations, ships repair, building construction, etc. [17–19]. In this paper, man-robot collaboration is delivered for a technological wall climbing robot system, including the underwater application of such kind of robot. In the paper the following prescribed tasks are solving: – – – –

wall climbing robot control motion along underwater surfaces; technological tasks solving with the help of man-operator; structure of the control system for man-operator collaboration with a robot; algorithms of the supervision control robot motion.

2 Wall Climbing Robot Control Motion Along Underwater Surfaces Presented design of wall climbing robot with vacuum gripper or vacuum contact devices (VCD) (Fig. 1) permits to realize it control motion along underwater surfaces. The UWCR consists of two platforms and includes the following devices (Fig. 1): body (1), rotary block (2), technological equipment with video camera (3), VCD (4), control system with output (5) and input (6), control line (7), feedback line (8), input/output (9) of UWCR. The control system of UWCR is suited higher than the level of liquid (10) and operator of the machine (11) may be on some safety distance form UWCR application for technological problem solving.

Human-Robot Cooperation in Technological Wall …

67 6

5

11

7 10

3 4

8 9 2 1

Fig. 1 Design and motion of wall climbing robot along underwater surfaces

The mood of robot motion lies in the fact that one platform connected with the group of VCD has the possibility to move relatively second platform connected with the group of WCD when another platform is contacted with the underwater surface.

3 Technological Tasks Solving with the Help of Man-Operator The solving UWCR problems may be performed partially on the automated mood of operation and with the help of a man-operator. In practice really it is collaborative interaction. It is possible to indicate tasks that solve technological equipment in automated mood and with the help of man-operator. Robot motion to the place of the technology is performing in automation mood. But sensor displacement is realized by means of man-operator in nearby zone of possible defects. Collaborative fulfillment is produced to solve assigned technological tasks such type for example as cutting, welding, inspection in the water pool or in decay tank. Presented Figs. 2, 3, 4 and 5 are illustrated the application of the wall climbing robot for execution some technological operations where man-operator with unit for control motion of robot along the wall (Fig. 2), robot performs non-distractive testing (Fig. 3), two arms robot produces cutting operations (Fig. 4), robot with sensor fulfills inspection of the surface (Fig. 5).

4 Structure of the Control System for the Collaboration of Man-Operator with Robot UWCR control system provides for collaboration of man-operator with the robot. The block-diagram of hierarchy for control (Fig. 6) determines the position of man-operator which connected with the control desk of the control system and has the possibility to receive information about the working situation from sensors. Depends

68

V. Gradetsky et al.

Fig. 2 Man-operator with control unit and wall climbing robot connected to the wall

Fig. 3 The wall climbing robot performs a non-destructive testing operation

on the situation, the operator can be included in the control process at any time. The structure of the control system (Fig. 7) includes five levels of control. The upper level is supervision one control with man-operator participation, then level intended to produce simple decision making with elements of artificial intelligence, after that levels for the program and drive control. Supervision control may be a remote type when the robot system is working in extreme conditions. Feedback interface between robot and man connects sensors, transducers and remote devices with supervision control of the operator. The interface between man

Human-Robot Cooperation in Technological Wall …

Fig. 4 Two arms robot produced cutting operation

Fig. 5 The wall climbing robot with a sensor fulfilled inspection of the surface

Fig. 6 Block-diagram of hierarchy for control and place of man-operator

69

70

V. Gradetsky et al.

Fig. 7 Structure and levels of the control system for UWCR

and robot organizes direct participation of man in the control system when it is necessary in critical situations or points of motion. Program control is based on information, analyzer and drive algorithms and produces program motion of robot by means of transducers and mechatronic drives of robot translation and rotation motion and VCD drives to connect robot body to the wall. Systems of UWCR (Fig. 8) produces all types of automation mood and permits to include man-operator to produce supervision control in every step of motion and technology realization. Man-operator intervention can happen when it is necessary in critical points of technological operation fulfillment on the time of the motion to the technology place, during the time of technological operation completion, or during an inspection of the quality of feasible work. Special attention pays on the quality of the production. In such a manner man-operator becomes a part of the feedback loop of the control system.

Human-Robot Cooperation in Technological Wall …

71

Fig. 8 System of UWCR

5 Algorithms of the Supervision Control Robot Motion To satisfy collaborative man-robot control UWCR motion the conforming algorithms were developed. Appropriate algorithms intended for satisfying as automatic control for typical kinds of robot motion, as so as supervision control with operator participation in critical points of every necessary period of time. Program software was realized on the base developed algorithms that satisfy translation, rotation transport motions and VCD motion for robot connecting to the walls. As examples, some block-diagrams of algorithms are presented on illustrations (Figs. 9, 10, 11). The structure of algorithm for supervision control of robot motion (Fig. 9) includes a level of man-operator and level of robot. In Fig. 9, where VCDC—vacuum contact device central, VCDO—vacuum contact device outside, EJ—ejection, PD—piston drive, PR—pressure reducer, JK—joystick. Structures of algorithms for horizontal and vertical forward motion are delivered on Figs. 10 and 11.

6 Conclusion The research was needed to determine the rational participation of the operator in UWCR with VCD that intended to motion over vertical underwater surfaces and to produce technological operation in extreme conditions. The design and simulation showed the effectiveness of suggested approaches.

72

V. Gradetsky et al.

Fig. 9 Structure of algorithm for supervision control robot motion

The paper shows how man-robot cooperated in the motion control system of UWCR with VCD. It was presented technological tasks solving with the help of man-operator, structure of the control system for man-operator collaboration with a robot, motion algorithms, including motion with supervision control.

Human-Robot Cooperation in Technological Wall …

Fig. 10 Structure of algorithm for program horizontal forward motion

73

74

Fig. 11 Structure of algorithm for program vertical forward motion

V. Gradetsky et al.

Human-Robot Cooperation in Technological Wall …

75

Acknowledgements This work was supported by the Russian Foundation for Basic Research, project no. 18-08-00357.

References 1. Kildal, J., Maurtua, I.: Revisiting the end user’s perspective in collaborative human-robot interaction. In: Proceedings of International CLAWAR2016 Conference. Advances in Cooperative Robotics, London, UK, pp. 196–204, 12–14 Sept. 2016 2. Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors J. Hum. Factors Ergon. Soc. 37, 32–64 (1995) 3. Muscato, G., Bonaccorso, F., Cantelli, L., Longo, D., Melita, C.D.: Volcanic environments: robots for exploration and measurement. IEEE Robot. Autom. Mag. 19(1), 40–49 (2012). https://doi.org/10.1109/mra.2011.2181683 4. Murphy, R.R.: Disaster Robotics. The MIT Press (2014) 5. Baudoin, Y.: TIRAMISU: FP7-project for an integrated toolbox in humanitarian demining, focus on UGV, UAV, technical survey and close-in detection. In: Proceedngs of the International Conference on Climbing and Walking Robots (CLAWAR2013), Sydney, Australia, pp. 14–17 (2013) 6. Cantelli, L., Magiameli, M., Melita, C.D., Muscato, G.: UAV/UGV cooperation for surveying operations in humanitarian demining. In: 11th IEEE International Symposium on Safety Security and Rescue Robotics, Oct. 21–26, 2013 Linkoping, Sweden (2013) 7. Habib, M.K., Baudoin, Y., Nagata, F.: Robotics for rescue and risky intervention. In: 37th Annual Conference on IEEE Industrial Electronics Society (IECON), pp. 3305–3310 (2011 Nov) 8. Fryman, J., Matthias, B.: Safety of Industrial Robots: From Conventional to Collaborative Applications, ROBOTIK. 12.1–5 (2012) 9. Brewster, S., Murray-Smith, R.: Haptic Human-Computer Interaction. Lecture Notes in Computer Science 2058, Springer, Berlin (2001) 10. Vasic, M., Billard, A.: Safety issues in human-robot interactions. In: ICRA’13, 197–204, IEEE (2013) 11. Malm, T., Viitaniemi, J., Latokartano, J., Lind, S., Venho-Ahonen, O., Schabel, J.: Safety of interactive robotics—learning from accidents. Int. J. Soc. Robot. 2, 221–227 (2010) 12. Carlson, T., Demiris, Y.: Using visual attention to evaluate collaborative control architectures for human robot interaction. In: SSAISB ’09, 38–43 (2009) 13. Behrens, R., Saenz, J., Vogel, C., Elkmann, N.: Upcoming technologies and fundamentals for safeguarding all form of human-robot collaboration. In: 8th International Conference Safety of Industrial Automated Systems (SIAS 2015), Königswinter, Germany 18–20 Nov. 2015. ISBN 987-3-86423-163-6 14. Pritzsche, M., Saenz, J., Penzlin, F.: A large scale tactile sensor for safe mobile robot manipulation. In: 11th International Conference on Human-Robot Interaction (HRI), New Zealand, 07–10 Mar. 2016 15. Mobile Manipulator VALERI. 47th International Symposium on Robotics—ISR 2016, Munich, Germany, June 21–22 2016 16. Kharlamov, A., Ermishin, K.: Voice dialogue with a collaborative robot driven by multimodal semantics. Lecture Notes in Computer Science. Interactive Collaborative Robotics. Proceedings of the First International Conference. ICR 2016, LNAI 9812, pp. 225–233 (2016)

76

V. Gradetsky et al.

17. Sattar, T., Hilton, P., Howlader, MDO. F.: Development of lazer cutting head with wall climbing robot for nuclear decommissioning. In: Advances in Cooperative Robotics: Proceedings of CLAWAR 2016 International Conference. London, UK, 12–14 Sept. 2016, pp. 725–732 (2016) 18. Gradetsky, V., Knyazkov, M., Sukhanov, A., Semenov, A.: Dynamics of vacuum contact devices of mobile climbing robots. In: Advances in Cooperative Robotics: Proceedings of CLAWAR 2016 International Conference. London, UK, 12–14 Sept. 2016, pp. 278–286 (2016) 19. Gradetsky, V., Knyazkov, M., Sukhanov, A., Chashchukhin, E., Kryukova, A.: Possibilities of using wall climbing robots for underwater applications. In: Proceedings of CLAWAR 2017. International Conference Human-Centric Robotics, Porto, Portugal, 11–13 Sept. 2017, pp. 239– 246 (2017)

Features of Human-Exoskeleton Interaction V. Gradetsky, I. Ermolov, M. Knyazkov, E. Semenov and A. Sukhanov

Abstract The operator is an important component of the Operator-Exoskeleton system. He is a source of signals and he controls the result of the motion of the actuators. The main source that drives the links of the human skeleton is the force of muscle contraction. A series of control algorithm tests were carried out to determine the efficiency of the exoskeleton when performing the task of positioning the exoskeleton link in different control modes. In the experimental studies, the time spent on the operation to achieve the target position was measured. In the Laboratory of Robotics and Mechatronics of the Ishlinsky Institute for Problems in Mechanics RAS a physical model of the arm, exoskeleton was made for experimental investigation. As a result of the experiments, the biopotential data of the operator muscle groups in the performance of various actions were obtained. The exoskeleton drive control will depend on the parameters of the controller and biopotential sensor. The choice of the parameters of these elements will significantly affect the dynamics of the exoskeleton. In this work, we have proposed the interaction model of the exoskeleton system with the operator modeled via biceps brachii action. The study simulated the effect of system parameters variations on the dynamics of the Operator-Exoskeleton system. Keywords Exoskeleton · Human-machine interface · Electromyography · Mechatronics · Control V. Gradetsky (B) · I. Ermolov · M. Knyazkov · E. Semenov · A. Sukhanov Ishlinsky Institute for Problems in Mechanics of the Russian Academy of Sciences, Moscow, Russian Federation e-mail: [email protected] I. Ermolov e-mail: [email protected] M. Knyazkov e-mail: [email protected] E. Semenov e-mail: [email protected] A. Sukhanov e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_7

77

78

V. Gradetsky et al.

1 Introduction The application field of exoskeleton devices is determined by the scientific and technical tasks assigned to such systems. The application of exoskeletons is relevant in emergency situations when performing tasks related to the movement of heavy loads and the implementation of power support in the debris removing, repair of agricultural machinery. The task of a rehabilitation exoskeleton device is the recovery of the motor function of the extremities of an injured person. Exoskeletons for virtual reality are developed in order to sense virtual objects and interact with them (Fig. 1). There are two main goals for researchers developing any exoskeleton device. The exoskeleton may perform compensation for the lost functions of a disabled person, or to expand these functions. On the other hand, the exoskeleton system can be performed as a technical device that increases human productivity or reduces the influence of external forces on any technological operation for increasing the efficiency of work. Researchers and developers of exoskeleton devices are faced with many problems that need to be solved to obtain a technical system that can efficiently perform their tasks. For example, special attention should be paid to energy system efficiency. An important part is devoted to the development of an operator monitoring system that will detect tremor and fatigue and configure parameters of the control system for the current psychophysiological state of the operator. When developing a control system for exoskeleton devices, it is very important to take into account the intuitiveness of the control. This is important in terms of reducing the time spent on operator training, as well as exoskeleton use in emergency situations. It is also necessary to take into account the safety of the control elements of the system and the usability of performing any movements.

2 Implementation of Operator-Exoskeleton Interaction In the Laboratory of Robotics and Mechatronics of the Ishlinsky Institute for Problems in Mechanics RAS, the development of exoskeleton systems has been practiced since the mid-90-is of the last century. The scientific basis for the design of such systems was set, various models and prototypes were created. They meet different

Fig. 1 Variation of field application of exoskeleton device

Features of Human-Exoskeleton Interaction

79

Fig. 2 Different types of exoskeleton devices

requirements and are able to perform various tasks. Special attention was paid to the design of control system algorithms, which had to take into account the intuitiveness of control of various drive systems—pneumatics and electromechanics (Fig. 2). In the laboratory of Robotics and Mechatronics for several years has been using one of the promising principles of control of the drive system of the exoskeleton—the use of biopotentials of the operator. The control signal for the drives of the exoskeleton is the muscular efforts of a person, familiar to him when performing manipulation operations with lighter objects. The use of human biopotentials in the control path of the exoskeleton device allows taking into account the physiological characteristics of the operator for the use of the exoskeleton in various fields of human activity, and also contributes to the rapid training of the operator due to the intuitiveness of the control signal formation. The information signal for the controller is the activation level a(t), which is the value determined by the ratio of the force Fdes(t), which is necessary to develop the muscle, to its maximum possible force, developed at the current parameters of the length lc(t) and the velocity of contraction Vc(t), determined by the calibration conditions. The muscle-tendon force F mt (t) depends on a variety of factors. It is a part of maximal isometric force Fmax , developed by the operator [1–5].  m   m   m  l (t) fV  V (t) + fd  V (t) Fmt (t) = Fmax a(t)fl 

(1)

m  Here f  l (t) is muscle-tendon dependence from the current muscle length,  m l  (t) is muscle-tendon dependence from the contraction velocity, and fV V  m   (t) corresponds to viscosity damper in the muscle. fd V

80

V. Gradetsky et al.

Many researchers propose the dependence from the current muscle length to be the next equation [7, 8]: (lm (t)−1) m  l (t) = e− γ fl  Here l (t) = m

lM , lM 0

2

(2)

where lM is current muscle length, lM 0 is optimal muscle length.

lM 0 depends on the muscle type and variates from 60 to 80% of the length of the passive muscle. γ = [0.42…0.54] is the shape factor.  m  fV  V (t) =

⎧ ⎨ ⎩

V (t)+1) 0.27( , m 0.31− V (t) m 2.51 V (t)+0.02 , m 1.24 V (t)+0.03 m

if  V (t) ≤ 0 m if  V (t) > 0 m

(3)

Here  V = VVmax , where V corresponds to current contraction velocity, Vmax is maximal contraction velocity of the current muscle. Usually, researches use Vmax = lM 0 , where τc = 0.1 s. τc m

 m  m V (t) V (t) = Bm  fd 

(4)

Here B m is the damping coefficient of the muscle. Let’s consider a person as an element of the “operator-exoskeleton” system (Fig. 3) [9, 10]. In this particular subsystem, there is a kind of a schedule that simulates the

Fig. 3 The “operator-exoskeleton” system

Features of Human-Exoskeleton Interaction

81

brain activity in encouraging the operator for action. This impulse is realized in the activation of motor neurons or synapses connecting the central nervous system and human muscles. The intensity of nerve impulses depends on the degree of motivation to the current action. This signal excites the membranes of cells of motor units, which leads to biochemical processes of ion penetration into these muscle cells, and this leads to muscle contraction. Internal subsystems of human perception control the process of the movement. The exoskeleton, as an element of the “operator-exoskeleton” system, also has its own subsystems connected with each other. The interface based on biopotential sensors can be considered as a link between the operator and the drive system of the exoskeleton. The penetration of ions into cells of motor units, as mentioned earlier, leads to a potential difference in different parts of the muscle. This information is used to form the control signal. Biopotential sensors allow receiving a signal at the stage of the beginning of muscle fiber contraction, which allows starting processing this signal in advance. The time of complete skeletal muscle contraction varies from 50 to 60 ms. Looping these elements, we get an “operator-exoskeleton” layer. This control scheme was used to form a functional model of the exoskeletal system. The experimental study of muscle activity in time during the process of holding loads of different weights was conducted. Data on the activity of the muscle group obtained experimentally revealed a close to the linear dependence of changes in the amplitude of stress in the muscles of the operator responsible for flexion and extension in the joint from the weight of the load held by the operator. However, while manipulating objects, these date changes in amplitude nonlinear. The fact is that the auxotonic contraction of the skeletal muscle depends on the position of the limb in space. This is a consequence of the influence of gravity on the object that the operator moves in space. Thus, the current state of electrical activity in one position does not equal to the current state of electrical activity in the other position. This effect is essential to understand while controlling the exoskeleton device since, in certain positions of the hand in space, the high sensitivity of the biopotential sensors will affect the creation of a high speed of movement of the control system link, which will lead to a dangerous situation.

3 Experimental Study Theoretical studies have been confirmed experimentally. As a result of the experiments, data on changes in the biopotential of the operator’s muscle groups during various movements were obtained. The figure shows a graph of the electrical activity of the biceps brachii in the process of lifting loads of 1, 2 and 3 kg weight, depending on the bending angle in the elbow joint (Fig. 4). The desired position of the hand in space corresponds to the maximum possible developed force Fmax that can be applied by the operator. To obtain this parameter, the control system must be calibrated to the individual characteristics of the operator, determining the maximum isometric load and evaluating the resulting angle.

82

V. Gradetsky et al.

Fig. 4 The experimental results

The mathematical description of the movement of the operator-exoskeleton system will be determined as follows. The “Arm–Drive” system can be considered as a complex system consisting of two interacting and interdependent subsystems. During free movement in relation to the exoskeleton system, the vector of generalized forces will be formed by the operator’s efforts and will depend on the activity of the operator’s muscle groups. However, given the delay in the formation of the control action, the vector of the generalized forces of the exoskeleton system will be formed with a delay, which can lead to a mismatch in the position of the operator’s arm and exoskeleton links. Suppose i is the maximum deviation in position. It should be noted that within the limits of the gap εi defined by the design, the operator’s hand will move freely, setting the movement of the exoskeleton, links by muscle force. In the presence of arm contact with the link of the exoskeleton, the not-holding relationship appears [11], generating reactive efforts, making changes in the dynamics of the nominal system and in the dynamics of the desired motion of the exoskeleton. ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨





∂ Ta ∂ Ta ∂ Wa d − ∂q + ∂q = i.a i.a dt ∂ q˙i.a ∂ Te ∂ Te ∂ We d − ∂qi.e + ∂qie = Mi.e dt ∂ q˙i.e

Mi.a + Ri li.a

− Mi.ext − Ri li.e ⎧ i f |qi.e (t) − qi.a (t)| ≤ εi ⎪ ⎨ 0, ⎪ ⎪ ⎪ R = −k − q − ε i f qi.e (t) − qi.a (t) > εi (q (t) (t) ), ⎪ i.a i ⎪ ⎩ i ⎩ i i.e ki (qi.e (t) − qi.a (t) + εi ), i f qi.e (t) − qi.a (t) < −εi

(5)

Here Te is the kinetic energy of the exoskeleton, We is the potential energy of the exoskeleton, qi.e ere joint coordinates associated with the parts of the system of the exoskeleton, Mi.e —generalized forces (torques) acting on an exoskeleton system, Mi.ext is a vector of external disturbing torques resulting from the interaction of links of the exoskeleton with the objects of the external environment. Where Ta is the kinetic energy of the arm, Wa is the potential energy of the arm, qi.a are joint coordinates associated with the operator’s arm, Mi.a —torques corresponding to their

Features of Human-Exoskeleton Interaction

83

coordinates qi.a . The kinetic energy of the considered arm can be found at the sum  of the kinetic energies of its components: Ta = ni=1 Ti.a , where n is the number of considered links. The supposed task of the exoskeleton drive is to form the speed of the relative rotation of its links and the compensation of external force in time [12–15]. During the design process, a synthesis of the servo system of the electric drive based on DCdrive with feedback loops of current and speed was carried out. Also, the frequency synthesis of regulators for this electric drive was carried out. As can be seen from the figure, this system allows compensating for the external moment, while forming the speed of rotation, according to the task. The level of muscle activity is a consequence of the involvement of a certain number of motor units through nerve impulses. Also, this parameter is a source of information for the biopotential sensor. Sending impulses, the brain generates tension in the muscle. This voltage is processed by an interaction block that uses the duty cycle of the nervous system of the operator and converts it for transmission to the drive control system. The controller unit is a controller that provides a control signal to the drive input of the exoskeleton. It has customizable settings for integration time, the Treg and transmission coefficient Kreg. Figure 5 shows a simplified model of operator interaction with an exoskeleton. There are three main components. The first is the subsystem of biological processes in the muscle. It reflects the reaction of the synapse to the stimulus, the dynamics of calcium metabolism and protein interaction, leading to an overall muscle contraction. At the stage of formation of the membrane potential begins recognition of the EMG action potential in the muscle and its further processing with the aim of generating control for the drive. It can be seen that in addition to the speed task, the drive also perceives the external load [16]. In this case, the external load is formed by the load from the object or cargo, as well as interaction with the operator’s hand. In this case, the moment of reaction of the operator is calculated on the basis of the mismatch of movements of the exoskeleton link and the operator’s hand. As can be seen from the figure, the delay of the operator’s muscle contraction contributes to the dynamic processes occurring in the drive.

Fig. 5 A simplified model of operator interaction with an exoskeleton

84

V. Gradetsky et al.

Figure 6 shows the results of the modeling of this system with a variation of the parameters of the regulators. The influence of various factors on the control of the exoskeleton was simulated. The change of the coefficients Treg and Kreg in the control subsystem leads to a change in the dynamics of the operator-exoskeleton system [17, 18]. For example, reducing the integration time Treg leads to position overshoot. In the process of modeling, the best result was achieved at Treg = 0.4 s. At this value, the hit in the 5% accuracy tube occurred with the shortest transition time. In further experiments, this value of the regulator parameter was used. The simulation showed that a significant decrease of the integration time Treg leads to the system stability loss, which is associated with the reaction of the operator reflected in the muscle model. The of the integration time Treg significantly increases

Fig. 6 The influence of Kreg a variation on the behavior of the drive system

Features of Human-Exoskeleton Interaction

85

Fig. 7 The Influence of the EMG sensor parameters on the dynamics of the system

the duration of position and speed transient processes for the exoskeleton drive, which does not satisfy the quality of control. The experiments also showed the effect of changes in the transfer coefficient of the biopotential sensor on the dynamics of the exoskeleton (Fig. 7). Increasing this parameter reduces the transient time by speed and position, but also results in beats, as does change Kreg. A similar situation can be observed when the integration time for the biopotential sensor changes. As shown by the simulation, the choice of the transmission coefficient in the controller is also important during the synthesis of the control. Reducing the coefficient of Kreg leads to a significant increase in transient processes on the position and speed. An increase in this parameter, however, leads to the appearance of beats, expressed an increase of the amplitude of the speed. This is due to the fact that the reaction of the operator to the changing of the position of the shaft is slow. The operator continues to impact the drive even after compensation for drive misalignments on the position that leads to mistiming of actions of the operator and the exoskeleton. Experiments have shown that these beats appear when the values Kreg > 1.2. Thus, it was shown that the formation of the control of the exoskeleton drive will depend on the parameters of the regulator and the biopotential sensor. The choice of parameters of these elements will significantly affect the dynamics of the exoskeleton. Of course, the proposed scheme is simplified. The mathematical model of the muscle is a nonlinear system that depends on the current parameters of muscle length, activity level, contraction velocity, etc. Also of great importance is the influence of

86

V. Gradetsky et al.

the antagonist’s muscle. In the real technical implementation, the most important parameter will be the maximum possible force developed by a particular operator. To obtain this parameter, the control system must be calibrated to the individual characteristics of the operator, determining the maximum isometric load and evaluating the resulting angle to calculate the optimal muscle length. We propose a method for determining the parameters depending on the physiological characteristics of the operator necessary for the exoskeleton control system. This method is used for the calibration of the exoskeleton. The feature of the method is the determination of the maximum possible force developed by the operator. To do this, the exoskeleton does not perceive the control from the sensor of biopotentials, and the action from the operator is perceived as an external disturbance. The operator needs to make a push with his muscles to change the position of the links of the exoskeleton. The force sensor located on the handle records the maximum value, and the position sensors record the current values of the generalized coordinates. Thus, the control system will identify the maximum force that the operator can develop for the current configuration. Based on the analysis of existing models of the human muscular system, the model developed by Zajac [6] was chosen and modified for the task of finding the contraction force depending on the changing parameter of the maximum force developed by the operator’s muscle. This modified model was used to create an algorithm for calibrating the exoskeleton control system and adjusting the parameters to the new operator. In this case, the speed task is calculated according to the model proposed by Hill [2]. This figure presents a completed model of interaction between the exoskeleton and human, which reflects the main subsystems. In this case, the output parameters of the muscle are calculated based on the current muscle length and contraction velocity. Also, there is a database of the current operator in the system. It takes into account the individual characteristics of the operator calculated during calibration. As shown by the simulation, the increase in the desired speed also affects the dynamics of the system. For example, proportional control in the formation of control was more effective in slow muscle contractions, and the integral component more effectively affected the dynamics at high contraction velocity. Thus, an algorithm for switching the modes of operation of the exoskeleton was proposed. Each of the modes comes from the requirements of operations that must perform the man in the exoskeleton. For example, the movement of the exoskeleton with low speed is necessary to perform some technological operations and requires increased accuracy of the desired movement. This is especially important when holding or moving fragile objects. The movement at high speeds mode allows quickly moving the links of the exoskeleton to the desired position in space for the operator. This mode is needed for use in situations where changes in direction of movement at high speed are required. A feature of this mode can be considered a requirement for a short transition time on the position and speed of the links of the exoskeleton. The simulation allowed determining that this algorithm improves the quality of control of the exoskeleton drive, using different laws of control formation for the corresponding tasks.

Features of Human-Exoskeleton Interaction

87

4 Conclusion The conducted experimental studies helped to identify the necessary values of the normalized EMG signal for the formation of the control for the drive system of the exoskeleton. The model was used for experimental studies of the possibility of controlling an exoskeleton device and improving its functionality with the use of information about changes in the biopotentials of the operator’s muscles. A series of tests of the control algorithm was carried out to determine the effectiveness of the exoskeleton when performing the task of positioning the tool in different control modes. In the process of experimental studies, the time spent on the operation to achieve the target position was measured. Tests of the control system showed that the implementation of the task by the proposed dual-mode control method is on average 24.3% faster than single-mode control, based on the proportional dependence of the developed torques on the activity of the biopotentials of the operator’s muscles. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAAA-A17117021310384-9.

References 1. Novoselov, V.S.: On mathematical models of molecular contraction of skeletal muscles. Vestnik SPbGU. Ser. 10. 3, 88–96 (2016) 2. Gasser, H.S., Hill, A.V.: The dynamics of muscle contraction. Proc. R. Soc. Lond. B 96, 398–437 (1924) 3. Hill, A.: Physiology of muscular activity, labor, and sports: a manual of physiology. Mech. Muscle Contraction (1969) 4. Haeufle, D.F., Grimmer, S., Seyfarth, A.: The role of intrinsic muscle properties for stable hopping–stability is achieved by the force-velocity relation. Bioinspiration Biomim. 5(1), 16004 (2010) 5. Sancho-Bru, J.L., Pérez-González, A., Mora, M.C., León, B., Vergara, M., Iserte, J.L., Rodríguez-Cervantes, P.J., & Morales, A.: Towards a Realistic and Self-Contained Biomechanical Model of the Hand (2011) 6. Zajac, F.E., Gordon, M.E.: Determining muscle’s force and action in multi-articular movement. Exerc. Sport Sci. Rev. 17, 187–230 (1989) 7. Zajac, F.E.: Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Crit. Rev. Biomed. Eng. 17(4), 359–411 (1989) 8. Ruzov, V.: Neuromodulation: action potential modeling. Master of Science Thesis in Biomedical Engineering, 130 p (2014) 9. Andersson, S.: Active muscle control in human body model simulations. Master’s Thesis in Automotive Engineering, CHALMERS, Applied Mechanics, Master’s Thesis, 62, 64 p. (2013) 10. Gradetsky, V., Ermolov, I., Knyazkov, M., Sukhanov, A.: Generalized approach to bilateral control for EMG driven exoskeleton. In: 12-th International Scientific-Technical Conference on Electromechanics and Robotics “Zavalishin’s Readings”, vol 113 (2017) 11. Ermolov, I.L., Sukhanov A., Knyaz’kov, M.M., Kryukova, A.A., Kryuchkov, B.I., Usov, V.M.: A sensory control and orientation system of an exoskeleton. In: 22nd Saint Petersburg International Conference on Integrated Navigation Systems, ICINS 2015—Proceedings, 181–185 (2015)

88

V. Gradetsky et al.

12. Zhechev, M.M.: Equations of motion for singular systems of massed and massless bodies. In: Proceedings of the Institution of Mechanical Engineers, Part K: Journal of Multi-Body Dynamics, vol. 221(4), pp. 591–597 (2007). https://doi.org/10.1243/14644193JMBD87 13. Pons, J.L., Rocon, E., Ruiz, A.F., Moreno, J.C.: 25 Upper-Limb Robotic Rehabilitation Exoskeleton: Tremor Suppression (2012) 14. Bergamasco, M., Allotta, B., Bosio, L., Ferretti, L., Parrini, G., Prisco, G.M., Salsedo, F., Sartini, G.: An arm exoskeleton system for teleoperation and virtual environments applications. In: IEEE International Conference on Robotics and Automation, pp. 1449–1454 (1994) 15. Burdea, G.C.: Force and Touch Feedback for Virtual Reality. Wiley, p. 368 (1996) 16. Ren, Y., Kang, S.H., Park, H.S., et al.: Developing a multi-joint upper limb exoskeleton robot for diagnosis, therapy, and outcome evaluation in neurorehabilitation. IEEE Trans. Neural Syst. Rehabil. Eng. 21(3), 490–499 (2013 May) 17. Thompson, R.L.: Integration of visual and haptic feedback for teleoperation: Ph.D. thesis, Trinity College Department of Engineering Science University of Oxford, P. 153. (2001) 18. Kazerooni, H.: Extenders: a case study for human-robot interaction via the transfer of power and information signals. Plenary Speaker at the 2nd IEEE International Workshop on Robot and Human Communication, Tokyo, Japan (1993) 19. Ishida, T., Kiyama, T., Osuka, K., Shirogauchi, G., Oya, R., Fujimoto, H.: Movement analysis of power-assistive machinery with high strength-amplification. In: Proceedings of SICE Annual Conference 2010, 2022–2025 (2010)

Analysis of Dynamics in Human—Exoskeleton Collaborative System Anna Matokhina, Alla G. Kravets, Daria Volodina, Stanislav Dragunov and Vladislav Shashkov

Abstract The article describes the synthesis module of design solutions for passive and active exoskeletons, taking into account the anthropometric parameters of the operator, the requirements and limitations imposed on the exoskeleton. Exoskeletons, presented on the market of their functional-parametric structure and technical characteristics, are investigated. The method of generating design solutions is considered by the example of an exoskeleton. An algorithm has been developed for the operator upper and lower extremities exoskeleton assembly synthesis, taking into account the anthropometric characteristics, requirements, and limitations of the exoskeleton, and also developed a system prototype for generating the assembly of the exoskeleton with regard to the anthropometric characteristics. The system for generating assemblies of different types of exoskeletons makes it easy to adapt to emerging markets and select the most suitable model taking into account various parameters. Keywords Collaborative system · Exoskeleton · Construing · Modeling · Analysis · Dynamic · 3D scanning · Assembly

A. Matokhina (B) · A. G. Kravets · D. Volodina · S. Dragunov · V. Shashkov Volgograd State Technical University, Volgograd, Russian Federation e-mail: [email protected] A. G. Kravets e-mail: [email protected] D. Volodina e-mail: [email protected] S. Dragunov e-mail: [email protected] V. Shashkov e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_8

89

90

A. Matokhina et al.

1 Introduction The scope of application of exoskeleton is constantly expanding, the functionality that gives the exoskeleton is constantly increasing. With the use of exoskeletons, a person can significantly increase their productivity, comfort in performing tasks, and their physical abilities. However, the ergonomics of the latter plays a big role in the interaction of the person from the exoskeleton. The presence of poorly set, dangling, poorly tuned or pressing parts can bring serious discomfort during prolonged wear. All types of wearable robots must be safe, comfortable and interact freely with the user. Principles of studying the possible and permissible kinematic structure of the exoskeleton are shown in [1, 2] and the fixing forces are recommended when fastened to the operator in [3, 4], the aggregation of exoskeleton implementation options is given taking into account the relationship between the exoskeleton and the human, and the donating and doffing procedure. However, modern technologies and methods allow the development of personalized robots that take into account the characteristics and precise anthropometric parameters of a person in [5, 6], methods of developing personalized robots are presented, used, for example, to recover from surgical intervention or injuries suffered. The article proposes a new method of analyzing the exoskeleton ergonomics that takes into account not only the anthropometric parameters of a person, but also allows modeling the mobility of the exoskeleton in conjunction with the geometric model of the operator, which will significantly reduce the iteration of setting and bringing the exoskeleton to the operator’s parameters. In addition, the developed method will allow producing the components of the exoskeleton that exactly correspond to the parameters of the operator, which will eliminate the customized parts and increase the reliability of the exoskeleton.

2 Method for Analyzing Human Interaction and Exoskeleton 2.1 Collection of Anthropometric Data For example, the article describes the process of collecting human anthropometric data for designing the exoskeleton of the lower extremities. When designing an exoskeleton, one should take into account the following anthropometric data: waist girth, hip girth, hip length, leg length, leg circumference, foot length, foot width (Fig. 1). In addition to measuring a person, the method involves obtaining a 3D model of a person. A 3D model of a person is obtained by scanning it in the position of the

Analysis of Dynamics in Human—Exoskeleton Collaborative System

91

Fig. 1 Anthropometric parameters of a person for designing an exoskeleton of the lower extremities

leg shoulder-width apart. Scanning was performed with an Artec EVA scanner with the following settings: • • • • •

quality of drawing (geometry)—high; the complexity of the object—the object is complex; hole filling method—closing loops; model resolution—300 dpi; the number of polygons is from 4000 to 12,000.

Scanning a person’s lower limbs takes from 15 to 20 min, with the specified settings. In the process of scanning the lower limbs of a person, it is recommended to get at least 4 layers: the front, the back of the legs, the right and left legs, the general scan. Such an arrangement makes it possible to obtain a complete model with almost no need for improvement. A variant of the model obtained by scanning, as well as all layers, are shown in Fig. 2a–d.

92

A. Matokhina et al.

Fig. 2 a The first scan layer of the model. b The second layer of the scan model. c Third layer scan model. d Fourth layer scan model

The resulting scan is processed, noise is removed from it, and discontinuities are smoothed out (Fig. 3a, b). The initial editing of the model is done directly in the Artec Studio software, which receives the layers and sticks them together into a general model. The ZBrush program smoothes the model and deletes merge errors with the environment that occurred during the scan (Fig. 4a, b). Initially, it was intended to scan a person in different positions, for example, in the position of one knee bent, but experiments showed that a person, without support, Fig. 3 a Assembling model layers. b Model processed after scanning using Artec Studio

Analysis of Dynamics in Human—Exoskeleton Collaborative System

93

Fig. 4 a Errors merging with the environment when scanning. b The final model is processed after scanning using ZBrush

could not maintain this position for more than 10 min and it was decided to simulate different positions. Requirements for the clothing of a person during scanning were built for further analysis of the interaction dynamics: • flat shoes, preferably sneakers; • tight pants, preferably jeans, not flared; • T-shirt would be raised to the waist jeans, but not tucked inside. The final model is loaded into the editor, where the construction of variants of movements of the model takes into account the tasks performed.

2.2 Exoskeleton Variants Analysis Today in the world there are more than 95 commercial models of exoskeletons. Exoskeletons production in various ways. To systematize the information, the ontology of exoskeletons is constructed (Fig. 5). The analysis revealed the types of the exoskeleton, the structure of the exoskeleton, the main options for controlling the exoskeleton, options for energy sources, executive mechanisms and much more. However, it was not possible to build a complete ontology with multiple instances of classes, because detailed information on the structure and characteristics of the constituent parts of exoskeletons is not presented in general access.

94

A. Matokhina et al.

Fig. 5 Exoskeleton ontology classes’ tree

2.3 Exoskeleton Structure The exoskeleton contains the following essential components [7]. Frame system of mechanical devices [8], which consists of working bodies, joints and steel rods and is represented as a set with successively connected to each other subsets of elements of different hierarchical levels. The external frame is a rigid, stationary frame of the exoskeleton, made in the form of a three-dimensional frame structure that defines the space for changing the position of the orthopedic modules of the internal structure [9]. The inner frame is a controlled, mobile frame of the exoskeleton, which is kinematically connected with the outer one [9]. A drive unit that includes control devices, power converters, and executive motors. A servo drive is any type of mechanical drive (device, working body) that includes a sensor (position, speed, effort, etc.) and a drive control unit (electronic circuit or mechanical system of rods) that automatically supports the necessary parameters on

Analysis of Dynamics in Human—Exoskeleton Collaborative System

95

the sensor (and, respectively, on the device) according to a given external value (the position of the control knob or the numerical value of other systems) [10]. Air muscles (BM) are devices that contract or stretch under the action of air pressure. They represent a hermetic shell in a casing with a braided fabric of inextensible filaments [11]. An actuator is an engine that is used to move or control the mechanics of a system [12]. The control system is a system that controls the movement of the exoskeleton, which performs the following functions: • control of mechanical movements in real-time; • organization of data exchange with peripheral devices, sensors and other devices of the system; • interaction with the human operator via the interface in other modes of system operation. Sensor—a measurement tool designed to generate a signal of measurement information in a form convenient for transmission, further transformation, processing and/or storage, but not amenable to direct perception by an observer [13]. A battery is a device for storing energy for the purpose of its subsequent use. Thus, it is possible to determine the composition of the main elements of the exoskeleton (Fig. 6): 1. Frame system – – – –

Outer frame Inner frame Shape memory frame Power frame

2. Drive unit – – – – –

Articulated actuators Hydraulic actuators Pneumatic actuators Servos Elastic drives

3. Control system – – – –

Control system based on sensor measurements Microcomputer-based control system Control system with remote control Control system based on combined microelectronics

4. Additional elements – Sensors – Battery – Actuator

96

A. Matokhina et al.

Fig. 6 Exoskeleton-user interaction model

A parametric model of the exoskeleton that takes into account modern trends in designing exoskeletons of the lower extremities, obtained during the analysis of existing analogs and an array of patent information on exoskeletons of this type, is built.

2.4 A Parametric Model of the Exoskeleton of the Lower Extremities A parametric model of the lower limbs exoskeleton has been developed. The model contains 41 elements and takes into account the anthropometric parameters described in paragraph 1. The assembly of the exoskeleton is shown in Fig. 7. Table 1 shows the specification of the assembly of the exoskeleton. The comments in the table show typical elements and parametric elements. For parametric elements, it is indicated taking into account which anthropometric parameters are being built. Typical items have typical sizes. The largest typical drive element is with a rod with a diameter of 60 mm. From the model of the assembly of the exoskeleton of

Analysis of Dynamics in Human—Exoskeleton Collaborative System

97

Fig. 7 Assembling the exoskeleton of the lower extremities

the lower extremities, limitations for humans were obtained. So the growth of the operator cannot be less than 155 cm and more than 190 cm. For the convenience of the user, an interface has been created for inputting anthropometric parameters and automatically transmitting them to the assembly to adopt the elements. In addition to assembling a passive exoskeleton of the lower extremities, an active exoskeleton assembly has been developed.

98

A. Matokhina et al.

Table 1 Assembly specifications of the exoskeleton of the lower extremities #

Identification

Description

Qty

1

Fastening the back to the hip joint

Swivel part

2

Comments

2

Back bolt 40 mm

40 mm bolt

14

Typical item

3

Nut for bolt 40 mm

Nut for bolt 40 mm

6

Typical item

4

Hip-to-back mounting with mirror hinge

The second part of the swivel

2

Typical item

5

Pivot bolt

Hinge bolt

2

Typical item

6

Hinge bolt internal

The second part of the hinge connection

4

Typical item

7

Electromagnetic drive 36 mm

The drive with a rod 36 mm

2

Typical item

8

Drive bolt 33 mm

Bolt for fixing the drive on the thigh

24

Typical item

9

Pressure meter

Pressure meter

12

Typical item

10

Bolt for the sensor in the shin

Bolt for mounting pressure sensor

48

Typical item

11

Electromagnetic drive 60 mm

The drive with a rod 60 mm

2

Typical item

12

Mount with foot

Fastening element

2

Typical item

13

Pivot foot bolt

Bolt for fixing the foot

4

Typical item

14

Hinge bolt internal store

Bolt 2 for fixing the foot

4

Typical item

15

Bolt articulated foot 1

Hinge for movement of the foot element

2

Typical item

17

Parameterization of the back is average

Back support frame

1

Parametric element is determined by the parameters of the length of the back and thigh circumference

20

Thigh carcass parametrization average

Thigh support frame

2

The parametric element is determined by the parameters of the length of the hip and thigh circumference

23

Belt lock middle

The element of fixing the operator in the belt

1

The parametric element is defined by the thigh circumference parameter

26

Belt mount medium

Fastening for foot fixers

4

Parametric element is determined automatically, depending on the parameters of the thigh and lower leg

Typical item

(continued)

Analysis of Dynamics in Human—Exoskeleton Collaborative System

99

Table 1 (continued) #

Identification

Description

29

The skeleton of the leg is parameterization average

Supporting skeleton

Qty 2

Comments Parametric element is determined by the parameters of the length of the leg and the circumference of the leg

32

Stop left parametrization average

Skeleton for the left foot

1

Parametric element is determined by the parameters of the length of the tops and width of the foot

35

Stop right parametrization average

Skeleton for the right foot

1

Parametric element is determined by the parameters of the length of the tops and width of the foot

38

Foot mediator

Operator’s foot lock

2

The parametric element is determined by the thigh circumference and lower leg circumference

41

Medium leg lock

Operator’s fixer

4

The parametric element is determined by the thigh circumference and lower leg circumference

2.5 Description of the Method for Analyzing the Mobility of an Operator with an Exoskeleton When Performing Certain Tasks 2.5.1

Description of Allowable Values for the Dynamics of the Exoskeleton

Currently, there is the problem of controlling the exoskeleton by the human operator due to the fact that the human body is a nonlinear system with distributed parameters [14]. The question of ergonomics has been studied in a number of studies devoted to the development of technical means of rehabilitation [14, 15]. To determine the level of ergonomics in the interaction of the human-exoskeleton system, an analysis is made of the implementation of such production and consumer tasks as lifting the load, performing surgical operations and moving the human operator with paraplegia of the lower extremities. In the process of performing tasks, physical overloads are possible due to physical dynamic load, the mass of lifted and moved cargo manually, stereotypical working movements, static load, long working posture, body tilt, movement in space. These overloads are often harmful and

100

A. Matokhina et al.

dangerous production factors and work, during which preliminary and/or periodic medical examinations must be carried out [16]. When designing an exoskeleton, it is necessary to take into account the peculiarities of human anatomy and to apply flexing restraints. Limitations include: (1) The excess of the bending angles of the structural element of the foot design is more than 60° backward and more than 30° forward from the direct position of the foot support (Fig. 8). (2) Bending of the structural element of the leg forward is unacceptable (Fig. 9).

Fig. 8 Modeling the movement of the foot in the exoskeleton of the passive type

Fig. 9 Invalid bend in the leg for an exoskeleton of the passive type

Analysis of Dynamics in Human—Exoskeleton Collaborative System

101

Fig. 10 Modeling the movement of the thigh in the exoskeleton of the passive type

(3) The excess of the angles of flexion of the structural element of the femoral joint is more than 90° to the top and more than 40° back from the straight position (Fig. 10).

2.5.2

Acceptable Tasks in the Human—Exoskeleton Collaborative System

The tasks those are available to the exoskeleton operator. (1) Task: stair climbing and descending Act: Lift: bending the leg at the knee, moving the support to the bent leg, extending the knee-bent leg Descent: bending the legs at the knee, moving the support to the straight-ahead of the standing leg Kinematic scheme of action: Fig. 11a–c. Description: The pelvic design of the exoskeleton has: – 3 degrees of freedom (flexion/extension, adduction/abduction, lateral meridional rotation), – 1 degree of freedom in the knee, which allows you to perform flexion and extension of the lower leg, – 1 degree of freedom in the foot (bending back and plantar flexion) that allows you to climb and descend the stairs.

102

A. Matokhina et al.

Fig. 11 a Kinematic scheme of the first action. b Kinematic scheme of the second action. c Kinematic scheme of the third action. d Kinematic scheme of the fourth action

(2) Tasks of the exoskeleton/restrictions: Remove the load from the muscles of the legs. Task: Human verticalization (position fixation). Act: Fixing the feet and maintaining balance. Kinematic scheme of action: Fig. 12. Description: The balance of the exoskeleton is maintained by means of two pneumatic muscles, the feet are fixed in the ankle element of the structure. Tasks of the exoskeleton/restrictions: Balance support. (3) Tasks of the exoskeleton/restrictions: Lifting loads Task: Knee bending while squatting; fixing the feet, lifting the load from the back. Kinematic scheme of action: Fig. 13a–c. Act: Knee bending while squatting; fixing the feet, lifting the load from the back Kinematic scheme of action: Description: Removal of the load from the back is made by the design to support the spine, fixation of the feet is made in the ankle element of the structure. Tasks of the exoskeleton/restrictions: Unloading, balance support.

Fig. 12 Kinematic scheme of Human verticalization

Analysis of Dynamics in Human—Exoskeleton Collaborative System

103

Fig. 13 a Kinematic scheme of the first action. b Kinematic scheme of the second action. c Kinematic scheme of the third action

(4) Task: Walking Act: Bending of the knees, maintaining balance, the possibility of rotation of the pelvis. Kinematic scheme of action: Fig. 14a–c. Description: Bending the knees, maintaining the design allows you to perform repeated walking cycles due to the possibility of turning the movement of the hip design element from one extreme position before the other, the support structure also maintains balance and assumes the correct position for the next step. Balance, the possibility of turning the pelvis. Tasks of the exoskeleton/restrictions: Balance, lifting the load from the muscles of the legs. The constructed tasks with kinetic schemes allow us to carry out a full analysis of the dynamics of the exoskeleton when interacting with a person and to evaluate ergonomics on a geometric model, taking into account parametric characteristics.

Fig. 14 a Kinematic scheme of the first action. b Kinematic scheme of the second action. c Kinematic scheme of the third action

104

2.5.3

A. Matokhina et al.

Description of Examples in the Human—Exoskeleton Collaborative System

The analysis of the dynamics of the Human—exoskeleton collaborative system is carried out using the primitives “Skeleton” in the system of geometric modeling (Fig. 15). The skeleton can be assembled from bones by hand, thus obtaining unique shapes. Plugins can be loaded into the program to simplify work. Also, there are many tools and modifiers for creating a realistic movement of the character and high-quality texturing of the resulting objects of the scene. For analysis, the primitive geometry is adjusted according to the human parameters obtained, the length of the feet, the lengths of the femoral and tibial spines, and the volumes of the thighs are changed. Next, the primitive is connected to the scanned model, i.e. the joints are combined with the nodal elements of the primitive. Using a primitive allows you to automate the process of building dynamic actions. Consider a few examples of the Human—exoskeleton collaborative system when performing various tasks.

Fig. 15 Human skeleton in CAD system. Kinematic primitive

Analysis of Dynamics in Human—Exoskeleton Collaborative System

105

Fig. 16 a Modeling knee bend with intersecting models. b Corrected model of the assembly of the exoskeleton based on the results of the structural analysis

During the lifting of cargo, taking into account the rules of biomechanics, the operator performs the following actions sequentially: 1. 2. 3. 4.

Stand next to the load so that you do not have to lean forward. Foot at a distance of 30 cm from each other, one foot slightly pushed forward. Bend the knees, keeping the torso upright. Press the load, do not straighten the knees.

When the human operator bends the leg, during which the angle between the thigh and lower leg decreases below 90°, the assembly parts intersect (Fig. 16a), which indicates that the current configuration of the exoskeleton does not meet the requirement. The design is changed in such a way (Fig. 16b) that the human operator can bend the knee to an angle of fewer than 90° without experiencing pressure from the parts. During surgeries, surgeons experience a heavy load on the spine and arms. Operation time can reach 10 h. Long stay in one unchanged posture in conditions of high humidity and temperature leads to pathologies such as radiculopathy of the cervicobrachial spine and osteochondrosis [17]. According to the data given in [18], the prevalence of back disorders was 44% among medical staff and it was 19% for acute low back pain, 17% for chronic low back pain and 8% for diagnosed lumbar disc hernia. The body of the surgeon during surgery is tilted by 45°, the head—by 60–80° with a norm of about 10°. A large load falls on the lower extremities: swelling of the lower leg increases, the foot flattens by 4–5 cm. Moving the blood to the extremities causes ischemia of the organs and brain, which can lead to dizziness and headaches. Working posture during surgery contributes to the compression of the abdominal organs [19]. When performing the operation, the following working postures are suggested for the surgeon’s lower limbs (Fig. 17).

106

A. Matokhina et al.

Fig. 17 Variants of poses in the surgeon when performing the operation

Fig. 18 Model of the exoskeleton view from the back

The exoskeleton of the lower extremities will remove the load from the legs, which will increase efficiency and will not bring restrictions in mobility for the upper part of the body (Fig. 18). For this purpose, a passive exoskeleton is used. Spinal cord injuries and spinal cord injuries are much less common than brain injuries and in peacetime are 1–4% in the structure of general injuries. In most cases, this is an indirect injury. The most common cause is a fall from the height on the buttocks, back, head, compression of a bent torso during collapses, hitting the head against the bottom when jumping into the water, etc. [20]. Spinal cord injuries in spinal injuries are characterized by serious consequences in the form of motor, sensory, trophic and other disorders leading to permanent disability. Since movement disorders are noted in 100% of patients, one of the main problems of the rehabilitation of this contingent is the restoration of the function of movement [21]. Restoration of motor functions, in particular, walking, in such patients is a complex task requiring the use of long-term complex treatment (surgical, medical, physiotherapy), physical therapy and orthotics. However, the use of a large complex of therapeutic measures is not always effective [22].

Analysis of Dynamics in Human—Exoskeleton Collaborative System

107

Fig. 19 Modeling a passive-type exoskeleton with an operator with paraplegia of the lower limbs in a sitting and standing position

The tasks that form the basis for the rehabilitation of patients with lower limb paraplegia due to spinal cord injury are highlighted (Fig. 19): 1. 2. 3. 4.

the ability to stand for a long time; improved resilience; acquisition of movement skills; reducing the risk of fractures as osteoporosis.

To solve the tasks it is proposed to use the active exoskeleton of the lower extremities. Its design allows the legs to be fixed in one position, which improves stability and allows the patient to stand for a long time. Additional motors of the driven exoskeleton will allow the patient with lower limb paraplegia to acquire movement skills.

3 Conclusion The developed method can significantly improve the ergonomics and comfort in the operation of exoskeletons. In general, the method consists of the following main steps: • Determination of anthropometric parameters of a person—exoskeleton operator. • Obtaining a 3D model of the human operator of the exoskeleton by scanning and further processing the model.

108

A. Matokhina et al.

• Obtaining the kinematic scheme of the exoskeleton operator using primitives of the human skeleton with the specified anthropometric parameters. • The combination of the scanned model of the human operator of the exoskeleton with the parameterized primitive of the human skeleton in the CAD system. • Construction of a parametric assembly of a human exoskeleton, taking into account the selected parameters and the size of a human operator. • Evaluation of ergonomics in a static standing position, in which the operator is scanned. The study of mutually intersecting or poorly fitting parts, assemblies, fasteners. • Bringing component assembly exoskeleton if necessary. Consideration of additional dimensions of the operator for further design if necessary. • Determination of typical variants of dynamic schemes during human interaction with the exoskeleton. • Simulation of dynamics in the interaction of man and exoskeleton in the CAD environment. Analysis of possible situations of discomfort in modeling the dynamics, friction, tension, reduced mobility, heating. • Making changes in the design or assembly elements of the exoskeleton according to the results of the analysis of the dynamics Conducting repeated research. In the future, it is planned to develop a module for automatic evaluation of geometry for fixing intersections or an excessive increase in the distance between the human body and the exoskeleton.

References 1. Schiele, A.: Fundamentals of Ergonomic Exoskeleton Robots. TU Delft, Delft University of Technology, Chicago (2008) 2. Munoz, L.: Ergonomics in the Industry 4.0: Exoskeletons (2017) 3. Näf, M., Junius, K., Rossini, M., Rodriguez-Guerrero, C., Vanderborght, B., Lefeber, D.: Misalignment compensation as a way to ensure full human-exoskeleton kinematic compatibility: state of the art and evaluation. Appl. Mech. Rev. 70, 050802 (2019). https://doi.org/10.1115/1. 4042523 4. Louie, D., Eng, J.: Powered robotic exoskeletons in post-stroke rehabilitation of gait: a scoping review. J. NeuroEng. Rehabil. 13, 53 (2016) 5. Yang, K., Fei Jiang, Q., Lai Wang, X., Wu Chen, Y., Yan Ma, X.: Structural design and modal analysis of exoskeleton robot for rehabilitation of lower limb. J. Phys. Conf. Ser. 1087, 062004 (2018). https://doi.org/10.1088/1742-6596/1087/6/062004 6. Long, Y., Du, Z., Wang, W., Dong, W.: Development of a wearable exoskeleton rehabilitation system based on hybrid control mode. Int. J. Adv. Robot. Syst. 13 (2016). https://doi.org/10. 1177/1729881416664847 7. Nacy, S., Hussein, N., Abdallh, M.M.M.: A review of lower limb exoskeletons. Innovative Syst. Des. Eng. 7, 1 (2016) 8. Frumento, Ch., Messier, E., Montero, V.: History and future of rehabilitation robotics (2010). https://web.wpi.edu/Pubs/E-project/Available/E-project-031010-112312/unrestricted/ HRRIQP_Final.pdf. Accessed 14 Mar 2019 9. Villoslada, A., Rivera, C., Escudero, N., Martín, F., Blanco, D., Moreno, L.: Hand exo-muscular system for assisting astronauts during extravehicular activities. Soft Robot. 6, 21–37 (2018). https://doi.org/10.1089/soro.2018.0020

Analysis of Dynamics in Human—Exoskeleton Collaborative System

109

10. Servo control facts. http://www.baldor.com/pdf/manuals/1205-394.pdf. Accessed 27 Apr 2018 11. Prabhakaran, A., Ghosal, A.: A survey on static modeling of miniaturized pneumatic artificial muscles with new model and experimental results. Appl. Mech. Rev. 70, 040802 (2018). https:// doi.org/10.1115/1.4041660 12. A great combination: pneumatic actuator, pneumatic timer, pneumatic valves, and pneumatic indicators. http://www.ekci.com/a-great-combination-pneumatic-actuator-timer-valvesand-indicators-ezp-69.html. Accessed 14 Apr 2019 13. Cecilia Villa-Parra, A., Delisle Rodriguez, D., Botelho, Th., Villarejo Mayor, J., Delis, A., Carelli, R., Frizera, A., Freire Bastos, T.: Control of a robotic knee exoskeleton for assistance and rehabilitation based on motion intention from sEMG. Res. Biomed. Eng. 34 (2018). https:// doi.org/10.1590/2446-4740.07417 14. Folgheraiter, M., Jordan, M., Straube, S., Seeland, A., Kyoung Kim, S., Kirchner, E.: Measuring the Improvement of the interaction comfort of a wearable exoskeleton: a multi-modal control mechanism based on force measurement and movement prediction. Int. J. Soc. Robot. 4, 285– 302 (2012). https://doi.org/10.1007/s12369-012-0147-x 15. Curtis, S., Kobetic, R., Bulea, T.C., et al.: Sensor-based hip control with hybrid neuroprosthesis for walking in paraplegia. J. Rehabil. Res. Dev. 51(2), 229–244 (2014). https://doi.org/10.1682/ JRRD.2012.10.0190 16. Schiele, A., Hirzinger, G.: A new generation of ergonomic exoskeletons—the high-performance X-Arm-2 for Space Robotics Telepresence. In: IEEE International Conference on Intelligent Robots and Systems, pp. 2158–2165 (2011). https://doi.org/10.1109/IROS.2011.6094868 17. Practical experience of applying Order 302n released by Russian Federation Health and Social Development Ministry on April 12, 2011 “On approving the lists of hazardous and (or) dangerous occupational factors and works that require obligatory preliminary and periodic medical examinations (workup) and on regimen of obligatory preliminary and periodic medical examinations (workup) for workers exposed to intense labor and to hazardous and (or) dangerous work conditions”. https://europepmc.org/abstract/med/23156064. Accessed 3 Mar 2019 18. Jiménez, P., Pavés, C.: Occupational hazards and diseases among workers in emergency services: a literature review with special emphasis on Chile. Medwave 15, e6239 (2015). [PubMed] [Google Scholar] 19. Olivier, B.: Factors Associated with Low Back Pain in Hospital Employees (2008) 20. Memon, A., Naeem, Z., Zaman, A., Zahid, F.: Occupational health related concerns among surgeons. Int. J. Health Sci. (Qassim) 10(2), 279–291 (2016) 21. Mallory, A., Stammen, J., Zhu, M.: Cervical and thoracic spine injury in pediatric motor vehicle crash passengers. Traffic Inj. Prev. 20, 1–9 (2019) 22. Thuc, T., Yamamoto, Sh: Development of a body weight support system using pneumatic muscle actuators: controlling and validation. Adv. Mech. Eng. 8, 1–13 (2016). https://doi.org/ 10.1177/1687814016683598 23. Lajeunesse, V., Vincent, C., Routhier, F., Careau, E., Michaud, F.: Exoskeletons’ design and usefulness evidence according to a systematic review of lower limb exoskeletons used for functional mobility by people with spinal cord injury. Disabil. Rehabil. Assistive Technol. 4, 1–13 (2015). https://doi.org/10.3109/17483107.2015.1080766 24. Baniqued, P., Dungao, J., Manguerra, M.V., Baldovino, R., Abad, A., Bugtai, N.: Biomimetics in the design of a robotic exoskeleton for upper limb therapy. In: AIP Conference Proceedings, vol. 1933, p. 040006 (2018). https://doi.org/10.1063/1.5023976 25. Tran, T., Cheng, H., Lin, X., Duong, K., Huang, R.: The relationship between physical humanexoskeleton interaction and dynamic factors: using a learning approach for control applications. Sci. China Inf. Sci. 57, 1–13 (2014). https://doi.org/10.1007/s11432-014-5203-8

Multi-agent Robotic Systems

This part defines intelligent control methods for robotic groups on the base of multi-agent theory and describes their control architecture. The authors study the multi-agent control methods implementation in conditions of uncertainty and dynamically changing environment. 1. S. L. Zenkevich, Jianwen Huo—Dynamic Switching Method of Multi-agent Formation in Unknown Obstacle Environment. 2. V. V. Serebrenny, A. M. Shereuzhev—Dependence of Dynamics of the Multi-robot System on Control Architecture. 3. A. V. Nazarova, Meixin Zhai—The Application of Multi-agent Robotic Systems for Earthquake Rescue. 4. E. S. Briskin, Ya. V. Kalinin—Adaptive Field Sprinkler Machine as a Multi-agent Robotic System.

Formation Control of Ground Multi-agent System Using Quadcopter with Camera Stanislav L. Zenkevich, Anaid V. Nazarova and Jianwen Huo

Abstract The paper is dedicated to discussing a method for ground multi-agent formation control in different obstacle environments. Use the data from the quadcopter vision system and the lidar sensor of the ground agent to create a global map. Then, we perform coordinate estimation of the ground agent by using the terrestrial agent’s odometer, GPS, and quadcopter vision system. Finally, we use the finite state machine theory to control the switching of ground multi-agent formation, and propose a strategy to avoid ground multi-agent collisions when the formation is switched. The effectiveness of this strategy is demonstrated through simulation experiments. Keywords Quadcopter · Ground multi-agent system · Environmental map · Coordinate estimation · Finite state machine theory · Formation switching · Collision avoidance strategy

1 Introduction Nowadays, with the continuous development of artificial intelligence and agent technology, the problem of multi-agent formation has become one of the hotspots of scholars. Multi-agent formation behavior can be seen everywhere in the natural biota, such as the wild geese flying in line, the fish swarm in groups in the water, the wolves flock in groups to prey. Compared with a single agent, the multi-agent system has incomparable advantages. For example, “coordination” and “cooperation” between multi-agent can improve the ability and efficiency of the system to complete complex The author Stanislav L. Zenkevich has expired. S. L. Zenkevich · A. V. Nazarova (B) · J. Huo Department of Robotic Systems and Mechatronics, Bauman Moscow State Technical University, Izmaylovskaya sq., 7, Moscow 1005007, Russian Federation e-mail: [email protected] J. Huo e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_9

113

114

S. L. Zenkevich et al.

tasks, and at the same time enhance system reliability [1]. For example, the literature [2] proposes a multi-agent system cooperative reconnaissance scheme consisting of a quadcopter and a ground agent. The scheme automatically maps urban areas through a multi-agent system and provides real-time data and image telemetry to remote human operators. The use of quadcopter facilitates the positioning of the ground agent. Multi-agent formation refers to a group of cooperative agents that sense the surrounding environment and their own state through sensors, form and maintain a specific formation shape, and realize the movement to the target in an environment with obstacles to complete the specified task. However, as the environment changes or tasks change, the multi-agent system is required to change the current formation to avoid obstacles or to complete tasks quickly. From the control form of the queue, multi-agent formation control can be divided into centralized and distributed formation control [3]. In a centralized structure, the queue has a centralized control unit, which can be composed of one or several master agents, and even the control unit can be acted by other smart units, which are independent of the formation agent group. For example, the literature [4] describes a “convoy” formation system consisting of a team leader agent and several follower agents and proposes a “convoy” formation control algorithm. Distributed formation control is mainly divided into behavior methods, Leader-Follower methods and graph theory [5]. The behavior methods are mainly divided into simulating the behavior of natural biotic populations and establishing a set of basic behaviors of agents. For example, the scholar Olfati-Saber [6] designed the distributed formation control algorithm of the multi-agent system moving in free space by using Reynolds’ three rules and carried out simulation experiments to verify the correctness of the algorithm. Haque et al. [7] modeled the food behavior of bottlenose dolphins and proposed three control strategies for multi-agent formation switching. However, the method of establishing the basic behavior set of the agent needs to define a behavior set containing the simple basic behavior of the agent, and then the overall formation behavior of the multiagent system is established by using the combination of behavior sets. For example, Saad and other scholars [8] studied the problems of the multi-agent formation and the agents in the face of complex situations. Besides, the authors considered the behaviors such as following, avoiding, moving into key circles, walking and sending, and designed controllers based on the five behaviors. They also provided queue members with the ability to handle complex situations in the formation process. The multi-agent formation switching is controlled based on the Leader-Follower methods, of which the basic idea is to change the relative position and relative direction between the leader and follower. For example, the literature [9, 10] controls the switching between multi-agent formations by switching the Leader-Follower controller to achieve a multi-agent system to go through gaps or avoid obstacles. Besides, the literature [11, 12] provides a formation control solution with a defined formation structure that can dynamically adapt to the environment. Moreover, the solution enables a multi-agent system to safely avoid obstacles and pass narrow passages, which also includes a collision-free strategy for multi-agent formation switching.

Formation Control of Ground Multi-Agent System Using Quadcopter …

115

In addition, the graph theory uses graph theory to construct the network dynamics of multi-agent topology and derives a reasonably distributed control algorithm to guide the agent to the desired position. For instance, Jaydev and Desai [13] used graph theory to establish a multi-agent position-control graph model and proposed a coordinated switching method between different formations to realize the movement of each agent between any formations while avoiding obstacles. Besides, Reference [14] used an undirected graph and a weighted adjacency matrix to describe the relationship between multi-agent and the set of formations of all possible topologies, and it switches the topological structure by a switching signal. Moreover, Literature [15] studied the construction and transformation of two-dimensional continuous graphs, which could be applied to multi-agent formation coordination and control. Nowadays, the work on multi-agent formation switching focuses more on independent formation switching (such as switching between line, column, triangle, and rhombus), but agents need to avoid obstacles and have the ability to switch formations when performing tasks in complex environments such as farms and earthquakes. Therefore, this paper discusses the dynamic switching of multi-agent formation in complex obstacle environments.

2 Problem Statement Assuming quadcopter and ground multi-agent system move in obstacle environment (as shown in Fig. 1), the quadcopter with the camera to obtain continuous ground images from the air and provides an environmental global map for terrestrial multiagents. In addition, the leader in terrestrial multi-agent systems needs to install a lidar sensor, whose collected data is used to create an environmental local map. Initially, the ground agents move in the XOY plane according to a certain formation. Then the formation of the ground multi-agent system needs to change with the environmental obstacles to ensure that the agents complete the specified task more efficiently and quickly. In order to solve the problem of the formation control of ground multiagent systems in an obstacle environment, it is necessary to complete three tasks: environment map establishment, agent positioning, and control strategy.

3 Environmental Map Building and Agent Positioning 3.1 Environmental Map Building The system coordinate is shown in Fig. 1. Let O − X Y Z be the absolute coordinate system, Or − X r Yr be the agent coordinate system, and Oq − X q Yq Z q be the quadcopter coordinate system. In order to simplify the model, it is assumed that the coordinate system of the two-dimensional lidar sensor is stationary relative to

116

S. L. Zenkevich et al.

Zq Yq Xq

Target

Z O

Xr Y

Yr

X Fig. 1 Schematic diagram of the group multi-agent system

the agent coordinate system, and the coordinate system of the camera is consistent with the quadcopter. Besides, assuming that the coordinate of a point M in the O − X Y Z coordinate system is r 0 , the coordinate of the point M in the Or − X r Yr and Oq − X q Yq Z q coordinate systems are: r 0 = T q r q and r 0 = T r r r ,

(1)

of which T q is Oq − X q Yq Z q relative to the O − X Y Z mapping matrix, and T r is Or − X r Yr relative to the O − X Y Z mapping matrix. It is thus possible to obtain the coordinates of the point M observed by the quadcopter in the coordinate system of the agent Or − X r Yr : r r = T r−1 T q r q

(2)

Assuming that r is coordinate of the point M observed by the lidar sensor on the agent, the local map established by the ground agent through the two-dimensional lidar sensor has the following relationship with the global map established by the quadcopter:

Formation Control of Ground Multi-Agent System Using Quadcopter …

r r = T r,

117

(3)

⎡

⎤ cθ −sθ x of which T = ⎣ sθ cθ y ⎦, θ is the rotation angle of the point between the 0 0 m two coordinate systems, x, y is the amount of translation, and m is the scale. Then the problem of establishing the environment map is transformed into solving four unknowns θ, x, y, m. In order to solve the four unknowns, the quadcopter uses the camera to take pictures and transmits them to the ground agent with the lidar sensor through the wireless network; to form the global map, the image processing technology (image filtering, image correction, image registration, image stitching) is used after the agent receives the image transmitted by the aircraft. Then the characteristics of the singleframe laser data are extracted to match the features of the global map. Moreover, the least-squares principle ε = minT r − r r 2 can be used to solve the matrix T: 

i =4 T = r ri r i−1 ; T = (r iT r i )−1 r i r ri ; i > 4

(4)

Among them, i = 1, 2, . . . , N ; when N is the number of matching feature points. By solving Eq. (4), an environmental map can be established to know the environmental obstacle information.

3.2 Agent Positioning When the multi-agent formation is switching, every agent’s position needs to be known. The basic idea to solve this problem is to use the odometer sensor and GPS of each agent, as well as the camera of the quadcopter to make parameter estimation to determine the position of each agent. Odometer. It is assumed that the radius of the i-th ground agent wheel is Ri , and the number of pulses output by one rotation of the rotary encoder is n 0 . In addition, it is assumed that the number of pulses of the right and left wheels collected is nri , n li during the time interval t. Therefore, the speed of the right and left wheels can be derived as:  nr vir = 2π · n 0 ·ti · Ri (5) nl vil = 2π · n 0 ·ti · Ri According to the kinematic model of the two-wheeled agent, the coordinates of the i-th ground agent in the absolute coordinate system can be derived as:

118

S. L. Zenkevich et al.

⎧ ⎨ xt+1 = xt + t · vi cos θt y = yt + t · vi sin θt ⎩ t+1 θt+1 = θt + ωi · t

(6)

Among them, x, y, θ are the position and direction of the ground agent; v, ω are respectively the linear velocity and angular velocity of the ground agent. However, the model (6) is the ideal position of the ground agent, and the following aspects need to be considered in real situation: the sliding of the ground agent wheels; incorrectly estimating the size of the wheels and the agent; the error value of the wheel partial rotation angle counter; the error between the steering wheel rotation and the true rotation caused by the uncertainty of the input. In these cases, the odometer measurement is incorrect. GPS. Another method of positioning a ground agent is to use satellite radio navigation systems. Therefore, in the open air, the ground agent uses a GPS receiver to receive signals of a plurality of satellites and calculates its own position information based on these signals. The calculation formula of the ideal model is as follows: d 2j = (x j − x)2 + (y j − y)2 + (z j − z)2 + [c(dt − dt j )]2

(7)

Among them, d j = ct j is the distance between satellite j and the receiver; c is the propagation speed of the GPS signal; t j is the time required for the signal of satellite j to reach the receiver; (x j , y j , z j ) is the satellite coordinate; dt is the receiver clock bias; dt j is the satellite j clock bias. However, when using GPS for positioning, it is affected by various factors, which will result in positioning errors. The main sources of error in GPS systems are GPS satellites, signal propagation processes, and the receivers. For example: satellite clock error and satellite ephemeris error; atmospheric delay after GPS signal passing through the atmosphere, multipath effect after electromagnetic waves are reflected by surrounding objects for many times; receiver clock error, receiver position error, various parts of electronic devices thermal noise, signal quantization error, algorithm error in determining code phase and carrier phase, and calculation error in receiver software. Machine vision system. According to the formula (1), the coordinates of the ground agent in the O − X Y Z coordinate system can be obtained. However, during the acquisition and transmission of the image, with the influence of the sensor material properties, the working environment, the electronic components and the circuit structure, there will be various noises, such as thermal noise caused by resistance, channel thermal noise of FET, etc., which will affect the measurement results. Ground agent coordinate estimation. It is assumed that the coordinates of ground agent obtained by the odometer, GPS, and camera are respectively r o , r g , r c . The coordinates of the estimated ground agent according to the multi-sensor fusion method proposed in [16] are: r = a1 r o + a2 r g + a3 r c

(8)

Formation Control of Ground Multi-Agent System Using Quadcopter …

119

 Among them, a1 , a2 , a3 are coefficients; a1 +a2 +a3 = 1 and ai = d1i / Nj=1 d1j , d is the variance of the sensor when measuring noise (standard normal distribution); N is the number of sensors.

4 Formation Switching Control Strategy First, all the formation structures of the ground multi-agent system are enumerated by the Polya theorem. All of the formations here are related to the shape of the obstacles in the map in Sect. 3.1 of this paper, which facilitates the control of the switching and movement of the ground multi-agent formation. Afterward, it is determined according to the formula (9) that the single ground agent i moves from a certain j position r io of the current formation structure F to a position r g in the target formation structure Fg . N

2 xi j r io − r gj → min

(9)

i, j=1

Among them, N is the number of ground agents; xi j is the relationship between a certain position of the current structure type Fo and a position in the target structure Fg . In addition, xi j ∈ {0, 1}, xi j = 1, indicates that there is a correlation. Moreover, the current position of agents and target formation position are already known, so the enumeration method can be used to find the position in the target formation of all the agents in the current formation.

4.1 Collision Avoidance Strategy It is assumed that the agents move in a straight line from the current formation position F to the target formation position Fg during the formation switching, a movement path from the initial position to the target position can be planned. When ground agents move along a cross trajectory, a certain ground agent stops moving or slows down the speed to allow other ground agents to pass the cross trajectory. Thus, the main flow of the proposed multi-agent mutual collision avoidance mechanism is shown in Algorithm 1. Algorithm 1 Ground multi-agent mutual collision avoidance algorithm Input: Current location of land-based multi-agent system F, Target formation position Fg , speed v, safety spacing between ground agents d; Output: speed v i ; Begin

120

S. L. Zenkevich et al.

2 (1) Using expressions dig (t) = (r i − r ig )T · (r i − r ig ) to calculate the distance from the current position F to the target formation position dig (t); (2) if dig (t) = 0,

Using expressions di2j (t) = (r i − r j )T · (r i − r j ) to calculate the distance between ground agents di j (t); if di j (t) ≤ d, if dig (t) ≤ d jg (t), ground robot i waits, v i = 0; else ground robot j waits, v j = 0; else v i = v; else move to the target position Fg , the algorithm ends.

4.2 Formation Switching Strategy On the basis of using finite state machine theory to solve a set of intelligent logic control tasks in the literature [17], the finite state machines A = [U, Z , X, f, h] (see Fig. 2) are used in different obstacle environments to dynamically switch the formation of ground multi-agent system. Besides, the entire control system includes the system top layer, logic layer, tactic layer, and execution layer; wherein the system top layer establishes the environment map, and sends a command U = {u 0 , u 1 , u 2 , u 3 } to the logic layer to start, stop or control the formation switching, of which u 0 — wait, u 1 —movement, u 2 —transformation, u 3 —stop. Moreover, there is a prepared set of formations on the logical layer. When the layer receives the command from the system top layer, the state of the finite state machine is changed X = {x0 , x1 , x2 } (wherein x0 is the initial state, x1 is the motion state, x2 is the formation switching state), and the command Z = {z 0 , z 1 , z 2 , z 3 } (where in z 0 —ready, z 1 —go, z 2 —transform, z 3 —pause) and the coordinates of the points in the target formation are sent to the lower layer. In addition, the output of the tactic layer is the agent motion control signal generated according to the relevant control law, and the state execution is fed back to the logic layer. Besides, In addition, the executive layer consists primarily of

/z

0

u3/z3

u0 x0

x1

u1/z1

/z 2

u1 /

z1

z2

/z 3 u3 u2

u2 /

Fig. 2 State transition diagram

x2

Formation Control of Ground Multi-Agent System Using Quadcopter …

121

environment and agent models. The entire control process requires commands from the top of the system or from the operator with a certain time interval. Because the properties of the obstacles encountered by the ground multi-agent system during the movement are different, it is possible to keep the current formation moving all the time, or it may not continuously switch the formation. It should be emphasized here that each ground agent is assumed to be at rest in the state x0 ; when in the state x1 , the ground agent moves according to the current topological structure; when in the state x2 , the ground multi-agent formation switches from the current structure to another; f is a state transfer function, x(t + 1) = f (x(t), u(t)). Besides, h is the output function. In this paper, the output function of the Moore type is used, which is z(t + 1) = h(z(t), u(t)).

5 Experiment and Result Analysis In order to verify the correctness and operability of the formation switching control strategy, simulation experiments are carried out in the ROS environment. During the simulation experiments, the environment global map and four agents are imported into the ros_stage, and four types of formation structures are used according to the obstacles in the map. It should be emphasized that the environment global map provided in this paper is not a mosaic of pictures taken by a multi-rotor aircraft camera. In addition, the detailed environment map creation and map matching methods are not elaborated here. During simulation experiments, the ground agent moves at a linear velocity vi = 0.5 m/s and an angular velocity ωi = αi −θi , of which αi is line direction between the starting position in the current formation of the ground agent i and the target position in the target formation, and θi is the current direction of the ground agent i. The initial formation of the four ground agents is set to a straight line, and the simulation is performed according to the formation switching and collision avoidance methods of this paper. The results are shown in Fig. 3. In Fig. 3, (a) is the initial state; (c), (d), (e), (g), (i), (l) are the motion states, which maintains the topological structure of the current formation; (b), (f), (h), (j), (k) are formation switching states. According to the shape of the obstacle, the formation is switching from one topological structure to another in order to quickly pass the obstacle. It can be seen from Fig. 3b that the motion trajectory of the ground agents 2 and 3, 1 and 2, 3 and 4, 1 and 3, 1 and 4 intersect with each other. Moreover, Figs. 4 and 5 are graphs showing the results of the distance and the speed between the ground agents in the formation switching state b). The correctness and effectiveness of the avoiding mutual collision algorithm of the ground multi-agent system can be reflected from Figs. 4 and 5.

122 (a)

(g)

S. L. Zenkevich et al. (b)

(c)

(h)

(i)

(d)

(f)

(e)

(j)

(k)

(l)

Fig. 3 Simulation results

robot 3

robot 0

robot 1

v (m/s)

robot 2

Time (s)

Fig. 4 Results of ground multi-agents motion speed

6 Conclusion Multi-agent formation control is widely used in geological exploration, military reconnaissance, rescue, and transportation. During the execution of a task, formation control makes the team members move and maintain a certain topological structure toward the target. However, in an environment with disordered obstacles, it is difficult for a multi-agent system to maintain a certain formation to achieve a predetermined target, therefore it is necessary to change the formation as the environment changes. This paper proposes a ground multi-agent formation switching strategy in different obstacle environments. Moreover, the strategy uses the quadcopter machine vision

Formation Control of Ground Multi-Agent System Using Quadcopter …

Distance between robots (m)

d03

123

d13

d01

d23

d02

d12

Time (s)

Fig. 5 Distance results between ground multi-agents

system and the ground agent lidar sensor to establish a global environment map, thus providing a priori obstacle information for a ground multi-agent system. In addition, the ground agent’s odometer, GPS and quadcopter machine vision are used to estimate the coordinate of the ground agent in the absolute coordinate system, thus solving the positioning problem of a single ground agent. Finally, the finite state machine theory is used to control the switching of ground multi-agent formation, and the collision avoidance strategy of the multi-agent system when the motion trajectory is crossed is considered in the formation switching process. However, this paper does not consider the control of the formation switching and maintaining a variety of formations (such as line, column, triangle, and rhombus, etc.), which is the research direction that needs to be carried out later.

References 1. Mohan, Y., Ponnambalam, S.G.: An extensive review of research in swarm robotics. In: 2009 World Congress on Nature and Biologically Inspired Computing (NaBIC), Coimbatore, India, pp. 140–145 (2009) 2. Chaimowicz, L., Grocholsky, B., Keller, J.F., Kumar, V., Taylor, C.J.: Experiments in multirobot air-ground coordination. In: IEEE International Conference on Robotics and Automation, New Orleans, LA, USA, pp. 4053–4058 (2004) 3. Parker, L.E.: Multiple Mobile Robot Systems. In: Siciliano, B., Khatib, O. (eds.) Springer Handbook of Robotics, pp. 921–941. Springer, Berlin, Heidelberg (2008) 4. Zenkevich, S.L., Zhu, H.: Control of a group of mobile robots moving in the convoy type formation. Mekhatronika, Avtomatizatsiya, Upravlenie. 18(1), 30–34 (2017) 5. Darmanin, R.N., Bugeja, M.K.: A review on multi-robot systems categorised by application domain. In: 25th Mediterranean Conference on Control and Automation (MED), Valletta, Malta, pp. 701–706 (2017)

124

S. L. Zenkevich et al.

6. Olfati-Saber, R.: Flocking for multi-agent dynamic systems: algorithms and theory. IEEE Trans. Autom. Control. 51(3), 401–420 (2006) 7. Haque, M.A. Egerstedt, M.: Decentralized formation selection mechanisms inspired by foraging bottlenose dolphins. Presented at the Mathematical Theory of Networks and Systems, Blacksburg, Virginia, USA, pp. 1–12 (2008) 8. Saad, E.M., Awadalla, M.H., Hamdy, A.M., Ali, H.I.: Robot formations using local sensing and limited range vision. In: 25th National Radio Science Conference (NRSC), Tanta University, Egypt, pp. C15:1–8 (2008) 9. Das, A.K., Fierro, R., Kumar, V., Ostrowski, J.P., Spletzer, J., Taylor, C.J.: A vision-based formation control framework. IEEE Trans. Robot. Autom. 18(5), 813–825 (2002) 10. Chen, J., Sun, D., Yang, J., Chen, H.: Leader-follower formation control of multiple nonholonomic mobile robots incorporating a receding-horizon scheme. Int. J. Robot. Res. 29(6), 727–747 (2010) 11. Gómez, J.V., Lumbier, A., Garrido, S., Moreno, L.: Planning robot formations with fast marching square including uncertainty conditions. Robot. Auton. Syst. 61(2), 137–152 (2013) 12. Dai, Y., Kim, Y.G., Wee, S.G., Lee, D.H., Lee, S.G.: A switching formation strategy for obstacle avoidance of a multi-robot system based on robot priority model. ISA Trans. 56, 123–134 (2015) 13. Jaydev, P., Desai, A.: Graph theoretic approach for modeling mobile robot team formations. J. Robot. Syst. 19(11), 511–525 (2002) 14. Hong, Y., Hu, J., Gao., L.: Tracking control for multi-agent consensus with an active leader and variable topology. Automatica. 42(7), 1177–1182 (2006) 15. Hendrickx, J.M., Fidan, B., Yu, C., Anderson, B.D.O., Blondel, V.D.: Formation reorganization by primitive operations on directed graphs. IEEE Trans. Autom. Control. 53(4), 968–979 (2008) 16. Zenkevich, S.L., Nazarova, A.V.: Combining sensor systems data. Mekhatronika, Avtomatizatsiya, Upravlenie 10(9), 71–73 (2010) 17. Zenkevich, S.L., Nazarova A.V., Zhu, H.: Logical control a group of mobile robots. In: Gorodetskiy A., Tarasova I. (eds.) Smart Electromechanical Systems. Studies in Systems, Decision and Control, vol. 174, pp. 31–43 (2019)

Dependence of Dynamics of Multi-robot System on Control Architecture Vladimir Serebrenny and Madin Shereuzhev

Abstract Nowadays multi-robot systems on the way to find widespread usage in industry, agriculture and other fields of human activity. Currently, there is multiple control systems architecture for multi-robot systems. Researches in which was carried out the mathematical description of robots group control systems by analogy with the electromechanical systems are known. E.g., analogies with the motion of charged particles in an electric potential field containing target points, and obstacles to avoid. In this paper, this approach has been developed and generalized. The problems of distributed, centralized and hybrid control of multi-robot systems in the presence of previously unknown environmental factors and changing operating conditions are considered. The main goal of the work is the development of an adaptive control system for a group of robots performing the assigned task regardless of external factors affecting the behavior of an individual robot. Unlike previously published works, a network model with a dynamic structure represents a mathematical model of the distributed control system of the robot group. The control system of an individual robot, including a number of subsystems (information processing, navigation, platform control, the user interface) described as a local network of finite-state machines. Algorithms for control the system as a whole, as well as algorithms for the behavior of each individual robot represented. The dynamic organization of clusters within the group control system, including situations with quantitative composition changing of the robot group, and with the occurrence of unknown obstacles in the working environment are developed. The results of the research can find application in solving various tasks of mobile robot group control, including UAVs in the tasks of exploration and monitoring of the environment; robots for agricultural purposes in the performance of a wide range of fieldwork; in the performance of extreme operations, including rescue operations, taken out after accidents and disasters. Keywords Multiagent systems · Network dynamics · Multi-robot system

V. Serebrenny (B) · M. Shereuzhev Bauman Moscow State Technical University, Moscow, Russia e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_10

125

126

V. Serebrenny and M. Shereuzhev

1 Introduction At the most general level, we can divide approaches to the creation of multi-robot systems into two large groups: collective swarm systems and intentionally cooperative systems. In swarm systems, robots need minimal knowledge about other robots in a group to perform their own tasks. A large number of homogeneous robots characterizes swarm systems. Robots use local control laws to generate consistent team behavior while minimizing data exchange between robots. Robots in intentionally cooperative systems are aware of the presence of other robots in the environment and work together based on information about the status, actions, and capabilities of other robots in a group to achieve a common goal. Intentionally cooperative systems differ in the extent to which robots take into account the actions or conditions of other robots, i.e. may vary the degree of cooperation. A strong degree of cooperation requires that robots act in concert to achieve the goal, performing not decomposed tasks into separate independent actions of each robot and require constant awareness of the state of all parts of the MRS. Usually, these approaches require a certain type of communication and synchronization between robots. Weakly cooperative solutions allow robots to have periods of independence, after coordinating their choice of tasks or roles. Intentionally cooperative MRS can be heterogeneous—the components (robots are team members) differ in sensory and effector possibilities. In these teams, coordination of robots can be very different from a swarm approach, since robots are not interchangeable. The formalization of the interaction of heterogeneous elements of MRS as different agents is a multi-agent system [1, 2]. Based on the interaction between groups of elements or between the terminal elements of MRS, one can distinguish centralized, hierarchical, distributed methods of organizing the management of a multiagent system [3]. The overall control system architecture for groups of robots has a significant impact on the reliability and scalability of the system [4]. The architecture of the group control system must take into account the interaction of robots and how from the behavior of individual robots in a team synthesizes the group’s behavior [5]. In centralized systems, a single control center manages the processes of decomposing a global task and processing information received by agents. In decentralized systems, there is no such a center, during the interaction of agents tasks are distributed, and agents carry out the processing of the information received [6]. It is theoretically possible to use centralized control system architectures, where coordination of the entire group of robots comes from a single control center. However, such systems are practically not feasible due to the difficulties of transferring the entire system state to the control center, as well as the impossibility of controlling in real-time. Approaches are relevant when a centralized controller has the ability to monitor robots, and can easily transmit group messages for all robots. Hierarchical architectures are realistic for some applications. In this approach to control, each robot observes the actions of a relatively small group of other robots, each of which, in turn, controls another group of robots, etc. Up to the lowest level,

Dependence of Dynamics of Multi-robot System on Control …

127

which simply performs its part of the task. This architecture scales much better than centralized and is reminiscent of military command. The weakness of the hierarchical control architecture is recovery from failures of robots located in the control tree. Decentralized control architectures are most common for groups of robots and usually require robots to take actions based only on local knowledge about the situation. These control approaches can be very resilient to failure since no robot is responsible for controlling any other robot. However, achieving global consistency in these systems can be difficult, since high-level goals must be included in the local control of each robot. If changes in targets are possible, it may be difficult to reconsider the behavior of individual robots. A hybrid system is a combination of centralized and decentralized systems. The hybrid system has the advantages of centralized and decentralized systems and is devoid of most of their drawbacks. Hybrid control architectures combine local management with a higher-level management approach to achieve both reliability and the ability to influence the actions of the whole team with the help of global goals, plans, or controls [7].

2 Multi-robot System Dynamics Robots in groups can communicate in different ways: 1. Through the world [5]. 2. Passive action recognition [8]. 3. Explicit communication [7]. Each of these mechanisms for exchanging information between robots has its own advantages and disadvantages. For special-purpose robot groups, it is common to use a hybrid communication system, to avoid the unmasking position of robots. Robots in groups use short time intervals for data exchange and cognitive recognition for continuous monitoring. In some cases, large groups of robots have to be subdivided into smaller groups. In this case, the communication process between subgroups may be constrained due to interconnections between robots in the subgroup. There arises a problem of determining the subgroup structure enabling them to complete the task with maximum efficiency. As the group of robots network structure can only be optimal in each point of time in each individual case, the network should be dynamic. In this paper, we consider homogeneous subgroups of robots connected with an information channel, which means, that robots are united in a computing network. The structure of such a network may be presented as a graph with robots in the nodes and information channels on edges as it is shown in Fig. 1. Each robot use information exchange protocol to communicate with other devices in the network (group), generally TCP or UDP. In such a computation network on the local level robot can be considered as an independent unit, that should be able to process data from the network and its inertial navigation system, which allows

128

V. Serebrenny and M. Shereuzhev R2

R1

R3

Rn

Ri

Fig. 1 Structure of the decentralized robot group network. Ri —a robot

to robot determine position relative to another member of the group. Therefore, the robot control system consists of communication and computer vision systems, environment model processing unit, map, includes information about global and local flotation ability and route planning subsystem, actuating system and servos (Fig. 2). In the simplest case robot can be described by the following parameters: robot ID, i ∈ I ⊂ Z; state xi ∈ Xi ⊂ Rn , control signal ui ∈ Ui ⊂ Rn , fi : Xi × Ui ⇒ TXi , determine dynamics x˙i = f (xi , u i ). From another state the robot network dynamics can be termed by the equation [2]: x˙ i = F(xi , Hi1 (xi , x1 ), . . . , Hin (xi , xn )), i = 1, . . . , n;

(1)

n—amount of dynamic nodes, xi = xi (t)—state column-matrix of i node in the t point of time, F(x, y1 , …, yn )—generalized nonlinear function, that determine mutual effects using intermediary function yj = Hij (xi , xj ), where yj = 0 means no interaction (e.g. disconnect). Thus, the robot system architecture implies pairwise interaction between robots. Statement of the problem that the robot group management system

Fig. 2 Robot control system

Dependence of Dynamics of Multi-robot System on Control … Fig. 3 Task decomposition example

129

W1

T1

T2

Wi-1

Ti Wi+1

W2

Wi T2

Tn

should solve: “Which structures of multi-robot systems, described as a network optimize the dynamics of the multi-robot system for a given function and what are the properties of such networks of robots?” Multi-robot task planning includes two aspects: task decomposition and task allocation. Task decomposition mainly refers to how the group mission to be completed. One complex task needs to be decomposed into several simple single subtasks (Fig. 3). Any robot can be assigned to perform any task. To achieve optimal task distribution there is a need for a robot group to perform the negotiating the process. Mathematically, a computational network formation process can be described in the following way. Each robot Ri , i = 1, …, n can perform any set of actions Ai = {A1i , A2i , …, Ami }. On the other hand, to solve any target task Tm in conditions fm = {fm1 , fm2 , …, fmn } robot should perform determined set of actions Tm = {T1m , T2m , …, Tqm }, at that each of them is characterized by a number of properties, e.g. type, capability, intensity, and others. Moreover, many another constraints can be imposed on target task Tm solving— Qm = {J(m)}—efficiency criteria, t(m)—time taken to target destination, n(m)—the number of robots in the group…}, related to the process of the target destination. The task of group organization may be to align nodes of the graph-model of the robot group to the nodes of graph-model of the target task in a way that to decrease the number of robots, involved in task performance. During solving of this problem, nodes of the graph can be united according to the following rules. If a task can’t be performed by a single robot, then Tvm ∈ {A1i (m), A2i (m), . . . , Ami (m)}, v = 1, . . . , qm

(2)

where n(m)—the number of robots in the group, in which every robot sets of contains actions all actions needed for target task Tm = {T1m , T2m , …, Tqm } to develop it in conditions fm . If every action Tqm can be done by robot Ri strength of the network Km , oriented on Tm target task performance, it will be equal to the number of actions Tvm . Generally, different robots can perform the same action Tvm with different efficiency. Let qiv (m)—Ri robot efficiency estimation, action on Tvm . Clearly, that it is expediently to include in the computational network Km , oriented on Tm target task performance, groups of robots that can perform a maximum of efficiency: J=

pm  v=1

qiv (m) → max

i∈[1...N ]

(3)

130

V. Serebrenny and M. Shereuzhev

where in the number of robots of computational network Km , oriented on Tm target task performance, need to be minimized: n(m) → min

(4)

Conditions (1)–(3) are in fact local rules of self-organization of computational network Km , oriented on Tm target task performance. On the other hand, computational network Km forms by observing these conditions. Robots reshape an initial model of the target task, changing the value of the nodes and edges, that define laboriousness and allowable time for performance, appropriating to nodes qiv (m) robot efficiency estimations. In this case (2) is the target function of computational network Km formation.

3 Experiments After task assignment, robot in-group needs to plan their action (moving in the environment, using effectors to complete tasks). It is necessary to determine the optimal structure of a group of robots and the optimal state of robots in the sense of the homogeneity of their states. If the task assigned to the robot is an internal disturbance, and the change in the environment is an external disturbance, then overcoming the external and internal disturbances for a group of robots as for a network means bringing the network to a homogeneous state of each node. In the case of a structural disturbance (e.g. the output of one robot or the loss of communication channel), it is necessary to convert the entire network into a new state with an optimal structure. For the class of systems described by Eq. (1), we consider network-homogeneous states given by x1 (t) = · · · = xn (t) = x ∗ (t)

(5)

where x satisfies the equation for an isolated node, xj F(x*, 0, …, 0). To encode the structure of the robot group there is a constant coupling matrix A = (A ij ). As  shown in work [2] for undirected networks with a fixed number of links m = i j>i Ai j optimization problem is minimi ze max (A) subject to Ai j ∈ {0, 1}, Ai j = 0, Ai j = A ji  Ai j = m i

j>i

(A)—is Lyapunov exponent measuring the stability of the network homogenous state [9].

Dependence of Dynamics of Multi-robot System on Control …

Multi – agent system

131

Multi – agent system

Fig. 4 Modeling centralized and federative control of the multi-robot system in Matlab

We decided to model the system with two types of architectures (Fig. 4) and identify the value of the Lyapunov exponent. The state of the robot in the simulation are linear and angular velocity and coordinate. To find the Lyapunov exponent, changes in the states of robots in groups were recorded in the form of time series. The simulation results show that the Lyapunov exponent has a negative sign. Thus, multi-agent systems with centralized and federated architecture are not chaotic.

4 Conclusion The paper adopted the first attempt to determine the dependence of the dynamics of a group of robots represented as a network depending on the control system architecture. An approach to modeling the system and identifying such properties as the randomness of the system is presented. In the future, we plan to examine a wide range of currently known architectures of control systems of a group of robots and multi-agent systems.

References 1. Vorotnikov, S., Ermishin, K., Nazarova, A., Yuschenko, A.: Multi-agent robotic systems in collaborative robotics. In: Ronzhin, A., Rigoll, G., Meshcheryakov, R. (eds.) Interactive Collaborative Robotics. ICR 2018. Lecture Notes in Computer Science, vol. 11097. Springer, Cham (2018) 2. Simmons, R., Singh, S., Hershberger, D., Ramos, J., Smith, T.: First results in the coordination of heterogeneous robots for large-scale assembly. In: Proceedings of 7th International Symposium on Experimental Robotics (ISER) (2000) 3. Minin, A.A., Nazarova, A.V., Ryzhova, T.P.: Distribution of tasks in a decentralized robotic system. Mech. Autom. Control. 11 (2014)

132

V. Serebrenny and M. Shereuzhev

4. Woolridge, M.: Intelligent agents. In: Multiagent Systems. A Modern Approach to Distributed Artificial Intelligence, pp. 27–78. Massachusetts Institute of Technology (1999) 5. Matari´c, M.J.: Issues and approaches in the design of collective autonomous agents. Robot. Auton. Syst. 16, 321–331 (1995) 6. Alili, S., Alami, R., Montreuil, V.: A task planner for an autonomous social robot. In: Asama, H., Kurokawa, H., Ota, J., Sekiyama, K. (eds.) Distributed Autonomous Robotic Systems 8. Springer, Berlin, Heidelberg (2009) 7. Parker, L.E.: ALLIANCE: an architecture for faulttolerant multi-robot cooperation. IEEE Trans. Robot. Autom. 14(2), 220–240 (1998) 8. Huber, M.J., Durfee, E.: Deciding when to commit to action during observation-based coordination. In: Proceedings of 1st International Conference Multi-agent Systems, pp. 163–170 (1995) 9. Nishikawa, T., Sun, J., Motter, A.E.: Sensitive dependence of optimal network dynamics on network structure. Phys. Rev. X 7, 041044 (2017)

The Application of Multi-agent Robotic Systems for Earthquake Rescue Anaid V. Nazarova and Meixin Zhai

Abstract Earthquake is one of the most serious catastrophes that cause a fatal threat to people and animals among all of the disasters. In order to rescue the victims as soon as possible, this paper proposed to apply the multi-agent robotic systems to the search and rescue operation to improve the rescue efficiency. Firstly, the rescue operations and the sequence of the rescue process are analyzed according to statistical data. Secondly, the search theory is researched based on probability theory. Thirdly, the intelligent optimization algorithm is used to ensure that agents complete the tasks more efficiently. At last, the collision avoidance trajectories of agents are planned in ROS. The development and application of rescue robotic systems are very important to modern society, especially involved in natural or man-made disasters because of their unpredictability and the possibility of large human losses. This paper provides a good model to solve the emergency with the help of multi-agents. Keywords Multi-agent system · Search theory · Probability theory · Intelligent optimization algorithm · Earthquake · Collision avoidance trajectory

1 Introduction At present, multi-agent robotic systems are widely used in industries that may lead to a risk to humans, such as in the chemical and nuclear industries, or in man-made, natural disasters and so on [1]. The advantages of using multi-agent robotic systems are obvious. Firstly, it extends the range actions; secondly, it expands the set of performed tasks; thirdly, it has a higher probability of successful completion of the task, achieved due to the possibility of redistributing tasks between the agent-robots when the agent-robots enter and exit the system [2]. Earthquake is one of the most serious A. V. Nazarova (B) · M. Zhai Department of Robotic Systems and Mechatronics, Bauman Moscow State Technical University, Izmaylovskaya sq., 7, Moscow 105037, Russian Federation e-mail: [email protected] M. Zhai e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_11

133

134

A. V. Nazarova and M. Zhai

catastrophes that poses a threat to people and animals. In addition, they often cause secondary disasters, such as the destruction of buildings, fires, and floods, which lead to additional disasters. The inability and inefficiency of rescue operations bring immeasurable losses to humans. Therefore, it is necessary to use groups of interacting agent-robots in the case of disasters. The solution to this problem is aimed at the practical implementation of multi-agent systems, which will significantly reduce the risk to human life and health when working in adverse or dangerous conditions. When applying multi-agent systems to complete rescue task, the following problems occur: design the structure of a multi-agent control system; which method and algorithm should apply to search victims; how to distribute tasks among agents to improve the rescue efficiency and choose the method of planning collision avoidance trajectory. Currently, there are three types of control systems of multi-agents—centralized, decentralized and hybrid [3–6]. A hybrid system is a combination of centralized and decentralized systems. Here the team is managed by exchanging information between the agents, and the data is processed by the managing center. The advantages of hybrid systems are the possibility of self-organization, reliability of work and low consumption of computing resources of agents. Coverage and search of multi-robots are being developed in a variety of areas: planning the trajectory of vacuum cleaners robots, equipment for plowing the land, harvesting, mowing lawns, painting and cleaning objects, as well as extinguishing fires and search operations performed as in the air, in the water and on the ground (example: after an earthquake) [7]. There are many algorithms for coverage and search problems, Balch and Arki [8, 9] proposed a method of stochastic search in an unknown area, which increases the efficiency by the dispersibility of robots in the search zone. However, this passive search method cannot avoid repetitions, given to lower search speed. Parker [10] has created a new distributed cooperative multi-agent model to improve search speed. This method is a fully distributed search method. Solanas [11], using the K-Means algorithm, divided the search area into several parts, and each robot performs the search tasks in its own zone. Knowing the net map, Moravec and Elfes [12] proposed a cell decomposition model that would evenly divide the search area into cells with the same size and shape. Gabriely and Rimon [13] performed the task of searching along a spanning tree, which divided the work area into discrete cells and the robot can cover each point only once. With regard to the partitioning of the search area, Latombe and Choset [14, 15] proposed a method of trapezoidal decomposition, but this method only fits into a flat polygonal space. As was shown above, the algorithms of coverage and search were discussed by many scientists [16], and they were especially keenly interested in the question of trajectory planning. However, the disadvantage of the existing works is that they only explained which trajectories the robots should move to cover the entire search area, and the issues of optimal speeds, probability of finding targets, the characteristics of the sensors and the search area, were not well covered. Such methods are not very suitable for solving real rescue tasks, especially such complex tasks as an earthquake, fire, etc.

The Application of Multi-agent Robotic Systems …

135

When involved in problems of task assignment, there are many methods are discussed. For centralized task assignments such as linear programming, swarm intelligence [17] and others are used. For distributed task assignment, game theory, Markov solutions, theory auction [18] are applied. The hybrid type is a combination of the two types mentioned above. The rest of this paper is organized as follows. Section 2 is the problem statement of using multi-agent in search and rescue operations. Section 3 gives details on a solution to the problems during the search and rescue operation by multi-agent. Finally, the fourth section presents a discussion of the conclusions and future work.

2 Problem Statement In order to use the multi-agent system in earthquakes, first, we have to analyze the emergencies. According to the real situation after the earthquake, the fundamental purposes of rescue operation are: analyze the situation of the disaster and create a map of the affected area; search for survivors and mark them on the map; find out the states of survivors, and plan the rescue routes and rescue them. According to the analysis above, the rescue system is created in Fig. 1. In the figure, the tethered drones are designed to create an air communication station. The search robots complete the task of search the survivors. The mobile robots provide emergency rescue. Therefore, our problems come down to

Tethered drones Search drones

Affected area Mobile robots

Control center

Fig. 1 The diagram of the rescue multi-robot system

136

A. V. Nazarova and M. Zhai

1. Analyze the rescue operations in earthquakes and plan the procedure of search and rescue operations. 2. Choose the principle of the communication system among agents. 3. Analyze the search problem based on theory probability. 4. Analyze the task assignment problem. 5. Solve the problem of motion planning of robots. Next, we will carefully study the above questions.

3 The Research of the Search and Rescue Problems 3.1 Analyze the Rescue Operations in Earthquakes In this part, we will plan the process of rescue operations. After finding out the condition of the affected area, it is necessary to detect the survivors as soon as possible and determine their condition, provide emergency assistance, such as medicines, water, food, and other necessary things, and transport them to a safe place. In accordance with this, we propose the following plan for the rescue process using unmanned aerial vehicles (UAV) and unmanned ground vehicles (UGV) of various “specializations” in Figs. 2 and 3. From the diagram, we can see that task 1 related to the investigation of disaster zones must be solved immediately. Tasks 2 and 3 begin to execute no later than two hours after receiving information from tasks 1. Tasks 4, 5 and 6 start after tasks 2 and 3. The effective execution time for tasks 3, 4, 5 and 6 is 72 h; however, the UGV and UAV will continue to work after 72 h. Tasks Number

Title

1

Disaster zone of inves ga on

2

Crea on of air communica on sta ons

3

Vic m detec on

4

Finding out the state of survivor

5

Emergency medical service

6

Delivery of vic ms to a safe area

Time a er the disaster 1ч 2ч 3ч 4ч 5ч 6ч

... 12ч 13ч ... 24ч 25ч ... 36ч 37ч ... 48ч 49ч ... 60ч 61ч ... 74ч 75ч ... 84ч 85ч ... 96ч 97ч ...

Fig. 2 The timing diagram of the rescue process

The Application of Multi-agent Robotic Systems …

137

Fig. 3 Search and rescue process

3.2 The Communication System of Search and Rescue System When it comes to rescuing, in the first place, should solve the problem of communication between agents, which includes global communication and local communication. As mentioned above, communication in the robotic system is one of the most important problems for successful realization motion the function of the system in real-time, which should ensure global data exchange, as well as local negotiations between agents, the implementation of tasks re-distribution, etc. In Fig. 4 shows the existing methods of communication. Since everything is destroyed in the affected area and the environment is uncertain and complex after the earthquake, therefore, it is necessary to create air communication stations to provide global communications. In addition, the environment in the destroyed zone are unstructured and complicated; as a result, mobile rescue robots may be lost. Therefore, it is necessary to create a ground-based local distributed communication that ensures reliable communication between robots on the ground. Wire communication Infrastructure based network Wireless communication

GSM, 3G,4G,WIFI... Ad Hoc

Infrastructureless network

Wireless Sensor Network Wireless Mesh Network

Fig. 4 Existing communication methods

138

A. V. Nazarova and M. Zhai

WIFI

WIFI

Access Point

WIFI

Access Point Access Point

network 1 network 3 network 2 Fig. 5 Multi-network combination architecture

In accordance with this, the architecture of multi-network combinations is proposed, shown in Fig. 5. Global communication is performed by tethered UAVs, which form an air communication station to cover the entire area with the wireless network; local communication will be performed by distributed networks—Ad hoc; the connection between Ad hoc network and the global wireless network through the gateway (access point).

3.3 Analyze the Search Process Based on Theory Probability In this part, we will study the search problem using probability theory. Firstly, we discuss different types of sensors for detecting survivors, such as acoustic, radar sounding, etc. Each one has its own advantages. However, for earthquakes, the radar detector and infrared sensor are more suitable due to their detection range. Based on these sensors, the detection models are created. As well as the search process by one agent in a discrete and continuous situation is investigated. After that, a number of collaborative search problems are solved. In the beginning, we created the search model and formulated the problem. Two probabilistic models are like this: (1) Map model: there is a map L x × L y , a priori information about the density˜of the probability distribution of the target in the map ρ(x, y), the total probability: ρ(x, y)d xd y ≤ 1; we discretize the continuous map M N into grid map M × N , then: m=1 n=1 c(m, n) ≤ 1. (2) Sensor model: there are two types of sensors observations: discrete observation and continuous observation. In a discrete search, the probability of detecting a target, in which the target is in the

The Application of Multi-agent Robotic Systems …

139 α(zk|dk)·△t

Fig. 6 Conditional probability

1 PD

PF

dk,m

0

din

dout

field of view of the sensor is—s or conditional probability. In the case of continuous search, each scan takes a small time t, while the conditional probability of target detection depends on the type of sensor and the average distance between the sensors and the target, s = αt, [α] = 1/s. ⎧ ⎪ ⎨ PD α(zk |dk )t = P D − ⎪ ⎩P F

(P D −PF )(dk −din ) dout −din

(dk < din ) (dout ≤ dk ≤ din ) (dk > dout )

(1)

where s = α(z k |dk )·t = λ(dk ) (Fig. 6) (discrete search also has the same function), where the observation zk ∈ {1, 0}, 1 means that the target is detected, and 0— not detected; dk —distance between sensors and targets. PD —probability of target detection by the sensor, PD ∈ [0, 1]; PF —probability of false alarm of sensor, PF ∈ [0, 1]; din —the largest distance from which the sensor observes the target with PD ; dout —the shortest distance from which the sensor cannot detect the target, but reports that it exists with a probability PF (false alarm of the sensor). Let T be the time allowed to search. PT is the probability of detecting a target in T. Our task is: based on the knowledge PT , to analyze the parameters and the search procedure. Let the agent scan K cells, one of which may have a target, during discrete examresearch, cell i is viewed ination each cell is viewed n i times.  KDuring continuous K n i = n, i=1 ti = T . Then we obtain the during the time ti (Fig. 7), where i=1 probability of finding a target: ⎧ K    ⎪ ⎪ ci 1 − (1 − s)ni ⎨ PKn = i=1

K    ⎪ ⎪ ⎩ PKT = ci 1 − e−αti

(2)

i=1

K where PKn is the probability of target detection after n = i=1 n i views, PKT is the K probability of target detection in time i=1 ti = T . k is the cell number; ci is the probability of finding a target in the i-th cell; s is the conditional probability of target

140

A. V. Nazarova and M. Zhai

(а)

(b)

C1

C2

C3

n1

n2

n3

t1

t2

t3

C1

C2

C3

...

...

Ci

C K-1

ni

ti

...

CK

n k-1

...

Ci

x

nK

tK-1

tK

CK-1

CK

x

Fig. 7 Target detection model: a in a discrete situation, b in a continuous situation

detection, when the target is in the cell, or s = αt, [α] = 1/s with continuous observation. The problem is that for a given observation time T find the observation time of each cell ti , i = 1, 2, …, K, maximizing the probability of detecting PKT under the K t . Using the Lagrange multiplier method leads to the following condition T = i=1 i c result: ti = KT − α1K Kj=1 ln cij , ci > 0, i = 1, 2, . . . , K . Next, let a group of agents explores the plane where a target is located. Then the following tasks occur: – how to choose the strategy of collaborative search; – how to divide the work area into sub-areas.

3.4 Analysis of Search Methods In this paper, four types of search methods are proposed (Fig. 8): 2K A (a) One agent inspects 2K cells, and each cell for the time tiA , i=1

ti = T . The  T  A 2K −αtiA ; probability of target detection is: P2K = i=1 ci 1 − e (b) Two agents simultaneously inspect 2K cells and each agent inspects their own cells for the time tiB and τ jB , respectively, Fig. 8b. Then the probability of target

(a)

(c)

(b)

agent1 t1

t2

t3

C1

C2

C3

agent2

...

tK

tK+1

CK

CK+1

...

t2K-1

t2K

C2K-1

C2K

agent1

x

agent2

t1

t2

t3

C1

C2

C3

t1

t2

t3

C1

C2

C3

...

tK

τK+1

CK

CK+1

tK

tK+1

CK

CK+1

...

τ2K-1 τ2K C2K-1

C2K

t2K-1

t2K

C2K-1

C2K

x

(d)

agent1

agent1

t1

t2

t3

C1

C2

C3

...

tK

tK+1

CK

CK+1

Fig. 8 The search method

...

t2K-1

t2K

C2K-1

C2K

x

agent2

...

...

x

The Application of Multi-agent Robotic Systems …

141

K B  T B detection for time i=1 ti = T , 2K τ jB = T has the form: (P2K ) = j=K+1

  K 2K −ατ jB −αtiB ; + j=K +1 c j e 1− i=1 ci e (c) Two agents inspect 2K cells, the first agent scans each cell with time tiC , after T /2 the second one starts to inspect the cell with time τ jC , shown in Fig. 8c. Then 2K C 2K C the probability of detecting a target with time i=1 ti = T /2, i=1 τi = T /2 2K C C T C has the form: (P2K ) = 1 − i=1 ci e− α(ti +τi ) ; (d) Two agents simultaneously inspect 2K cells and each agent inspects the cells with time t D and τ D , Fig. 8d. Then the probability of target detection in time 2K D i 2Kj D  D D t = T, τ = T has the form: (P T ) D = 1 − 2K ci e−α(ti +τi ) . i=1 i

j=1

j

2K

i=1

Let ci be symmetrically distributed, that is, cK = cK+1 , cK−1 = cK+2 . . ., comparing the four methods: (1) 0, because in the second situation this means that if the the search time is twice more; (2) search time is doubled, the probability of target detection is less than doubled. (3)  T (A)  T (B) P2K if the search parameters are con− P2K = 0, stant, then different search methods lead to the same results, and if the search time is the same, then the values of the detection probabilities are the same.

3.5 Method of Dividing the Search Area Analytically this problem is difficult to solve. Therefore, we simulate some specific situations. Let two agents perform the search task in T time, the number of cells is M × N . The first agent searches R × N cells, the remaining cells (M − R) × N are scanned by the second agent, see Fig. 9a. Our task is: to find R, maximizing the probability of target detection. The probability of finding a target in the cells—see Fig. 9b. The partitioning results are shown in Fig. 10a, b. The conclusion is made as follows: If the probabilities of finding a target in the first area and the second area are

(a) First area

Second area

N

0

(b)

R

Probability of finding targets in cells

N

M

Fig. 9 a Schematic diagram of the process of partitioning the region; b the probability of finding the target in the cells

142

A. V. Nazarova and M. Zhai (b)

The probability of detection target

The probability of detection target

(а)

The ratio of the probability of finding a target in the first area to the probability of finding a target in the second area

R

Fig. 10 a Probability of target detection depends on the ratio of the average probability of finding a target in the first area to the average probability of finding a target in the second area; b Probability of target detection depending on the position of R

the same or not very different from each other, then the probability of target detection reaches the maximum.

3.6 Task Allocation in Multi-agents Robotic System In this part, the problem of tasks distribution among agents is solved (after survivors are found, it is necessary to distribute targets (survivors) among rescue agents in order to rescue survivors faster), including global and local distribution. With the global distribution (centralized), particle swarm optimization (PSO) and the genetic algorithm (GA) are considered, for the local distribution (distributed) the auction algorithm and game theory are investigated. ➀ Global distribution The global distribution is carried out by the control center. Let M = {m1 , m2 , …, mk } be the set of tasks (may change in the process), N = {n1 , n2 , …, np } be the set of robots-agents (may also change). Matrices are given Q = {qij }, R = {rij } and L = {lij } with dimension k × p, where qij is the reward when performing the task i  {1, …, k} by agent j  {1, …, p}, rij —the cost when agent j performs the task i, and lij —the ability to perform task i by the robot j, lij = {0,1}. Our goal is to distribute    p k  tasks M among agents N to ensure the benefit C = j=1 i=1 lij ∗ qij − rij will be maximum. The core of PSO is: a flock of birds random disperses in space. Each bird is a certain distribution decision, knows its own distance to food (the optimal solution), as well as the distance to the food of all other birds in the flock. The process of algorithm is: (1) random initialization of a “swarm of particles”, i.e. agents are scattered randomly throughout the search area and each has a random speed and direction of movement. Let 5 robots (agents) with numbers 1, 2, 3, 4, 5, save 6 people. The coordinates of robots and humans are known, then, the costs are determined by the distances between robots and humans. (2) The calculation of costs for every bird. (3) Adjusting the position of the particle (bird) so that it does not go beyond

The Application of Multi-agent Robotic Systems …

143

the search space. υi = υi + a1 · r nd() · ( pbesti − xi ) + a2 · r nd() · (gbesti − xi ), xi = xi + υi . Where υi is the velocity of the current particle i; xi is the position of the current particle i; a1 and a2 are constants corresponding to a local and global solution. pbesti is the best point found by the particle; gbesti is the best point found by all particles; r nd() is a random number from 0 to 1; (4) The stop conditions (number of cycles): when the conditions are met, the search ends, otherwise—continue to cycle. For example, the sequence [2 3 5 1 4 2] is one way of distribution (particle position): it means that the second robot will perform the first task, the third robot will perform the second task, etc. If the velocity of a given particle is [1 0 1 1 1 2], then its new position is [3 3 4 2 5 4]. Studies have shown that PSO allows finding a better solution than GA with the same quantities of population and iteration. Therefore, in this connection, with a centralized solution of the task, we will use the PSO. ➁ Local distribution When in the process of performing tasks, if agents are broken or lost, then their tasks must be distributed among the other free agents through negotiations among themselves. The process of tasks distribution is realized by a negotiation model based on the exchange of information between individual agents. In this situation, without the presence of the control center, tasks are distributed among the agents using the game theory and auction algorithm. “busy”}, the vector of possibilities C ri =  states Ai : {“free”,  1Let2 the agent k r ci , ci , . . . , ci , whose elements ci ≥ 0, k = 1, 2, …, r corresponds to the necessary competencies to solve tasks. The task T j is related to the vector C rj , of dimension r:

C rj = c1j , c2j , . . . , crj , ckj ≥ 0, k = 1, 2, …, r, characterizing the necessary competencies to solve it. Our goal is to distribute tasks between agents to make the total costs be minimal. Game theory studies optimal strategies in games and the planning of the behavior of participants (players), in which two or more parties participate, leading the struggle for the realization of their interests. When using game theory to distribute tasks: players are robots-agents or tasks; the way of task allocation among agents— strategies; payment matrices—the costs of agents when performing any task. Our goal is to achieve the optimal strategy, i.e. “Nash equilibrium”, which means that the player  alwayschooses the best, from the point of utility, minimizing costs U∗ : U∗ u ∗1 , u ∗2 , u ∗3 ≤ U(u 1 , u 2 , u 3 ). The solution process is as follows: step 1: find pure strategies for solving each task; step 2: choose the best strategy for all tasks among the pure strategies obtained in step 1; Step 3: Check the strategies are obtained in step 2 so that there is no conflict. If there is a conflict within the strategies, then the negotiations will appear among the agents. The auction algorithm is fulfilled based on the exchange of information between individual agents. The process of auction is Step 1: the local auction center reports the information of tasks to every agent. Step 2: agents send their information to the center—state (free/busy) and current coordinates.

144

A. V. Nazarova and M. Zhai

Step 3: the local auction center starts processing auction: (a) forming the cost matrix—the distances between the robots and the tasks; (b) sorts all tasks in order of increasing needs; (c) starts the auction from the first task. Step 4: The local auction center reports the results to the agents, and the agents begin performing the tasks. The results of the redistribution of tasks based on game theory and the auction algorithm are shown that using game theory, we can always find the optimal solution, but using the auction algorithm gets a suboptimal solution. However, the process of games takes a lot of time, especially when there are many participants (agents). Let us distribute N tasks between M agents, then the number of sort by the auction algorithm is N + 1; the number of sort by the game theory: 

N, the set of pure strategy N + C N1 , the set of pure strategy and negotiations

From above, we see, in the case of local tasks distribution, use game theory to get the optimal solution, when the number of tasks is large, for exempla more than 5, then we suggest applying the auction algorithm.

3.7 The Movement Planning of Agents Further, after the distribution of the tasks, agents begin to perform their tasks, that is, move toward goals. The ROS is used for computer simulation to plan the trajectories of the agents when performing the search and rescue operations. The process of planning the optimal trajectory of robots is realized by the extension ROS Navigation Stack. In our system, the trajectory is planned from two positions: global planning and local planning. Global planning is to find a path from the starting point to the goal using the A * algorithm. Local planning can avoid dynamic obstacles with the help of a dynamic window algorithm and the Rollout trajectory algorithm shown in Fig. 11.

Fig. 11 a Construction of global trajectories; b Local planning to avoid obstacles

The Application of Multi-agent Robotic Systems …

(a)

145

(b) /control center

①

/monitoring center1

/monitoring center2

/robot_0

/robot_N

/robot_1

/robot_2

/stage

...

/robot_i

...

② ③

④

Fig. 12 a Diagram of the task distribution process; b Process modeling at ROS/stage

The process of simulation is like this: (1) several mobile agents are located on the ground to perform the rescue tasks. (2) The control center distributes tasks among agents (particle swarm optimization). (3) Agents begin to perform their tasks. When agents cannot perform their tasks, they will redistribute their tasks among other free agents (game theory). See Fig. 12. In Fig. 12a level ➀ is the center of global distribution; level ➁—local distribution center or monitoring center; Level ➂—motion control of robots; Level ➃—model of the environment and robots (executive level). In Fig. 12b the red blocks are target points, black blocks are new target points, the other small blocks are robots.

4 Conclusion In this paper, the control system of multi-agents is applied to solve all of the problems in process search and rescue: firstly, analyze the rescue operations in earthquakes and formulate the sequence of rescue process with robots of various specializations based on the statistical data of earthquakes that have occurred in history. Secondly, choose the principle of the communication system between agents; the wireless network is selected for global communication and Ad hoc for the local communication. Thirdly, analyze the characteristics of the search process based on theory probability; we got the relationship between the probability of target detection, conditional probability, and the other parameters. Fourthly, the intelligent optimization algorithm is used to distribute the tasks among agents to make the process of task execution more efficiently. At last, solve the problem of motion planning of agents. The development and application of rescue robotic systems are very important and urgent to modern society, especially when eliminating natural or man-made disasters because of their unpredictability and the possibility of large human losses. This paper provides a good model to solve the emergency with the help of multi-agents.

146

A. V. Nazarova and M. Zhai

References 1. Yogesh, V.B.: An overview on search and rescue robots during earthquake and natural calamities. IJISET Int. J. Innov. Sci. Eng. Technol. 2(5) (2015) 2. Nazarova, A.V., Ryzhova, T.P.: Methods and algorithms for multi-agent control of robotic system. Vestnik BMSTU N.E. Bauman, Moscow, Russia, pp. 93–105 (2012) 3. Parker, L.E.: Multiple mobile robot systems. In: Siciliano, B., Khatib, O. (eds.) Springer Handbook of Robotics, pp. 921–941. Springer, Berlin, Heidelberg (2008) 4. Yurevich, I.E., et al.: Intellectual robots. M.: Mashinostroenie, 360 p. (2007) 5. Haque, M.A., Egerstedt, M.: Decentralized formation selection mechanisms inspired by foraging bottlenose dolphins. In: Presented at the Mathematical Theory of Networks and Systems, 2008, Blacksburg, Virginia, USA, pp. 1–12 (2008) 6. Bernussou, J., Sebe, N.: About decentralized feedback: a survey. In: Proceedings of the 9th IFAC/IFORS/IMACS/IFIP Symposium on Large Scale Systems: Theory and Applications, pp. 22–33 (2002) 7. Bagnitsky, A.V., Inzartsev, A.V.: Planning and implementation of the motion paths of an autonomous underwater robot performing monitoring in different types of waters. In: Underwater Research and Robotics, vol. 2, no. 22, pp. 25–35 (2016) 8. MacKenzie, D., Balch, T.: Making a clean sweep: behavior based vacuuming. In: AAAI Fall Symposium, Instationating Real-World Agents (1996) 9. Arkin, R.C., Balch, T., Nitz, E.: Communication of behavioral state in multi-agent retrieval tasks. In: IEEE International Conference on Robotics and Automation, vol. 1, pp. 678, Atlanta, GA (1993) 10. Parker, L.E.: An architecture for fault-tolerant multi-robot cooperation. IEEE Trans. Robot. Autom. 14(2), 220–240 (1998) 11. Solanas, A., Garcia, M.A.: Coordinated multi-robot exploration through unsupervised clustering of unknown space. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Sendai. Japan, pp. 717–721 (2004) 12. Moravec, H.P., Elfes, A.: High resolution maps from wide-angle sonar. In: IEEE International Conference Robotics Automation, St. Louis, MO, pp. 116–121 (1985) 13. Gabriely, Y., Rimon, E.: Spanning-tree based coverage of continuous areas by a mobile robot. Ann. Math. Artificial Intell. 31, 77–98 (2001) 14. Choset, H., Lynch, K., Hutchinson, S., Kantor, G., Burgard, W., Kavraki, L., Thrun, S.: Principles of Robot Motion: Theory Algorithms and Implementation. The MIT Press, Cambridge, MA (2005) 15. Latombe, J.C.: Robot Motion Planning. Kluwer Academic Publishers (1991) 16. Shehory, O., Kraus, S.: Methods for task allocation via agent coalition formation. Artif. Intell. 101, 165–200 (1998) 17. Meng, Y., Gan, J.: Self-adaptive distributed multi-task allocation in a multi-robot system. In: IEEE Congress on Evolutionary Computation, pp. 398–404 (2008) 18. Dias, M., Stentz, A.: Opportunistic optimization for market-based multi-robot control. In: International Conference on Intelligent Robots and Systems, vol. 3, pp. 2714–2720 (2002)

About Controlling the Shape of the Sprinkler Eugene Briskin, Yaroslav Kalinin, Konstantin Lepetukhin and Alexander Maloletov

Abstract In modern agriculture, various methods of land reclamation are used. Applied sprinklers provide processing of rectangular fields with rectilinear motion and round-shaped fields when using circular irrigation sprinklers. Therefore, it is important to develop methods to increase the irrigation of fertile soils, including through the use of sprinkler machines that can irrigate fields not only round and rectangular in shape, but also arbitrary. The task is to control the shape of a multisection sprinkler by controlling the speed of each support trolley in order to achieve the required configuration and change it in accordance with the field boundary profile. This is achieved by solving the problem of the controlled motion of the multi-tier multi-agent robot, which is such a sprinkler. As a result of solving the set task, for fields with a form that is close to circular, an increase in the irrigation area is achieved by 5–10%. Solving this task will allow increasing the area of irrigated soils only by changing the control system of the sprinkler. Keywords Sprinkler · Shape control · Increased efficiency · Increased area

1 Introduction In modern agriculture, various methods of land reclamation are used [1]. This and improvement of soil quality due to fertilization, rational crop rotation, as well as various methods of land reclamation [2, 3]. Irrigation is the main reclamation technique in the conditions of soils with high fertility achieved [4, 5]. Various irrigation E. Briskin (B) · Y. Kalinin · K. Lepetukhin · A. Maloletov Volgograd State Technical University, Volgograd, Russian Federation e-mail: [email protected] A. Maloletov e-mail: [email protected] E. Briskin · Y. Kalinin · A. Maloletov Center for Technology Components of Robotics and Mechatronics, Innopolis University, Innopolis, Russia © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_12

147

148

E. Briskin et al.

options are possible: general sprinkler irrigation, drip irrigation, and others. However, for large irrigated areas in the conditions of open ground, the main type is still general sprinkler irrigation using specialized sprinkler (irrigation) machines. Used in modern agriculture sprinkler machines provide processing of rectangular fields in the rectilinear movement of the machine and the fields of circular shape when using circular irrigation sprinklers [6, 7]. However, real plots of fertile land are rarely plots of regular round or rectangular shapes [8]. Therefore, it is important to develop methods to increase the irrigation of fertile soils, including through the use of sprinkler machines that can irrigate fields not only round and rectangular in shape, but also arbitrary [9–11]. Figure 1 shows real photographs of irrigated fields on which circular irrigation machines are used. It can be seen that significant areas of potentially fertile land are being lost. Known solutions aimed at increasing the irrigated area are based on the rational distribution of the centers of such machines, the choice of their dimensions and the processing of well-defined sectors (Fig. 2) or on changing the shape of the sprinkler caused by turning its farms relative to each other (Fig. 3).

1

2

Fig. 1 Forms of circular fields: 1—irrigated areas, 2—non-irrigated (lost) areas

Fig. 2 Modified forms of irrigated fields: 1—irrigated area, 2—a non-irrigated area 1 2

About Controlling the Shape of the Sprinkler

149

Fig. 3 Sprinkler with a changeable configuration

However, in the latter case, the support cart must be able to rotate relative to the truss for the subsequent joint movement of all the farms of the sprinkler as a single solid body. In this case, the number of drives increases and the design of the support carriage and control system becomes more complicated. Therefore, the actual may be the task of changing the shape of the machine without increasing the number of drives and complicating the control system. Such a task, due to the need for coordinated management of the group of supporting traction carts, is a type of the task of group management of robots, to which increased attention has recently been paid, and its peculiarity is an obvious practical significance.

2 Methodology The system of 2N bodies pivotally connected to each other is used as the calculated kinematic scheme (Fig. 4). A directional mechanical system consisting of N-shaped and N-bearing carriages and perfect flat movements imposes 4N-geometric connections on N-shaped hinges and N-linear kinematic connections. These connections do not prevent the change of the shape of the sprinkler. Therefore, to ensure its movement is enough N drives, and it drives the course movement. Indeed, in the absence of at least one of such drives, the propellers of the corresponding support car interacting with the ground do not implement traction and are a break, which is unacceptable. However, in the general case, in the hinges connecting sprinkler’s truss and the support carriage, controlled drives of their rotation or brakes can be installed to prevent such a turn. The form of the irrigated field is considered in the form of sectors (Fig. 4), and as an example, a sprinkler is shown, consisting of five articulated farms. In this regard, the laws of motion are determined in three stages. As an example, the following algorithm can be considered:

150

E. Briskin et al.

Fig. 4 The design scheme of the circular sprinkler: 1—central fixed support, 2—one of the farms of the sprinkler, pivotally (4) connected to the neighboring ones, 3—supporting traction cart, pivotally connected to the farm of the sprinkler (5), 6—edge of the field

– the first stage corresponds to the curvature of the shape of the sprinkler due to the successive rotation of the sections relative to different axes and consists of 4 sub-steps (in general, if the machine consists of N farms from (N − 1) sub-steps). Thus, in the first sub-step, the first section is fixed, and all the others, as one solid, rotate around the axis connecting it with the next, on each of the j sub-steps, all sections with numbers k > j remain stationary, j axis. The angles of rotation of the sections are determined in accordance with the established optimal configuration [12]; – at the second stage, the movement of the most distant support carriage occurs along the edge of the field. The machine has a constant configuration and rotates around a point O as a rigid body. In particular, this can be achieved by installing controlled brakes that prevent one section from turning relative to another; – the third stage corresponds to the restoration of the rectilinear form of the sprinkler. This can be achieved due to the rotational motion on each j substage of k = j sections, and the translational motion of the rest until the rectilinear configuration of the sprinkler is restored. The translational motion of the corresponding sections can be provided if the speed Vk (1) of each k carriage is determined by the expression.

Vk = ωr j cos γk

(1)

where ω is the angular velocity of the first j sections as a solid, rj is the distance from the fixed center O to the junction axis j and j + 1 farms of the sprinkler (approximately can be read to the installation axis j of the support carriage), γk is the angle between the longitudinal axes j and k carts. Figure 4 shows the corresponding values for j = 1. The solution to the problem is based on the hypothesis of self-orientation of carriages of mobile supports and self-installation of mobile supports.

About Controlling the Shape of the Sprinkler

151

Self-orientation is an effect expressed in the automatic orientation of carriages of movable supports perpendicular to the line connecting the fixed support with the axis of attachment of the cart of movable support to the sprinkler section. Self-setting is an effect that is expressed current radius depending movement of the movable support on the ratio of the speeds of movement of the other movable supports. In the considered movement, the axis of each support trolley should be oriented tangentially to the trajectory of movement. For steady motion along a circle, the axis of the cart is oriented perpendicular to the line connecting the center of the circle and the wheel of the cart. However, due to the fact that the size of the carriage is negligible compared to the distances to the axis rotation, we can assume that the axis of the cart is oriented perpendicular to the line connecting the center of the field and the node point of the section. In this case, the support carts should be located behind the section. The diagram (Fig. 5) shows that when the supporting carts deviate from the perpendicular to the line connecting the axis of rotation and the carriage, a lateral rolling resistance of the wheel arises, restoring the carriage perpendicular position when the carriage is in the back (Fig. 5a, b), and contributing to an increase in the deviation of the cart from the perpendicular otherwise (Fig. 5c, d). For clarity, Fig. 5 shows the carts on an enlarged scale.

Fig. 5 Scheme of self-orientation of the supporting carts: a, b the movement is stable, c, d the movement is unstable, ω is the angular speed of rotation of the sprinkler section relative to the fixed support, F 0 is the lateral rolling resistance of the wheel, V is the linear velocity of the cart

152

E. Briskin et al.

To fulfill the installation requirements for a driving propulsion unit behind the trolley attachment axis in the direction of travel, a generalized kinematic scheme has been developed in which any type of propulsion unit can be used, including wheeled or walking. Its difference is that the axis of attachment of the carriage is located not in the center, but is shifted to the front edge (Fig. 6). Figure 6a shows the construction of the sprinkler support with the use of wheel propulsion. Figure 6b shows the design of the support of the sprinkler with the use of a dual lambda-shaped walking mechanism. When the multi-section sprinkler rotates as a solid around the central support, the movable supports describe concentric circles, and the speed of each support is proportional to the distance from the support to the center. Thus, when changing the speeds of movement of the support carts, they will tend to take a position that the distance from them to the center of the field is proportional to their linear velocities. With such a movement, the axis of each supporting carriage should be oriented tangentially to the trajectory of movement. For steady motion on the circumference, the axis of the cart is oriented perpendicular to the line connecting the center of the circle and the wheel of the cart, and the radii of movement of each of the supporting carriages are proportional to the linear velocities (2). However, due to the fact that

Fig. 6 Variants of the design of the mobile support of the sprinkler: a with wheel propulsion; b with walking propulsion; 1—support beam, 2—axis, 3—side stops, 4—wheeled (a) or walking (b) propulsion, 5—hinges, to ensure the section turns relative to the trolley, 6—reinforcing elements, 7—pipeline, 8—hinges, providing rotation of the sections relative to each other, 9—elastic knee

About Controlling the Shape of the Sprinkler

153

Fig. 7 The design scheme of the cart

the size of the cart is negligible compared to the distances to the axis of rotation, we can assume that the axis of the cart is oriented perpendicular to the line connecting the center of the field and the node’s nodal point (Fig. 7). V V Vk = 1 = 2, Rk R1 R2

(2)

The sprinkler is considered as a system of solids connected by hinges. This system can be divided into N subsystems, each of which contains two solids—a section and a trolley (Fig. 7). Each of these solid bodies is associated with a moving coordinate system with the origin at the center of the mass of the corresponding body. The coordinates of the center of mass in the fixed frame of reference ξ η and the angle of rotation of the body ϕ are chosen as independent generalized coordinates. Section i interacts with the cart i at point Ai . We denote the interaction forces: Pi is the force with which the section acts on the trolley, P−i is the force with which the cart acts on the section. Section i interacts with section i − 1 at point Ai–1 and section i +1 at point Ai . Let Ri denote the force with which the i − 1 section acts on the i-section and through R−i+1 —the force with which the i +1 section acts on the i-section. For i = 1, the point A0 is the point of interaction with the support. For i = N, the next section is missing, so the force is R−N+1 = 0. The positions of the points Ai and Ai−1 are set in the moving reference system associated with the i section. In the general case, the attachment points of carts to sections and sections to each other can be different, and their coordinates in moving reference systems can take arbitrary values. However, in the well-known designs of the sprinkling machines, the attachment points of the sections and carriages practically coincide, and their coordinates along the axes yi are zero.

154

E. Briskin et al.

Then, taking into account the ratios Pi = −Pi− , Ri = −Ri− ,

(3)

differential equations of motion of a subsystem can be written as: m i ξ¨i = Ri,ξ − Ri+1,ξ − Pi,ξ , m i η¨ i = Ri,η − Ri+1,η − Pi,η , Ji ϕ¨i = −Ri,ξ xi,i−1 sin(ϕi ) + Ri,η xi,i−1 cos(ϕi ) + Ri+1,ξ xi,i sin(ϕi ) m i ξ¨i m i η¨ i Ji ϕ¨i

− Ri+1,η xi,i cos(ϕi ) + Pi,ξ xi,i sin(ϕi ) − Pi,η xi,i cos(ϕi ),     = Pi,ξ + F0i sin ϕi + FMi cos ϕi ,     = Pi,η − F0i cos ϕi + FMi sin ϕi ,       = −Pi,ξ xi sin ϕi − Pi,η xi cos ϕi − FMi y Bi − F0i x Bi ,

(4)

where mi , m i are the masses of the i-th section and carts; J i , Ji are the moments of inertia of the i-th section and cart; ξ i , ηi , ϕ i , ξi , ηi , ϕi are the coordinates of the center of mass and the angles of rotation of the i-th section and cart; x ij —the coordinate of the point Aj in the moving coordinate system associated with the i-th section, xi —the coordinate of the point Aj in the coordinate system associated with the i-th cart,   , y Bi —coordinates of the center of the i-th Bi wheel in the coordinate system x Bi associated with the i-th cart, F Mi , F Oi —the longitudinal and transverse forces of the interaction of the i-th wheel with supporting surface. To determine the forces of interaction of the wheel with the ground is used linear dependence of forces on relative slip values of the reference point wheels on the bearing surface: VBi,x − Vi , Vi VBi,y , F0i = −k0 Vi

FMi = −k M

(5)

where V i is the speed of the center of the wheel, which he would have when rolling without slippage; V Bi,x , V Bi,y —projections of the real speed of the wheel’s Bi center on longitudinal and transverse axes; k M , k 0 —coefficients depending on conditions coupling the wheel to the supporting surface in the longitudinal and transverse direction and from the normal reaction of the bearing surface to the wheel. Such the model cannot be used when braking the wheel or when extremely large values of relative slip. It gives acceptable results in the case of relatively slow, but not the intermittent movement, which is observed in sprinkling machines. For adhesion coefficient

About Controlling the Shape of the Sprinkler

155

estimates for the numerical simulation used data known from the theory of wheeled vehicles and experimental studies. To determine the forces of interaction between the bodies of the system are recorded equations:     Di,ξ = ξi + xi,i−1 cos(ϕi ) − ξi−1 + xi−1,i−1 cos(ϕi−1 ) ,     Di,η = ηi + xi,i−1 sin(ϕi ) − ηi−1 + xi−1,i−1 sin(ϕi−1 ) ,       = ξi + xi,i cos(ϕi ) − ξi + xi cos ϕi , Di,ξ       = ηi + xi,i sin(ϕi ) − ηi + xi sin ϕi , Di,η

(6)

which are algebraic equations and in general system differential equations are included in the form: D¨ i,ξ + 2a D˙ i,ξ + b2 Di,ξ = 0, D¨ i,η + 2a D˙ i,η + b2 Di,η = 0,       D¨ i,ξ + 2a D˙ i,ξ + b2 Di,ξ = 0, D¨ i,η + 2a D˙ i,η + b2 Di,η = 0,

(7)

where a and b are constant coefficients whose values are chosen experimentally to ensure the sustainability of a numerical solution.

3 Discussion: Research and Results The developed mathematical models were implemented as software. Were developed: a program for calculating energy consumption during the movement of individual sections of the sprinkler [13] and a program for modeling driving dynamics of a multi-section sprinkler. To test the effects of self-alignment and self-orientation, a laboratory model of a two-section wheeled drive sprinkler was created (Fig. 8). The control system of this model is based on the Arduino Mega 2560, with an optional driver installed for controlling drives (DC motors) and the Bluetooth module HC-05 for receiving

Fig. 8 Laboratory model

156

E. Briskin et al.

Fig. 9 Results of numerical modeling

commands from a computer or smartphone. The model is powered by two types of 18,560 rechargeable batteries. To control the model, commands are used that allow one to change the rotational speed of each of the wires individually or jointly, set predetermined motion parameters, change the direction of motion, stop motion. Experiments with a laboratory model (Fig. 8) and Numerical Simulation of the movement of a multi-section sprinkler (Fig. 9) confirmed the possibility of changing the angle between sections due to control the speed of movement of the mobile support of the sprinkler. As a result of solving the set task, for fields with a form that is close to circular, an increase in the irrigation area is achieved by 5–10%. Solving this task will allow increasing the area of irrigated soils only by changing the control system of the sprinkler.

4 Conclusion Methods of group control of the movement of the sprinkler sections to ensure its optimal configuration, consisting of regulating the speeds of the support carts in accordance with the established laws, are proposed. The mechanical effect of the self-orientation of the support cart of the sprinkler was established. Acknowledgements This work was supported by RSF (project No. 18-71-10069).

About Controlling the Shape of the Sprinkler

157

References 1. Pereira, L.: Higher performance through combined improvements in irrigation methods and scheduling: a discussion. Agric. Water Manag. 40, 153–169 (1999) 2. Varshney, R.S.: Modem methods of irrigation. GeoJournal 35, 59–63 (1995) 3. Singh, G.: Farm machinery. In: McNulty, P., Grace, P.M. (eds.) Agricultural Mechanization & Automation, Encyclopedia of Life Support Systems (EOLSS). EOLSS, Oxford (2002) 4. Sharma, S.K.: Principles and Practices of Irrigation Engineering. Oxford and IBH Publication Co., New York (1984) 5. James, L.G.: Principles of Farm Irrigation System Design. Wiley, New York (1988) 6. Merriam, J.L., Styles, S.W., Freeman, B.J.: Flexible irrigation systems: concept, design, and application. J. Irrig. Drain. Eng. 133(1), 2–11 (2007) 7. Keller, J., Bliesner, R.D.: Evaluation of Sprinkle Irrigation System, pp. 202–212. Van Nastand and Reinhold, New York (1990) 8. Zhiyu, Z., Zhihong, Q.I.: The Calculation of Sprinkler Irrigation Uniformity Based on Radial Basis Function Neural Network and the Optimization of Sprinkler Combination Spacing. Department of Water Resource of Agriculture University of Hebei, Baoding, China (2007) 9. McCarthy, A.C., Hancock, N.H., Raine, S.R.: Advanced process control of irrigation: the current state and an analysis to aid future development. Irrig. Sci. 31(3), 183–192 (2013) 10. Park, G., Lee, M.S., Kim, S.J.: Networking Model of Paddy Irrigation System Using Archyhydro GIS. ASAE Paper No. 052079. ASAE, St. Joseph (2005) 11. Hashim, S., Mahmood, S., Afzal, M., Azmat, M., Rehman, H.A.: Performance evaluation of hose-reel sprinkler irrigation system. Arab. J. Sci. Eng. 41(3923), 2016 (2016). https://doi.org/ 10.1007/s13369-015-1953-x 12. Briskin, E., Maloletov, A.: On optimal laws of groups of walking robots motion while solving formation task. In: Proceedings of the 8th ECCOMAS Thematic Conference on Multibody Dynamics, MBD 2017, pp. 601–606 (2017) 13. Maloletov, A.V., Kalinin, I.V., Lepetukhin, K.Yu., Barsov, V.S.: Calculation of Energy Efficient Turning Conditions Corps of the Walking Robot in a Plane Motion. Certificate state of the computer program registration number 2018661384 dated 6 Sept 2018 Russian Federation. VSTU (2018)

Space Robotics

In this part, the authors substantiate the scientific and methodological approaches to studying modern mechanisms of space robotic development and control. Breakthrough achievements of the last decade defined novel design imperative for space robotics. The authors determine the main features of space robotic design and the key control technologies. 1. P. P. Belonozhko—Modern Space Robotics. History, Trends, Outlook. 2. P. P. Belonozhko—Moving-Base Space Robots. Applying Eigen-Dynamics of a Reduced System to Synthesize Controls Modern Space Robotics. 3. I. V. Asharina—The Concept of Failure- And Fault-Tolerance for Distributed Control Systems of Spacecraft Groupings.

Modern Space Robotics History, Trends, Prospects Pavel P. Belonozhko

Abstract Today’s advancements in robotics are largely driven by two contradictory trends. On the one hand, humanity has been successfully operating specific space robotics in orbit, which experience, coupled with the unique design and high costs of such robotics, implies that the existing designs must evolve towards better functionality. The technological continuity being highly desirable is another argument in favor of such evolution. On the other hand, the sheer scale, complexity, and diversity of newly arising problems, solving which implies the use of robotics, make novel designs imperative. Research, development, and implementation of such designs will require not only a breakthrough in key robotics but also a long-term strategy in place. This paper analyzes the logic behind advancements in space robotics as observed in the operation of specific systems, first of all, shuttle handlers and ISS robotics. The literature review has identified the outstanding challenges of space robotics. Promising concepts are reported. The paper presents what the author believes to be the most feasible strategy for further advancement in assembly and maintenance robotics. Keywords Space robotics · Orbital robotic service · Assembly of large-size space objects · Assembly and service robotic space modules

1 Introduction To date, the experience of real operation has been accumulated mainly in relation to space robotics, which is an integral part of multifunctional manned systems [1–11]. First of all, it is necessary to name the reusable «Space Shuttle» transport system and the International Space Station (ISS). The logic of development implies the further improvement of space robotics as part of the objects of the promising space infrastructure, the prototype of which are manned space stations. At the same time, the analysis of promising projects [1–4, 7–10, 12–18] shows a shift in emphasis towards the use of Autonomous service robotics devices that do not P. P. Belonozhko (B) Bauman Moscow State Technical University, 5, 2-ya Baumanskaya str., Moscow 105005, Russia e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_13

161

162

P. P. Belonozhko

have currently operated analogs. It should be noted that the proposed Autonomous robotic means are more complex than actually working in orbit devices. It is especially necessary to allocate the expense of building a traditional concept of installation and service space of Autonomous robotic modules [1–3, 12–14, 19–23]. The problems of medical robotics in space, as well as anthropomorphic robotic crew assistants, are very relevant [24–26]. Thus, the further development of space robotics requires a unified long-term strategy and intensive progress in the field of key robotic technologies in space. We emphasize the relevance of research related to the creation of devices for robotic installation of large space structures [27–32].

2 Experience in Orbital Operation of Space Robotics Devices In [1–3], the importance of the approach to the development of advanced space robotics devices based on the improvement of well-functioning prototypes in the direction of expanding functionality was noted. The effectiveness of this approach is confirmed by the history of successful development of currently existing space robotics [4–10]. The manipulator “Canadarm” (Figs. 1 and 2) served as a prototype of the manipulator “Canadarm-2” (Fig. 4), which is the basis of the mobile service system, successfully operated today as part of the ISS. “Canadarm” represents the level of development of space robotics in the 1980s– 1990s and is one of the most effective and reliable devices of the “Space Shuttle” operated as part of the reusable transport ships” [1–10]. The Canadarm was first successfully tested in space during the STS-2 mission in November 1981. The last flight took place in July 2011 (STS-135 mission). Thus, the manipulator was successfully operated for 30 years.

Fig. 1 «Canadarm» [5] a construction elements b ground testing

Modern Space Robotics

163

Fig. 2 The first deployment in space of the manipulator «Canadarm». STS-2, November 1981, «Columbia» (foto NASA, [10])

Manipulator “Canadarm” is designed to move the payload from the cargo compartment to some point in the working area with the desired orientation, for example, when the satellite is put into orbit. Able to capture free object, place and lock him in the cargo hold. With the help of the manipulator “Canadarm” the astronauts working in the open space were supported, including their movement. Quite often the joint work of astronauts was used, one of which is fixed on the manipulator, and the second has the ability to move freely in the area of work. After the destruction of the Columbia Shuttle (STS-107 mission), during each of the subsequent Canadarm missions, together with the Orbiter Boom Sensor System (OBSS), which contains tools placed on the extension of the manipulator, was used for a thorough examination of the external surface of the Shuttle in order to detect possible damage to the heat shield. The workplace of the operator of the manipulator in the stern of the orbital ship is equipped with manual controls for rotation and translational movement of the cargo. The operators of the manipulator observe their actions both directly through the Windows and using the screens of the surveillance system located next to the controls, while one of the team members can assist as a camera operator. Video cameras are also attached to the manipulator (in the elbow and wrist part). The length of the manipulator “Canadarm” is about 15 m. There were produced five manipulators “Canadarm”. A typical example of the use of the manipulator “Canadarm” can be successfully implemented in May 2009, the mission of the Shuttle “Atlantis” STS-125 to service the telescope “Hubble”. The manipulator for the «Buran» spacecraft (Fig. 3), functionally and constructively similar to the «Canadarm» manipulator, was developed at the State scientific center—the Central research and development Institute of robotics and technical Cybernetics (SSC of the RTC of the Russian Federation) (St. Petersburg) [11]. The principle of expanding the functionality of the prototype can be illustrated by the example of improvements “Canadarm–2” compared to “Canadarm” (Fig. 4). The possibility of permanent stay of the manipulator in orbit is realized; the seventh degree of mobility is added and symmetry of the manipulator is provided—the working body can be fixed relative to the station and serve as a basis; the design of hinges

164

P. P. Belonozhko

Fig. 3 The manipulator of the spaceship «Buran» [11]

Fig. 4 «Canadarm–2» (SSRMS) i «Dextre» (SPDM) [9, 10]

is improved to increase the possible angle of rotation of adjacent links, force and moment sensors are added. Part of the mobile servicing system (ISS Mobile Servicing System MSS) service, in addition to the manipulator “Canadarm–2” also includes manipulator “Dextre” (Special Purpose Dextrous Manipulator—SPDM), mobile Transporter (mobile Transporter—MT), moving along the rails laid along the main farm of the ISS, mobile base system (Mobile Base System—MBS), installed on a mobile conveyor (Fig. 5). SPDM, which was put into operation on the ISS in 2008, is one of the most striking examples of “dexterous”-space robotics devices. Seredin et al. [16] describes the transport-handling system to service the space station and support the extravehicular activity of astronauts (RTC, TSNIIMASH). As noted in [16], the developed system for its intended purpose, operating conditions, and design are similar to SPDM.

Modern Space Robotics

165

Fig. 5 ISS Mobile Servicing System (MSS) [9]

3 Experimental Development of Space Robotics Devices At the commissioning of the manipulator “Canadarm” consecutive testing of its functionality was carried out. Contact interaction with the object of the known form placed in the cargo compartment, captured by the working body of the power type of devices for capture in conditions of limited visibility, a gradual increase in the mass of the moving goods was carried out. Due to the impossibility of full-scale ground experimental development, this approach is the most appropriate for the creation of new space robotics devices. It is fully implemented in relation to MSS within the ISS. Of particular note is the use of MSS and SPDM manipulator for debugging advanced robotic technologies in the framework of the Robotic Refueling Mission (RRM) [17, 18], the ultimate goal of which is to create service satellites that provide automated orbital refueling of client satellites, including those not initially adapted for service. A series of experiments on debugging the corresponding robotic technologies are based on the experience of ETS-VII and Orbital Express missions, as well as the experience of orbital servicing of the Hubble telescope [9]. The ETS-VII experiment was conducted in 1997–1999 [8, 19–21]. As noted in [8], it is considered the first demonstration flight for the purpose of verification of Autonomous rendezvous and docking technologies for robotic service in space. The operations of rendezvous and docking were practiced, both with the use of a docking system without the use of a manipulator installed on the base satellite, and by capturing the client satellite with the manipulator of the base satellite. Experiments were carried out on Autonomous replacement of blocks on the satellite-client, orbital Assembly, deployment of transformable elements. Worked out the mode of remote control from the ground point. The dynamics of controlled motion of the

166

P. P. Belonozhko

free—growing system “basic satellite—manipulator-satellite-client”, combining the movement of the movable base of the manipulator (basic satellite) with the process of manipulation, was studied experimentally. In 2007, a similar experiment was carried out by Orbital Express [8, 22, 23]. The service satellite ASTRO and the Next Sat client satellite were used for experimental testing of Autonomous rendezvous and docking, robotic replacement of units, inorbit refueling, and mutual maneuvering operations for visual inspection of the client satellite (Fig. 6). There are also a number of similar projects that have not been brought to the stage of the flight experiment. As an example, the concept of a service satellite equipped with robotic support, developed in the Central Research Institute of the RTC, described in [13, 14], can be given. To carry out the experiment, RRM was put into orbit and placed on the ISS truss structure a special platform with equipment placed on it and replaceable tools for the SPDM manipulator (Fig. 7). During the experiment, a number of technological operations were successfully carried out, in particular, the dismantling of

Fig. 6 Orbital express [9]

Fig. 7 Preparing for the experiment robotic refueling mission (RRM) [9]

Modern Space Robotics

167

heat-protective coatings, cutting of the wire fixing the protective cover, installation of robotic refueling elements, simulation of fuel pumping. The Restore-L program based on the experience of the Orbital Express and Robotic Refueling Mission experiments in [9] is described. It is assumed the satellite maintenance-clients with manipulators service satellite.

4 Assembly and Service Robotic Space Modules An important feature of space robotics is that certain work in orbit cannot be performed by man. For example, in [8] the following groups of problems are identified, the solution of which involves the use of space robotics: • • • • • • •

docking, berthing; refueling; repairing; upgrading; transporting; rescuing; orbital debris removal.

It can be concluded that there is a potential constructive variety of promising space robotics devices. The concept of installation and service robotic space modules equipped with manipulators should be highlighted [1–3, 12–14, 19–23]. For example, in [27] the orbital Assembly of a large space structure using a heterogeneous group of robots is considered (Fig. 8). An analog of such an Assembly can be the creation of a truss structure of the ISS.

Fig. 8 Orbital assembly of large-size space structure using a heterogeneous group of robots [27]

168

P. P. Belonozhko

Depending on the functionality of specific Autonomous robotic space modules, different strategies of installation and service operations can be implemented. Suppose the following stages of robotic installation of a large space structure of building blocks (for example, fragments of trusses): 1. 2. 3. 4. 5.

Delivery of blocks into orbit. Removing blocks from the transport compartment. Capture blocks of space robotic module. Transportation of blocks to the Assembly site. Connection of blocks to the mounted structure.

Various modes of operation of the Autonomous robotic space module can be distinguished: • controlled movement of the module without load; • docking of the module to the base station or mounted structure without the use of a manipulator; • docking of the module to the base station or mounted structure using a manipulator; • capture by the manipulator of the unit fixed relative to the base station; • the capture of the manipulator in inertial free space block; • controlled movement of the module with the load held by the manipulator; • managed the movement of goods by means of a manipulator; • controlled movement of the module with the load fixed on the base; • connection of the unit to the mounted structure. As an example of a combination of possible modes of operation of the installation and service robotic space module, the procedure described in [8] for capturing the target satellite by the service satellite can be given.

5 Conclusion The need for large-scale multifunctional stackable space systems is one of the factors that determine the prospects of space robotics. The following trends in the development of space robotics can be identified: – improvement of the tested prototypes in the direction of expansion of functionality; – development of an interrelated set of various space robotics tools; – coordinated design of serviced space objects and robotic systems. Commissioning devices for space robotics involves complex orbital testing that it is expedient to combine with the orbital operation. A good example of this approach is the robotic mobile service system of the International space station. The effective implementation of this approach could be the creation of a largesized modular uninhabited (occasionally visited) space object, the installation and maintenance of which would be carried out by autonomous robotics, which is its integral part.

Modern Space Robotics

169

In the future, it is advisable to use a heterogeneous set of assembly and service autonomous robotic space modules.

References 1. Belonozhko, P.P.: Space robotics: past experience and future considerations. Aerosp. Sphere 1, 84–93 (2018) (in Russian) 2. Belonozhko, P.P.: Space robotics. Current state, future challenges, development trends. Analytical review. Sci. Educ. Bauman MSTU 12, 110–153 (2016). https://doi.org/10.7463/1216. 0853919 (in Russian) 3. Belonozhko, P.P.: Advanced assembly and service robotic space modules. Robot. Tech. Cybern. 2, 18–23 (2015) (in Russian) 4. Thronson, H.A., Akin, D., Lester, J.: The evolution and primise of robotic in-space servising. In: AIAA SPACE 2009 Conference and Exposition, Pasadena, California, 14–17 Sept 2009 5. Kaupp, E., Bains, E., Flores, R., Jorgensen, G., Kuo, Y.M., White, H.: Shuttle Robotic Arm. Engineering Innovations, pp. 286–301. http://www.nasa.gov/centers/johnson/pdf/ 584734main_Wings-ch4h-pgs286–301.pdf. Accessed 09 Nov 2018 6. Stockman, B., Boyle, J., Bacon, J.: International Space Station Systems Engineering Case Study. http://spacese.spacegrant.org/uploads/images/ISS/ISS%20SE%20Case%20Study.pdf. Accessed 09 Nov 2018 7. Ower, C., Poynter, L., Keenan, A.: The role of dexterous robotics in ongoing maintenance of the ISS. In: 63rd International Astronautical Congress 2012 (IAC 2012), Naples, Italy, 1–5 Oct 2012, vol. 5 of 14 (2012) 8. Flores-Abad, A., Ma, O., Pham, K., Ulrich, S.: A review of space robotics technologies for on-orbit servicing. Prog. Aerosp. Sci. 68, 1–26 (2014) 9. NASA: NASA: Official Website. http://www.nasa.gov/. Accessed 09 Nov 2018 10. CSA: CSA: official website. http://www.asc-csa.gc.ca/eng/default.asp. Accessed 09 Nov 2018 11. Buran.ru. http://www.buran.ru. Accessed 09 Nov 2018 12. Lysyj, S.R.: Scientific and technical problems and prospects of development of robotics of special (space) purpose. In: Proceedings of the International Scientific and Technical Conference on Extreme Robotics, pp. 29–32. Publishing House «Polytechnic-service», St. Petersburg (2015) (in Russian) 13. Dalyayev, I.Y., Shardyko, I.V., Kuznetsova, E.M.: The technical configuration of robotic tools security service satellite, intended to extend the active lifetime of the spacecraft. In: Proceedings of the International Scientific and Technical Conference on Extreme Robotics. pp. 181–186. Publishing House «Polytechnic-service», St. Petersburg (2015) (in Russian) 14. Dalyayev, I.Y., Shardyko, I.V., Kuznetsova, E.M.: The prospect of creating robotic service satellites for maintenance and extension of the active life of spacecraft. Robot. Tech. Cybern. 3, 27–31 (2015) (in Russian) 15. Lonchakov, Y.V., Sivolap V.A., Sokhin I.G.: Ergonomic problems of creation and application of anthropomorphic robotic assistants of crews of perspective space missions. In: Proceedings of the International Scientific and Technical Conference on Extreme Robotics, , pp. 195–199. Publishing House «Polytechnic-service» , St. Petersburg (2015) (in Russian) 16. Seredin, S.V., Lysyy, S.R., Semenov, V.V., Abalikhin, O.Y., Emel’dyashcheva, O.V., Fomina, V.V., Kondrat’ev A.S., Gradovtsev, A.A., Konyshev, V.A.: Space robotic systems to support the activities of the crew of orbital alien modules. In: Proceedings of the International Scientific and Technical Conference on Extreme Robotics, pp. 199–202. Publishing House «Polytechnicservice», St. Petersburg (2015) (inRussian) 17. NASA’s Robotic Refueling Mission Practices New Satellite-Servicing Tasks//NASA. http:// www.nasa.gov/mission_pages/station/research/news/rrm_practice.html. Accessed 05 Nov 2018

170

P. P. Belonozhko

18. Robotic refueling mission (RRM). https://sspd.gsfc.nasa.gov/robotic_refueling_mission.html. Accessed 04 Nov 2018 19. Yoshida, K.: Space robot dynamics and control: To orbit, from orbit, and Future. In: Hollerbach, J.M., Koditschek, D.E. (eds.) Robotics Research. Springer, London, pp. 449–456 20. Yoshida, K.: Space robot dynamics and control: a historical perspective. J. Robot. Mechatron. 12(4), 402–410 (2000) 21. Yoshida, K., Hashizume, K.: Zero reaction maneuver: flight velication with ETS-VII space robot and extension to kinematically redundant Arm. In: Proceedings of the 2001 IEEE International Conference on Robotics and Automation, Seoul, Korea, 21–26 May (2001) 22. Mulder, T.A.: Orbital express autonomous rendezvous and capture flight operations. In: AIAA/AAS Astrodynamics. Specialist Conference and Exhibit, Honolulu, HI, 2008. AIAA Paper 2008–6768, pp. 1–22 (2008) 23. Friend, R.B.: Orbital express program summary and mission overview. In: Proceedings SPIE 6958, Sensors and Systems for Space Applications II, p. 695803 (2008) 24. Ushakov, I.B., Polyakov, A.V., Karpov, A.A., Usov, V.M.: Medical robotics as a new stage in the development of on-board simulators and biotechnical systems at the orbital station. In: Proceedings of the International Scientific and Technical Conference on Extreme Robotics, pp. 47–52. Publishing House «Polytechnic-service», St. Petersburg (2015) (in Russian) 25. Bogacheva, R.A., Zonov, A.A., Konyshev, V.A.: Prospects of using social robots for psychological and information support of cosmonauts in long-term space flights. In: Proceedings of the International Scientific and Technical Conference on Extreme Robotics, pp. 55–59. Publishing House «Polytechnic-service», St. Petersburg (2015) (in Russian) 26. R2 Robonaut. https://robonaut.jsc.nasa.gov/R2. Accessed 12 Nov 2018 27. Dubowsky, S., Boning, P.: Coordinated control of space robot teams for the On-orbit construction of large flexible space structures. Adv. Robot. 24(3), 303–332 (2012) 28. Artemenko, Y.N., Karpenko, A.P., Belonozhko, P.P.: Features of manipulator dynamics modeling into account a movable platform. In: Smart Electromechanical Systems. Studies in Systems, Decision and Control, vol. 49, pp. 177–190 (2016) 29. Artemenko, Y.N., Karpenko, A.P., Belonozhko, P.P.: Synthesis of control of hinged bodies relative motion ensuring move of orientable body to necessary absolute position. In: Smart Electromechanical Systems: The Central Nervous System. Studies in Systems, Decision and Control, vol. 95, pp. 231–239 (2017) 30. Belonozhko, P.P.: Methodical features of acquisition of independent dynamic equation of relative movement of one-degree of freedom manipulator on movable foundation as control object. In: Smart Electromechanical Systems: The Central Nervous System. Studies in Systems, Decision and Control, vol. 95, pp. 261–270 (2017) 31. Artemenko, Y.N., Karpenko, A.P., Belonozhko, P.P.: Synthesis of the program motion of a robotic space module acting as the element of an assembly and servicing system for emerging orbital facilities. In: Smart Electromechanical Systems: Group Interaction. Studies in Systems, Decision The literature and Control, vol. 174, pp. 217–227, Springer International Publishing, Switzerland, Cham (2019) 32. Belonozhko, P.P.: Robotic assembly and servicing space module peculiarities of dynamic study of given system. In: Smart Electromechanical Systems: Group Interaction. Studies in Systems, Decision and Control, vol. 174, pp. 287–296, Springer International Publishing Switzerland, Cham (2019)

Moving-Base Space Robots—Applying Eigen-Dynamics of a Reduced System to Synthesize Controls Pavel P. Belonozhko

Abstract A review of trends in space robotics identifies a promising class of assembly and service robotic space modules (ASRSM). This paper considers the topical problem of automating the assembly of large space objects. For certain ASRSM operations that are of practical interest, engineers can apply a design model comprising a moving base, a handler, and a payload. A massless single-DoF handler is an important special case. It is shown that in the absence of forces external to the system, the payload displacement with respect to the base can be described by an independent differential equation derived by Routhian formalism. In the common case where the angular momentum of the original system is non-zero, this equation is the dynamics equation for some non-linear oscillatory system, which, using the traditional terminology of analytical mechanics, we referred to as a reduced system and is seen as a controllable object. The proposed approach is convenient for studying the Eigen movement of a system by the internal degree of freedom, which movement is caused by the non-zero values both on the (neglected) cyclic axis, and on the hinged positional axis. Using these movements to synthesize the controls makes sense from the point of view of an important principle applicable to robotic movement arrangements; the principle consists in harmonizing the free and forced movements of the handler. The paper considers a modeling problem, which is of practical interest in its own right. The results can be generalized to apply to a general case of spatial motion of a system. Keywords Assembly and service robotic space modules · Proper inertial motion by the degree of freedom of manipulator · Reduced system · Routh equation · Phase portrait

P. P. Belonozhko (B) Bauman Moscow State Technical University, 5, 2-ya Baumanskaya str., Moscow 105005, Russia e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_14

171

172

P. P. Belonozhko

1 Introduction Ensuring the accord between free and forced motions of a manipulator is essential in managing robot movement. The vital task of automated assembly of large-sized space facilities is defined in [1–5]. In [6–9] the authors specified the possibility to develop distinctive modes for controlled motion of assembly and service robotic space modules characterized by absent external forces and moments in relation to the system. The plane motion of a system of two rigid bodies (moving platform and load) bound by a perfect single-pivot point ball-socket joint is reviewed as an example. It is through this system that the author reveals that, absent any external force and moments, the independent dynamic equation with respect to pivot point coordinates in Routh equation form may be associated with the reduced nonlinear oscillating system. The problem of using the proper inertial motions of the reduced system in the synthesis of control is considered.

2 Kinetic Energy of the System “Space Module—Massless Manipulator—Payload” For systems of this class is important is the ability to ignore the weight of a mass of the manipulation mechanism compared to the masses of the movable base—space module (SM) and payload (PL) emissions [8–10]. Assume that the external forces on the system “KM–manipulator–PG” does not apply. We introduce the following coordinate systems. XYZ—the non-rotating coordinate system with the origin at the center of mass of the system, in the absence of external forces, will be inertial. The values related to this coordinate system are indicated by the index “inr” (“inertial”). X b Y b Z b —a coordinate system with the origin at the center of mass SM and axes directed along the principal Central axis of inertia SM. The values related to this coordinate system will be distributed by the index “b” (“base”). X p Y p Z p —a coordinate system with the origin at the center of mass PL and axes directed along the main Central axis of inertia PL. The values related to this coordinate system will be distributed with the index “p” (payload). In (18) it is shown that the kinetic energy of motion of the “SM–manipulator–PL” system with respect to the coordinate system XYZ, taking into account the assumptions about the zero mass of the manipulative mechanism and the absence of external forces, can be represented as T =

1 bbT 1 bbT T bb inr b T inr b V · A1 · V pbb + V pbb · A2 · bb · A−1 , p +  p · A3 ·  p + L 0 ·L 2 p 2 (1)

Moving-Base Space Robots—Applying Eigen-Dynamics …

173

where V pbb —the matrix-column of the coordinates in the system X b Yb Z b of the vector of the linear speed of the center of mass PL (origin of X p Y p Z p –) relative to X b Yb Z b ; bb p —the matrix-column of the coordinates in the system X b Yb Z b of the vector of angular speed PL relative to X b Yb Z b ; L inrb —the matrix-column of the coordinates in the system X b Yb Z b ; of the vector of the kinetic moment of the system “SM–manipulator–PL”; L¯ inr relative to the origin of XYZ. Matrices A0 , A1 , A2 and A3 are determined by the relations ˆ bb A0 = θbb + θ pb − m˜ Rˆ bb p · Rp ,

(2)

−1 ˆ bb A1 = m(E ˜ + m˜ Rˆ bb p · A0 · R p ),

(3)

−1 b A2 = m˜ Rˆ bb p · A0 · θ p ,

(4)

b A3 = θ pb (E − A−1 0 · θ p ),

(5)

where θbb —matrix component of SM inertia tensor relative to its mass center in X b Yb Z b ; θ pb —matrix component of PL inertia tensor relative to its mass center in X b Yb Z b ; m m m˜ = m pp+mbb , m b —SM mass, m p —PL mass; Rˆ bb p —skew-symmetric matrix put in correspondence to matrix column of vector coordinates in X b Yb Z b of the radius vector of PL mass center (origin of X p Y p Z p ) relative to SM mass center (origin of X b Yb Z b ); E—a single 3 × 3 matrix. It follows from (1)–(5) that it is important to distinguish between cases L¯ inr = 0

(6)

L¯ inr = 0.

(7)

and

In the case of (7), the expression (1) takes the form T =

1 bbT 1 bbT T bb V p · A1 · V pbb + V pbb · A2 · bb p +  p · A3 ·  p , 2 2

(8)

that is, the kinetic energy of the system is a quadratic form of relative velocities with coefficients that are functions of relative coordinates, and is completely determined by the motion of the PL relative to SM.

174

P. P. Belonozhko

Further, in accordance with [8–10], it is shown that in the case of two-link plane motion, the kinetic energy (8) of the initial system can be interpreted as the kinetic energy of the reduced system, and the independent equation of the dynamics of relative motion in the form of the Routh equation can be obtained as a Lagrange equation of the 2nd kind for the reduced system.

3 Equation of Dynamics of the Reduced System Let us consider the plane motion of a system of two rigid bodies 1 and 2 (SM and PL) bound by a perfect single-pivot point ball-socket joint (Fig. 1). Masses of bodies 1 and 2—m 1 and m 2 , respectively, J1 and J2 —moments of body inertia 1 and 2 with respect to the centers of mass C1 and C2 , l1 and l2 —the distance between centers of mass and a joint. The motion is viewed with respect to the nonrotating coordinate system XCY originating in the system’s center of mass C, which will be inertial should there be no external forces and moments applied to the system. Position relative to XCY is defined by angle ϕ1 , which describes the absolute motion of the platform, and joint angle q, which describes the motion of the load with respect to the platform (Fig. 1). The control moment is applied to the joint bounding platform and load M. The kinetic energy of the system ˙ q) = T = T (ϕ˙1 , q,

1 1 aϕ˙1 ϕ˙12 + aϕ˙1 q˙ ϕ˙1 q˙ + aq˙ q˙ 2 2 2

where ˜ 12 + ml ˜ 22 + 2ml ˜ 1l1 cos q, aϕ˙1 = aϕ˙1 (q) = J1 + J2 + ml aϕ˙1 q˙ = aϕ˙1 q˙ (q) = J2 + ml ˜ 22 + ml ˜ 1l1 cos q,

Fig. 1 The system “SM–manipulator–PL”

(9)

Moving-Base Space Robots—Applying Eigen-Dynamics …

175

aq˙ = aq˙ (q) = J2 + ml ˜ 22 , m1m2 . m˜ = m1 + m2

(10)

In (9) and thereafter, a dot marks a time derivative t. Assume related to the system’s external forces and moments are absent. In this case, the coordinate q is positional, while coordinate ϕ1 is cyclic and cyclic integral is involved ∂T = aϕ˙1 ϕ˙1 + aϕ˙1 q˙ q˙ = L = const, ∂ ϕ˙ 1

(11)

which reflects that the angular momentum of the system is constant L when external forces and moments are absent. Express ϕ˙1 from (11) ˙ q, L) = − ϕ˙ 1 = f ϕ˙1 = f ϕ˙1 (q,

aϕ˙1 q˙ 1 q˙ + L. aϕ˙1 aϕ˙1

(12)

Substitute (12) in (9) T ∗ = T ∗ (q, ˙ q, L) = T (ϕ˙1 , q, ˙ q)|ϕ˙1 = fϕ˙1 = =

1 1 aϕ˙1 f ϕ˙21 + aϕ˙1 q˙ f ϕ˙1 q˙ + aq˙ q˙ 2 . 2 2

(13)

Let’s write down the Rous function ˙ q, L) − f ϕ˙1 (q, ˙ q, L) · L , R = R(q, ˙ q, L) = T ∗ (q,   aϕ2˙1 q˙ 2 aϕ˙1 q˙ 1 1 2 1 L q˙ − L . R= q˙ + aq˙ − 2 aϕ˙1 aϕ˙1 2 aϕ˙1

(14)

(15)

The independent equation of dynamics of the controlled relative motion for the considered system is the Routh equation d dt



∂R ∂ q˙

 −

∂R = M, ∂q

(16)

and, taking into account (15), has the form  q¨ aq˙ −

aϕ2˙1 q˙ aϕ˙1



 + q˙

2

2

aϕ˙ q˙ ∂aϕ˙1 q˙ 1 aϕ˙1 q˙ ∂aϕ˙1 1 ∂aq˙ − 1 + 2 2 aϕ˙1 ∂q aϕ˙1 ∂q 2 ∂q

 − L2

1 1 ∂aϕ˙1 = M. 2 aϕ2˙1 ∂q (17)

176

P. P. Belonozhko

We introduce notations for the summands of the right part (15)   aϕ2˙1 q˙ 2 1 R2 = q˙ , aq˙ − 2 aϕ˙1

R1 =

aϕ˙1 q˙ L q, ˙ aϕ˙1

R0 = −

1 1 2 L . 2 aϕ˙1

(18)

Given the obvious ratios a˙ ϕ˙1 =

∂aϕ˙1 q˙ ∂aq˙ d ∂aϕ˙1 d d aϕ˙1 = q, ˙ a˙ ϕ˙1 q˙ = aϕ˙1 q˙ = q, ˙ a˙ q˙ = aq˙ = q, ˙ (19) dt ∂q dt ∂q dt ∂q

have aϕ˙ q˙ ∂ R1 = 1 L, ∂ q˙ aϕ˙1 ⎡ ⎤

 ∂aϕ˙1 q˙ ∂aϕ˙1   qa ˙ − qa ˙ ϕ ˙ ϕ ˙ q ˙ ∂a a 1 d ∂ R1 ∂a 1 1 ∂q ∂q ϕ ˙ q ˙ ϕ ˙ q ˙ ϕ ˙ 1 1 1 ⎦L = =⎣ − 2 q˙ L , dt ∂ q˙ aϕ˙1 ∂q aϕ2˙1 aϕ˙1 ∂q ⎡ ⎤

 ∂aϕ˙1 q˙ ∂aϕ˙1 a − a ϕ ˙ ϕ ˙ q ˙ a ∂a 1 ∂a ∂ R1 1 1 ∂q ∂q ϕ ˙ q ˙ ϕ ˙ q ˙ ϕ ˙ 1 1 1 ⎦q˙ L = =⎣ − 2 q˙ L . (20) ∂q aϕ˙1 ∂q aϕ2˙1 aϕ˙1 ∂q Thus d dt



∂ R1 ∂ q˙

 −

∂ R1 = 0. ∂q

(21)

We will consider (17) as an equation of some reduced mechanical system, which in the considered case is the object of control. Then, by virtue of (21), the function   aϕ2˙1 q˙ 2 1 1 2 1 L R = R2 + R0 = L , q˙ − aq˙ − 2 aϕ˙1 2 aϕ˙1

(22)

just as the function (15) can be considered as the Lagrange function of the reduced system. This means that Eq. (17) can be obtained both in the form (16) and in the form d dt



∂LR ∂ q˙

 −

∂LR = M. ∂q

(23)

Considering the Lagrange function as a function L R , we can give the system a visual physical interpretation.

Moving-Base Space Robots—Applying Eigen-Dynamics …

177

4 Applying Eigen Dynamics of a Reduced System to Synthesize Controls Substituting (10) in (17), we obtain the equation of the reduced system in the form



˜ 12 J2 + ml ˜ 22 − m˜ 2 l12 l22 cos2 q J1 + ml

J1 + J2 + ml ˜ 12 + ml ˜ 22 + 2ml ˜ 1l2 cos q



˜ 1l2 sin q J1 + ml ˜ 12 + ml ˜ 1l2 cos q J2 + ml ˜ 22 + ml ˜ 1l2 cos q 2 ml + q˙

2 J1 + J2 + ml ˜ 12 + ml ˜ 22 + 2ml ˜ 1l2 cos q ml ˜ 1l2 sin q + L2

2 = M. 2 J1 + J2 + ml ˜ 1 + ml ˜ 22 + 2ml ˜ 1l2 cos q

q¨

(24)

Equation (24) describes the motion of a nonlinear oscillatory system with one degree of freedom under the action of the control moment M. If M = 0 we obtain the equation of proper motions of the reduced system. Kinetic and potential energy of the reduced system   aϕ2˙1 q˙ 2 1 Ek = q˙ , aq˙ − 2 aϕ˙1

Ep =

1 1 2 L 2 aϕ˙1

(25)

Substituting (10) in (25), we obtain



˜ 12 J2 + ml ˜ 22 − m˜ 2 l12 l22 cos2 q 2 1 J1 + ml

q˙ Ek = 2 J1 + J2 + ml ˜ 12 + ml ˜ 22 + 2ml ˜ 1l2 cos q

(26)

1 1

L2 2 2 J1 + J2 + ml ˜ 1 + ml ˜ 22 + 2ml ˜ 1l2 cos q

(27)

Ep =

A general view of the phase portrait of the equation of proper motions of the reduced system is shown in Fig. 2. The total energy of the reduced system E ∗ = Ek + E p ,

(28)

performing its own movements remains constant. As an example of the use of its own dynamics of the above system, consider the control in the form of two pulses, which translates the system from position 1 to position 5 (Fig. 3).

178

P. P. Belonozhko

Fig. 2 Phase portrait of the equation of proper motions of the reduced system

Fig. 3 An example of a control that moves a point from position 1 to position 5 using the system’s own motion

When moving the reduced system from position 1 to position 2, the kinetic energy changes by a jump in the amount of E 0∗ − E 2∗ < 0, the potential energy remains unchanged.

(29)

Moving-Base Space Robots—Applying Eigen-Dynamics …

179

Moving from position 2 to position 3 is carried out on its own “ballistic” trajectory, the control moment is zero, the total energy remains unchanged. When moving the reduced system from position 3 to position 4, the kinetic energy changes by a jump in the amount of E 1∗ − E 0∗ > 0,

(30)

the potential energy remains unchanged. Moving from position 4 to position 5 is carried out on its own “ballistic” trajectory, the control moment is zero, the total energy remains unchanged.

5 Conclusion According to the results of the analysis of trends in the development of space robotics, the actual problem of automated assembly of large space objects with the help of installation and service robotic space modules is highlighted. By the example of the plane motion of the system of two hinged bodies it is shown that when controlling the movement of the payload relative to the space module, the nonlinear oscillatory system can be considered as a control object. An example of the use of own “ballistic” movements of the reduced system in the synthesis of control is considered.

References 1. Angel Flores-Abad, A., Ma, O., Pham, K., Ulrich, S.: A review of space robotics technologies for on-orbit servicing. Prog. Aerosp. Sci. 68, 1–26 (2014). https://doi.org/10.1016/j.paerosci. 2014.03.002 2. Vovk, A.V., Legostaev, V.P., Lopota, V.A.: Promising concepts and technologies for creating space technology based on mechatronics and microsystem technology. In: Proceedings of the Academy of Sciences, vol. 3, pp. 3–11. Power Industry (2011) (in Russian) 3. Lysyj, S.R.: Scientific and technical problems and prospects for the development of specialpurpose (cosmic) robotics. In: Extreme Robotics. Proceedings of the International Scientific and Technical Conference, pp. 29–32. Politekhnika-Servis, Saint Petersburg (2015) (in Russian) 4. Moosavian, S., Ali, A., Papadopoulos, E.: Free-flying robots in space: an overview of dynamics modeling, planning and control. Robotica 25, 537–547 (2007). https://doi.org/10.1017/ S0263574707003438 5. Rutkovskiy, V.Y., Sukhanov, V.M., Glumov, V.M.: Control of a multimode space robot when performing manipulation operations in the external environment. Autom. Remote. Control. (11), 96–111 (2010) (in Russian) 6. Artemenko, YN., Karpenko, A.P., Belonozhko, P.P.: Features of manipulator dynamics modeling into account a movable platform. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems. Studies in Systems, Decision and Control, vol. 49, pp. 177–190. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27547-5_17

180

P. P. Belonozhko

7. Artemenko, YN., Karpenko, A.P., Belonozhko, P.P.: Synthesis of control of hinged bodies relative motion ensuring move of orientable body to necessary absolute position. In: Gorodetskiy, A.E., Kurbanov, V.G. (eds.) Smart Electromechanical Systems: The Central Nervous System. Studies in Systems, Decision and Control, vol. 95, pp. 231–239. Springer, Cham (2017). https:// doi.org/10.1007/978-3-319-53327-8_16 8. Belonozhko, P.P.: Methodical features of acquisition of independent dynamic equation of relative movement of one-degree of freedom manipulator on movable foundation as control object. In: Gorodetskiy, A.E., Kurbanov, V.G. (eds.) Smart Electromechanical Systems: The Central Nervous System. Studies in Systems, Decision and Control, vol. 95, pp. 261–270. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-53327-8_19 9. Artemenko, Y.N., Karpenko, A.P., Belonozhko, P.P.: Synthesis of the program motion of a robotic space module acting as the element of an assembly and servicing system for emerging orbital facilities. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems: Group Interaction. Studies in Systems, Decision and Control, vol. 174, pp. 217–227. Springer, Cham (2019). ISBN: 978-3-319-99759-9_18 10. Belonozhko P.P.: Robotic assembly and servicing space module peculiarities of dynamic study of given system. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems: Group Interaction. Studies in Systems, Decision and Control, vol. 174, pp. 287–296. Springer, Cham (2019). ISBN 978-3-319-99759-9_23

The Concept of Failureand Fault-Tolerance on Base of the Dynamic Redundancy for Distributed Control Systems of Spacecraft Groups Irina V. Asharina Abstract The chapter considers the existing based on the use of static redundancy approach to the design of failure- and fault-tolerant control systems for multifunctional spacecraft and their groupings, the aggregate computing and control means of which are distributed computing systems of the network structure, and the concept of failure- and fault-tolerance of such systems used in this case. The paper shows the inapplicability of this approach and this concept for the currently developed failureand fault-tolerant self-repairing self-reconfigurable systems with given durations of self-managed degradation and times of active existence of spacecraft grouping control systems. The aim of the work is to formulate a new definition of fault tolerance based on the use of dynamic redundancy for such distributed information- and control systems, which also determines the main features of their design technology. The proposed new definition of failure- and fault-tolerance for these systems simultaneously sets a new approach in their development, based on the preliminary construction of the required trajectories of controlled reconfiguration and degradation of such a system and its subsequent design in accordance with these trajectories. As a result of the application of the proposed definition of fault tolerance and the corresponding design method, the specified trajectory of self-managed reconfiguration and degradation of such system are realized. When the critical level of failure- and fault-tolerance is reached and the next fault occurs, as well as when unacceptable faults occur, the system must enter the safe stop mode and wait for the instructions provided in the design of the distributed multi-machine control system. Then the system must perform instructions, are received from the external environment. Keywords Multifunctional spacecraft · Groups of spacecraft · Multiprocessor systems · Task replication · Fault-tolerance · Mutual information agreement · Dynamic redundancy

I. V. Asharina (B) Joint-Stock Company “Scientific Research Institute ‘Submicron’”, Zelenograd, Russia e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_15

181

182

I. V. Asharina

1 Introduction On-Board digital computer systems (BDCS) of modern on-Board control systems of automatic spacecraft and their groups are multi-machine computer systems (MMCS), or multicomputers, or computer networks, or distributed multiprocessors. And these BDCS in combination with ground infrastructure inevitably become more and more distributed computing systems, corresponding to the definition of such systems in [1]. Let us consider a number of works related to the solution of actual problem—the development of space robotic arms, artificial satellites and their groups. Li et al. [2] describes various types of space activities in near-earth and long-range space missions, examples of Autonomous systems deployed in space to date; provides the current state of the relationship between trusted Autonomous systems and Autonomous space systems; presents various possible scenarios for the development of near-earth space and deep space. Kormushev and Ahmadzadeh [3] is devoted to the problem of fault detection filtration for space robot-manipulators based on the Markov model. The purpose of the work is to design a fault detection filter so that the filter error system is stochastically stable and the effectiveness of limiting probabilistic characteristics can be guaranteed. Approaches to training space robotic arms that help to raise robot autonomy to the next level, the so-called “permanent autonomy” are described in [4]. It’s about the robot’s ability to perform targeted work over long periods of time in dynamic, uncertain conditions without the need for human help. In particular, permanent autonomy is extremely important for robots in hard-to-reach environments such as underwater, rescue and space robotics. There are many aspects of permanent autonomy, such as: overcoming uncertainty, responding to changing conditions, deviation interference, fault-tolerance, energy efficiency, etc. [4] presents a wide range of approaches to training robots. The formulation of the problem in [5, 6] is the group use of robot-manipulators and servicing spacecraft modules (RASSM), which includes the solution of the problem of program motion synthesis for each RASSM taking into account its own inertial movements in terms of internal degrees of mobility. The obtained results are of interest from the point of view realization of the important principle of movement of robots, coordination of free and forced motions of the manipulator in the synthesis of control for space robot module, as part of the assembly and maintenance of systems for the creating orbital objects. In all mentioned works actual goals and tasks of creation of robots-manipulators, single or as a part of groups of SC are put, but these tasks do not affect questions of ensuring reliability and fault tolerance of control systems of such groups. According to [1] in a distributed system, the presence of numerous autonomous computers is invisible to the user, from whose point of view it is a single connected system. Usually, there is a layer of software at a certain level (above the operating system), which is called linking software and is responsible for implementing this

The Concept of Failure- and Fault-Tolerance on Base …

183

idea. A distributed system is a software system built on the basis of a network. This software system provides the necessary degree of connectivity and transparency of network elements. Thus, the difference between a computer network and a distributed system is in the software (especially in the operating system), not in the hardware. Such a computer system has a multi-level organization of interacting hardware and software, at the lower level of which (Fig. 1) there is the hardware of such a system (computers, network elements, communication channels). The following levels represent the software that bottom-up includes: 1. basic software that enables interaction between upper levels software and hardware; 2. general, system software, including, in particular, the operating system, a subsystem I/O, etc.; 3. service level, which programs which provide interaction a basic level and the system level with software higher-level; 4. special (application, target) software—designed to solve the user’s target tasks (special software is often called a software application or simply an application).

Fig. 1 Multilevel organization of interacting hardware and software

184

I. V. Asharina

Principles of construction of control systems of a multifunctional spacecraft or a groups of spacecraft [7]: (1) openness (interaction with the external environment), (2) self-organization, (3) weak hierarchy in the circuit of making agreed decisions, (4) parallel and simultaneous solution of interacting targets in real-time, (5) ensuring information security (given the reliability of the issued information, given failureand fault-tolerance for each of the solved targets of critical application and network environments of their interaction). The features of such systems are: (a) the autonomy of the Central computer, (b) the lack of shared memory, (c) inter-machine interaction on point-to-point and bus communication channels; (d) multi-level system and the lack of centralized control body; (e) the need for self-synchronization and self-organization of the system to ensure the necessary adaptation, scaling, protection from external influences, the effects of faults and design errors; (e) real-time operation; (g) a long period of active existence; (h) high requirements for reliability and reliability of the results. The criticality of applications spacecraft and their groups, very high cost of errors in their design and operation more give the highest demands on reliability, failure tolerance BDCS them, and these requirements still remain one of the most important and most urgent tasks in the development of their management systems. At the same time, a paradoxical situation arises—the basic concept of failure- and fault-tolerance for such distributed and network computing systems is formulated insufficiently fully and correctly, requires clarification even at the level of International standards of the relevant industries, which inevitably leads to design errors and heavy economic and other losses, but such refinement is not carried out. One of the first concepts of fault-tolerance, currently widely used, was introduced by A. Avizienis, who formulated it as follows: “fault-tolerance is a property of the architecture of digital systems, which allows a logical machine to continue to work even when a variety of component failures occur in the real system, which is its carrier” [8, 9]. The disadvantage of this definition is a clear contradiction: due to the lack of restrictions on the number of simultaneous failures of components, it turns out that any system, no matter how it was built, can’t be initially fault-tolerant, since when all components in the system malfunction (which is allowed by this definition), it is impossible to create a logical machine specified in the definition. International Standards define the concept of fault-tolerance as follows. According to [10] the concept of fault-tolerance is the ability of a system, product or component to operate as intended, despite the presence of defects in software or hardware. This definition has the same drawback as the above definition. The paper proposes a new technology for the development of control systems for multifunctional spacecraft and their groupings, based on a new definition of the concept of failure- and fault-tolerant system.

The Concept of Failure- and Fault-Tolerance on Base …

185

2 Definition of the Failure- and Fault-Tolerance Concept of Multifunctional Spacecraft and Their Groups Control Systems Suppose that the developed control system of a multifunctional spacecraft or a group of spacecraft should contribute to the implementation of a number of target functional tasks, for which this spacecraft or group of spacecraft is created. Under the fault tolerance of distributed multimachine computing systems (DMCS), which include system management of multifunctional spacecraft or groups of spacecraft, we will be understand its ability to correctly carry out the target tasks in case of valid sets of simultaneous faults (failures) acceptable models and acceptable sequences of such sets [11]. While DMCS is in the form of a set of interrelated, further indivisible elements [11], and any failure or any concurrent combination of such a separate element is considered a separate single fault DMCS, and the number of simultaneously faulty elements in DMCS is the multiplicity of existing faults. Difference to determine fault tolerance in [8, 12] is that, given the definition of the failure tolerance DMCS at the initial stage of its design and requires the consideration of three input parameters, first, allowable models of faults [11], second, the allowed multiplicity of simultaneous sets of such faults, and, thirdly, acceptable sequences of such sets of faults, which must be taken into account next apply design methods DMCS. Moreover, in the projected DMCS shall be provided all possible mechanisms for protection in cases, when the situation in DMCS beyond permissible, for example, receiving a set of invalid multiplicity faults or malfunctions of such models that are “not covered” by a valid model of the faulty, and must be provided possible methods and means of exit DMCS of these situations, for example, the transfer of part or DMCS overall in the state of unconditional safe stop, then await instructions from the external environment and their possible execution. In this case, the fault model is understood as the features of its manifestation (possible errors) in the behavior and/or output information of the faulty element relative to the serviceable elements of the DMCS.

3 The Proposed Concept of Failure- and Fault-Tolerant Control Systems for Multifunctional Spacecraft and Their Groups Design Based on the definition of the failure tolerance DMCS offers the concept of building a distributed failure- and fault-tolerant control systems of promising multifunctional spacecraft and their groupings that take into account their complexity and degree of integration [7]. Modern and especially perspective computer systems of control of multifunctional spacecraft and their groupings are intended for control of parallel execution of several interacting target tasks Z 1 , Z 2 , …, Z k providing external target functional

186

I. V. Asharina

requirements of the Customer. At the same time, each of the target tasks has its own degree of importance for the Customer and the more important target task should have higher protection against its possible non-execution or improper execution than the less important target task. This requirement in DMCS is affecting assigned for subsystems of each target task Z i its individual failure tolerance µi that specifies the allowed multiplicity of simultaneous faults of the allowable models for the elements of this subsystem, when the subsystems of the other target tasks that interacting with it the will be able is guaranteed to get the correct information from a subsystem task Z i . In this work, it is proposed to achieve failure- and fault-tolerant execution of the target task Z i in the DMCS due to its parallel execution (replication) on several independent, but interconnected and interacting computers that make up the complex K i of this target task Z i , in which all computers exchange calculated copies of the output information and by majoring or quorum determine its correct value in each serviceable computer of this complex, provided that no more than µi copies of this information can be erroneous [13, 14]. The excess above µi the number of erroneous copies the output of the complex, i.e. the fault is more than µi computers in the complex, considered as elements of the complex, is considered the fault of the complex. The complexity of algorithms, methods, and mechanisms that provide a theoretically grounded failure and failover solutions targets DMCS depends on accept the model of fault elements DMCS. Of all known and used fault models, the most common is the hostile fault model, in which the behavior of a faulty computer can be completely arbitrary, unequal to other computers of the DMCS, and even similar to malicious [7, 15–18]. Hostile fault protection will be guaranteed to protect against faults of the same multiplicity of at any other fault model. There are two types of redundancy used in the DMCS to implement failure- and fault-tolerance in them: static and dynamic. Static redundancy guarantees only the correctness of the output information in the event of permissible sets of faults, i.e. performs “masking” of the manifestations of faults due to the excess number of copies (majority, quorum), assuming that only a smaller part of these results can be erroneous. The advantage of static redundancy is the relative ease of implementation. The disadvantage is that static redundancy does not guarantee and not ensures the long life of DMCS due to the lack of control over the entered in DMCS redundancy, and also because of the possibility of undetectable faults of the subsystems RMS or the entire DMCS in general. Dynamic redundancy provides: 1. parrying of manifestations of malfunctions (errors) in the output information of the system, 2. simultaneous and consistent detection of manifestations of malfunctions (errors) in the subsystems and in the DMCS as a whole, 3. continuous and end-to-end, simultaneous and consistent identification of manifestations of malfunctions at the place of their occurrence and by type (failure, program failure, fault),

The Concept of Failure- and Fault-Tolerance on Base …

187

4. consistent information recovery of, and retraction vos-case elements in the joint task work DMCS (as separate elements DMCS can be considered separate computers, their interface devices with communication channels, communication channels and other network elements (hubs, routers, etc.)), 5. the required self-managed reconfiguration DMCS, 6. self-governing degradation of the DMCS with reduced reliability and/or functional characteristics DMCS, 7. if unable to perform self-governed degradation (due to the exhaustion of resources or the occurrence of invalid sets of faults or invalid sequences of faults), must be simultaneous and coordinated unconditional jump DMCS or part thereof in a state of safe stop and must be pending guidance from the environment with the following possible execution of the received instructions. Dynamic redundancy implies continuous and end-to-end functional diagnosis [11, 19] of DMCS. If the spacecraft or their grouping is intended for long-term use (longterm active existence), the DMCS should be built only on the basis of the introduction of dynamic redundancy. It is this redundancy that allows DMCS to implement selfgoverning reconfiguration and self-governing degradation in cases where the valid sets of faults to implement envisaged in its design, that is, to implement self-managed work recovery DMCS for target tasks execution. The third input parameter (set of admissible sequences of admissible sets of simultaneous malfunctions) of the process of designing the DMCS using dynamic redundancy sets the necessary qualitative requirements for the mechanisms of dynamic redundancy to be implemented in the DMCS. Each of the allowable sequences of sets of simultaneous faults sets a certain trajectory of the process of continuous and end-to-end functional diagnosis DMCS, possible self-recovery of software failures identified, it is possible to self-governing after the reconfiguration of the identified component failure, the possible degradation DMCS self-governing and self-healing it is possible to continue target work or the necessary transition to a state of safe shutdown subsystems or DMCS in general. And all possible admissible sequences of sets of simultaneous malfunctions of DMCS set all possible such trajectories which are required to be realized in the develop DMCS. Thus, the longest degradation trajectories and the longest periods of active existence can be achieved in DMCS. Permissible sequences of permissible sets of faults can be the basis for determining the necessary redundancy levels of individual control subsystems and the control system as a whole, introduced at the design stage of the DMCS. Dynamic redundancy provides not only long-term degradation trajectories of DMCS but also its reconfiguration (build-up) in order to increase functionality. The proposed concept allows you to perform such actions. This happens in the case of launching additional spacecraft or their groups, which have their own targets. According to [11, 13, 14] DMCS complexes for fault-tolerant solutions of target tasks are built, if possible, on the basis of a network of an arbitrary, but the known structure, which does not contradict the principles of construction of multifunctional spacecraft or their groups. At certain points in time, it is possible to allocate new complexes for new targets and draw them into the overall target work of the DMCS.

188

I. V. Asharina

In addition to the above-mentioned mechanisms to ensure the failure- and faulttolerance of the DMCS management of spacecraft and their groupings, mechanisms, and methods are also needed: • exclusion of external unauthorized access, • protection against design errors. Target properties, qualities, characteristics of DMCS and methods of their implementation should be defined, developed and guaranteed by target developers of the system. Properties, qualities, safety characteristics, failure- and fault-tolerance and methods of their implementation are determined, developed and guaranteed by the relevant specialists in close cooperation with the target application specialists. Currently, the following technology is used to develop failure- and fault-tolerant control systems for spacecraft and their groups. The target developers of the spacecraft control system or their groups without the participation of failure- and fault-tolerance specialists perform the following sequence of actions (Fig. 2): 1. The construction of the unformalized objective of the project. 2. Development of assumptions, hypotheses, limitations, analysis of existing theories, formulation of conditions for the application of DMCS. 3. Formalization of the project goal within the framework of accepted assumptions, constraints, existing theories and application conditions.

Fig. 2 The existing top-down design process of the target function and the failure- and faulttolerance function of the spacecraft control system or their group

The Concept of Failure- and Fault-Tolerance on Base …

189

4. Development of a generalized and justified method and algorithm for the implementation of the formulated project goal, its modeling, and evaluation without taking into account the requirements for failure and fault tolerance. 5. Decomposition of the generalized algorithm into hardware and software parts, their modeling and evaluation. Paragraphs 1–5 of the development process are, in fact, the study of the architectural part of the project without taking into account the requirements for failure- and fault-tolerance. In the execution of the following points of the development process, specialists are already involved in ensuring the characteristics of reliability and failure-and faulttolerance. Thus to them, the requirement of protection against single malfunctions of elements of system familiar to target developers of the model of this malfunction is imposed: 6. Development of technical specifications of the hardware part and technical specifications of the software part. 7. Realization of hardware and software parts of DMCS parts. 8. Docking and debugging of hardware and software parts. 9. Comprehensive testing and evaluation of decisions taken and implemented. If the results are unsatisfactory, return to the previous paragraphs and rework the project. Such a design process often leads to the appearance of negative synergistic effects—emergence, i.e. improper behavior of the system due to the occurrence of unforeseen system phenomena, the inadequacy of accepted models, constraints or theories to the realities. Such effects are extremely difficult to analyze and are usually “written off” as yet unexplored or not worked out elements of technology or protection from external influences (for example, insufficient radiation resistance of electro-radio products). It is, therefore, necessary in the initial architectural design stages of the system solve the problem of ensuring the fault-tolerance, are applying adequate models, constraints, assumptions and theories, to what are hostile model of faulty item DMCS, the allowed multiplicity of simultaneous faults DMCS, admissible set of sequences of simultaneous faults DMCS.

4 Conclusion The correctness of the definitions used largely determines the reliability of the conducted scientific research, and the clues laid down in these definitions of the necessary further steps of research pushes the researcher to move in the right direction. The concept of failure- and fault-tolerance at the present stage requires revision, also by the State Standard. The issues of ensuring fault tolerance of any system should be accompanied by the formulation of requirements for the model of system failure,

190

I. V. Asharina

the permissible multiplicity of simultaneous sets of such faults and the permissible sequences of these permissible sets. The target work of the DMCS and its interaction with the failure- and faulttolerance interface implemented in this DMCS [20], should be based on theoretically proven methods and mechanisms for system self-diagnosis DMCS, parry permissible manifestations of a fault with their identification by place of occurrence and by type, reconfiguration and recovery DMCS target work DMCS after failures and software faults, self-degradation DMCS at faults, mode transition to the safe stop DMCS in the exhaustion of spare resources DMCS or the occurrence of inadmissible sets of faults or unacceptable sequences of such sets [20]. The use of dynamic redundancy should not only lengthen the trajectory of selfmanaged degradation, but also reconfigure the DMCS to increase its functionality. The chapter proposes a new approach for the development of control systems for multifunctional spacecraft and their groupings, based on a new definition of the concept of failure- and fault-tolerant system.

References 1. Boyce, R., Griffin, D.: Future trusted autonomous space scenarios. In: Studies in Systems, Decision and Control, vol. 117, pp. 355–364 (2018) 2. Li, F., Shi, P., Wu, L.: Fault detection for underactuated manipulators modeled by MJS. In: Studies in Systems, Decision and Control, vol. 81, pp. 171–193 (2017) 3. Kormushev, P., Ahmadzadeh, S.R.: Robot learning for persistent autonomy. In: Studies in Systems, Decision and Control, vol. 42, pp. 3–28 (2015) 4. Artemenko, Y.N., Karpenko, A.P., Belonozhko, P.P.: Synthesis of the program motion of a robotic space module acting as the element of an assembly and servicing system for emerging orbital facilities. In: Studies in Systems, Decision and Control, vol. 174, pp. 217–227 (2019) 5. Belonozhko, P.P.: Robotic assembly and servicing space module peculiarities of dynamic study of given system. In: Studies in Systems, Decision and Control, vol. 174, pp. 287–296 (2019) 6. Tanenbaum, A., Wetherall, D.: Computer Networks, 5th edn., p. 07458. Prentise Hall, New Jersey (2011) 7. Lobanov, A.V., Asharina, I.V., Grishin, V.Ju., Sirenko, V.G.: A prototype of a highly adaptive, distributed, net-centric, multicomplex malfunction- and fault-tolerant control system—a topical problem. H&ES Res. 10(1), 48–55 (2018). https://doi.org/10.24411/2409-5419-2018-10019. (in Russian) 8. Avizienis, A.: Fault-tolerance: the survival attribute of digital systems. Proc. IEEE 66(10), 1109–1125 (1978) 9. Avizienis, A., Laprie, J.-C.: Dependable computing: from concepts to design diversity. Proc. IEEE 76(5), 629–638 (1986) 10. ISO/IEC 25010: 2011 systems and software engineering—systems and software quality requirements and evaluation (SQuaRE)—system and software quality model 11. Lobanov, A.V.: Models of closed multimachine computer systems with transient-fault-tolerance and fault-tolerance on the basis of replication under byzantine faults. Autom. Remote Control 70, 328 (2009). https://doi.org/10.1134/S0005117909020131 12. Karavay, M.F.: Application of the theory of symmetry to the analysis and synthesis of faulttolerant systems. Autom. Remote Control 57(6), 899–910 (1996) 13. Asharina, I.V., Lobanov, A.V.: Extracting complexes that ensure sufficient structural conditions for system mutual informational coordination in multicomplex systems. Autom. Remote Control 75(6), 1078–1089 (2014). https://doi.org/10.1134/S0005117914060083

The Concept of Failure- and Fault-Tolerance on Base …

191

14. Asharina, I.V., Lobanov, A.V.: Extraction of the structural environment of mutual system information agreement in multicomplex systems. Autom. Remote Control 75(8), 1471–1478 (2014). https://doi.org/10.1134/S0005117914080104 15. Pease, M., Shostak, R., Lamport, L.: Reaching agreement in the presence of faults. J. ACM 27(2), 228–237 (1980) 16. Asharina, I.V., Lobanov, A.V., Mishenko, I.G.: Mutual informational coordination in incompletely connected multicomputer systems. Autom. Remote Control 64(5), 843–850 (2003). https://doi.org/10.1023/A:1023679325384 17. Barborak, M., Malek, M.: The consensus problem in fault-tolerant computing. ACM Comput. Surv. 25(2), 171–220 (1993) 18. Kopyltsov, A.V., Kravets, A.G., Abrahamyan, G.V., Katasonova, G.R., Sotnikov, A.D., Atayan, A.M.: Algorithm of estimation and correction of wireless telecommunications quality. In: 2018 9th International Conference on Information, Intelligence, Systems and Applications, IISA 2018, Art. No. 8633620 (2019) 19. Parkhomenko, P.P. (ed.): Fundamentals of Technical Diagnostics (Optimizing Diagnostic Algorithms and Hardware). Energiya, Moscow (1981) 20. Lobanov, A.V., Asharina, I.V.: Restoration of target work in automatic failure and fault-tolerant multitasking distributed information-control system. Eng. J.: Sci. Innov. 7 (2019). https://doi. org/10.18698/2308-6033-2019-7-1902

Industrial Robotics

This part defines theoretical foundations of industrial robotic control, formulates its concept within the frameworks of intelligent control methods and theories, and views the peculiarities of control technologies as a specific sphere of modern industrial technologies, which distinguished in the Industry 4.0. The authors study the genesis of mathematical and control methods transformation from directly human-controlled machines to knowledge-based complex robotic systems in the twenty-first century. 1. I. L. Ermolov—Industrial Robotics Review. 2. O. I. Gladkova, V. V. Veltishev, S. A. Egorov—Development of an Information Control System for a Remotely Operated Vehicle with Hybrid Propulsion System. 3. I. N. Egorov—Control of the Launcher of Unmanned Aerial Vehicles with Parallel Kinematics in the Conditions of Position–Force Uncertainty. 4. A. Schwandt, A. S. Yuschenko—Intuitive Industrial Robot Programming Interface with Augmented Reality Environment.

Industrial Robotics Review Ivan Ermolov

Abstract Industrial robotics plays an important role in modern industrial automation technology. One cannot imagine transfer to the next manufacturing paradigm without industrial robotics being involved in various phases of production. This work has been done as a part of a perspective view towards the fundamental research in industrial robotics. Keywords Industrial robotics · Robots · Research areas

1 General Trends Review Once being end-users oriented researches should imply their plans together with current industry trends and perspectives [1, 2]. For this purpose this study has covered at a glimpse main trends of modern industrial robotics which are the following.

1.1 Expansion of New Application Areas of Industrial Robots Previously, the main area of industrial robotics installations was the automotive industry with spot welding as the main application area. Now industrial robots are used actively in assembly [3], including microelectronics and gadgets’ assembly. This has affected both the manipulators of robots and their periphery.

I. Ermolov (B) Ishlinsky Institute for Problems in Mechanics RAS, Moscow, Russian Federation e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_16

195

196

I. Ermolov

1.2 Expansion of Robots’ Payloads and Sizes Traditionally manufacturers of industrial robots produced robots of 5–500 kg payload range. However, nowadays there is a clear trend of payload expansion. We recognize that almost all main industrial robots’ manufacturers have abated lower payload down to 0.5–1 kg (Fig. 1). Also, some of them have done an increase of heavy payload robots up to 1700 kg (Fig. 2).

Fig. 1 Microrobot by Cobotta

Fig. 2 This giant from Fanuc can lift a car at once

Industrial Robotics Review

197

Fig. 3 SCARA manipulators are popular again

Fig. 4 A “Zoo” of Delta-type manipulators

1.3 Application of New Kinematic Schemes in Manipulators Traditionally PUMA-alike configuration was dominating the market. SCARAconfiguration was familiar to robotics researchers, however, was used quite rarely. Now SCARA-configuration is presented among light-payload manipulators as it’s more efficient for assembly operations (Fig. 3). Delta-configuration also is being presented by most of the robots’ manufacturers (Fig. 4).

1.4 Collaborative Robotics Collaborative robotics continues to expand its applications. It seems like one of its main targets to robotize SMEs, which usually are restricted in high-skilled personnel and spare spaces [4, 5]. Collaborative robots allow for a human to work safely (Fig. 5) right next to a moving robot (paying by reduction of speeds of course) and to simplify the teaching of robots (Fig. 6) by personnel with minimal knowledge about robots [6].

198

I. Ermolov

Fig. 5 AIRSKIN protects human from undesirable contact with the robot

Fig. 6 “Green line” of collaborative robots by FANUC

1.5 The Flexible Basement of Manipulators In the past, there were strict rules from all manufacturers of robots regarding basement of manipulators. This dealt with a thickness of ceiling, type, and sizes of tacks, etc. Nowadays a lot of manipulators undermine to be moved during their lifetime. Moreover, some of the manipulators are using a self-propelled basement for operational movements (Fig. 7).

Industrial Robotics Review

199

Fig. 7 KUKA-manipulator on the wheels

1.6 Industrial Robots’ Tools and Grippers As soon as robots were mostly used for welding in the past, the weld gun was the main type of robots’ periphery. Now, these are complemented with various grippers, screwing devices, paint guns, etc. Machine vision was always a state of art equipment. In the industry of the 2000s [7], it was considered as an “expensive toy”. Now a lot of manufacturers of industrial robots propose on the market robots equipped with sophisticated [8] machine vision equipment. Also, it’s a new challenge that machine vision systems now can not only recognize an object but also measure its sizes with high efficiency (Fig. 8). Fig. 8 World optics manufacturer Nikon now tries to replace coordinate inspection centers

200

I. Ermolov

Fig. 9 Robotic palletizer by MiR

1.7 Industrial Mobile Robots Mobile robots now are more actively used in industry as well then it was before [9]. As a rule, it’s a 6-wheels based flat platform with 2 driving wheels in the center and 4 supporting wheels aside, some robots use omnidirectional wheels. However, some platforms may be equipped with manipulators or palletizing equipment (Fig. 9).

1.8 Exoskeletons Exoskeletons [10] have left research labs and now are coming to the factory. There they are usually used for assembly of heavy parts in the automotive industry. An interesting fact is a trend that some industrial robots’ manufacturers have started to produce exoskeletons (Fig. 10).

1.9 Industrial Robots with Low-Cost Mechanics In [11] there was presented a trend of transfer of functional load from mechanical components of machines towards intelligent parts of machines. This trend has found its extra proof in industrial robotics where there have appeared new manufacturers of industrial robots with simplified kinematics and drastically reduced the cost of the robot (Fig. 11). It’s too early to make conclusions regarding the efficiency and reliability of these robots (remember case of modular robots [12, 13]), however, in case of success this trend seems to be very promising.

Industrial Robotics Review

201

Fig. 10 This young lady is especially strong with an exoskeleton by COMAU

Fig. 11 Low-cost robots’ “Zoo”

1.10 Dual-Manipulators Almost all main manufacturers of industrial robots present dual-manipulators scheme. The latter should be specifically efficient for sophisticated assembly (Fig. 12). However, upon the author’s point of view the dual-manipulators scheme has not become very popular. Force-sensing is used both in collaborative robotics and in the adaptation of robot towards technological processes.

202

I. Ermolov

Fig. 12 Dual-manipulators robot by ABB

2 Industry 4.0 and Its Developments In first positions, we must mention further development of algorithms [14] and sensors [11], including data fusion [15], remote diagnostics and repair of robots, operational information dissemination throughout the whole enterprise. Also, we must mention new industrial protocol OPC UA which is actively presented by many companies. These also outline emergency of information within industrial robotics, including cloud calculation, data storage, programming [16] and protection, etc. An approach to expand variety of input devices into industrial robotics is being widely developed (e.g. see Fig. 13).

Fig. 13 Now one can program an industrial robot from a PAD

Industrial Robotics Review

203

Basing on this author suggests to concentrate on following areas of fundamental research in industrial robotics: • Modeling of kinematics and dynamics of manipulators of new architecture including soft robotics, optimization, and studies of ergonomics [4, 17, 18] of robots’ workspace. • Modeling of various technological operations, including force contact and mobile technological robots [19]. • Control of robots [20] on complaint base performing various technological operations. • Machine vision applications for measurement and control, including dynamic measurements.

3 Conclusions Fundamental research can and hence must play an important role in the further development of industrial robotics. Proposed themes may be included into perspective Programme of Fundamental Research in Russian Federation. Acknowledgements Presentation of these materials was supported by the Russian Ministry of Science and Higher Education (Project Reg. No. AAAA-A17-117021310384-9). Part of research was done under Program “Advanced Topics of Robotic Systems” of the Presidium of the Russian Academy of Sciences. Author thanks Bosch-Rexroth for their assistance to attend Automatika-2018 exhibition.

References 1. Robotics 2020, 2014–2020 Strategic Research Agenda for Robotics in Europe, euRobotics aisbl, 2013 2. Executive Summary World Robotics 2018 Industrial Robots, IFR, 2018 3. Gilday, K., Hughes, J., Iida, F.: Achieving flexible assembly using autonomous robotic systems. In: Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Madrid, Spain, 1–5 Oct 2018 4. Haddadin, S., Albu-Schäffer, A., Eiberger, O., Hirzinger, G.: New insights concerning intrinsic joint elasticity for safety. In: IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010—Conference Proceedings. Art. No. 5652037, pp. 2181–2187 (2010) 5. Bermejo, J., Chibani, A., Gonçalves, P., Li, H., Jordan, S.R., Olivares, A., Olszewska, J.I., Prestes, E., Fiorini, S.R., Sanz, R.: Collaboratively working towards ontology-based standards for robotics and automation. In: Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Madrid, Spain, 1–5 Oct 2018 6. Lin, H.-C., Ray, P., Howard, M.: Learning task constraints in operational space formulation. In: Proceedings of IEEE International Conference on Robotics and Automation. Art. No. 7989039, pp. 309–315 (2017)

204

I. Ermolov

7. Ermolov, I.L.: Industrial robotics: the modern state and perspectives (The review of the exhibits of the section “the industrial automation” of the Hanover Fair 2002). Mechatron. Autom. Control 6 (2003) 8. Serebrennyi, V., Boshliakov, A., Ovsiankin, G.: Active stabilization in robotic vision systems. In: Proceedings of 13th International Scientific-Technical Conference on Electromechanics and Robotics “Zavalishin’s Readings”—2018, MATEC Web Conference, vol. 161 (2018) 9. Ermoiov, I.L.: Special robotics: the modern state and perspectives (The review of the exhibits of the section “the industrial automation” of the Hanover Fair 2002). 7 (2003) 10. Gradetsky, V., Ermolov, I., Knyazkov, M., et al.: The influence of various factors on the control of the exoskeleton. Proceedings of the 3rd International Conference on Interactive Collaborative Robotics, ICR 2018, pp. 60–70. Springer Nature Switzerland AG 2018 Printforce, The Netherlands (2018) 11. Poduraev, J.: Mechatronics: Fundamentals, Methodology, Applications. Mashinostroenie, Moscow (2006) 12. Dashevskiy, V., Budkov, V., Ronzhin, A.: Survey of modular robots and developed embedded devices for constructive and computing components. In: International Conference on Interactive Collaborative Robotics, LNAI 10459, pp. 50–58. Springer, Cham (2017). https://doi.org/10. 1007/978-3-319-66471-2_6 13. Andreev, V.P., Poduraev, J.V., Design of heterogene mobile robots using functional modules approach. In: Proceedings of “Extreme Robotics” International Conference, 2016, vol. 1, № 1. pp. 39–49 14. Cheng, F.-T., Tieng, H., Yang, H.-C., Hung, M.-H., Lin, Y.-C., Wei, C.-F., Shieh, Z.-Y.: Industry 4.1 for wheel machining automation IEEE robotics and automation. In: Proceedings of 2016 IEEE International Conference on Robotics and Automation (ICRA) Stockholm, Sweden, 16–21 May 2016 15. Ermolov, I.L.: Hierarchical data fusion architecture for unmanned vehicles. In: Smart Electromechanical Systems: The Central Nervous System, vol. 95 of Studies in Systems, Decision and Control (2017) 16. Biggs, G., MacDonald, B.: A survey of robot programming systems. Proceedings of the Australasian Conference on Robotics and Automation (2003) 17. Ermolov, I.: Increasing efficiency of industrial robots through optimization of their trajectory location in workspace. In: Kaliaev, I.A. (ed.) 10th Russian National Multi-Conference on Control Problems, pp. 265–267 (2017). ISBN: 978-5-9275-2429-7 (T. 2) 18. Kronander, K., Billard, A.: Online learning of varying stiffness through physical human-robot interaction. In: Proceedings of IEEE International Conference on Robotics and Automation. Art. No. 6224877, pp. 1842–1849 19. Filaretov, V.F., Zuev, A.V., Gubankov, A.: Manipulators’ Control for Various Technological Operations. Nauka, Moscow (2018) 20. Afonin, V., Iljuchin, Y.: Intelligent control in Robotics and Mechatronics. Moscow (2018)

Development of an Information Control System for a Remotely Operated Vehicle with Hybrid Propulsion System Olga I. Gladkova, Vadim V. Veltishev and Sergey A. Egorov

Abstract The need for periodic monitoring of the technical conditions of various underwater metal structures with limited access for humans stimulates the development of new specialized underwater robotic vehicles. One of the most important requirements for such types of works is to ensure high accuracy of motion and positioning of the vehicle. It can be reached the means of remotely operated vehicles (ROVs) with hybrid propulsion systems (including both propellers and wheels). Complex processing of linear and angular movement sensors is used to solve the navigation problem. A comparative analysis of various combinations of these sensors to determine the coordinates of the ROV relative to the inspected object was carried out. Three algorithms of the trajectory movement were tested in trials. Experimental data led to the conclusions about the practical applicability of the proposed approach to the organization of survey work and the attainability of the positioning accuracy of the ROV. The correctness of the implemented algorithms was verified and the ways of system improvement were specified. Keywords Remotely operated vehicle (ROV) · Hybrid propulsion system · Underwater inspection · Flaw detection · Underwater local navigation systems

1 Introduction Carrying out many operations of diagnosing the current state of such underwater structures, such as pipeline lines, ship hulls, port facilities and internal surfaces of water-cooled power reactors, due to the dangerous environment for humans and large volumes of surfaces requiring detailed analysis, makes it increasingly important to switch to the use of robotic tools. The use of remotely operated underwater vehicles (ROV) is one of the most optimal solutions in terms of reducing the risks to the life and health of divers, time and financial costs, as well as improving the quality of survey results. Analysis of the existing robotic tools that position themselves as O. I. Gladkova (B) · V. V. Veltishev · S. A. Egorov Bauman Moscow State Technical University, Moscow, Russia e-mail: [email protected] © Springer Nature Switzerland AG 2020 A. G. Kravets (ed.), Robotics: Industry 4.0 Issues & New Intelligent Control Paradigms, Studies in Systems, Decision and Control 272, https://doi.org/10.1007/978-3-030-37841-7_17

205

206

O. I. Gladkova et al.

vehicles for carrying out a survey of underwater structures showed that many of them are capable of performing only a limited range of the entire volume of operations [1]. The use of vehicles with hybrid propulsion systems, including contact propulsion system (wheeled or tracked) along with propellers (traditional for underwater robotics), allows to ensure the presence of guaranteed contact or a small gap (several millimeters) between the sensors used for flaw detection and surface of survey, as well as potentially solve the problem of high-precision positioning [2]. The purpose of this work is to consider the features of building an information control system for a remotely operated vehicle with a hybrid propulsion system, designed to monitor the state of these underwater objects.

2 ROV Baseline Requirements To develop the initial requirements for the movement parameters of the robottechnical equipment of this class, industry-specific international and state standards, rules of operating and controlling organizations, including the standards of sea and river navigation [3, 4], subsea pipeline systems [5–7], equipment for nuclear power plants were considered [8, 9]. Their detailed analysis led to the conclusion that the largest volume of operations with a fundamentally similar methodology is required to carry for the underwater parts of the ship hulls survey. Therefore, in the course of further research, issues related to the conduction of flaw detection of the underwater parts of ships without drydocking were considered. The essential feature of this type of work is the need for the vehicle to provide high-precision positioning and navigation fixation of the detected damage to the coordinate system of the object being examined. The required accuracy is due to the diagnostic methods used and the parameters of the defect meters used. Nowadays, to identify hidden or inaccessible by visual survey of discontinuities subsurface, acoustic methods of non-destructive testing (NDT) [11–13]—an ultrasound method (USM) and electromagnetic acoustic transducers (EMAT), which allow flaw detection without restrictions on the speed of the sensors relative to the surface, regardless of the curvature of the test object, the ambient temperature surfaces and surfaces for flaw detection, as well as in the presence of small dirt on the surface [14, 15]. Some of the most stringent requirements are given in the American Society for Testing and Materials “Standard Practice for Mechanized Ultrasonic Examination of Girth Welds Using Zonal Discrimination with Focused Search Units” ASTM E196116 [16]. According to this document, the meter of the current position of the USM tool relative to the weld should provide: • a distance-measuring circuit or acquisition system shall provide a means of electronically determining weld distance to an accuracy of typically ±1 cm or better (including circumferential weld distance);

Development of an Information Control System for a Remotely …

207

Fig. 1 Scanning options: a transverse-longitudinal scan, b longitudinal-transverse scan

• the recording or marking system shall clearly indicate the location of discontinuities relative to the marked starting position of the scan, with a ±1 cm accuracy. The trajectory of the sensors and, therefore, the underwater vehicle on which they are placed, depends on the method of ultrasonic scanning (see Fig. 1—transverselongitudinal or longitudinal-transverse). It represents traverse with a width of transverse steps Ct and longitudinal CL , determined by the design dimensions, the number, and location of the EMAT, as well as the thickness of the welded joint being controlled. For example, when using this type of piezoelectric transducers with a diameter of 50 mm on a ROV, the trajectory of the carrier vehicle with the transverse-longitudinal scanning method will be “micro traverse” with a pitch of 25 mm. Accordingly, when placing several sensors (for example, when using 6 transducers with the same geometrical dimensions without overlapping zones between the sensors), the traverse pitch can be increased to 150 mm. In this case, the only limitation of the speed of movement of the sensors is the processing speed of the computing module information from the converters. Thus, the possible options for organizing movement in the process of inspecting a ship hull afloat using robotic tools can be reduced to the following (see Fig. 2): 1. Complete fault detection of the ship’s hull: the movement of the vehicle by horizontal or vertical traverses with the capture of the flaw detection sensors (or their system) of the maximum possible surface area; 2. Partial fault detection of the ship’s hull: access to a given point and detailed examination of certain areas by “micro traverses”. At the same time, the task of ensuring high accuracy of positioning and navigation measurement binding to the current location of the technical means, reaching about ±1 cm, is currently completely new for underwater robotics.

208

O. I. Gladkova et al.

Fig. 2 Examples of options for organizing the movement of ROV during the survey of the hull of the vessel in the mode: a full fault detection, b partial fault detection

3 Development of an Information Control System Requirements for the structure and composition of an information control system components are formed on the basis of the requirements for the vehicle and, as a result, for the control system. The information-measuring complex, which consists of the hardware and software parts forms all the necessary information about the motion parameters for the control system. The required composition of the complex was determined on the basis of an analysis of local navigation methods that could potentially provide this high accuracy.

Development of an Information Control System for a Remotely …

209

3.1 Analysis of the Applicability of Underwater Local Navigation Systems In the process of flaw detection, the ROV, which is the carrier of the measuring equipment, must solve the problem of local navigation that is determining its location relative to the object being examined. Therefore, in the present work, the term “navigation” will be understood in a narrow sense as a solution to the problem of positioning the vehicle in the coordinate system of the object of the survey. According to the way of determining the location of the ROV, navigation methods can be divided into three groups [17]: positional methods, dead-reckoning methods, surveycomparative methods. Figure 3 shows a diagram of methods and navigation systems of ROV, used for surveying underwater structures. Acoustic positioning systems (APS) form the basis of the navigation systems of the overwhelming number of autonomous and remotely operated underwater vehicles created at different times in different countries. By the principle of operation, they are divided into systems with a long baseline (LBL), a short baseline (SBL) and systems with an ultrashort baseline (USBL) [18, 19]. The most accurate system is the LBL, one of the variants of its application for the survey of ship hull-type structures is considered in [20]. The positioning accuracy of the navigation object using a shortbase antenna is noticeably lower than the long-base one. The values range from 0.15 to 0.5% of the slant range (for NASDrill RS925 [21] and Kongsberg HPR 406 [22], respectively). For example, at a distance of 80 m according to [23], the root mean square and maximum value of the position estimation error with USBL are 1.64 m and 1.94 m respectively (2% and 2.4%). However, with the integrated use of APS data and the inertial measurement system (IMS), as it is widespread for unmanned

Fig. 3 Methods and systems of local navigation ROV

210

O. I. Gladkova et al.

aerial vehicles [24, 25], the corresponding measures of inaccuracy can be reduced to 0.19 m and 0.28 m, respectively (0.2% and 0.35%) [23]. Path reckoning methods based on measurements of the component velocity vectors (Doppler navigation) or accelerations (IMS) in a certain coordinate system and their subsequent integration or double integration over time to calculate the coordinates of the vehicle from known initial values have one major drawback—increasing inaccuracy of measurements, which is proportional to time. In this regard, the use of only Doppler or inertial methods (and compensating algorithms, e.g. [26]) for calculating the path over large time intervals (for example, when carrying out a complete hull fault detection, see Fig. 2a) is inexpedient. So, the use of hydroacoustic Doppler velocity log (DVL) in conjunction with LBL APS operating at higher frequencies and with a limited range of distances (100 m) reduced the positioning errors of the ROV during movement: with a trajectory in the form of a square with a side of 5 m, the positioning error was 0.1–0.2 m [20]. The use of visual odometry (including algorithmic processing of images from video cameras) makes it possible to achieve accuracy of the order of 1.5% of the distance covered. For example, due to the use of a stereo camera, according to [27], with the rectilinear motion of the vehicle along the pool wall 22.86 m long, the results of the reckonings were 21.49–22.17 m, which were within the experimental accuracy (average error of 0.46 m would correspond to 2% of the traveled distance). Approaches based on simultaneous localization and mapping (SLAM) have gained particular popularity today. The advantage of this approach is not only the ability to build a map of the surrounding space, but also the ability to estimate navigation parameters by comparing chart-based information generated at the current time and recorded in memory from the results of past measurements [10]. If for the construction of a 3D model of a ship’s hull due to 2D Imaging Sonar [28, 29], the accuracy of determining the coordinates is not critical, then for navigating the identified defects, as shown in [30], using SLAM and DVL together allows accuracy is much higher than for standard dead-reckoning: the maximum difference between the measured path and the distance traveled using SLAM algorithms at a distance of 2.5 km was 1.10 m, and using the dead-reckoning sensor because of the presence of sensor drifts—21.39 m. The use of the hybrid propulsion system makes it possible to use an odometer reckoning system, which is not typical for underwater vehicles but is standard for mobile ground robots. Within the framework of solving the problem of high-precision navigation fixation of measured structural defects, such systems allow to achieve accuracy theoretically limited only by the bit depth of the sensors used (encoders), since for them, unlike inertial measurement and Doppler systems, there is no increase of inaccuracy in measure with time. The results obtained in practice make it possible to use as a guideline the expected value of the measure of inaccuracy in the reckoning of the track due to odometer wheels of a few centimeters. In this work, the results of navigation measurements obtained through this approach are of particular interest. The measurement ranges for various types of navigation systems, as well as the presence of significant features and measure of inaccuracy for each of them, allow us to conclude that their integrated use (as it is shown in [31]) is necessary.

Development of an Information Control System for a Remotely …

211

Table 1 Options of ROV’s navigation systems Navigation system

Heading angle

APS autonomously

APSF (APS with low-pass filter)

Gyro-azimuth/odometer/video system

Odometer left autonomously

Odometer left with correction according to APSF

Gyro-azimuth/odometer/video system

Odometer right autonomously

Odometer right with correction by APSF

Average odometer autonomously

Average odometer with correction according to APSF

Visual odometer system autonomously

Visual odometry with correction by APSF

Gyro-azimuth/odometer/video system

SLAM autonomous

SLAM with correction according to APSF

Gyro-azimuth/odometer/video system

3.2 Configurations of ROV’s Navigation Systems The navigation problem with increased demands on the accuracy of determining the location of the vehicle on the ship hull is solved by the complex use of measurements of linear and angular displacements of the ROV. Measurement of the linear movements of the vehicle during the movement of the ROV on the wheels is carried out by one of the options or their combination: according to one of two or two odometers (dead-reckoning system), a television camera (visual odometry), an acoustic positioning system with a short baseline (similar to [32]) or using SLAM algorithms. The heading angle of the vehicle, which is the angle between the directions of the longitudinal axes of the ship and the ROV, is determined at one time point by one of the methods: integrating the readings of the precision angular velocity sensor (gyro azimuth heading angle), processing the data of two odometers (odometer heading angle), or calculating the course by the video system (the heading angle of the video system). Ultimately, to determine the coordinates of the ROV in the ship’s coordinate system, one of the twelve navigation systems can be selected (see Table 1). At the same time, for each block of navigation systems (using any of the reckoning systems, APS or SLAM using the indications of one of the dead-reckoning system), as well as for the control system, one of the three-course headings listed above can be selected.

3.3 Information-Measuring Complex of ROV In Bauman Moscow State Technical University developed ROV “Iznos” (Fig. 4), designed to survey ship hulls afloat. It has a modular design and includes the vehicle

212

O. I. Gladkova et al.

Fig. 4 ROV “Iznos”

with propellers itself, the chassis module with four wheels and the technical means of non-destructive testing. Based on the requirements for ROV hardware, the vehicle’s part of informationmeasurement complex includes (see Table 2):

Table 2 Technical specification of navigation sensors Sensor

Variable

Range

Resolution/parameter

Precision

Trax

Heading (compass mode)

0…360°

0.1º

Up to 0.3°

Roll

±180°

0.01º

Up to 0.2°

Pitch

±90°

Angular velocity (vertical axis of ROV)

120°/s

In run bias stability

0.3°/h

Angle random walk

√ 0.01°/ h

VG035PD

VG910D

Angular velocity (longitudinal and transverse axes of ROV)

D11

Pressure

LIR-DA235T

Angular displacement

240°/s

Scale factor error