New Trends in Medical and Service Robotics: MESROB 2020 [1st ed.] 9783030581039, 9783030581046

Citation preview

Mechanisms and Machine Science

Georg Rauter · Philippe C. Cattin · Azhar Zam · Robert Riener · Giuseppe Carbone · Doina Pisla   Editors

New Trends in Medical and Service Robotics MESROB 2020

Mechanisms and Machine Science Volume 93

Series Editor Marco Ceccarelli, Department of Industrial Engineering, University of Rome Tor Vergata, Roma, Italy Editorial Board Members Alfonso Hernandez, Mechanical Engineering, University of the Basque Country, Bilbao, Vizcaya, Spain Tian Huang, Department of Mechatronical Engineering, Tianjin University, Tianjin, China Yukio Takeda, Mechanical Engineering, Tokyo Institute of Technology, Tokyo, Japan Burkhard Corves, Institute of Mechanism Theory, Machine Dynamics and Robotics, RWTH Aachen University, Aachen, Nordrhein-Westfalen, Germany Sunil Agrawal, Department of Mechanical Engineering, Columbia University, New York, NY, USA

This book series establishes a well-defined forum for monographs, edited Books, and proceedings on mechanical engineering with particular emphasis on MMS (Mechanism and Machine Science). The final goal is the publication of research that shows the development of mechanical engineering and particularly MMS in all technical aspects, even in very recent assessments. Published works share an approach by which technical details and formulation are discussed, and discuss modern formalisms with the aim to circulate research and technical achievements for use in professional, research, academic, and teaching activities. This technical approach is an essential characteristic of the series. By discussing technical details and formulations in terms of modern formalisms, the possibility is created not only to show technical developments but also to explain achievements for technical teaching and research activity today and for the future. The book series is intended to collect technical views on developments of the broad field of MMS in a unique frame that can be seen in its totality as an Encyclopaedia of MMS but with the additional purpose of archiving and teaching MMS achievements. Therefore, the book series will be of use not only for researchers and teachers in Mechanical Engineering but also for professionals and students for their formation and future work. The series is promoted under the auspices of International Federation for the Promotion of Mechanism and Machine Science (IFToMM). Prospective authors and editors can contact Mr. Pierpaolo Riva (publishing editor, Springer) at: [email protected] Indexed by SCOPUS and Google Scholar.

More information about this series at http://www.springer.com/series/8779

Georg Rauter Philippe C. Cattin Azhar Zam Robert Riener Giuseppe Carbone Doina Pisla •

•

•

•

•

Editors

New Trends in Medical and Service Robotics MESROB 2020

123

Editors Georg Rauter Bio-Inspired RObots for MEDicine-Lab (BIROMED-Lab), Department of Biomedical Engineering University of Basel Allschwil, Switzerland Azhar Zam Biomedical Laser and Optics Group (BLOG), Department of Biomedical Engineering University of Basel Allschwil, Switzerland Giuseppe Carbone University of Calabria Arcavacata di Rende, Italy

Philippe C. Cattin Center for medical Image Analysis & Navigation (CIAN), Department of Biomedical Engineering University of Basel Allschwil, Switzerland Robert Riener Sensory-Motor Systems Lab ETH Zurich Zürich, Switzerland Doina Pisla Technical University of Cluj-Napoca Cluj-Napoca, Romania

ISSN 2211-0984 ISSN 2211-0992 (electronic) Mechanisms and Machine Science ISBN 978-3-030-58103-9 ISBN 978-3-030-58104-6 (eBook) https://doi.org/10.1007/978-3-030-58104-6 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

Medical and service robots face growing demands on their functionality and performance in a broad range of applications. Therefore, strengthening our community through interdisciplinary work is beneficial for all parties involved: researchers, technology providers, medical healthcare personnel, and most importantly patients. This year, methods from laser physics and virtual/augmented reality-based surgical planning have found their way to augment the functionality, possibilities, and safety of medical and service robots. This year, we had to face difficult circumstances due to the worldwide pandemic situation with COVID-19 that prevented us from realizing MESROB 2020, the 7th International Workshop on New Trends in Medical and Service Robotics, Basel, Switzerland. Nevertheless, our faithful community keeps supporting our successful story of MESROB workshops, which led to this book. The oral presentations of the accepted papers in these proceedings are planned for MESROB 2020, which should be finally held at the University Hospital Basel, Basel, Switzerland. The entire story of MESROB conference events started with the first of its kind in 2012 in Cluj-Napoca, Romania. Following events were MESROB 2013 at Institute “Mihailo Pupin” in Belgrade, Serbia; MESROB 2014 at EPFL in Lausanne, Switzerland; MESROB 2015 at IRCCyN in Nantes, France; MESROB 2016 co-organized by the University of Innsbruck and Joanneum Research in Graz, Austria, and MESROB 2018 at the School of Engineering of the University of Cassino and South Latium in Cassino, Italy. This workshop series is also sponsored by IFToMM, the “International Federation for the Promotion of Mechanism and Machine Science” and is one of the main conferences for the IFToMM Technical Committees on Biomechanical Engineering, Robotics and Mechatronics, and Computational Kinematics. The content of the MESROB 2020 book covers a wide range of aspects and topics such as 1) assistive devices, 2) surgical robotics, 3) lasers, planning, and navigation in surgery, 4) performance evaluation, 5) mobile and service robots, and 6) tissue modeling. These contributions are provided as a collection of 37 papers that were selected among the 49 submitted contributions on the basis of a blind peer-review process. v

vi

Preface

We wish to express our gratitude to the authors, the reviewers, and the scientific committee for their valuable contribution to ensure the scientific quality of MESROB 2020. July 2020

Georg Rauter Azhar Zam Philippe C. Cattin Robert Riener Giuseppe Carbone Doina Pisla

Organization

General Chair Georg Rauter

University of Basel, Switzerland

Co-conference Chairs Azhar Zam Philippe Cattin Robert Riener Giuseppe Carbone

University of Basel, Switzerland University of Basel, Switzerland ETH Zurich, Switzerland University of Calabria, Italy

Scientific Committee Bernard Bayle Hannes Bleuler Giuseppe Carbone Philippe Cattin Marco Ceccarelli Gery Colombo Carlo Ferraresi Nicolas Gerig Yeongmi Kim Jean-Pierre Merlet Domen Novak Constanze Pfeiffer Doina Pisla Annika Raatz Georg Rauter

University of Strasbourg, France École polytechnique fédérale de Lausanne, Switzerland University of Calabria, Italy University of Basel, Switzerland University of Rome Tor Vergata, Italy Self-employed, Switzerland Politecnico di Torino, Italy University of Basel MCI, Austria Inria, France University of Wyoming, USA DBE, Switzerland Technical University of Cluj-Napoca, Romania Institut für Montagetechnik, Leibniz Universität Hannover, Germany University of Basel, Switzerland

vii

viii

Daniela Tarnita Philippe Wenger Akio Yamamoto Azhar Zam

Organization

University of Craiova, Romania CNRS, France University of Tokyo, Japan University of Basel, Switzerland

Contents

Assistive Devices Trajectory Planning and Fuzzy Control of a Hand Exoskeleton for Assisted Rehabilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Berith A. De La Cruz-Sánchez, Manuel Arias-Montiel, and Esther Lugo-González Design of a Novel Robot for Upper Limb Rehabilitation . . . . . . . . . . . . Elio Matteo Curcio and Giuseppe Carbone Comfort Perception Analysis of Human Models Interfacing with Novel Biped-Wheeled-Exoskeletons . . . . . . . . . . . . . . . . . . . . . . . . Angelica Zoccali and Giovanni Gerardo Muscolo Ankle Rehabilitation of Stroke Survivors Using Kuka LBR Iiwa . . . . . . Paul Tucan, Ionut Ulinici, Nicoleta Pop, Ferenc Puskas, Giuseppe Carbone, Bogdan Gherman, Iosif Luchian, and Doina Pisla A Compliant Parallel Manipulator for Trunk Rehabilitation After Stroke . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Diaz-Caneja, Francisco J. Campa, Oscar Altuzarra, Mikel Diez, Ion Lascurain-Aguirrebeña, Leire Santisteban, and Ana Bengoetxea Design and Motion Analysis of an Exoskeleton Robot for Assisting Human Locomotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ionut Geonea, Nicolae Dumitru, Daniela Tarnita, Cristian Copilusi, and Adrian Sorin Rosca Alternative Methods for Direct Kinematic Analysis of a Parallel Robot for Ankle Rehabilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erick D. Flores-Salazar, Manuel Arias-Montiel, Esther Lugo-González, Jaime Gallardo-Alvarado, and Ricardo Tapia-Herrera

3

12

21 29

37

44

53

ix

x

Contents

First Clinical Evaluation of a Spherical Robotic System for Shoulder Rehabilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Calin Vaida, Ionut Ulinici, Alexandru Banica, Alin Burz, Bogdan Gherman, Paul Tucan, Adrian Pisla, Giuseppe Carbone, and Doina Pisla Experimental Characterization of a Cable-Driven Device for Elbow Motion Assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arnaud Kozisek, Marco Ceccarelli, Med Amine Laribi, and Lucia Ferrara

62

71

Surgical Robotics Injection Device for Percutaneous Osteoplasty . . . . . . . . . . . . . . . . . . . . Julien Garnon, Laurence Meylheuc, Léo Harrer, Guillaume Koch, Afshin Gangi, and Bernard Bayle A New Correction Coefficient Formula for the Simplified Dynamic Model of a Surgical Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Orhan Ayit and Mehmet İsmet Can Dede Designing Adaptive Nonlinear Controller for Optimal Tracking of Laparoscopic Robotic Arm with Nonholonomic Constraints . . . . . . . Amir Aminzadeh Ghavifekr, Roushanak Haji Hassani, and Andrea Calanca

81

89

97

Manipulation of an Wide Angle Endoscope in Minimally Invasive Robotic Surgery and Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 John Mannion and Yeongmi Kim Tendon Force Control Evaluation for an Endoscope with Series Elastic Actuation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Lorin Fasel, Nicolas Gerig, Philippe C. Cattin, and Georg Rauter Design Evaluation of a Stabilized, Walking Endoscope Tip . . . . . . . . . . 127 Manuela Eugster, Melanie Oliveira Barros, Philippe C. Cattin, and Georg Rauter Experimental Evaluation of Needle Tip Force Sensing Associated to Tactile Feedback for Improving Needle Remote Insertion . . . . . . . . . 136 Charlélie Saudrais, Lennart Rubbert, Lisa Bonnefoy, Rui Zhu, Hubert Schneegans, Charles Baur, Ulrich Mescheder, and Pierre Renaud A Compliant Mechanism as a Sternum Prosthesis . . . . . . . . . . . . . . . . . 143 Octavio Ramirez, Christopher R. Torres-San-Miguel, Marco Ceccarelli, José Luis Rueda Arreguín, and Guillermo Urriolagoitia-Calderón Design and Lab Experiences for a Fixator of Rib Fractures . . . . . . . . . 152 Ludovica Sommariva, Josè Luis Arreguin, Cuauhtémoc Morales Cruz, Marco Ceccarelli, Vincenzo Ambrogi, and Lucrezia Puglisi

Contents

xi

Learned Task Space Control to Reduce the Effort in Controlling Redundant Surgical Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Murali Karnam, Manuela Eugster, Riccardo Parini, Philippe C. Cattin, Elena De Momi, Georg Rauter, and Nicolas Gerig Design, Static and Performance Analysis of a Parallel Robot for Head Stabilisation in Vitreoretinal Surgery . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Hans Natalius, Patrice Lambert, Manish K. Tiwari, Lyndon da Cruz, and Christos Bergeles Lasers, Planning, and Navigation in Surgery Multimodal Risk-Map for Navigation Planning in Neurosurgical Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 Maximilian Gerst, Christian Kunz, Pit Henrich, and Franziska Mathis-Ullrich Augmented Reality Based Surgical Navigation of the Periacetabular Osteotomy of Ganz – A Pilot Cadaveric Study . . . . . . . . . . . . . . . . . . . . 192 Armando Hoch, Florentin Liebmann, Fabio Carrillo, Mazda Farshad, Stefan Rahm, Patrick O. Zingg, and Philipp Fürnstahl Optoacoustic Tissue Classification for Laser Osteotomes Using Mahalanobis Distance-Based Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Hervé Nguendon Kenhagho, Yakub Aqib Bayhaqi, Ferda Canbaz, Raphael Guzman, Tomas E. Gomez Alvarez-Arenas, Philippe C. Cattin, and Azhar Zam Simulation of Echellogram Using Zemax OpticStudio and Matlab for LIBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 Hamed Abbasi, Negin Sahraei, Ferda Canbaz, Philippe C. Cattin, and Azhar Zam Robot- and Laser-Assisted Bio-Sample Preparation: Development of an Integrated, Intuitive System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Cédric Duverney, Hamed Abbasi, Lina M. Beltrán Bernal, Tino Stauber, Jess G. Snedeker, Philippe C. Cattin, Azhar Zam, and Georg Rauter Modelling and Performance Evaluation Lab Experiences on Impact Biomechanics of Human Head . . . . . . . . . . 229 José Luis Rueda Arreguín, Marco Ceccarelli, Christopher R. Torres-San-Miguel, and Cuauhtémoc Morales Cruz Nonlinear Dynamic Analysis of Human Sit-to-Stand Movement with Application to the Robotic Structures . . . . . . . . . . . . . . . . . . . . . . 238 Daniela Tarnita, Alin Petcu, Marius Georgescu, Ionut Geonea, and Danut Tarnita

xii

Contents

Human Squat Motion: Joint Torques Estimation with a 3D Model and a Sagittal Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Olivier Bordron, Clément Huneau, Éric Le Carpentier, and Yannick Aoustin Visuo-Otolithic and Electrodermal Interactions in Experimental 3D Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 Irini Giannopulu, A. Pisla, and D. Pisla Development and Characterization of a Versatile, Force-Range Adjustable, Low-Cost, Tri-Axial Force Sensor . . . . . . . . . . . . . . . . . . . . 265 Ivan Sušić, Philippe C. Cattin, Raphael Guzman, and Georg Rauter Development of an Automatic Perturbator for Dynamic Posturographic Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 Carlo Ferraresi, Carlo De Benedictis, Giovanni Gerardo Muscolo, Oliviero Walter Pica, Marco Genovese, Daniela Maffiodo, Walter Franco, Maria Paterna, Silvestro Roatta, and Zeevi Dvir An Indirect Method Based on Capture Data Is Usable for Muscle Fatigue Treatment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Olfa Jemaa, Sami Bennour, David Daney, and Lotfi Romdhane Daily Life Activities Analysis for Rehabilitation Purposes . . . . . . . . . . . 290 Ferdaws Ennaiem, Abdelbadiâ Chaker, Med Amine Laribi, Juan Sandoval, Sami Bennour, Abdelfattah Mlika, Lotfi Romdhane, and Saïd Zeghloul Method for 3D Modeling of Human Osteo-Articular System Bones . . . . 298 Cristian Copilusi, Nicolae Dumitru, Alexandru Margine, Ionut Geonea, Adrian Sorin Rosca, and Eugen Rosu Mobile and Service Robots Human Gait Analysis Using Non-invasive Methods with a ROS-Based Mobile Robotic Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 Diego Guffanti, Alberto Brunete, and Miguel Hernando Gutierrez Introducing a Modular Framework for Human Tracking with Inhomogeneous Sensor Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 Nils Mandischer, Mathias Huesing, and Burkhard Corves Mobile Robot with Wheeled-Legs for Inspection of Pipes with Variable Diameter and Elbow Shapes . . . . . . . . . . . . . . . . . . . . . . 325 Eduardo Castillo-Castañeda and Ana Rocío Córdoba-Malaver Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333

Assistive Devices

Trajectory Planning and Fuzzy Control of a Hand Exoskeleton for Assisted Rehabilitation Berith A. De La Cruz-S´ anchez1(B) , Manuel Arias-Montiel2 , and Esther Lugo-Gonz´ alez2 1

2

Division of Postgrade Studies, Universidad Tecnol´ ogica de la Mixteca, Oaxaca, Mexico [email protected] Institute of Electronics and Mechatronics, Universidad Tecnol´ ogica de la Mixteca, Oaxaca, Mexico {mam,elugog}@mixteco.utm.mx

Abstract. One of the current trends within service robotics is the development of exoskeletons for rehabilitation. It has been reported in literature that the use of exoskeletons can help to solve some problems relating to repetitive tasks that involve rehabilitation therapies. The present paper deals with dynamic modeling and the development of a controller based on fuzzy logic for the trajectory tracking that describes basic movements used in rehabilitation therapy. The ranges of movement are obtained by video analysis software and the paths are designed using B´ezier curves. The proposed controller is evaluated numerically with simulations using the obtained dynamic equations.

Keywords: Exoskeleton

1

· Fuzzy logic · Trajectory planning.

Introduction

In recent years, the development of robotic systems to support rehabilitation has increased considerably. It is possible to find options for these type of devices for different parts of the body in the market. The importance of improving the function of the hand after suffering an injury has stimulated the increase in the development of exoskeletal devices at a commercial and research level. Reports of the control of these devices are scarce in literature. An example of an alternative technique that works for controlling these devices is the fuzzy logic controller reported in [1,2]. The application of fuzzy logic in the design and implementation of control systems is one of the main areas of advancements. The main advantage in the use of fuzzy logic systems is the approximation of its behavior where there are no analytical functions or numerical relationships as well as having the capacity to understand biological, medical, social, economic or political systems [3]. The use of artificial intelligence (AI) in the area of rehabilitation robotics has increased substantially c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 3–11, 2021. https://doi.org/10.1007/978-3-030-58104-6_1

4

B. A. De La Cruz-S´ anchez et al.

in recent years. An example of this is the development of algorithms for fuzzy logic which allow the control of devices and classify signals from the body, thus providing information and robustness. The control of the devices developed at a research level has a tendency to focus on the mechanism position. The implementation of its basics control systems does not consider safety, comfort, or the correct accomplishment of the patient’s rehabilitation tasks. These data are not reported in literature [4]. The selection and application of a control technique must depend on the focus and type of routine for the rehabilitation that will be implemented in the device. It must be carried out in conjunction with the implementation of an instrumentation system that acquires the necessary variables to develop precise control and functionality for the device [5]. This work presents the dynamic model and the control based on fuzzy logic of a hand exoskeleton for assisted rehabilitation. The proposed trajectories take into account the basic movements used in rehabilitation therapies. The ranges of movement are obtained by video analysis software. The paths are designed using B´ezier polynomials in order to obtain smooth profiles for position, velocity and acceleration of the mechanism.

2

Exoskeleton Design

The exoskeleton is designed for independent movement of each of the fingers. The transmission of power for the movement of each phalanx is carried out through a pinion-rack mechanism while the prismatic and angular movements are performed through gear-rack kits. These two movements give the mechanism the ability to follow instant finger velocity centers, which minimizes the error that exists between the instantaneous center of velocity of the mechanism with regard to the center of rotation of the phalanges. This allows coverage of the movement ranges of a healthy hand and avoid the mechanical interference between the exoskeleton and the user’s finger. Each finger of the exoskeleton was divided into three blocks. The first block is located on the metacarpal bone and secured to the base of the exoskeleton. This block is responsible for the movement of the proximal phalanx. The second block is located on the proximal phalanx and is responsible for the movement of the middle phalanx. The last block operates as a receptor of the movement and is located on the distal phalanx. The exoskeleton has 2 active and 2 passive Degrees of Freedom (DOF) for the index, middle and annular fingers, while the thumb and the little finger have 1 active and 1 passive DOF. The movement of each finger is controlled independently and the weight of the mechanical prototype is 750 g. Figure 1 shows a partial view of the exoskeleton. Details about design and kinematic analysis of the proposed hand exoskeleton are presented in [6].

Trajectory Planning and Fuzzy Control of a Hand Exoskeleton

5

Fig. 1. Final prototype assembly

2.1

Dynamic Model

For each degree of freedom there is an actuator. For each phalanx, a motor is responsible for the movement that allows the finger to be modelled as a planar manipulator with 2 DOF. The exoskeleton finger model is shown in Fig. 2.

Fig. 2. Arm model of 2 degrees of freedom.

Using the Euler-Lagrange method [7], the dynamic model is obtained and it can be written in the general form as: τ = M (q)¨ q + C(q)q˙ + g(q)

(1)

where: q is the angular displacement of the joint; M (q) is the inertia matrix of the manipulator; C(q) is the centrifugal force and the Coriolis force; G(q) is the gravity term. q1 and q2 represent the angular displacement of the proximal and middle phalanx.

3

Modeling of Trajectories

The basic hand movements considered during the development of the project and found during rehabilitation therapies are: fist, cylindrical grip and tip pinch

6

B. A. De La Cruz-S´ anchez et al.

[8]. For the cylindrical grip exercises the user takes a plastic cylinder with a diameter of 8 cm and a length of 12 cm. For the development of the tip pinch the user takes a marker of trade with the tips of the thumb and index finger. The reference for the system’s tracking problem is obtained using B´ezier curves and with the help of a Vicon motion capture system. This determines the starting and ending positions for the phalanges in each movement to be performed. In each phalanx a mark was placed to follow its trajectory in cartesian coordinates; a reference mark is positioned on the wrist. The tests performed are available in [9] as a free download. Rotation matrices were applied to the data to analyze them in an XZ plane, and this data was processed in Matlab. The results of different tests for the angles for each phalanx in the three movements are shown in Table 1. The data values are in degrees (◦ ). Table 1. Phalange movement ranges for three movements. Movement

Phalanx

Fist

Proximal 54.1086 38.6238

60.2354

33.1456

54.1086

Middle

–

79.8522

87.4303

62.9804

Distal

62.9804 81.3978

76.847

81.3978

83.4123

Proximal 54.8130 54.8130

61.4474

61.5974

54.8130

Middle

–

45.1992

45.5139

49.8300

45.1992

Distal

Tip pinch

Thumb Index finger Middle finger Ring finger Little finger 82.7016

50.3636 50.3836

52.9135

50.0200

50.3636

Cylindrical Proximal 35.2300 17.8251

2.0200

1.9800

1.9500

Grasp

Middle

–

49.1760

3.0200

3.4500

3.3800

Distal

44.9800 37.2291

2.9801

2.9800

3.0200

With the data obtained from the final angle for each of the phalanges, it is possible to determine the trajectories that the exoskeleton should follow. In order to accomplish this, the use of B´ezier curves is proposed. A soft path will prevent injury to the user and comply with the desired position of a safe way for the patient and the device [10]. The B´ezier curve can be obtained through equation (2), [11]. n    n (2) Pi (i − t)n−i ti B(t) = i i=0 where: P0 , Pn are the starting and ending points of the B´ezier curve; P1 ...Pn−1 are the control points of the B´ezier curve and t is the parameter that influences the distribution of the interpolation of the points. To obtain the trajectory, the B´ezier polynomial was implemented in Simulink, where the parameters to be introduced are given by Table 1. These desired displacement values for the proximal and middle phalanx are represented in the system as the variables q1d and q2d .

Trajectory Planning and Fuzzy Control of a Hand Exoskeleton

4

7

Fuzzy PD Controller

Fuzzy logic control was based on the patterns in [12], where it is part of a classic PD that combines the error and change in error which results in a fuzzy version of PD controller. A closed loop is needed in the PD controller to perform the comparison of the reference input with the output produced by the controller. In addition, a controller can be MIMO or SISO. The typical SISO controller regulates a control signal according to an error signal. The exoskeleton system contains a motor for the movement of each phalanx which results in the ability to have a decentralized SISO control type for the movement of each of the motors. For the implementation of the fuzzy control, the inputs of the system are the reference values in degrees that each of the phalanges has to reach during movement. The angular position of the control systems output is subtracted from the angular position of reference in the input, thus obtaining an error from which it is possible to determine its variation in time. 4.1

Fuzzification

The fuzzy type controllers have linguistic variables as input. For this controller, the variables are error and change ratio. At this point in the driver development, the input variables remain membership functions. The output of the controller is a value corresponding to the high cycle of the pulse width modulator (PWM), which controls the speed of the actuator. The functions that model linguistic terms are trapezoidal, and the error is defined betwen [−1, 1]. Each encoder reads the error that is updated and normalized. The PWM signal will be generated by an Arduino board, which works on 8-bit resolution [0–255] equivalent to 0–100% of the duty cycle. For the undertaking of the tests the maximum duty cycle is 21%, which is equivalent to 8 bits in range of [−53, 53]. Similarly, the same linguistic terms of Table 2 are applied. Table 2. Linguistic terms and Fuzzy rules. Linguistic variable Abbreviation Linguistic term

Rules

Error

NE ZE PE

Negative error Zero error Positive error

If (NE) and (NC) Then (NV) If (NE) and (ZC) Then (NV) If (NE) and (PC) Then (NV)

Change

NC ZC PC

Negative change If (ZE) and (NC) Then (NV) Zero change If (ZE) and (ZC) Then (ZV) Positive change If (ZE) and (PC) Then (PV)

Voltaje

NV ZV PV

Negative voltaje If (PE) and (NC) Then (PV) Zero voltaje If (PE) and (ZC) Then (PV) Positive voltaje If (PE) and (PC) Then (PV)

8

4.2

B. A. De La Cruz-S´ anchez et al.

Fuzzy Rules

To relate entries with the outputs it is necessary to establish input groups of fuzzy rule combinations. Since the system has two entries with three linguistic terms each, a total of 9 rules are obtained. The rules can be seen in Table 2. 4.3

Inference System

The Mandani method was used as the inference method for the fuzzy intersection. The Matlab Fuzzy Logic Toolbox was used for this phase because it is a practical tool for the calibration of a diffuse system due its graphical interface and the possibility of evaluating the results with the sliding surface. The sliding surface obtained from this diffuse system is shown in Fig. 3. The location of the parameters of the membership functions becomes a less complicated task, otherwise the adjustment of these parameters must be tested with an error of the real system.

Fig. 3. Sliding surface obtained for the linguistic variables of the system.

5

Results

The simulation of the fuzzy controller was carried out using the Simulink software, taking into account that the dynamic equation (1) was used for simulation purposes. It is worth noting that at this point in the simulation, the dynamics of the motor were not taken into account as the controller’s output is obtained through the torque corresponding to the motor. Figure 4 shows the Simulink block diagram for the control test of the system. The generation of the trajectory in the physical system will be determined by the recognition of electromyographic signals for the detection of the movements. It is later reproduced by the mechanism, wich takes the form of a type of bilateral assisted therapy. The advantage of having DC motors for each of the phalanx movements is the use of separate fuzzy controllers for each of the motors. The movement of a grip cylinder and fist that the proximal and middle phalanx of the index finger performs are shown in Figs. 5 and 6.

Trajectory Planning and Fuzzy Control of a Hand Exoskeleton

Fig. 4. Control implementation block diagram in Simulink.

Fig. 5. System response for cylindrical grip.

Fig. 6. System response for fist.

9

10

B. A. De La Cruz-S´ anchez et al.

The fuzzy controller allowed the tracking of the trajectory that has been introduced to the model of the exoskeleton. The response of the system with the controller has been successful. The results obtained were applied to the model of a finger of the exoskeleton. The replication of this control is applied to each one of the fingers because they move independently. For the tests performed, the average error generated by the five fingers corresponds to 2.1% for the proximal phalanx and 2.4% for the middle phalanx, its equivalent in degrees is 1.19◦ and 1.04◦ correspondingly. With some previous tests carried out with different classic controllers like the PD, approximate variations in the response of 0.2◦ were obtained compared to the results of a fuzzy controller. This technique was implemented because the gains are adaptable without the need to re-synchronize them in their future physical implementation, unlike other control techniques where the gain tuning becomes a complicated task.

6

Conclusions

The implementation of fuzzy controllers represents an advantage for systems in which obtaining the model is tedious or in which it fails to accurately represent the system. Obtaining the dynamic model of the exoskeleton is an approximation which can make a big difference to the real system, as there are components that have been generalized. Contrary to other control techniques based on a model whose parameters vary in a real implementation, the technique chosen for this paper shows the guidelines for the development of a decentralized fuzzy controller that allows the control of both motors in order to obtain more natural movements.

References 1. Irastorza-Landa, N., et al.: Design of continuous EMG classification approaches towards the control of a robotic exoskeleton in reaching movements. In: Proceedings of 2017 International Conference on Rehabilitation Robotics (ICORR), IEEE, pp. 128–133 (2017) 2. Silva, A.F., et al.: Fuzzy control of a robotic finger actuated by shape memory alloy wires. J. Dyn. Syst. Meas. Control 140(6), 064502 (2018) 3. De Silva, C.W.: Intelligent Control: Fuzzy Logic Applications. CRC Press, New York (2018) 4. Basteris, A., Nijenhuis, S.M., et al.: Training modalities in robot-mediated upper rehabilitation in stroke: a framework for classification based on a systematic review. J. Neuroeng. Rehabil. 11(1), 1–15 (2014) 5. Leonardis, D., Barsotti, M., Loconsole, C., et al.: An EMG-controlled robotic hand exoskeleton for bilateral rehabilitation. IEEE Trans. Haptics 8(2), 140–151 (2015) 6. De la Cruz S´ anchez, B., Arias Montiel, M., Lugo-Gonz´ alez, E.: Development of hand exoskeleton prototype for assisted rehabilitation. In: Gasparetto, A., Ceccarelli, M. (eds.) Mechanism Design for Robotics. Mechanisms and Machine Science, vol. 66, pp. 378–385. Springer, Cham (2019) 7. Jazar, R.N.: Theory of Applied Robotics: Kinematics, Dynamics, and Control, 2nd edn., p. XXIII, 883. Springer, USA (2010)

Trajectory Planning and Fuzzy Control of a Hand Exoskeleton

11

8. Cailliet, R.: Hand Pain and Impairment, vol. 3. FA Davis Company (1971) 9. De la Cruz S´ anchez, B., Arias Montiel, M., Lugo Gonz´ alez, E.: MOCAP data base for three hand movements. Mendeley Data (2020) 10. Xu, Z., Wei, S., Wang, N., Zhang, X.: Trajectory planning with Bezier curve in cartesian space for industrial gluing robot. In: Zhang, X., Liu, H., Chen, Z., Wang, N. (eds.) Intelligent Robotics and Applications. ICIRA 2014. Lecture Notes in Computer Science, vol. 8918, pp. 146–154. Springer, Cham (2014). https://doi. org/10.1007/978-3-319-13963-0 15 11. Ju Mei, Z., and Hong Lun,W.: Inserting control points for Bezier curve approximation. In: Proceedings of the 31st Chinese Control Conference, IEEE, pp. 2416–2420 (2012) 12. Jantzen, J.: Foundations of Fuzzy Control: A Practical Approach. Wiley, Chichester (2013)

Design of a Novel Robot for Upper Limb Rehabilitation Elio Matteo Curcio and Giuseppe Carbone(B) DIMEG, University of Calabria, Rende, Italy [email protected]

Abstract. This paper addresses the design of a novel robotic device for upper limb rehabilitation tasks. The main goals of the design process have been to achieve a design of a rehabilitation device, which can be easily portable also for home use. Specific attention has been devoted to design of the main mechatronic components by developing specific kinematics and dynamics models. The design process includes the implementation of a specific control hardware and software. Preliminary experimental tests are reported to show the effectiveness and feasibility of the proposed design solution. Keywords: Design · Simulations · Parallel robots · Upper limb rehabilitation

1 Introduction Robotic assisted rehabilitation is a relatively young and rapidly growing field with increasing applications in clinical care, as reported for example in [1–10]. The development of robots for the rehabilitation of the upper limbs are characterized by a complexity that is increasing with the movement to be simulated, [1]. Most famous commercial upper limb rehabilitation robots are the MIT-MANUS, [2], now sold as InMotion ARM (Interactive Motion Technologies, USA) and ARMin (now sold as ArmeoPower by Hocoma AG, Switzerland), [3], but also two other models used today in clinics: ArmeoSpring [4] and ReoGo [5]. The MANUS is a five-bar SCARA robot with only two translational degrees of freedom (DOFs) for the movement of the elbow and forearm. Motorska’s ReoGo is a robotic system for upper limb therapy with up to 3 DOFs. ArmeoSpring is a passive orthosis for upper limb. It offers a large 3D workspace, where one can detect the 3D position and gripping force of the arm. ArmeoPower is a motorized version of ArmeoSpring, with 6 DOFs, capable of supporting the weight of a patient’s arm and assisting the patient during specific exercises, adapting to the patient’s capabilities, in a large 3D workspace, [4]. Several researchers are still researching and developing novel design solutions for limb rehabilitation, including exoskeletons, or new kinematic architectures or cable driven parallel architectures as for example in [6–12]. This paper, addresses this open problem by proposing a new design solution that can be easily portable also for home use and at same time have performances similar to © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 12–20, 2021. https://doi.org/10.1007/978-3-030-58104-6_2

Design of a Novel Robot for Upper Limb Rehabilitation

13

existing design solutions. Accordingly, a specific design process is herewith outlined by focusing at main mechanical components. Preliminary experimental tests are reported to show the effectiveness and feasibility of the proposed design solution.

2 Requirements of the Proposed Device A preliminary analysis was conducted on the characteristics of existing commercial devices for upper limb rehabilitation. A comparison has been made on basis of costs, workspace and payload. Results are summarized in Table 1. Given the high costs, none of the existing devices can be considered suitable for home use. Among the existing devices ReoGo is the cheapest solution with 3 active DOFs and a payload of 5 kg. Accordingly, further studies and clinical tests have been made with a ReoGo device, which authors tested as courtesy of ANMIC rehabilitation clinic in Crotone, Italy, Fig. 1. Although the most promising commercial product, ReoGo is too bulky and expansive for a home use. This gives the motivation for designing a novel robotic device to be much more compact, lightweight and cheap while keeping a workspace and payload comparable with the ReoGo device.

price weight Work- X space Y Z payload

$ 100.000 83 Kg 400 mm 400 mm Not allowed ~5 Kg

$ 85.000 79 Kg 400 mm 400 mm 200 mm ~5 Kg

Risolution DOFs Arm motions

Not available 2DOF Planar movements

Not available 3 DOF 3D movements

$62.500 82 Kg 600 mm 400 mm 600 mm ~2.5 Kg (forearm) ~3.8 Kg (arm) 10 ml) volume of cement, which is hardly achievable with manual injection due to the rapid onset of polymerization (10– 15 min). Hence, injection of a large volume of cement might benefit from the device with a passive exchanger used to cool down the PMMA. Ideally, the device has to be adapted to be compatible with larger syringes (30 ml would be adapted to the real clinical practice). This adaptation comes however with specific issues that need to be addressed such as the pressure of injection, the nature and thickness of the insulation material and the design of the syringe that might impact the shear rate of cement and therefore its viscosity. Another advantage of the device is the possibility for the operator to perform the injection distant from the source of X-ray, which is associated with a decrease of the scattered radiation for the interventional radiologist. This is of utmost importance especially for cases where large amount of cement is injected, and for which continuous fluoroscopic monitoring is performed for several minutes.

5 Conclusion Extra-spinal cementoplasty of volumes of cement greater than 10 ml using an injection device designed for spine is feasible, but technically demanding due to the need for manipulations during syringe exchanges. Development of an injection device adapted for the delivery of larger volume of cement (up to 30 ml) would theoretically allow to inject one single cohesive PMMA ball while offering a prolonged time for injection

88

J. Garnon et al.

that is mandatory to carefully check progressive filling of the lesion and avoid cement leakage.

References 1. Galibert, P., Deramond, H., Rosat, P., Le Gars, D.: Preliminary note on the treatment of vertebral angioma by percutaneous acrylic vertebroplasty. Neurochirurgie 33, 166–168 (1987) 2. Wang, Z., et al.: CT fluoroscopy-guided percutaneous osteoplasty for the treatment of osteolytic lung cancer bone metastases to the spine and pelvis. J. Vasc. Interv. Radiol. 23, 1135–1142 (2012). https://doi.org/10.1016/j.jvir.2012.06.007 3. Lepoutre, N., Bara, G.I., Meylheuc, L., Schmitt, F., Barbé, L., Bayle, B.: Design and control of a thermal device for bone cement injection. In: 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 508–513 (2016). https://doi.org/10. 1109/BIOROB.2016.7523677 4. Lepoutre, N., et al.: Robotically assisted injection of orthopedic cement: system design, control and modeling. In: 2016 European Control Conference (ECC), pp. 2127–2132 (2016). https:// doi.org/10.1109/ECC.2016.7810606 5. Lepoutre, N., Meylheuc, L., Bara, G.I., Barbé, L., Bayle, B.: Bone cement modeling for percutaneous vertebroplasty. J. Biomed. Mater. Res. Part B Appl. Biomater. 107, 1504–1515 (2019). https://doi.org/10.1002/jbm.b.34242 6. Zaharie, D.T., Phillips, A.T.M.: Pelvic construct prediction of trabecular and cortical bone structural architecture. J. Biomech. Eng. 140 (2018). https://doi.org/10.1115/1.4039894 7. Turner, C.H., Rho, J., Takano, Y., Tsui, T.Y., Pharr, G.M.: The elastic properties of trabecular and cortical bone tissues are similar: results from two microscopic measurement techniques. J. Biomech. 32, 437–441 (1999). https://doi.org/10.1016/s0021-9290(98)00177-8 8. Kühn, K.-D.: PMMA Cements. Springer Medizin, Berlin (2014)

A New Correction Coefficient Formula for the Simplified Dynamic Model of a Surgical Robot ˙ Orhan Ayit(B) and Mehmet Ismet Can Dede Izmir Institute of Technology, Izmir, Turkey [email protected], [email protected]

Abstract. Execution of model-based control algorithms such as computed torque technique requires the knowledge of the dynamic model of the robotic system. In our work, the active part of the surgical robot, NeuRoboScope, has a parallel kinematics architecture and the dynamic model is relatively complicated to run in a microprocessor with limited computing capabilities. For this reason, we formulated a simplified dynamic model to run in the selected microprocessor. In this work, a new formula for calculating the correction coefficients is described to minimize the errors in the whole orientation range of the robot’s base platform. This new formula is examined in terms of execution time and the result is reported in this paper.

Keywords: Surgical robot robot

1

· Dynamic modeling · Endoscope holder

Introduction

In the last 30 years, the minimally invasive surgery method has become prominent due to advantages over traditional open surgery such as less pain, fast recovery and small or no scarring [2,8]. In this method, medical instruments are inserted through one or more small incision/s and an imaging system (usually termed as the endoscope) is used to observe the operation space during surgery. While the endoscope is handled by the surgeon in the conventional minimally invasive surgery procedures, in the last two decades, robotic endoscope holders have been developed. These robotic holders are incorporated in the surgery to eliminate some drawbacks of manually handling an endoscope such as surgeon’s fatigue and tremor, and miscommunication with an assistant surgeon that handles the endoscope [3]. Hence, the performance and safety measure of minimally invasive surgeries are improved by the use of robotic endoscope holders [9]. Supported by The Scientific and Technological Research Council of Turkey via grant number 115E726. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 89–96, 2021. https://doi.org/10.1007/978-3-030-58104-6_11

90

˙ C. Dede O. Ayit and M. I.

A robotic endoscope holder, NeuRoboScope, was designed in Izmir Institute of Technology Human and Robot interaction laboratory (HuR) for use in a minimally invasive pituitary tumor surgery. The NeuRoboScope robot has two parts which are labeled active part and passive part as it is presented in Fig. 1. The active part is a 3 degree of freedom (DoF) remote center of motion mechanism (RCM) with parallel kinematics. The active arm controls the direction and translation of the endoscope during the surgery. The rotation angles about the remote center of motion (which is also called the pivot point) that define the direction of the endoscope are φ and ψ. The translational motion is defined by the d distance measured from the pivot point along the endoscope’s telescope axis [10]. The passive part is a 6 DoF serial manipulator that carries the active part. The first joint of the passive part is an active prismatic joint which is used only in the beginning of the surgery to level the active part’s base platform with respect to the patient. The passive arm is statically balanced. The rest of the passive arm has 5 passive revolute joints with brakes and angular position sensors. When brakes are in off-states, the passive part becomes back-drivable. Therefore, the passive part is moved for locating the active arm properly in the surgical workspace.

Fig. 1. The left part shows the active part of the NeuRoboScope and the right part shows the passive part of the NeuRoboScope. p shows the pivot point of active arm and the shown coordinate frame is right-handed.

A preferred control algorithm (computed torque method/feedback linearization) of NeuRoboScope requires to know the dynamic model of the active part. For executing the control algorithm with high frequency, a simplified dynamic model was proposed in [1]. According to this previous work, the dynamic model was designed with respect to 3 assumptions. In the first assumption, inertia matrices of the links and actuators are neglected. In the second assumption, in addition to the first assumption, masses of the links and actuators are neglected.

A New Correction Coefficient Formula for the Simplified Dynamic Model

91

In the third assumption, in addition to the previous assumptions, Coriolis and centripetal terms in the rigid body dynamic equation are neglected. By using these assumptions, the simplified dynamic model was derived as follows; ˆ corr (M ˆ (¯ ¯ q )) T¯ = K q )¨ q + G(¯

(1)

ˆ = JˆT (Vˆ ∗ )T mend (Vˆ ∗ )Jˆ + Jˆw T (R ˆ T + Iˆact )Jˆw ˆ 2 Iˆend R M end end 2

(2)

¯ = −JˆT F¯end G

(3)

ˆ, G ¯ and q¯ are defined as generalized torque vector, generalized where T¯, M mass matrix, generalized gravity vector, and vector of generalized coordinates  T ˆ corr is a 3 by 3 diagonal matrix with constant (¯ q = θ1 θ2 θ3 ), respectively. K correction coefficients K1 , K2 and K3 . mend and Iend are mass and inertia matrix of the endoscope. Jˆ and Jˆw refer to Jacobian and modified Jacobian matrices as defined below. ⎡˙ ⎤ ⎡ ⎤ ⎡ ˙⎤ ⎡˙ ⎤ θ1 φ θ1 φ˙ ˆ ⎣θ˙2 ⎦ , ⎣ψ˙ ⎦ = J ˆw ⎣θ˙2 ⎦ ⎣ψ˙ ⎦ = J (4) 0 d˙ θ˙3 θ˙3 ∗ Vˆend is a matrix which relates the linear velocity of the endoscope’s center of ˙ ψ, ˙ mass (CoM) to the endoscope’s velocities defined about the pivot point (φ, ˙ ˆ ¯ ¯ d). R2 is a rotation matrix, which is explained in the next section. Lend and Vend refer to the position and linear velocity of the endoscope’s CoM. ⎡ ˙⎤ φ ¯ end ) ˆ2L d( R ∗ ⎣ ˙⎦ (5) V¯end = → V¯end = Vˆend ψ dt d˙

The inertia matrix of the actuator Iˆact appears in Eq. (2) is the modified inertia matrix of the actuator which is located on the middle link. This actuator is not fixed to the active part’s base and rotates about the x-axis of the frame denoted in Fig. 1 (frame 1). Hence, its rotation about the x-axis is to be included in the dynamic equation. Consequently, the only non-zero element in the modified inertia matrix of the actuator is the top left element (corresponding to the moment of inertia about the x-axis originating from the pivot point). F¯end refers to forces/torques represented in frame 1, induced by the weight of the endoscope. The correction coefficients are obtained as a result of simulation tests. These coefficients are calculated by dividing the torques calculated from the full dynamic model by the torques obtained from the proposed model. At various locations of the workspace, simulation tests are repeated, and the correction coefficients are re-calculated. The obtained coefficients at different locations of the workspace are similar to each other and for this reason, the mean of these coefficients is taken as the correction coefficients for the dynamic model within the whole workspace (K1 = 1.6078, K2 = 1.6411 and K3 = 2.0343). For

92

˙ C. Dede O. Ayit and M. I.

verifying the proposed model in [1], the torques from this model and the full dynamic model were compared and the root-mean-squares (RMS) of the torque errors were obtained as 0.0223 Nm, 0.0251 Nm, and 0.0223 Nm for a range of 2.2043 Nm, 2.6099 Nm, and 2.5916 Nm, respectively. The execution time of the proposed dynamic model was measured as 2.7 ms, which is obtained by using an ARM Cortex M4 processor. However, this previous work did not consider the change in the orientation of the active arm’s base platform. Later, the embedded coding used in the previous study is further optimized and the execution time is measured as 2.1 ms. In this current paper, we aim to improve the proposed model by considering the orientation of the active part’s base. In the next section, the simplified dynamic model with a new correction factor method is proposed and the results are explained in Sect. 3. In the last section, the results obtained via the new method are compared with the results of the method in [1].

2

Methodology

In the NeuRoboScope system, frame 0 refers to the world coordinate. Frames 1 and 2 are attached to the base of active part and endoscope, respectively as shown in Fig. 2. The rotation matrices from frame 0 to frame 1, R1 , and the rotation matrix from frame 2 to frame 1, R2 , are used in the formulations so that the calculations are carried out in frame 1. Also, it is known that the orientation between frame 0 and frame 1 is due to the angular motion of 5th and 6th joints of the passive arm, however, the 6th joint’s motion is neglected in this study. The range of rotation of 5th joint which is responsible for the orientation of the active part’s base about y0 – direction, is limited between 0 and -80 degrees. This limitation is assigned with respect to the required range of motion of the surgeon.

Fig. 2. Coordinate frames assigned for the NeuRoboScope system. [10].

By using the Eq. (1) to (5), correction coefficients are calculated within the 5th joint’s motion range with a step size of 1◦ . The correction coefficients are obtained as shown in Fig. 3a.

A New Correction Coefficient Formula for the Simplified Dynamic Model

93

Fig. 3. (a) Correction coefficient K1 vs orientation of the active part’s base, (b) obtained coefficients K1∗ vs their corresponding torque values at different locations of the workspace when orientation angle is −40◦ .

As it is seen in Fig. 3b, the correction coefficients receive unexpected values when rotation angles are between −37 and −80◦ and the reasons of this can be listed as; – It is observed that the torque values of the simplified dynamic model are closed to zero in the mentioned range. The coefficient values (division of the torques calculated from the full dynamic model by the torques obtained from the proposed model) go to infinity. The mean of them is taken as a correction coefficient and hence, it is also a high value. – The coefficients get positive and negative values between −37 and −80 degrees, therefore, the mean of the coefficients gives imperfect results. For solving the mentioned problems, a simple solution is proposed by adding an offset to the torque values while calculating the correction coefficients. The modified formulation is presented below. ∗ = Ki,m

∗ +n Ti,m Ti,m + n

mmax Ki =

m=1

∗ Ki,m

mmax

(6)

(7)

∗ is the mth coefficient of the ith actuator corresponding to mth locawhere Ki,m tion in the workspace for a specific orientation. mmax is the number of locations ∗ refers to ith in the workspace on which the calculations are carried out. Ti,m th actuator’s torque calculated at the m location using the full dynamic model and Ti,m refers to torque calculated via the proposed dynamic model. n is an offset value, which is chosen as 5, in order to have positive torque values for all calculations. The full dynamic model of the NeuRoboScope is shown in Eq. (8) under assumptions such as no friction and deformation (rigid body) where T¯∗ , ˆ f , Cˆf and G ¯ f refer to generalized torque vector model, Coriolis and centripetal M matrix, generalized mass matrix and generalized gravity vector.

ˆ f (q)¨ T¯∗ = M q + Cf (q, q) ˙ q˙ + Gf (q)

(8)

94

˙ C. Dede O. Ayit and M. I.

To use the correction coefficients in Eq. (7), Eq. (1) is modified as follows. ˆ corr (M ˆ (¯ ¯ q) + N ¯) − N ¯ T¯ = K q )¨ q + G(¯ (9)  T ¯ = 555 . where N ∗ are similar Equation (7) provides usable results when the coefficients Ki,m ∗ to each other. Unfortunately, Ki,m values calculated by Eq. (6) are not always similar to each other at certain orientation values of the active part’s base. As an example of such a case, in Fig. 3b, these calculated values are presented for the −40◦ orientation of the active part’s base. The simulation tests carried out between the orientation angles −37◦ and −80◦ show that the coefficients K1∗ , K2∗ and K3∗ have similar behavior to the results shown in Fig. 3b. According to the results, the correction coefficients K1 , K2 and K3 are designed as a function of the obtained torques from the simplified model rather than a constant value and the linear regression method is employed for this study. The correction coefficient formula is obtained as; Ki = b1,i + b2,i Ti

(10)

where b1,i and b2,i are linear regression gains. When this method is applied, different linear regression gains are calculated for each orientation angle. In this study, 44 b1,i and 44 b2,i are obtained for each Ti where i = 1, 2, 3. These gains can be modeled as a function of orientation or obtained from a lookup table for experimental studies.

3

Results

The simplified dynamic model with the correction coefficients is verified by using the NeuRoboScope robot’s dynamic model in simulation tests. The simulation tests are carried out in Matlab/Simulink R2019a where solver and step time are chosen as Euler and 0.001 respectively. The motion trajectories for the simulation tests are obtained from workspace studies of NeuRoboScope in [5] and the motion ranges are shown in Table 1. Table 1. Motion Ranges used in the Simulation Tests φ(rad)

ψ(rad)

d(mm)

±0.205 ±0.2043 240 – 250 or 150 – 160 rad rad ˙ ˙ φm a x ( s ) ψm a x ( s ) d˙m a x ( msm ) 0.0685

0.1336

0.0548

Initially, the correction coefficient formula, shown in Eq. (10) is used when the orientation angle is between −37◦ and −80◦ and in addition to that, the correction coefficients are calculated by the proposed method in [1] when the orientation angles between 0◦ and −36◦ . As a second study, the correction coefficient formula is used for the whole range of the orientation angle.

A New Correction Coefficient Formula for the Simplified Dynamic Model

RMS torque error for T1

95

RMS torque error for T2

RMS torque error for T3

Fig. 4. RMS torque errors for each orientation angle. Red line is obtained when the correction coefficient formula is used for all orientation. Blue one refers to RMS errors when the correction coefficient formula is used between −37◦ and −80◦ and the method in [1] is applied when orientation angle is between 0◦ and −36◦ .

The RMS errors of the simplified model with respect to the full dynamic model at each orientation of the active part’s base are given in Fig. 4. The total RMS error for the case when the correction coefficient formula is used for all the range and between −37◦ and −80◦ is calculated as 0.0236 Nm and 0.0340 Nm. Hence, it is deduced that using the correction coefficient formula for the whole range of orientation produces better results. Among these relatively better results, the maximum errors in all simulation tests are −0.0990 Nm for T1 , 0.0482 Nm for T2 and −0.0470 Nm for T3 where the obtained torque values from the simplified dynamic model are −3.0757 Nm, −1.8283 Nm and 3.0138 Nm respectively. The largest percentage errors are obtained when the obtained torques values are close to zero; for instance, the errors are 0.0684 Nm, −0.0316 Nm and 0.0324 Nm where T1 = −0.0278 Nm, T2 = 0.0105 Nm and T3 = 0.0207 Nm, respectively. The results show that the torque errors are tolerable for the NeuRoboScope system therefore, the simplified dynamic model can be used as a substitute for the full dynamic model.

4

Conclusion

In a previous study [1], a method was proposed to formulate the simplified dynamic model of a robotic endoscope holder, NeuRoboScope. The main motivation for this study was to have the control algorithm, which is using the dynamic model, executed at high frequency. However, the previous work neglected the effects of orientation change of the base platform due to the passive arm’s wrist motion. In this paper, a new correction coefficient formula is proposed to formulate the simplified dynamic model according to the orientation changes. If

96

˙ C. Dede O. Ayit and M. I.

constant coefficients were used for the whole range, the RMS error value in estimating the torques is 8.8705 Nm and when this new correction coefficient formula is used, the RMS value dropped to 0.0236 Nm. As proposed formulation, neglecting the parameters that have minimal impact on the dynamic model is a generally preferred way to simplify the dynamic model such as neglecting the mass and inertia of the particular elements [4,6] also, the elimination of Coriolis and centripetal terms for slow moving manipulator [7]. However, the usage of the correction coefficient is a novel approach for simplifying the dynamic model. In order to check the execution time, ARM Cortex M4 is chosen as the main processor. The code of the simplified model is written and compiled by using the STM32Cube IDE program. The execution time is calculated as 2.1 ms, which is the same value received for the constant coefficient method applied in the previous work.

References 1. Ayit, O., Ya¸sır, A., Vardarlı, E., Kiper, G., Dede, M.I.C.: Bir Ameliyat Robotunun Denetimi i¸cin Basitle¸stirilmi¸s Dinamik Modeli. In: TOK 2018 Otomatik Kontrol Ulusal Toplantısı Bildiriler Kitabi, pp. 265–270. TOK, Kayseri (2018) 2. Bardaro, S.J., Swanstr¨ om, L.: Development of advanced endoscopes for natural orifice transluminal endoscopic surgery. Minim. Invasive Ther. Allied Technol. 15(6), 99–110 (2006) 3. Bax, K.N.M.A., Georgeson, K.E., Rothenberg, S.S., Valla, J.-S., Yeung, C.K.: Endoscopic Surgery in Infants and Children, 1st edn. Springer, NewYork (2018) 4. Carbonari, L., Battistelli, M., Callegari, M., Palpacelli, M.-C.: Dynamic modelling of a 3-CPU parallel robot via screw theory. Mech. Sci. 4(1), 185–197 (2006) ˙ et al.: Cerrahın anlık y¨ 5. Can Dede, M. I., onlendirilebildi˘ gi robot yardımlı endoskop kontrol sistemi mimarisi-NeuRoboScope. In: TORK 2018 T¨ urkiye Robotbilim Kon¨ ˙ feransı. Bo˘ gazi¸ci Universitesi, Istanbul (2018) 6. Kumar, S., Martensen, J., Mueller, A., Kirchner, F.: Model simplification for dynamic control of series-parallel hybrid robots - a representative study on the effects of neglected dynamics. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5701–5708. China (2019) 7. Lebret, G., Liu, K., Lewis, F.L.: Dynamic analysis and control of a stewart platform manipulator. J. Robot. Syst. 10(5), 629–655 (1993) 8. Smith Jr., J.A., Herrell, S.D.: Robotic-assisted laparoscopic prostatectomy: do minimally invasive approaches offer significant advantages? J. Clin. Oncol. 23(32), 8170–8175 (2005) R robotic camera holder 9. Stolzenburg, J.-U., et al.: Comparison of the FreeHand with human assistants during endoscopic extraperitoneal radical prostatectomy. BJU Int. 107(6), 970–974 (2011) 10. Ya¸sır, A.: Design of a 2R1T mechanism with remote center of motion for minimally invasive transnasal surgery applications. M.Sc Thesis. Izmir Institute of Technology, Turkey (2018)

Designing Adaptive Nonlinear Controller for Optimal Tracking of Laparoscopic Robotic Arm with Nonholonomic Constraints Amir Aminzadeh Ghavifekr1(B) , Roushanak Haji Hassani2 , and Andrea Calanca3 1 Department of Electrical and Computer Engineering, University of Tabriz, Tabriz, Iran

[email protected] 2 BIROMED-Lab, Deptartment of Biomedical Engineering, University of Basel, Basel,

Switzerland [email protected] 3 Department of Computer Science, University of Verona, Verona, Italy [email protected]

Abstract. Laparoscopic surgery is one kind of minimally invasive surgery in which surgeons can control and evaluate the tracking of the robot via special monitoring. Preserving stability, precise tracking, and uncertainties estimation are essential factors in controller design for these systems. Designing autonomous graspers would improve the perception of surgeons and reduce the unwanted errors of operations. An adaptive inverse dynamic controller is proposed in this paper to improve the performance of the tracking desired trajectories for gripper of the surgical robots used in laparoscopic surgeries. By considering nonholonomic conditions for the grasper of the robot, the workspace is generalized, and using the Ritz approximation method, the optimal steering problem for nonholonomic gripper has been solved. In this case, the surgeon can steer the gasper of the surgical robot from any initial condition to any desired destination point. The performance of the proposed controller for nonholonomic conditions is demonstrated by simulation results. Keywords: Laparoscopic arm · Adaptive inverse dynamic controller · Nonholonomic constraints · Optimal tracking · Surgical robots

1 Introduction In recent years, the minimally invasive surgery systems have developed rapidly, and numerous studies have been presented in this area [1, 2]. Combining the surgeon’s skill with the robot’s precision, increases the accuracy of surgeries and reduces the postoperative pain and recovery time for the patients. Laparoscopic surgery is one kind of minimally invasive surgery in which some small cavities have been made on the chest, and camera and performing instruments are penetrated to the body. Surgeons control and evaluate the tracking of the robot © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 97–107, 2021. https://doi.org/10.1007/978-3-030-58104-6_12

98

A. Aminzadeh Ghavifekr et al.

via a special monitoring [3]. Laparoscopic surgical systems include master and slave robots. These surgical robots are computer-controlled devices that can be programmed to aid the manipulation and positioning of surgical instruments. Recently these robots have been developed to address the limitations of laparoscopy, such as two-dimensional visualization, ergonomic limitations, and incomplete articulation of instruments [4–6]. Grasper of the surgical robots has an important role in accurate performing [7, 8]. The tracking control problem for the end-effectors of surgical robots has been extensively studied. For surgical robots with completely known parameters, methods such as inverse dynamic controllers, computed torque scheme, and feedback linearization technique have been suggested to improve the accuracy of tracking. However, these controllers have constant parameters and may cause unwanted injuries in the tissues. Also, in the presence of the uncertainties such as type of tissue and imprecise physical parameters, the tracking errors are significant and non-negligible for the aforementioned controllers and can cause traumas to the lateral tissues of the surgical site [9–11]. When parameters of the system are subjected to uncertainties, adaptive, and robust control methods and intelligence control have been mostly suggested [12, 13]. In this paper, adaptive inverse dynamics controller has been applied to improve the tracking performance in the presence of the uncertainties besides preserving the stability. This paper addresses how for two points of laparoscopic robot’s workspace such as q0 , qf ∈ R3 , the desired trajectory of q(t) can be applied by the surgeon which is satisfying the Pfaffian constraints. The set of all qf points that can be connected to q0 and satisfies Pfaffian constraints is called reachable points corresponding to the q0 . Therefore, we would like to know under what conditions the domain of the reachable area will be R3 . One necessary condition for this purpose is that all constraints should be nonholonomic [14]. Investigating of this case for a set of Pfaffian constraint has a complicated and challenging process. Using Chow’s theorem [15] can just investigate the local controllability and existence of the desired trajectory, and does not provide a method for its designing. Several methods have been introduced in the literature for steering of these systems, such as using sinusoids [16], chained form transformation, and Fourier techniques [17]. In this paper, Ritz approximation method [18], has been applied for solving the optimal steering problem of the gripper. The rest of this paper is organized as follows: In Sect. 2, mechanical architecture of the laparoscopic gripper is described. Section 3 explains the dynamic model, and process of designing a nonlinear adaptive controller for the grasper. In Sect. 4, using the Ritz approximation method, the constrained optimal steering problem is solved for the nonholonomic dynamic of the gripper. In this case, the surgeon can steer the gasper of the surgical robot from any initial condition to any desired destination point. Finally, simulation results are presented for the proposed controller in Sect. 5.

2 Mechanical Model of Laparoscopic Grasper In the laparoscopic robot design section, a basic scheme of the gripper is initially designed. Then the motors (servomotors) are selected according to the forces and torques required for the moving parts of the gripper with good reliability coefficients. Figure 1 represents the simulated model of the grasper for the Laparoscopic robot.

Designing Adaptive Nonlinear Controller for Optimal Tracking

99

Fig. 1. Virtual prototype of the grasper for the Laparoscopic robot

Grippers are the end-effecters of surgical robots that have various types and applications and are used for sensitive medical operations such as stitching and grabbing. Figure 2 illustrates different types of conventional grippers and Fig. 3 shows the simulated structure of the proposed laparoscopic grasper.

Fig. 2. Different conventional grippers used in surgical applications

Fig. 3. Simulated structure of the proposed grasper

Movement transmitters to the gripper include spring and towing cables. The spring connects rod (316 stainless steel) to the gripper and enables its rotation inside the connected rod. Power transmission wires pass through the connector tube and transfer the required movement to the gripper. One end of the rod is coupled to the cylindrical compartment and the other end to the gripper. Also, five towing cables (0.8 mm in diameter) transfer power from servomotors to the gripper. Indeed, the motors rotations to the right and left causes the wires to be pulled and thus different forms of the movement for the gripper can be performed. Figure 4 shows the overall structure of the aforementioned details. Three gears are utilized to drive the connecting rod. The rotational property of the connecting rod can provide one of the freedom degrees of the system. By fixing the upper and the lower arms whole cylinder box and its inner rod can be rotated (Fig. 1). There are three servomotors inside the compartment, which the middle one is used to push and pull the rod cable to open and close it. The other servo motors provide the remaining degrees of freedom by pulling the cables connected to the spring.

100

A. Aminzadeh Ghavifekr et al.

Fig. 4. Assembled structure of the gripper’s actuator

3 Dynamic Model and Designing Nonlinear Adaptive Controller for the Grasper To determine the position of the grasper of the robot relative to the ground in the Cartesian coordinate, Lagrangian formulation has been used for dynamic modeling. For this purpose and simplification of the calculation, the grasper’s spring has been exchanged by three joints. As it is shown in Fig. 5 two of these joints are revolute around z1 and z2 axes and one of them is prismatic around z3 to represent the effect of the length increase of spring on all rotations.

Fig. 5. Geometrical Model of the grasper

The well-known nonlinear dynamic equation of the gripper can be stated as: M (q)¨q + C(q, q˙ )˙q + g(q) = u

(1)

where M (q) ∈ Rn × n and C(q, q˙ )˙q ∈ Rn are the inertia and Coriolis matrices, respectively. g(q) ∈ Rn indicates the gravitational force and u is the control input. q ∈ Rn is the position signal. This dynamic can be rewritten in a linear form as: M (q)¨q + C(q, q˙ )˙q + g(q) = Y (q, q˙ , q¨ )θd

(2)

where θd = (θd 1 , θd 2 , . . . , θdp )T is a set of physical parameters and Y (q, q˙ , q¨ ) ∈ Rn × p is the regressor matrix. Using Jacobian matrix concept, the velocity of gripper in the task space is redefined by using dynamic parameters of ak as: x˙ = J (q)˙q = Yk (q, q˙ )ak

(3)

where YK (q, q˙ ) is a dynamic regressor matrix. In order to avoid the velocity measurement in the workspace, the following low pass filter is used y˙ + λy = λ˙x, y =

λ λ Y (q, q˙ ) x˙ = Wk (t)ak ,Wk (t) = λ+p λ + pk

(4)

Designing Adaptive Nonlinear Controller for Optimal Tracking

101

where y is the filtered velocity output with initial values of zero. It is assumed that xd , x˙ d , x¨ d are desired position, velocity and acceleration signals, respectively. The following well-known controller has been suggested in the literature to eliminate nonlinear terms from (1): ˙ − Kp x − J˙ (q)˙q)] + C(q, q˙ )˙q + g(q) u = M (q)[J −1 (q)(¨xd − Kv x

(5)

˙ where x = x − xd and x = x˙ − x˙ d are tracking error of the position and velocity, respectively. Kp , Kv are positive definite matrices and J˙ (q) is the derivative of the Jacobian matrix. However, in the presence of the uncertainties such as type of tissue, the control law given in (5) cannot be used. Thus, an estimation of unknown parameters has been used in the proposed controller law as: ˆ ˆ (q)[Jˆ −1 (q)(¨xd − Kv ˆ ˙ x − Kp x − J˙ˆ (q)˙q)] + C(q, u=M q˙ )˙q + g(q) ˆ

(6)

where Jˆ (q) and aˆ k are the estimations of the Jacobian matrix and kinematic parameters, respectively and will be calculated by the update rules. Also, we have xˆ˙ = xˆ˙ − x˙ d where xˆ˙ = Jˆ (q)˙q. xˆ˙ is the estimation of the velocity of the gripper. The controller’s update laws are proposed as: ˆ −1 Y (q, q˙ , q¨ ))T (xˆ˙ + αx) a˙ˆ d = −d (Jˆ M aˆ˙ k = k [YkT (Kp + αKv )x + αYkT xˆ˙ − WkT Rk (Wk aˆ k − y)]

(7)

where Rk , d , k are positive definite matrices and α > 0 is a positive constant which is satisfied Kv − αI ≥ βI for all positive β. Also, Y (q, q˙ , q¨ ) can be calculated by (2), and YK (q, q˙ ) which is the kinematic regressor matrix can be obtained by:   − sin(q1 )˙q1 − sin(q1 + q2 )(˙q1 + q˙ 2 ) Yk (q, q˙ ) = (8) cos(q1 )˙q1 cos(q1 + q2 )(˙q1 + q˙ 2 ) It can be proved that by using the update rules of (7) for the proposed controller in (6), besides preserving the stability of the system, the tracking errors of the position and velocity signals converge to Zero. The following Lyapunov function has been applied for this proof: V =

1 ˆ 1 1 1 (x˙ + αx)T (xˆ˙ + αx) + xT (Kp + αKv − α 2 I )x + adT Γd−1 ad + akT Γk−1 ak 2 2 2 2

(9)

The details of the proof have been excluded for brevity. If the velocity signal is measurable in the workspace, the control law and update rules can be rewritten as: a˙ˆ d

ˆ ˆ (q)[Jˆ −1 (q)(¨xd − Kv x ˙ − Kp x − J˙ˆ (q)˙q)] + C(q, u = M q˙ )˙q + g(q) ˆ −1 T T T T T ˆ Y (q, q˙ , q¨ )) B Pξ a˙ˆ k = k [Y B Pξ − Y Rk (Yk aˆ k − x˙ )] (10) = −d (Jˆ M k k

where Rk , d , k are positive   definite matrices, P is the answer of Lyapunov equation, x 0 and ξ = ,B = . ˙x I

102

A. Aminzadeh Ghavifekr et al.

4 Optimal Steering of Nonholonomic Grasper The least square optimal control problem is obtaining a path for q˙ =

m  j=1

gj (q)uj from

initial point of q(0) = q0 to final point of q(t) = qf during [0–1] s. The Cost function is defined as: J =

1 t ∫u(t)22 dt 20

(11)

¯ Assuming (q) = Tq Rn and applying Chow theorem, it is deduced that the system is locally controllable if: u(t)2 = u(0)2 ; ∀t ∈ [0, 1]

(12)

Let {ψ0 (t), basis for L2 [0, 1], where:  ψ1 (t), . . . , ψ2k (t)} be an orthonormal  1 2 L2 [0, 1] = f : [0, 1] → R∫ f (t)dt < ∞ 0 √ = 2 cos 2kπ t; ∀t ∈ [0, 1], ψ2k (t) = = 1, ψ And let: ψ (t) 0 2k−1 (t) √ 2 sin 2kπ t; ∀t ∈ [0, 1] A Ritz approximation is assumed to define as: ui (t) =

N 

αik ψk (t), i = 1, . . . , m, t ∈ [0, 1]

(13)

k=0

Also, we have 1

N 1

0

0

∫ ui2 (t) = ∫

k=0

αik ψk (t)

N 

αij ψj (t)dt =

N  N  k =0j =0

j=0

1

N 

0

k=0

αik αij ∫ ψk (t)ψj (t)dt =

1

N 

0

k=0

2 ∫ ψk2 dt = αik

2 αik

(14)

where αi = (αip + . . . + αiN ) ∈ RN +1 and N is the number of Ritz approximation’s terms. Using equality of 11 1  2 1 ∫u(t)22 dt = αi 22 αik = 20 2 2 m

N

i=1 k=0

m

(15)

i=1

Leads to the following constrained optimization problem: 1 αi 22 , s.t q(1; α1 , . . . , αm ) = qf 2 m

min

(16)

i=1

where unknown parameters of α should be calculated for optimal steering from q0 to qf . Assume θ is the angle of the body of the grasper with respect to the horizontal, and ψ1 , ψ2 are the angles of the grasper’s arms with respect to the end-effecter of the laparoscopic robot. By applying Lagrangian method: Pm =

∂L = a13 ψ˙ 1 + a23 ψ˙ 2 + a33 θ˙ ∂ θ˙

(17)

Designing Adaptive Nonlinear Controller for Optimal Tracking

103

where Pm represents the angular momentum of the robot. Giving the following constraint equation: a13 ψ˙ 1 + a23 ψ˙ 2 + a33 θ˙ = 0

(18)

Input signals can be defined as: u1 = ψ˙ 1 , u2 = ψ˙ 2 Substituting these inputs in (17) and defining q = (ψ1 , ψ2 , θ )T the dynamic of the system can be rewritten as: Σ : q˙ = g1 (q)u1 + g2 (q)u2 where ⎡ ⎡ ⎤ ⎤ 1 0 ⎦ ,g2 (q) = ⎣ ⎦ g1 (q) = ⎣ (19) 0 1 −a13 (ψ)/a33 (ψ) −a23 (ψ)/a33 (ψ) According to (15), the constraint optimization problem can be stated as: 1 minJ := 2

1 u(t; α)22 dt, s.t. q(1; α, q0 ) = qf

(20)

0

Indeed, we are looking for appropriate adaptive controller parameters in (6), which minimizes the aforementioned cost function.

5 Simulation In this section, the performance of the proposed controller for nonholonomic conditions is demonstrated by the simulation results. The physical parameters of the gripper are presented in Table 1. Table 1. Physical parameters of the gripper ith mi (Kg) Ii (Kgm2 ) li (m) lci (m) link 1

0.5

0.15

0.2

0.1

2

0.5

0.25

0.2

0.1

where mi , Ii , li , lci are the masses of the links, moments of inertia, length of links, and the centers of the masses, respectively. The used values of parameters in the simulation are:  0.8 + 0.3 cos(π t) , q(0) = [π/3 − 2π/3]T , x(0) = [1.1 − 0.1732]T 0.3 sin(π t) s aˆ d (0) = [4 0.5 0.5]T aˆ k (0) = [1.4 1.5]T , q¨ = q˙ , λa = 1/60, Kp = 400I , Kv = 40I , λ = 10 λa s + 1 

xd (t) =

(21)

104

A. Aminzadeh Ghavifekr et al.

where xd (t) is the desired movement by the surgeon, and aˆ d (0), aˆ k (0) are the initial values of the dynamic and kinematic parameters, respectively. Figure 6 illustrates the tracking error of the gripper by applying controller (6). Figure 7 depicts the evolution of the estimation of dynamic parameters. It is obvious that the estimated parameters are bounded and converge to a constant value.

Fig. 6. The tracking error of the gripper

Fig. 7. Estimation of the dynamic parameters

The Number of Ritz approximation terms assumed N = 7. The evolution of the optimal states of the grasper over [0, 1] s is depicted in Fig. 8.

Fig. 8. Evolution of the optimal states of the grasper over [0, 1] s for N = 7

Designing Adaptive Nonlinear Controller for Optimal Tracking

105

Figure 9 shows the evolution of the optimal control inputs over [0, 1] s.

Fig. 9. Evolution of the optimal control inputs over [0, 1] s for N = 7

To illustrate the effect of N on minimizing cost function, the trajectory of grasper is compared for N = 7 and N = 8 in Fig. 10. The trajectory from proposed initial conditions to the final configuration has been compared in Fig. 10.

Fig. 10. Trajectory of grasper for N = 7 and N = 8

It is obvious that, the higher number of approximation terms (N) results in a smaller final cost function. Also, it must be taken into consideration that the obtained solution is not unique and by changing of N the answer may change specifically.

6 Conclusion The adaptive inverse dynamic controller is proposed in this paper to improve the tracking ability of the desired trajectories for gripper of the surgical robots used in laparoscopic

106

A. Aminzadeh Ghavifekr et al.

surgeries. It was demonstrated that the tracking error is acceptable for the proposed controller when the dynamics of the robot is subjected to uncertainties such as type of tissue and imprecise physical parameters. Using a presented Lyapunov function, it is acclaimed that the stability of the system has been preserved besides tracking improvement. By considering nonholonomic conditions for the grasper of the robot, the workspace is generalized and using the Ritz approximation method, the optimal steering problem for nonholonomic gripper has been solved. The trajectory of the grasper for the different numbers of Ritz approximation terms is compared and shows that a higher number of terms lead to the smaller final cost function. The simulation results provided a theoretical basis for further structural progress and optimization of the robotic arms for laparoscopic surgeries.

References 1. Haidegger, T.: Autonomy for surgical robots: concepts and paradigms. IEEE Trans. Med. Robot. and Bionics 1, 65–76 (2019) 2. Wang, P., Su, Y.-J., Jia, C.-Y.: Current surgical practices of robotic-assisted tissue repair and reconstruction. Chin. J. Traumatol. 22, 88–92 (2019) 3. Elizondo, R.A., Au, J.K., Song, S.H., Huang, G.O., Zhang, W., Zhu, H., et al.: Open versus robot-assisted laparoscopic ureteral reimplantation: hospital charges analysis and outcomes at a single institution. J. Pediatr. Surg. (2020) 4. Choi, H., Kwak, H.-S., Lim, Y.-A., Kim, H.-J.: Surgical robot for single-incision laparoscopic surgery. IEEE Trans. Biomed. Eng. 61, 2458–2466 (2014) 5. Jo, Y.. Kim, Y.J., Moon, H.-M., Kim, S.: Development of virtual reality-vision system in robotassisted laparoscopic surgery. In: 2018 18th International Conference on Control, Automation and Systems (ICCAS), pp. 1708–1712 (2018) 6. Jayarathne, U.L., Chen, E.C., Moore, J., Peters, T.M.: Robust, intrinsic tracking of a laparoscopic ultrasound probe for ultrasound-augmented laparoscopy. IEEE Trans. Med. Imaging 38, 460–469 (2018) 7. Guo, S., Yang, C., Bao, X., Xiao, N.: A novel design of grasper for the interventional surgical robot. In: 2017 IEEE International Conference on Mechatronics and Automation (ICMA), pp. 469–473 (2017) 8. Ghavifekr, A.A., Badamchizadeh, M., Alizadeh, G., Arjmandi, A.: Designing inverse dynamic controller with integral action for motion planning of surgical robot in the presence of bounded disturbances. In: 2013 21st Iranian Conference on Electrical Engineering (ICEE), pp. 1–6 (2013) 9. Azimian, H., Patel, R.V., Naish, M.D., Kiaii, B.: A semi-infinite programming approach to preoperative planning of robotic cardiac surgery under geometric uncertainty. IEEE J. Biomed. Health Inform. 17, 172–182 (2012) 10. Ghiasi, A., Ghavifekr, A., Hagh, Y.S., SeyedGholami, H.: Designing adaptive robust extended Kalman filter based on Lyapunov-based controller for robotics manipulators. In: 2015 6th International Conference on Modeling, Simulation, and Applied Optimization (ICMSAO), pp. 1–6 (2015) 11. Ghavifekr, A., Ghaemi, S., Behinfaraz, R.: A modified biogeography based optimization (bbo) algorithm for time optimal motion planning of 5 dof pc-based gryphon robot. Int. J. Eng. Works (2014) 12. Wang, H., Xie, Y.: Adaptive inverse dynamics control of robots with uncertain kinematics and dynamics. Automatica 45, 2114–2119 (2009)

Designing Adaptive Nonlinear Controller for Optimal Tracking

107

13. Guo, J., Yang, S., Guo, S., Meng, C., Qi, L.: Study on robust control for the vascular interventional surgical robot. In: 2019 IEEE International Conference on Mechatronics and Automation (ICMA), pp. 1361–1366 (2019) 14. Spong, M.W., Hutchinson, S., Vidyasagar, M.: Robot Modeling and Control. John Wiley & Sons, Hoboken (2020) 15. Murray, R.M.: Geometric phases, control theory, and robotics. In: Motion, Control, and Geometry: Proceedings of a Symposium (1997) 16. Murray, R.M., Sastry, S.S.: Nonholonomic motion planning: steering using sinusoids. IEEE Trans. Autom. Control 38, 700–716 (1993) 17. Tan, Y., Jiang, Z., Zhou, Z.: A nonholonomic motion planning and control based on chained form transformation. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3149–3153 (2006) 18. Fernandes, C., Gurvits, L., Li, Z.: Near-optimal nonholonomic motion planning for a system of coupled rigid bodies. IEEE Trans. Autom. Control 39, 450–463 (1994)

Manipulation of an Wide Angle Endoscope in Minimally Invasive Robotic Surgery and Training John Mannion and Yeongmi Kim(B) MCI, University of Applied Science, Innsbruck, Austria [email protected], [email protected]

Abstract. Wide angle cameras may be beneficial in minimally invasive laparoscopic and colorectal surgeries. Due to the extended range of the wide angle camera, conventional input devices may not be as effective in navigating around wide angle endoscopic views. In this paper we design and develop two input interfaces used for manipulating the views of a wide angle endoscopic camera. The interface is designed to be mounted on the master manipulators of a dVRK in an ergonomic position to allow the surgeon instantaneous access to both the robotic patient-side manipulators and the wide angle endoscopic view. Furthermore, the two interface devices are tested by novice users to examine the efficiency of control of the wide angle endoscopic view. Results show that the devices allow for effective movement across four degrees of freedom and can be used simultaneously with the master manipulator arms in a cohesive manner. Keywords: dVRK · Input control interface · Minimally invasive surgery · Robotic assisted surgery · Wide angle view · Endoscope

1

Introduction

Modern surgical robots incorporate a variety of devices and techniques to perform their tasks in a safe and precise manner. In laparoscopic surgeries performed with the aid of a surgical robot, the view is generally limited to a front facing view from the endoscopic camera. A wide angle endoscopic camera, which could provide up to 360◦ of viewing spectrum, may provide an enhanced surgical experience compared to the standard front facing endoscopic views. With the access to a greater view of the scene, a surgeon may be able to better perform surgery. Colonoscopies commonly use wide angle lens to get a greater view along the folds of the intestine. For example, Fuse (EndoChoice Inc., USA) is a full-spectrum endoscopy platform providing 330◦ of viewing space achieved by stitching the feed from three sources [1]. Mihori et al. [2] introduce a hybrid wide angle endoscopic system for performing vitrectomy in conjunction with a 3D visualisation system. This wide field of view is essential for clearly observing the fundus of c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 108–117, 2021. https://doi.org/10.1007/978-3-030-58104-6_13

Manipulation of an Wide Angle Endoscope

109

the eye and demonstrates the usefulness of such wide angle views in surgical procedures. Qin et al. [3] propose a solution to the issues of trading off between spatial resolution and acquired field of view by demonstrating a multi-resolution foveated laparoscope system. In this system, the surgeon has access to a wide angled overview of the scene and a zoomed-in view which has a much higher resolution. This wide angled probe is useful in providing an overall situational awareness for the surgeon and demonstrates also the usefulness of wide angled views in surgery over narrower views provided by standard front-facing endoscopes. Saneso (Saneso Inc., USA) [4] uses an omnidirectional camera to view inside colons. For its endoscope, five individual camera feeds are stitched together to create a full 360◦ view. The surgical controls for this system involve a handheld device with various dials to rotate the tip of the endoscope and change the view. There have been various methods to manipulate endoscopic cameras. Common manipulator designs incorporate various transducers such as rotational dials, buttons and joysticks. The Flex robotics system (Medrobotics Corporation, USA) [5] provides a flexible double-scope endoscope which can be used for trans-oral procedures. A handle is gripped and used for pitch and yaw movements while a rotating dial is used for the movement of roll. The da Vinci surgical system (Intuitive Surgical Inc., USA) is a tele-operated surgical system. A surgeon controls the remote slave instrument arms and endoscope by operating master manipulators attached to the surgeon console. By pressing the camera foot pedals it is possible to alter the control mode between the surgical tools and endoscope. Recently Hong et al. [6] developed a head mounted master interface comprised of force sensors and a hall sensor to control the endoscopic camera in the da Vinci Research Kit (dVRK). This method allows continuous operation to avoid interrupting the surgeon’s workflow, as the master manipulators can always be used for manipulating the surgical tools instead of being used for altering the view. The Senhance system (TransEnterix, Inc., USA) [7], a surgical system used for performing colonoscopies, uses eye tracking functionality to allow the surgeon to simultaneously operate the surgical instruments and move the endoscopic camera. The monarch platform (Auris Health, Inc., USA) [8] is a teleoperated surgical system used for performing bronchoscopic procedures. The controller is a video-game-console styled device, with two joysticks for moving the endoscopic tip. The endoscopic tip, at which the camera is located, can be tilted in a pitch and yaw motion using the right joystick. The left joystick is used to move the endoscope forward or to retract it. Other more diverse examples of camera manipulation proposed for use in surgical environments include voice commands [9], eye tracking [10], computer vision techniques [11] and gesture recognition [12]. A study performed by Xia et al. [13] compares the use of difference input devices in terms of user efficiency and usability for navigating virtual scenes. Another study carried out by Youssef et al. [14] focuses on combinations of different techniques and devices used for navigating laparoscopic surgeries. This study analyses aspects such as the cognitive effort required to operate the devices.

110

J. Mannion and Y. Kim

Two-handed and one-handed techniques for performing the tasks were compared against each other. In order to properly assess the effectiveness or usefulness of an wide angle camera for a surgical robotic system, an interface by which to control the camera must be designed. Such an interface must not hinder the original operation of the robot in any way, but rather enhance the surgical procedure by providing the surgeon with additional ways to view the scene. Therefore, the interface must be integrated with the existing controls to allow fluid and intuitive operation without complicating the original procedure. In terms of the da Vinci surgical system, this means any physical devices of the interface have to be readily available for use by the surgeon. The devices are also to allow access to the endoscopic camera manipulator, therefore allowing the surgeon to switch between the operation of the wide angle camera and the physical camera arm at will while they would simultaneously have continuous access to the patient side manipulator arms. We designed two types of input interfaces to explore throughout a wide field of view and integrated them into the dVRK system. The two interfaces were compared against each other in order to examine the efficiency and usability in the wide angle (up to 360◦ ) scene. They have the same core functionalities by which they allow for the navigation of a wide angled view without preventing the original use of the surgeon console’s master manipulators.

2

Methods

The dVRK is an open source research platform which includes hardware and software components for a teleoperated robotic surgical system based on the da Vinci surgical robot [15]. This platform allows for the reading of sensor data (motor current, encoder rotation angle, magnetic sensing) of each joint of the master tool manipulators, patient side manipulators and endoscopic camera manipulators in order to control the physical actuators. The Linux PC incorporates ROS (Robot Operating System) which is used to operate the dVRK system. The proposed interfaces are mounted on the master manipulators and can control the viewing position of the wide angle camera. 2.1

Design and Implementation of the Control Interfaces

Two input interfaces were designed as controllers for a wide angle endoscopic camera operated by the index finger(s). The first is a one-handed interface and the other, a two-handed interface. The former is attached on the right side of the master manipulator clipping ring as depicted in Fig. 1 and the latter has two sub-devices which are mounted on the left and right surgeon console’s existing master manipulators respectively, in specific positions designed to be ergonomic for the user. The interfaces support four degrees of freedom (i.e. rotation about the three camera axes and zooming) for navigating a wide field of view. The onehanded interface consists of three force-sensitive buttons and a trackball encased in an adjustable 3D printed housing. The trackball is used for the movements

Manipulation of an Wide Angle Endoscope

111

Fig. 1. Two different control interfaces and their mounting procedure for a da Vinci master manipulator. Thumb and middle fingers are inserted into the black Velcro rings which secure them to the master arms in order to manipulate four degrees of freedom. The index fingers activate the transducers of the devices.

of both pitch and yaw simultaneously. Two of the force sensors (HSFPAR303A, Alps Electric) are used to perform the motion of rolling. The left-most rotates the camera anticlockwise, while the right-most rotates the camera clockwise. The middle force sensor zooms the camera in and out. The trackball can be pushed downwards acting as a push-button which can swap the direction in which the camera zooms. The two-handed interface consists of three buttons with corresponding LEDs for showing each of their individual activation. The trackball is encased in a secondary housing. Each button allows the user access to a different set of DOFs in the camera view. These axes are then rotated by using the trackball. The interfaces are designed so as not to collide with any orientation of the master manipulator links. The sensor signals are amplified before being connected to the micro-controller (ATmega16). 2.2

Software

Since the endoscopic camera with the dVRK does not support wide angle view, renderings of simulated wide views from an existing omnidirectional camera were implemented in order to test the proposed control interfaces. In this regard, the simulated images are displayed on a sphere in 3D virtual space. OpenGL, a crossplatform API used for rendering 2D and 3D environments, was integrated into a ROS executable so as the display could be accessed and manipulated through the use of the devices connected to the ROS system. The devices described in Sect. 2.1 are used to publish messages to the instance of ROS running. These messages are then read by the ROS node in which OpenGL is being run. In order to load a 360◦ video, the OpenCV library was also used. The cv bridge libraries allow for OpenCV images to be converted to ROS interpretable images and vice versa.

112

J. Mannion and Y. Kim

Fig. 2. ROS nodes and topics used for the system. Orange and black lines represent subscribing and publishing processes respectively from the nodes to the topics. Hardware components are grouped within the green rectangles.

The ROS framework is used to connect the control input interfaces to the OpenGL scene and manipulate the virtual camera accordingly. ROS uses a publisher and subscriber system to allow communication between the control input interfaces and the OpenGL 360◦ scene. Data can be sent to specifically named buses in the ROS system, called topics. ROS messages contain the data which is sent to the ROS topics. By this method of communication the micro-controllers are used to publish messages to a ROS topic via a USB serial interface. Once the messages are sent to the topics, they can be accessed by ROS nodes. ROS nodes are processes which perform any form of computation. They can exist on the ROS PC itself, or in devices connected to the PC. The messages published by the micro-controllers are in the format of 16 bit integer arrays. The OpenGL simulation, acting as a node, can then subscribe to the ROS topic and read the data from the incoming arrays. A ROS communication protocol is provided to the micro-controller scripts which allows for the micro-controller to operate as a ROS node. From within the firmware, a new topic is specified, named omniSimulation. The data received from the devices is published to this topic. Figure 2 shows the communication setup of ROS with the various topics, nodes and connected hardware. Figure 3 shows a close-up of one of the da Vinci’s master manipulators, along with the one-handed device attached to it.

3

User Study

Investigating the optimal manipulation method for the wide angle endoscope is the ultimate goal of designing the input control interface. In terms of efficiency, usability and ergonomics, we hypothesized that the one-handed interface

Manipulation of an Wide Angle Endoscope

113

Fig. 3. The one handed interface mounted on a master manipulator with the thumb and middle finger inserted into the Velcro straps.

improves the navigational performance of the user and receives higher preference over the two-handed interface. Eight novice participants (three females and five males between the ages of 23 and 34) took part in the study and the study protocols had been approved by the Institutional MCI Research Ethics Group, Austria. A questionnaire was presented to each of the participants post-testing in order to compare the two devices. Factors such as ergonomics and usability are scored for the individual devices by each of the participants. Statistical tests are employed to compare the two interfaces regarding which would be a better a wide angle view manipulation method. 3.1

Experimental Setup and Procedure

Four different testing scenes were generated and presented to all participants during the study. These are 360◦ images mapped to the inside of a sphere in the developed ROS-OpenGL application. Each overall scene consists of 70 images, 50 of which are randomised. The participants were asked to find a specific test image, of which there were twenty copies within the scene. They were to find as many of the specific image shown within five minutes for each scene. Every image has a randomly assigned specific serial number. Once the participant recognises an image, the participant states its serial number verbally, which the experiment coordinator then records. Figure 4 displays a part of a 360◦ image used as one of the scenes. The image is texture-mapped across the OpenGL sphere, which also ensures none of the distortions visible in a 2D representation occur, and that all edges meet each other cohesively. The sub-images displayed to the right of Fig. 4 were used to generate each of the four test scenes. Each participant took part in the study with four patterns of images and two control interfaces. The order of patterns and interfaces were randomised. 3.2

Data Analysis

Normality of the data sets was checked using Shapiro-Wilk, the results of which were then used to determine whether a paired samples t-test or Wilcoxon test

114

J. Mannion and Y. Kim

Fig. 4. A 360◦ image used for the study (left). Sub-images used to generate the test scenes (right)

should be used on each set of data. The user performance was analysed by recording the total number of correct and incorrect patterns found across the two interfaces. Furthermore, the Wilcoxon signed-ranked test was employed to compare the two interfaces based on the subjective ratings. All statistical analysis was conducted in IBM SPSS Statistics 25.

4

Results

Table 1 summarizes the results regarding the mean percentages and the average number of the correct and incorrect images found using both the one-handed and two-handed interface. The two-handed interface results in a slightly higher correct mean value and also a lower incorrect mean value of images found compared to the one-handed interface. According to the Shapiro-Wilk test, data is normally distributed for the collection of correct images (p > 0.05) but not for the incorrect images set (p < 0.05). Regarding the correct images set, a paired sample t-test did not reveal any significant difference between two interfaces on the number of accurate images (t(7) = 0.045, p = 0.965). Similarly the Wilcoxon Table 1. The mean percentages and average number of the correct and incorrect images found using both devices across all testing procedures. 100% represents all the correct available images which could be found by the participants. There are a total of 20 possible correct images for each testing procedure. Correct images found Incorrect images found Mean (%) Mean number Mean (%) Mean number Two-handed 34.20%

6.81

0.63%

0.125

One-handed 34%

6.75

2.13%

0.375

Manipulation of an Wide Angle Endoscope .

Ra ngs for different device characteris cs

10

5

8

4

6

3

4

2

2

1

2-H 1-H

115

2-H 1-H

0

0

Ergonomics

Usability

Roll

Pitch/Yaw

Zoom

Fig. 5. The average ratings of the individual characteristics and their standard deviations. The graph on the left has a scale from 1 to 10, while the graph on the right has a scale from 1 to 5. Note that 1-H and 2-H refer to the one-handed and two-handed interfaces respectively.

test also shows insignificant difference between the interfaces in terms inaccuracy (Z = −0.962, p = 0.336). The results of the participants’ ratings in terms of usability, ergonomics, and different manipulation functions are depicted in Fig. 5. Overall, the one-handed interface received a higher average score except for the zooming function. Analogous to the accurate/inaccurate image matching tasks, two interfaces did not significantly show differences in ratings for ergonomics or usability, nor for the roll and pitch/yaw functions (p > 0.05). However, the function of zooming is deemed preferable using two-handed interface rather than the one-handed interface. The Wilcoxon signed-rank test revealed a significance (Z = −2.06, p < 0.05) in favour of the two-handed interface being more effective.

5

Discussion

In this paper we proposed the two control interfaces for manipulating a wide angle endoscopic camera view. The integrated software provides an interface between the input device and camera view in a dVRK system. From the user study, we found that both interfaces support similar user performances and subjective ratings, while the two-handed interface provides statistically higher preference for the zooming function. A study performed by Xia et al. [13] compares bimanual (two-handed) and unimanual (one-handed) navigation tasks across a virtual scene. According to this study, unimanual operations (involving just one hand and one device) were more efficient than the two-handed system in terms of task completion time. Comparisons can be made between this paper’s results and those of Xia et al. in that the one-handed device received a higher average score than that of its counterpart. There was no significant difference between the two interfaces regarding

116

J. Mannion and Y. Kim

performance in the allotted test time. Buxton et al. [16] also compares twohanded and one-handed devices for navigation and selection tasks. This study displays the degree of parallel activity engaged by the participants, whereby both the left and right-handed devices were used simultaneously. In this study, it was found the group using the two-handed system outperformed the counter group. This suggests that there is validity in using a two-handed system, and that depending on the design it could be preferable for navigational tasks as opposed to a one-handed device. However, the specific navigational task will determine whether a one-handed or two-handed system is more advisable. The specific transducers which make up the devices themselves are also important when considering efficiency, degree of user-friendliness and ergonomics. In addition to the efficiency comparison over the course of a certain time period, Youssef et al. [14] explored the cognitive effort apparent when different hand techniques and standing positions are used for laparoscopic procedures. Regarding the operation time and level of complexity existing in the tasks, there was no difference found in either device’s performance. However, the two-handed device technique required higher mental workload than the one-handed technique and was worse from an ergonomics standpoint in certain body positions of the participants. This is similar to the study presented in this paper in that the one-handed system seemed to be more ergonomic on average, but the outcome of the quantitative tests (regarding the number of images found by the participants) was surprisingly quite similar, showing no discernible difference between either system.

6

Conclusion

The two interfaces designed and presented here allow the user to control a wide angle endoscopic view and to be integrated into a tele-operating interface in the dVRK. Our study results show that only zooming function is outperformed with the two-handed interface. In the one-handed interface, manipulation of roll and pitch/yaw motion correspond to the physical rotation or spatial mapping, while the zoom button is physically less intuitive. Further research will comprise of more investigations as to how the manipulation of the zooming motion can be improved in terms of usability and ergonomics. Another thread of future work would be to compare different one-handed manipulation devices. A further line of research could involve a secondary device whose purpose would be relegated to purely selection tasks for a GUI which could overlay the surgeon’s view providing additional information, while the primary device would perform only navigational tasks. Acknowledgment. The authors would like to thank Prof. Dr. Nikolaos Bonaros, Stephan J¨ ackel, Benigna Meussling, Andreas Ascher and Stefan Spiss for their advice and support in the development of this project.

Manipulation of an Wide Angle Endoscope

117

References 1. Gkolfakis, P., Tziatzios, G., Dimitriadis, G., Triantafyllou, K.: New endoscopes and add-on devices to improve colonoscopy performance. World J. Gastroenterol. 23, 3784 (2017) 2. Kita, M., Mori, Y., Hama, S.: Hybrid wide-angle viewing-endoscopic vitrectomy using a 3d visualization system. Clin. Ophthalmol. 12, 313–317 (2018) 3. Qin, Y., Hua, H.: Optical design and system engineering of a multiresolution foveated laparoscope. Appl. Opt. 55, 3058 (2016) 4. Saneso Homepage. http://saneso.com/. Accessed 21 Feb 2019 5. Medrobotics Corporation. https://medrobotics.com/gateway/instruments/. Accessed 14 June 2019 6. Hong, N., Kim, M., Lee, C., Kim, S.: Head-mounted interface for intuitive vision control and continuous surgical operation in a surgical robot system. Med. Biol. Eng. Comput. 57, 601–614 (2019) 7. Senhance Homepage. https://www.senhance.com/us/digitallaparoscopy. Accessed 14 Feb 2019 8. Auris Health Homepage. https://www.aurishealth.com/monarch-platform. Accessed 14 Feb 2019 9. Reichenspurner, H., Damiano, R.J., Mack, M., Boehm, D.H., Gulbins, H., Detter, C., Meiser, B., Ellgass, R., Reichart, B.: Use of the voice-controlled and computerassisted surgical system ZEUS for endoscopic coronary artery bypass grafting. J. Thorac. Cardiovasc. Surg. 118(1), 11–16 (1999) 10. Staub, C., Can, S., Jensen, B., Knoll, A., Kohlbecher, S.: Human-computer interfaces for interaction with surgical tools in robotic surgery. Biomed. Robot. Biomechatron. (BioRob) 4, 81–86 (2012) 11. Staub, C., Lenz, C., Panin, G., Knoll, A., Bauernschmitt, R.: Contour-based surgical instrument tracking supported by kinematic prediction. In: Editor, F., Editor, S. (eds.) Biomedical Robotics and Biomechatronics (BioRob 2010), 3rd IEEE RAS and EMBS International Conference, pp. 746–752 (2016). https://doi.org/10.1109/ BIOROB.2010.5628075 12. Jalaliniya, S., Smith, J., Sousa, M., B¨ uthe, L., Pederson, T.: Touch-less interaction with medical images using hand & foot gestures. In: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 1265–1274. Association for Computing Machinery, New York (2013) 13. Xia, X., Irani, P., Wang, J.: Evaluation of Guiard’s theory of bimanual control for navigation and selection. In: International Conference on Ergonomics and Health Aspects of Work with Computers, Orlando, FL, USA, pp. 368–377. Springer (2011) 14. Youssef, Y., Lee, G.I., Godinez, C.: Laparoscopic cholecystectomy poses physical injury risk to surgeons: analysis of hand technique and standing position. Surg. Endosc. 25(7), 2168–2174 (2011) 15. Kazanzides, P., Chen, Z., Deguet, A., Fischer, G.S., Taylor, R.H., DiMaio, S.P.: An open-source research kit for the da Vinci(R) surgical system. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 6434–6439. IEEE, Hong Kong (2014) 16. Buxton, W., Myers, B.: A study in two-handed input. In: Proceedings of CHI 1986, pp. 321–326. ACM (1986)

Tendon Force Control Evaluation for an Endoscope with Series Elastic Actuation Lorin Fasel1(B) , Nicolas Gerig1 , Philippe C. Cattin2 , and Georg Rauter1 1

BIROMED-Lab, Department of Biomedical Engineering, University of Basel, Basel, Switzerland [email protected] 2 CIAN, Department of Biomedical Engineering, University of Basel, Basel, Switzerland

Abstract. While telemanipulating a surgical robot, surgeons usually miss haptic feedback, i.e., they cannot feel interactions between the surgical instruments and the tissue. The lack of haptic feedback makes the surgeons solely dependent on visual information, which might render the handling of surgical robots unintuitive and thereby poses a potential safety threat. To overcome these limitations, we propose to use Series Elastic Actuation (SEA) for tendon-driven endoscopes. In this regard, we developed an endoscope prototype with SEA to demonstrate its potential for safer robot-tissue interactions: A spring was integrated into the tendon transmission between the motor and the movable endoscope tip. The deflection of the spring was measured and used to estimate the tendon force. The estimated tendon force was then fed back to a force controller. Our tendon force estimation had lower noise levels than that of a commercial force/torque sensor, but showed a systematic error, which we assume to originate from friction. In response to an input step, the tendon force settled to the desired value within 50 ms and the steady-state error was below 0.05 N. These results showed that SEA is a promising concept for force measurements and robust force control in tendon-driven endoscopes. Keywords: Series elastic actuation · Surgical robotics endoscope · Tendon-driven endoscope

1

· Articulated

Introduction

Over the last decades, surgeons have aimed at reducing the size and number of skin incisions required for an intervention. Although these minimally invasive surgeries (MIS) bring many benefits to the patient, it is complicated and often unintuitive for the surgeon to manually manipulate the surgical instruments through a small opening in the skin [9]. Introducing robots to assist with MIS could make instrument manipulation easier, which would allow the surgeon to focus on the actual surgical tasks. One possible setup for robot-assisted surgeries c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 118–126, 2021. https://doi.org/10.1007/978-3-030-58104-6_14

Tendon Force Control Evaluation for an Endoscope with SEA

119

features the robot at the operating table and the surgeon controlling it from a telemanipulation console, therefore eliminating the direct mechanical connection between surgeon and surgical instruments. In general, the main advantage of telemanipulation over manual MIS is precise motion control of the surgical instruments (e.g., by filtering tremor and motion scaling). However, to steer the surgical instruments, the surgeon currently relies solely on visual feedback but does generally not feel the interaction with the tissue. In comparison to conventional surgeries, this lack of haptic feedback makes it challenging to perform certain surgical tasks (e.g., palpation) despite high performance in device positioning. In other situations, the lack of haptic feedback might even pose a safety threat, e.g., when collisions between the surgical instruments and the surrounding tissue occur outside the visual field of the endoscope camera. Other work has identified the lack of haptic feedback as one of the main drawbacks of current robotic surgical systems [9,11] and efforts have been made to provide haptic feedback to the surgeon. However, different factors have obstructed the implementation of haptic feedback into robotic surgical systems. Haptic perception is not a passive, unidirectional perception like vision or hearing, but it always involves the exchange of mechanical energy in two directions. Because of this bidirectionality, it is difficult to achieve stable control for devices with haptic feedback [2]. Furthermore, it is necessary to measure contact forces between the endoscope and the tissue, but embedding force, torque, or pressure sensors into the endoscope tip has proven to be difficult. Sensors would need to be small, reliable, and able to withstand sterilization procedures [6]. While some attempts have been made to include a miniature force sensor directly into the endoscope tip (e.g., [8,10]), another possibility is to measure the forces indirectly, e.g., by measuring the current of the external motors [13] or by using an external force/torque sensor [12]. Often, the end-effector of robotic endoscopes for MIS is controlled with remote electric motors, and tendons transmit the motion to the end-effector. The tendon tension can then be measured externally with a load cell [4]. Force measurements from stiff load cells, however, have high noise levels and are sensitive to temperature changes. Another approach is to measure the tendon elongation and to use Hooke’s law to estimate the tendon tension [3]. An algorithm was proposed to control the position of the end-effector tip by varying the estimated tendon tension [1]. However, these approaches create the need to place a position sensor at the distal end of the endoscope. Instead of modeling the nonlinear elongation of the tendons, our idea is to deliberately include an elastic element with a known linear stress-strain behavior (e.g., a spring) in the transmission between the motor and the load. This kind of actuation is called Series Elastic Actuation (SEA) [5]. It stands in contrast to traditional actuation and transmission systems, which are designed to be as stiff as possible. For a revolute SEA, the motor position qm and the load position ql can be used to compute the deflection of the spring and therefore the torque applied on the load as (1) τl = ks (qm − ql ) ,

120

L. Fasel et al.

where ks is the torsional spring stiffness. The spring, and therefore the encoders, can be placed close to the motors outside the endoscope. Even though this configuration does not eliminate the effects of tendon friction on the force measurement, it allows for a more robust measurement because it turns the force measurement problem into a position measurement problem. The spring acts as a low-pass filter between the motor and the load [5]. On one hand it reduces peak forces in the transmission and thus improves the shock tolerance, on the other hand it also limits the achievable bandwidth of the output force, creating a trade-off in the system design. However, surgical tasks do usually not involve highly dynamic movements. Active research on SEA is done in other fields of robotics (e.g., legged robotics or rehabilitation robotics), but to our best knowledge, SEA has not been applied to tendon-driven endoscopes. With this paper we aim to demonstrate the potential of employing SEA in surgical robots by presenting preliminary results of an endoscope prototype with SEA.

2

Methods

To prove the concept of using SEA in a tendon-driven endoscope, we designed and manufactured an experimental setup (Fig. 1). The endoscope prototype had two degrees of freedom, but only one degree of freedom was used for this first experiment: Two antagonistic tendons (tendon number is denoted by index i) were used to pitch the endoscope tip up (i = 1) and down (i = 2).

Fig. 1. The experimental setup consisted of an endoscope prototype, an actuation pack, and a control system (not shown). Two antagonistic tendons were used to pitch the endoscope up- and downwards. The tendons were pulled by winches, which were connected to electric motors through torsional springs.

At one end, the tendons were attached at the endoscope tip. At the other end, they were wound up on a winch. Applying a torque on the winch resulted in

Tendon Force Control Evaluation for an Endoscope with SEA

121

a force Fi on tendon number i, and consequently a movement ϕ of the endoscope tip. The torque on the winch was applied by a DC motor (Maxon Motor AG, Sachseln, Switzerland). However, the winch and the motor were not connected through a rigid axis but through a torsional spring with stiffness ks . Two 14bit absolute encoders (AMT23, CUI Devices, Tualatin, OR, USA) were used per axis i: One was placed between the motor and the spring to measure the motor position qi,m ; the other one between the spring and the winch to measure the position of the load qi,l . This placement of encoders enabled computing the deflection of the spring and therefore the torque applied on the load according to Eq. (1). The estimated force on tendon i could be computed as follows: ks (qi,m − qi,l ) Fˆi = , rw

(2)

where rw is the winch radius. The motors were operated in torque control mode (provided by the motor drives). For axis i, the commanded motor torque τi,cmd was computed in a control algorithm (Fig. 2) running at a cycle time of 1 ms on a real-time PC (Beckhoff Automation GmbH & Co. KG, Verl, Germany). This torque command was the output of the force controller Ci,force , which was minimizing the error between the desired tendon force Fi,des and the estimated tendon force Fˆi , as computed in Eq. (2). Since the torque can be directly commanded to the motor drive, this controller added the desired tendon force as a feedforward term to the integral of the error:   KI T s  Fi,des − Fˆi τi,cmd = rw Fi,des + , (3) z−1 where rw is the winch radius, KI is the integral gain, and Ts = 1 ms is the control cycle time. While the feedforward term was used to react quickly to changes in Fdes , the integral feedback part mainly served to eliminate the steady-state error. This force control could then be used for different applications. As a test case, we implemented a position controller Cpos in configuration space, thus a ˆ A controller to minimize the position error of the endoscope joint Δϕ = ϕdes − ϕ. detailed description and evaluation of the position control was not in the scope of this paper. However, the output of such a controller would be the endoscope joint torque command τϕ,cmd . From that, the desired tendon force Fi,des can be derived in the force command computation. When computing the desired tendon force, it is important to keep in mind that only pulling forces can be exerted on the tendon. To avoid tendon slack (and thus backlash), a minimal pre-tension force Fmin was applied to the tendons. The output of the position controller was added to this pre-tension force only if the controller output corresponded to a pulling force on the tendon. 2.1

Determining Spring Stiffness

Before ordering the torsional springs, we evaluated the requirements on the springs and selected the spring stiffness accordingly. The requirements will

122

L. Fasel et al.

Fig. 2. The proposed control scheme for a tendon-driven endoscope with SEA has a separate force controller Ci,force for each of the antagonistic tendons 1 and 2. The controller Ci,force is minimizing the error between the estimated tendon force Fˆi and the desired tendon force Fi,des that it receives from the high-level control. The form of this high-level control depends on the application – as an example, a configurationspace position controller Cpos is shown (dashed lines), which also ensures a minimal pre-tension force Fmin .

depend on the exact surgical application, but some assumptions can be made. For example, the necessary control bandwidth (i.e., up to which frequencies the output force can follow the reference) is assumed to be low for surgical applications. Generally, the lower the required bandwidth, the lower is the allowed spring stiffness [7]. Furthermore, the output impedance (i.e., with how much force the endoscope resists when subjected to a disturbance motion) should also be low in surgeries to maintain small interaction forces when getting into undesired contact with human tissue. Decreasing the spring stiffness will reduce the mechanical impedance, especially at high frequencies [7]. Without identifying the exact surgical requirements, for our test stand, we chose a torsional spring with stiffness ks = 9 × 10−04 Nm/◦ (Durovis AG, Perlen, Switzerland). 2.2

Parameter Tuning and Control Evaluation

Once the actuation pack was built, the force estimation model was evaluated. For that, one tendon was not yet fixed to the endoscope but attached to a force sensor (Mini45 F/T Sensor, ATI Industrial Automation, Inc., Apex, NC, USA). The motor positions were held fixed and the force sensor was moved by hand to apply a pulling force on the tendons. The tendon force was measured and compared with the estimated values as computed in Eq. (2). In this first experiment, no spring parameter calibration was performed, but instead, the product’s specified stiffness was used. The control parameters of the inner control loop (i.e., the integral force controller with feedforward term) were tuned first. To facilitate the tuning procedure and to prevent damage to the system, an aluminum tube was placed over the endoscope to block any movement of the endoscope tip. The tuning of the inner control loop was evaluated by analyzing the time response of the system: By commanding a step from 0.5 N to 1.5 N in the desired tendon force and by commanding a constant tendon force of 0.5 N. The step was not

Tendon Force Control Evaluation for an Endoscope with SEA

123

started from 0 N but from 0.5 N because the system is under constant pre-tension when in operation.

3

Results

For the evaluation of the tendon force estimation, data was recorded over a period of 27 s (Fig. 3). In this period, the mean absolute error between the estimated and measured values was 0.33 N, the maximum absolute error was 0.89 N.

Fig. 3. The tendon force estimation using spring deflection (Fˆ2 ) was compared to the tendon force measured with a force sensor (Fmini45 ). The left side shows the measured data over a period of 27 s, the right side is a magnification at the location of the maximum absolute error (shown in grey on the left side). On the bottom, the absolute error between measured and estimated data is shown.

The tuning of the force controller (I-Gain KI = 7.9) resulted in closed-loop stability of the inner control loop. The response of the system to a step in desired tendon force from 0.5 N to 1.5 N was analyzed to characterize the control performance (Fig. 4). For the tendon corresponding to downward motion (i = 2), the step resulted in an overshoot of 0.57 N (rise time t90 = 11 ms). The measured tendon forces of both tendons had a settling time (±5%, i.e., ±0.05 N) of less than 50 ms. The commanded force for each tendon (i.e., the control signal to the motors) settled to 1.62 N and 1.51 N after 100 ms. At a constant low tendon force of 0.5 N, the system entered a limit cycle but stayed within an error band of ±0.02 N (Fig. 5).

124

L. Fasel et al.

Fig. 4. Tendon force step response: The solid lines (Fˆ1 and Fˆ2 ) represent the tendon forces measured through spring deflection, the dotted lines (F1,cmd and F2,cmd ) represent the force commanded to the motors.

4

Fig. 5. Limit cycle at a constant desired tendon force of 0.5 N: The measured force never settled at the exact desired value, thus the integral of the error built up until the commanded force was large enough to overcome static friction.

Discussion

Measuring spring deflection resulted in a good estimate of the tendon force, even without calibrating spring stiffness parameters (Fig. 3). The measured signal showed less noise than the signal of a commercial strain gauge force sensor. However, the measurements had a gain and offset error. The gain error likely occurred because the actual spring stiffness differed from the value in the specifications, which can be accounted for with calibration. We assume that the offset error originated from the friction in the system. Since the axis of the winch and the axis of the motor shaft were slightly misaligned, the frictional resistance varied depending on the winch position. The effects of friction can also be observed in the time response of the force control system. High static friction in combination with integral control can generate limit cycles as seen in Fig. 5, because the motor will never settle precisely at the correct location. This causes the error integral to build up until the torque command τcmd overcomes the resistance from static friction. Since the Coulomb friction is low, the actual tendon force overshoots the setpoint again, causing the system to go into an oscillation around the setpoint. This stick-slip behavior happened mainly at low tendon forces, where the torque command is not large enough to overcome static friction. No limit cycles were observed at 1.5 N pre-tension. Hence, increasing the minimal tendon force Fmin could prevent limit cycles. The pre-tension will determine the endoscope’s stiffness, which will be chosen according to clinical requirements (e.g., which contact force the surrounding tissue can tolerate). Since the desired stiffness might vary throughout a surgery, the readily adjustable pre-tension is an advantage of our setup. The stiffness could be increased for tasks where precise

Tendon Force Control Evaluation for an Endoscope with SEA

125

motion control is required, and it could be decreased when the endoscope needs to be compliant.

5

Conclusion

In this paper we have reported the successful commissioning of a tendon-driven endoscope prototype with SEA. The conducted experiments showed that the use of SEA is a promising method for force measurements and robust force control in tendon-driven devices. Our proposed configuration with antagonistic tendon pairs and two motors per endoscope joint makes it possible to vary the endoscope joint stiffness during the surgery. A high-level control suitable for surgical applications has yet to be implemented, but could lead towards safer robot-tissue interactions and the implementation of haptic feedback in surgical robotics. Acknowledgements. We would like to thank Sara Pensotti, who helped in the development of the tendon-driven endoscope prototype with SEA during her master thesis. The work in this paper was conducted as part of the MIRACLE project and we are grateful for the generous funding by the Werner Siemens Foundation.

References 1. Au, S.K.W., Prisco, G.M.: Tension control in actuation of multi-joint medical instruments. US Patent 9,101,379 B2 (2015) 2. Enayati, N., De Momi, E., Ferrigno, G.: Haptics in robot-assisted surgery: challenges and benefits. IEEE Rev. Biomed. Eng. 9, 49–65 (2016) 3. Haghighipanah, M., Miyasaka, M., Hannaford, B.: Utilizing elasticity of cabledriven surgical robot to estimate cable tension and external force. IEEE Robotics Autom. Lett. 2(3), 1593–1600 (2017) 4. Phee, S.J., Low, S.C., Dario, P., Menciassi, A.: Tendon sheath analysis for estimation of distal end force and elongation for sensorless distal end. Robotica 28(7), 1073–1082 (2010) 5. Pratt, G., Williamson, M.: Series elastic actuators. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE Computer Society Press (1995) 6. Puangmali, P., Althoefer, K., Seneviratne, L.D., Murphy, D., Dasgupta, P.: Stateof-the-art in force and tactile sensing for minimally invasive surgery. IEEE Sens. J. 8(4), 371–381 (2008) 7. Robinson, D.W.: Design and analysis of series elasticity in closed-loop actuator force control. Ph.D. thesis, Massachusetts Institute of Technology (2000) 8. Seibold, U., Kubler, B., Hirzinger, G.: Prototype of instrument for minimally invasive surgery with 6-axis force sensing capability. In: Proceedings of IEEE International Conference on Robotics Automation. IEEE (2005) 9. Simaan, N., Yasin, R.M., Wang, L.: Medical technologies and challenges of robotassisted minimally invasive intervention and diagnostics. Annu. Rev. Control Robotics Auton. Syst. 1(1), 465–490 (2018)

126

L. Fasel et al.

10. Suˇsi´c, I., Cattin, P., Zam, A., Rauter, G.: Versatile, force range-adjustable, triaxial force sensor with integrated micro camera for the tip of endoscopic devices. In: Hamlyn Symposium on Medical Robotics. Imperial College London, June 2018 11. Vitiello, V., Lee, S.L., Cundy, T.P., Yang, G.Z.: Emerging robotic platforms for minimally invasive surgery. IEEE Rev. Biomed. Eng. 6, 111–126 (2013) 12. Zemiti, N., Ortmaier, T., Morel, G.: A new robot for force control in minimally invasive surgery. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE (2004) 13. Zhao, B., Nelson, C.A.: Sensorless force sensing for minimally invasive surgery. J. Med. Devices 9(4), 0410121–0410124 (2015)

Design Evaluation of a Stabilized, Walking Endoscope Tip Manuela Eugster1(B) , Melanie Oliveira Barros1 , Philippe C. Cattin2 , and Georg Rauter1 1

BIROMED-Lab, Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland [email protected] 2 CIAN, Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland https://biromed.dbe.unibas.ch

Abstract. A rigid attachment of a surgical tool to the target tissue can improve the tool manipulation accuracy in minimally invasive robotic surgery. The main challenges in developing an attachment concept for stabilizing a surgical tool are not to restrict the task-relevant workspace of the device and not to increase the invasiveness of the procedure by either the attachment concept itself or additional incisions needed to reposition the surgical tool. We developed an upscaled prototype of a parallel mechanism for minimally invasive surgery integrated into an endoscope tip that can attach to the target tissue with its two legs. An actuated leg design that can fold, unfold, and attach to the target surface was designed, manufactured, and evaluated. We demonstrate that by the actuated leg design, the parallel robot can reposition its two legs independently and can “walk” along the target surface. Keywords: Medical robot Laserosteotomy

1

· Parallel robot · Workspace · Stability ·

Introduction

In medical robotics, accurate tool manipulation is of great importance. Especially for minimally invasive procedures, a stable geometric relation between the active tool and the target tissue is an essential advantage. Devices such as the HeartLander, a parallel wire-robot, which attaches to the heart [1], the trackguided ultrasound probe for application on the liver [8], or the MARS robot, which is mounted on the bone to allow precise drilling or needle positioning in surgical procedures [7], already put this principle into practice. Further examples can be found in the fields of orthopeadic surgery [5], flexible endoscopy [6], and cochlear implant surgery [4]. The main advantages of attaching the surgical tool to the target tissue are, to our understanding, increased robustness against mechanical disturbances, e.g., c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 127–135, 2021. https://doi.org/10.1007/978-3-030-58104-6_15

128

M. Eugster et al.

Fig. 1. Overall design: The device consists of a robotic endoscope (1) with a parallel mechanism at its tip (2). The parallel mechanism has two arms (3) on each side, which are rotatably mounted on legs (4) that can attach to the bone. The main body of the parallel mechanism (2) is used to guide the toll, e.g., a laser (5) for accurate bone cutting. The design of these legs has to allow two states: The storage state (S), where the legs are retracted to length ls and released, as well as the attachment state (A), where the legs are extended to length la and fixed to the bone.

patient movement, the increased tool positioning accuracy, and the decreased amount of needed intra-operative registration, tracking, and imaging processes. The main challenges in developing an attachment concept for stabilizing a surgical tool are not to restrict the task-relevant workspace of the device, and not to increase the invasiveness of the procedure by either the attachment concept or additional incisions needed to reposition the device. We developed an upscaled prototype of a parallel mechanism for minimally invasive surgery integrated into an endoscope tip [2]. This mechanism has two legs that connect the moving platform of the parallel mechanism (endoscope tip) with the bone (Fig. 1). One possible application of the device is minimally invasive cutting of bone (laserosteotomy), e.g., in joint replacement surgeries. The robot’s two legs need to attach to the bone surface and provide a stable relative connection between the target tissue and the laser tool to allow an accurate laser cutting process. To avoid the limitation of the device’s operating range to its workspace, in addition to stabilizing the tool, the topology of the parallel mechanism allows the mechanism to reposition its two legs independently and, therefore, to “walk” along the bone surface. This walking functionality enables the device to expand its workspace without requiring an additional skin incision. The walking functionality requires that the legs, which connect the device to the target tissue, allow independent attachment and detachment of the parallel mechanism to the environment, e.g., bone. During the repositioning of the leg, the leg length needs to be reduced in order not to collide with the surface below. Furthermore, the legs need to be storable close to the mechanism particularly during incision and retraction to reduce the overall device’s dimension and, therefore the required incision size.

Design Evaluation of a Stabilized, Walking Endoscope Tip

129

Fig. 2. Dimensions of the suction cup design (millimeters) and the two 3D-printed suction cups (alphacam GmbH, Schorndorf, Switzerland). We ordered two suction cups with different shore hardness for comparison (50 A and 60 A).

This paper presents two different leg designs that allow folding and unfolding of the leg structure. We further present a suction mechanism for attaching and releasing the leg to/from the target tissue. The designs are manufactured and tested for an up-scaled prototype of the mechanism that was presented in previous work [3]. Finally, also the successful realization of the device’s functionality to reposition itself and, therefore, expand its workspace is presented.

2 2.1

Methods Attachment Concept

To stabilize a tool with respect to the target tissue, we believe that a concept including a connection between the tool and the target tissue increases robustness against mechanical disturbances on the patient or the robotic endoscope. The attachment method should also be effective on uneven and moist surfaces. Furthermore, the attachment element should not increase the skin opening for inserting the robotic endoscope. Therefore, we considered different principles based on soft materials such as suction cups and suction pads (Fig. 2). Concepts that entail an additional level of invasiveness, such as bone pins and bone screws, were not considered. 2.2

Foldable Leg Structure

The foldable leg structure must allow storing the legs such that the skin incision for the insertion of the device is minimized. At the same time, the design should ensure that the needed leg length for attachment of the device can be reached. Also, each leg needs to house two rotary joints on which two arms of the parallel mechanism are mounted. The two proposed designs of the foldable structure were inspired by tendon telescopes and pneumatic telescopes. Based on the dimensions of the up-scaled prototype, which was built with a scaling factor of 5 : 1, the leg design needs to allow a storage length of ls ≤ 30 mm and an unfolded length of la ≥ 40 mm.

130

M. Eugster et al.

Fig. 3. Foldable pneumatic leg structure: The figure shows the different components in the assembled (left) and exploded (right) state. Folding (1) and unfolding (2) is enabled by pneumatics. An air inlet (blue tube) can provide pressurized air or vacuum. On the bottom right side, the manufactured prototype is shown in the storage state (S) and attached state (A).

Pneumatic Design. The pneumatic leg design is based on nested aluminum and brass cylinders. This combination of materials improves the sliding and sealing properties of the mechanism. The structure consists of three cylinders, two of which are telescopic and can slide out from another cylinder to lengthen the leg (Fig. 3). A tube mounted on top of the leg structure provides an air inlet for the first cylinder. The tube is directly screwed in a corresponding thread in the cap of the first cylinder. The inner thread of the cap directly cuts an outer thread into the tube, which leads to a sealing preventing the outflow of air. When pressure is applied via the tube, air fills the cylinder chambers, and the telescopic leg structure unfolds. When a vacuum is applied, the telescopic leg structure folds back to the storage state. The used components for generating pressurized air and vacuum are a Laval nozzle (VN-05-H-T3-PQ2-VQ2-RO1) and a manual pressure valve (LRS-1/8-D-I-MINI) by Festo (Festo AG & Co. KG, Esslingen, Germany). Tendon Based Design. The tendon design consists of nested aluminum and brass cylinders with milled tendon guides to avoid wear of the tendons. In this first design, we used fishing wire with a diameter of 0.12 mm and a break load of 7 kg. The actuation of the leg structure is based on two tendon mechanisms. One tendon mechanism has to be pulled while the other one is released and vice versa to fold and unfold the leg structure. The folding tendon mechanism consists of two tendons that are attached to the innermost cylinder. Pulling these two tendons leads to a folding of the telescopic structure. The unfolding tendon

Design Evaluation of a Stabilized, Walking Endoscope Tip

131

Fig. 4. Tendon based leg structure: Folding (1) and unfolding (2) is achieved by pulling the corresponding tendons. The bottom right shows the manufactured prototype in the storage state (S) and the attachment state (A).

mechanism is based on one tendon that enters the leg structure from one side, is guided through the cylinders, and exits the leg structure on the opposite side. The unfolding tendon is connecting the cylinders in such a way that pulling this tendon at both ends leads to an unfolding of the leg structure (Fig. 4). 2.3

Performed Experiments

The theoretical attachment force of a suction cup can be calculated as: F = Δp · A,

(1)

where A = 157 mm2 is the suction area and Δp = 1 atm is the applied pressure difference, resulting in a theoretical force of F = 7.96 N. The effective attachment force on the bone will be smaller. Therefore, we measured the break-away force of the suction cup orthogonal to the attachment surface on a piece of pig bone using a six-axis force/torque sensor (Nano17, ATI Industrial Automation, NC, USA). The force/torque sensor was mounted between the suction cup and a setup to pull on the suction cup. The break-away force of the suction cup was measured ten times. For each measurement, the suction cup was attached to the pig bone and then slowly and orthogonally pulled away from the bone until it released, while the force/torque sensor measured the forces acting on the suction cup (Fig. 5). The two foldable leg designs were manufactured, assembled, and evaluated based on their performance during repeated unfolding and folding cycles. The better design was selected to be implemented in the final leg design for the upscaled prototype. The final leg design was integrated into the upscaled prototype and the walking functionality to expand the workspace of the mechanism was tested. Therefore, the parallel mechanism and the pneumatic components were controlled to

132

M. Eugster et al.

Fig. 5. Break-away force measurement setup: The suction cup is mounted on a fixture that allows pulling the suction cup away from the attachment surface on the pig bone. A force/torque sensor is mounted between the suction cup and the fixture to measure the break-away force.

subsequently detach, move, and attach each of the parallel mechanism’s two legs. The topology, kinematics, and control of the parallel mechanism are described in previous work [3].

3

Results

The suction cups reached a mean break-away force of 3.78 N and 3.95 N with a standard deviation of 0.19 N and 0.12 N (shore hardness of 50 A and 60 A, respectively). This value is about half of the theoretical suction force (Eq. 1). The pneumatic telescope design performed better during evaluation since the tendon based concept has the disadvantage that the tendons show wear and tend to tear apart after several usage cycles. For the final design, we selected the pneumatic leg structure in combination with the suction cups. Two pneumatic tubes are connected to the leg. One is screwed to the top, which enables folding and unfolding of the leg structure by applying pressure or vacuum. The other is screwed on the connection element and allows applying suction for attachment of the suction cup (Fig. 6). We mounted the final leg design on the prototype, and the different phases of repositioning the mechanism to expand its workspace were successfully carried out (Fig. 7).

4

Discussion

We presented two different designs for a folding leg structure with an attachment element for tool stabilization in minimally invasive surgery. In the experiments performed, the pneumatic leg design showed better performance than the tendon based design, mainly due to the wear of the tendons after repeated usage cycles. We expect that the tendon wear will get worse when the design is miniaturized due to smaller bending radii and sharper edges. However, we are aware that the

Design Evaluation of a Stabilized, Walking Endoscope Tip

133

Fig. 6. Final design of the foldable leg structure, where the pneumatic leg structure is combined with a suction cup. Two air inlets (blue tubes) can provide pressurized air or vacuum. One tube allows folding and unfolding of the leg structure, while the other tube allows attachment and release of the suction cup by applying pressurized air or vacuum. The suction cup attaches to the surface below the mechanism. For the fast release of the suction cup, pressure can be applied to the second tube.

evaluation of further performance measures such as the stiffness of the leg in the unfolded state, or the force that can be counteracted by the leg when unfolded, might lead to a better performance of the tendon based design. A final design was implemented and tested on the up-scaled prototype for minimally invasive laserosteotomy. The current design requires two tubes to supply the actuated leg, one entering from the top, the other from the side. We believe that decreasing the tube diameters will allow better integration close to the mechanism and through the endoscope. However, reducing the tube diameter will influence the pneumatic supply. The measured break-away force of the suction cups was about half of the calculated theoretical suction force. We assume that this reduction is due to a smaller pressure difference, sealing problems between the suction cup and the bone, and the uneven bone surface. We consider the miniaturization of the actuated leg a challenging task, especially since the attachment force decreases with the square of the suction cups’ radius. Therefore, an alternative design of the attachment concept with multiple suction cups [9], foldable suction cups [10], or even a combination with a more invasive attachment concept, such as bone pins or screws, might be necessary to provide the needed stability. The walking functionality of the mechanism with the designed actuated leg could be demonstrated successfully in a lab setting. Further experiments have to be carried out to evaluate the walking functionality in a more realistic setting.

134

M. Eugster et al.

Fig. 7. Insertion and walking function of the endoscope tip. Upper row: top view, lower row: side view. The developed leg design is mounted on the prototype, and the different phases of operation are illustrated: Insertion (1), where the legs are stored close to the mechanism. Attachment (2) at the initial location (blue frame), where the legs are unfolded, and the suction cups are attached to the ground. In this state, the mechanism can perform laser cutting (L). Successive repositioning of the legs (3 to 5) to reposition the mechanism. Reattachment of the mechanism (5) at the new location (red frame), where the cutting is continued (L). After the cut is finished, the mechanism can be retracted by detaching, folding and storing the legs.

5

Conclusion

In this work, we developed, manufactured, and tested two designs for an actuated leg that can fold, unfold, and attach to a surface. In combination with the parallel mechanism that guides a surgical tool, this actuated leg will allow stabilizing the surgical tool by an attachment to the target tissue, while not restricting the task-relevant workspace of the device. The leg was implemented in an upscaled prototype of the parallel mechanism, and the walking functionality to expand the robot’s workspace was demonstrated successfully. Acknowledgements. We gratefully acknowledge funding of the Werner Siemens Foundation through the MIRACLE project and we thank Prof. Dr. med. Niklaus F. Friederich for his continuous support with respect to medical questions.

Design Evaluation of a Stabilized, Walking Endoscope Tip

135

References 1. Costanza, A.D., Wood, N.A., Passineau, M.J., Moraca, R.J., Bailey, S.H., Yoshizumi, T., Riviere, C.N.: A parallel wire robot for epicardial interventions. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 6155–6158. IEEE (2014). https://doi.org/10.1109/EMBC. 2014.6945034 2. Eugster, M., Weber, P., Cattin, P., Zam, A., Kosa, G., Rauter, G.: Positioning and stabilization of a minimally invasive laser osteotome. In: Hamlyn Symposium on Medical Robotics, vol. 10, pp. 21–22 (2017) 3. Eugster, M., Cattin, P.C., Zam, A., Rauter, G.: A parallel robotic mechanism for the stabilization and guidance of an endoscope tip in laser osteotomy. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1306–1311. IEEE (2018). https://doi.org/10.1109/IROS.2018.8594188 4. Kratchman, L.B., Blachon, G.S., Withrow, T.J., Balachandran, R., Labadie, R.F., Webster, R.J.: Design of a bone-attached parallel robot for percutaneous cochlear implantation. IEEE Trans. Biomed. Eng. 58(10), 2904–2910 (2011). https://doi. org/10.1109/TBME.2011.2162512 5. Plaskos, C., Cinquin, P., Lavall´ee, S., Hodgson, A.: Praxiteles: a miniature bonemounted robot for minimal access total knee arthroplasty. Int. J. Med. Robot. Comput. Assist. Surg. 1(4), 67–79 (2005). https://doi.org/10.1002/rcs.59 6. Ranzani, T., Russo, S., Schwab, F., Walsh, C.J., Wood, R.J.: Deployable stabilization mechanisms for endoscopic procedures. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1125–1131. IEEE (2017). https:// doi.org/10.1109/ICRA.2017.7989134 7. Shoham, M., Burman, M., Zehavi, E., Joskowicz, L., Batkilin, E., Kunicher, Y.: Bone-mounted miniature robot for surgical procedures: concept and clinical applications. IEEE Trans. Robot. Autom. 19(5), 893–901 (2003). https://doi.org/10. 1109/TRA.2003.817075 8. Stilli, A., Dimitrakakis, E., Tran, M., Stoyanov, D.: Track-guided ultrasound scanning for tumour margins outlining in robot-assisted partial nephrectomy. In: Joint Workshop on New Technologies for Computer/Robot Assisted Surgery (CRAS 2018), vol. 8. CRAS (2018) 9. Thanh-Vinh, N., Takahashi, H., Kan, T., Noda, K., Matsumoto, K., Shimoyama, I.: Micro suction cup array for wet/dry adhesion. In: 2011 IEEE 24th International Conference on Micro Electro Mechanical Systems, pp. 284–287. IEEE (2011). https://doi.org/10.1109/MEMSYS.2011.5734417 10. Zhakypov, Z., Heremans, F., Billard, A., Paik, J.: An origami-inspired reconfigurable suction gripper for picking objects with variable shape and size. IEEE Robot. Autom. Lett. 3(4), 2894–2901 (2018). https://doi.org/10.1109/LRA.2018. 2847403

Experimental Evaluation of Needle Tip Force Sensing Associated to Tactile Feedback for Improving Needle Remote Insertion Charl´elie Saudrais1 , Lennart Rubbert1 , Lisa Bonnefoy2 , Rui Zhu3 , Hubert Schneegans2 , Charles Baur2 , Ulrich Mescheder3 , and Pierre Renaud1(B) 1

AVR-ICube, Strasbourg University, INSA Strasbourg, Strasbourg, France [email protected] 2 Instant-Lab, EPFL, Lausanne, Switzerland 3 Hochschule Furtwangen, Furtwangen, Germany http://spirits.icube.unistra.fr/en/

Abstract. In this paper, we present an experimental evaluation of tactile feedback for remote needle insertion in soft tissues. A needle with tip force sensing is used to detect changes of tissues and puncture events during insertion. This information is provided to the user during teleoperation by means of a tactile display. Evaluation is conducted by reproducing needle insertion as performed during biopsy procedures in the liver. Needle tip force sensing makes the measurement of tissue change and puncture easier compared to force measurement at the base of the needle. Puncture events are correctly understood by the users. Change of tissue is also satisfactorily detected, which could be an interesting additional information for the user to perform surgical gesture. Keywords: Interventional radiology feedback · Force sensing

1

· Needle insertion · Tactile

Introduction

In interventional radiology, the use of X-ray based imaging devices represents a safety issue for radiologists due to radiations. MRI scanners also have the issue of limited access to the patient due to the bore geometry. Remote manipulation of needle could then improve the conditions for tasks such as biopsy performed by radiologists with these modalities. Radiologists achieve needle insertion relying on images and also insertion forces [4]. Because of the multiple visual sources of information in the operating room that the radiologist has to deal with, remote manipulation with force feedback to inform about the puncture of tissues and the change of tissues are then of interest [1]. In the following we focus on this objective, considering kidney and liver biopsy procedures. Such procedures consist in local anesthesia at the biopsy needle insertion point, before skin incision. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 136–142, 2021. https://doi.org/10.1007/978-3-030-58104-6_16

Tactile Feedback for Improving Needle Remote Insertion

137

A coaxial biopsy needle is then inserted after deep anesthesia along the planned path. The biopsy needle trajectory is monitored with a medical imaging device when going through abdominal muscle, fat and the organ of interest. When the needle tip reaches the point of interest, the biopsy is performed with a trigger mechanism before extracting the needle carrying the desired tissue specimen [2]. Teleoperation with force feedback to reflect insertion forces during the needle progression is possible, but challenging in terms of control [3]. An easier approach is to provide to the user only specific events such as the puncture of membranes surrounding tissues like muscle or liver, that also possess different stiffnesses and mechanical properties [4]. In [1], we showed that we were able to measure in lab conditions the punctures of membranes with great accuracy thanks to a Fabry-Perot load cell placed inside the needle, close to the tip. In [5], we investigated how to render information with a tactile feedback using a dynamic Braille display. In this paper, their combination is evaluated experimentally to investigate the capacity of users to understand change of tissues and occurrence of punctures during a needle insertion, and therefore to add to the medical image additional information with tactile feedback.

2 2.1

Methods Set-Up

A 16 G needle (diameter of 1.6 mm) with tip force sensing capability, which has been presented in [1] is inserted using a linear table (FB075, Nanomotion). To reproduce the biopsy, the needle insertion is performed at constant speed in a phantom composed of 3 layers (Fig. 1). These layers are produced with gelatin, using concentrations of 7%, 45% and 18% [6] to reproduce respectively the biomechanical properties, i.e. stiffness, of fat, muscle and organ (liver or kidney). The layers are separated by membranes composed of polyethylene cling film in order to reproduce the faciæ existing between the anatomical structures. The signal associated to the membrane puncture can be delivered to the user or not, so that three scenarios can be reproduced with two, one or no membrane between the layers. The phantom is rotated after 3 insertions that are shifted radially by 8 mm, so the insertions are never performed in the same volume of the phantom. 22.5◦ rotations are applied, therefore 48 insertions can be performed with a single phantom. The Fabry-Perrot load cell [1] located in the instrumented needle is connected to a FISOTM white light interferometer. The signal is then transferred to a PC for signal processing using Labview platform. For some needle insertions, force measurement at the base of the needle is performed for comparison purpose using a force sensor Scaime K1107 mounted between the needle and the linear table. The tactile feedback is provided by means of a so-called tactuator, composed of a Braille display based on a tactile actuator SC5 (KGS Corporation). The tactile display is an array of 8 × 8 dots (25 × 25 mm2 ) to produce dynamic patterns using an Arduino platform. A push button is also provided to the user to record puncture events. According to the initial results described in [5], the following two patterns are being used (Fig. 3): a moving horizontal line which frequency varies with the nature of the gel as detected with the needle tip, and a single ON/OFF signal generated at

138

C. Saudrais et al.

Fig. 1. The linear table holding the needle with tip force sensing capability and the phantom.

each puncture detection. The frequency varies proportionally with the measured force between 10 Hz and 25 Hz. In order to set the frequency range, one insertion was used to determine the minimum value of force, associated to 10 Hz, and the maximum value, associated to 25 Hz (Fig. 2). 2.2

Protocol

The evaluation is performed on 11 healthy subjects (9 male and 2 female subjects) between 22 and 47 years old. Each subject is first introduced to the protocol and the tactile feedback using a prerecorded sequence, in order to get familiar with the dynamic patterns generated with the tactuator. The training phase is two and half minutes long, to keep the learning phase short. Each subject is equipped with noise-canceling headset (Sony WH-1000XM3) and is placed in such a manner he cannot see nor hear the needle progression. Insertion start is indicated to the subject by a visual sign. When the subject considers a membrane puncture occurred, he presses the push button. In addition, he mentions orally if a tissue change occurs. After each insertion, the subject leaves the room keeping the noise canceling headset while the next insertion settings are prepared. In order to avoid learning of the sequence by the subject, the insertion sequences are generated randomly, modifying two parameters. First, the insertion speed is set to 2 mm/s or 4 mm/s (S = 1 or S = 2), so the time to go through a layer is not necessarily constant between two experiments. Second, the presence of the number of membrane (up to two) is not always reflected to the subject. Puncture of the first membrane (M1 = 0 or M1 = 1) and the second membrane (M2 = 0 or M2 = 1) are varied between two experiments. A total of 8 combinations of conditions S, M1, M2 exists. For each subject, 3 insertions are achieved choosing randomly a set of conditions. The subject is only aware of the number of layers in the phantom, and the possible presence of membranes.

Tactile Feedback for Improving Needle Remote Insertion

3 3.1

139

Results Needle Tip Force Sensing Performances

In Fig. 4, the force measurements as functions of the needle progression are represented for insertions at speeds of 2 mm/s (3 insertions) and 4 mm/s (3 insertions). The membrane punctures M1 and M2 are recorded synchronously for all measurements as expected. The evolution of the tip force is not dependent on the speed and needle insertion depth whereas the proximal force profiles exhibit different slopes. Using the tip force, it is quite simple to discriminate the nature of the layers. On the contrary the proximal force measurement is dependent on the needle velocity, so it is difficult then to discriminate the layers. In addition, after noise signal filtering, membrane puncture appears clearly on the tip force measurement, while for the first membrane it can be seen that the peak of proximal force measurement is hardly visible.

Fig. 2. Experimental set-up during evaluation with subject.

Fig. 3. Patterns for the puncture event (A) and the change of tissue (B).

140

C. Saudrais et al.

Fig. 4. Distal and proximal needle axial force measurement evolution with respect to the insertion depth at speed 2 mm/s and 4 mm/s in three various gels separated by membranes.

3.2

Experimental Evaluation

The Table 1 provides the results of the experimental evaluation with the tip force sensing needle (distal). The experiment has been performed on two different days with thus two different phantoms, but with the same gel compositions. The first 7 subjects starting from the left column of the table did the experiment the first day in a row, and the 4 remaining on the second day in a row. As a summary, 65 out of 66 membrane puncture detections (98%) have been successfully performed. 59 out of the 66 frequency changes (89%) have been successfully detected.

4

Discussion

With the considered instrumented needle, the ability to measure the force at the tip of the needle is proven to be clearly an added-value to build force feedback during needle insertion, as also observed in [7]. In fact, the insertion profile of the force measurement at the distal end of the needle allows us to detect in a simple way the puncture during the whole insertion. The phantom was here based on homogeneous layers. It will be interesting to further investigate force measurements with this design of needle based on Fabry-Perot interferometry, and how to make the signal processing robust to signals collected in in vivo conditions. The approach we propose to reflect to the user events related to the needle progression is quite simple. We actually limit the number of information sent to

Tactile Feedback for Improving Needle Remote Insertion

141

Table 1. Results of the experimental evaluation. The highlighted column can be interpreted as follows. The male subject is 23 years old. The first insertion has been set to an insertion speed of 4 mm/s (speed #2), simulating an absence of membrane M1 between the first and second gel layers, by not sending a signal to the tactuator, but with a membrane M2 between the second and third gel layers. The results are split in two parts: membrane puncture detection, and change of tissue. In case of the nonoccurrence of membrane M1 puncture, the subject did not press the push button, which was correct (1). In case of membrane puncture, the ON/OFF pattern has been sent to the tactuator and the subject pressed the push button which was also correct (1). The second part of the evaluation is the ability to detect a change of tissue which is interpreted by an increase or decrease of the frequency of the moving line pattern. For the change between the first and second layers (T12 with no membrane M1) the subject did not detect any change (0). However, for the second change between the second and third layers (T23 with membrane M2), the subject did detect the change (1).

Subject

Sex M M M M M M M F

F

M M

Age 27 26 23 23 26 26 47 25 25 22 35 S

122111121212121212221211121212221

Settings M1 001010101001110111111100110100010 M2 111111101100010010010011011111110 M1 111111111111111111111111111111111 Results

M2 111111110111111111111111111111111 T12111111111011110011111111111011111 T23110001111111111111111111111111111

the user, providing only one piece of information related to the nature of tissue and another one related to the occurrence of membrane puncture. As one can see, the absence of transition detection occurs equivalently for both transition (T12 and T23) and are not correlated with the absence or presence of a membrane (M1 or M2). The evaluation results are excellent (98%) for the puncture detection which is very encouraging in terms of safety improvement during teleoperated needle insertion as the tactile feedback comes as a complementary information to the visual feedback provided by the medical imaging device. Puncture information to take a decision about needle progression could be sufficient, as they have a priori knowledge about the anatomy. However, the feedback about change of tissue seems also feasible, and it could be a valuable complementary source of information if unexpected changes occur.

142

5

C. Saudrais et al.

Conclusion

In this paper, the interest of needle tip force sensing capabilities in remote needle insertion was presented and discussed as well as its experimental evaluation in combination with tactile force feedback. The interest of needle tip force sensing is demonstrated by comparing the insertion profiles of distal versus proximal force sensing when inserted in a phantom. Based on the profile of tip force measurement, puncture and changes of tissues could be well established and reconstructed as a tactile information with two distinct dynamic patterns. The experimental evaluation on 11 subjects with a phantom demonstrates that it is possible to detect puncture event which could improve the safety during remote needle insertion. Other information as the change of tissues which seems understandable through a tactile feedback could be a crucial information to avoid critical zones. Acknowledgements. This work was supported by the INTERREG Upper Rhine program from the ERDF (European Regional Development Fund), SPIRITS project (interreg-spirits.eu), and also by Investissements d’Avenir program (Robotex ANR10-EQPX-44, Labex CAMI ANR-11- LABX-0004). The authors thank Sensoptic SA for the work on needle with tip force sensing capabilities and Help Tech GmbH for providing the Braille display.

References 1. Schneegans, H., Rubbert, L., Rivera, J., Fifanski, S., Renaud, P., Henein, S., Baur, C.: Fiber optic Fabry-Perot interferometry for a biopsy needle with tip force sensing, IEEE IROS, Macau, China, 4-8 November 2019 (2019) 2. Pfeil, A., Barb´e, L., Wach, B., Cazzato, R., Gangi, A., Renaud, P.: Observations and experiments for the definition of a new robotic device dedicated To CT, CBCT and MRI-guided percutaneous procedures. In: Proceedings of International Conference of the IEEE Engineering in Medicine and Biology Society (2018) 3. Barb´e, L., Bayle, B., Laroche, E., de Mathelin, M.: User adapted control of force feedback teleoperators: evaluation and robustness analysis. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2008), Nice, France, September 2008 (2008) 4. Elayaperumal, S., Bae, J.H., Daniel, B.L., Cutkosky, M.R.: Detection of membrane puncture with haptic feedback using a tip-force sensing needle. In: IEEE International Conference on Intelligent Robots and Systems, (IROS), pp. 3975–3981 (2014) 5. Zhu, R., Rubbert, L., Renaud, P., Mescheder, U.: Determination of a tactile feedback strategy for use in robotized percutaneous procedures. In: 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, July 2019. IEEE (2019) 6. Ng, K.W., Goh, J.Q., Foo, S.L., Ting, P.H., Lee, T.K.: Needle insertion forces studies for optimal surgical modeling. Int. J. Biosci. Biochem. Bioinform. 3, 187–191 (2013) 7. Elayaperumal, S., Bae, J.H., Christensen, D., Cutkosky, M.R., Daniel, B.L., Costa, J.M., Black, R.J., Faridian, F., Moslehi, B.: MR-compatible biopsy needle with enhanced tip force sensing. In: Joint Eurohaptics Conference Symposium Haptic Interfaces Virtual Environ Teleoperator Systems (2015)

A Compliant Mechanism as a Sternum Prosthesis Octavio Ramirez1(B) , Christopher R. Torres-San-Miguel1 , Marco Ceccarelli2 , José Luis Rueda Arreguín1,2 , and Guillermo Urriolagoitia-Calderón1 1 Instituto Politécnico Nacional, Escuela Superior de Ingeniería Mecánica y Eléctrica Sección

de Estudios de Posgrado e Investigación, Unidad Profesional “Adolfo López Mateos” Zacatenco, Mexico City, Mexico [email protected] 2 LARM2: Laboratory of Robotics and Mechatronics, Department of Industrial Engineering, University of Rome Tor Vergata, 00133 Rome, Italy

Abstract. This paper presents a three-dimensional design of a compliant mechanism as a transmission system for its application in a novel human sternum prosthesis. The proposed compliant mechanism design is able to generate a coordinated movement between ribs during the respiratory process in human beings. Finite Element Method (FEM) analysis was performed in order to describe maximum stress values from the deformed elements of the mechanism by applying input forces. FEM results show that the compliant mechanism movements are proper for sternum prosthesis application, also it was possible to prove that maximum stress values generated in the mechanism, did not exceed the yield point of the material. Finally, by performing a fatigue analysis of the mechanism, it was possible to determine the maximum number of cycles that it can resist. Obtained results will be used to develop a complete design of the compliant mechanism and the sternum prosthesis in a prototype solution. Keywords: Compliant mechanism · Sternum prosthesis · Ribs movement

1 Introduction The design of medical devices like sternum prosthesis must consider different parameters to develop a suitable rehabilitation option for patients. This kind of medical devices are used to help for several kinds of pathological illnesses, like cancer. Although tumors in the sternum are not very common. They necessitate the removal of the sternum from the thoracic cage [1]. When it happens, a medical device (prosthesis) must be used in order to help patients to recover the natural function that they lost, but the development of these medical devices must consider a methodology in which characteristics are considered according to the patient, the pathology, age and the indications of the surgeon to establish the best alternatives that allow replacing the eliminated elements [2, 3]. Several sternum prostheses have been developed to replace a missing element. In 2015, the first sternum prosthesis was manufactured by 3D printing titanium alloy [4]. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 143–151, 2021. https://doi.org/10.1007/978-3-030-58104-6_17

144

O. Ramirez et al.

The reconstruction of this sternum was realized by using Computed Tomography (CT) of a thorax in order to keep the thoracic morphology. Another similar case was reported by Kamel [5], in which the sternum prosthesis was manufactured by 3D printing titanium alloy, with a porous part of polyethylene located in the middle section to allow the growth of the bone. In 2019 Aranda [6] developed a modular prosthesis with three independent elements, a central one and two pieces located on each side, that was in order to manufacture a low-cost prosthesis. There is only one report of a flexible sternum prosthesis that was developed in 2017 by Cano [7]. This prosthesis had a flexible system which allows free movements of the ribs during respiratory cycle. It was manufactured by 3D printing titanium alloy. Once the prosthesis was implanted, it provided good flexibility. A radioscopy study of the patient’s respiratory movements showed that the prosthesis moved well-coordinated with the rest of the ribcage, without restraints. Although this prosthesis has a flexible system, it is not considered as a compliant mechanism. The present work discusses the numerical analysis for a three-dimensional design of a compliant mechanism in order to verify if the obtained movement is suitable for its application in a novel design of sternum prosthesis. It is also verified that the mechanism works in the elastic range during its operation. The obtained results are presented in plots that show the produced stress on the mechanism as well as its displacements. As background of this work, the study of the mechanical behavior of fractured human ribs when using bone osteosynthesis implants have been developed [8] as well as the design of machines that have allowed to evaluate the behavior of the thoracic cage during an impact scenario [9].

2 Three-Dimensional Model and Numerical Analysis The use of compliant mechanisms is a good option for biomedical implants, because they have a low wear, the ability to be manufactured with biocompatible materials and it is also possible to develop very compact designs [10]. The compliant mechanism design presented below represents the author’s own development, which seeks to provide a coordinated movement among the human ribs, by using all the elements of the mechanism that are attached to a central element that is pushed by each rib. The respiratory process in living beings is determined by 2 stages, inspiration and expiration. The inspiratory muscles conduct air to the lung, while the expiratory muscles remove air from the lungs. The muscles involved in inspiration are the diaphragm, external intercostals, parasternal, sternomastoid and scalene. On the other hand, the expiratory muscles are the internal intercostal, rectus abdominis, external and internal oblique, finally the transverse abdominis muscles [11]. During inspiration, there is an expansion of the rib cage caused by the movement of the ribs, the sternum and the action of the inspiratory muscles [12]. The external intercostal muscles extend obliquely down from each rib to the rib below, the contraction of this muscle exerts a force that acts on the lower rib. The net effect of contracting these muscles elevates the rib cage (see Fig. 1). The sternomastoid and scalene muscles are secondary muscles in inspiration, the contraction of these muscles raises the sternum and the first two ribs [13].

A Compliant Mechanism as a Sternum Prosthesis

145

Fig. 1. Contraction of the external intercostal muscles during inspiration

The shape of the mechanism was designed to attend only the lifting movements that ribs carry out during the inhalation process in order to promote a coordinated movement that helps to have good ventilation in human beings. The dimensions of the mechanism are 60 mm width, 165 mm length and 8 mm thickness, real human sternum dimensions on an adult were established, these dimensions have a range between 183 to 209 mm length, 60 to 68 mm breadth and 10 to 13 mm thickness in accordance [14]. By considering the force that generates elevation of each rib (input), the compliant mechanism is able to generate an upward movement that in coordination with the corresponding rib generate a uniform lifting movement (output) (see Fig. 2). This mechanism has several arms on each side, each arm is connected at each free end of a rib. The compliant mechanism that is evaluated in this work was designed in Solid works® student version.

Fig. 2. A Computer Aided Design (CAD) model of the compliant mechanism as sternum prosthesis

146

O. Ramirez et al.

The mechanism consists of several arms on each side, these arms are used to connect to the free segments of each rib by means of an anchoring device that is attached to the rib cartilages by using a screw since this prosthesis replaces the sternum that has been removed. The arms are attached to a central structure that allows a coordinated movement in all the ribs during inhalation. The smaller cross-sectional elements are used to transmit the movement, while these elements are deformed by the action of the input force, an output displacement is generated as a consequence (see Fig. 3).

Fig. 3. Scheme of prosthesis application

Before starting the numerical analysis, it was necessary to determine the mechanical properties of the material, the proposed material was titanium alloy because it is a biocompatible material, it has good flexibility and it can currently be used in 3D printing machines [15]. The mechanical properties of this material are; Young’s module of 110 GPa and the Poisson ratio of 0, 3 and Yield strength of 827 MPa [16]. The analysis was considered as linear, elastic and isotropic. The function of this compliant mechanism is to encourage a coordinated movement of elevation in the ribs during inhalation. To analyze the movement, the pairs of ribs 3 to 7 are considered, and the external intercostal muscles are mainly responsible for generating this movement in those ribs. In a study by Ratnovsky, he developed a realistic two-dimensional model of the human trunk to analyze the contribution of seven groups of muscles involved in breathing, he was able to obtain the forces generated by the inspiratory and expiratory muscles by using electromyography (EMG) electrodes placed on the skin. According to this study, the forces produced by each external intercostal muscle were equal, reaching a maximum value of 242 N for the 24 external intercostal muscles analyzed in the high effort of breathing. According to the author, the short ribs have 2 external intercostal muscles and the longer ribs have 3 muscles [17]. The last 5 pairs of ribs are considered thus they are the longest, each intercostal space contains 3 external intercostal muscles, exerting a force of 30.25 N on each rib.

A Compliant Mechanism as a Sternum Prosthesis

147

For the boundary conditions, the mechanism was fixed at all holes and the force of 30.25 N was placed at the end of each arm, where it is fixed with the free segment of the rib (see Fig. 4).

Fig. 4. Meshing and boundary conditions for FEM analysis

3 Results Three relevant results were obtained from the numerical analysis; the maximum stress concentration, the displacements and the maximum life of the mechanism. The values of the Von-Misses stress in the prosthesis were computed in order to determine if proposed dimensions can withstand the payloads of human breathing (see Fig. 5). FEM analysis showed the areas with the greatest stress concentration in the mechanism, with a maximum value of 558 MPa identified in red, while the zones with the lowest stress concentration are identified in dark blue with stress values of 181 Pa. There are also areas in green, which reach a stress concentration of 827 MPa. It was possible to obtain the numerical simulation of the whole movement mechanism to perform the inhalation process (see Fig. 6). It can also be observed that the maximum displacement in the upper arms of the mechanism was reached a value of 14,9 mm, while in the lower arms displacements between 7,45 to 9,93 mm were generated. A fatigue analysis was performed to determine the cycles that compliant mechanism resists. In this case it was analyzed under a type of load that reaches its maximum value and decreases to zero. Results shown that almost the entire mechanism can resist high cycling, reaching 1 × 107 cycles, but there are areas where the useful life decreases, as in the blue zones (see Fig. 7), where the number of cycles decreases to 9,05 × 105. Figure 8 shows the stress values that the mechanism can reach for a certain number of cycles. It can be observed that by having lower stress values work cycles are greater, when the stress values increase the work cycles decrease.

148

O. Ramirez et al.

Fig. 5. Plot results for Von-Misses stress on mechanism

Fig. 6. Displacements on y axis of the mechanism

4 Discussion In this work, the design and numerical analysis of a compliant mechanism for its subsequent application as a sternum prosthesis was performed. The design of the mechanism is the author’s own creation. This seeks to improve the ventilatory dynamics in humans who have had their sternum partially or totally removed.

A Compliant Mechanism as a Sternum Prosthesis

149

Fig. 7. Fatigue test results for the mechanism

Fig. 8. Plot results of alternating stress (S) and cycles (N) for titanium alloy compliant mechanism

Discretization of the three-dimensional model was performed by using tetrahedral elements of 1 mm. By performing the discretization, they were obtained, 75,806 nodes and 28,430 elements were developed, 92% of the elements reached a skewness of 1. By performing the numerical analysis of the mechanism, the results of the stress distribution, the useful life and the displacements generated in the arms of the mechanism were obtained. The displacements obtained from the numerical analysis for the fifth arm of the mechanism (located at the bottom), were similar to the results reported by [18] where ribs 8 and 9 performs a vertical displacement of 0,81 cm by action of the external intercostal muscles.

150

O. Ramirez et al.

5 Conclusion This paper presents a numerical analysis that was performed on a flexible mechanism to evaluate the effects of a payload and to determine if it can be used for further development of a medical device such as a sternum prosthesis. The obtained results showed that both the shape and the material are able to resist to the applied load, since the maximum stress concentration reached the value of 558 MPa, while the material yield stress is 827 MPa. It was observed in the simulation that a coordinated proper movement was generated in the arms of the mechanism. It was observed that the mechanism can perform a long useful life, even so, a morphological optimization can be carried out to produce that the useful life is greater in the parts where the greatest damage occurs. This work is the beginning for developing a new sternum prosthesis that uses a compliant mechanism that generates a coordinated movement among the ribs. As a future work, it would be necessary to consider the first two ribs together with the action of the muscles that exert the lifting forces on these. It would also be necessary to consider a clinical case, since the development of prostheses is carried out according to the specific characteristics of a patient. Acknowledgments. The authors wish to gratefully acknowledge Consejo Nacional de Ciencia y Tecnologia (CONACyT) and Instituto Politécnico Nacional through the support projects 20201964 and 20200930, as well as an EDI grant, all by SIP/IPN.

References 1. Herreros, J., Glock, I., Echavé, V., Teijeira, J.: Surgical treatment of sternum tumors (Tratamiento quirúrgico de los tumores de esternón). J. Med. Univ. Navarra (Revista de Medicina de la Universidad de Navarra) 26, 241–243 (1982) 2. Torres-San-Miguel, C.R., Hernández-Gómez, J., Urriolagoitia-Sosa, G., Romero-Ángeles, B., Martínez-Sáez, L.: Design and manufacture of a customised temporomandibular prosthesis. Int. J. Numer. Methods Calculation Des. Eng. (Revista internacional de métodos numéricos para cálculo y diseño en ingeniería) 35, 1–10 (2019) 3. Torres-San-Miguel, C.R.: Metal mandibular prosthesis for subtotal hemimandibulectomy, capable of resembling a natural bite, allowing fixation of the denture implant, (Prótesis mandibular metálico para hemimandibulectomía subtotal, capaz de asemejar la mordida natural, permitiendo la fijación de dentaduran). MX/a/2015/006792, Mexico City, 28 May 2015 4. Aranda, J.L., Jiménez, M.F., Rodriguez, M., Varela, G.: Tridimensional titanium printed custom-made prosthesis for sternocostal reconstruction. Eur. J. Cardiothorac. Surg. 48, 92–94 (2015) 5. Kamel, M.K., Cheng, A., Vaughan, B., Stiles, B., Altorki, N., Spector, J., Port, J.: Sternal reconstruction using customized 3D-printed titanium implants. Ann. Thorac. Surg. 109, 1–7 (2019) 6. Aranda, J.L., Novoa, N., Jiménez, M.F.: Thoracic customized modular titanium printed prosthesis. AME Case Rep. 3, 1–4 (2019) 7. Cano, J.R., Hernández, E.F., Pérez, A.D., Lopez, R.L.: Reconstruction of the anterior chest wall with a 3D printed biodynamic prosthesics. J. Thorac. Cardiovasc. Surg. 155, 1–9 (2017)

A Compliant Mechanism as a Sternum Prosthesis

151

8. Ramirez, O., Ceccarelli, M., Russo, M., Torres-San-Miguel, C.R., Urriolagoitia-Calderon, G.: Experimental dynamic tests of rib implants. In: The Second International Conference of IFToMM ITALY 2018, vol. 68, pp. 353–361. Springer, Cham (2019) 9. Ramirez, O., Ceccarelli, M., Torres-San-Miguel, C.R., Urriolagoitia-Calderon, G.: Experimental characterization of an osteosynthesis implant. In: Advances in Mechanism and Machine Science, IFToMM WC 2019. Mechanisms and Machine Science, vol. 73, pp. 53–62. Springer, Cham (2019) 10. Howell, L.L.: Compliant mechanisms. In: McCarthy, J. (ed.) 21st Century Kinematics, pp. 189–205. Springer, London (2013) 11. De Troyer, A., Kirkwood, P., Wilson, A.: Respiratory action of the intercostal muscles. Physiol. Rev. 85, 717–756 (2005) 12. Torres-San-Miguel, C.R., García, G., Aguilar, L.A., Martínez, L.: Design of a rib impactor equipment. J. Phys.: Conf. Ser. 792, 1–6 (2017) 13. Ratnovsky, A., Elad, D., Halpern, P.: Mechanics of respiratory muscles. Respir. Physiol. Neurobiol. 163, 82–89 (2008) 14. Selthofer, R., Nikolic, V., Mrcela, T., Radic, R., Leksan, I., Rudez, I., Selthofer, K.: Morphometric analysis of the sternum. Coll. Antropol. 30, 43–47 (2006) 15. Dzian, A., Živˇcák, J., Penciak, R., Hudák, R.: Implantation of a 3D-printed titanium sternum in a patient with a sternal tumor. World J. Surg. Oncol. 16, 1–4 (2018) 16. Niinomi, M.: Mechanical properties of biomedical titanium alloys. Mater. Sci. Eng. 243, 231–236 (1998) 17. Ratnovsky, A., Elad, D.: Anatomical model of the human trunk for analysis of respiratory muscles mechanics. Respir. Physiol. Neurobiol. 148, 245–262 (2005) 18. Loring, S.H., Woodbridge, J.A.: Intercostal muscle action inferred from finite-element analysis. J. Appl. Physiol. 70, 2712–2718 (1991)

Design and Lab Experiences for a Fixator of Rib Fractures Ludovica Sommariva1 , Josè Luis Arreguin2,3 , Cuauhtémoc Morales Cruz2,3 , Marco Ceccarelli2(B) , Vincenzo Ambrogi1 , and Lucrezia Puglisi1 1 Department of Surgical Sciences, University of Rome Tor Vergata, 00133 Rome, Italy 2 LARM2, Department of Industrial Engineering, University of Rome Tor Vergata, 00133

Rome, Italy [email protected] 3 Instituto Politécnico Nacional, Mexico City, Mexico

Abstract. A non-invasive bioabsorbable rib fixator is presented with its mechanical design and concepts for surgery implementation. Its feasibility and efficiency are investigated though lab testing whose results are discussed with satisfactory outcomes. The rib fixator is designed with small size for fairly easy manufacturing and surgery implantation with short-time tissue absorption. The aim of the paper as well its content refers to a feasibility check and operation characterization of a rib implant for osteosynthesis of rib fractures. Keywords: Biomechanics · Rib fixators · Design · Experimental testing

1 Introduction Rib fractures treatment is not yet uniquely defined in thoracic surgery. This is due to the particular anatomical and physiological characteristics of the ribs that require continuous movement to allow breathing in patients. The anatomical features of the rib imply a difficult treatment of a fracture. Indeed, the ribs are thin, flat and brittle bones not easy to be pierced by usual system of osteosynthesis. They have also a peculiar function, which entails continuous movement to allow breathing. Today the main osteosynthesis system uses titanium plates and bicortical screws, that are not indicated for rib fixation because they can cause further rib fracture and persisting instability, as reported for example in [1–3]. Other techniques can be used, such as the cerclage wiring or Judet’s Agraphes, [4], and however these implants anchoring themselves to the lower edge of the rib, can damage the intercostal nerve provoking chronic pain syndrome and causing neural tumor, [5]. Several osteosynthesis systems have also been developed, including U-shaped plates, that can be fixed only on the upper edge of the rib, but they still require a use of screws as in the cases reported in [1–3, 6, 7]. All the techniques are related with several complication and involve the use of permanent devices that often need to be removed through a second surgery, thus increasing morbidity and mortality, as reported for example in [8]. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 152–160, 2021. https://doi.org/10.1007/978-3-030-58104-6_18

Design and Lab Experiences for a Fixator of Rib Fractures

153

In general, fracture consolidation requires one-month time approximately, thereafter a fixation device is no longer useful. For this reason, an ideal osteosynthesis system should be made of absorbable materials and be fixed without the use of screws or wires. In this work a Mg-Zn-Mg alloy is used for an absorbable rib fixator together with a biological glue as suggested in [9, 10]. In addition, testing activity on rib fixators is not reported in the literature with activity to characterize the mechanical behavior of the implants and their effects on the biomechanics of the rib functioning during the osteosynthesis, [12]. In this work, laboratory experiences are presented to support both the proposed design of a new rib fixators and to characterize the biomechanics of fractured ribs under repair with rib fixators.

2 Medical Surgery Requirements According to some minimalistic theories rib fractures can be managed by sedating the pain originated by the fracture itself as for example in [5]. This is true only for simple or limited multiple fractures. On the other hand, multiple pluri-regional and displaced rib fractures as in Fig. 1 associated with internal injury must be treated aggressively by open surgery and those rib fractures represent a frequent occurrence in thoracic surgery. Between these two extremes there are many complicated conditions that can require many different types of therapy, being each of them associated with variable complications.

Fig. 1. Example of flail chest, multiple adjacent ribs broken in multiple places, [13]

The main goal of a treatment is to create an osteosynthesis system that ensures fixation of the broken stumps, avoiding their painful displacement and possible lesions

154

L. Sommariva et al.

of the intercostal nerve and surrounding tissues. At the same time a fixation ensures the movement of the ribs, avoiding complications related to their forced immobilization for the thorax during breathing. The most common sources causing rib fractures can be considered blunt trauma due to car accident, falls from height or domestic accidents. Thoracic traumas with rib fractures are also associated with pulmonary contusions, or other severe injury such as pneumothorax or hemothorax, [11]. The occurrence of pulmonary contusions is associated with higher mortality in several studies, especially because it frequently evolved to gas exchange impairment, delayed acute respiratory distress syndrome (ARDS) and/or multi-organ failure, [8]. Thoracic Trauma Severity (TTS) score as in Table 1 gives prediction about the risk of ARDS in trauma patients. The score takes into account age, respiratory status, the gravity of thoracic injuries (pleural effusions, pulmonary contusions, rib fractures). For this reason, it is of vital importance to correctly diagnose and treat any type of rib fractures allowing reduction of complications and mortality, even with surgery implants. Table 1. Thoracic trauma severity (TTS) score, [14]

Single rib fracture with no displacement of stumps can be conservatively treated with drugs to control the pain allowing normal respiratory physiology and reducing long-term complications such as pneumonia. Multiple bilateral, poli-district and displaced fractures associated with tissue injuries need to be surgically treated through different techniques, each of them presenting different complications, as for example in [15]. When fractures are so extended to create an area of the wall chest completely free from the remaining ribcage, that is called flail chest, Fig. 1, there is high risk of death. In this case mechanical ventilation under general anesthesia is required for long periods, exposing the patient to high risk of infection and other typical complications of intensive care unit.

Design and Lab Experiences for a Fixator of Rib Fractures

155

3 A New Design for Rib Fixators The aim of the here-in reported work is to propose a resorbable rib fixator to help osteosynthesis process, with features to overcome all the complications of the techniques currently used with screw-fixed fixators. The goal is to obtain not only an absorbable prosthesis fixator, but also by using an anchor system without screws and wiring, so that the surgery can be less traumatic and less painful for the patient but preserving the safety of the treatment itself. Among the new materials, biodegradable metals have been shown to be significantly resistant and are completely reabsorbed in vivo, avoiding the need for a second intervention. The new rib fixator (n.1) in Fig. 2 shows a flat rectangular-shaped plate with two surfaces of adhesion (n.2) to the rib and a central part (n.3) with reduced section to give flexibility in the bending necessary to the rib movement during patient breathing, [9]. The design is characterized by reduced dimensions in terms of width, length and thickness, with values (as for example 10 × 30 × 2 mm) that allow to fix the plate on the two shoreline rib segments to align the fracture stumps adequately to prevent any excessive motion or relative rotation between them. Absorbable materials like magnesium alloy have been used to create the plate - but other bioabsorbable non-metallic materials can be considered as well - to be absorbed from tissues in order not to require subsequent removal surgeries. The plate is fixed on the periosteum (n.5) of a rib (n.6) through the two fixing surfaces with biological glue (n.4), that can be made of cyanoacrylate, so that it does not remain in situ with biological invasive effects and with the aim of ensuring adequate adhesion of each part of the fixing plate to a part of the respective coastal segment.

a)

b)

Fig. 2. A novel design for rib fixators: a) concept (1: rib fixator with 2 plates of adhesion and 3 flexural joint; 4: biological glue; 5: periosteum; 6 rib segments); b) a lab demonstrator prototype

The motion of the implant, made of magnesium alloy, will follow the rib motion due to breathing with a relative bending of the two adhesive plates (2) around the flexural joint (3) in the structure of the rib fixator (1) as shown in Fig. 2. The motion range and related stiffness are sized as near as possible to the characteristic of the rib under repair, i.e. the relative bending rotation of the adhesive plates (2) are limited to less than 10°. The manufacturing is indicated as a low-cost solution in having the fixator structure as a small rectangular plate with a small milled part in the center to produce the flexural joint (3) that in the prototypes used in the tests is of 3 mm width. The mechanical design is

156

L. Sommariva et al.

conceived and presented as a fairly simple as possible to let also the surgeons understand the mechanical solution and its operation feasibility.

4 Lab Testing and Results For the tests a machine TEMARI, Fig. 3, [12, 16], was used for simulating breathing and coughing with compression and relaxation cycles on rib samples during expiration and inspiration, Fig. 4. The TEMARI testing machine is powered with a 12 V DC motor that is used to move a cam mechanism to generate an impact shift of 40 mm.

a)

b) Fig. 3. Experimental layout of TEMARI testing machine: a) scheme; b) a lab setting.

It is provided of, Fig. 3a) and 3b): a linear potentiometer (n. 6) to measure the displacement with a maximum value of 6 cm; a IMU accelerometer (n.1) positioned on the head of the follower of the cam mechanism; two load cells (n.5), to measure one the reaction force that is impressed on the coast at the point of impact, the other the force at the opposite end; two strain gauges (n. 2 and 3), placed on rib fixator; and a

Design and Lab Experiences for a Fixator of Rib Fractures

157

IMU accelerometer (n.4) positioned on grasped rib set at the fixing point. A fixature clamp (7), provided also of screws for fixing, fixs a rib sample on a fixed extremity. Two rib fixators (n. 2 and 3), were used in the testing as positioned at the fracture line, respectively a metal one on the external surface of the rib, and a plastic one on the internal surface, Fig. 3b). TEMARI testing machine simulates human breathing in rib specimen of any size with characteristics that can be adjusted by regulating the speed of cam rotation to have a desired breathing frequency from 10 to 100 cycles per minute and rib displacement using part of the cam-follower stroke of 8 cm, with monitoring using the set of sensor as specified in Fig. 3. Specimens of rib fixator were used in lab testing as built with AZ31 magnesium alloy that is successfully used in surgical implants for other applications with proper biocompatibility as reported for example in reference [10]. An extrapolation to the cases in human ribs can be understood considering the similarity of pig ribs to the human ribs, but nevertheless it is to be further investigated in a future planned activity. Preliminary tests were carried out to characterize the human breathing and coughing by using the same IMU sensor in TEMARI to acquire acceleration in rib motions. Figure 5 shows the results indicating maximum values for normal breathing of less than 0.2 g, for deep breathing of about 0.05 g, and for coughing of about 0.25 g, (g is 9.8 m/sec2 ). Experiments were carried out on samples consisting of a pig single rib in the two conditions of fractured and repaired, [12]. Additional tests were performed on rib set samples consisting of pig three-rib set with the central one fractured and repaired similarly to the case with one rib sample. Figure 4 shows an experimental setup during testing with a rib set sample of pig three ribs. Tests with pig ribs were carried out to evaluate the acceleration of the ribs during normal and deep breathing and coughing with the aim to have a comparison with the results of human data in Fig. 5. In both tests values are acquired from sensors in TEMARI testing machine as reported in the result examples

Fig. 4. An experimental test setup with pig three-rib set in TEMARI

158

L. Sommariva et al.

Fig. 5. Rib acceleration during 3 normal breaths, 3 coughs and 3 deep breaths.

in Figs. 6 and 7 in terms of characteristics of the impact and response of the rib an a summary is listed in Table 2.

a)

b)

c) Fig. 6. Results of test no.1 in Table 2 in terms of: a) Acceleration; b) Strain; c) Force.

Design and Lab Experiences for a Fixator of Rib Fractures

159

a)

b)

c) Fig. 7. Results of test no.2 in Table 2 in terms of: a) Acceleration; b) Strain; c) Force.

Table 2. Data of the experimental tests Test Sample

Max Acceleration (g) Max impact force (N)

1

One Rib

0.32

7.50

2

Three Ribs 0.55

9.65

In particular, the acquired test results in Figs. 6 and 7 show the breathing human frequency with regular peaks whose values represent how the impact is well transmitted and distributed within a rib structure and a rib set, respectively, with a proper functioning of the rib fixator. It can be noted that the rib set responds in better smoothing and larger resistance to the impact, as expected, indicating that the proposed osteosynthesis system is able to resist the physiological stresses that occur inside the wall chest.

160

L. Sommariva et al.

5 Conclusions Feasibility and efficiency of a new rib fixator is investigated with lab testing of pig rib samples in a testbed machine simulating breathing and coughing motions. Results are obtained in terms of force and motion characteristics on pig fractured ribs reproducing human-like situations so that the proposed rib fixator can be considered suitable for surgery implantation with the proposed minimally invasive absorbable design solution.

References 1. Nirula, R., et al.: Rib fracture repair: indications, technical issues, and future directions. World J. Surg. 33, 14–22 (2009) 2. Paris, F., et al.: Surgical stabilization of traumatic flail chest. J. Thorax 30, 521–526 (1975) 3. Bemelman, M., et al.: Historic overview of treatment techniques for rib fractures and flail chest. Trauma Emergency Surgery 36, 407–415 (2010) 4. Menard, A., et al.: Treatment of flail chest with Judet’s struts. J. Thorac. Cardiovasc. Surg. 86(2), 300–305 (1983) 5. Wu, W.M., et al.: Which is better to multiple rib fractures, surgical treatment or conservative treatment? Int. J. Clin. Exp. Med. 8(5), 7930–7936 (2015) 6. Sales, J., et al.: Biomechanical testing of a novel, minimally invasive rib fracture plating system. J. Trauma 65, 1270–1271 (2008) 7. Bottlang, M., Long, W.B., Phelan, D.: Surgical stabilization of flail chest injuries with MatrixRIB implants: a prospective observational study. J. Care Injured 44, 232–238 (2012) 8. Lin, F.C., et al.: Morbidity, mortality, associated injuries, and management of traumatic rib fractures. J. Chin. Med. Assoc. 79(6), 329–334 (2016) 9. Ambrogi, V., Ceccarelli, M.: Fixing plate for osteosynthesis of fractured ribs), patent request n. 102019000005638, Italy. Accessed 12 Apr 2019 10. Arreguin, J.L., et al.: Design solutions from material selection for rib fixators. J. Mater. Process. Trans Tech Publications, Vienna 2020 (submitted) 11. Flagel, B.T., Luchette, F.A., Reed, R.L., et al.: Half-a-dozen ribs: the breakpoint for mortality. Surg 138, 717–725 (2005) 12. Ramirez, O., Torres-San-Miguel, C.R., Ceccarelli, M., Urriolagoitia-Calderon, G.: Experimental characterization of an osteosynthesis implant. Mechanisms and Machine Science. MMS, pp. 53–62. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20131-9_6 13. Loizzi, M., Oliaro, A.: Respiratory diseases. Pneumatology and thoracic surgery. Minerva Medica, 2015. (in Italian) 14. Daurat, A., et al.: Thoracic Trauma Severity score on admission allows to determine the risk of delayed ARDS in trauma patients with pulmonary contusion injury. Int. J. Care Injured 47, 147–153 (2016) 15. Mineo, T.C., Ambrogi, V., Cristino, B., Pompeo, E., Pistolese, C.: Changing indications for thoracotomy in blunt chest trauma after the introduction of video-thoracoscopy. J. Trauma 47, 1088–1091 (1999) 16. Ramirez, O., Ceccarelli, M., Russo, M., Torres-San-Miguel, C.R., Urriolagoitia-Calderon, G.: Experimental dynamic tests of rib implants. In: Carbone, G., Gasparetto, A. (eds.) Advances in Italian Mechanism Science- Proceedings of the Second International Conference of IFToMM Italy, Springer, Cham, pp. 353–361 (2019). https://doi.org/10.1007/978-3-030-03320-0_38

Learned Task Space Control to Reduce the Effort in Controlling Redundant Surgical Robots Murali Karnam1(B) , Manuela Eugster1 , Riccardo Parini1,3 , Philippe C. Cattin2 , Elena De Momi3 , Georg Rauter1 , and Nicolas Gerig1 1

BIROMED-Lab, Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland [email protected] 2 CIAN, Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland 3 NearLab Medical Robotics, Department of Electronics, Information and Bioengineering, Politecnico di Milano, 20133 Milan, Italy https://biromed.dbe.unibas.ch

Abstract. Redundant robots allow multiple robot joint configurations for the same end-effector pose by moving only in null space. Robot’s motions in null space are not intuitive to predict in general and in particular for medical personnel. In this work, we present a control concept that allows the operator to focus on the correct end-effector pose during time-critical tasks, e.g. change of the endoscope pose during a surgical intervention, while the shape of the redundant robotic structure is handled autonomously based on previously learnt preferred shapes close to the actual end-effector pose. We investigated the benefit of the proposed learned task space control over naive task space control that required an operator to manually control a virtual robot in task space and null space independently. In a first user study, we found that learned task space control significantly reduced the effort – as measured by task duration and task load – for operators compared to naive task space control. Keywords: Redundancy

1

· Null space · Control · Medical robots

Introduction

Redundant robots have more degrees of freedom (DoF) than required for a primary task (end-effector pose control). Null space of redundant robots enables changing the robot shape (joint configuration) as a secondary task with lower priority, and not affect the primary task [1]. Secondary tasks could for example be avoiding obstacles or joint limits. Typical methods to control robots for c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 161–168, 2021. https://doi.org/10.1007/978-3-030-58104-6_19

162

M. Karnam et al.

secondary tasks may rely on mathematically defined objective functions [2,3], making null space motion challenging to predict for medical personnel. Alternative methods take user input to control the null space [4,5] and are considered more intuitive since the user has complete control over the robot motion. Nevertheless, methods based on user input may increase the overall effort for the operator as both task space and null space must be controlled simultaneously. We aim to increase the intuitiveness in controlling redundant robots and surgical robots in particular by decreasing the effort for medical personnel in handling non-intuitive motion of the redundant DoF of the robot. We achieve this by recording the desired robot joint configurations during preoperative planning. While the operator controls the robot intraoperatively to perform the surgery (primary task), the robot automatically moves in null space towards the closest recorded joint configuration (Fig. 1). To reach our goal, we present a learned task space control method, which controls both task space and null space of a redundant robot. We hypothesize that the operator’s effort to control a robot is reduced with the proposed control method as compared to a naive task space control method where the operator actively controls task space and null space.

Fig. 1. Workflow overview: during preoperative planning, the operator uses naive task space control to move the robot to reach a desired end-effector pose (pink cone). At the desired end-effector pose, the robot’s joint configuration can be reshaped in null space. The operator can record the current end-effector pose and related joint configuration. Intraoperatively, the operator can move the robotic arm using learned task space control to any desired end-effector pose. As the operator controls the robot in task space, the robot’s joints automatically move towards the closest recorded joint configuration.

Learned Task Space Control

2 2.1

163

Materials and Methods Apparatus

A virtual 7-DoF redundant robot (KUKA LBR iiwa, KUKA AG, Augsburg, Germany) with a rigid endoscope as the end-effector was visualized on a screen. This end-effector was controlled with a table-fixed force/torque (F/T) sensor with a handle on top (Fig. 2).

Fig. 2. Schematic of the setup. A handle with two buttons allows switching control modes and enables moving to robot poses by applying force on a F/T sensor (Mini45, ATI Industrial Automation). Real-time calculations are performed on a CPU (CX2020, Beckhoff Automation GmbH & Co. KG, Verl, Germany) that reads the forces (fˆ ) that the user applies to the handle, and calculates the corresponding desired joint configuration q cmd . A GUI shows the robot configuration to the user. In the GUI, obstacle regions are indicated as red spheres and the target pose for the robot’s endeffector is shown as a cone. Once the end-effector has reached the target pose, the color of the cone is changed from pink to green. The status of the handle buttons, the number of existing recorded poses, and the robot status when it reaches a work-, joint-, or null space limit are shown with text to the user.

2.2

Control Modes

The two buttons on the handle (Fig. 2) allow the user recording poses and switching between different control modes – null space control, naive task space control, and learned task space control (Fig. 3). Measured user forces (fˆ ) were filtered using a moving median filter with a 100 ms time window. Record Pose. The interface allows a user to record the current estimated endeffector pose and joint configuration as (xreci , q reci ). The pairs of all recorded poses (X rec ) and configurations (Qrec ), are used in learned task space control. If an end-effector pose is recorded that is too close to an already recorded one, the previous pose is overwritten with the new pose and joint configuration. Two poses (xa and xb ) were considered too close if the dissimilarity metrics in position

164

M. Karnam et al.

Fig. 3. User control modes overview: the red button allows the user to switch between task space and null space control. The green button allows the user to record the robot pose and joint configuration used for learned task space control.

(Euclidean distance [6]) and orientation (deviation of the rotation matrix from the identity matrix [7]) between them is below a predefined threshold: ||Δxpos ||2 < 0.15 m, and ||I − a Rb ||F < 0.73,

(1)

where xpos and a Rb are the position vector and the rotational transformation matrix between the two end-effector poses (xa and xb ), I is a 3 × 3 identity matrix, and || · ||F denotes the Frobenius norm of the matrix.

Null Space Control. Based on the force applied by the user, q cmd is calculated such that the end-effector pose is unchanged, while the joint configuration changes only in the null space. The details have been presented previously [8].

Naive Task Space Control. The commanded robot joint configuration q cmd ˆ , and the desired endis calculated based on the estimated joint configuration q effector velocities in task space x˙ cmd (Fig. 4, left): ˆ + Ts · J+ · x˙ cmd , q cmd = q

 −1 J+ = JT JJT − λ2 I

(2)

where J+ is the damped Moore-Penrose pseudo-inverse of the Jacobian, with the damping parameter λ = 0.001, and Ts = 0.001 s is the cycle time of the realtime system. The desired end-effector pose velocity, x˙ cmd , consists of velocities in position (v cmd ) and orientation (ω cmd ). They are calculated from the measured user forces as, v cmd = cv · fˆ for position control and v cmd = cω · fˆ for orientation m and cω = 1.0 rad control, where cv = 1.4 Ns Ns were used. The commanded velocity was saturated to 0.035 m/s in position and 0.026 rad/s in orientation.

Learned Task Space Control

165

Fig. 4. Task space control modes: naive task space control (left) and learned task space control (right). At each time step, the control modes provide a commanded joint configˆ and commanded end-effector uration q cmd based on the estimated joint configuration q velocities x˙ cmd . The learned task space control further considers recorded end-effector poses Xrec and the corresponding robot configurations Qrec .

Learned Task Space Control. For each time step Ts the learned task space control (Fig. 4, right) requires as inputs the commanded end-effector velocity x˙ cmd , the recorded end-effector poses Xrec , the related robot’s joint configuration ˆ . The output is the resulting Qrec , and the estimated robot’s joint configuration q commanded joint configuration qcmd : ˆ + Ts · (J+ · x˙ cmd + q˙ LRN ), qcmd = q

(3)

which is a summation of the naive task space control joint command and a learned joint velocity (q˙ LRN ). The learned joint velocity is calculated as the gradient of an objective function (f (x, q)) projected into null space:   q˙ LRN = N · ∇q f (x, q)x=ˆx , (4) q =ˆ q

ˆ is the estimated end-effector pose and N is the null space projection where x matrix. The objective function is formulated such that it has a minimum at the closest recorded pose and joint configuration: 2  w1 · (q ∗rec − q)  q range , f (x, q) = q˙max · (5) Φpos (x∗rec , x) + w2 where, x∗rec is the closest among all recorded end-effector poses as per the dissimilarity metric Eq. (1) and q ∗rec is its corresponding joint configuration. The weight, w1 = 0.07 m, is set manually to tune the effect of learned task space control, and a damping term, w2 = 0.06 m, is used to ensure the function is bounded. A maximum robot joint velocity q˙max = 30◦ /s is used to limit the learned joint velocity and not exceed the maximum allowed joint speeds. The difference between the estimated and closest recorded joint configuration is normalized by the joint ranges of the robot q range , where the Hadamard division symbol , denotes element-wise division of the two vectors. 2.3

Participants

Six healthy volunteers (all male, age 25 to 45 years, mean age 29.5 years) participated in the pilot study. The participants did not have prior experience with

166

M. Karnam et al.

the user-interface to control a virtual robot, and they were all researchers at the Department of Biomedical Engineering, University of Basel. 2.4

Experimental Protocol

The objective for the participant was to move the robot from a predefined initial configuration to a goal pose (visualized as a cone) and configuration (no collision with obstacles). The study consisted of three tasks: Task P (planning). Naive task space control followed by null space control was used to complete the objective, and the final robot configuration was recorded. Task N (naive). The objective was completed using naive task space control. The controller was manually set to ignore the recorded poses. Task L (learned). The objective was completed using learned task space control, and the controller used the recorded poses from planning. The trial began by allowing each participant to use the interface and move the robot freely for five minutes to get acquainted with the system. The participants were instructed to complete the tasks as fast as possible. At the end of each task, the participants were asked to fill a NASA Task Load Index (NASATLX) questionnaire. With two task space control modes (N and L), the study had a crossover design with one independent variable, leading to two conditions per participant (N-L or L-N). The participants were randomly divided into two groups. Group 1 performed the tasks in the order P-N-L, and Group 2 in the order P-L-N. Each participant performed the three tasks once. 2.5

Evaluation Metrics

The effort for the user for each task was measured with two metrics; (1) The task duration (s), i.e., the time taken by the participant to complete the task; (2) The NASA-TLX scores between 0 and 100. A one-sided Wilcoxon signed-rank test with a confidence level of 5% was performed independently on task duration and task load. The null hypothesis was that the effort to perform the tasks with naive task space control was greater than or equal to the effort for learned task space control.

3

Results

Task duration (p < .03, W = 20, r = .95) and task load (p < .02, W = 21, r = 1.0) were significantly reduced (Wcritical = 17) using learned task space control compared to naive task space control (Fig. 5).

Learned Task Space Control

167

Fig. 5. Task duration (top-left) and task load (top-right): participants are represented by different markers that are connected by a dashed line across the three tasks. The two lower plots show the difference in the task duration (bottom-left) and task load (bottom-right) for each participant between using naive and learned task space control. The * indicates a significant reduction in the one-sided Wilcoxon signed-rank test.

4

Discussion

The results showed a statistically significant reduction in the effort of participants using learned task space control compared to naive task space control as evaluated quantitatively by a one sided Wilcoxon signed-rank test. Although the study included only six participants, the pilot study shows that using learned task space control for robot control holds potential to reduce medical personnel’s effort during surgery and its utility needs to be investigated with a physical robot in a surgical setting and medical personnel as participants. We observed that the reduction in the task duration from using naive task space control to learned task space control was smaller than the time taken to plan and record poses. However, we expect that the robot is re-positioned multiple times in a real intervention, thereby not scaling the time taken to plan the surgery proportionally. Furthermore, the time duration was only used as a measure of effort. Reducing the overall surgery duration was not a goal of this study. A limitation of the study was the use of a virtual robot visualized on a screen, which might have affected the performance of the users to control the robot in task space. Whether the effort reduces with a physical robot needs to be investigated. We believe that an interface directly mounted on the robot would reduce the effort to control the robot in all modes and make all modes more intuitive to the operator.

168

5

M. Karnam et al.

Conclusion

In this work, we proposed learned task space control of a redundant robot. We implemented the controller and assessed the reduction in the operator’s effort to control a virtual robot. In the pilot user study with six participants, using learned task space control significantly reduced the effort compared to naive task space control. Acknowledgments. We gratefully acknowledge funding by the Werner Siemens Foundation through the MIRACLE project, and we thank Prof. Dr. med. Niklaus F. Friederich for his continuous support with respect to medical questions. We thank the participants for their time and volunteering to take part in the study.

References 1. Siciliano, B.: Kinematic control of redundant robot manipulators: a tutorial. J. Intell. Robot. Syst. 3(3), 201–212 (1990) 2. Moradi, H., Lee, S.: Joint limit analysis and elbow movement minimization for redundant manipulators using closed form method. In: International Conference on Intelligent Computing, pp. 423–432. Springer, Berlin (2005) 3. Flacco, F., De Luca, A., Khatib, O.: Motion control of redundant robots under joint constraints: saturation in the null space. In: 2012 IEEE International Conference on Robotics and Automation, pp. 285–292. IEEE (2012) 4. Sandoval, J., Su, H., Vieyres, P., Poisson, G., Ferrigno, G., De Momi, E.: Collaborative framework for robot-assisted minimally invasive surgery using a 7-DoF anthropomorphic robot. Robot. Auton. Syst. 106, 95–106 (2018) 5. Vigoriti, F., Ruggiero, F., Lippiello, V., Villani, L.: Tracking control of redundant manipulators with singularity-free orientation representation and null-space compliant behaviour. In: Ficuciello, F., Ruggiero, F., Finzi, A. (eds.) Human Friendly Robotics, pp. 15–28. Springer, Cham (2019) 6. Celebi, M.E., Celiker, F., Kingravi, H.A.: On Euclidean norm approximations. Pattern Recogn. 44(2), 278–283 (2011) 7. Huynh, D.Q.: Metrics for 3D rotations: comparison and analysis. J. Math. Imaging Vis. 35(2), 155–164 (2009) 8. Karnam, M., Parini, R., Eugster, M., Cattin, P., Rauter, G., Gerig, N.: An intuitive interface for null space visualization and control of redundant surgical robots. In: Proceedings on Automation in Medical Engineering, vol. 1. Infinite Science Publishing (2020). https://doi.org/10.18416/AUTOMED.2020

Design, Static and Performance Analysis of a Parallel Robot for Head Stabilisation in Vitreoretinal Surgery Hans Natalius1(B) , Patrice Lambert2 , Manish K. Tiwari1 , Lyndon da Cruz1,3 , and Christos Bergeles2 1

3

University College London, Gower St, London WC1E 6BT, UK [email protected] 2 King’s College London, Strand, London WC2R 2LS, UK Moorfields Eye Hospital, 162 City Rd, London EC1V 2PD, UK

Abstract. This paper explores the requirements-based design and static analysis of a 6 Degree-of-Freedom parallel robotic headrest, of novel architecture, to counter head motion in vitreoretinal surgery. Upcoming therapy delivery interventions require micro-precision but should ideally take place under local anaesthesia. Therefore, breathing, spasmodic motions, and even snoring that often occurs need to be accounted for and if possible counteracted. Passive approaches that aim to constrain the patient’s head have not yet been fruitful, while invasive stereotactic fixation is naturally not an option. The proposed design respects ergonomic and surgical constraints to act as a headrest that will ultimately counteract patient motion. Static models are developed to understand the architecture’s characteristics, and performance metrics are devised for design evaluation. Finally, a prototype is presented.

Keywords: Parallel robot

1

· Surgical robot · Kinematics

Introduction

Stem cell implantation and gene vector delivery are envisioned as sight-restoring vitreoretinal surgical interventions [1,2]. To maximise the efficacy of therapeutics, injection precision of 10 µm is required. The surgeon’s physiological hand tremor and patient’s head movement, however, are proving a challenge in achieving the required positioning accuracy. Physiological hand tremor is on the order of 200 µm [3], while patient head motion can be as much as 11 mm [4]. While many solutions to mitigate hand tremor are being researched [5], methods to reduce the patient’s head movement have been less explored. Head movements that occur such as when a patient snores, breathes, or sneezes, raise the risk of complications. Therefore, a method to stabilize the patient’s head during ophthalmic surgery needs to be devised. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 169–179, 2021. https://doi.org/10.1007/978-3-030-58104-6_20

170

H. Natalius et al.

Examined approaches that aim to mitigate head movement have so far focused on trying to restrain the head. Examples include the head fixation device for iRAM!S robot [6] and the Granular-Jamming Headband [7]. Our article explores the possibility of a robotic headrest that can actively counter the patient’s head movements. As the headrest needs to perform motions whilst supporting the weight of the head, a parallel robot was chosen as the base system. Parallel manipulators combine high rigidity and low inertia, resulting in a faster dynamical response than serial manipulators. They also present higher accuracy due to their rigidity against unwanted movement. Although currently there are many available parallel robotic systems with 6 Degrees of Freedom (DoF) on the market, for example the Physik Instrumente H-825 6-Axis hexapod [8] and the HexGen HEX300230HL Hexapod [9], design constraints and operational requirements arising in ophthalmic surgery render the existing manipulators unsuitable as active headrests. Therefore, a robotic headrest with a new parallel kinematic architecture alongside performance evaluation metrics related to our application’s requirements were developed.

2

Methods

This section presents the manipulator design approach, its static model, and analysis of its workspace and performance against clinical requirements.

Fig. 1. Possible locations of the neck space in: (a) traditional Hexapod with a section between two actuators removed, and (b) the proposed manipulator.

Manipulator Requirements: For acceptance in a vitreoretinal surgical setting, the proposed manipulator needs to mitigate patient’s head movement whilst preserving the ergonomics for the surgeons and retaining patient comfort. To design a manipulator that can fulfill these functions, a set of requirements is defined.

Analysis of Robotic Headrest for Vitreoretinal Surgery

171

To stabilize a lying patient’s head, the manipulator needs to apply a countermotion with the same magnitude. A recent study showed that during surgery the patient’s head drifts for up to 11 mm along the horizontal plane (XY ). However, patient’s head drift along the global vertical direction (Z) and rotation along an arbitrary axis needs to be anticipated as well [4]. Since the head moves and rotates in 3D space, the proposed manipulator architecture needs to possess 6-DoF, to reach any position within its workspace under any orientation. The workspace needs to be at least as big as the magnitude of the head drift, while also adjustable to adapt to different patient’s head position. Table 1. Manipulator target specification Requirement

Value

Resolution

10 µm

Translational Workspace diameter and height 88 mm, 61 mm Rotational Workspace θx , θy , θz

±5◦ , ±5◦ , ±5◦

DoF

6

Load capacity

30 kg

Maximum height at Workspace Bottom Point 300 mm

As a safety measure in head stabilization, a cylindrical-shaped workspace that fits a 50 × 50 × 50 mm3 cube within it was chosen, with the addition of 11 mm on all directions to account for the magnitude of the head drift. To cover the required volume, the cylindrical-shaped workspace should have a minimum diameter of 88 mm, and vertical motion of 61 mm. Furthermore, while there is no identified value of a patient head’s rotation during surgery, we select a ±5◦ rotation relative to all three global axes indicated in Fig. 1, as a requirement to accommodate extreme motions. The resolution target for the headset was selected to be the same as the 10 µm resolution required for retinal therapy delivery. In terms of load carrying, the manipulator must support the weight of the head, the headrest pillow, and all attached components in a dynamic setting. The human head on average weights 4.3–5.3 kg [10], while the headrest pillow (including metal supports) is approximated at 4–6 kg. The weight of the manipulator components to be carried is estimated at a maximum 3.5 kg. Therefore, the rounded up weight that the manipulator needs to carry is considered 15 kg. Considering a safety factor of at least 2 to account for purposeful/forceful motions for head-posture adaptation, we arrive at a requirement of 30 kg as the manipulator’s load-bearing capacity under motion. Finally, the manipulator must preserve the ergonomics for the surgeon and retain patient comfort. This can be achieved by embedding the headrest within the manipulator. As the height of existing headrests are on the order of 180 mm (measured in the clinic), the maximum height of the proposed manipulator when the upper platform is at its lowest operational position, which is referred as the workspace bottom point, should be maximum 300 mm. The requirements for the proposed manipulator are summarised in Table 1.

172

H. Natalius et al.

Manipulator Design Architecture: As is common in 6-DoF parallel robots, the general design of the manipulator architecture comprises a lower platform that functions as a base, an upper platform that acts as the end-effector of the system, and 6 linear actuators connecting the two. The headrest pillow will be mounted within the perimeter of the upper platform, as shown in Fig. 1b, so that the height of the manipulator respects the requirements listed in Table 1. The advantage of this configuration is that the center of mass of the head will be close to the upper platform centre-point, which increases the mechanical stability of the system. However, as the patient’s head will rest deeper within the upper platform perimeter, dedicated space is required to accommodate their neck. One way to achieve this is by making space between two of the manipulator’s actuator attachment points on the upper platform. In traditional hexapods, cutting the upper platform between the two actuator attachment points as shown in Fig. 1(a) will not provide enough space to accommodate the patient’s neck due to the interference caused by the axis-symmetric arrangement of the actuators. To maintain and enhance manipulator mechanical stability while enabling the platform to accommodate lower pillow position, we designed a parallel architecture where the 6 actuators are grouped into 3 Leg-Pairs. Each Leg-Pair always forms an imaginary local plane (Leg-Plane), perpendicular to its normal vector n ˆ regardless of the position and orientation of the upper platform. The manipulator upper platform is connected by three 1-DoF rotary joints to three upper links, with their rotation axis located on the local XY plane of the s1 , perpendicular to the upper platform at point U0 and oriented along the vector ˆ vector c, as shown in Fig. 2. Each upper link, together with two prismatic linear actuators and a lower link form a single Leg-Pair. The upper link is connected on both ends to the two actuators by 1-DoF rotary joints located at U1 and U2 , whereas the two actuators are also connected to the lower links by 1-DoF rotary joints, located at B1 and B2 . The rotation axes of these 4 1-DoF rotary joints are parallel to the normal n ˆ of the Leg-Plane . The two prismatic linear actuators are located between points U1 and B1 , and U2 and B2 , respectively. Finally, each lower link is connected to the lower platform by a 2-DoF universal joint located at point s2 and 1 joint parallel to the the B0 , with 1 joint oriented on the Leg-Plane along ˆ global Z Axis along ˆ s3 . The four-bar linkage actuator unit possess 3-DOF, which combined with the 3 1-DOF rotary joint, provide the manipulator upper platform the capability to move to any position and orientation within the workspace. By combining the linear actuators two-by-two in each leg-pairs, the proposed arrangement can provide space for the patient’s neck by increasing the separation angle between two of its leg-pairs.

Analysis of Robotic Headrest for Vitreoretinal Surgery

173

Fig. 2. The proposed manipulator architecture: The notation used in the kinematic model. The 3 identical Leg-Pairs can be modelled the same way.

Inverse Position Kinematics: Inverse position kinematics are briefly introduced in support of the expanded statics derivation in the subsequent section. Figure 2 shows the notation used to establish the relation between the position and orienT  tation of the end-effector v = x y z θx θy θz and all the kinematic elements of the design, including the length of the prismatic actuators. For each leg, position of joints U1 and U2 are invariant to the upper platform local frame O, while the base point B0 is invariant in the base frame. The orientations of the joint axis vectors sˆ1 and sˆ3 are also invariant in their respective frames. The only unknown is therefore the orientation of sˆ2 . This vector is perpendicular to both sˆ3 and n = u1 × u2 , ˆ is known for each leg, the i.e. the normal to the plane of the leg. Once sˆ2 = sˆ3 × n configuration of the device can be fully established. Static Model and Jacobian Matrix: This section presents the analytical model for the force transmission from the 6 actuators to the end-effector. The derived model was cross-validated with numerical derivatives of the inverse kinematics model. To achieve static equilibrium in a given configuration, the sum of forces and moments on all rigid bodies must be zero. As it is the case for any parallel structure, the total forces and moments acting on the top platform are a linear combination of the sum of the forces and moments on each leg. The static analysis can therefore be calculated initially for each leg independently by the leg Jacobian matrix Jl . We will use the notation of Fig. 2 for a single leg-pair and retrieve the linear relation     fe(3×1) f = Jl(6×2) 1 , (1) τe (3×1) f2 where f1 and f2 are the forces produced by the two actuators of the leg and fe and τe are the forces and moments at the attachment point U on (top link, middle).

174

H. Natalius et al.

The total forces and moments can be broken down into a component that lies in the plane of the leg, and a component that is normal to that plane such that fe = fe,p + fe,n and τe = τe,p + τe,n . Note that the vectors in this section can be computed in any reference frame as long as the same frame is used for all vectors. On one hand, the calculation of the reaction forces in the plane (fe,p ), and the moments normal to the plane (τe,n ) are straightforward. In matrix form, we have       ˆ2 ˆ d fe,p d f1 = ˆ 1 (2) ˆ 2 × a2 = f2 τe,n d1 × a1 d On the other hand, the force normal to the plane fe,n and the moments in the plane τe,n require more elaborate calculations. Conceptually, these forces and moments are created because the universal joint at base point Bon cannot constrain the moment normal to the plane without creating additional moments in that plane. Indeed, the moment applied on the base platform by the actuators is given by ˆ 1 × b1 ) + f2 (−d ˆ 1 × b1 ), which is always directed along n ˆ , i.e. τb,n = f1 (−d normal to the plane. However, the only reaction moment allowed by the universal joint on the base is along the direction perpendicular to both of its joints, i.e. along s2 . Therefore, the total reaction moment on the base τb = τb (ˆ s3 ׈ s2 ) is such ˆ s3 × ˆ that its vector projection onto n ˆ is equal to τb,n . While the τb,n component of τb reaches static equilibrium due to the forces f1 and f2 created by the actuators, the residual moment τb,p = τb − τb,n = τb,p (ˆ n× ˆ s2 ) lies in the Leg-Pair plane and must be compensated by forces and moments applied at the leg attachment point U on . In matrix form:       τb   ˆ 2 × b2 f1 ˆ 1 × b1 d −ˆ s3 × ˆ s2 n ˆ ׈ s2 = d (3) τb,p f2 For given forces f1 and f2 , this constitutes a system of three equations and two unknowns. A Moore-Penrose pseudoinverse (+) can be used to find a least-squares solution to a system of linear equations that lacks a unique solution. However, since all vectors in this system lie in the same plane normal to ˆ s2 , only 2 out of the 3 equations are independent. Thus, the use of the pseudoinverse produces the exact system solution, and the residual moment τb,p is    +    ˆ 1 × b1 f1 ˆ 1 × b1 d ˆ ׈ s2 −ˆ s2 n ˆ ׈ s2 s3 × ˆ τb,p = 03×1 n d f2

(4)

 T To keep notation compact, (4) will be denoted as τb,p = J1 f1 f2 , where index 1 in J1 simply refers to the first building block of the complete Jacobian matrix J.

Analysis of Robotic Headrest for Vitreoretinal Surgery

175

The residual moment τb,p at the base of the leg lies in the plane of the Leg-Pair and must be compensated by forces and moments applied at the leg attachment point U on . The two ways of creating moments in that plane are by a linear force fe,n perpendicular to the plane, and a moment τe,p in the plane itself, which are respectively orthogonal to vectors fe,p and τe,n described in (2). Because of the revolute joint with axis ˆ s1 located at U on , the direction of τe,p at this point must be aligned with n ˆ ׈ s1 while a force fe,n at this point will create a moment fe,n (ˆ n × p), with p being a vector from U on to Bon . In static equilibrium:       fe,n f n ˆ×pn ˆ ׈ s1 = −J1 1 (5) τe,p f2 Since all the vectors lie in the same plane, only 2 of the 3 equations are independent. The pseudoinverse can be used for the exact solution for fe,n and τe,p :       + fe,n n ˆ 03×1  f n ˆ×pn ˆ ׈ s1 J1 1 =− (6) τe,p 03×1 n ˆ ׈ s1 f2 Adding (6) and (2) gives us the linear relation between the actuator forces f1 and f2 and the total reaction forces and moments fe and τe described in (1). The latter, however, only describes the forces for one leg at its attachment point. We are interested in the effect of all three legs at the point O located in the middle of the end-effector platform. The force of leg i at point O is given by         I(3×3) 0(3×3) f1i f fi  = J = Jli 1i (7) ci × I(3×3) i f2i τi f2i   where ci × is the cross-product matrix of vector ci going from leg attachment point U oi to end-effector point O. Now, we can assemble all three legs as: ⎡ ⎤   Jl1  T f = ⎣Jl2 ⎦ f11 f21 f12 f22 f13 f23 = JF (8) τ Jl3 Due to the power conservation principle, J can be used to establish the inverse velocity kinematics as q = J T v where q is the length of the 6 actuators. Another way to obtain J T is by numerical derivation of the inverse position kinematics. Numerical evaluations of J T was performed all over the workspace, converging to the same value for the Jacobian matrix J obtained analytically in this section. Metrics for Performance Analysis: We consider the reachable workspace, endeffector resolution for unit actuation stroke inputs (resolution ratio), and endeffector force output for unit actuation force input (force ratio). Stiffness and dynamics were not evaluated, and are left for future work. It is well known that parallel manipulators are prone to parallel singularities inside their workspace where J T vp = 0 and control in a particular direction vp is lost. As manipulator workspace, we define the volume in which the upper platform can move and rotate without encountering any singularity. It was analyzed

176

H. Natalius et al.

by sampling the determinant and the condition number of the manipulator Jacobian matrix J within a range of positions and orientations that cover the desired workspace. To numerically identify singularities given the sampling intervals, we detected sign changes of the Jacobian determinant. The final robot workspace should not exhibit singularities. The second performance metric is the end-effector resolution given unit actuation inputs, see [11]. This method maps a 6D hyper-cube corresponding to the resolution in the actuator space to a zonotope corresponding to the resolution in the end-effector space, the farthest points of which in each dimension correspond to the end-effector resolution in that dimension. The final robot should exhibit a resolution ratio that, for appropriate actuators, should fulfill the requirements of Table 1. Finally, the end-effector force ratio was evaluated within the entire workspace in the X, Y , and Z direction, following the methodology presented in [12]. Using (8), the force ratio corresponds to the highest absolute element in F needed among the 6 actuators to produce a unit force on the end-effector in one direction. This ratio will be used to determine the actuator force needed to meet the force requirements.

3

Results

The metrics described previously aid in understanding how a manipulator’s performance varies as the design parameters change. We employed them in a design evaluation approach that will not be fully described in this submission. In the end, to accommodate the patient’s neck, Γ1 was set at 86◦ , and subsequently Γ2 , and Γ3 , at 180◦ , and 274◦ , respectively. Furthermore, as the upper platform needs to be large enough to support both the head and the headrest, ru1 , ru2 and ru3 were set at 140 mm. To prevent the three upper links colliding with each other, lu was constrained to maximum 260 mm. The design parameters are shown in Table 2. Manipulator Workspace: Workspace analysis of the final design resulted in a set of condition number maps, where each map describes the condition number of the manipulator Jacobian Matrix at sampling points on an X-Y plane, given an initial height and orientation for the manipulator. Condition number maps were evaluated for representative sets of heights and orientations to ensure the absence of singularities within the desired workspace. Figure 3a shows the inverted value of the condition number for each sampling point for the manipulator at Z = 280 mm, which represents the bottom of the overall workspace and its most common starting configuration. As can be seen, no singularities are present. The condition number is fairly constant within the desired workspace (encircled), implying consistent manipulator behaviour. The design satisfies the requirements of Table 1. End-Effector Resolution: End-effector resolution analysis performed on the proposed manipulator design resulted in six resolution ratio maps, where each map details the resolution of the upper platform in a translational or rotational axis relative to the resolution of the actuator sensor, which was represented as qmax , at each sampling point. In translational motion along the X axis direction, the resolution factor of the upper platform is observed to be between 1.14 and 1.91, with the

Analysis of Robotic Headrest for Vitreoretinal Surgery

177

Table 2. Manipulator design parameter values Design parameter

Value

run : Length of imaginary line that connects upper platform center ru1 = 140 mm point to the rotary joint that connects the upper platform and the ru2 = 140 mm nth Leg-Pair upper link. Also referred as upper platform nth radius. ru3 = 140 mm Γn : Angle of separation between the upper platform nth radius and the x axis on the local XY Plane.

Γ1 = 86◦ Γ2 = 180◦ Γ3 = 274◦

lu : Upper link length.

lu = 260 mm

rbn : Length of imaginary line that connects lower platform center rb1 = 175 mm point to the universal joint that connects the lower platform and the r = 180 mm b2 nth Leg-Pair lower link. Also referred as lower platform nth radius. rb3 = 175 mm Θn : Angle of separation between the lower platform nth radius and the x axis on the global XY Plane.

Θ1 = 60◦ Θ2 = 180◦ Θ3 = 300◦

lb : Lower link length.

lb = 76 mm

lo : Vertical offset between the center of lower platform to the lower link axis of rotation.

lo = 40 mm

average value of 1.66. The maximum and average resolution factor values remain similar to that of the X direction, with the maximum value of 1.89 for both the translational motion on the Y and Z axis, and average value of 1.68 and 1.66 for the translation motion on the Y and Z axis respectively. Finally, the resolution factor for rotational motion relative to the global axes is similar to that for translational motion. End-Effector Force: The end-effector force ratio value for each sampling point within the manipulator theoretical workspace is shown as the manipulator endeffector force map, with three examples of the force map shown in Fig. 3b. Examination of the end-effector force ratio values within the desired workspace revealed the maximum end-effector force ratio value of 5.63 to 1, minimum value of 5.45 to 1, and average of 5.56 to 1. The minimum end-effector force ratio value will be used in determining the minimum actuator force that one actuator would need to exert. Manipulator Prototype: A prototype was created based on the design parameters to evaluate the manipulator’s performance in a manually actuated setting. The manipulator upper and lower platforms were made using acrylic plates, whereas complex components such as the joint were made using sintered nylon. Because the 1st iteration manipulator prototype will only be used for design evaluation, it is equipped with 6 non-motorized struts that represent the actuators. The manipulator prototype is shown in Fig. 4 alongside a Styrofoam head.

178

H. Natalius et al.

Fig. 3. Performance metric map for the proposed manipulator, Left: Sample condition number map with the workspace of interest marked within a circle, Right: End-effector force-ratio map for representative top platform orientations

Fig. 4. The assembled prototype of the proposed manipulator. The large reachable workspace is illustrated, while the phantom head serves to indicate the accessibility of the manipulator’s interior.

4

Discussion and Conclusion

The designed manipulator needs to use sensors and actuators with a particular minimum specification in order to fulfill the operational requirements. The end-effector resolution analysis gave the maximum upper platform resolution factor value for translational motion of 1.91. Therefore, as the resolution factor value was rounded to 2, the manipulator needs to have linear actuator sensors with minimum resolution of at least 5 µm to fulfill the end-effector resolution target of 10µm. Furthermore, because the manipulator needs to support a weight of 30 kg or 294 N, each actuator will need to be able to exert force of at least 54.04 N as the minimum value of the end-effector force ratio is 5.45 to 1. Therefore, the performance requirements are turned into actuation component requirements, and pave the road for motorisation. Acknowledgement. This research was supported by the Sir Michael Uren Foundation. H. Natalius is supported by an Overseas Research Scholarship from University College London. H. Natalius, M. K. Tiwari and L. da Cruz are with University College London,

Analysis of Robotic Headrest for Vitreoretinal Surgery

179

UK. Further, L. da Cruz is also with Moorfields Eye Hospital, London, UK. P. Lambert and C. Bergeles are with King’s College London. H. Natalius and P. Lambert are equally contributing first authors, while C. Bergeles and L. Da Cruz are equally contributing senior authors. [email protected].

References 1. Da Cruz, L., Fynes, K., Georgiadis, O., Kerby, J., Luo, Y.H., Ahmado, A., Vernon, A., Daniels, J.T., Nommiste, B., Hasan, S.M., Gooljar, S.B., Carr, A.J.F., Vugler, A., Ramsden, C.M., Bictash, M., Fenster, M., Steer, J., Harbinson, T., Wilbrey, A., Tufail, A., Feng, G., Whitlock, M., Robson, A.G., Holder, G.E., Sagoo, M.S., Loudon, P.T., Whiting, P., Coffey, P.J.: Phase 1 clinical study of an embryonic stem cellderived retinal pigment epithelium patch in age-related macular degeneration. Nat. Biotechnol. 36, 1–10 (2018) 2. REP1 Gene Replacement Therapy for Choroideremia - Full Text View - ClinicalTrials.gov. https://bit.ly/37MLG5N. Accessed 02 Dec 2019 3. Riviere, C.N., Scott Rader, R., Thakor, N.V.: Adaptive canceling of physiological tremor for improved precision in microsurgery. IEEE Trans. Biomed. Eng. 45, 839– 845 (1998) 4. Brogan, K., Dawar, B., Lockington, D., Ramaesh, K.: Intraoperative head drift and eye movement: two under addressed challenges during cataract surgery. Eye. 32, 1111–1116 (2018) 5. Vander Poorten, E., Riviere, C.N., Abbott, J.J., Bergeles, C., Nasseri, M.A., Kang, J.U., Sznitman, R., Faridpooya, K., Iordachita, I.: Robotic retinal surgery. In: Handbook of Robotic and Image-Guided Surgery (2020) 6. Huang, K., Zhou, M., Lajblich, C., Lohmann, C.P., Knoll, A., Ling, Y., Lin, H., Nasseri, M.A.: A flexible head fixation for ophthalmic microsurgery. In: Proceedings - 2017 Chinese Automation Congress, CAC 2017, pp. 6707–6710. Institute of Electrical and Electronics Engineers Inc. (2017) 7. Wirz, R., Lathrop, R.A., Godage, I.S., Burgner-Kahrs, J., Russell, P.T., Webster, R.J.: Can coffee improve image guidance? In: Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling, p. 941513 (2015) 8. H-825 6-Axis Hexapod. https://bit.ly/31by9Cc. Accessed 02 Dec 2019 9. HexGen HEX300-230HL—Aerotech, Inc. https://bit.ly/2u7CCd4. Accessed 02 Dec 2019 10. Clauser, C., McConville, J., Young, J.: Weight, volume and center of mass of the human body. Springf. NTIS. (1969) 11. Cardou, P., Bouchard, S., Gosselin, C.: Kinematic-sensitivity indices for dimensionally nonhomogeneous jacobian matrices. IEEE Trans. Robot. 26, 166–173 (2010) 12. Nokleby, S.B., Fisher, R., Podhorodeski, R.P., Firmani, F.: Force capabilities of redundantly-actuated parallel manipulators. Mech. Mach. Theor. 40, 578–599 (2005)

Lasers, Planning, and Navigation in Surgery

Multimodal Risk-Map for Navigation Planning in Neurosurgical Interventions Maximilian Gerst, Christian Kunz, Pit Henrich, and Franziska Mathis-Ullrich(B) Institute for Anthropomatics and Robotics, Health Robotics and Automation Lab, Karlsruhe Institute of Technology, Karlsruhe, Germany [email protected]

Abstract. Planning a keyhole neurosurgical intervention is challenging due to the complex brain anatomy and the risk of penetrating vital structures. This work presents a neurosurgical planning tool for safe and effective interventions in the brain by minimizing risk due to optimized access planning. The tool uses a modular architecture that allows integration of various risk structures, combined in a holistic model represented as volumetric risk maps. Risk evaluations are computed for linear and nonlinear trajectories and intuitively visualized as a projection onto the head surface. Neurosurgeons can choose between two separate modes for automatic access planning according to the type and complexity of the neurosurgical procedure. An evaluation with clinical experts demonstrates the practical relevance.

Keywords: Neurosurgical planning assisted interventions

1

· Risk determination · Computer

Introduction

Brain tumors are responsible for less than 2% of the total cancer cases in humans. However, they cause considerable morbidity as the human brain is a very sensitive organ. Depending on the type of cancer, the five-year survival rate in Germany was 23% on average in 2016 [1]. Due to the sensitivity of the brain and the high-risk surgical process, navigated minimally invasive procedures are the gold standard of care. Minimally invasive procedures typically requires the insertion of a needle-shaped instrument into the cranial structures. Thus, a wellchosen trajectory for the instrument is essential for the rapid and safe execution of neurosurgical interventions in the brain and the minimization of injury to non-tumorous tissue. Even for experienced surgeons, planning a keyhole brain surgery is often a very demanding task. The challenge arises due to the complex morphology of the human brain and the mental demand of avoiding all different M. Gerst and C. Kunz—The authors contributed equally to this work. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 183–191, 2021. https://doi.org/10.1007/978-3-030-58104-6_21

184

M. Gerst et al.

risk structures. In the standard of care, trajectory planning is performed manually, and therefore relies on the experience of the surgeon. To verify the chosen trajectory and assess the treatment risk, a surgeon has to consider each image slice along the trajectory and the different risk structures, which is a complex and time-consuming task. To support the surgeon during planning, several systems for planning brain tumor resections have been proposed. Navkar et al. and the subsequent work by Rinc´ on-Nigro et al. present an approach to calculate the risk of linear trajectories through brain tissue and the projection onto the cranial surface [2,3]. However, this work does not take into account the surgical planning workflow and therefore neglects non-formalizable expert knowledge. Shamir et al. [4] evaluates different methods of risk calculation and Le´ on-Cuevas et al. [5] use fuzzy logic for risk calculation. Neither work considers the surgical planning process. Extending on this research, Essert et al. [6] utilize a global trajectory optimization strategy that combines cost functions of strict and soft constraints together with user-dependent weightings of these cost functions. The mentioned research only considers blood vessels and the ventricular system as risk structures inside the brain, and thus, does not provide an underlying holistic multimodal planning model that consolidates various risk structures. Vaillancourt et al. [7] on the other hand consider fiber tracts and visualize the corresponding risk. Diepenbrock et al. [8] examine the interaction of structures (blood vessels and functional areas) with a tumor and their visualization. However, these studies do not provide the projection of a risk map onto the cranial surface to allow fast trajectory planning. This work aims to support neurosurgeons by providing a tool for simple and efficient 3D risk structure visualization and faster planning of neurosurgical interventions. The proposed system supports the surgeon to effectively choose the most appropriate entry point and access route, avoid vital structures, and minimize potential trauma to healthy tissue. We extend the research by incorporating different risk structures into a holistic, easily extensible model and by taking into account non-formalizable expert knowledge. We provide neurosurgeons with a comprehensive and interactive decision support for the task of access planning with the possibility of different user-adjustable risk calculation methods and the ability to plan on the cortical brain surface.

2

Methods

The required user inputs and the procedure for surgical trajectory planning are presented in Fig. 1. After acquiring multimodal 3D medical images (e.g. CT and MRI) of the patient, several risk maps are created, representing vital structures inside the patient’s head, such as blood vessels, functional areas, nerve fiber tracts, as well as the skin, skull and ventricular system. Additionally, a risk map of tumorous tissue is created. As a result, a risk map represents anatomical structures or pathological tissue. Due to the software’s modular architecture, imaging modalities and pathologies are not restricted and it is thus possible to implement additional risk structures, such as hemorrhages or specialized functional areas.

Multimodal Risk-Map for Navigation Planning

185

Each of these risk maps is stored in its own data structure to enable more flexible processing capabilities. The risk maps are extracted from a patient’s segmented 3D medical image data and are therefore limited to previously acquired image modalities. Blood vessels are segmented using a digital subtraction angiography (DSA) and refined with a modified Vesselness filter [9]. The algorithm by Kunz et al. [10] is used to segment the ventricular system, skin and skull from CT data. A tumor mask is used for segmentation of the tumor and a threshold is used for segmentation of the functional areas. In the dataset used, the fibrous webs were already segmented. Figure 2 shows the segmented and differentiated risk structures forming the 3D risk maps. Furthermore, to allow for comparison between the various modalities, a registration is carried out, such that each voxel of the different modalities represents the same physical location within the patient’s head. Standard registration algorithms are used, such as the head-and-hat and the iterative closest point algorithm, depending on the image modality. The proposed software allows the surgeon to show and hide the individual relevant risk maps. Predefined weights for all risk maps reflect the importance of the represented anatomical structure. Furthermore, surface models of the patient’s skin and brain surface are calculated, onto which the risk maps are later projected. A surgeon then picks the target point for navigation in the patient’s head. The vertices of the surface model mesh represent possible entry points into the skull during the surgical procedure. The resolution of the mesh directly corresponds to the number of considered entry points. A trajectory is calculated from each vertex to the target point for a surgical instrument as selected by the surgeon. The proposed planning tool offers two different modes for risk calculation. Mode 1 calculates the projection map using the selected risk maps as parameterized by the surgeon (see Fig. 3). Possible surgical trajectories with minimal probability of penetrating risk structures are suggested by the software (refer to Sect. 2.1). This mode offers information to assist the surgeon during decision making for highly complex surgeries that can not be performed completely risk-free. In Mode 2 additional hard and soft constraints are added. The soft constraints consider the path length and the distance to risk structures. The hard constraint blocks a trajectory if any risk structure is hit. The cumulative risk of each trajectory is projected onto the skin or cortical surface so that the surgeon can intuitively choose an entry point that, according to surgical experience, minimizes the risk. 2.1

Computation of Projection Risk Maps

In Mode 1 the risk of a trajectory Tj in the 3D risk map Mi is defined as r(Mi , Tj ) and calculated by integrating over discrete space along the trajectory. The number of considered risk structures is described by i. The total risk of Tj is calculated as a weighted normalized sum using Eq. 1, with preselected weights

186

M. Gerst et al. Surgeon Choose Risk Structures

Software Registration of Risk Structures

Define Target Point

Segmentation of Risk Structures

Risk Calcualtion

Amplification Function

Mode 1 Mode 2 Restrain with Mode? Strict Constraints

Mesh Extraction

Risk Map Consolidation Risk Calculation based on soft constraints

Skin Projection Map

Define Risk Weights

Fig. 1. Information flow diagram of the proposed system.

wi for each risk structure. ˆ j) = R(T



r(Mi , Tj ) · wi

i

(1)

ˆ j) R(T R(Tj ) = ˆ k) maxk R(T

Blood vessels

Ventricular system

Functional areas

Fiber tracts

Tumor

Holistic representation

Fig. 2. Risk maps representing different anatomical and pathological structures.

To increase safety in close proximity to voxels associated with high risk, R(Tj ) is scaled utilizing an amplification function, described in Eq. 2. The result-

Multimodal Risk-Map for Navigation Planning

187

ing risk R1 (Tj ) is subsequently projected onto the brain surface or skin surface model to form the color map that guides the surgeon during complex procedures, such as the treatment of deep tumors. R1 (Tj ) = 1 − e−C·R(Tj )

(2)

During risk calculation in Mode 2, two types of restrictive constraints are implemented. Hard constraints (HC) represent risks that must be avoided during surgery, whereas soft constraints (SC) define risks that need to be minimized. An optimal surgical trajectory must comply with all hard constraints while minimizing risk defined by the soft constraints. The hard constraints defined ensure the surgical trajectory length is shorter than the total length of an inserted instrument (HC1), and the trajectory does not penetrate any risk structures (HC2), such as blood vessels, the ventricular system, or functional areas. After applying the hard constraints to the trajectories, all remaining solutions are rated according to their total lengths, and the distance to risk structures. While the trajectory’s length should be minimized to cause the least damage to healthy brain tissue (SC1), the distance to risk structures is maximized to counteract small path deviations and brain shift during surgery (SC2). While the risk associated with hard constraints is binary (i.e., rh (·, ·) ∈ {0, 1}), soft constraints are defined on a continuous scale. Assuming a linear relationship between the length of trajectory Tj and associated risk, SC1 is expressed as a normalized risk function Rl , where lmin = mink length(Tk ) and lmax = maxk length(Tk ) represent the longest and shortest trajectories, respectively. length(Tj ) − lmin Rl (Tj ) = (3) lmax − lmin To construct a safety margin around vital regions, a modified Gaussian filter is applied to the risk maps Mi . The resulting normalized risk function Rd , using the dilated high risk voxel structures, is given as  ˆ d (Tj ) = R r(Gi , Tj ) · wi i

ˆ d (Tj ) R Rd (Tj ) = ˆ d (Tk ) maxk R

(4)

where Gi is obtained by applying a modified Gaussian filter to Mi . The modification ensures that risk is never decreased, but is only dilated. The total risk for a given trajectory Tj is defined as  1 |{Mi |rh (Mi , Tj ) = 1}| ≥ 1 R2 (Tj ) = 1 (5) (R (T ) + R (T )) else l j d j 2 where weights wi are predefined by the surgeon. For an intuitive visualization of trajectories, originating at the vertices of the surface model, the vertices are colored according to the risk of the associated

188

M. Gerst et al.

trajectory. This results in the projection of the internal risk structures for a given trajectory onto the cranial or cortical surfaces. To evaluate the proposed method in regard to the clinical workflow, a surgical planning tool is developed on the basis of the Insight Segmentation and Registration Toolkit (ITK) as well as the Visualization Toolkit (VTK) (Fig. 4).

Fig. 3. Risk projection onto the head surface.

2.2

Fig. 4. User interface of the planning tool.

Robot Operating System Implementation

The surgical planning tool was first implemented as a standalone prototype and in a second step within the Robot Operating System (ROS) as nodes. The 3D risk maps of every risk structure, as well as the consolidated 3D risk map (including all structures) can be queried over a ROS service. The volumetric risk maps can be used to plan the risk of any trajectory through a patient’s cranial structure. Out of the 3D risk maps and the cranial or skin surface, the risk projection maps of every risk structure and the consolidated risk projection map is calculated and also provided via a service. For visualization the colored cranial or cortical surface mesh can be queried.

3

Results

The evaluation of the presented neurosurgical planner includes a quantitative analysis of its computational performance as well as a qualitative user study with medical experts. For evaluation, two sets of medical images were available, one of which was an open source dataset supplied by the IEEE Visualization Contest 2010. These datasets consists of anatomical and functional images of a human brain. The second dataset was supplied for internal use. For the evaluation we combined a T1-weighted, a T1-weighted image with a contrast agent, a computed tomography (CT), a functional magnetic resonance image (fMRI) of a fingertapping task, and a diffusion tensor imaging (DTI) image.

Multimodal Risk-Map for Navigation Planning

3.1

189

Computational Performance

The runtime for computation of the risk maps and the risk projection were measured for both datasets to quantitatively analyze our approach. Evaluation was performed for Mode 1, being the computationally more intensive mode. Table 1 shows the resulting initial computation and update time of the 3D risk maps and the projection map in correspondence to the number of considered risk structures inside the brain. The computational evaluation was performed on a mesh with 43280 vertices. The initial computation was implemented using parallel processing on the CPU. All measurements were performed on a workstation with the following configuration: Intel Core i7 6820HQ, Nvidia Quadro M3000M, 16 GB DDR4 RAM, SSD. Table 1. Runtime behavior of the presented approach. Initial computation time: Calculation of 3D risk maps and the consolidation as weighted sum. Update time: Recalculation with changed weights. Number of risk structures Initial computation time (s) Update time (s)

3.2

1

11

1,2

2

15

1,3

3

21,5

1,4

4

27

1,5

Survey with Clinical Experts

To obtain qualified feedback on the presented approach, five neurosurgeons from two German hospitals were included in a user study. The task was to plan an optimal trajectory for a ventricular puncture, a common neurosurgical procedure. Qualitative feedback was obtained both from “thinking-out-loud” sessions and from a formal survey. The user study resulted in a ranking of considered risk structures during a ventricular puncture according to their risk level, with functional areas being of highest importance, followed by damage of blood vessels and fiber tracts, path lengths of the surgical instrument trajectory, accidental puncture of air-filled sinuses, and cosmetic considerations. In general, the interviewed neurosurgeons rated the demonstrator as useful (average points given: 4/5), with the most important features being the time saved during surgical planning, the great potential for educational purposes, the ability to integrate the planner into the complete surgical workflow, and a meaningful use of the otherwise less required 3D view. Additionally, it was suggested to improve the surgical planning tool by adding a more detailed way to inspect selected trajectories. In conclusion, the proposed planning tool shows the potential to close the gap between planning and practical surgical experience.

190

4

M. Gerst et al.

Discussion

The developed surgical planner targets the combination of arbitrary risk structures, such as blood vessels and tumorous tissue, in a holistic risk map. Projection of the risk values onto the skin and brain surfaces can be used for intuitive entry point planning. Advantages of the system include ease-of-use due to the direct integration into the surgical planning workflow, the possibility to parameterize risk weights as well as the easily extendable modular architecture. The initial calculation time of the risk maps and projection risk map takes up to 30 s. Once the projection has been calculated, the risk data is stored in memory to allow faster calculation when a new weighting of the risk structures is selected, with an update time of 2 s. During initialization, a longer computation time is acceptable as calculations are performed as soon as the image data is available prior to the planning procedure. During planning an update time of 2 s is acceptable but can be further improved. This work was limited by the availability of only a few datasets for evaluation purposes. In addition, positive effects on the surgical workflow have so far only been shown to a limited extent. To address these limitations, future work should improve the planner with a focus on obtaining measurable results of time and risk reduction. The modular system architecture makes it easy to expand the risk structures considered. It allows the calculation of linear and non-linear trajectories as illustrated in Fig. 5. Trajectory A shows a linear and B a non-linear trajectory, circumnavigating risk structures.

Linear trajectory.

Non-linear trajectory.

Fig. 5. Planned trajectories through risk structures and projection onto the skin.

5

Conclusion

The presented method allows for an accelerated and information-based process of planning a neurosurgical instrument’s trajectory, thus, rendering cranial interventions safer. By integrating the system into the surgical workflow, an intuitive planning tool for neurosurgical interventions was created that supports the OR

Multimodal Risk-Map for Navigation Planning

191

staff and potentially improves the outcome for the patient. Depending on the procedure and the surgeon’s assessment, both the weighting of the risk structures and the method of risk computation may be adjusted. Thus, the planner incorporates non-formalizable knowledge and experience of the surgeon. Due to its modular architecture, the functionality of this system can be extended with minimal effort to consider other registered risk maps. We are thus broadening the state of the art with a novel methodology for risk minimization during neurosurgical planning, based on a holistic, extendable model from various anatomical risk structures, resulting in fast access planning. Acknowledgment. The authors acknowledge the expertise provided by neurosurgeons from the Departments of Neurosurgery at the Klinikum Karlsruhe and the University Hospital Ulm/G¨ unzburg.

References 1. Robert Koch-Institut, Gesellschaft der epidemiologischen Krebsregister in Deutschland e.V.: Krebs in Deutschland 2015/2016 (2019). https://doi.org/10.25646/5977 2. Navkar, N.V., et al.: Visualization and planning of neurosurgical interventions with straight access. In: Navab, N., Jannin, P. (eds.) IPCAI 2010. LNCS, vol. 6135, pp. 1–11. Springer, Heidelberg (2010) 3. Rinc´ on-Nigro, M., et al.: GPU-accelerated interactive visualization and planning of neurosurgical interventions. Comput. Graph. Appl. 34, 22–31 (2014) 4. Shamir, R.R., et al.: Reduced risk trajectory planning in image-guided keyhole neurosurgery. Med. Phys. 39(5), 2885–2895 (2012) 5. De Le´ on-Cuevas, A., et al.: Risk map generation for keyhole neurosurgery using fuzzy logic for trajectory evaluation. Neurocomputing 233, 81–89 (2017) 6. Essert, C., et al.: Automatic computation of electrode trajectories for Deep Brain Stimulation: a hybrid symbolic and numerical approach. Int. J. CARS 7, 517–532 (2012) 7. Vaillancourt, O., et al.: A fiber navigator for neurosurgical planning (NeuroPlanningNavigator). In: 16th IEEE VIsualization Conference, Utah (2010) 8. Diepenbrock, S., et al.: 2010 IEEE visualization contest winner: interactive planning for brain tumor resections. IEEE Comput. Graph. Appl. 31(5), 6–13 (2011) 9. Frangi, A.F., et al.: Multiscale vessel enhancement filtering. In: Wells, W.M., Colchester, A., Delp, S. (eds.) MICCAI 1998. LNCS, vol. 1496, pp. 130–137. Springer, Heidelberg (1998) 10. Kunz, C., et al.: Fast volumetric auto-segmentation of head CT images in emergency situations for ventricular punctures. In: CURAC 2019, pp. 41–46, Reutlingen (2019)

Augmented Reality Based Surgical Navigation of the Periacetabular Osteotomy of Ganz – A Pilot Cadaveric Study Armando Hoch1,2(B) , Florentin Liebmann1,3 , Fabio Carrillo1 , Mazda urnstahl1 Farshad2 , Stefan Rahm2 , Patrick O. Zingg2 , and Philipp F¨ 1

2 3

Research in Orthopaedic Computer Science, Balgrist University Hospital, University of Zurich, Zurich, Switzerland [email protected] Orthopaedic Department, Balgrist University Hospital, University of Zurich, Zurich, Switzerland Laboratory for Orthopaedic Biomechanics, ETH Zurich, Zurich, Switzerland

Abstract. Introduction: The periacetabular osteotomy of Ganz is one of the most demanding procedures in orthopaedic surgery and requires a profound three-dimensional understanding of the human anatomy. The use of augmented reality offers new possibilities for computer-assisted interventions. The aim of our study was to investigate if the navigation of the periacetabular osteotomy of Ganz by the Microsoft HoloLens is feasible in a pilot cadaveric study. Material and Methods: An augmented reality based registration and navigation method for the periacetabular osteotomy of Ganz was developed. The pelvic bone was registered using landmarks and surface digitization and the osteotomies as well as the reorientation of the acetabular fragment were guided holographically. An orthopedic surgeon performed the procedure under realistic operating room conditions on one human cadaver for evaluation. The performed osteotomy starting points, the angles between the corresponding connecting lines and the reorientation of the acetabular fragment were compared to the preoperative planning. Results: Distances between planned and performed osteotomy starting points were 12.3 mm, 1.4 mm, 3.0 mm and 9.6 mm. Projected angles between the connecting lines of planned and performed osteotomy starting points were 7.8◦ , 2.1◦ and 15.2◦ . The projected angle between supraand retroacetabular osteotomy deviated 6.4◦ and the angle between retro- and infraacetabular osteotomy deviated 16.4◦ from the planning, respectively. Performed reorientation of the acetabular fragment was 3.3◦ (planned 10◦ ) in the frontal and 2.1◦ (planned 15◦ ) in the sagittal plane. Conclusion: The execution of complex osteotomies is feasible with navigation through the Microsoft HoloLens with a satisfying realization of the preoperative plan. Nevertheless, further work is needed to also improve the navigation of the reorientation of the acetabular fragment. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 192–201, 2021. https://doi.org/10.1007/978-3-030-58104-6_22

AR Guided PAO Keywords: Periacetabular osteotomy Augmented reality

1

193

· Surgical navigation ·

Introduction

The periacetabular osteotomy of Ganz (PAO), introduced in 1988, is one of the most demanding procedures in orthopaedic surgery. As it requires a profound three-dimensional (3D) understanding of the human anatomy [7], the surgery is typically performed by experienced hip surgeons. The objective of the PAO is to improve the osseous containment of the hip joint in young adults with residual developmental dysplasia of the hip. The typical patient is female and in childbearing age. The acetabular fragment is cut free from the pelvic bone by four osteotomies, namely the supra-acetabular, the pubic, the ischial and the retroacetabular osteotomy. The mobilized fragment is then reoriented in space and fixed to the remaining pelvis with screws. The surgeon uses anatomical landmarks to define the osteotomy starting points. Thereby, the ischial and parts of the retroacetabular osteotomy have to be performed without direct sight on the anatomy. For these osteotomies, the surgeon typically depends on his tactile sensation and the intensive use of intraoperative fluoroscopy. The reorientation of the acetabular fragment has to be verified also under intraoperative fluoroscopy. Overall, the patient is exposed to a considerable amount of radiation. Incorrectly performed osteotomies can lead to fatal consequences like damage to the articular cartilage or discontinuity and subsequent instability of the pelvic ring. Furthermore, a malpositioning of the acetabular fragment may lead to a mechanical conflict between the femoral head-neck junction and the acetabular rim which in turn may need revision surgery. Through recent developments in augmented reality (AR) in form of offthe-shelf optical see-through head-mounted displays (OST-HMD), intraoperative visualization of 3D preoperative plannings and holographic surgical navigation has become not only possible, but also easier and more affordable [4,5,9,13,16,17]. The purpose of this feasibility study was to find out if holographic surgical navigation of a PAO with an OST-HMD (Microsoft HoloLens, Microsoft Corporation, Redmond, WA, U.S.A.) can be performed on a human cadaver. We wanted to particularly investigate how precise the preoperative planning of the osteotomies can be realized and how accurate the reorientation of the acetabular fragment can be performed.

2

Materials and Methods

One hip of one thawed fresh-frozen human cadaver was used for this study. The specimen had no history of trauma, malformation, tumor or surgery in this region. This study was approved by the local ethical committee (KEK Zurich BASEC Nr. 2018-00922).

194

2.1

A. Hoch et al.

Preoperative Planning

A Computed Tomography (CT) scan of the cadaver was performed at our instituR device (Siemens, Erlangen, Germany). A slice tion using a Somatom Edge CT thickness 1.0 mm and an in-plane resolution (x-y) of 0.4 0.4 mm was used. The CT data was segmented using the global thresholding and region growing functionalities of a commercial segmentation software (Mimics Medical, Materialise NV, Leuven, Belgium) to generate 3D models of the pelvis and femur [6,12,24]. The 3D preoperative planning was performed by two experienced orthopaedic surgeons (S.R., P.O.Z.) using our in-house developed preoperative planning software CASPA (Balgrist University Hospital, Zurich, Switzerland). With the help of CASPA, the starting points of the osteotomies and the 3D cutting planes were placed according on the technique initially described by Ganz [7]. As the cadaver had a physiological hip anatomy, the amount and direction of the reorientation of the acetabular fragment were chosen such that they resembled a typical correction in developmental dysplasia of the hip. A rotation of 10◦ in the anterior-posterior plane around the predefined y-axis and 15◦ in the sagittal plane around the predefined z-axis was applied using center of rotation was given by the center of rotation of the hip joint (see Fig. 1).

Fig. 1. (a) Preoperative planning defining the 3 osteotomy planes (light blue). (b) Planned reorientation of the acetabular fragment shown in purple. The red, green and blue arrows define the x-axis, y-axis, z-axis, respectively, of the anatomical coordinate system.

AR Guided PAO

2.2

195

AR Based Holographic Surgical Navigation

The main goal of our method was to enable the surgeon to perform the osteotomy cuts and reorientation of the acetabular fragment under holographic surgical navigation according to the preoperative plan. This required the following major steps: coarse registration between pre- and intraoperative anatomy, fine registration, tracking of the pelvis, visualization of osteotomy planes and tracking of the fragment. The procedure was executed by an orthopaedic surgeon under realistic operating room (OR) conditions. The standard surgical approach of the PAO described by Ganz was conducted [7]. Tracking of the pelvis and the acetabular fragment was achieved by attaching two custom-made 3D printed mounts equipped with markers (Fig. 2b) to the pelvis (pelvic mount and marker) and the assumed region of the fragment (fragment mount and marker). The mounts were designed and positioned such that they did not interfere with the surgical procedure nor lead to additional harm on the native anatomy. The part performed under holographic surgical navigation was then performed as follows. 1. Registration (a) Definition of 3 anatomical landmarks (Fig. 2a) using a pointing device (PD; Fig. 2b) (b) Digitization of the accessible surface of the pelvic bone (Fig. 2a) using the PD (c) Visual verification of registration result based on holographic overlay (d) Registration of the pelvic and fragment markers. 2. AR based navigation of the osteotomies (a) Navigation of the supraacetabular and retroacetabular osteotomies by visualization of osteotomy planes. The ischial osteotomies is navigated by visualization the chisel poses. (b) Pubic osteotomy is not navigated and performed in a freehand fashion. 3. AR based navigation of the acetabular fragment (a) Reorientation of the mobilized acetabular fragment (b) Visual verification of the fragment target pose based on holographic overlay. After reorientation, the acetabular fragment was fixated to the pelvic bone with screws and the surgical wound was closed layer by layer. 2.3

Implementation

Our registration approach based on our previous work on radiation-free surface digitization [13] using the PD mentioned above (Fig. 2b). The sterile marker (Clear Guide Medical, Baltimore MD, USA) integrated in the PD shows an AprilTag pattern [19,25] that can be tracked in 6 degrees of freedom (DoF) by the two front-facing environment tracking cameras of the HoloLens using OpenCV [3] and the ArUco library [8,21]. In this study, a coarse alignment was

196

A. Hoch et al.

achieved by manually capturing three landmarks on the pelvic bone (Fig. 2a) using the PD. Afterwards, the accessible region on the pelvic bone was digitized into a 3D point cloud by tracking the continuous motion of the PD tip when following the anatomy. This point cloud was then aligned to the preoperative 3D model using the iterative closest point (ICP) surface registration [2].

Fig. 2. (a) The 3 anatomical landmarks used for the initial registration are shown as red, green, blue spheres. The accessible bone surface that will be digitized and used for fine registration is shown in brown. (b) Custom-made 3D printed mount for the markers (left) and pointing device used for registration (right). (c) Visualization of cutting planes and chisel as presented holographically to the surgeon during navigation of the osteotomies.

Applying osteotomy cuts with a chisel leads to motion of the anatomy and, consequently, to a loss of the registration. We addressed this issue by extending the original method with a continuous motion compensation strategy. To this end, after successful registration, the pelvis pose was stored with respect to the pelvic marker (Sect. 2.2, 1d). The osteotomies were guided visually by rendering planes for supra- and retroacetabular cuts and chisel positions for the ischial cut, respectively. Thereby, the pelvic bone was rendered in black color making it appear transparent. This gave the surgeon a better visual perception of the intersection between the virtual plane and real pelvic anatomy. For the reorientation of the mobilized acetabular fragment, the surgeon could choose between visual guidance where the fragment was rendered at the planned postoperative position and a tracking-based guidance with a display showing the angular and positional deviation between the current and the postoperative fragment pose. The latter was computed after successful registration with respect to the fragment marker (Fig. 2a). Our method was implemented as a holographic Universal Windows Platform application for the Microsoft HoloLens (Microsoft Corporation, Redmond, WA, USA) using Unity (Unity Technologies, San Francisco, CA, USA).

AR Guided PAO

2.4

197

Outcome Measurements

A CT scan of the cadaver after PAO was acquired to obtain 3D bone models in the same manner as for preoperative planning. The CASPA software was used to superimpose the pre- and postoperative bone models by using ICP [2,22] and followed by a manual correction of the alignment. One key measure of the outcome evaluation were the planned and performed osteotomy starting points P L1 . . . P L4 and P F1 . . . P F4 , and their resulting vectors V L1 . . . V L4 and V F 1 . . . V F 4 , respectively. They were defined as follows: – P L1 , P F1 : most lateral point on intersection between supraacetabular osteotomy plane and corresponding pelvic 3D model (pre- or postoperative) – P L2 , P F2 : most superior points on intersecting line between supraacetabular and retroacetabular osteotomy planes – P L3 , P F3 : most superior points on intersecting line between retroacetabular and ischial osteotomy planes – P L4 , P F4 : most inferior point on intersection between ischial osteotomy plane and corresponding pelvic 3D model – V L1 = P L1 − P L2 , V L2 = P L3 − P L2 , V L3 = P L2 − P L3 and V L4 = P L4 − P L3 – V F 1 . . . V F 4 according to V L1 . . . V L4 . For 2D measurements, all vectors were projected onto a plane V P defined as the best fit in a least-squares sense by P L1 . . . P L4 . The projected vectors are denoted as V Li and V F i . For each pair of starting points P Li and P Fi , the 3D distance was quantified. For each pair of V Li and V F i , the 2D angle was measured. Furthermore, the 2D angle between V F 1 and V F 2 was compared to the one between V L1 and V L2 . Accordingly, the same was done for vectors V F 3 , V F 4 and V L3 , V L4 . The reorientation error of the acetabular fragment was measured by calculating the difference of the rotational correction between the planned and the performed position of the acetabular fragment according to the anatomical planes and the corresponding coordinate system.

3

Results

The 3D distances between each pair of starting points P Li and P Fi were 12.3 mm, 1.4 mm, 3.0 mm and 9.6 mm (Fig. 3a). The 2D angles between each pair of starting point connecting vectors V Li and V F i were 7.8◦ , 2.1◦ and 15.2◦ (Fig. 3b). The 2D angle between supra- and retroacetabular osteotomy deviated 6.4◦ and the angle between retroacetabular and ischial osteotomy deviated 16.4◦ from the planning, respectively (Fig. 3b). Regarding the reorientation of the acetabular fragment the surgeon faced certain issues when trying to reach its target position because the pelvis bone and the soft tissue anatomy interfered. The performed reorientation of the acetabular fragment was 3.3◦ (planned 10◦ ) in the frontal and 2.1◦ (planned 15◦ ) in the sagittal plane (Fig. 3c).

198

A. Hoch et al.

Fig. 3. (a) Pre- (bright) and postoperative (dark) 3D model of pelvic bone with planned (green) and performed (orange) osteotomy starting points. (b) Starting points projected onto plane P (blue) and resulting 2D lines used for angle measurements. (c) Planned (green) and performed (orange) reorientation of acetabular fragment and the coordinate system used for planning (see Sect. 2.1). (d) Overlap between planned (green) and obtained (orange) acetabular fragment.

4

Discussion

The periacetabular osteotomy of Ganz is typically performed by experienced hip surgeons and the outcome is highly dependent on the surgeon’s performance. Incorrectly performed osteotomies can lead to fatal consequences like damage to the articular cartilage or discontinuity and subsequent instability of the pelvic ring. Furthermore, the procedure requires an operating time of approximately two hours, which is associated with a continuous blood loss and a steadily increasing risk of infection. A considerable amount of time is spent on the complicated execution of the osteotomies and fragment reorientation under radiographic control. Thus, a navigation method to improve these aspects would be highly desirable. In this study, we present a radiation-free navigation solution for the periacetabular osteotomy of Ganz based on surface digitization and intuitive holographic surgical navigation. In the literature, several methods for AR based surgical navigation were presented and most of them focused on spine surgery [5,9,13,14,17]. Other anatomies and applications were targeted as well, e.g. recently co-calibration of a C-arm and an OST-HMD [1]. However, in computer-assisted interventions not only the question of visualization, but also the issue of registration plays an important role. Many different approaches are available, of which most are dependent on intraoperative x-ray usage [15]. Surface digitization methods, such as the one we employed, offer a radiation-free solution and are implemented in state-of-the-art navigation system, e.g. for spine surgery [18]. Ongoing technological advance may even allow more automated workflows, such as in [10] with a markerless registration for head and neck cancer surgery. Regarding the PAO of Ganz no holographic surgical navigation but two approaches with optical tracking systems have been published. Hsieh et al. [11] performed a clinical study with 36 patients [11]. For the 18 patients in the

AR Guided PAO

199

navigated group a 3D reconstruction of the patient’s pelvis was generated and displayed on a workstation in the operating room. The patient’s pelvis was intraoperatively registered by surface digitization and the surgical instruments were equipped with infrared reflecting spheres, so that the position of the instruments could be tracked and monitored on the off-field workstation in relation to the bone anatomy. The pelvic osteotomies or the reorientation of the acetabular fragment were neither preoperatively planned nor tracked three-dimensionally. Pfugi et al. [20] performed a study on synthetic bone models and on a cadaver, in which they monitored the reorientation of the acetabular fragment with an optical tracking system on an off-field workstation. The proposed optical tracking systems as well as AR based methods require tracked instruments to be equipped with markers. However, optical tracking systems require an external camera which could cause more occlusion problems than the first-person view of an OST-HMD with a direct line-of-sight. Additionally, these methods offer the possibility of visualizing instruments at their targeted position (e.g. chisels in this study) and thus are not dependent on tracking every single instrument. Furthermore, the optical tracking navigation in the work mentioned above can only be displayed off-field which interferes with a quick and safe surgical procedure. The most important findings of our study were that holographic surgical navigation of the PAO of Ganz is feasible. Particularly, the achieved precision of the osteotomies already meets clinical demands. Regarding the reorientation of the acetabular fragment, we still face relevant problems leading to a clinically not acceptable result. These problems are less related to the navigation procedure than to the preoperative planning. Although it was conducted in accordance to the original description of the procedure by Ganz [7], in our 3D planning the acetabular fragment did not undergo a lateral translation as it would be necessary to not to provoke a conflict between the acetabular fragment and the remaining pelvis. This fact is a limitation of this study and further work is needed to develop an improved 3D preoperative planning method which takes the possibility of a translation of the acetabular fragment into account. Lastly, the proposed method is partly constrained by current technical limitations of the Microsoft HoloLens. Holograms are prone to drift once they have been placed [23]. Despite our motion compensation strategy, this may have negatively influenced our results.

5

Conclusion

The execution of complex osteotomies is feasible with navigation through the Microsoft HoloLens with a satisfying realisation of the preoperative plan. Nevertheless, further work is needed to also improve the navigation of the reorientation of the acetabular fragment.

200

A. Hoch et al.

Acknowledgment. This project is part of SURGENT under the umbrella of University Medicine Zurich/Hochschulmedizin Z¨ urich. Imaging was performed with the support of the Swiss Center for Musculoskeletal Imaging, SCMI, Balgrist Campus AG, Zurich.

References 1. Andress, S., Johnson, A., Unberath, M., Winkler, A.F., Yu, K., Fotouhi, J., Weidert, S., Osgood, G., Navab, N.: On-the-fly augmented reality for orthopedic surgery using a multimodal fiducial. J. Med. Imaging 5(2), 021209 (2018) 2. Besl, P.: A method for registration of 3-D shapes. Robot Tentat 14(2), 239–256 (1992) 3. Bradski, G., Kaehler, A.: OpenCV. Dobb’s J. Softw. Tools 3, 120–125 (2000) 4. Fan Chiang, C.Y., Tsai, T.T., Chen, L.H., Lai, P.L., Fu, T.S., Niu, C.C., Chen, W.J.: Computed tomography-based navigation-assisted pedicle screw insertion for thoracic and lumbar spine fractures. Chang Gung Med. J. 35(4), 332–338 (2012). https://doi.org/10.4103/2319-4170.106137. https://www.ncbi.nlm.nih.gov/pubme d/22913860 5. Fida, B., Cutolo, F., di Franco, G., Ferrari, M., Ferrari, V.: Augmented reality in open surgery. Updates Surg. 70(3), 389–400 (2018) 6. Furnstahl, P., Vlachopoulos, L., Schweizer, A., Fucentese, S.F., Koch, P.P.: Complex osteotomies of tibial plateau malunions using computer-assisted planning and patient-specific surgical guides. J. Orthop. Trauma 29(8), e270–e276 (2015). https://doi.org/10.1097/BOT.0000000000000301. http://www.ncbi.nlm.nih.gov/p ubmed/25932528 nih.gov/pubmed/25932528 7. Ganz, R., Klaue, K., Vinh, T.S., Mast, J.W.: A new periacetabular osteotomy for the treatment of hip dysplasias. Technique and preliminary results. Clin. Orthop. Relat. Res. 232, 26–36 (1988). https://www.ncbi.nlm.nih.gov/pubmed/3383491 8. Garrido-Jurado, S., Munoz-Salinas, R., Madrid-Cuevas, F.J., Medina-Carnicer, R.: Generation of fiducial marker dictionaries using mixed integer linear programming. Pattern Recogn. 51, 481–491 (2016) 9. Gibby, J.T., Swenson, S.A., Cvetko, S., Rao, R., Javan, R.: Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography. Int. J. Comput. Assist. Radiol. Surg. 14(3), 525–535 (2019). https://doi.org/ 10.1007/s11548-018-1814-7. https://www.ncbi.nlm.nih.gov/pubmed/29934792 10. Gsaxner, C., Pepe, A., Wallner, J., Schmalstieg, D., Egger, J.: Markerless imageto-face registration for untethered augmented reality in head and neck surgery. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 236–244. Springer (2019) 11. Hsieh, P.H., Chang, Y.H., Shih, C.H.: Image-guided periacetabular osteotomy: computer-assisted navigation compared with the conventional technique: a randomized study of 36 patients followed for 2 years. Acta Orthop. 77(4), 591–597 (2006). https://doi.org/10.1080/17453670610012656. https://www.ncbi.nlm.nih.gov/pubmed/16929435 12. Jentzsch, T., Vlachopoulos, L., Furnstahl, P., Muller, D.A., Fuchs, B.: Tumor resection at the pelvis using three-dimensional planning and patient-specific instruments: a case series. World J. Surg. Oncol. 14(1), 249 (2016). https://doi.org/10. 1186/s12957-016-1006-2. http://www.ncbi.nlm.nih.gov/pubmed/27729037

AR Guided PAO

201

13. Liebmann, F., Roner, S., von Atzigen, M., Scaramuzza, D., Sutter, R., Snedeker, J., Farshad, M., Furnstahl, P.: Pedicle screw navigation using surface digitization on the Microsoft HoloLens. Int. J. Comput. Assist. Radiol. Surg. 14(7), 1157–1165 (2019). https://doi.org/10.1007/s11548-019-01973-7. ://WOS:000471635000006 14. Ma, L., Zhao, Z., Chen, F., Zhang, B., Fu, L., Liao, H.: Augmented reality surgical navigation with ultrasound-assisted registration for pedicle screw placement: a pilot study. Int. J. Comput. Assist. Radiol. Surg. 12(12), 2205–2215 (2017) 15. Markelj, P., Tomaˇzeviˇc, D., Likar, B., Pernuˇs, F.: A review of 3D/2D registration methods for image-guided interventions. Med. Image Anal. 16(3), 642–661 (2012) 16. Navab, N., Blum, T., Wang, L.J., Okur, A., Wendler, T.: First deployments of augmented reality in operating rooms. Computer 45(7), 48–55 (2012). https:// doi.org/10.1109/Mc.2012.75. ://WOS:000305983400018 17. Nikolaou, V.S., Chytas, D., Malachias, M.A.: Augmented reality in orthopedics: current state and future directions. Front. Surg. 6, 38 (2019) 18. Nottmeier, E.W., Crosby, T.L.: Timing of paired points and surface matching registration in three-dimensional (3D) image-guided spinal surgery. J. Spinal Disord. Tech. 20(4), 268–270 (2007). https://doi.org/10.1097/01.bsd.0000211282.06519. ab. https://www.ncbi.nlm.nih.gov/pubmed/17538349 19. Olson, E.: AprilTag: a robust and flexible visual fiducial system. In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 3400–3407. IEEE (2011) 20. Pflugi, S., Vasireddy, R., Lerch, T., Ecker, T.M., Tannast, M., Boemke, N., Siebenrock, K., Zheng, G.: Augmented marker tracking for periacetabular osteotomy surgery. Int. J. Comput. Assist. Radiol. Surg. 13(2), 291–304 (2018). https://doi.org/10.1007/s11548-017-1690-6. https://www.ncbi.nlm.nih.gov/pubmed/29188423 21. Romero-Ramirez, F.J., Mu˜ noz-Salinas, R., Medina-Carnicer, R.: Speeded up detection of squared fiducial markers. Image Vis. Comput. 76, 38–47 (2018) 22. Roner, S., Vlachopoulos, L., Nagy, L., Schweizer, A., Furnstahl, P.: Accuracy and early clinical outcome of 3-dimensional planned and guided single-cut osteotomies of malunited forearm bones. J. Hand Surg. Am. 42(12), 1031 e1–1031 e8 (2017). https://doi.org/10.1016/j.jhsa.2017.07.002. https://www.ncbi.nlm.nih.gov/pubmed/28888571 23. Vassallo, R., Rankin, A., Chen, E.C., Peters, T.M.: Hologram stability evaluation for Microsoft HoloLens. In: Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, vol. 10136, p. 1013614. International Society for Optics and Photonics (2017) 24. Vlachopoulos, L., Schweizer, A., Meyer, D.C., Gerber, C., Furnstahl, P.: Three-dimensional corrective osteotomies of complex malunited humeral fractures using patient-specific guides. J. Shoulder Elbow Surg. 25(12), 2040–2047 (2016). https://doi.org/10.1016/j.jse.2016.04.038. http://www.ncbi.nlm.nih.gov/pubmed/27503533 25. Wang, J., Olson, E.: AprilTag 2: efficient and robust fiducial detection. In: IROS, pp. 4193–4198 (2016)

Optoacoustic Tissue Classification for Laser Osteotomes Using Mahalanobis Distance-Based Method Hervé Nguendon Kenhagho1(B) , Yakub Aqib Bayhaqi1 , Ferda Canbaz1 , Raphael Guzman2 , Tomas E. Gomez Alvarez-Arenas3 , Philippe C. Cattin4 , and Azhar Zam1 1 Biomedical Laser and Optics Group, Department of Biomedical Engineering,

University of Basel, Gewerbestrasse 14, Allschwil, Switzerland [email protected] 2 Department of Neurosurgery, University Hospital Basel, Spitalstrasse 21, Basel, Switzerland 3 Department of Ultrasonic and Sensors Technologies, Information and Physical Technologies Institute ITEFI, Spanish National Research Council (CSIC), Serrano 144, Madrid, Spain 4 Center for Medical Image Analysis and Navigation, Department of Biomedical Engineering, University of Basel, Gewerbestrasse 14, Allschwil, Switzerland

Abstract. The use of lasers for bone cutting holds many advantages over mechanical tools, including more functional cuts, contactless interaction and faster wound healing. To avoid undesirable tissue damage, a method to classify the tissue being cut is needed. We classified four tissue types—hard and soft bone, muscle and fat from a proximal and a distal fresh porcine femur—by measuring acoustic shock waves generated using an air-coupled transducer during the ablation process. A nanosecond Nd:YAG laser at 532 nm and a microsecond Er:YAG laser at 2940 nm were used to create ten craters on the surface of each tissue type. We performed the Principal Component Analysis (PCA) combined with the Mahalanobis distancebased method to classify tissue types. A set of 2520 data points (or 840 average of three spectra) measured from the first seven craters in one proximal and distal femurs was used as “training data”, while a set of 1080 data points (or 360 average three spectra) measured from last three craters in the remaining proximal and distal femur was considered as “testing data” for both lasers. It was possible to classify each tissue, with an average classification error for all tissues of 7.98% and 36.88%, during laser ablation with the Nd:YAG and Er:YAG, respectively. However using the Er:YAG, it was challenging to classify between soft tissue types. These results show that the Mahalanobis method could be used as feedback for laser osteotomes. Keywords: Laser ablation · Acoustic signal · Principal Component Analysis · Mahalanobis distance

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 202–210, 2021. https://doi.org/10.1007/978-3-030-58104-6_23

Optoacoustic Tissue Classification for Laser Osteotomes

203

1 Introduction Standard surgeries depend on mechanical tools that often lead to mechanical trauma, bacterial contamination and collateral damage to soft tissues [1]. These side effects lead to extended wound healing periods. Therefore, scientists are studying how to overcome the drawbacks of mechanical tools by using laser technology to cut bone [2]. Lasers have emerged and evolved in recent years to have several advantages over standard surgical tools, including precision cutting, sterility, no mechanical trauma, and shorter recovery times [1]. While laser technologies appear to offer a sophisticated solution to the disadvantages of mechanical tool, a control system is needed to prevent the laser from unwanted tissue damage. Thus, the challenge of distinguishing tissue type during ablation becomes a critical focus of research. When ablating tissues with a laser, energy is generated in various forms such as acoustic shock wave (ASW) emission [3]. The ASW generated varies with the laser parameters and more importantly mainly depends on the type of ablated tissue. Furthermore, when water is applied to the ablated spot, energy absorption at a wavelength of 2940 nm is very high compared to that at 532 nm [4, 5]. Thus, in a wet tissue environment, the frequency-doubled neodymium-doped yttrium aluminium garnet (Q-switched Nd:YAG) laser at 532 nm seems well suited to clinical studies for applications such as meniscus/knee surgery. The erbium-doped yttrium aluminum garnet (Er:YAG) laser at 2940 nm could potentially be used to achieve deeper ablation, with a proper cooling system in place. Based on the emitted ASW measured by a transducer, tissues differentiation method can be performed. To decrease the computational time during tissue differentiation, the dimensions of each ASW can be reduced using Principal Component Analysis (PCA) [6]. PCA was used to reduce the complexity of high-dimensional data by finding similarity patterns and trends of the measured ASW [7]. Based on the PCA-scores, we used the Mahalanobis distance measure to classify tissue types, because it can be visualized as the distance of a score from the center of a class and translated onto an ellipse whose main direction is that of the data. The ellipse is plotted based on the scores which have the same Mahalanobis distance from the centroid. In the 3-dimensions, axes of the ellipse are scaled with the same length for scores at equal Mahalanobis distance from the center [6]. In this study, we ablated hard bone, soft bone, fat, and muscle tissues using a nanosecond-Nd:YAG pulsed laser at 532 nm and with a microsecond-Er:YAG pulsed laser at 2940 nm. We measured the emitted ASWs using a high efficiency, broadband piezoelectric air-coupled ultrasonic transducer. To simultaneously differentiate tissue types, we performed the PCA combined with the Mahalanobis distance-based method.

2 Method 2.1 Sample Preparation In the laser tissue ablation experiments, we used one fresh porcine proximal and distal femur purchased from a local butcher (Fig. 1). Scalpels were used to separate the connective tissues. The specimen was rinsed in distilled water prior to the laser experiments. The dimension of compact bone, soft bone, fat and muscle were all 10 × 50 × 5 mm3 .

204

H. Nguendon Kenhagho et al.

Fig. 1. Tissue samples from one fresh porcine femur. Proximal femur: hard bone (a), soft bone (b), muscle (c), and fat (d); distal femur: hard bone (e), soft bone (f), muscle (g), and fat (h).

3 Laser Ablation System Excitation sources for the experiments came from a Q-switched frequency-doubled Nd:YAG laser (Q-smart 850, Quantel, Paris, France) at 532 nm (producing 5 ns pulses) and from an Erbium-doped yttrium aluminum garnet laser (Syneron litetouch LIFG0001A, AOT, Basel, Switzerland) at 2940 nm (producing 400 µs pulses). Several craters were produced ex-vivo on the specimens, using two different lasers (Fig. 2). Each sample was exposed to 180 laser pulses, at a repetition rate of 2 Hz at a single location, for both lasers. This procedure was repeated at ten different ablation locations, spaced 4 mm apart.

Fig. 2. Schematic of the experimental set-up for laser-induced acoustic shock wave measurement using (a) the nanosecond Q-switched Nd:YAG laser and (b) the microsecond Er:YAG laser during tissue ablation.

The applied pulse energy of the Nd:YAG laser and the Er:YAG laser were 200 mJ and 940 mJ, respectively. Figure 2a shows the ns-frequency-doubled Nd:YAG pulse laser and a data recording element embedded in a single opto-acoustic system for tissue ablation and ASW measurement. An NB1-K13 mirror (Nd:YAG Laser Line Mirrors) placed in the beam path of the Nd:YAG laser (Fig. 2a) was used to reflect the laser pulse at a 90º angle. A CaF2 bi-convex lens with a focal length of 30 mm served to focus the laser beam onto the surface of the target specimen. Fig. 2b shows the microsecond (µs) Er:YAG laser with a pulse energy of 940 mJ and operating at a wavelength of 2940 nm, which was used for tissue ablation. A CaF2 window was placed in front of the laser head to split the incident laser beam into two parts—transmitted and reflected light—to allow for a triggering signal. A 30 mm lens was used to focus the transmitted

Optoacoustic Tissue Classification for Laser Osteotomes

205

light onto the surface of the target specimen. The reflected light was collected by a fast PbSe photodiode (PbSe Fixed Gain Detector, PDA20H, 1.5–4.8 µm, Thorlabs, Munich, Germany).

4 Detection and Collection of the Acoustic Shock Wave Signal Acoustic shock waves (ASWs) emitted from the ablation spot were detected with a self-developed, custom-made air-coupled piezoelectric transducer (manufactured in the ITEFI-Instituto de Technologias Fisicas y de la Information, CSIC, Madrid, Spain), with a resonance frequency of 0.4 MHz, usable frequency band of 0.1–0.8 MHz and 15 mm aperture. The transducer was placed at 45º angle and 5 cm away from the ablated spot while recording the laser-induced acoustic wave during the ablation process. The temporal profile of ASW signals detected by the transducer were amplified by 30 dB and digitized by a PCI Express Data Acquisition Card (16-bit resolution, 4 channels at 10 MS/s each, M4i.44xx-x8, Spectrum Microelectronic GmbH, Grosshansdorf, Germany). The Data were collected using LabVIEW (version 2016a), and the information was extracted from the samples using MATLAB (version R2018b) software. When using the Nd:YAG laser, the laser’s CMOS (Complementary metal–oxide–semiconductor) trigger provided the triggering signal for the transducer and data acquisition. For the Er:YAG laser, 4% pulse energy (reflected from the surface of the CaF2 window, Fig. 2b) was used as the triggering signal. For both lasers, data recording of the temporal profile for ASW signals took place during time windows of 0.82 ms.

5 Signal Processing and Classification Method A summary of the signal processing and classification methods for tissue classification is given in Fig. 3. Statistical analysis and calculations were performed in MATLAB software. We suppressed the phase shift of the measured ASW signal by converting the time domain signal into the frequency domain, using the Fast Fourier Transform (FFT). The average of three ASW spectra was calculated to improve the signal-to-noise ratio between each measured ASW in the time domain. We split the amplitude spectrum into three equal frequency bands (Low-, Mid- and High-Frequency). Each frequency band was used as an input for PCA. Here, we identified the frequency band which produce the lowest classification error for tissue classification.

Signal Acquisition

Dimensionality Reduction (PCA)

Fourier Transform

Bandpass Filter

Mahalanobis Distance

Classification Error

Fig. 3. Summary of the signal processing methods for tissue classification.

206

H. Nguendon Kenhagho et al.

For tissue differentiation, we performed the PCA combined with a Mahalanobis distance-based method. We used the first three PCA scores from the set of “training data”, combined with the 95% confidence of each ellipsoid and plotted against three orthogonal eigenvectors from the training data scores—we used three-PC scores with the most variance of the “training data” —. Testing scores lying within and close to the plotted ellipsoid were considered true positives and false positives otherwise. We differentiated tissues based on the ellipsoid, where the covariance of the ASW score and the scales of the different variables were considered. Therefore, ellipsoid is useful for detecting members of the same group and even outliers. A set of 2520 data points (or 840 average of three spectra) measured from the first seven craters in one proximal and one distal femur was used as “training data”, while a set of 1080 data points (or 360 average three spectra) measured from last three craters in the remaining proximal and distal femur was considered as “testing data” for both lasers. From the confusion matrix, errors in “testing data”-based scores from each sample were calculated.

6 Results and Discussion Comparing the ASW of hard bone to surrounding tissues, we found that the peak-topeak amplitudes of the ASW generated by the hard bone specimen and measured by the air-coupled transducer were higher than those generated by the surrounding/soft tissues (Figs. 4a and 4c). Soft tissues contain around 79% water, while hard bone is made up of 85–95% carbonated hydroxyapatite [8]. Thus, we believe that the carbonated hydroxyapatite resulted in a higher amplitude of sound due to its compact structure [9]. The peak-to-peak value of the ASWs generated for each tissue with the Nd:YAG laser was ~7 times higher than those generated with the Er:YAG laser. The greater peak-to-peak pressure generated by the Nd:YAG laser was expected, as the ablation is based on plasma mediation, which increases the pressure energy measured by the transducer. The Er:YAG laser depends on thermal ablation, thus, most of the light energy is absorbed by the exposed tissue and transformed into heat [10]. This is also the reason the acoustic signal duration (wt) generated by the Nd:YAG laser (wt = 1 ms) was longer compared to the one generated by the Er:YAG laser (wt = 0.7 ms). The corresponding frequency domain of the acoustic waves for each tissue is depicted in Figs. 4b and d. The amplitude spectrum of hard bone is higher compared to the surrounding tissues. By splitting the spectrum into three, equal bands, as in Figs. 4b and 4d—the mid-frequency was found to be between 0.27–0.53 MHz—classification between hard bone and surrounding tissues was more accurate than in other bands. Midfrequency region overlaps with the frequency band in which the transducer has higher sensitivity. The features that were chosen for PCA show 94.74% and 90.36% of the total variance in the acoustic waves generated with the Nd:YAG and the Er:YAG laser, respectively. The first three principal components PC1, PC2, and PC3 demonstrated most of the variation in ASWs. Based on the percentage of total variance, PC1, PC2 and PC3 were used to differentiate tissue types based on the analysis of the measured acoustic shock waves (Fig. 4 and 5).

Optoacoustic Tissue Classification for Laser Osteotomes

207

Fig. 4. Measured acoustic shock wave: Time domain of the ASW signal using (a) Nd:YAG laser and (c) Er:YAG laser; frequency domain of the ASW signal using (b) Nd:YAG laser and (d) Er:YAG laser.

During tissue classification, it was possible to differentiate each tissue with an average classification error for all tissues of 7.98% and 36.88%, during laser ablation with the Nd:YAG and Er:YAG, respectively (Table 3). The average classification error for all tissues was higher during Er:YAG cutting than during Nd:YAG cutting. Because, during Er:YAG cutting, it was challenging to differentiate between soft tissues (classification errors in the confusion matrices are in Table 1 and 2); a part of the ellipsoids for soft tissues overlapped (Fig. 5c and d). Furthermore, when cutting tissues with the Nd:YAG, we did not obtain a single wrong classification with this method, we only found unknown tissue types. This is way better than classifying some tissues as the wrong type, as if the result is unknown, the user must manually check what tissue is being cut. Depending on the ablation process, we believe that important parameters of the ablated material can be found in the emitted ASWs. It is possible that, the classification error during Er:YAG cutting was worse because the ablation process is thermal ablation, where most power of the emitted light of the pulsed laser is absorbed in the exposed material and light is transformed into heat. Thus, only a fraction of light energy contributes to the emitted acoustic waves [3]. This is probably the reason the Mahalanobis method could not properly classify them into the relevant tissue types. It is not the case for the Nd:YAG, where most photons are absorbed by

208

H. Nguendon Kenhagho et al.

Fig. 5. Ellipsoids based on 840 scores from “training data” (a and c; using ns-Nd:YAG laser and µs-Er:YAG laser, respectively) and classification of 360 scores from the test data (b and d; using ns-Nd:YAG laser and µs-Er:YAG laser, respectively) in each ellipsoid of the “training data” for hard and soft bone, muscle and fat.

the material which created free electrons at the surface of the exposed zone (plasma mediated). When the free electrons returned to equilibrium state, they emitted ASWs [3]. Table 1. Confusion matrix for tissue types during ablation with Nd:YAG laser. Tissue

Classified as Hard bone

Hard bone

Classification error Soft bone

Muscle

Fat

Unknown

332

0

0

0

28

7.7%

Soft bone

0

302

0

0

58

16.1%

Muscle

0

0

358

0

2

0. 6%

Fat

0

0

0

333

27

7.5%

Optoacoustic Tissue Classification for Laser Osteotomes

209

Table 2. Confusion matrix for tissue types during ablation with Er:YAG laser. Tissue

Classified as Hard bone

Hard bone

Classification error Soft bone

Fat

Muscle

Unknown

327

0

0

0

33

9.17%

Soft bone

0

122

96

62

80

66.11%

Muscle

0

31

26

282

21

21.67%

Fat

0

106

178

70

6

50.56%

Table 3. Average classification error of all tissues for both lasers using Mahalanobis distancebased method. Nd:YAG laser

7.98%

Er:YAG laser

36.88%

7 Conclusion Our aim was to simultaneously classify tissue types during laser ablation by measuring the generated ASW with an air-coupled transducer and process the information using the Mahalanobis method. The Mahalanobis method was combined with PCA during classification experiments. In the experiments, we used two different lasers to generate acoustic waves. We found that the ns-Nd:YAG laser generates higher acoustic amplitudes than the µs-Er:YAG laser. It was possible to differentiate each tissue with a classification error for all tissues of 7.98% and 36.88%, during laser ablation with the Nd:YAG and Er:YAG, respectively. Therefore, Mahalanobis distance-based method could be used as optoacoustic feedback for smart laser osteotomes. Future work includes reducing the average classification errors between soft tissues. We will also investigate different methods to classify tissue types such as the Support Vector Machine (SVMs) and Artificial Neural Network (ANN). Acknowledgement. The authors gratefully acknowledge funding of the Werner Siemens Foundation through the MIRACLE (stands for Minimally Invasive Robot-Assisted Computer-guided LaserosteotomE) project.

References 1. Baek, K.W., et al.: A comparative investigation of bone surface after cutting with mechanical tools and Er:YAG laser. Lasers Surg. Med. 47(5), 426–432 (2015). https://doi.org/10.1002/ lsm.22352 2. Nguendon Kenhagho, H., et al.: Characterization of ablated bone and muscle for long-pulsed laser ablation in dry and wet conditions. Materials 12(8), 1338 (2019). https://www.mdpi. com/1996-1944/12/8/1338

210

H. Nguendon Kenhagho et al.

3. Nguendon, H.K., et al.: Characterization of ablated porcine bone and muscle using laserinduced acoustic wave method for tissue differentiation. In: European Conference on Biomedical Optics, p. 104170N. Optical Society of America (2017) 4. Tulea, C.-A., Caron, J., Gehlich, N., Lenenbach, A., Noll, R., Loosen, P.: Laser cutting of bone tissue under bulk water with a pulsed ps-laser at 532 nm. In: SPIE, vol. 20, p. 9 (2015) 5. Kuscer, L., Diaci, J.: Measurements of erbium laser-ablation efficiency in hard dental tissues under different water cooling conditions. In: SPIE, vol. 18, p. 11 (2013) 6. Brereton, R.G.: The Mahalanobis distance and its relationship to principal component scores. J. Chemom. 29(3), 143–145 (2015). https://doi.org/10.1002/cem.2692 7. Lever, J., Krzywinski, M., Altman, N.: Principal component analysis. Nat. Methods 14, 641 (2017). https://doi.org/10.1038/nmeth.4346 8. Curzon, M., Featherstone, J.: Chemical composition of enamel. In: Lazzan, E.P. (ed.) Handbook of Experimental Aspects of Oral Biochemistry, pp. 123–135. CRC Press, Boca Raton (1983) 9. Apel, C., Meister, J., Ioana, R., Franzen, R., Hering, P., Gutknecht, N.: The ablation threshold of Er:YAG and Er:YSGG laser radiation in dental enamel. Lasers Med. Sci. 17(4), 246–252 (2002) 10. Kenhagho, H.N., Rauter, G., Guzman, R., Cattin, P.C., Zam, A.: Comparison of acoustic shock waves generated by micro and nanosecond lasers for a smart laser surgery system. In: SPIE BiOS, SPIE, vol. 10484, p. 9 (2018)

Simulation of Echellogram Using Zemax OpticStudio and Matlab for LIBS Hamed Abbasi1(B) , Negin Sahraei2 , Ferda Canbaz1 , Philippe C. Cattin2 , and Azhar Zam1 1 Biomedical Laser and Optics Group (BLOG), Department of Biomedical Engineering,

University of Basel, 4123 Allschwil, Switzerland [email protected] 2 Center for Medical Image Analysis and Navigation (CIAN), Department of Biomedical Engineering, University of Basel, 4123 Allschwil, Switzerland

Abstract. Echelle spectrometers offer high-resolution (ca. one Angstrom or higher) and wide range (ca. half a micron or wider), simultaneously, without having a mechanical moving part. However, they require a complicated optical set-up as compared to conventional one-dimensional-dispersion/diffraction spectrometers, e.g., Czerny-Turner. Therefore, the simulation of the Echellogram is required before building the spectrometer. This paper aims to develop an application for visualizing the Echellogram using MATLAB App Designer followed by simulation of the optical aberrations by Zemax OpticStudio to optimize the spectrometer parameters. The developed application provides an interactive graphical interface with a user-friendly dashboard to control and monitor the required parameters. This spectrometer is being used in laser-induced breakdown spectroscopy (LIBS) system application lied of robot-guided laser ablation of biological tissues. Keywords: Echelle spectrometer · Simulation · Zemax OpticStudio · Matlab · Diffraction grating

1 Introduction Echelle spectrometers are widely used in observatories [1–3] and Laser-Induced Breakdown Spectroscopy (LIBS) experiments [4–6]. The ones used in observatories are typically up to several meters to have the highest possible resolving power [7–9]. The ones used in LIBS utilize rather small designs to make the system portable (trade-off between size and resolving power) [10, 11]. However, in both cases, the Free Spectral Range (FSR) and optical throughput are required to be wide and high enough, respectively, complicated due to low light intensity (in observatories) [12] and low exposure time (time-resolved measurements in LIBS) [13]. Unlike conventional prism- [14] or grating-based [15] monochromators/spectrometers with one-dimensionaldispersion/diffraction, Echelle spectrometers have dispersion/diffraction in two dimensions, one caused by the Echelle grating and the other caused by the cross-dispenser to © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 211–218, 2021. https://doi.org/10.1007/978-3-030-58104-6_24

212

H. Abbasi et al.

separate the different order of diffractions from each other. Therefore, without employing any mechanical moving part, a large number of pixels in the camera can be used to separate wavelengths from each other, to achieve a high resolving power, without reducing FSR. Although designing an Echelle spectrometer has its challenges, the performance (etendue, resolution, bandwidth, etc.) can still be improved by utilizing an applicationoriented design [16, 17]. Therefore, several toolboxes, based on different platforms, have been developed to simulate the Echellogram (frequency-separated image of the slit in the camera plane) focusing mainly on astronomical application [18–23]. To overcome the design challenges of Echelle spectrometers for LIBS application, a user-friendly simulation approach can foster the design, by reducing the final cost and designing time. In this work, therefore, an application for simulating Echellograms has been developed. This application allows rapid calculation of the properties of an Echelle spectrometer, primarily with LIBS application in mind. This design-aid tool can quickly simulate the images seen by the camera. However, the widening of the slit image in the Echellogram due to optical aberrations is not considered; therefore, the simulation is followed by a secondary simulation by Zemax OpticStudio to consider the effect of induced aberration on its resolution. The final step includes combining these two simulation platforms to develop a user-friendly designing tool for Echelle spectrometer in LIBS systems.

2 Spectrometer Design We previously performed some initial experiments on LIBS measurements of biological tissues [24–26] employing a commercially available Echelle spectrometer. In these experiments, we aimed to find the most important atomic and molecular lines for differentiation of tissue types to be used as feedback for smart laser surgery. Based on these preliminary experimental results, the design requirements were obtained. A bandwidth from 330 to 830 nm and resolution of ca. one Angstrom has been selected for the design with an emphasis on maximizing the optical throughput as compared to the commercially available Echelle spectrometers [6]. Therefore, we set up our spectrometer using off-axis parabolic mirrors with 2-in. diameter to reduce the F-number and consequently increasing the optical throughput. Figure 1 shows a CAD representation of the spectrometer set-up. The input fiber (A) delivers light to the first collimator (B), which reflects the parallel light to the Echelle grating positioned at the Littrow angle at point C. After being reflected back to point B, due to a small off-axis angle induced to the Echelle grating, the light passes slightly above the fiber and goes to the second collimator (D) which reflects the light to the second diffraction grating (E) for separating the Echelle orders (cross-dispenser). Separated orders are imaged to the camera through the imaging lens (F). Note that, since both collimators have the same properties (size, focal length, etc.), the design can also be simulated considering only a single collimator.

Simulation of Echellogram Using Zemax OpticStudio and Matlab for LIBS

213

Fig. 1. CAD representation of the spectrometer set-up (A: input fiber, B: first collimating mirror, C: Echelle grating, D: second collimating mirror, E: cross-dispenser grating, F: imaging lens). The light pass starts from point A, goes to B, and C, comes back to B, passes above point A, goes to D, E, F, and Camera, accordingly. The optical base plate is 90 × 60 cm. The size of the set-up without considering the camera and its lens is 44 (l) × 36 (w) × 13 (h) cm3. The camera should be located outside of the spectrometer in order to not obstruct the camera cooling fan. The CAD representation was designed using SolidWorks 2017.

3 Echellogram Simulator Echellogram simulator application was developed using MATLAB app designer (R2018b) to simulate the grating cross-dispense Echelle spectrometers. The developed application provides an interactive graphical user interface (GUI) with a user-friendly control panel. It asks the user to enter the required 22 design parameters and returns 27 calculated output parameters plus the depicted Echellogram. Figure 2 shows the user interface of the developed application. In the user interface, the required input and output parameters are shown in dark blue and black, respectively. The suggested pixel width/height is given according to the Nyquist criterion to avoid “undersampling” i.e. aliasing [12]. Smaller pixels fulfill the Nyquist–Shannon sampling theorem as well and will offer more points for peak fitting; however, it does not narrow down the FWHM of the peaks. The offered tilt angle for the camera helps to align the pixels with the order rows to be able to do binning (to increase the frame rate). The suggested input slit width/height shows the counterproductive limit, i.e., smaller slits would not result in improved resolution but only reduce the intensity of the entered light [27]. To eval-

214

H. Abbasi et al.

uate the performance of the developed application, the results for modified OPTIMA (commercially available Echelle spectrometer) simulation have been compared by the one simulated with “SimEchelle” [12]. SimEchelle is an excel-based simulator without a GUI. The results from both simulators were in agreement. The developed application provides some additional useful information (including requirements for a matching camera sensor and its placement angle) for designing an Echelle spectrometer as compared to similar previously developed software. The input numbers shown in the application interface after starting the application (default numbers) are those from the modified OPTIMA simulation, as shown in Fig. 2.

Fig. 2. The user interface of the developed application. The required input parameters are entered in the dark blue boxes, knobs, and sliders. The output parameters are shown in black; the simulated Echellogram is depicted once the plot button is pressed. Positions on the Echellogram are in reference to the diffracted light at Littrow configuration for both gratings. The black circles on the sides of each order show the overlapping border.

Besides the inserted figure in the application, some additional pop-up figures additionally are shown by the application, e.g., diffraction orders vs. diffraction angle/wavelength (with or without spatial filtering). Figure 3 shows the pop-up figures for the spectrometer designed to cover 330 to 830 nm for the analysis of biological samples by LIBS.

Simulation of Echellogram Using Zemax OpticStudio and Matlab for LIBS

215

Fig. 3. Auxiliary pop-up figures created by the developed application. Part (a) shows all possible diffraction orders without spatial filtering. Part (b) shows all diffractable wavelengths at each diffraction order. Part (c) is the same as (a) with spatial filtering. Part (d) is the Echellogram. By holding the mouse cursor on each point, its properties will be shown to the user.

4 Zemax OpticStudio Simulation The Echellogram simulator application showed the position of the exit slit on the camera plane, which can be used to define the required camera properties for the desired FSR based on the chosen grating and lenses/mirrors. However, induced optical aberrations were not considered. Therefore, an additional simulation step is required to show the effect of aberrations on the resolution of the system. Optical aberrations can reduce the resolution of the system and potentially cause crosstalk between the orders. Zemax OpticStudio is the most commonly used software for spectrometer aberration simulation [28, 29]. Figure 4 shows the simulation of the designed spectrometer by Zemax OpticStudio for the exemplary sodium (Na) doublets. The sodium doublets are the closest peaks we expect important for differentiating tissues [24]. A clear separation can be observed in the spot diagram for the chosen parameters.

216

H. Abbasi et al.

Fig. 4. Simulation of the designed spectrometer using Zemax OpticStudio 16.5 standard edition. The spot diagram shows the separation of doublet of sodium (Na) with 5.97 Angstrom distance (from 5889.95 to 5895.92 Angstrom). An astigmatism aberration is visible in the spot diagram; however, it did not reduce the required resolution.

5 Conclusions A simulator application for quick calculation of Echellograms (image seen by the camera) properties has been developed. According to user input parameters such as groove density, placement angles of gratings, focal length, and sizes of lenses/mirrors, the Echellogram simulator application returns the requirements for a matching camera sensor, including the size and number of required pixels as well as camera tilt angle. Secondary, simulation has also been conducted to determine that the induced aberrations do not result in insufficient resolution. The result of both simulations showed that the designed spectrometer with high optical throughput had the potential to be used in the desired spectral range with enough resolving power based on the previous experimental data for differentiating tissues using LIBS. Future work might be combining both simulator platforms into a single software application. Acknowledgments. The authors gratefully acknowledge funding of the Werner Siemens Foundation through the Minimally Invasive Robot-Assisted Computer-guided LaserosteotomE (MIRACLE) project. Moreover, the authors acknowledge the assistance given by Mr. Arsham Hamidi and Mr. Yakub Bayhaqi.

Simulation of Echellogram Using Zemax OpticStudio and Matlab for LIBS

217

References 1. Telting, J.H., Avila, G., Buchhave, L., Frandsen, S., Gandolfi, D., Lindberg, B., Stempels, H.C., Staff, T.N., Prins, S.: FIES: the high-resolution Fiber-fed Echelle Spectrograph at the Nordic Optical Telescope. Astronomische Nachrichten 335, 41–45 (2014) 2. Pepe, F., Cristiani, S., Rebolo Lopez, R., Santos, N., Amorim, A., et al.: ESPRESSO: the Echelle spectrograph for rocky exoplanets and stable spectroscopic observations. In: SPIE Astronomical Telescopes + Instrumentation (SPIE, 2010), vol. 7735 (2010) 3. Tala, M., Heeren, P., Grill, M., Harris, R., Stürmer, J., Schwab, C., et al.: A high-resolution spectrograph for the 72 cm Waltz telescope at Landessternwarte, Heidelberg. In: SPIE Astronomical Telescopes + Instrumentation (SPIE, 2016), vol. 9908 (2016) 4. Sabsabi, M., Detalle, V., Harith, M.A., Tawfik, W., Imam, H.: Comparative study of two new commercial echelle spectrometers equipped with intensified CCD for analysis of laser-induced breakdown spectroscopy. Appl. Opt. 42, 6094–6098 (2003) 5. Fichet, P., Menut, D., Brennetot, R., Vors, E., Rivoallan, A.: Analysis by laser-induced breakdown spectroscopy of complex solids, liquids, and powders with an echelle spectrometer. Appl. Opt. 42, 6029–6035 (2003) 6. Abbasi, H., Rauter, G., Guzman, R., Cattin, P.C., Zam, A.: Design and implementation of a compact high-throughput echelle spectrometer using off-the-shelf off-axis parabolic mirrors for analysis of biological samples by LIBS (Conference Presentation). In: SPIE Optical Metrology (SPIE, 2019), vol. 11060 (2019) 7. Schwab, C., Rakich, A., Gong, Q., Mahadevan, S., Halverson, S., Roy, A., Terrien, R., et al.: Design of NEID, an extreme precision Doppler spectrograph for WIYN, SPIE Astronomical Telescopes + Instrumentation (SPIE, 2016), vol. 9908 (2016) 8. Lentini, N., Kuo Tiong, B., DuBois, L., Feger, T., Coutts, D., Fallscheer, C., Schwab, C.: A high resolution echelle spectrograph for exoplanet searches with small aperture telescopes. In: SPIE Astronomical Telescopes + Instrumentation (SPIE, 2018), vol. 10702 (2018) 9. Chakraborty, A., Mahadevan, S., Roy, A., Dixit, V., Harvey Richardson, E., Dongre, V., et al.: The PRL stabilized high-resolution echelle fiber-fed spectrograph: instrument description and first radial velocity results. Publications of the Astronomical Society of the Pacific 126, 133–147 (2014) 10. Fu, X., Duan, F., Jiang, J., Huang, T., Ma, L., Lv, C.: Astigmatism-corrected echelle spectrometer using an off-the-shelf cylindrical lens. Appl. Opt. 56, 7861–7868 (2017) 11. Mohamed, W.T.Y.: Study of the matrix effect on the plasma characterization of six elements in aluminum alloys using LIBS with a portable Echelle spectrometer. Progress in physics 2, 42–49 (2007) 12. Eversberg, T., Vollmann, K.: Spectroscopic Instrumentation, Springer, Heidelberg (2015) 13. Abbasi, H., Rauter, G., Guzman, R., Cattin, P.C., Zam, A.: Plasma plume expansion dynamics in nanosecond Nd:YAG laserosteotome. In: SPIE BiOS (SPIE, 2018), vol. 10505 14. Abbasi, H., Nazeri, M., Mireei, S.A.: Design and development of a LabVIEW-based LED-induced fluorescence spectroscopy system with applications in non-destructive quality assessment of agricultural products. J. Phys: Conf. Ser. 672, 012010 (2016) 15. Nazeri, M., Majd, A.E., Massudi, R., Tavassoli, S.H., Mesbahinia, A., Abbasi, H.: Laserinduced breakdown spectroscopy via the spatially resolved technique using non-gated detector. J. Russian Laser Res. 37(2), 164–171 (2016). https://doi.org/10.1007/s10946-0169556-x 16. Probst, R.A., Steinmetz, T., Wu, Y., Grupp, F., Udem, T., Holzwarth, R.: A compact echelle spectrograph for characterization of astro-combs. Appl. Phys. B 123(3), 1–8 (2017). https:// doi.org/10.1007/s00340-016-6628-0

218

H. Abbasi et al.

17. Bykov, S.V., Sharma, B., Asher, S.A.: High-throughput, high-resolution echelle deep-UV Raman Spectrometer. Appl. Spectroscopy 67, 873–883 (2013) 18. Dantzler, A.A.: Echelle spectrograph software design aid. Appl. Opt. 24, 4504–4508 (1985) 19. Barnes, S.: The design and performance of high resolution échelle spectrographs in astronomy (2004) 20. Barnes, S.: EchMod: a MATLAB toolbox for modeling high resolution échelle spectrographs. In: SPIE Optical Systems Design (SPIE, 2012), vol. 8550 (2012) 21. Eversberg, T., Vollmann, K.: “Considerations for Designing an Echelle Spectrometer,” in Spectroscopic Instrumentation: Fundamentals and Guidelines for Astronomers, pp. 229–279. Springer, Heidelberg (2015) 22. Stürmer, J., Seifahrt, A., Robertson, Z., Schwab, C., Bean, J.L.: Echelle ++, a fast generic spectrum simulator. Publ. Astron. Soc. Pac. 131, 024502 (2018) 23. Sablowski, D.P., Vollmann, K.: SILENT: Spectrograph simulation and emulation tool. New Astron. 75, 101303 (2020) 24. Abbasi, H., Rauter, G., Guzman, R., Cattin, P.C., Zam, A.: Differentiation of femur bone from surrounding soft tissue using laser-induced breakdown spectroscopy as a feedback system for smart laserosteotomy. In: SPIE Photonics Europe (SPIE, 2018), vol. 10685 (2018) 25. Abbasi, H., Rauter, G., Guzman, R., Cattin, P.C., Zam, A.: Laser-induced breakdown spectroscopy as a potential tool for autocarbonization detection in laserosteotomy. J. Biomed. Optics 23(1–7), 7 (2018) 26. Abbasi, H., Sugiarto, I., Rauter, G., Guzman, R., Cattin, P.C., Zam, A.: Pilot ex vivo study of laser-induced breakdown spectroscopy to detect bone dehydration: an approach for irrigation feedback in laserosteotomy. In: International Conference on Electrical Engineering & Computer Science (ICEECS 2018), (Bali, Indonesia) (2018) 27. Demtröder, W.: Laser Spectroscopy 1: Basic Principles, Springer, Heidelberg (2014) 28. Bagusat, V., Kraus, M., Förster, E., Thomae, D., Hönle, T., Brüning, R., Hillmer, H., Brunner, R.: Concept and optical design of a compact cross-grating spectrometer. J. Opt. Soc. Am. A 36, 345–352 (2019) 29. Scheeline, A.: How to design a spectrometer. Appl. Spectroscopy 71, 2237–2252 (2017)

Robot- and Laser-Assisted Bio-Sample Preparation: Development of an Integrated, Intuitive System C´edric Duverney1(B) , Hamed Abbasi2 , Lina M. Beltr´ an Bernal2 , Tino Stauber3 , Jess G. Snedeker3 , Philippe C. Cattin4 , Azhar Zam2 , and Georg Rauter1 1

3 4

BIROMED-Lab, Department of Biomedical Engineering, University of Basel, Allschwil, Switzerland [email protected] 2 BLOG, Department of Biomedical Engineering, University of Basel, Allschwil, Switzerland Laboratory for Orthopaedic Biomechanics, ETH Z¨ urich, Z¨ urich, Switzerland CIAN, Department of Biomedical Engineering, University of Basel, Allschwil, Switzerland https://biromed.dbe.unibas.ch

Abstract. The preparation of small-sized biological samples is traditionally performed manually utilizing mechanical tools such as scalpels. The main drawbacks of such methods are a lack of accuracy and repeatability of the resulting cuts and damage to the surrounding tissue due to the high interaction forces and the accompanying mechanical stresses. One way to circumvent these issues is to substitute the mechanical tools for laser light. When used in conjunction with a high-accuracy positioning system, such a preparation procedure enables repeatable cutting of arbitrary geometries while largely preserving the integrity of the surrounding tissue. In this paper, a system leveraging the potential of laser-based ablation for bio-sample preparation is proposed. It integrates and synchronizes all key components with extensive safety features and an intuitive user interface, allowing novice operators to perform sample preparations easily. As a first application, the proposed system has been utilized to create microdamages in mouse tail tendon fascicles. Promising results could be obtained, but careful tuning of the laser parameters and further optimization of the mechanical setup is still required to attain the high repeatability striven for. Keywords: Sample preparation

1

· Laser ablation · Micropositioning

Introduction

Manual preparation of small-sized biological samples for further processing or experimentation is commonly done using mechanical tools such as scalpels. This c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 219–226, 2021. https://doi.org/10.1007/978-3-030-58104-6_25

220

C. Duverney et al.

for evident reasons, as the tools are affordable, readily available, no specific training is required in handling them, and the process is generally comparably fast. Furthermore, as they do not require large installations, the preparation process is highly flexible, meaning that various types and geometries of tissues can be prepared using similar tools, without time-consuming modifications. These benefits, however, come at the expense of relatively low cutting accuracy, poor repeatability, and potential damage to the tissue portions surrounding the cutting areas due to the occurring mechanical stresses. An interesting alternative is to utilize laser light instead of mechanical tools for performing the cut, as done increasingly frequently in recent biomedical applications [3]. Substantial advantages of doing so have been demonstrated especially in terms of low surrounding tissue damage and superior cut quality [2]. The main challenge in laser-based preparation of bio-samples is a reliable control of the ablation process. It has been shown that the tissue optics, as well as the pulse duration of the laser, have a significant impact on the ablation results [5]. These parameters, in conjunction with wavelength and irradiance of the laser, need to be carefully matched to obtain a type of laser-tissue interaction feasible for a particular application and producing satisfactory results [4,6]. Further critical engineering challenges are an accurate, automated sample positioning ensuring accurate and repeatable cuts, an intuitive user interface, comprehensive safety features, and real-time synchronization of all components. The objective of this project is to develop a system for laser-based bio-sample preparation combining versatility and ease of use. It should offer the possibility to ablate various types of tissues and to generate arbitrary geometries, all while minimizing the setup and control efforts required by the user. This would enable non-technical personnel to easily perform preparations required for their experiments without large technical overhead, leaving more time for the scientific work. These benefits shall be demonstrated in a first exemplary application.

2

Methods

A system for laser-based preparation of bio-samples has been designed. It integrates all components required for accurate positioning and controlled ablation of samples and provides an intuitive graphical user interface (GUI) to control and monitor the preparation process in real-time. 2.1

Hardware Design

In terms of hardware, two main component groups can be identified: One associated with sample positioning, and one with sample ablation. These two groups are interfaced with a control system that assures their precise synchronization. The requirements imposed on the positioning group are a high accuracy (5 cm in all three dimensions). For the ablation group, the requirements are the ability to ablate hard- and soft-tissue in

Robot- and Laser-Assisted Bio-Sample Preparation

221

a controlled way, without damaging or carbonizing the surrounding tissue. Furthermore, as living samples generally need to be kept inside a nutrient solution at all times to maintain their cells, the ablation process requires to function within liquids as well. Figure 1 shows a computer aided design (CAD) rendering and a photograph of the proposed hardware design. The key components marked therein shall now be detailed and related to the aforementioned system requirements:

Fig. 1. CAD rendering (optics simplified) and photograph of the system, showing all key components: 1 robotic stage, 2 Nd:YAG pulsed laser, 3 Er:YAG pulsed laser, 4 Nd:YAG CW laser, 5 photodetector, 6 camera, 7 user interface, and 8 bio-sample holder. The colors of the underlines match the corresponding components’ optical paths as shown in the CAD rendering.

1 The robotic stage used for positioning the bio-samples relies on a custom design based on a microscope stage (H101A, Prior Scientific Instruments Ltd., Fulbourn, United Kingdom), a laboratory jack (L490/M, Thorlabs Inc., Newton, United States), and low-voltage servomotors (AM8111, Beckhoff Automation GmbH, Verl, Germany). These components provide high rigidity, low backlash, and an accuracy and workspace size satisfying the given requirements. 2 The Nd:YAG pulsed laser (Q-smart 450, Quantel Technologies, Les Ulis, France) is operating at a wavelength of either 532 nm or 1064 nm, a pulse

222

3

4

5

6

7 8

C. Duverney et al.

duration of 5 ns, an adjustable energy between 2 mJ and 170 mJ, and is used for electro-mechanical ablation. As it is not being absorbed by water, it may be used to prepare living bio-samples that need to remain inside nutrient solutions at all times. The Er:YAG pulsed laser (LiteTouch, Syneron Medical Ltd., San Francisco, United States) is operating at a wavelength of 2940 nm, a pulse duration in the microseconds regime, a maximum energy of 1000 mJ, and is used for photo-ablation. It is largely absorbed by water, and can hence not be used on samples within aqueous solutions. The Nd:YAG CW laser (LSR532NL, Lilly Electronics Ltd., Wuhan, China) is operating at a wavelength of 532 nm, a maximum power of 500 mW, and ablates the samples due to photothermal interaction effects. In order to generate single pulses, as needed for a precisely controlled ablation process, a rotating shutter is periodically obstructing its optical path. By adjusting the shutter’s rotational speed, the pulse duration can be set to any value between 144 µs and infinity. The photodetector (PDA20H, Thorlabs Inc., Newton, United States) registers shots of the pulsed Er:YAG laser. It subsequently sends a trigger signal to the control cabinet, which will, in turn, trigger a shot of the Nd:YAG pulsed laser, enabling precise synchronization of the two lasers. The camera (LifeCam Studio, Microsoft Corporation, Redmond, United States) enables real-time monitoring of the sample preparation. Its optical axis is coaxial to the three ablation lasers. The user interface enables easy control of the sample preparation by mouse and keyboard. A dedicated holder carries the bio-sample and is directly fixed to the stage.

The two hardware component groups have been interfaced with a custombuilt control cabinet running an automation software (TwinCAT 3, Beckhoff Automation GmbH, Verl, Germany). The control cabinet ensures precise synchronization of all components controlled by the GUI. It further integrates all safety features, ensuring in particular that in case of an emergency, all laser paths are immediately physically blocked, and all power supplied to the laser sources and motors cut. 2.2

Software Design

On the software side, a custom GUI enables quick, intuitive access to all relevant system functions for non-technical personnel. A live screenshot of the proposed layout is pictured in Fig. 2. The GUI is subdivided into four main sections: (i) camera feed, (ii) general application control, (iii) laser ablation control, and (iv) stage positioning control; the latter two are further discussed here. The laser ablation control portion of the GUI offers three modes, accessible through the three tabs at its top. In the first mode, only the pulsed Nd:YAG laser is used for ablation. Its repetition frequency and energy can be altered in real-time. In the second mode, the pulsed Nd:YAG laser is used in conjunction

Robot- and Laser-Assisted Bio-Sample Preparation

223

with the pulsed Er:YAG laser. For the pulsed Er:YAG laser, both its repetition frequency and energy need to be set via its own external control terminal. For the pulsed Nd:YAG laser, only the energy may be altered, as its repetition frequency is being synchronized with the Er:YAG laser pulses via the trigger signal provided by the photodetector. In the third mode, only the Nd:YAG CW laser is used. Its repetition frequency can be altered in real-time by adjusting the angular speed of its rotational shutter, while its power is altered via its external control terminal. The stage positioning control part of the GUI provides three modes as well, again accessible through the three tabs at the top, side by side with the measured real-time stage position. While the first mode has been fully implemented, tested, and used during ablation experiments, the second and third modes remain in the late development stage at the time of writing of this article. The first mode can be utilized for purely manual control of the stage. Both, continuous motion with a selectable speed as well as stepwise motion with a selectable step size, are possible. The second mode allows the operator to store the current stage position in persistent memory, and to later have the stage autonomously return to that exact position. This feature may for instance be used to define fixed positions for sample loading and initial sample positioning. The third mode moves the stage along a predefined trajectory, which the operator may supply to the system in the form of an appropriately formatted text file. This text file may also include instructions for the lasers, allowing pulses to be delivered at specific points of the trajectory. This feature enables accurate, repeatable laser-cutting of complex shapes into the bio-samples.

Fig. 2. Screenshot of the live GUI. The camera feed is displaying a mouse tail tendon fascicle (∅100 µm) placed in the sample holder.

224

2.3

C. Duverney et al.

Exemplary Application

A first application serving as a demonstration of the proposed system’s benefits is related to the investigation of the intrinsic repair capabilities of tendon tissue [7]. A recent study has shown that the load-bearing tendon compartment is unable to repair stretch-induced microdamage [8]. In these trials, tendon explant models [9] were manually damaged by applying a single-stretch loading protocol and then monitored over several days. Unfortunately, this procedure distributes tissue microdamage randomly over the whole explant and therefore minimizes the available pool of repair-competent cells. The proposed laser-based ablation procedure on the other hand allows for tightly controllable and repeatable generation of locally restricted microdamage while only marginally affecting the resident cell population. The basic feasibility of this procedure has been demonstrated using dead, dry tendon tissue. For this simplified setting, sterility did not need to be maintained during the entire procedure. The tendon was placed in a dedicated sample holder fixed to the robotic stage. Next, the stage position was manually adjusted until the laser’s focal spot coincided with the tendon’s centerline. The tendon was then exposed to a single shot of the pulsed Nd:YAG laser with an energy of approximately 8 mJ. The process was monitored using the system’s integrated camera, and pictures of the tendon before and after ablation were recorded with a digital microscope (DVM6, Leica Microsystems, Wetzlar, Germany) for qualitative analysis of the generated microdamage.

3

Results

Figure 3 shows pictures of the mouse tail tendon fascicle before and after ablation following the procedure described above. As expected, the laser pulse created a microdamage in the tendon. It has been observed that a tendon in this damaged state will break apart after a few additional laser shots; often already the second shot was sufficient for complete rupture. Further experiments have shown the wetness of the tendons to have a noticeable impact on the outcome of the ablation process. Dry tendons could generally

Fig. 3. Wet mouse tail tendon fascicles (∅100 µm, magnification factor 600) before (left column) and after (right column) ablation with a single laser pulse, as observed with a digital microscope (top row) and the camera integrated in the setup (bottom row).

Robot- and Laser-Assisted Bio-Sample Preparation

225

endure at least five laser pulses before rupturing. This is significantly more than the one to two pulses for wet tendons.

4

Discussion

In the first experiments, it could be shown that the proposed robot- and laserassisted bio-sample preparation approach is feasible and functional. However, not all aspects of the system are completely satisfying yet. In particular, positioning the sample correctly with respect to the laser is difficult due to poor image quality of the currently used camera. This camera also remains to be the only component that is not well synchronized to the system. Obtaining synchronized camera data would allow capturing frames at a specified time interval after the laser pulses are delivered. In this way, the camera would record less noise originating from the laser light reaching the image sensor. Currently, vertical positioning errors can also be observed. These errors lead to bigger holes than desired, as the laser’s focal plane does not precisely coincide with the tendon’s centerline. Therefore, the laser beam diameter at the height of the tendon is bigger than the desired beam size found at the focal spot. Presently, the focal plane is determined by horizontally moving the focal spot away from the sample and delivering laser pulses to generate plasma in the air. The sample is then carefully moved vertically until it lies at the same height as the plasma, which corresponds to the height of the focal plane. Finally, the sample is positioned horizontally to begin with ablation. This adjustment procedure is prone to inaccuracies, as the air plasma has a diameter of around 1 mm and a slightly different shape at every pulse [1], rendering the visual alignment with the tendon’s centerline difficult. To overcome current focal spot alignment issues, new calibration procedures are being developed. It should be noted that even in the case of perfect alignment of the laser’s focal spot with the tendon’s centerline, it will not be possible to obtain a clean hole with the same diameter as the focal spot (roughly 4 µm). This is due to the fact that the plasma generated by the laser around the focal spot will always be larger than the focal spot itself, and hence an accordingly larger portion of tissue will be ablated. The ripped hole borders clearly visible in Fig. 3 will remain as well, as these stem from the specific material response of this particular type of tissue to the laser. As tendons are mainly constituted of tiny individual collagen fibers, some of these fibers are completely cut during the ablation process and start to fray, while other fibers remain intact, leading to the fuzzy borders observed. The effect of the tendon wetness on the result of the ablation is related to differences in size of the generated plasma for the two cases. If the tissue still contains a certain amount of water, a larger plasma will be generated, leading to a larger damage to the tendon. For dry tissue, the opposite is the case. Repeatable ablation results, as needed to assess the tissue regeneration properties reliably, are hence difficult to achieve with the current setup, as tissue wetness is not being precisely controlled. Since living tissue will be used in future experiments, sterile conditions will furthermore need to be guaranteed during the entire procedure, which again is currently not possible. An extension of the system addressing these issues is being developed.

226

5

C. Duverney et al.

Conclusion

The benefits and functionality of the proposed preparation system have been demonstrated in first experiments. It has been shown that it is feasible to create microdamages in soft-tissue for the case of mouse tail tendon fascicles. However, the resulting microdamages were not as small as anticipated, which is due to inaccuracies in sample positioning. Furthermore, no clean and sharp borders could be realized by reason of the specific material response to the laser pulses. The next steps will hence involve the integration of a better imaging system, and the elaboration of a more accurate method for aligning the laser’s focal plane with the tendon’s centerline. Later, the above-mentioned system extension controlling tissue wetness and sterility shall be designed, such that further trials with dead and eventually living mouse tail tendon fascicles can be conducted. Acknowledgments. The authors gratefully acknowledge funding by the Werner Siemens Foundation through the MIRACLE project. They would further like to thank Sascha Martin and team, Department of Physics, University of Basel, Switzerland, for their invaluable support during the mechanical design and manufacturing phases.

References 1. Abbasi, H., Rauter, G., Guzman, R., Cattin, P.C., Zam, A.: Plasma plume expansion dynamics in nanosecond Nd:YAG laserosteotome. In: SPIE 10505, HighSpeed Biomedical Imaging and Spectroscopy III (2018). https://doi.org/10.1117/ 12.2290980 2. Baek, K.W., et al.: A comparative investigation of bone surface after cutting with mechanical tools and Er:YAG laser. Lasers Surg. Med. 47(5), 426–432 (2015). https://doi.org/10.1002/lsm.22352 3. Beltr´ an Bernal, L.M., Abbasi, H., Zam, A.: Laser in bone surgery. In: St¨ ubinger, S., Kl¨ ampfl, F., Schmidt, M., Zeilhofer, H.F. (eds.) Lasers in Oral and Maxillofacial Surgery. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-29604-9 9 4. Boulnois, J.L.: Photophysical processes in recent medical laser developments: a review. Lasers Med. Sci. 1, 47–66 (1986). https://doi.org/10.1007/BF02030737 5. Jacques, S.L.: Role of tissue optics and pulse duration on tissue effects during highpower laser irridation. Appl. Opt. 32, 2447–2454 (1993). https://doi.org/10.1364/ AO.32.002447 6. Niemz, M.H.: Laser-Tissue Interactions, 3rd edn. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-030-11917-1 7. Snedeker, J.G., Foolen, J.: Tendon injury and repair - a perspective on the basic mechanisms of tendon disease and future clinical therapy. Acta Biomater. 63, 18–36 (2017). https://doi.org/10.1016/j.actbio.2017.08.032 8. Stauber, T., Blache, U., Snedeker, J.G.: Tendon tissue microdamage and the limits of intrinsic repair. Matrix Biology 85, 68–79 (2020). https://doi.org/10.1016/j.matbio. 2019.07.008 9. Wunderli, S.L., Blache, U., Snedeker, J.G.: Tendon explant models for physiologically relevant in vitro study of tissue biology - a perspective. Connective Tissue Research 61(3–4), 262–277 (2020). https://doi.org/10.1080/03008207.2019.1700962

Modelling and Performance Evaluation

Lab Experiences on Impact Biomechanics of Human Head José Luis Rueda Arreguín1,2(B) , Marco Ceccarelli2(B) , Christopher R. Torres-San-Miguel1 , and Cuauhtémoc Morales Cruz2,3 1 Instituto Politécnico Nacional, Sección de Estudios de Posgrado e Investigación, Escuela

Superior de Ingeniería Mecánica y Eléctrica, 07738 Mexico City, Mexico [email protected], [email protected] 2 Department of Industrial Engineering, Laboratory of Robot Mechatronics, University of Rome Tor Vergata, 00133 Rome, Italy [email protected] 3 Instituto Politécnico Nacional, Centro de Innovación y Desarrollo Tecnológico en Cómputo, 07738 Mexico City, Mexico [email protected]

Abstract. In this paper experimental experiences in LARM 2 in Rome are presented to analyze impacts on human head. The experiments are worked out with on a commercial head mannequin that is hit by a rigid object. Two Inertial Measurement Unit (IMU) sensors and three force sensors are used to measure the impact characteristics. The sensors are located on suitable head points in order to monitor force, acceleration and angular displacement on lateral impact events. Results of illustrative tests are discussed to investigate and to characterize the biomechanics in human head impacts. Considerations from results are used to formulate a new criterion for neck-head injury by impacts. Keywords: Biomechanics · Head impact · Lab tests · Analysis · Injury criterion

1 Introduction Head injuries due to sudden impact are quite common during sports events, vehicle crashes and daily activities. In the literature, several impact testing methods are presented to improve head protective equipment and to establish standards for vehicles and sports safety, and to replicate different head impact events for its study as reported in [1–7]. Currently, there are impact test methods performing head impact tests for different events, like clash in sports, impact by projectiles, an injury suffered during vehicles accidents. Head injury and neck injury criteria are index related to the acceleration experience during impacts to make an estimation of the injurie level that a person can suffered [8–14]. Each impact test method is characterized by parameters as velocity and acceleration of the impact, the head zones where the impact occurs, and the sensors required for the data acquisition. The biomechanics of head impact is investigated by authors’ teams for safety index formulation [8], applications [9, 10] and testbed designs [10–12]. In this © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 229–237, 2021. https://doi.org/10.1007/978-3-030-58104-6_26

230

J. L. Rueda Arreguín et al.

paper lab experiences in LARM2 in Rome are presented with results on lateral impacts by using a commercial head mannequin and suitable sensors for data acquisition.

2 Safety Standards and Requirements Different methods for impact testing are available in literature, from the ones that are used in sport safety standards to the ones that evaluate vehicle accidents. Figure 1 shows the most popular impact test methods. The most typical impact test method involves a linear guided drop of the head model on an impact surface, as shown in Fig. 1a). This test allows the head movement on a single axis while its rotation is constrained [2]. A variation of this method is specified in ISO standards for hockey helmets, this consist of a free-falling head model with a guiding carrier, allowing the head model motion after the impact [3]. The predominant impact surfaces in these drop tests are anvils of various shapes or modular elastomer pads to represent a ground surface on which sport participants fall. In baseball and softball helmet standards, the most common methods consist of firing a projectile as in Fig. 1b). Figure 1c) shows a linear pneumatic ram in football and rugby helmet standards to perform an impact [4]. Another common test method in martial arts and soccer events consists on using an anthropometric head-neck mannequin and performing impacts using a rigid striker, as shown in Fig. 1d) [5]. In general, the vehicle accidents impact test methods are more complex. For example, Euro NCAP is a safety program for cars. They develop several tests in order to publish safety reports on new cars and one of these is the rear impact test. This method uses a full-body human anthropometric mannequin to test the whiplash effect [6]. This occurs when an acceleration force produces a neck injury as consequence of a sudden movement of the head.

a)

b)

c)

d)

Fig. 1. Head impact testing methods: a) linear guided drop [2]; b) projectile impact test [3]; c) pneumatic ram impact test [4]; d) lateral striker impact test [5].

Lab Experiences on Impact Biomechanics of Human Head

231

Requirements for a suitable testing can be defined in terms of testing setup and testing mode. For testing set up it is convenient to consider the following aspects: • A mannequin head model can be designed for a simulation of head impacts by monitoring angular displacements and acceleration of strategic zones of the head. • A pendulum mechanism can be dropped against mannequin head from a certain angle to ensure test repeatability with predeterminate impact energy. • Sensors of proper size and capacity can be used to measure impact force, angular acceleration and rotation of the mannequin head. For testing mode requirements can be identified in: • Giving orientation of the mannequin head that can be used to perform different impact test. • Using a pendulum angle that can be used to obtain a proper impact force for simulating feasible impacts on human head. Testing with head mannequin can be also designed with low-cost solutions and fairly simple operation with the aim both of affordable tests up to breaking conditions and facilitating a full understanding of biomechanics in head impacts. In summary, a suitable testing design can be arranged with commercial sensors and data elaboration equipment for test experiences with commercial head mannequin at impact acceleration of 10 g up to it is full damage.

3 Design of Experimental Tests The experiment proposed in this work consists on a low-cost test for head impact analysis. The main objective is to characterize head impact in terms of acceleration and force. Figure 2a) shows a scheme for the experimental test design to perform impact tests, and in Fig. 2b) the lab solution built at LARM2 in Rome to obtain the force, acceleration and angular displacement response on the head when impacted by a rigid object. The elements of the experimental test are one commercial rubber head mannequin, two IMU sensors and three force sensors. The head mannequin has a spherical joint that allows the head to move after having experienced an impact, simulating the action of the neck. The mannequin is fixed on a table. Referring to Fig. 2a) IMU sensors are located on the forehead (1) and on the neck (2), in order to analyze the acceleration force difference between the two head zones and the angular displacement of the head model. IMU sensors used in these experiments are BMI 160 which consist on an accelerometer and gyroscope that are able to measure the orientation, the body acceleration and the angular rate. These sensors are fixed inside 3D printed boxes to ensure a better fixation to the rubber head mannequin. Three force sensors are used to capture the force used to hit the head model to further on simulate the impact. These sensors consist on a resistor that modifies its values when a force, pressure or stress is applied. The ones used in this experiment have a maximum range of 100 N, and the diameter of the resistive area is 5 mm. They are located on the lateral (1), rear (2) and top (3) areas of the head, as

232

J. L. Rueda Arreguín et al.

indicated in Fig. 2a), to simulate events like car crashes, sports accidents or a falling from a high floor.

a)

b)

Fig. 2. Experimental layout at LARM2 in Rome: a) a scheme; b) a lab solution.

The impact is generated by a pendulum mechanism. The elements used in the pendulum are a 34 mm diameter steel sphere attached to a 37.5 cm stainless-steel pipe. A revolute joint is used to connect the stainless-steel pipe to a 70 cm rigid frame. The revolute joint also helps to keep the trajectory of the pendulum to ensure the impact on the sensitive area of the force sensor. Data acquisition is achieved by connecting a force sensor and IMUs to an Arduino MEGA 2560. The data transmission speed is increased up to 115200 bits/s to prevent loss of information. Figure 3 shows a block diagram for the proposed experimental test layout. All the data from sensors are saved as a “.xls” file, for their analysis and evaluation.

Force Sensor Pendulum Impact

Data elaboration

Head Model IMUs

Fig. 3. A block diagram for the structure of experimental tests at LARM2 in Fig. 2.

Lab Experiences on Impact Biomechanics of Human Head

233

In principle, the testing layout in Figs. 2 and 3 can be used also in experiences with humans when proper safety protocols can determinate proper range of safe impacts with the aim to have a validation and transfer of the biomechanics results that are obtained with the mannequin head.

4 Testing and Results Several tests have been performed at LARM 2 with different layouts and modes with the above-mentioned features for low-cost solution and fairly single understanding. In this paper, an illustrative test is reported to discuss the testing and to characterize the results. The tests consist on an impact that is generated against a head model using a rigid object, that is designed as a steel sphere in a pendulum extremity. The pendulum in Fig. 2 is dropped from an angle α of 60° to give a kinematic energy of 0.45 J with an equivalent force of 9 N. Figure 4 shows a snapshot of a lateral impact experiment with the motion response of the head model after receiving the impact from the pendulum weight. The angle of the pendulum is set by a small 3D printed triangle shape to ensure the repeatability of the tests. The elements are settled to ensure that the pendulum hits the mannequin only one time. Table 1 shows a summary of results from the tests that were performed in LARM 2. Five lateral impact tests were performed to discuss and to validate the repeatability of the experiment with its acquired sensor measures.

a)

b)

c)

Fig. 4. A snapshot of a lab test for lateral impact: a) starting; b) impact; c) after impact.

As an illustrative example, Fig. 5 shows the graphics of the results from test reported in Table 1. Figure 5a) shows the force applied by the pendulum. The impact consists in one single hit on mannequin’s head with impulsive action. As it is seen the impact occurs in less than 0.1 s. Figure 5b) shows the angular displacement in terms of roll and pitch for the lateral impact test performed. A displacement of 10° for the mannequin head in Fig. 5b) can be noted during the pitch rotation that matches with the lateral impact as in Fig. 4c). It is also measured a slightly rotation on the roll angle. This rotation means that the head perform a small

234

J. L. Rueda Arreguín et al. Table 1. Test results from lateral impact from Figs. 4 and 5. Test Force Head acceleration Neck acceleration Pitch (N) (g) (g) (°) 5.57

5.86

2.43

– 2.42 10.2

2

6.53

7.18

2.42

– 2.51 10.85

3

5.41

6.42

2.68

– 2.21 10.14

4

5.74

6.6

3.29

– 2.15

9.9

5

5.87

5.62

2.92

– 2.52

8.84

Force [N]

Angular displacement [°]

1

Roll: Pitch:

Time [s]

Time [s] b)

a) Acceleration [g]

Roll (°)

IMU 1 (head): IMU 2 (neck):

Time [s] c) Fig. 5. Experimental results for test 1 of lateral impact in Table 1 in terms of: a) impact force by sensor 1; b) angular displacement; c) accelerations by IMU sensors.

backward movement due the pitch negative value. The impact is characterized by an impulsive force, Fig. 4a), that is sensed also with an accelerated motion both of the head and neck that is nevertheless perturbated for a while yet, although with a quite smooth angular motion. Figure 5c) shows the plot of the accelerations on head and neck as referring to IMU 1 and 2, respectively. It is to note that head acceleration is more than twice that acceleration on the neck. The plot shows small waves after the acceleration peak that could be related to the action of bouncing after the head reaches it maximum

Lab Experiences on Impact Biomechanics of Human Head

235

displacement. Results shows similar force values except for test number 2 as reported in Table 1. In this one the force is larger as well as head acceleration and pitch angles. This shows a relation in which due a larger force, the mannequin presents larger acceleration and angular displacement which is justified according to energy conservation law. The test results show a characteristic of the impact in a straightforward relation between the impact force and consequent head acceleration while the neck suffers the impact yet but with reduced effects, and the motion of the head and neck continues for a while with smooth but perturbated time history.

5 A New Neck Injury Criterion Neck Injury Criterion (NIC) was proposed by Boström et al. [13], to predict neck injuries at low speed impact crashes. This criterion evaluates the acceleration and velocity from vertebra C7 to vertebra C1 using the formula 2 NIC = arel ∗ l + vrel

(1)

in which arel is the relative linear acceleration between vertebrae C7 and vertebrae C1, l is the distance between these two cervical vertebras and vrel is the relative velocity of C1 with respect to C7. The mathematical model Eq. (1) evaluates the relative horizontal acceleration considering the distance between the cervical as a constant [14]. The experiment here presented allows a user to obtain significant parameters as the angular displacement in terms of roll and pitch angles during the impact as in Fig. 5b). With this information, it is possible to know the position vector Xt of the head after the impact using rotation matrix [R] considering the roll and pitch angles in Xt =[R]x, where x is the initial position vector of the head. By using the test results in Table 1 and Figs. 4 and 5, a better impact evaluation can be expressed considering the impact force, the impact energy, the acceleration of the head and the neck, and the new position vector of the head. Likewise in previous work as reported in [8], it seems convenient to formulate a specific criterion for a specific evaluation of the biomechanics in head impacts considering contributions from neck response. In particular, if the consequent angular displacement is included in the risk evaluation, a new criterion can be formulated when considering an impact in a time of t ≤ 0.1 s, as NICN =

F ah |Xt − X | Mh v2 an

(2)

in which F is the applied force, Mh is the weight of the head; v2 is the velocity from the pendulum impulse representing the impact energy; ah is the head acceleration measured by IMU 1; an is the neck acceleration obtained from IMU 2; Xt is the position of the head at after the impact. Thus, it is possible to determine the NICN as a function of the impact impulse, head structure, and response of both head and neck. Examples are evaluated in the lab experiences as in the reported Table 2 with a clear characterization. From the values in Table 2 it is to note that the new criterion gives a better differentiation of the results and the neck contribution makes more evident the impact consequences. In fact, the riskiest test results to be number 5 in Table 1 for which the impact affected significant both head and neck.

236

J. L. Rueda Arreguín et al. Table 2 Comparison between NIC and NICN. Test NIC

NICN

1

2.155 3.704

2

2.507 3.754

3

2.304 2.701

4

2.352 2.981

5

2.091 4.308

6 Conclusions The reported lab experiences show a successful characterization of the biomechanics of human head impact by using a mannequin head with a low-cost testing layout. During tests, the acceleration that is sensed by IMU 1 keeps its value as double than in the IMU 2 on the neck. This means that during a head impact, the energy of the hit reaches the neck zone with enough acceleration to cause some damage in neck yet. In addition, it is to note during a very similar period of time a relation is observable between the impulsive impact force and the consequent acceleration on head. The reported lab experiences have suggested a new formulation of the NIC criterion to take into account the consequent angular motion with perturbation in an evaluation of the risk severity of a head impact. Acknowledgments. The first author wishes to gratefully acknowledge Consejo Nacional de Ciencia y Tecnologia (CONACyT) and Instituto Politécnico Nacional through scholarship “Becas de Movilidad 2019 al Extranjero” for permitting his period of study at LARM2 of University of Tor Vergata in the A.Y. 2019-20 within a double PhD degree program and also thank the support of projects 20201964 and 20200930, as well as an EDI grant, all by SIP/IPN.

References 1. Bina, A.J., Batt, G.S., Des Jardins, J.D.: A review of laboratory methods and results used to evaluate protective headgear in American football. Proc. Inst. Mech. Eng. Part P: J. Sports Eng. Technol. 232(4), 360–368 (2018) 2. ASTM, Standard Test Methods for Equipment and Procedures Used in Evaluating the Performance Characteristics of Protective Headgear, ASTM International, F1446-15b (2015) 3. Menickelli, J., Cooper, C.A., Withnall, C., Wonnacott, M.: Analysis and comparison of lateral head impacts using various golf discs and a Hybrid III head form. Sports Biomech. 18(6), 1–11 (2019) 4. NOCSAE, NOCSAE 081-18am19a: Standard Pneumatic Ram Test Method and Equipment Used in Evaluating the Performance Characteristics of Protective Headgear and Face Guards, National Operating Committee on Standards for Athletic Equipment (2019) 5. O’Sullivan, D.M., Fife, G.P.: Impact attenuation of protective boxing and taekwondo headgear. Eur. J. Sport Sci. 16(8), 1219–1225 (2016) 6. EURO NCAP, The Dynamic Assessment of Car Seats for Neck Injury Protection Testing Protocol, European New Car Assessment Programme, Version 4.1 (2019)

Lab Experiences on Impact Biomechanics of Human Head

237

7. Echávarri, J., Ceccarelli, M., Carbone, G., Alén, C., Muñoz, J. L., Díaz, A. Muñoz Guijosa, J. M.: Towards a safety index for assessing head injury potential in service robotics. Adv. Robot. 27(11), 831–844 (2013) 8. Alén-Cordero, C., Carbone, G., Ceccarelli, M., Echávarri, J., Muñoz, J.L.: Experimental tests in human–robot collision evaluation and characterization of a new safety index for robot operation. Mech. Mach. Theory 80, 184–199 (2014) 9. Cortes-Vasquez, O., Torres-San-Miguel, C.-R., Urriolagoitia-Sosa, G., Cruz-Jaramillo, I.-L., Aguilar-Pérez, L.-A., Martínez-Sáez, L., Romero-Ángeles, B., Urriolagoitia-Calderón, G.M.: Head injury criterion (HIC) numerical comparison in run over conditions at different speeds with a sedan vehicle type. Dyna 92(5), 489 (2017) 10. Cortés-Vásquez, O., Cruz-Jaramillo, I.L., Torres-San Miguel, C.R., Reyes-Jiménez, G.A., Verduzco-Cedeño, V.F., Rodríguez-Martínez, R., Urriolagoitia-Sosa, G.: Numerical simulation of the encephalic injury index caused by a vehicular accident in different collision scenarios. Científica 20(2), 93–101 (2016) 11. Rueda Arreguín, J.L.,Torres San Miguel, C.R., Ceccarelli, M., Ramírez Vela, V. Urriolagoitia Calderón, G.M.: Design of a test bench to simulate cranial sudden impact. Mech. Mach. Sci. 65, 225–234 (2019) 12. Rueda Arreguín, J.L., Ceccarelli, M., Torres San Miguel, C.R.: Design and simulation of a parallel-mechanism testbed for head impact. In: 29th International Conference on Robotics in Alpe-Adria-Danube Region, RAAD 2020, Poitiers (2020). (under review) 13. Boström, O., Svennson, M.Y., Aldman, B., Hansson, H.A., Haland, Y., Lövsund, P., Seeman, T., Suneson, A., Säljö, A., Örtengren, T.: A new neck injury criterion candidate — based on injury findings in the cervical spine ganglia after experimental neck extension trauma. In: Proceedings of the International Conference on the Biomechanics of Impact, Dublin, Ireland (1996) 14. Croft, A.C., Herring, P., Freeman, M.D., Haneline, M.T.: The neck injury criterion: future considerations. Accid. Anal. Prev. 34(2), 247–255 (2002)

Nonlinear Dynamic Analysis of Human Sit-to-Stand Movement with Application to the Robotic Structures Daniela Tarnita1(B) , Alin Petcu1 , Marius Georgescu1 , Ionut Geonea1 , and Danut Tarnita2 1 University of Craiova, Craiova, Romania

[email protected] 2 University of Medicine and Pharmacy, Craiova, Romania

Abstract. The purpose of this study was to estimate Lyapunov exponents (LE) in order to quantify the local dynamic stability of the lower limb joints during sit-to-stand movement. The values of the Lyapunov exponents are obtained based on the experimental time series of the flexion-extension movements in sagittal plane and the rotational movements in frontal plane for all six main joints of both lower limbs, collected from a sample of healthy subjects and a sample of patients suffering by knee osteoarthritis before total knee replacement. The values of LEs obtained for the osteoarthritic lower limb are associated with more divergence and less stability, being generally, larger than those obtained for the other patient’ limb, and larger than those of healthy joints. Keywords: Sit-to-stand · Osteoarthritic knee · Stability · Lyapunov exponents · Phase-planes

1 Introduction In recent years, researchers are increasingly inspired by biological systems and their self-stabilization to design robots or robotic structures that must maintain their stability. Many works investigate the natural behaviours of human subjects and the biomechanics of their movements [1–11] in order to transfer knowledge in the study of bioinspired robotic structures and of the modules used for medical applications, especially in the field of rehabilitation, prosthetics and orthotics [2, 7, 10–20], and minimally invasive surgery [21–23]. Standing up from a seated position is an essential condition to maintain the stability of the person by locating the vertical component of the reaction force with the ground in the support area [6] and for walking and therefore for the functional independence of an individual [8, 9]. It has been reported that people who have difficulties in standing up from a seated position are more likely to lose their stability and to fall during walking [9]. The inability to stand was related to death among the elderly people [6, 8, 9]. Chaos analysis and LEs have been used to quantify the local dynamic stability of human walking kinematics [3–5] or chaotic mobile robots and chaotic behaviour of © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 238–246, 2021. https://doi.org/10.1007/978-3-030-58104-6_27

Nonlinear Dynamic Analysis of Human Sit-to-Stand Movement with Application

239

Robot-environment interaction [12–15]. In this paper, we apply a nonlinear analysis approach to quantify local dynamic stability of human lower limb joints during sit-tostand movement, by using LE which are computed based on the experimental time series collected for all six joints in sagittal plane and frontal plane.

2 Experimental Protocol For human sit-to-stand experimental analysis, a Biometrics data acquisition system based on electrogoniometers is used [7, 24–25]. The SG twin axis goniometers are used to measure joint angles simultaneously in two perpendicular planes of movement. In Fig. 1, the acquisition system composed by Biometrics software installed on the notebook, by electrogoniometers and Datalog devices mounted on the human subject, is shown. The sensors are connected to Biometrics DataLog through which the data are transferred to the computer via Bluetooth interface. The tests were performed by a sample of seven healthy subjects with any musculoskeletal disorders and a sample of 3 patients with osteoarthritis disease of the left knee, two days before the total knee replacement surgery. Each subject and patient executed 15 consecutive sit-to stand cycles. The subjects and patients gave their written consent for performing the tests. Ethics Committee of the University of Craiova approved the research. The data series collected with a frequency equal to 500 Hz from the left and right joints (both movements: sagittal plane and frontal plane) are used in order to quantify the local dynamic stability of the joints’ movements. In Table 1, the mean values and standard deviations for anthropometric data are presented.

Fig. 1. Sit-to-stand of a subject with mounted acquisition system; electrogoniometers and Datalog Table 1. Mean values and standard deviations of subjects’ and patients’ anthropometric data

Subjects

Patients

Average

Age [years]

Weight [kg]

Height [cm]

Hip–knee length [cm]

Knee–ankle length [cm]

30.26

75.6

172.54

44.03

39.5

St. Dev.

3.73

6.18

4.12

3.14

2.59

Average

59.23

79.84

165.32

42.16

39.42

St. Dev.

4.35

6.18

5.27

3.65

3.26

240

D. Tarnita et al.

3 Results The data files of the angular amplitudes of joints flexion-extension movements in sagittal plane and of rotational movements in frontal plane during sit-to-stand movement were obtained for each subject and patient from the report generated by the acquisition system. In Fig. 2 consecutive cycles of the right lower limb joints movements in sagittal plane and in frontal plane, corresponding to the sit-to-stand experimental data acquired by using Biometrics system are shown. Similar diagrams were obtained for the left lower limb joints and for patients’ sample.

Fig. 2. Time series collected by Biometrics system for the movements’ angles of the right lower limb joints, subject 2

Taken into account the natural biological variability from one individual to another, but also from one movement cycle to other, nine consecutive standing and sitting cycles were selected for each subject, eliminating the first three and last three cycles in order to remove the transient data. For a more accurate representation of human joint variability, unfiltered data were analysed in this study. These reduced data files were normalized by

Fig. 3. Mean cycle of sit to stand movement for the right lower limb joints: a) normalized consecutive cycles for right ankle; b) ankle mean cycle; c) knee mean cycle; d) hip mean cycle

Nonlinear Dynamic Analysis of Human Sit-to-Stand Movement with Application

241

interpolation and reported on the abscissa at a scaled range of 0 to 100%, using Simi Motion software [11, 24–25]. For each subject and patient, the normalized curves of the angles corresponding to each movement, as well as the curve corresponding to the mean cycle were obtained, as shown in Fig. 3.

4 Dynamic Analysis The objective of this study consists on quantifying nonlinear sit-to-stand motion of human lower limbs joints for healthy subjects and for patients suffering of knee osteoarthritis, using nonlinear dynamics stability analysis. In this study, the human lower limb is considered as a nonlinear system that can achieve dynamic equilibrium. In order to characterize the kinematics of the system when this equilibrium is attained, phase plane portraits are used, by correlating the joint angular rotations with the respective joint angular velocities [24–26]. For all patients, the left knee is the osteoarthritic one. The phase-plane graphics corresponding to the right joints movements of healthy subject 2 with those corresponding to the left joints movements of Patient 2 (Fig. 4) are compared. It can be seen that for healthy subject the cycle’s curves show less divergence in their trajectories, are more compact, the amplitudes tend to be constant and their spread is decreasing, while the curves traced for the patient’ joints show more divergence in their trajectories. One of the known methods to reconstruct the state space S is to generate the delay coordinates vectors [26]. The space constructed by using the vector: xn = [s(t0 + nTs ); s(t0 + nTs + T), . . . . s(t0 + nTs + (dE − 1)T)];

(1)

is called the reconstructed space, where the integer dE is the embedding dimension. In relation (1) the notation s(.) is a measured scalar function, Ts is the sampling time, n = 1,2,…,dE and T = kTs is an appropriately chosen time delay. In [27] is demonstrated that the dynamics in the reconstructed state space is equivalent to the original dynamics. In Fig. 5 some examples of state-space reconstruction are shown. An appropriate time lag, T, was computed for each time series, by using the average mutual information (AMI) function [28], which sets the time lag equal to the value of delay corresponding to the first minimum of the AMI function (Fig. 6). A suitable embedding dimension which is the minimum value that trajectories of the reconstructed state vector may not cross over each other in state space, was chosen by using the false nearest neighbour method [29]. The LEs are computed using the Rosenstein algorithm [30], ‘lyap_r’ routine available in TISEAN package. An increase of LE implies a decrease of local dynamic stability. The LEs calculated for all-time series were positive, that means the time series of the human angular movements were unstable and the human lower limb is a deterministic chaotic system. In Table 2 the average values of the computed LEs for all joints movements are presented, while in Fig. 7 these values are plotted. Comparing with the joints of healthy subjects, the LE values are bigger for the normal knees of the patients and much bigger for osteoarthritic knees. Larger values of short LEs obtained for the right lower limb joints of patients are explained by the influence

242

D. Tarnita et al.

Fig. 4. Phase plane plots for the movements of Subject 3 joints and of Patient 2 joints

Nonlinear Dynamic Analysis of Human Sit-to-Stand Movement with Application

243

Fig. 5. State space reconstruction for healthy subjects and patients

Fig. 6. AMI function diagram, FNN diagram, LE diagram

of the pain and instability of the osteoarthritic knees. Larger values of LEs obtained for patient’s left lower limb joints are associated with more divergence, more instability and movements’ variability, while smaller values obtained for healthy subjects reflect a local stability, less divergence and less variability. The values of LEs for each test were positive values that suggest that human motions show chaotic characteristics. The values of LEs obtained for the osteoarthritic lower limb are associated with more divergence and less stability, being generally, larger than those obtained for the other patient’ limb, and larger than those of healthy joints.

244

D. Tarnita et al. Table 2. The average values of the computed LEs for all joints movements

Joint movement

Average LE, Patients

Average LE, Subjects

Left ankle flexion-extension (la fl-ext)

0.34106687

0.25717034

Left ankle inversion-eversion (la inv-ev)

0.12865239

0.09948355

Left knee flexion-extension (lk fl-ext)

0.55329236

0.48458887

Left knee varus-valgus (lk vv)

0.20290664

0.12070048

Left hip flexion-extension (lh fl-ext)

0.52835276

0.37977435

Left hip abduction-adduction (lh abd-add)

0.19503632

0.07003171

Right ankle flexion-extension (ra fl-ext)

0.45485171

0.21928868

Right ankle inversion-eversion (ra inv-ev)

0.19362316

0.01053878

Right knee flexion-extension (rk fl-ext)

0.41391815

0.38708896

Right knee varus-valgus (rk vv)

0.19772702

0.12038476

Right hip flexion-extension (rh fl-ext)

0.49370409

0.37772131

Right hip abduction-adduction (rh abd-add)

0.22478291

0.14212969

AVERAGE LYAPUNOV EXPONENT Unstable Stable 0.6 0.4 0.2 0

Fig. 7. Average values of the computed LEs

5 Conclusions This paper presents a study of the influence of the osteoarthritic knee on the stability of human lower limb joints movements on a sample of patients suffering by this disease. The LEs calculated for healthy joints are smaller than those obtained for osteoarthritic lower limb joints, and they are similar with those obtained for the joints of the other lower limb of patients. This study is important for the correct approach the design and construction of artificial devices used for rehabilitation in order to improve the dynamic stability of human sit-to-stand. Biologically inspired control structure for biped robots and their agreement with the behavioral and biological facts could be developed and help modeling human motor control system.

Nonlinear Dynamic Analysis of Human Sit-to-Stand Movement with Application

245

References 1. Galli, M., et al.: Quantitative analysis of sit to stand movement: experimental set-up definition and application to healthy and hemiplegic adults. Gait Posture 28, 80–85 (2008) 2. Tarnita, D., et al.: Orthopedic modular implants based on shape memory alloys. chapter in Biomedical engineering – from theory to applications. In: Tech, Viena, pp.431–468 (2008). 3. Dingwell, J.B., Cusumano, J.P.: Nonlinear time series analysis of normal and pathological human walking. Chaos 10(4), 848–863 (2000) 4. Scafetta, N., et al.: Understanding the complexity of human gait dynamics. Chaosm 19, 026108 (2009) 5. McAndrew, P., Wilken, J., Dingwell, J.: Dynamic stability of human walking in visually and mechanically destabilizing environments. J. Biomech 44, 644–649 (2011) 6. Rodosky, M.W., Andriacchi, T.P., et al.: The influence of chair height on lower-limb mechanics during rising. J. Orthop. Res. 7, 266–271 (1989) 7. Tarnita, D., Pisla, D., Geonea, I., et al.: Static and dynamic analysis of osteoarthritic and orthotic human knee. J. Bionic Eng. 16, 514–525 (2019) 8. Dehail, P., et al.: Kinematic and electromyographic analysis of rising from a chair during a “Sit-to-Walk” task în elderly subjects: role of strength. Clin Biomech 22(10), 1096–1103 (2007) 9. Lord, S.R., et al.: Sit-to-stand performance depends on speed, balance, and psychol status în addition to strength în older people. J. Gerontol. Biol. Sci. Med. Sci. 57(8), 539–543 (2002) 10. Berceanu, C., et al.: About an experimental approach used to determine the kinematics of the human. J. Solid State Phenom. 166–167, 45–50 (2010) 11. Copilusi, C., et al.: Cam mechanism kinematic analysis used in a human ankle prosthesis structure. In: World Congress on Engineering (WCE 2010), London, UK (2010). 12. Osuka, K., Kirihara, K.: Analysis and experiments of passive walking robot QUARTET II. In: IEEE International Conference on Robotics and Automation (2000). 13. Goswami, A., Thuilot, B., Espiau, B.: A study of the passive gait of a compass-like biped robot: symmetry and chaos. Int. J. Robot. Res. 7, 1282–1301 (1998) 14. Nakamura, Y., Sekiguchi, A.: The chaotic mobile robot, "Robotics and Automation”. IEEE Trans. Robot. Autom. 17, 898–904 (2001) 15. Nehmzow, U.: Scientific Methods in Mobile Robotics: Quantitative Analysis of Agent Behavior. Springer, Berlin (2006) 16. Geonea, I., Tarnita, D.: Design and evaluation of a new exoskeleton for gait rehabilitation. Mech. Sci. 8(2), 307–322 (2017) 17. Dumitru, N., et al.: Dynamic analysis of an exoskeleton new ankle joint mechanism. New Trends Mech. Mach. Sci. 24, 709–717 (2015) 18. Tarnita, D., Berceanu, C., et al.: The three-dimensional printing–a modern technology used for biomedical prototypes. Mater. Plast. 47(3), 328–334 (2010) 19. Enescu, M.L., Alexandru, C.: Modeling and simulation of A 6 DOF Robo. In: Advanced Materials Research, pp. 1116–1119 (2012). 20. Emadi, M., Bahrami, F.: A biologically inspired modular structure to control the Sit-to-Stand transfer of a biped robot. In: IEEE Proceed (2007). 21. Pisla, D., Plitea, N., Vaida, C.: Kinematic modeling and workspace generation for a new parallel robot used in minimally invasive surgery. In: Advances in Robot Kinematics: Analysis and Design, pp. 459–468 (2008). 22. Gherman, B., et al.: On the singularity-free workspace of a parallel robot for lower-limb rehabilitation. Proc. Rom. Acad. 20(4), 383–391 (2019) 23. Pisla, D., Gherman, B., Plitea, N., et al.: PARASURG hybrid parallel robot for minimally invasive surgery. Chirurgia 106(5), 619–625 (2011)

246

D. Tarnita et al.

24. Tarnita, D., Marghitu, D.: Nonlinear dynamics of normal and osteoarthritic human knee. In: Proceedings of the Romanian Academy, pp. 353–360 (2017). 25. Tarnita, D., et al.: Applications of nonlinear dynamics to human knee movem on plane & inclined treadmill. New Trends Med. Serv. Robots 39, 59–73 (2016) 26. Packard, N.H., Crutchfield, J.P., Farmer, J.D., Shaw, R.S.: Geometry from a time series. Phys. Rev. Lett. 45, 712–716 (1980) 27. Nayfeh, A.H.: Introduction to Perturbation Techniques. Wiley, New York (1981) 28. Fraser, A.M., Swinney, H.L.: Independent coordinates for strange attractors from mutual information. Phys. Rev. A33, 1134–1140 (1986) 29. Kennel, M.B., Brown, R., Abarbanel, H.D.I.: Determining minimum embedding dimension using a geometrical construction. Phys. Rev. A45, 3403–3411 (1992) 30. Rosenstein, M.T., Collins, I.J., Deluca, C.J.: A practical method for calculating largest Lyapunov exponents from small data sets. Phys. D. 65, 117–134 (1993)

Human Squat Motion: Joint Torques Estimation with a 3D Model and a Sagittal Model ´ Olivier Bordron, Cl´ement Huneau, Eric Le Carpentier, and Yannick Aoustin(B) Laboratoire des Sciences du Num´erique de Nantes, UMR CNRS 6004, Centrale Nantes, Universit´e de Nantes, 1 Rue de la No¨e, 44321 Nantes Cedex 3, France {o.bordron,c.huneau,e.lecarpentier,yannick.aoustin}@univ-nantes.fr

Abstract. A specific half squat motion is analyzed as a planar movement. Experimental data about this motion are recorded with a motion capture device and two force plates. Joint torques are estimated with -a 3D Opensim model and a sagittal Matlab model. Torque variations from the two models are consistent even if the magnitudes of the corresponding variables differ. These results are attributable to the different anthropometric tables which the two models are based on. Another strategy developed here consists in estimating joint torques without the measured ground reaction forces. In that case, global vertical reaction force is well estimated. The use of the sagittal Matlab model is an efficient way to preliminarily analyze squat trajectories. The next step in this work is the study of the influence of the load represented by the weight of a knee prosthesis. Keywords: Squat motion plate

1

· Dynamic model · Motion capture · Force

Introduction

A repetitive movement over a long period of time, habituation to poor posture, handling heavy load can lead to musculoskeletal disorders (MSDs). They are the leading cost drivers in the workers compensation system [5]. The most important thing is to prevent MSDs by automating repetitive tasks. Another solution is to assist the operator with a passive exoskeleton [2] to maintain posture or active [1,8]. But it is also necessary to help a patient with rehabilitation when a MDS has appeared [13]. Regarding the human musculoskeletal system of movement analyzes, joint torques developed by muscles are useful to prevent or treat MSDs. Often for these analyzes the squat motion is routinely prescribed by physical therapists and sport medicine physicians [3]. Authors demonstrate a technique to calculate the EMG instantaneous median frequency to assess muscle fatigue during a dynamic exercise commonly prescribed in patients with an anterior cruciate ligament deficiency. A lot of studies are based on the squat c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 247–255, 2021. https://doi.org/10.1007/978-3-030-58104-6_28

248

O. Bordron et al.

movement. Without being exhaustive we can mention the following interesting works. Hwang et al. [7] show that for two different symmetrical lifting techniques squat and stoop, there are no significant differences in maximum lumbar joint moments. Bonnet et al. [4] estimate in real-time the lower-limb joint and torso kinematics by using a single inertial measurement unit placed on the lower back. Wei et al. [11] compare the kinematics of young and elderly subjects during normal squatting activity. They observe that the elderly enjoys more hip flexion/extension angles than the young, with the squatting posture assumed. Their purpose is to establish a standard of designing lower extremity prosthesis. These studies focus mainly on the kinematics of movement. However, there is no investigation into the estimation of joint torques developed by muscles. The knowledge of these torques is important to evaluate the possibilities of assisting a joint with an orthosis for example. Our study is also based on a squat movement. Considering that this movement takes place in the sagittal plane our strategy is to evaluate the torques with a simplified sagittal model as follows. During the squat experimental data are recorded with a motion capture device and two force plates. An Opensim Model allows us to estimate the position, velocity and acceleration of the ankle, knee an hip joints of the human subject. The main advantage of musculoskeletal models is to provide a non-invasive means to study human movement [6,10]. Then a simplified inverse dynamic model of a biped, taking into account explicitly the contact with the ground, is designed. Several cases are considered to compare the articular torques calculated thanks to a sagittal model with the torques estimated with OpenSim. Our objective is to define an efficient but simple tool, which is based on the dynamical effects of the lower-limbs to size an orthosis. This paper is outlined as follows. Section 2 presents the used material and the methods. Section 3, the results are analyzed. Section 4 a discussion is conducted about the assumption symmetric effort distribution for the sagittal model. Section 5 offers our conclusion and perspectives.

2 2.1

Materials and Methods Squat Motion

The squat movement studied is defined as a four-step cycle so that it can be repeated in a continuous loop by the subject. The total duration of that cycle is T = 4 s and each step lasts one second. In its initial state, the subject stands upright with his knees slightly bent (Fig. 1a)). The arms are stretched horizontally. The trunk is also slightly inclined with respect to the longitudinal axis. The cycle is defined as follows: 1. The subject goes down bending his knees while keeping his arms straight and his trunk slightly inclined forward. At the end of this step, the subject’s thighs are parallel to the ground in order to perform a half squat (Fig. 1b)). 2. This pose is then maintained before starting the ascent step. 3. The subject returns to the initial pose (Fig. 1c)). 4. The initial pose is maintained.

Human Squat Motion: Joint Torques Estimation with a 3D Model

2.2

249

Data Acquisition

The Motion Capture experiment was conducted with a 25-year-old woman, height H = 1.73 m and mass M = 62 kg. A set of 17 marker arrays were placed on the subject’s head, trunk, arms, legs and feet. During the movement, the spatial coordinates of the marker arrays were estimated by the ART IR acquisition system using eight cameras with an acquisition frequency 60 Hz. Two force plates were used to measure the center of pressure of each foot of the subject on the ground as well as the ground reaction forces and moments. N = 20 squats were performed by the subject according to the motion defined in 2.1. To eliminate the measurement noise, we applied a Butterworth filter to all signals with a cut-off frequency 5 Hz. From the data of the force plates and more specifically the global vertical ground reaction, a rupture detection method -the CUSUM algorithm [9]- was employed to detect initial cycle times. All data presented in Sect. 3 were averaged with respect to the N cycles.

a)

b)

c)

Fig. 1. a) Initial configuration of biped. b) Intermediate configuration. c) Final configuration − initial one in the next squat motion.

2.3

Opensim Model

The model developed on the software Opensim is based on the Rajagopal model [10]. It is a complete musculoskeletal model designed to perform dynamic simulations of human movement. For the dynamic analysis of squat movement, this model has been adapted with unilateral constraints between the ground and both feet so that they are in full flat contact with the ground (Fig. 2). Virtual markers are defined in respect to the different bodies of the model. The estimation of the articular torques is done in three steps. First, a scaling step is required to adjust the size, mass distribution and position of the Opensim model’s virtual markers. From a static trial, the position data of the experimental markers are recorded via the motion capture system. Then, the deviation between the experimental markers and their corresponding virtual marker is minimized in the sense of least squares. The optimization variables are the segment lengths.

250

O. Bordron et al.

Then, several squat trials are carried out by the subject. The position of the experimental markers are imported on Opensim. The inverse kinematic step consists in minimizing a least squares problem in order to model the performed motions. Finally, the inverse dynamic step enables the joint torques estimation from the forces measured by the force plates and the motions obtained by the inverse kinematic model.

A

B

C

Fig. 2. Full-Body Musculoskeletal Opensim Model for the squat movement.

2.4

Matlab Model

Anthropomorphic Parameters. A sagittal model of a nine-link planar biped is designed in order to evaluate the joint torques, which are developed during the squat motion. The physical parameters of this model are obtained from Winter’s

o

g67 m

k

q67 i

g

q33

n a

g44

j b

g33

h

c

q44 q22

g55

f d

g22

q55 q11

g66 e

p22 g11 l

p11

l

Fig. 3. Model of the planed biped. a) Parameterization of the biped. Note that angles are positive in counterclockwise. b) Length segments and position of centers of mass.

Human Squat Motion: Joint Torques Estimation with a 3D Model

251

anthropometric table [12] by considering the height and weight of the subject. The planar biped has a trunk, two arms, two thighs, two shins and two feet, see Fig. 3. For this planar biped Table 1 gathers the length, mass, center of mass position and inertia moment of each segment. Table 1. Anthropomorphic parameters of the human planar model. Calf

Thigh

Trunk

Arm

Segment Weight (kg) mf = 0.90

Foot

mc = 2.88

mt = 6.20

mT = 35.8

ma = 3.10

Segment Length (m)

Lc = 0.426

Lt = 0.424

LT = 0.813

La = 0.761

Lf = 0.068 Ld = 0.196

Ha = 0.498

Hf = 0.068 Inertia (kg.m2 )

If = 0.0087 Ic = 0.0476 It = 0.1162 IT = 2.2508 Ia = 0.1385

Center of mass (m)

sfx = 0.098 sc = 0.184

st = 0.184

sT = 0.329

sa = 0.304

sfy = 0.034

The biped configuration is described with q, Fig. 3:   q = q1 , q2 , q3 , q4 , q5 , q6 , q7 , qp1 , qp2 , xh , yh . Unilateral Constraints Between the Ground and the Stance Foot. For the studied squat motion, both feet have a flat contact with the ground. In the sagittal plane, the action of the ground on each foot can be modeled as a wrench with three components, expressed at Ai , as illustrated in Fig. 4 such as: ri = [rix riy Miz ] , for i ∈ {1, 2}.

(1)

e f

k a b

d j

i

h

c

g Fig. 4. Torques and forces acting on the foot.

252

O. Bordron et al.

The conditions of feet flat contact on the ground are as follows: [xAi yAi qpi ] = [0 − Hf 0] , for i ∈ {1, 2}

(2)

By deriving (2) in time, we obtain the conditions in velocity and acceleration of each foot and the ground, expressed in point Ai : ¨ + J˙ i q˙ 2 = 03×1 , for i ∈ {1, 2}. Ji q˙ = 03×1 and Ji q

(3)

The non-rotation of the supporting feet reflects the dynamic equilibrium of the feet during the squat motion. This state can be described by the trajectory of the centre of pressure (CoP) of each foot. This point, called Pi , represents the application point of the ground reaction forces such as Miz = 0, for i ∈ {1, 2}. That definition leads to: xPi =

Γia + mf gsfx − Hf rix and − Ld ≤ xPi ≤ Lf , for i ∈ {1, 2} riy

(4)

The condition of dynamic equilibrium of the feet can be expressed by (4). Dynamic Model. The dynamic model of the nine-links planar model is defined as follows:  ˙ q˙ + G(q) = BΓ + J D(q)¨ q + C(q, q) (5) 1 r1 + J2 r2 Here Γ8×1 is the torque vector, the 3 × 1 ground reaction wrenches r1 and r2 are applied to respectively the feet 1 and 2. B11×8 is a constant matrix to  take into account of the influence the torques in each joint. J 1 and J2 are the 11 × 3 transposed Jacobian matrices, which convert ground reaction wrenches at feet 1 and 2 in torques applied to the d joints. D11×11 is the symmetric positive inertia matrix, C11×11 represents the Coriolis and centrifugal effects, and G11 defines the gravity effect. Calculation of Γ, r1 , and r2 . By the motion capture acquisition system, kinematic data related to the squat motion were recorded for each sampling time. Consequently, the left part of (5) can be calculated. The unknown variables are the 8 joint torques and the 2 × 3 components of r1 and r2 . Then, the global model can be considered as a system with 11 equations and 14 unknown variables. By considering the sagittal plane as a symmetric plane for the motion and the distribution of masses, let us consider that r1 = r2 = r. In this way,   the inverse  dynamic model can be totally resolved by using (6). The matrix B J 1 + J2 has always been invertible in our numerical tests.      Γ  −1 ˙ q˙ + G(q)) = B J (D(q)¨ q + C(q, q) (6) 1 + J2 r

Human Squat Motion: Joint Torques Estimation with a 3D Model

253

ˆ of the Joint Torques by the Least Mean Squares EstimaEstimation Γ tion. The ground reaction forces applied to feet 1 and 2 are recorded with two force plates. These data can be used to inverse (5). The global system has more equations than unknown variables. The least mean squares estimation allows us ˆ with (7), where B+ is the 8 × 11 pseudo-inverse to estimate the joint torques Γ matrix of B.    ˆ = B+ D(q)¨ ˙ q˙ + G(q) − J q + C(q, q) (7) Γ 1 r1 − J2 r2

3 3.1

Results Kinematic and Force Plates Data of the Squat Motion

The data resulting from the motion capture session were imported into Opensim. The inverse kinematic model was solved according to Sect. 2.3 to obtain angular variable trajectories for each joint. For the use of the sagittal model, motion trajectories have been projected in the sagittal plane. The analysis of the measured average trajectory of the CoP shows the presence of a privileged direction. The trajectory of the global CoP -the average of the trajectories of the CoPs for the left and right foot- evolves mainly along an axis parallel to the x axis. This result is consistent with the fact that the squat movement is a plane-type movement. 3.2

Comparison of the Estimation of the Joint Torques Between a 3D Opensim model and a sagittal Matlab model.

From the motion obtained in Sect. 3.1, inverse dynamic model has been first resolved with the 3D Opensim model. From the projected trajectories in the sagittal plane, the resolution was carried out by using (7). Figure 5 illustrates the results obtained from both resolutions. The torque estimation made by the two models at the ankle and knee joints are almost the same for both left and right sides. Nevertheless, the hip torque estimation made by the Matlab model differs from the Opensim model. Variations in the anthropomorphic data used in both models can lead to these differences. The few torques variations of both shoulders are due to the gravity effect in order to maintain a stable position.

4

Discussion

In a simulation approach, to study other squat trajectories, it may be interesting to solve the dynamic model by freeing us from the measured data. In this case, an assumption on the efforts distribution is necessary. As shown with (6), joint torques and reaction forces can be calculated by assuming that the subject distributes the forces equally over each of his feet. The magnitude of the vertical reaction is observed greater for the right foot than for the left one in average. This means that the subject uses the right foot to a higher extent during the experiment. Consequently, the assumption of equally distribution on both feet is not valid. Nevertheless, the calculation shows that the global vertical reaction is quite well estimated in average. The use of the sagittal Matlab model

254

O. Bordron et al.

Fig. 5. Joint torques calculated by the Matlab model (green lines) and estimated by the Opensim model where left and right sides have been averaged (violet lines).

combined with the extracted kinematic data is in accordance with the global external forces measured. Under the assumption symmetric effort distribution, the calculated joint torques are the same at the left side of the human model than at the right one.

Human Squat Motion: Joint Torques Estimation with a 3D Model

5

255

Conclusion

The results show a coherence between a 3D model and a 2D model for a squat motion. It highlights a variability according to the anthropometric data used. From the 2D model, it is possible to solve the inverse dynamic model by making an assumption on the distribution of forces. These results underline the interest of using a planar model for a squat movement. The perspective is to investigate the influence of the load represented by the weight of a knee prosthesis.

References 1. Aoustin, Y.: Walking gait of a biped with a wearable walking assist device. Int. J. Humanoid Robot. 12(2), 1550018 (2015) 2. Aoustin, Y., Formalskii, A.: Walking of biped with passive exoskeleton: Evaluation of energy consumption. Multibody Syst. Dyn. 43(1), 71–96 (2018) 3. Bonato, P., Cheng, M.S., Gonzalez-Cueto, J., O’Connor, A.L.J., Roy, S.H.: EMGbased measures of fatigue during a repetitive squat exercice. IEEE Eng. Med. Biol. Mag. 20(6), 133–143 (2018) 4. Bonnet, V., Mazz´ a, C., Fraisse, P.: Real-time estimate of body kinematics during a planar squat task using a single inertial measurement unit. IEEE Trans. Biomed. Eng. 60(7), 1920–1926 (2013) 5. Cusimano-Reaston, M.R., Carney, B.: Legal changes necessitate proactive management of musculokeletal disorders: The role of electrodiagnostic functional assessment soft tissue management program. In: Proceedings of 33rd Annual International Conference of IEEE EMBS, Boston, Massachusetts, USA, pp. 7570–7573. (2011) 6. Durandau, G., Farina, D., Sartori, M.: Robust real-time musculoskeletal modeling driven by electromyograms. IEEE Trans. Biomed. Eng. 65(3), 556–564 (2018) 7. Hwang, S., Kim, Y., Kim, Y.: Joint kinetics and lumber curvatures during symmetric lifting: squat and stoop. In: Proceedings of International Conference on BioMedical Enginneering and Informatics, Sanya, Hainan, China, vol. 8, pp. 818– 822 (2008) 8. Kazerooni, H., Steger, R.: The Berkeley lower extremity exoskeleton. Trans. ASME J. Dyn. Syst. Meas. Control 128(1), 14–25 (2006) 9. Page, E.S.: Continuous inspection schemes. Biometrika 41(1-2), 100–115 (1954) 10. Rajagopal, A., Dembia, C.L., DeMers, M.S., Delp, D.D., Hicks, J.L., Delp, S.L.: Full-body musculoskeletal model for muscle-driven simulation of human gait. IEEE Trans. Biomed. Eng. 63(10), 2068–2079 (2016) 11. Wei, S., Zhou, H., Li, X., Wang, C.: Kinematics of lower extremity for the young and elderly chinese population during squatting. In: Proceedings of International Conference on BioMedical Enginneering and Informatics, Dalian, China, pp. 578– 582 (2014) 12. Winter, D.A.: Biomechanics and Motor Control of Human Movement. John Wiley and Sons, Hoboken (2009) 13. Yusof, A.S., I.Che-Ani, A., Hussain, Z., Hamzah, N., Boudville, R., Rahman, M.: Back-drivability of powered knee orthosis for knee free swing and knee extension. In: Proceedings of 7th IEEE International Conference on Control Systems, Computing and Engineering (ICCSCE 2017), Penang, Malaysia, pp. 331–335 (2017)

Visuo-Otolithic and Electrodermal Interactions in Experimental 3D Environments Irini Giannopulu1(B) , A. Pisla2 , and D. Pisla2 1 ICAM (Interdisciplinary Centre for the Artificial Mind), FSD, Bond University, Gold Coast,

Australia [email protected] 2 Design Engineering and Robotics, Universitaa Tehnica Cluj-Napoca, Cluj-Napoca, Romania {adrian.Pisla,doina.pisla}@muri.utcluj.ro

Abstract. Among the components of the ANS (Autonomic Nervous System), electrodermal activity variations (EDA) are less involved in self-motion. However, from neuroanatomical point of view, EDA variations are essentially modulated by sympathetic endings that are interconnected with the vestibular system. The aim of the present study was to analyse the latencies for reporting vertical self-motion (upward and downward) and the variations of electrodermal activity in healthy adults (17 men and 18 women aged 22 years old on average; sd 1 year and 2 month) when they were visually exposed to an experimental 3D environment via a head mounted display (HMD). The results revealed that the otolithic saccular maculae contribute differently to latencies and electrodermal activations. When self-motion was towards earth gravity, latencies and electrodermal activity were shorter in downward self-motion than in upward self-motion. We suggest that a top-down organisation connected with the visuo-otolithic information along the vertical axis, which essentially contributes to self-position, equilibrium and bodyconsciousness, could provide a neuromorphic model to improve space location and representation of humanoid robots. Keywords: Visuo-otolith interaction · Vertical self-motion · Electrodermal activity · Healthy participants

1 Introduction Visuo-vestibular cooperation, both visuo-canalar and visuo-otolithic, offers the frame of reference for the body position, orientation and navigation in 3D gravitational space. Visuo-otolithic interaction in particular, provides a scheme for the synergistic integration of linear acceleration visually induced along the three motion axes: sagittal, lateral and vertical [1]. This interaction refers to a complex neurophysiological phenomenon which necessitates the involvement of central and peripheral autonomic correlates such as heart rate, respiration and electrodermal activity at a minimum. With the present paper, we aim to analyse the visuo-otolithic interaction of vertical self-motion via the investigation of latencies and the variations of electrodermal activity. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 256–264, 2021. https://doi.org/10.1007/978-3-030-58104-6_29

Visuo-Otolithic and Electrodermal Interactions in Experimental

257

From a neuroanatomical viewpoint, the visuo-otolithic interaction has privileged relationships with the brainstem and more particularly with the cerebellum. The gravisensing otolithic organs are directly and indirectly interconnected. These organs also make connections with various cerebellum areas and more specifically with the floculo-nodular lobe, which essentially constitutes the vestibular cerebellum [2]. As per nature and function, the cerebellum is an important structure for motion, body-earth coordination and body-earth position [3]. It has been consistently reported that visuovestibular afferents transported via the vestibule-thalamic pathways are analysed and integrated into the frontal, temporal, parietal and prefrontal cortices [4]. It seems that the vestibular projection area is associated with the neural activity of the somatosensory cortex, the primary and premotor cortex and the cingulate cortex [5]. This visuo-vestibular and cerebellar pathway is participating in adjusting changes in gravitational environment and plays a critical role in the egocentric perception in both static and dynamic conditions [6]. The visuo-otolithic interaction is also involved in the regulation of the autonomic system (i.e. ANS). This is due to the fact that the ANS is structurally composed of visceral afferent pathways whose integration areas are the medulla oblongata, the brainstem, the thalamus and the hypothalamus and cerebral cortices [7]. As such, if visuo-otolith afferents are involved in the regulation of ANS during self-motion, it would be expected that modifications in ANS components would occur during otolithic modifications. Among the components of the ANS, electrodermal activity (EDA) variations are less involved in self-motion. However, from neuroanatomical point of view, EDA variations are essentially modulated by sympathetic endings [8] that are interconnected with the vestibular system. It has been reported that the stimulation of vestibular fibers adjusts the activity of sympathetic fibers and regulates somesthetic inputs such as, the heart rate (i.e. HR) and/or the electrodermal activity (i.e. EDA). For instance, overstimulation of the vestibular system that generally provokes motion sickness significantly modifies EDA by increasing its activity. Predominately controlled by central motor commands, static and dynamic motion increases EDA [9], and is paralleled by cardiovascular changes. There is no direct evidence of EDA modifications in relation to self-motion, but based on neuroanatomical arguments, it seems reasonable to suggest that EDA activity would be significantly affected by the otolithic variations. With regard to visuo-vestibular interaction and self-motion, it has been reported that any decrease of vestibular canals input facilitates self-motion by reducing its latencies [10–12]. Not only do patients with bilateral or right labyrinthine damage have shorter latencies for reporting self-motion than healthy people, but healthy people report shorter latencies of self-motion in weightlessness or in microgravity than in earth gravity [11]. When only vestibular otoliths and saccular maculae in particular are considered, this latter behaves differently according to its relationship towards or away from the earth’s gravity. The less the saccular maculae are involved, the faster the self-motion induced. In other words, downward self-motion, is more easily induced than upward self-motion [10]. In the present study, we analyse the latencies for reporting vertical self-motion (upward and downward), and the electrodermal activity in healthy adults when they are visually exposed to an experimental environment via a head mounted display (HMD).

258

I. Giannopulu et al.

It is expected that if the latency for reporting self-motion and electrodermal autonomic activity are under visuo-otolithic influence, significant modifications would be observed accordingly. In other words, both latency and electrodermal activity would be “vestibuloform”.

2 Method 2.1 Participants The study sample of the study was composed of forty healthy young adults (17 men and 18 women) aged 22 years old on average (sd 1 year and 2 months). They were all exposed to an experimental optokinetic stimulation via a Head Mounted Display (HMD). This stimulation was able to induce whole upward vs downward self-motion (along the vertical axis, i.e. Z axis). All participants declared normal or corrected-to-normal vision and were free of vestibular, cardiac or neurological disorders including spatial perception and orientation. Five participants (three women and two men) were excluded due to dropping out (three participants) or being nonsensitive to self-motion (two participants). The study was approved by the local ethics committee and was conformed to the declaration of Helsinki 2.0. 2.2 Experimental Apparatus The optokinetic stimulation was displayed by a head-mounted display (HMD, SONY HMZ-T1) whose visual field was 51.6° diagonal at full overplay (45° H × 25.3° V). The stimulation was a 3D black-white lamellar flux combined with a pseudorandom dot pattern [10]. Its spatial frequency was about 0.08 cycle deg−1 and its vertical oscillation about 0.6 Hz [13]. As per our previous similar study, the optokinetic environment was controlled by a PC computer. We used two response devices: a key response to record latency for reporting self-motion and an electrodermal device. Specifically, the Affectiva Q Sensor bracelet was considered to record electrodermal activity of the skin. The Q Sensor registered electrodermal activity at 32 Hz sample rate. This recording involved a tiny amount of direct electric current between two electrodes in contact with the skin [14]. The unit of measurement was microSiemens (µS). 2.3 Procedure The participants entered the experimental room where they carried the HMD. The Affectiva Q sensor was put on the participant’s right hand; the key response was placed on the participant’s right hand (Fig. 1a). As in the previous experiment [10] the participants performed several trials. A typical trial is described hereafter. The participants were comfortably seated in an armchair, their vertical axis was aligned with gravitational force. All participants were told to orient their head straight ahead. As soon as the head was in the correct position and after the participants declared that they were ready, the visual environment was presented on the HMD’s screen, and the participants were told to fixate on a central point. The participants performed two phases: a selection and an experimental phase.

Visuo-Otolithic and Electrodermal Interactions in Experimental

259

2.3.1 Selection Phase This phase was conceived in order to familiarise the participants with the procedure and to verify their sensitivity to vertical self-motion. The participants were told that the visual environment would move and that during its motion they could be (or not) in self-motion upward or downward. They were instructed to click on the key response as soon as they felt their own motion (and not to click on the key in the opposite situation). The order of visual directional (upward and downward) was randomised across the participants. Each trial lasted 60 s (1 min), with an inter-trial interval of around 5 s. Within each visual direction, a trial started as soon as the participants reported that they were ready. This phase was completed if the participants provided two correct consecutive trials. A trial was considered as being correct if the participant provided a click response and gave a formal explanation of self-motion. Only the subjects who provided two correct consecutive trials were counted for the experimental phase. 2.3.2 Experimental Phase During the experimental phase, the participants were immersed in the same conditions as previously. All participants were given six trials for each condition in a randomised order whose inter-trial interval was about 5 s. The participants were verbally redirected to align their head and gaze straight ahead. Once again the participants were instructed to click on the key response only as soon as they felt in self-motion. A total number of twelve trials (6 trials x 2 directions) was given to each participant, each trial lasted 1 min. At the end of the experiment, the participants were invited to verbalise their impressions. 2.4 Statistical Analysis and EDA Preprocessing Two dependent variables were considered a) the latency for reporting vertical selfmotion; b) the electrodermal activity across the skin. For the statistical analysis we used SPSS 24.0. Linearity was not met after visual inspection of a scatterplot matrix.

Fig. 1. Head Mounted Display (a) and visual environment (b) [reproduced from Giannopulu et al. 2015].

260

I. Giannopulu et al.

Similarly, the assessment of sphericity via Mauchly’s test gave a significant value of 0.544 indicating a violation of the assumption. Normality was therefore assessed via Shapiro-Wilk test. All data distributions resulted in a significance level of less than 0.05 indicating a violation of the assumption of normality. Based on the above, the Wilcoxon rank test was accomplished to compare latencies of reporting self-motion and electrodermal activity between upward and downward directions, and rho Spearman to assess the correlation between latencies and electrodermal activity. Finally, to filter out noise from raw EDA data, a low-pass filter set at 10 Hz was utilised.

3 Results Only the selected participants were considered, that is, all the participants who reported two correct consecutive trials (n = 35). Note that none of the participants experienced motion sickness. We analysed the differences between upward and downward selfmotion and the associated electrodermal activity as well as the correlations between the variables. With respect to the intensity of the visual-vestibular interaction, essentially due to the vestibular involvement along the vertical axis, we hypothesised that the electrodermal activity would be higher for upward than downward self-motion perception. Figure 2 illustrates (a) the latency for reporting upward and downward self-motion perception for the one side, and (b) and the electrodermal activity (µS) in relation with upward and downward self-motion perception from the other side. The results shown that the median latency for upward self-motion perception was higher than for downward self-motion perception (Wilcoxon T = 5, p < 0.001. In addition, the median electrodermal activity was higher in the upward than in the downward self-motion perception (Wilcoxon T = 111, p = .002).

Fig. 2. (a) Latency (b) electrodermal activity for downward and upward self-motion

Visuo-Otolithic and Electrodermal Interactions in Experimental

261

If the intensity of the visuo-vestibular interaction affects both the latency of reporting self-motion (downward vs upward) and the electrodermal activity, both variables would be interdependent. At correlational level, a positive relationship was observed between latency of self-motion perception and electrodermal activity for upward self-motion perception (Spearman rho = .974, p < 0.001) (Fig. 3a), i.e. the more the electrodermal activity, the more the latency for reporting self-motion. In contrast, negative relationship for downward self-motion perception was observed (Spearman rho = –.371, p < 0.001), i.e. the more the electrodermal activity, the less the latency for reporting self-motion (Fig. 3b).

Fig. 3. Correlation between electrodermal activity and latency for reporting (a) upward and (b) downward self-motion.

262

I. Giannopulu et al.

4 Discussion The aim of the present study was to analyse the modifications of electrodermal activity in a context of self-motion perception along the vertical axis in healthy participants in earth gravity. The findings indicated significant differences between 1) the latency for reporting downward self-motion and electrodermal activity and 2) significant correlations between latency and electrodermal activity. As such, the results are coherent with our hypotheses which anticipated that the magnitude of the visuo-otolithic interaction would affect the duration time for reporting vertical self-motion and the variation of electrodermal activity. Note that the self-motion perception experience reflects specific incoherence between visual and vestibular information. From the one side, vestibular information indicates self-immobility and the consequent visual information indicates self-motion. According to Zacharias and Young (1981) model [15], in such situations, a visually signal of reference is generated that corresponds to the vestibular dynamics that would be expected if a person was really in movement in 3D terrestrial space. As a result, the visual-vestibular information would correspond to the difference between the ongoing dynamics of the vestibular information that corresponds to self-immobility and the expected vestibular dynamic information associated with self-motion. With regard to the latency for reporting self-motion, it has been constantly reported that in normal earth conditions, healthy participants experience shorter self-motion latencies along the vertical rather than sagittal axes [10]. Such inter-axis difference is understood as being regulated by the intensity of the visuo-vestibular interaction [11, 12]. More particularly, visuo-vestibular intensity is weaker in the vertical than in the sagittal axis and this is because of the alignment of the vertical axis with earth gravity. If we now transpose such assumption into the vertical axis, it would signify that reduced vestibular information would reduce self-motion latency. In other words, due to the decrease of visuo-vestibular intensity, self-motion towards earth gravity would require shorter latency than self-motion away from earth gravity. By demonstrating a significant intraaxis difference in latencies for reporting vertical self-motion, our findings are coherent with several studies according to which vestibular modifications, in gravity or microgravity conditions, modify the latencies for reporting self-motion [16–19]. All in all, the latency for reporting vertical sea-motion appears to be “vestibuloform”. The results also show that autonomic electrodermal activity is modified with respect to the self-motion direction. It was lower for downward than for upward self-motion. Such results are coherent with our hypothesis that the autonomic activity and more precisely, the electrodermal activity, would be “vestibuloform” specifically because of its interconnections with the vestibular -otolithic system. Indeed, the electrodermal activity refers to electrical modifications of the skin as a reaction to brain signals. Several studies have reported changes in electrodermal activity during spatial navigation and motion sickness [20, 21]. Moreover, physiological experiments have revealed that during passive or active movement, inputs from otolith organs contribute to the control of autonomic activity (i.e. both electrodermal and heart rate) [22]. In that context, if the magnitude of the visuo-vestibular interaction is modified, the electrodermal activity would be affected too. Similarly, any attenuation of the vestibular input, in the case of a unilateral or bilateral labyrinthine disease [16, 23, 24], produces a reduction in electrodermal activity [25]. It is also established that the development of motion sickness leads to the disintegration of

Visuo-Otolithic and Electrodermal Interactions in Experimental

263

the autonomic nervous system balance due to the activation of the sympathetic activity, which in turn increases the electrodermal activity. The adaptation to motion sickness restores electrodermal activity balance [26, 27].

5 Conclusion With the above in mind, it can be concluded that during self-motion along the vertical axis, the otolithic saccular maculae contribute differently to latencies and electrodermal activation. When self-motion is towards earth gravity, downward self-motion latencies are shorter than upward self-motion latencies. Analogously, the electrodermal autonomic activation follows a similar pattern: it is lower for downward than for upward self-motion. Our findings are in line with several data which suggest that under normal terrestrial conditions, vertical self-motion is rapidly performed and significantly involves electrodermal decrease when alined towards gravitational acceleration. Given the implication of visuo-otolithic and electrodermal interactions to self-motion, balance and body-consciousness and in accordance with our previous studies where significant interactions were also observed between the visuo-vestibular system and another component of the autonomic system, i.e. heart rate [10, 23, 24], we suggest that the dynamic visuo-autonomico-vestibular multimodal interaction should be integrated into the humanoid robot navigation system. Here, the assumption is that such a top-down integration would improve the sensing models and algorithms that can contribute, in turn, to improve the space location and representation of humanoid robots. Acknowledgments. We are thankful to all the participants, to Gilles Ratureau, Pierre Leboucher and Roland Jouvent for their technical support, and to the CNRS and the French National Department of Education and Research for their scientific support.

References 1. Howard, I.P.: The vestibular system. In: Boff, K.R., Kaufman, L., Thomas, J.P. (eds.), Handbook of Perception and Human Performance, vol. I, pp. 11–3–11-26. Wiley, New York (1986) 2. Kawamura, A., Patel, K., Stotter, J., Noori-Hoshyar, A., Marty, B., Giannopulu, I.: A Neurootolithic Mathematical Approach of Self-motion Mental Imagery (2020). (submitted) 3. Ito, M.: Cerebellar circuit as a neuronal machine. Prog. Neurobiol. 78, 272–303 (2006) 4. Deutschländer, A., Bense, S., Stephan, T., Schwaiger, M., Dieterich, M., Brandt, T.: Rollvection versus linearvection: comparison of brain activations in PET. Hum. Brain Mapp. 21(3), 143–153 (2004) 5. Pfeiffer, C., Serino, A., Blanke, O.: The vestibular system: a spatial reference for bodily self-consciousness. Front. Integr. Neurosci. 8, 31 (2014) 6. Indovina, I., Maffei, V., Pauwels, K., Macaluso, E., Orban, G.A., Lacquaniti, F.: Simulated self-motion in a visual gravity field: sensitivity to vertical and horizontal heading in the human brain. Neuroimage 71, 114–124 (2013) 7. Becker-Bense, S., Buchholz, H.G., zu Eulenburg, P., Best, C., Bartenstein, P., Schreckenberger, M., Dieterich, M.: Ventral and dorsal streams processing visual motion perception FDG-PET study. BMC Neurosci. 16, 13–81 (2012)

264

I. Giannopulu et al.

8. Shields, S.A., Macdowell, K.A., Fairchild, S.B., Campbell, M.L.: Is mediation of sweating cholinergic, adrenergic, or both. A comment on the literature. Psychophysiology 24, 312–319 (1987) 9. Chritchley, H.D.: Electrodermal responses: what happens in the brain. Neuroscientist 8, 132– 142 (2002) 10. Giannopulu, I., Leboucher, P., Rautureau, G., Israël, I, Jouvent, R.: Spatial Vertical Navigation in Healthy Adults. New Trends in Medical and Service Robots, pp. 101–112. Springer, Cham (2015) 11. De Saedeleer, C., Vidal, M., Lipshits, M., Bengoetxea, A., Cebolla, A.M., Berthoz, A., et al.: Weightlessness alters up/down asymmetries in the perception of self-motion. Exp. Brain Res. 226, 95–106 (2013) 12. Giannopulu, I., Lepecq, J.C.: Linear vection chronometry along spinal and sagittal bodymotion. Perception 27, 363–449 (1998) 13. Wright, G.W.: Linear vection in virtual environments can be strengthened by discordant inertial input. In: 31th Annual International Conference of the IEEE EMBS Minneapolis, Minnesota, USA, 2–6 September, pp. 1157-1160. (2009) 14. Picard, R.W., Fedor, S.: Ayzenberg, Y.: Multiple arousal theory and daily-life electrodermal activity asymmetry. Emot Rev. 8, 62–75 (2016) 15. Zacharias, G.L., Young, L.R.: Influence of combined visual and vestibular cues on human perception and control of horizontal rotation. Exp. Brain Res. 41, 159–171 (1981) 16. Cheung, B.S.K., Howard, I.P., Nedzelski, J.M., Landolt, J.P.: Circular vection about horizontal axes in bilateral labyrinthine-defective subjects. Acta Otolaryngol (Stockh) 108, 336–344 (1989) 17. Wong, S.C.P., Frost, B.J.: The effect of visual-vestibular conflict on the latency of steady-state visually induced subjective rotation. Percept. Psychophys. 30(3), 228–236 (1981) 18. Young, L.R., Shelhamer, M.: Weightlessness enhances the relative contribution of visuallyinduced self motion. In: R. Warren, Wertheim, A.H. (eds.) Perception and Control of SelfMotion, pp. 523–538. Erlbaum, Hillsdale (1990) 19. Cheung, B.S.K., Howard, I.P., Money, K.E.: Visually-induced tilt during parabolic flights. Exp. Brain Res. 81, 391–397 (1990) 20. Yates, B.J., Bolton, P.S., Macefield, V.G.: Vestibulo-sympathetic responses. Compr Physiol. 4(2), 851–887 (2014) 21. Westmoreland, D., Krell, R.W., Self, B.P.: Physiological responses to the Coriolis illusion: effects of head position and vision. Aviat. Space Environ. Med. 78, 985–999 (2007) 22. Chen, W., Chao, J.G., Chen, X.W., Wang, J.K., Tan, C.: Quantitative orientation preference and susceptibility to space motion sickness simulated in a virtual reality environment. Brain Res. Bull. 113, 17–26 (2015) 23. Giannopulu, I.: Visuo-vestibular and somesthetic contributions to spatial navigation in children and adults. In: Pissaloux, E., Velázquez, R. (eds.) Mobility of Visually Impaired People, pp. 201–233. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-54446-5_7 24. Giannopulu, I.: Neuroscience, Rbotics and Virtual Reality: Internalised vs Externalised mind/brain. Springer, Cham (2018) 25. Ray, C.A., Monahan, K.D.: Aging attenuates the vestibulosympathetic reflex in humans. Circulation 105(8), 956–961 (2002) 26. Hu, S., Grant, W.F., Stern, R.M., Koch, K.L.: Motion sickness severity and physiological correlates during repeated exposures to a rotating optokinetic drum. Aviat. Space Environ. Med. 62, 308–314 (1991) 27. Wood, S.J., Reschke, M.F., Sarmiento, L.A., Clement, G.: Tilt and translation motion perception during off-vertical axis rotation. Exp. Brain Res. 182, 365–377 (2007)

Development and Characterization of a Versatile, Force-Range Adjustable, Low-Cost, Tri-Axial Force Sensor Ivan Suši´c1(B) , Philippe C. Cattin2 , Raphael Guzman3 , and Georg Rauter1 1 Bio-Inspired RObots for MEDicine-Lab (BIROMED-Lab), Department of Biomedical

Engineering, University of Basel, Gewerbestrasse 14 (Room 12.03.008), 4123 Allschwil, Switzerland {ivan.susic,georg.rauter}@unibas.ch 2 Center for Medical Image Analysis and Navigation (CIAN), Department of Biomedical Engineering, University of Basel, Gewerbestrasse 14, 4123 Allschwil, Switzerland [email protected] 3 Department of Neurosurgery, University of Basel, Hebelstrasse 20, 4031 Basel, Switzerland [email protected]

Abstract. Three-axial force sensors enable quantification of 3D interaction forces (one normal and two shear components), which appear when a sensor gets in physical contact with an object. Typically, these sensors are used in applications such as minimal-invasive palpation, closed-loop force control, or safety, where the information on quantified interaction forces bring beneficial information to a human or a system. However, general-purpose industrial tri-axial force sensors may be costly (i.e. range of several hundred or thousand CHF). To overcome this drawback, we developed a tri-axial force sensor that costs more than ten times less compared with the commercially available sensors of similar size and specifications. In this paper, we present a versatile, force-range adjustable, low-cost, tri-axial force sensor. The tri-axial force sensor was developed using a rapid prototyping technique where most mechanical components are 3D printed. The sensor houses three low-cost, unidirectional force sensors, washer-like and conical shape-like silicone structures for enabling mechanical compliance, and a force-range adjustment mechanism. A multipurpose testing machine equipped with a 2 kN load cell and an industrial force-torque sensor Mini45 were used to characterize the sensor’s properties. With this set-up, we characterized the properties of the lowcost tri-axial force sensor, which may be used as a substitution for costly sensors currently available on the market in applications where similar force ranges, but lower resolution and accuracy are sufficient. Keywords: Force sensing · Three-axial force sensor · Force-range adjustment

I. Suši´c and G. Rauter—Contributed equally. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 G. Rauter et al. (Eds.): MESROB 2020, MMS 93, pp. 265–272, 2021. https://doi.org/10.1007/978-3-030-58104-6_30

266

I. Suši´c et al.

1 Introduction Force sensors are devices that quantify interaction forces when they get in physical contact with their surroundings. They have been developed for different applications and therefore have distinct properties that can be used for sensor classification such as the number of measured force directions, physical measurement principle, force range, sensitivity, size, overload capacity, or sealing. Typical fields, where force sensors are used are human-robot interaction [1] robotics [2] or biomedical devices [3, 4]. Information on interaction forces is particularly important for systems that require safe interaction with the environment, such as human-machine interaction [5]. In many respective applications, the knowledge on normal and share forces is required, e.g. to detect slipping of grasped objects [6], or measure crucial shear forces when manipulating an endoscope [7]. Implementation of multidirectional force sensors may increase the costs of a system, so we used rapid prototyping technologies to develop a low-cost three-dimensional force sensor that can replace expensive commercially available force sensors. A force sensor able to quantify interaction forces in longitudinal and both transversal directions is called a tri-axial force sensor. The core of a tri-axial force sensor consists of force-sensing elements which are the basic units for force-sensing. The main task of force-sensing elements is the transformation of a deformation caused by mechanical stress into an electrical signal which is then processed and provided to a user or a system. The working principle of force-sensing elements, thus force sensors, can be based on different force-sensing technologies [8]. However, most of the force sensor technologies have in common that the manufacturing of highly accurate mechanical parts, the precise placement of the sensing elements with respect to position and orientation on the mechanical structure, and the calibration of the sensor are costly. In many applications, however, a three-axial version of a low-cost force sensor with less sensitivity and resolution but a similar force range would be highly appreciated. Example applications can be robot-assisted arm [1] or gait rehabilitation [9], where force-sensing improves the transparency of the human-robot interaction. To overcome the problem particularly related to hardware costs, we developed a low-cost, multipurpose, Tri-axial Force Sensor (TFS). To do so, we used rapid prototyping technology, where all the mechanical parts are 3D printed or molded. Besides, we integrated three unidirectional low-cost force-sensing elements. The main idea was to develop a low-cost replica of the commercially available sensor Mini45 (ATI Industrial Automation, Apex, USA), which measures all three forces but also all three torques. However, the current version of the TFS measurement of torques is not implemented yet. In this paper, we present the mechanical design of the self-developed tri-axial force sensor, the testing setup, and the results from the characterization experiments that revealed the main properties of the TFS. In the end, we compared the properties of the TFS in the three force directions with those of the commercially available six degrees of freedom force/torque sensor Mini45.

Development and Characterization of a Versatile

267

2 Materials and Methods A CAD software SolidWorks 2018 (Dassault Systèmes SolidWorks Corporation, Waltham, Massachusetts, USA) was used to design the mechanical parts which were then 3D printed with a Stratasys FORTRUS 250mc (Stratasys, Ltd. Eden Prairie, Minnesota, USA), using the standard thermoplastic material ABS P430. The washer-like silicone structures for enabling mechanical compliance were molded using silicones SF-45 2k A45 and SF-33 2k (Silikon Fabrik, Bad Schwartau, Germany) - stiffness Shore A45 and Shore 33, respectively. For a force measurement inside the TFS, we used three commercially available, unidirectional force sensors HSFPAR303A (ALPS Co, Japan). The main characteristics of these sensors are price (6.53 CHF/piece), size (2.0 × 1.6 × 0.66 mm3 ), force range (0–8 N), overload protection (up to 200 N), high-sensitivity (3.7 mV/N), and linearity (