March 2022 IEEE Robotics and Automation Magazine

Citation preview

Displaying the world in your hands. Inventing new ways to interact. Force Dimension designs and manufactures the finest master haptic devices for leading-edge applications in research, medical, industry, and human exploration.

The ScanTrainer from Intelligent Ultrasound is an educational tool that uses real patient scans and curriculum-based teaching across obstetrics, gynecology, general medicine, and emergency medicine. The system integrated a customized delta.3 haptic device. Force Dimension Switzerland www.forcedimension.com [email protected]

Vol. 29, No. 1 MARCH 2022 ISSN 1070-9932 http://www.ieee-ras.org/publications/ram

FEATURES 10

Augmented Reality-Assisted Reconfiguration and Workspace Visualization of Malleable Robots Workspace Modification Through Holographic Guidance By Alex Ranne, Angus B. Clark, and Nicolas Rojas

22

Proactive Robot Assistance

35

A Tool for Organizing Key Characteristics of Virtual, Augmented, and Mixed Reality for Human–Robot Interaction Systems

Affordance-Aware Augmented Reality User Interfaces By Rodrigo Chacón Quesada and Yiannis Demiris

(a)

Synthesizing VAM-HRI Trends and Takeaways By Thomas R. Groechel, Michael E. Walker, Christine T. Chang, Eric Rosen, and Jessica Zosa Forde

45

Spatial Computing and Intuitive Interaction

Bringing Mixed Reality and Robotics Together By Jeffrey Delmerico, Roi Poranne, Federica Bogo, Helen Oleynikova, Eric Vollenweider, Stelian Coros, Juan Nieto, and Marc Pollefeys

58

ARviz

68

A Study on the Dexterity of Surgical Robotic Tools in a Highly Immersive Virtual Environment

An Augmented Reality-Enabled Visualization Platform for ROS Applications By Khoa C. Hoang, Wesley P. Chan, Steven Lay, Akansel Cosgun, (b) and Elizabeth A. Croft

Assessing Usability and Efficacy By Andrea Danioni, Gulfem Ceren Yavuz, Defne Ege Ozan, Elena De Momi, Anthony Koupparis, and Sanja Dogramadzi

76 ON THE COVER This special issue of IEEE Robotics and Automation Magazine explores the recent advances in AR, VR, and MR for human–robot interaction (HRI) in the field of robotics.

Metrics for 3D Object Pointing and Manipulation in Virtual Reality

The Introduction and Validation of a Novel Approach in Measuring Human Performance By Eleftherios Triantafyllidis, Wenbin Hu, Christopher McGreavy, and Zhibin Li

©SHUTTERSTOCK.COM/KDDESIGNPHOTO

If you like an article, click this icon to record your opinion. This capability is available for online Web browsers and offline PDF reading on a connected device. Digital Object Identifier 10.1109/MRA.2022.3143272

MARCH 2022

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

1

A Publication of the IEEE ROBOTICS AND AUTOMATION SOCIETY Vol. 29, No. 1 March 2022 ISSN 1070-9932 http://www.ieee-ras.org/publications/ram

EDITORIAL BOARD Editor-in-Chief Yi Guo ([email protected]) Stevens Institute of Technology (USA)

COLUMNS & DEPARTMENTS 4 FROM THE EDITOR’S DESK 6 PRESIDENT’S MESSAGE 8 FROM THE GUEST EDITORS 92 INDUSTRY ACTIVITIES 99 TC SPOTLIGHT 102 COMPETITIONS 108 SOCIETY NEWS 120 EDUCATION 124 CALENDAR

Editors Elena De Momi Politecnico di Milano (Italy) Jindong Tan University of Tennessee (USA) Associate Editors Ming Cao University of Groningen (The Netherlands) Feifei Chen Jiao Tong University (China) Carlos A. Cifuentes Universidad del Rosario (Colombia) Kingsley Fregene Lockheed Martin (USA) Antonio Frisoli Scuola Superiore Sant’Anna (Italy) Jonathan Kelly University of Toronto (Canada) Ka-Wai Kwok The University of Hong Kong (Hong Kong) Xiang Li Tsinghua University (China) Perla Maiolino University of Cambridge (UK) Surya G. Nurzaman Monash University (Malaysia) Weihua Sheng Oklahoma State University (USA) Yue Wang Clemson University (USA) Enrica Zereik CNR-INM (Italy) Houxiang Zhang Norwegian University of Science and Technology (Norway) Past Editor-in-Chief Bram Vanderborght Vrije Universiteit Brussel (Belgium) RAM Column Manager Amy Reeder (USA) RAM Editorial Assistant Joyce Arnold (USA) COLUMNS Competitions: Minoru Asada (Japan) and Yu Sun (USA) Education: Vacant From the Editor’s Desk: Yi Guo (USA) Industry News: Tamas Haidegger (Hungary)

2

•

Humanitarian Technology: Kaspar Althoefer (UK) Standards: Craig Schlenoff (USA) President’s Message: Frank Park (Korea) Regional Spotlight: Megan Emmons (USA) Student’s Corner: Francesco Missiroli (Germany) TC Spotlight: Yasuhisa Hirata (Japan) Women in Engineering: Lydia Tapia (USA) IEEE RAS Vice-President of Publication Activities Todd Murphey (USA) RAM home page: http://www.ieee-ras.org/publications/ram IEEE Robotics and Automation Society Executive Office Peter Sobel Executive Director Amy Reeder Program Specialist [email protected] Advertising Sales Mark David Director, Business Development—Media & Advertising Tel: +1 732 465 6473 Fax: +1 732 981 1855 [email protected] IEEE Periodicals Magazines Department Kristin Falco LaFleur Journals Production Manager Patrick Kempf Senior Manager Journals Production Janet Dudar Senior Art Director Gail A. Schnitzer Associate Art Director Theresa L. Smith Production Coordinator Felicia Spagnoli Advertising Production Manager Peter M. Tuohy Production Director Kevin Lisankie Editorial Services Director Dawn M. Melley Staff Director, Publishing Operations IEEE-RAS Membership and Subscription Information: +1 800 678 IEEE (4333) Fax: +1 732 463 3657 http://www.ieee.org/membership_services/ membership/societies/ras.html

Digital Object Identifier 10.1109/MRA.2022.3143274

IEEE prohibits discrimination, harassment, and bullying. For more information, visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

IEEE Robotics and Automation Magazine (ISSN 1070-9932) (IRAMEB) is published quarterly by the Institute of Electrical and Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA, Telephone: +1 212 419 7900. Responsibility for the content rests upon the authors and not upon the IEEE, the Society or its members. IEEE Service Center (for orders, subscriptions, address changes): 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ 08855 USA. Telephone: +1 732 981 0060. Individual copies: IEEE Members US$20.00 (first copy only), non-Members US$140 per copy. Subscription rates: Annual subscription rates included in IEEE Robotics and Automation Society member dues. Subscription rates available on request. Copyright and reprint permission: Abstracting is permitted with credit to the source. Libraries are permitted to

photocopy beyond the limits of U.S. Copyright law for the private use of patrons 1) those post-1977 articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without a fee. For other copying, reprint, or republication permission, write Copyrights and Permissions Department, IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854. Copyright © 2022 by the Institute of Electrical and Electronics Engineers Inc. All rights reserved. Periodicals postage paid at New York and additional mailing offices. Postmaster: Send address changes to IEEE Robotics and Automation Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ 08854 USA. Canadian GST #125634188 PRINTED IN THE U.S.A.

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

MARCH 2022

�creating a new robotic world

Looking for�the next robotic innovator The KUKA Innovation Award 2023 has some big news: for the first time, the competition will be based on our new iiQKA robot operating system and ecosystem. And in this spirit, we chose the new topic of the award: the Open Platform Challenge. This means that there are almost no limitations to what applicants can do this year. The selected teams will be provided with a KUKA LBR iisy and 3D vision sensor from Roboception to use as well as robotics training and coaching by KUKA, free of charge throughout the whole competition. After implementing their ideas, the teams can then present their results at a major trade fair, with the winner receiving € 20,000. The award is aimed at the wider robotics community and is open to developers, graduates, and research teams from companies or universities.

Apply now All details about the Challenge are available at www.kuka.com/InnovationAward2023

KUKA Innovation Award 2023 www.kuka.com/InnovationAward2023

FROM THE EDITOR’S DESK

Reality Check By Yi Guo

T

wo years into the pandemic, many places in the world experienced another wave of spiking COVID cases. People got used to remote shopping, work­ ing from home, and Zoom meetings more than ever. The world’s predict­ able unpredictability has become the new normal, and we are trying to live with a disease that has not yet settled into its The world’s endemic state. predictable In the race between virus unpredictability has evolution and become the new the science de­ velopment to normal, and we are tackle it, the ar­ trying to live with a tificial intel­ ligence (AI)disease that has not driven software yet settled into its developed for protein folding endemic state. by DeepMind, AlphaFold, stands out as a breakthrough in 2021, and it is used to model the effect of mutations in the omicron variant’s spike protein [1]. It is exciting to see that the “protein-folding problem,” a 50-year-old grand challenge in biol­ Digital Object Identifier 10.1109/MRA.2022.3143185 Date of current version: 22 March 2022

4

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

ogy, is now solvable by AI-powered algorithms. In the tech world, metaverse, a new world built in virtual reality (VR), became a buzzword as Facebook changed its name to Meta in 2021. While the history of VR can be dated back to the 1950s, the technology has evolved significantly over the past few years, benefitting from improve­ ments in hardware, software, AI, and cloud technologies. In 2021, the biggest semiconductor chip manufacturer, Intel, announced its vision to make the metaverse a reality through its plan for next-generation immersive Internet experienced via augmented reality (AR) and VR. To roboticists, VR, AR, and mixed reality are not just buzzwords but useful tools for telepresence and medical robots as well as many other robotics applications where enhanced human– robot interaction is desired. Advances in VR and AR will provide new tools and opportunities for robotics. On the other hand, robotics will offer challeng­ ing problems and emerging applica­ tions in the forefront of technology. This March issue of IEEE Robotics and Automation Magazine (RAM) includes exemplar research develop­ ment on extended reality in robotics. I am thankful to the guest editors, Mahdi

MARCH 2022

Tavakoli, Mark A. Minor, Jef­ frey Delmerico, Paul Chip­ pendale, and Giovanni Ros­ sini as well as the leading RAM editor, Elena De Momi, and assisting RAM associate editor (AE), Antonio Frisoli, for their great efforts promoting and editing this special issue. I would also like to express my grati­ tude and appreciation to the retiring RAM editorial board members Fabio Bonsignorio, Perla Maiolino, and Xiang Li for their dedicated service to the magazine. We have selected a few new AEs, Ming Cao, Houxiang Zhang, Car­ los A. Cifuentes, and Feifei Chen, to join the editorial board starting in Janu­ ary 2022. They were selected based on expertise, technical areas, and geo­ graphical representation. We hope to receive more nomina­ tions from professionals from industry and government as well as underrepre­ sented groups, including women. Nom­ inations (including self-nominations) are accepted through the RAM web­ site during the submission window between August 1 and September 1 each year. Enjoy the issue! Reference [1] R. F. Service, “Protein structures for all,” Science, vol. 374, no. 6574, pp. 1426–1427, 2021. doi: 10.1126/science.acz9822.

PRESIDENT’S MESSAGE

Greetings From the 2022 IEEE Robotics and Automation Society President By Frank Park

A

s I write my first editorial and reflect on past President Seth Hutchinson’s observation that we are in the middle of several inflection points in history, I can’t help but wonder where we will be in March when this article comes to press. To observe that we are still living in very trying times is now a tired cliché, and yet it’s hard to suppress the optimism that perhaps, just possibly, 2022 could finally be the year we return to some semblance of normal. It’s probably not a gross exaggeration to say that the outgoing executive committee endured two of the most trying years in the history of the Robotics and It may come as a Automation Sosurprise to some ciety (RAS), and I am deeply that, among the 46 grateful to everytechnical Societies one—President Seth Hutchinson, and councils in Secretary Paolo IEEE, we are now Fiorini, Treasurer Venkat Krovi, the fifth largest. Vice President (VP) Conference Activities Torsten Kroeger, VP Electronic Products and Services Zhidong Wang, VP Financial Activities Yasushi Nakauchi, VP Industrial Activities Robert Ambrose, VP Member Activities Nancy Amato, VP Publication Activities Aude Billard, VP Technical Activities Tony Maciejewski, and ParDigital Object Identifier 10.1109/MRA.2022.3144883 Date of current version: 22 March 2022

6

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

liamentarian Hong Zhang— for not only holding the ship together, but achieving so much under such difficult conditions. This included organizing and running our flagship IEEE International Conference on Robotics and Automation (ICRA) conference online with less than four months to prepare, making diversity and inclusion a core part of our identity and conduct, taking steps to increase our administrative support staff, seeking greater synergy between automation and robotics, launching an educational activities committee, and much more. The easy thing would have been to blame COVID-19 and simply “phone it in.” I thank everyone for their selfless dedication in serving our entire global community of roboticists. I’m also excited to welcome the incoming executive committee for 2022–2023: President-Elect Aude Billard of Ecole Polytechnique Federale de Lausanne, Secretary Katja Mombaur of the University of Waterloo, Treasurer Tony Maciejewski of Colorado State University, VP Conference Activities Paul Oh of the University of Nevada, Las Vegas, VP Electronic Products and Services Bram Vanderborght of Vrije University, VP Financial Activities Hiromi Mochiyama of the University of Tsukuba, VP Industrial Activities Andra Keay of Silicon Valley Robotics, VP Member Activities Stefano Stramigioli of the University of Twente, VP Publication Activities Todd Murphey of Northwestern University, VP Technical Activities Kyujin Cho of Seoul National

MARCH 2022

University, and Parliamentarian Patrick Wensing of the University of Notre Dame. All of the VPs have, in turn, recruited a large number of volunteers to serve their boards. To all who are committing so much of their personal time and effort in service to our Society, I extend my deepest gratitude. Over the next few months, I hope to communicate to you details of our ongoing efforts as well as some new initiatives we have planned for 2022–2023. Here is a brief preview. ●●  It may come as a surprise to some that, among the 46 technical Societies and councils in IEEE, we are now the fifth largest. Organizers of ICRA, our flagship conference, are now routinely asked to prepare for a minimum of 5,000 participants and often up to 8,000. Perhaps it’s because of the highly interdisciplinary nature of robotics, but I find that there is a distinct and common unifying thread to our culture, one of openness and inclusivity, of breaking down barriers not only in our work but also in how we engage with each other. One of the challenges we face during this phase of unprecedented growth is how to preserve this common culture. I believe an important key is to rejuvenate and empower our technical committees, to make them focal centers of interaction while continuing to maintain a large and inclusive RAS tent. By creating synergy among technical committees and also reengaging with communities

●

like automation that have drifted somewhat from robotics, my hope is that our conferences, publications, and member outreach activities can better respond to the diverse needs of our membership. Anyone who’s recently attended the annual Consumer Electronics Show (CES) in Las Vegas will probably agree that at least half of it is a consumer robotics show. I think it’s safe to say that the robotics and automation industry is also going through a series of inflection points, and that there is now considerable interest within our community in start-ups and entrepreneurship. At the same time, there is growing interest from industry in connecting with our members, seeking employees, con-

●

sultants, and business partners, and just keeping up with the latest technology and advances in robotics. Our industrial activities board, in partnership with our conference, technical activities, and electronic products and services boards, will be devising ways to allow our members to more actively engage with industry, through job fairs at our flagship conferences, RAS satellite events at CES, entrepreneurship events at global locations, and much more. If anything, the past three years have confirmed that online learning—in the form of massive open online courses, online classes, and various archived content—is here to stay. While the creators of the best robotics educational content are all within

our Society, that content is spread out over a disparate collection of platforms, with uneven accessibility at best. An important component of our Societal mission is education. As such, we will be investigating ways to provide the best robotics educational content to our community and beyond, developing any needed platforms and infrastructure, and offering incentives to our content creators. I very much look forward to serving you, RAS, and the broader community of roboticists over the next two years. I invite you to share with me your concerns, interests, and any other issues that you feel impact our members. Both personal communications and letters to the magazine are welcome.

ROBOTIC END - EFFECTORS

Measure all six components of force and torque in a compact, rugged sensor. Interface Structure

High-Speed Electronics

high-strength alloy provides IP60, IP65, and IP68 environmental protection as needed

interfaces for Ethernet, PROFINET, EtherNet/IP, Analog, USB, CAN EtherCAT, Wireless, and more

Sensing Beams and Flexures

Silicon Strain Gages

designed for high stiffness and overload protection protec ion withou without compromising resolutionn resolutio

provide high noise immunity, accuracy, and high factor-of-safety, standard on all F/T models

Engineered for high-performance and maximum stiffness, with the highest resolution and accuracy available, it’s the ultimate force/torque sensor. Only from ATI. www.ati-ia.com 919.772.0115

MARCH 2022

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

7

FROM THE GUEST EDITORS

Extended Reality in Robotics By Elena De Momi, Mahdi Tavakoli, Jeffrey Delmerico, Antonio Frisoli, Mark A. Minor, Giovanni Rossini, and Paul Chippendale

E

xtended reality (XR), which combines the real and virtual worlds, is greatly enhancing interaction possibilities between robots and humans, leading to a paradigm shift where the two entities can intuitively cooperate to perform shared-target tasks. Many XR devices are essentially performing the same spatial perception tasks as mobile robots (e.g., visual simultaneous localization and mapping), and thus XR provides an opportunity for robots and the humans using these de­­vices to colocalize through a common un­­ derstanding of their space, which also enables easier human–robot A key enabler interactions. XR interfaces of human–robot can be realized collaboration is through augmented reality the colocalization (AR), where an and shared spatial operator’s perception of the intelligence that AR real world is en­­ and MR can provide. hanced through the superimposition of virtual objects and in­­formation; virtual reality (VR), where the operator is immersed in a 3D virtual world; or mixed reality (MR), where the user can both see and interact with digital content that is superimposed over the real world. A key enabler of human–robot collaboration is the colocalization and shared Digital Object Identifier 10.1109/MRA.2022.3143186 Date of current version: 22 March 2022

8

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

spatial intelligence that AR and MR can provide. The COVID-19 pandemic has highlighted an unprecedented need for immersive platforms that can facilitate safer working conditions through telerobotic control. Health-care workers, for example, can use XR technologies to enable intuitive collaborations for remote perception and control, such as diagnosis, treatment planning, or patient monitoring. In this context, research efforts have received a significant push toward realizing fascinating promises in several application fields, with XR being a prime contender to offer immersive experiences, even from remote locations. For instance, they can enable immersive remote inspection in dangerous environments. This special issue explores the recent advances in AR, VR, and MR for human–robot interaction (HRI) in the field of robotics. In the first article, Ranne et al. propose a novel AR interface for controlling and visualizing a malleable robot arm. Because of the adaptive topology of the variable-stiffness links in these robots, different reconfigurations of the malleable link result in unique workspaces, which can be nonintuitive for the user to predict based just on the endeffector pose. The authors leverage AR to enable the user to manipulate a digital twin of the robot’s end effector to a desired position and then select a reconfiguration from several candidates whose workspaces are visualized through the head-mounted AR display, overlaid on the real world. The proposed

MARCH 2022

interface allows the user to accurately align the physical robot to the virtual one, resulting in the desired reconfiguration and workspace, without adding a significant workload for the user. In the second article, Quesada and Demiris propose an affordance-aware proactive planning, whereby an AR user interface proactively identifies user affordances, i.e., action options that are possible in the current context. The tool combines natural language processing and planning algorithms to provide the most relevant and feasible options based on the user’s context as well as to reduce the time needed for these options to be generated and presented to the user. The resulting affordance-aware AR user interfaces for robot control combine atomic actions and provide higher-level options to the user. In the third article, Groechel et al. apply the tool for organizing key characteristics of virtual, augmented, and mixed reality Technologies in human– robot interaction (VAM-HRI) framework to several papers presented in the fourth VAM-HRI workshop. The framework introduces a proper taxonomy and the interaction cube, which captures characteristics about the design elements involved (expressivity of the view and flexibility of control) and the virtuality they implement (from real to fully virtual). Metrics for systems evaluation are presented as well. In the fourth article, Delmerico et al. explore several novel applications of MR for HRI: mission planning for inspection, gesture-based control, and immersive teleoperation. Each of these

applications takes advantage of the egocentric sensing on MR devices to capture multimodal human inputs and leverages their onboard spatial computing capabilities to translate these signals into robot commands that utilize the spatial context. Their case studies illustrate how mixed reality offers promising new possibilities in HRI, whether from a shared spatial understanding between the robot and the user or the embodied control that immersive MR devices can provide for teleoperation. In the fifth article, Hoang et al. introduce their versatile, extendable AR visualization platform [developed in the Robot Operating System (ROS) framework] for robot applications. They provide a universal visualization platform with the capability of displaying ROS message data types in AR as well as a multimodal user interface for interacting with robots. Their solution offers a collection of plug-ins that

provides visualization and/or interaction capabilities, with the option for users to extend the platform to meet specific needs by implementing their own plug-ins. In the sixth article, Danioni et al. present a highly immersive virtual environment with simulated surgical tools with extended degrees of freedom with respect to the standard ones. The motion of the tools was teleoperated by placing inertial sensors on the user arms and hands. The novel interface was tested by users with different levels of expertise, performing tasks at varying levels of complexity. The majority of the users achieved better results using the articulated tools, but they required more physical and cognitive loads. In the final article, Triantafyllidis et al. introduce new metrics for 3D object pointing and manipulation in a VR environment to assess human motor performances. The metrics are

compared with others in the literature under different experimental settings. Additionally, the work presents recommendations for the design of pointThe COVID-19 ing and manippandemic has ulation tasks in VR and telehighlighted an operation. unprecedented It is a common thought of need for immersive the guest editors platforms that can that this special issue would be a facilitate safer starting point for working conditions further research in the field of through telerobotic XR to further excontrol. tend the human p o ssibi lit ies when interfacing with robotic devices, offering better controllability and safety of the interactions.

TAP. CONNECT. NETWORK. SHARE. Connect to IEEE–no matter where you are–with the IEEE App. Stay up-to-date with the latest news

Schedule, manage, or join meetups virtually

Get geo and interest-based recommendations

Read and download your IEEE magazines

Create a personalized experience

Locate IEEE members by location, interests, and affiliations

MARCH 2022

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

9

©SHUTTERSTOCK.COM/BLUE PLANET STUDIO

Augmented RealityAssisted Reconfiguration and Workspace Visualization of Malleable Robots Workspace Modification Through Holographic Guidance

By Alex Ranne, Angus B. Clark, and Nicolas Rojas

Digital Object Identifier 10.1109/MRA.2022.3141973 Date of current version: 28 January 2022

10

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

MARCH 2022

M

alleable robots are a type of reconfigurable serial robot capable of adapting their topology, through the use of variable stiffness malleable links, to desired tasks and workspaces by varying the relative positioning between their revolute joints. However, their reconfiguration is nontrivial, lacking intuitive communication between the human and the robot, and a method of efficiently aligning the end effector to a desired position. In this article, we present the design of an interactive augmented reality (AR) alignment interface, which helps a malleable robot understand the user’s task requirements, visualizes to the user the requested robot’s configuration and its workspace, and guides the user in reconfiguring the robot to achieve that configuration. Through motion tracking of a physical two degree-of-freedom (2 DoF) malleable robot, which can achieve an infinite number of workspaces, we compute the 1070-9932/22©2022IEEE

accuracy of the system in terms of initial calibration and overall accuracy, and demonstrate its viability. The results demonstrated a good performance, with an average repositioning accuracy of 9.64 ± 2.06 mm and an average base alignment accuracy of 10.54 ± 4.32 mm in an environment the size of 2,000 mm × 2,000 mm × 1,200 mm. Introduction The difficulty of designing, trajectory planning, and orientation control of a multi-DoF robot are common problems faced by researchers in task-oriented robot design [1]. Although these issues have been mitigated as robotics continues to evolve, with better computational tools developed to control manipulators with higher accuracy and repeatability implemented [2], one aspect that remains challenging is the development of solutions where a robot must complete tasks in collaboration with a user, instead of operating fully autonomously. Despite the progress made in the last decade with the deployment of collaborative robots [3]—and in human–robot interaction (HRI) research [4]—under an industrial setting, robots still excel at performing precise, accurate, and repetitive tasks while being kept away from humans as it is still difficult to visualize their intentions in real time and communicate that to a nearby user [5]. In this article, we introduce an AR-enabled reconfiguration interface for serial, malleable robots, thus improving the accuracy of end-effector positioning following a robot reconfiguration. Malleable robots are a type of reconfigurable serial robot developed by Clark and Rojas [6], [7], based around the design of a malleable link—a variable-stiffness link between the revolute joints of the robot that allows for their variable, relative positioning. This results in a low-DoF robot having significantly increased task versatility due to the variable workspace of the robot. For extrinsic, malleable robots, where their reconfiguration is performed externally by a user directly reshaping the malleable link, one of the key issues that arises is performing this alignment accurately. For example, visualizing the new configuration of the robot is difficult for a user and results in a limited positioning accuracy [7]. The developed AR system enables the user to smoothly generate and place a virtual end effector to within the maximum reachable space that the robot can move (referred to as reconfiguration space). Using this desired position and orientation, the system can advise optimal end-effector transformations, compute the workspace for each topology, and ultimately overlay them in front of the user’s eyes. This allows for easier and more accurate alignment of the malleable robot to an optimal position by the user. The Significance of Extended Reality in HRI Despite HRI being a well-studied field, many researchers agree that there is a need for a shared understanding of a robot’s workspace to enhance collaboration and situational awareness. Although this method of interaction is still novel, a study by Solyman et al. [8] with 2D workspace visualization has demonstrated the potential of these workflows to

accelerate robot-assembly tasks. However, the biggest drawback of showing the workspace on a PC is that it requires the user to constantly switch between looking at the display and the robot, which can become tiresome. Furthermore, it can be challenging for the user to map a 2D projection of an image to the 3D world, leading to poor positioning accuracy and increasing operation time. This problem is particularly critical in medical applications, where the lack of information on image depth may compromise patient safety due to incorrect placement of visual markers [9]. Overall, these challenges were commonly encountered in 2D workspace visualization studies in literature, thus demonstrating the importance of a 3D immersive experience. One solution to solve these issues is taking the surrounding environment completely out of the problem via the use of virtual reality (VR)-enabled teleoperation. Unfortunately, this introduces the problem of mapping a user’s reference frame onto the robot’s reference frame, where an inaccurate bidirectional mapping leads to poor user experience [10]. With that said, simpler solutions to that problem have been found within the Baxter’s Homonculus system developed by Lipton et al. [11], which leverages off-the-shelf VR goggles to give the user the same point of view as the in-action robot by introducing an intermediate “VR control room” that takes control of the mapping. This decouples the provision of sensory stimuli directly from the robot to the user’s VR scene and allows the user to select which of their movements translates to the robot. Although VR technology may provide a user-friendly interface, it does not enable the user and the robot to collaborate harmoniously within the same shared space. When evaluating its accuracy, the Baxter’s Homonculus study also presents little insight into the degree of error in the mapping, instead focusing on other practical aspects in manufacturing such as time consumption of assembly and number of grasps. Following the introduction of AR, image overlay of 3D holograms onto the users vision was made possible. AR systems have seen applications in a wide range of working scenarios, fulfilling the purposes of motion planning [12], control [13], and visualization [14]. The main benefits of having an AR-assistive system was well summarized by Makhataeva and Varol’s review [15]. The studies examined in their work displayed varying levels of accuracy, with some AR-enabled teleoperation procedures displaying 60–80% accuracy when trying to replicate motions with digital cameras [16]. The other systems used in neurosurgical applications achieved mean target errors as low as 1.11 ± 0.42 mm [17] by incorporating fiducial markers and using intraoperative reimaging to adjust for tissue shifting. Recently, the growth in popularity of the Microsoft HoloLens has attracted developers to enter the head-mounted displays (HMDs) AR community. Together with the Mixed Reality ToolKit (MRTK) [18], developers are able to create interactions through methods such as gaze, speech, and gesturing in their AR apps while preserving the benefits of AR. An interesting study conducted by Rosen et al. asked users to label whether collision occurs for a set of trajectories [12], MARCH 2022

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

11

with some users given no AR support, some given 2D AR visualization, and the remaining given HMD AR support. The results indicated that the users of HMD yielded better accuracy ratings (75%) compared to 2D AR users (65%). One reason behind this was that having a 3D holographic object mitigates errors caused by the user trying to map a 2D projection of 3D objects onto their vision. The potential of AR-/HMD-driven robotic systems do not stop there. In Bolano et al.’s work [19], they used 3D cameras and point clouds to predict robot collision via trajectory and swept volume visualization. What is interesting in this work was the ability of their system to project dynamic robot workspaces to AR such that it can enhance user awareness and enable path replanning. Robot reprogramming was also seen in Kousi et al.’s work [20], where markerless, AR-based HRI was used to aid the user with controlling a mobile robot in a production environment in the case of an unexpected error. Virtual buttons were implemented to initiate robot movements, and Microsoft HoloLens

OptiTrack Tracking System

Malleable Robot

Workbench Hologram

Lectern Hologram Distal Link Hologram

Figure 1. The developed AR-assisted reconfiguration system of a malleable robot. The user is shown interacting with the AR-assisted points-placement scene overlaid on top of the robot base, with the presence of a transparent configuration space. The lectern to the right prints out the instructions for the task the user needs to conduct.

P4

D2

4

P6

P5

P3

3

D

D1

P2

4

13

D2

P1 Figure 2. The previously developed 2-DoF malleable robot to be reconfigured, highlighting the point (P1 - 5) representation and interpoint distances (D13, D14, D23, D24) that define the malleable-link topology.

12

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

MARCH 2022

the robot provided real-time information on active tasks at the workstations. In summary, AR in HMDs is a promising technology that is worth exploiting for enhancing HRI. AR System Design and Development The AR application designed in this article was for a malleable robot system presented in [7] and preliminarily discussed in [6]. It is a 2-DoF, reconfigurable robot formed of a vertical, revolute joint at the base, with a malleable link connecting this base to a second revolute joint, and a rigid link connecting the second revolute joint to the end effector. This is an extrinsic, malleable robot, and thus we believe that there is potential for an AR system to recommend the end-effector placement. We deployed our AR system on the first generation of Microsoft HoloLens. AR/MR was selected over VR as it allows for direct manipulation and co-located visualization of the malleable robot reconfiguration. The HoloLens was selected because it possesses many sensors that capture environmental information and the user’s gestures by using its four environment-understanding cameras and one depth camera. All the applications for the HoloLens were developed on the Unity game engine (Version 2018.4.27f1) and the MRTK (Version 2017.4.3.0). As the HoloLens behaves similarly to a mobile device, no dedicated PC is needed to run the applications, except for loading the application in the first place. However, the developed system does require an additional workstation to compute, then generate, the robot’s workspace as well as the desired end-effector positions. OptiTrack Motive (Corvallis, Oregon, USA) software was also run using this workstation to perform the motion tracking. The specifications of the workstation were a Ryzen 9 5900X CPU and 64-GB random-access memory (RAM). The Malleable Robot-Reconfiguration Workspace Given the desired end-effector position and orientation, a set of optimal robot topologies (that is, specific reconfigurations) can be generated that achieve this [7]. With that said, the developed system should allow for efficient reconfiguration by classifying whether the user’s demands are valid, demonstrating visually what is achievable, and feeding this information back to the user through an AR-enabled interface (see Figure 1). Ultimately, this implies robustly defining a reconfiguration workspace, defined as the volume in which all points within this space can be reached by the malleable robot through reconfiguration (not through joint movement) of the robot’s end effector when it is not fixed in a certain topology. For the developed AR application, it was necessary to compute the reconfiguration workspace for the end effector (P5, as shown in Figure 2), aiding the user in ensuring that the position of the desired end effector is within it. However, this is difficult due to the variability in the malleable link (bend, extension, compression, and twist). To simplify the determination of the reconfiguration workspace, a twist of the malleable link was not considered. Using the minimum radius of curvature (rmin), the involute and cycloid curves of the malleable link at its maximum

(L max) and minimum (L min) lengths, respectively, were computed, as presented in Figure 3. For the malleable link, with y in the vertical direction and x in the horizontal, the involute curve is defined as

x = r (1 - (cos (t) + (t - a) sin (t))(1)



y = r (sin (t) - (t - a) cos (t)), (2)

and the cycloid curve is defined as

x = r (1 - cos (t))(3)



y = r (t - sin (t)) + L min, (4)

was recognized that having a visible workspace can not only speed up the reconfiguration, as the user can easily anticipate which positions and orientations are achievable, it can also visualize the range of robot motions a priori to confirm that it will not coincide with objects in the physical space. Therefore, the availability to toggle this workspace on and off was provided in the developed system, and an on-demand feasibility-checking system (initiated using a button press) was adopted. Malleable Robot Optimal Solution Visualization For the developed AR application, it was also necessary to guide the user in reconfiguring the robot once an ideal configuration was determined. From the topology computation equations, four distances (D13, D14, D23, and D24, shown in Figure 2) are returned that represent the topology of the robot based on the interpoint distances between P1 at the robot base origin, P2 directly above the origin, and P3 and P4, which define either side of the distal joint. Visualizing solely interpoint distances is quite difficult. In our previous work on end-effector alignment, a non-AR, tracking-based alignment strategy was proposed [7]. An OptiTrack system with six cameras was set up around the malleable robot. The user was given real-time feedback on the error between the expected and current positions during alignment, despite it being extremely difficult for the user to use such errors to correct the movement as there was no feedback regarding which direction this error was pointing toward. One alternative approach could be to use the ARinteractive environment to mark the desired P3–5 positions in the user’s vision, and physically attaching such points on the robot before performing the alignment. However, such a system design will impact comfort and ease of use as having only the four points means that the user must deduce the position and orientation of the distal link. To offer a more intuitive alignment strategy, we propose the following: representing the new robot topology using a mesh model of the distal link, located at the final optimal

Malleable-Link-Length Minimum

Malleable-Link-Length Maximum

n

e mi

t olu Inv

in

id m

lo

yc

C

where a is the maximum bending angle of the malleable link, t is the variable curvature of the malleable link, and r is the radius of curvature. The values of rmin = 110, L max = 700, L min = 550, and a = 5 rad were used. The total area captured within these curves represents the distal joint at the end of the malleable-link (P3/4, shown in Figure 3) reconfiguration space. Then, using the length of the distal link, it was possible to compute the resulting end-effector reconfiguration workspace based on the reconfiguration workspace of P3/4. Then, by sweeping this 2D area around the vertical base joint, a 3D theoretical reconfiguration workspace of the end effector was generated. To validate this, a reconfiguration space-exploration experiment was carried out, where the position of P3/4 and P5 were tracked while the robot was moved throughout its reconfiguration space. These 3D results were then rotated around the vertical base axis, removing the z component, to form a 2D plot for increased readability in comparison against the theoretical reconfiguration space. These results can be seen in Figure 4. The robot’s mounting platform was also added, highlighting why negative y values cannot be seen. From these results, we can see that the theoretical reconfiguration space is a good representation of the actual reconfiguration space of the robot, assuming that the theoretical space is adjusted to remove all negative y values. A 3D model of the theoretical reconfiguration workspace of the end effector, with this adjustment made, was then produced for use in y the AR application. Inv olu te With the generated reconfigurama x Cy x c tion workspace, several design deciloi d Path Traced by P5 ma sions were made with regard to how x P3/4 it will interact with the other holoP5 graphic objects. Initially, we considered utilizing the reconfiguration workspace purely to provide feedback to the user, without it appearing P3/4 in the user’s vision, by constantly rmin Reconfiguration Space checking for collisions among the P2 P1 objects and changing the color of the holograms once it collides with the Figure 3. The involute and cycloid curves traced by the malleable link in both maximum reconfiguration workspace. However, and minimum lengths, demonstrating how the reconfigurable space of P3/4 (blue) and following preliminary user feedback, it P5 (green) were theoretically calculated. MARCH 2022

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

13

position defined by the four distances, and using it to infer the positions of P3–5 (which is trivial as the relative position of P3–5 does not change with respect to the distal link). To obtain this, first the new locations of P3 and 4 were determined using the distances. The mesh model is defined with P3 at [0, 0, 0], with P4 directly along the z-axis. We can then compute the translation of the mesh model based on the new position of P3, and then we can rotate the model around P3 to align to the new P4 using a rotation matrix. The rotation matrix is computed using the initial model (p34 model = [0, 0, 50]) and new-world (p34 world = P4 - P3) vectors that define the orientation of the mesh model.

1,200 1,000 800 Y (mm)

600 400 200 0 –200 –400 –600

–1,000

–500

0 X (mm)

P3/4 Experimental P5 Theoretical P5 Experimental

500

1,000

Robot-Mounting Platform P3/4 Theoretical Achievable P5 Theoretical

Figure 4. Results of the reconfiguration space exploration, shown for the end of the malleable link (P3/4 experimental) and the robot’s end effector (P5 experimental). The theoretical reconfiguration space is overlaid for both, along with the positioning of the robot mounting platform.

Coordinate System and Initial Alignment To combine the reconfiguration space and the optimal solution visualization algorithm into an HRI platform, an appropriate AR interface was developed to enhance the user experience in manipulating malleable robots. Prior to this, this section introduces the reference frames attached to each holographic object in the AR space as such relations lay the foundations of an accurate alignment. The ultimate goal of understanding scene-coordinate systems is to infer the location of the end effector with respect to the base. The AR environment consists of four coordinate systems {H}, {E}, {B}, and {W}, illustrated in Figure 5, and corresponding to the HoloLens, end effector, base, and world origin, respectively. Using its four tracking cameras, the HoloLens is able to infer its own transformation {H} with respect to a stationary world origin {W}. It is also able to resolve spatial relationships between virtual objects. Before proceeding with the AR application reconfiguration, the virtual and real environments must be aligned at the base {B} of the robot. This can be performed automatically, with the addition of error-metric feedback; however, due to the computational limits of the HoloLens, this was performed manually by the user, allowing the HoloLens to focus entirely on performing at its maximum rendering frequency. During the initial alignment phase, the table and robot base holograms are freely movable in terms of position, but with its orientation in x and y fixed. This was done to improve the comfort of the alignment process, but more importantly it limits the alignment error and its effects on the system. By default, the Unity game engine environment used for the development of this app defines all the holographic objects with respect to {W}, hence, mathematically, it needs to be transformed into a frame {B}. For example, to find BE T, the transformation from the base to the end effector, given the relationship between the end effector EW T with the origin, and the base BW T with the origin:

{H}

{E}

{B}

{W} Figure 5. The coordinate systems of the scene. W: world origin; B: robot base; E: end effector; H: HoloLens.

14

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

MARCH 2022

B E

T = EW T - 1 BW T.(5)

Under Unity, this procedure can be accomplished by defining the local position of virtual objects with respect to the base frame {B} using a parent-child relationship. AR Interface We begin by defining the tools that are necessary to perform a pose reconfiguration in the context of malleable robots. This includes all the information needed to define the desired endeffector position and orientation as well as the postreconfiguration robot workspace. The virtual world consists of two main hologram modules: the robot base and the lectern. Each module consists of its own suite of peripheral holograms, which are designed to assist the user and provide instructions. The idea behind choosing these two objects as the main HRI medium is simple. For the robot base, as it is an integral part to the robot, it is perfectly logical that it is displayed as a hologram to allow

for initial alignment with the real world. During development, the use of the robot base alone without it being coupled with the table was also trialed. However, the table was kept because it is visually more convenient to align a larger object, and because the table was later used for robot-reconfiguration guidance, once an ideal distal link transformation is generated. On the other hand, the choice of the lectern in place of a “tag-along” floating canvas was made, even though the latter was seen more commonly in AR applications, and the user employed voice commands to advance to the next task or return to the previous task. This choice was made because the canvas often gets in the way of the user and interferes with his or her work, while having a fixed canvas located at a comfortable position and orientation is more ergonomic. Both of the hologram modules were defined with respect to frame {B}. Following the initial alignment, the user interacts almost exclusively with the contents on the lectern and the physical robot, with the exception of placing the desired end-effector hologram, which is grouped under the base module. The lectern module consists of a lectern hologram with a dynamic instruction panel overlaid on top. The user works through the instructions and provides the system feedback via button presses. The lectern design was chosen as it is ergonomically designed for the user to interact with as they are standing. Its slanted and large surface tilts the panel toward the user so he or she can conveniently view its content without straining his or her neck. In addition, it can be placed next to the working environment and take over the job of a traditional 2D display. Conversely, scene-content holograms and functional classes that are related to the reconfiguration of the robot are grouped under the robot’s base module. This consists of the table, robot base, desired end-effector model, reconfiguration workspace, and postreconfiguration robot workspace unique to its topology. Following the initial alignment of the base and the table [Figure 6(a)], the user instructs the robot base to “anchor” via voice commands, fixing the base in place. The anchoring process also places the scene origin as a reference point at the center of the hologram. The instructions that the user then follows are in line with the reconfiguration workflow presented in Figure 7 and can be summarized as follows. First, the user starts a new configuration [see Figure 6(a)]. Then the user enters task 1, a scene consisting of the reconfiguration space computed in the “Augmented Reality System Design and Development” section. From the lectern shown in Figure 6(b), the user can instantiate the hologram of the distal link, and then use gaze and gestures to position and orientate it to a desired configuration. In conjunction, an end-effector-checking algorithm was implemented to confirm whether the holograms lie within the configuration space [Figure 6(c) and (d)] and further prevents points that construct a topology, which is physically unachievable. On the user’s side, following a press of the “check” button, the end-effector hologram will change its color depending on its current state. A “green” color indicates that it is within the workspace, and a “red” color suggests the opposite. Once the hologram turns to green, a “ready” button is activated, and the

user is prompted to proceed [see Figure 6(d)]. The user then confirms the coordinates of distal-link points P5 and 6, and the system saves the configuration presented in the scene [see Figure 6(e)]; this records the transformation of all the instantiated holograms, closes the app, and sends the information to a workstation. The workstation is responsible for computing the optimal topologies of the malleable robot, defined by distance geometry using a sampling of dihedral angles between the triangles formed by the desired point and orientation of the end effector and the fixed robot topology [7], which are used to return mesh models of the achievable robot workspaces that obtains the desired end-effector position and orientation as well as the robot’s topology transformations, represented by a mesh model of the distal rigid link and distal joint in the new optimal topology configurations. Scene saving occurs after the user correctly places their desired end effector in the reconfiguration space, but before the workstation returns the mesh models of achievable workspaces, along with their end-effector robot topologies. The design decision of using scene saving and loading functions over TCP/IP communication between the workbench and the HoloLens was contemplated during development. The implementation of sockets drastically reduces the frames per second on the HoloLens and negatively impacts user experience; however, it ensures a seamless transition between the initial points-positioning stage and the reconfiguration stage of the workflow, whereas a traditional saving and loading introduces a break point. Ultimately, we prioritized user experience and speed of reconfiguration over ensuring a seamless transition as a jittery scene greatly diminishes the immersive experience that an AR system should provide. Reconfiguration and Alignment To complete the necessary elements needed to define a reconfiguration, the new robot’s workspace and end-effector models are loaded into the system [Figure 6(g) and (h)]. Note that the workspace defined here differs from the reconfiguration workspace as it is the volume that the robot can reach given a particular end-effector reconfiguration, assuming that the malleable link is rigid. In other words, the robot’s workspace is a subset of the reconfiguration workspace, defined when the malleable link is rigid and the end effector is placed at a unique position and orientation. The system requests that the user provide a final confirmation of the configuration scene [Figure 6(i)]. This is done to ensure that the scene has successfully loaded the configuration consisting of the desired end-effector pose, and that a robot’s workspace, which successfully encapsulates the end effector, was generated. If any of these conditions are not met, the user is free to reject the configuration and restart the process. Content with the generated workspace, a mesh of the distal link, positioned in the new robot configuration, is shown in the virtual environment [Figure 6(j)]. This visualizes to the user exactly where to reconfigure and position the malleable robot. Finally, the user is walked through the reconfiguration process via the lectern’s instructions. During this stage, MARCH 2022

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

15

Desired Distal Link Position

AR World Anchor

Malleable Robot

Confirmed Within Reconfiguration Workspace

Reconfiguration Workspace

(a)

(b)

(c)

(d)

Suggested Reconfiguration and Workspace

Desired End-Effector Vector (e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 6. An illustration of the workflow of the AR-assisted reconfiguration app, where the top row is the scene at the robot base, and the bottom row is the corresponding instruction panel. (a) An initial alignment of the robot base; (b) the reconfiguration workspace of the robot is shown; (c) a holographic end effector is generated for the user to place at a desired location; (d) checking whether the end effector is situated within the reconfiguration space; (e) the system displaying the vector of the end effector; (f) reloading the holograms, and the user confirms the desired end-effector location; (g)–(i) the suggested reconfigurations of the robot and the resulting workspaces they generate are shown; (j) the user selects a reconfiguration and proceeds with reconfiguration assistance; (k) an animated guidance with holograms aids the user in reconfiguration; and (l) a correctly aligned end effector postreconfiguration. This workflow can also be viewed in the linked time stamp of the supplementary video at https://youtu.be/ HgHEnruNgeI?t=30.

16

•

IEEE ROBOTICS & AUTOMATION MAGAZINE

•

MARCH 2022

animations and dynamic markers, as shown in Figure 6(k), were designed to guide the user through the procedures used to pressurize and depressurize the malleable link. Once the user is happy with his or her positioning of the distal link with the virtual distal link, he or she can lock it in place, completing the malleable robot’s reconfiguration [Figure 6(l)].

Workbench

Microsoft HoloLens

Computation of Desired Malleable Robot Topology

Computation of Workspace

Align World Anchor

Load Configuration

Create New Configuration

Display Workspace

Performance Evaluation of the AR Application

Computation of Desired EndEffector Location

User-Desired End-Effector Hologram

Reject Confirm Initial Alignment Accuracy HoloLens Configuration Configuration We first evaluate the accuracy of manFeedback ually aligning the holographic environment to the real environment’s Confirm Hologram origin, located at the malleable robot’s Positioning P1 (see Figure 2). Six OptiTrack Flex3 cameras surround the robot, with reflective tracking markers located on Save Scene the points P1–5, defining the malleaReconfiguration ble robot’s topology [6]. The calibraGuidance Exit and tion report of the cameras detailed a Reload mean 3D error for overall projection of 0.426 mm. The holographic environment was then manually aligned to Figure 7. The reconfiguration workflow achieved by the AR-HRI system. the real environment by a user dragging the robot’s base hologram onto the real robot’s base. Once satisfied with the alignment, the To better illustrate this metric, a labeled diagram including holographic environment’s position was locked. the real and holographic points, along with the error, is The initial alignment error is difficult to measure direct- defined graphically in Figure 8. In this experiment, five ly via optical tracking, but it can be inferred using three sets of data were collected, and the results are shown in randomly distributed subpoints within the achievable P5 Table 1. Theoretically, the ideal alignment accuracy would theoretical space, defined by the P5 theoretical space when show an offset of 0 mm, however, due to human and sysY > 0 (see Figure 4), which is visually bounded by the tem errors, a good accuracy can be defined as