HumanRobot Interaction: Control, Analysis, and Design 1527557405, 9781527557406

Citation preview

Human–Robot Interaction

Human–Robot Interaction: Control, Analysis, and Design Edited by

Dan Zhang and Bin Wei

Human–Robot Interaction: Control, Analysis, and Design Edited by Dan Zhang and Bin Wei This book first published 2020 Cambridge Scholars Publishing Lady Stephenson Library, Newcastle upon Tyne, NE6 2PA, UK British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Copyright © 2020 by Dan Zhang, Bin Wei and contributors All rights for this book reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the copyright owner. ISBN (10): 1-5275-5740-5 ISBN (13): 978-1-5275-5740-6

TABLE OF CONTENTS

Preface ....................................................................................................... vi Chapter 1 .................................................................................................... 1 Trust and the Discrepancy between Expectations and Actual Capabilities of Social Robots Bertram Malle, Kerstin Fischer, James Young, AJung Moon, Emily Collins Chapter 2 .................................................................................................. 24 Talking to Robots at Depth Robert Codd-Downey, Andrew Speers, Michael Jenkin Chapter 3 .................................................................................................. 45 Towards the Ideal Haptic Device: Review of Actuation Techniques for Human-Machine Interfaces Maciej Lacki, Carlos Rossa Chapter 4 .................................................................................................. 75 New Research Avenues in Human-Robot Interaction Frauke Zeller Chapter 5 .................................................................................................. 93 Interpreting Bioelectrical Signals for Control of Wearable Mechatronic Devices Tyler Desplenter, Jacob Tryon, Emma Farago, Taylor Stanbury, Ana Luisa Trejos Chapter 6 ................................................................................................ 147 Human-Robot Interaction Strategy in Robotic-assisted Balance Rehabilitation Training Jiancheng Ji, Shuai Guo, Jeff Xi, Jin Liu Chapter 7 ................................................................................................ 171 Development of a Wearable Exoskeleton Suit for Paraplegic Parents Bing Chen, Bin Zi, Ling Qin, Wei-Hsin Liao

PREFACE

Robotics have been used in industry and other fields for the past decade, however human-robot interaction is at its early stage. This book, Human – Robot Interaction: Control, Analysis, and Design, will focus on the topics of human-robot interaction, its applications and current challenges. We would like to thank all the authors for their contributions to the book. We are also grateful to the publisher for supporting this project. We hope the readers find this book informative and useful. This book consists of 7 chapters. Chapter 1 takes trust to be a set of expectations about the robot’s capabilities and explores the risks of discrepancies between a person’s expectations and the robot’s actual capabilities. The major sources of these discrepancies and ways to mitigate their detrimental effects are examined. Chapter 2 has concentrated primarily on diver to robot communication. Communication from the robot to the diver, especially when approaches such as gesture-based are used, is also an issue. Chapter 3 reviews recent advancements in the field of passive and hybrid haptic actuation. The authors highlight the design considerations and trade-offs associated with these actuation methods and provide guidelines on how their use can help with development of the ultimate haptic device. Chapter 4 introduces an extended HRI research model, which is adapted from communication and mass communication studies, and focuses on the social dimension of social robots. Chapter 5 highlights some existing methods for interpreting EEG and EMG signals that are useful for the control of wearable mechatronic devices. These methods are focused on modelling motion for the purpose of controlling wearable mechatronic devices that target musculoskeletal rehabilitation of the upper limb. Chapter 6 discusses a training method for patient balance rehabilitation based on human-robot interaction. Chapter 7 develops a wearable exoskeleton suit that involves human-robot interaction to help the individuals with mobility disorders caused by a stroke, spinal cord injury or other related diseases. Finally, the editors would like to acknowledge all the friends and colleagues who have contributed to this book. Dan Zhang, Toronto, Ontario, Canada Bin Wei, Sault Ste Marie, Ontario, Canada February 25, 2020

CHAPTER 1 TRUST AND THE DISCREPANCY BETWEEN EXPECTATIONS AND ACTUAL CAPABILITIES OF SOCIAL ROBOTS BERTRAM F. MALLE, KERSTIN FISCHER, JAMES E. YOUNG, AJUNG MOON, EMILY COLLINS Corresponding author: Bertram F. Malle, Professor Department of Cognitive, Linguistic, and Psychological Sciences Brown University 190 Thayer St. Providence, RI 02912, USA [email protected] +1 (401) 863-6820 Kerstin Fischer, Professor (WSR) Department of Design and Communication University of Southern Denmark Alsion 2 DK-6400 Sonderborg, Denmark [email protected] Phone: +45-6550-1220 James E. Young, Associate Professor Department of Computer Science University of Manitoba Winnipeg, Manitoba R3T 2N2, Canada Email: [email protected] Phone: (lab) +1-204-474-6791

AJung Moon, Assistant Professor Department of Electrical and Computer Engineering McGill University 3480 University Street Montreal, Quebec H3A 0E9, Canada [email protected] Phone: +1-514-398-1694 Emily C. Collins, Research Associate Department of Computer Science University of Liverpool Ashton Street Liverpool L69 3BX, UK [email protected] Phone: +44 (0)151 795 4271

2

Chapter 1

Abstract From collaborators in factories to companions in homes, social robots hold the promise to intuitively and efficiently assist and work alongside people. However, human trust in robotic systems is crucial if these robots are to be adopted and used in home and work. In this chapter we take trust to be a set of expectations about the robot’s capabilities and explore the risks of discrepancies between a person’s expectations and the robot’s actual capabilities. We examine major sources of these discrepancies and ways to mitigate their detrimental effects. No simple recipe exists to help build justified trust in human-robot interaction. Rather, we must try to understand humans’ expectations and harmonize them with robot design over time.

Introduction As robots continue to be developed for a range of contexts where they work with people, including factories, museums, airports, hospitals, and homes, the field of Human-Robot Interaction explores how well people will work with these machines, and what kinds of challenges will arise in their interaction patterns. Social robotics focuses on the social and relational aspects of Human-Robot Interaction, investigating how people respond to robots cognitively and emotionally, how they use their basic interpersonal skills when interacting with robots, and how robots themselves can be designed to facilitate successful human-machine interactions. Trust is a topic that currently receives much attention in human-robot interaction research. If people do not trust robots, they will not collaborate with them or accept their advice, let alone purchase them and delegate to them the important tasks they have been designed for. Building trust is therefore highly desirable from the perspective of robot developers. A closer look at trust in human-robot interaction, however, reveals that the concept of trust itself is multidimensional. For instance, one could trust another human (or perhaps robot) that they will carry out a particular task reliably and without errors, and that they are competent to carry out the task. But in some contexts, people trust another agent to be honest in their communication, sincere in their promises, and to value another person’s, or the larger community’s interests. In short, people may trust agents based on evidence of reliability, competence, sincerity, or ethical integrity

Trust and Discrepancy

3

[1], [2] 1. What unites trust along all these dimensions is that it is an expectation—expecting that the other is reliable, competent, sincere, or ethical. Expectations, of course, can be disappointed. When the other was not as reliable, capable, or sincere as one thought, one’s trust was misplaced. Our goal in this chapter is to explore some of the ways in which people’s expectations of robots may be raised too high and therefore be vulnerable to disappointment. To avert disappointed expectations, at least two paths of action are available. One is to rapidly expand robots’ capacities, which is what most designers and engineers strive for. But progress has been slow [3], and the social and communicative skills of artificial agents are still far from what seems desirable [4], [5]. Another path is to ensure that people trust a robot to be just as reliable, capable, and ethical as it really is able to; that is, to ensure that people understand the robot’s actual abilities and limitations. This path focuses on one aspect of transparency: providing human users with information about the capabilities of a system. Such transparency, we argue, is a precondition for justified trust in any autonomous machine, and social robots in particular [6], [7]. In this chapter, we describe some of the sources of discrepancies between people’s expectations and robots’ real capabilities. We argue the discrepancies are often caused by superficial properties of robots that elicit feelings of trust in humans without validly indicating the underlying property the person trusts in. We therefore need to understand the complex human responses triggered by the morphology and behaviour of autonomous machines, and we need to build a systematic understanding of the effects that specific design choices have on people’s cognitive, emotional, and relational reactions to robots. In the second part of the chapter we lay out a number of ways to combat these discrepancies.

Discrepancies Between Human Expectations and Actual Robot Capabilities In robot design and human-robot interaction research, the tendency to build ever more social cues into robots (from facial expressions to emotional tone of voice) is undeniable. Intuitively, this makes sense since robots that exhibit social cues are assumed to facilitate social interaction by leveraging people’s existing social skill sets and experience, and they 1

The authors have provided a measure of these multiple dimensions of trust and invite readers to use that measure for their human-robot interaction studies: http://bit.ly/MDMT_Scale

4

Chapter 1

would fit seamlessly into social spaces without constantly being in the way [8]. However, in humans, the display of social cues is indicative of certain underlying mental properties, such as thoughts, emotions, intentions, or abilities. The problem is that robots can exhibit these same cues, through careful design or specific technologies, even though they do not have the same, or even similar, underlying properties. For example, in human interaction, following another person’s gaze is an invitation to joint attention [9]; and in communication, joint attention signals the listener’s understanding of the speaker’s communicative intention. Robots using such gaze cues [10] are similarly interpreted as indicating joint attention and of understanding a speaker’s instructions [11], [12]. However, robots can produce these behaviors naïvely using simple algorithms, without having any concept of joint attention or any actual understanding of the speaker’s communication. Thus, when a robot displays these social cues, they are not symptoms of the expected underlying processes, and a person observing this robot may erroneously attribute a range of (often human-like) properties to the robot [13]. Erroneous assumptions about other people are not always harmful. Higher expectations than initially warranted can aid human development (when caregivers “scaffold” the infant’s budding abilities; [14], can generate learning success [15], and can foster prosocial behaviors [16]. But such processes are, at least currently, wholly absent with robots. Overestimating a robot’s capacities poses manifest risks to users, developers, and the public at large. When users entrust a robot with tasks that the robot ends up not being equipped to do, people may be disappointed and frustrated when they discover the robot’s limited actual capabilities [17]; and there may be distress or harm if they discover these limitations too late. Likewise, developers who consistently oversell their products will be faced with increasing numbers of disappointed, frustrated, or distressed users who no longer use the product, write terrible public reviews (quite a significant impact factor for consumer technology), or even sue the manufacturer. Finally, the public at large could be deprived of genuine benefits if a few oversold robotic products cause serious harm, destroy consumer trust, and lead to stifling regulation. Broadly speaking, discrepancies between expectations and reality have been well documented and explored under the umbrella of “expectancy violation,” from the domains of perception [18] to human interaction [19]. In human-robot interaction research, such violations have been studied, for example, by comparing expectations from media to interactions with a real robot [20] or by quantifying updated capability estimates after interacting with a robot [21]. Our discussion builds on this line of inquiry, but we do

Trust and Discrepancy

5

not focus on cases when an expectancy violation has occurred, which assumes that the person has become aware of the discrepancy (and is likely to lose trust in the robot). Instead, we focus on sources of such discrepancies and avenues for making a person aware of the robot’s limitations before they encounter a violation (and thus before a loss of trust).

Sources of Discrepancies There are multiple sources of discrepancies between the perceived and actual capacities of a robot. Obvious sources are the entertainment industry and public media, which frequently exaggerate technical realities of robotic systems. We discuss here more psychological processes, from misleading and deceptive design and presentation to automatic inferences from a robot’s superficial behavior to deep underlying capabilities.

Misleading design Equipping a robot with outward social cues that have no corresponding abilities is, at best, misleading. Such a strategy violates German designer Dieter Rams’ concept of honest design, which is the commitment to design that “does not make a product more innovative, powerful or valuable than it really is” [22]; see also [23], [24]. Honest design is a commitment to transparency—enabling the user to “see through” the outward appearance and to accurately infer the robot’s capacities. In the HRI laboratory, researchers often violate this commitment to transparency when they use Wizard-of-Oz (WoZ) methods to make participants believe that they are interacting with an autonomous, capable robot. Though such misperceptions are rarely harmful, they do contribute to false beliefs and overly high expectations about robots outside the laboratory. Moreover, thorough debriefing at the end of such experiments is not always provided [25], which would reset people’s generalizations about technical realities.

Deception When a mismatch between apparent and real capacities is specifically intended—for example, to sell the robot or impress the media—it arguably turns into deception and even exploitation [26]. And people are undoubtedly vulnerable to such exploitation. A recent study suggested that people were willing to unlock the door to a university dormitory building for a verbally communicating robot that had the seeming authority of a

6

Chapter 1

food delivery agent. Deception is not always objectionable; in some instances it is used for the benefit of the end user [27], [28], such as in calming individuals with dementia [29] or encouraging children on the autism spectrum to form social bonds [30]. However, these instances must involve careful management of the risks involved in the deception—risks for the individual user, the surrounding social community, and the precedent it sets for other, perhaps less justified cases of deception.

Impact of norms At times, people are well aware that they are interacting with a machine in human-like ways because they are engaging with the robot in a joint pretense [31] or because it is the normatively correct way to behave. For example, if a robot greets a person, the appropriate response is to reciprocate the greeting; if the speaker asks a question, the appropriate response is to answer the question. Robots may not recognize the underlying social norm and they may not be insulted if the user violates the norm, but the user, and the surrounding community (e.g., children who are learning these norms), benefit from the fact that both parties uphold relevant social practices and thus a cooperative, respectful social order [32]. The more specific the roles that robots are assigned (e.g., nurse assistant, parking lot attendant), the more these norms and practices will influence people’s behavior toward the robot [33]. If robots are equipped with the norms that apply to their roles (which is a significant challenge; [34], this may improve interaction quality and user satisfaction. Further, robots can actively leverage norms to shape how people interact with it, but perhaps even in manipulative fashion [35]. Norm-appropriate behavior is also inherently trust-building, because norms are commitments to act, and expectations that others will act, in ways that benefit the other (thus invoking the dimension of ethical trust; [36], norm violations become all the more powerful in threatening trust.

Expanded inferences Whereas attributions of norm competence to a robot are well grounded in the robot’s actual behavior, a robot that displays seemingly natural communicative skills can compel people to infer (and genuinely assume to be present) many other abilities that the robot probably is unlikely to have [37]. In particular, seeing that a robot has some higher-level abilities, people are likely to assume that it will also possess more basic abilities that in humans would be a prerequisite for the higher-level ability. For

Trust and Discrepancy

7

instance, a robot may greet someone with “Hi, how are you?” but be unable itself to answer the same question when the greeting is reciprocated, and it may not even have any speech understanding capabilities at all. Furthermore, a robot’s syntactically correct sentences do not mean it has a full-blown semantics or grasps anything about conversational dynamics [38]. Likewise, seeing that a robot has one skill, we must expect people to assume that it also is has other skills that in humans are highly correlated with the first. For example, a robot may be able to entertain or even tutor a child but be unable to recognize when the child is choking on a toy. People find it hard to imagine that a being can have selected, isolated abilities that do not build upon each other [39]. Though it is desirable that, say, a manufacturer provides explicit and understandable documentation of a system’s safety and performance parameters [40], [41], making explicit what a robot can and cannot do will often fail. That is because some displayed behaviors set off a cascade of inferences that people have evolved and practiced countless times with human beings [32]. As a result, spontaneous reactions to robots in social contexts and their explicit beliefs on what mental capacities robots possess can come apart [42], [43].

Automatic inferences Some inferences or emotional responses are automatic, at least upon initial encounters with artificial agents. Previous research has shown that people treat computers and related technology (including robots) in some ways just like human beings (e.g., applying politeness and reciprocity), and often do so mindlessly [44]. The field of human-robot interaction has since identified numerous instances in which people show basic socialcognitive responses when responding to humanlike robots—for example, by following the “gaze” of a robot [45] or by taking its visual perspective [46]. Beyond such largely automatic reactions, a robot’s humanlike appearance seems to invite a wide array of inferences about the robot’s intelligence, autonomy, or mental capacities more generally [47]–[49]. But even if these appearance-to-mind inferences are automatic, they are not simplistic; they do not merely translate some degree of humanlikeness into a proportional degree of “having a mind.” People represent both humanlike appearance and mental capacities along multiple dimensions [50]–[52], and specific dimensions of humanlike appearance trigger people’s inferences for specific dimensions of mind. For example, features of the Body Manipulator dimension (e.g., torso, arms, fingers) elicit inferences about capacities of reality interaction, which include perception,

8

Chapter 1

learning, acting, and communicating. By contrast, facial and surface features (e.g., eyelashes, skin, apparel) elicit inferences about affective capacities, including feelings and basic emotions, as well as moral capacities, including telling right from wrong and upholding moral values [53].

Variations We should note, however, that people’s responses to robots are neither constant nor universal. They show variation within person, manifesting sometimes as cognitive, emotional, or social-relational reactions, can be in the foreground or background at different moments in time, and change with extended interactions with the robot [8], [32]. They also show substantial interpersonal variation, as a function of levels of expertise [54], personal style [55], and psychosocial predispositions such as loneliness [56].

Status quo The fact remains, however, that people are vulnerable to the impact of a robot’s behavior and appearance [57]. We must expect that, in real life as in the laboratory, people will be willing to disclose negative personal information to humanoid agents [58], [59], trust and rely on them [60], empathize with them [61], [62], give in to a robot’s obedience-like pressure to continue tedious work [63] or perform erroneous tasks [64]. Further, in comparison to a mechanical robot, people are more prone to take advice from a humanoid robot [65], trust and rely on them more [60], and are more likely to comply with their requests [66]. None of these behaviors are inherently faulty; but currently they are unjustified, because they are generated by superficial cues rather than by an underlying reality [57]. At present, neither mechanical nor humanoid robots have more knowledge to share than Wikipedia, are no more trustworthy to keep secrets than one’s iPhone, and have no more needs or suffering than a cartoon character. They may in the future, but until that future, we have to ask how we can prevent people from having unrealistic expectations of robots, especially humanlike ones.

How to Combat Discrepancies We have seen that discrepancies between perceived and actual capacities exist at multiple levels and are fed from numerous sources. How can people recover from these mismatches or avoid them in the first place? In this section, we provide potential paths for both short- and long-term

Trust and Discrepancy

9

solutions to the problem of expectation discrepancy when dealing with social robots.

Waiting for the future An easy solution may be to simply wait for the robots of the future to make true the promises of the present. However, that would mean an extended time of misperceived reality, and numerous opportunities for misplaced trust, disappointment, and non-use. It is unclear whether recovery from such prolonged negative experiences is possible. Another strategy to overcome said discrepancies may be to encourage users to acquire minimally necessary technical knowledge to better evaluate artificial agents, perhaps encouraging children to program machines and thus see their mechanical and electronic insides. However, given the widespread disparities in access to quality education in most of the world’s countries, the technical-knowledge path would leave poorer people misled, deceived, and more exploitable than ever before. Moreover, whereas the knowledge strategy would combat some of the sources we discussed (e.g., deception, expanded inferences), it would leave automatic inferences intact, as they are likely grounded in biologically or culturally evolved response patterns.

Experiencing the cold truth Another strategy might be to practically force people to experience the mechanical and lifeless nature of machines—such as by asking people to inspect the skinless plastic insides of an animal robot like Paro or by unscrewing a robot’s head and handing it to the person. It is, however, not clear that this will provide more clarity for human-robot interactions. A study of the effects of demonstrating the mechanistic nature of robots to children in fact showed that the children still interacted with the robot in the same social ways as children to whom the robotic side of robots had not been pointed out [67]. Furthermore, if people have already formed emotional attachments, such acts will be seen as cruel and distasteful, rather than have any corrective effects on discrepant perceptions.

Revealing real capacities Perhaps most obvious would be truth in advertising. Robot designers and manufacturers, organizations and companies that deploy robots in hotel lobbies, hospitals, or school yards would signal to users what the

10

Chapter 1

robot can and cannot do. But there are numerous obstacles to designers and manufacturers offering responsible and modest explanations of the machine’s real capacities. They are under pressure to produce within the constraints of their contracts; they are beholden to funders; they need to satisfy the curiosity of journalists and policy makers, who are also keen to present positive images of developing technologies. Further, even if designers or manufacturers adequately reveal the machine’s limited capabilities, human users may resist such information. If the information is in a manual, people won’t read it. If it is offered during purchase, training, or first encounters, it may still be ineffective. That is because the abovementioned human tendency to perceive agency and mind in machines that have the tell-tale signs of self-propelled motion, eyes, and verbal communication is difficult to overcome. Given the eliciting power of these cues, it is questionable (though empirically testable) whether explicit information can ever counteract a user’s inappropriate mental model of the machine.

Legibility and explainability An alternative approach is to make the robot itself “legible”— something that a growing group of scholars is concerned with [68]. But whereas a robot’s intentions and goals can be made legible—e.g., in a projection of the robot’s intended motion path or in the motion itself— capabilities and other dispositions are not easily expressed in this way. At the same time, the robot can correct unrealistic expectations by indicating some of its limits of capability in failed actions [69] or, even more informative, in explicit statements that it is unable or forbidden to act a certain way [70]. A step further would be to design the robot in such a way that it can explicate its own actions, reasoning, and capabilities. But whereas giving users access to the robot’s ongoing decision making and perhaps offering insightful and human-tailored explanations of its performed actions may be desirable [71], “explaining” one’s capacities is highly unusual. Most of this kind of communication among humans is done indirectly, by providing information about, say, one’s occupation [72] or acquaintance with a place [73]. Understanding such indirect speech requires access to shared perceptions, background knowledge, and acquired common ground that humans typically do not have with robots. Moreover, a robot’s attempts to communicate its knowledge, skills, and limitations can also disrupt an ongoing activity or even backfire if talk about capabilities makes users suspect that there is a problem with the interaction [32]. There

Trust and Discrepancy

11

is, however, a context in which talk about capabilities is natural— educational settings. Here, one agent learns new knowledge, skills, abilities, often from another agent, and both might comment freely on the learner’s capabilities already in place, others still developing, and yet others clearly absent. If we consider a robot an ever-learning agent, then perhaps talk about capabilities and limitations can be rather natural. One potential drawback of robots that explain themselves must be mentioned. Such robots would appear extremely sophisticated, and one might then worry which other capacities people will infer from this explanatory prowess. Detailed insights into reasoning may invite inferences of deeper self-awareness, even wisdom, and user-tailored explanations may invite inferences of caring and understanding of the user’s needs. But perhaps by the time full-blown explainability can really be implemented, some of these other capacities will too; then the discrepancies would all lift at once.

Managing expectations But until that time, we are better off with a strategy of managing expectations and ensuring performance that matches these expectations and lets trust build upon solid evidence. Managing expectations will rely on some of the legibility and explainability strategies just mentioned along with attempts to explicitly set expectations low, which may be easily exceeded to positive effect [74]. However, such explicit strategies would be unlikely to keep automatic inferences in check. For example, in one study, Zhao et al. (submitted) showed that people take a highly humanlike robot’s visual perspective even when they are told it is a wax figure. The power of the mere humanlike appearance was enough to trigger the basic social-cognitive act of perspective taking. Thus, we also need something we might call restrained design— attempts to avoid overpromising signals in behavior, communication, and appearance, as well as limiting the robot’s roles so that people form limited, role- and context-adequate expectations. As a special case of such an approach we describe here the possible benefit of an incremental robot design strategy—the commitment to advance robot capacities in small steps, each of which is well grounded in user studies and reliability testing.

Incremental Design Why would designing and implementing small changes in a robot prevent discrepancies between a person’s understanding of the robot’s

12

Chapter 1

capacities and its actual capacities? Well-designed small changes may be barely noticeable and, unless in a known, significant dimension (e.g., having eyes after never having had eyes), will limit the number of new inferences that would be elicited by it. Further, even when noticed, the user may be able to more easily adapt to a small change, and integrate it into their existing knowledge and understanding of the robot, without having to alter their entire mental model of the robot. Consider the iRobot Roomba robotic vacuum cleaner. The Roomba has a well-defined, functional role in households as a cleaning appliance. From its first iteration, any discrepancy between people’s perceptions of the robot’s capacities and its actual capacities were likely related to the robot’s cleaning abilities, which could be quickly resolved by using the robot in practice. As new models hit the market, Roomba’s functional capacities improved only incrementally—for example, beep-sequence error codes were replaced by pre-recorded verbal announcements, or random-walk cleaning modes were replaced by rudimentary mapping technology. In these cases, the human users have to accommodate only minor novel elements in their mental models, each changing only very few parameters. Consider, by contrast, Softbank’s Pepper robot. From the original version, Pepper was equipped with a humanoid form including arms and hands that appeared to gesture, and a head with eyes and an actuated neck, such that it appeared to look at and follow people. Further, marketing material emphasized the robot’s emotional capacities, using such terms as “perception modules” and an “emotional engine.” We can expect that these features encourage people to infer complex capacities in this robot, even beyond perception and emotion. Observing the robot seemingly gaze at us and follow a person’s movements suggests attention and interest; the promise of emotional capacities suggests sympathy and understanding. However, beyond pre-coded sentences intended to be cute or funny, the robot currently has no internal programmed emotional model at all. As a result, we expect there to be large discrepancies between a person’s elicited expectations and the robot’s actual abilities. Assumptions of deep understanding in conversation and willingness toward risky personal disclosure may then be followed by likely frustration or disappointment. The discrepancy in Pepper’s case stems in part from the jump in expectation that the designers invite the human to take and the actual reality of Pepper’s abilities. Compared with other technologies people may be familiar with, a highly humanoid appearance, human-like social signaling behaviors, and purported emotional abilities trigger a leap in inference people make from “robots can't do much” to “they can do a lot.” But that leap is not matched by Pepper’s actual capabilities. As a result,

Trust and Discrepancy

13

encountering Pepper creates a large discrepancy that will be quite difficult to overcome. A more incremental approach would curtail the humanoid form and focus on the robot’s gaze-following abilities, without claims of emotional processing. If the gaze following behavior actually supports successful person recognition and communication turn taking, then a more humanoid form may be warranted. And only if actual emotion recognition and the functional equivalent of emotional states in the robot are achieved would Pepper’s “emotion engine” be promoted. Incremental approaches have been implemented in other technological fields. For example, commercial car products have in recent years increasingly included small technical changes that point toward eventual autonomous driving abilities, such as cruise control, active automatic breaking systems, lane violation detection and correction, and the like. More advanced cars, such as Tesla’s Model S, have an “auto-pilot” mode that takes a further step toward autonomous driving in currently highly constrained circumstances. The system still frequently reminds the user to keep their hands on the steering wheel and to take over when those constrained circumstances no longer hold (e.g., no painted lane information). However, the success of this shared autonomy situation depends on how a product is marketed. Other recent cars may include a great deal of autonomy in their onboard computing system but are not marketed as autonomous or self-driving but are called “Traffic Jam Assist” or “Super Cruise.” Such labeling decisions limit what the human users expects of the car and therefore what they entrust it to do. A recent study confirms that labeling matters: People overestimate Tesla cars’ capacities more than other comparable brands [75]. And perhaps unsurprisingly, the few highly-publicized accidents with Teslas are typically the result of vast overestimation of what the car can do [76], [77]. Within self-driving vehicle research and development, a category system is in place to express the gradually increasing levels of autonomy of the system in question. In this space, however, the incremental approach may still take steps that are too big. In the case of vehicle control, people's adjustment to continuously increasing autonomy is not itself continuous but takes a qualitative leap. People either drive themselves, assisted up to a point, or they let someone else (or something else) drive; they become passengers. In regular cars, actual passengers give up control, take naps, read books, chat on the phone, and would not be ready to instantly take the wheel when the main driver requests it. Once people take on the unengaged passenger role with increasingly (but not yet fully) autonomous vehicles, the situation will result in over-trust (the human will take naps, read books, etc.). And if there remains a small chance that the car needs

14

Chapter 1

the driver’s attention but the driver has slipped into the passenger role, the situation could prove catastrophic. The human would not be able to take the wheel quickly enough when the car requests it because it takes time for a human to shift attention, observe their surroundings, develop situational awareness, make a plan, and act [78]. Thus, even an incremental approach would not be able to avert the human’s jump to believing the car can handle virtually all situations, when in fact the car cannot. Aside from incremental strategies, the more general restrained design approach must ultimately be evidence-based design. Decisions about form and function must be informed by evidence into which of the robot’s signals elicit what expectations in the human. Such insights are still rather sparse and often highly specific to certain robots. It therefore takes a serious research agenda to address this challenge, with a full arsenal of scientific approaches: carefully controlled experiments to establish causal relations between robot characteristics and a person’s expectations; examination of the stability of these response patterns by comparing young children and adults as well as people from different cultures; and longitudinal studies to establish how those responses will change or stabilize in the wake of interacting with robots over time. We close our analysis by discussing the strengths and challenges that come with longitudinal studies.

Longitudinal Research Longitudinal studies would be the ideal data source to elucidate the source of and remedy for discrepancies between perceived and actual robot capacities. That is because, first, they can distinguish between initial reactions to robots and more enduring response patterns. We have learned from human-human social perception research that initial responses, even if they change over time, can strongly influence the range of possible longterm responses; in particular, initial negative responses tend to improve more slowly than positive initial reactions deteriorate [79]. In human-robot encounters, some responses may be automatic and have a lasting impact, whereas others may initially be automatic but could be changeable over time. Furthermore, some responses may reflect an initial lack of understanding of the encountered novel agent, and with time a search for meaning may improve this understanding [80]. Longitudinal studies can also track how expectations clash with new observations and how trust fluctuates as a result. High-quality longitudinal research is undoubtedly difficult to conduct because of cost, time and management commitments, participant attrition,

Trust and Discrepancy

15

ethical concerns of privacy and unforeseen impacts on daily living, and the high rate of mechanical robot failures. A somewhat more modest goal might be to study short-term temporal dynamics that will advance knowledge but also provide a launching pad for genuine longitudinal research. For the question of recovery from expectation-reality discrepancies we can focus on a few feasible but informative paradigms. A first paradigm is to measure people’s responses to a robot with or without information about the true capacities of the robot. In comparison to spontaneous inferences about the robot’s capacities, would people adjust their inferences when given credible information? One could compare the differential effectiveness of (a) inoculation (providing the ground-truth information before the encounter with the robot) and (b) correction (providing it after the encounter). In human persuasion research, inoculation is successful when the persuasive attempt operates at an explicit, rational level [81]. By analogy, the comparison of inoculation and post-hoc correction in the human-robot perception case may help clarify which human responses to robots lie at the more explicit and which at the more implicit level. A second paradigm is to present the robot twice during a single experimental session, separated by some time delay or unrelated other activities. What happens to people’s representations formed in the first encounter that are either confirmed or disconfirmed in the second encounter? If the initial reactions are mere novelty effects, they would subside independent of the new information; if they are deeply entrenched, they would remain even after disconfirmation; and if they are systematically responsive to evidence, they would stay the same under confirmation and change under disconfirmation [82]. In addition, different response dimensions may behave differently. Beliefs about the robot’s reliability and competence may change more rapidly whereas beliefs about its benevolence may be more stable. In a third paradigm, repeated-encounter but short-term experiments could bring participants back to the laboratory more than once. Such studies could distinguish people’s adjustments to specific robots (if they encounter the same robot again) from adjustments of their general beliefs about robots (if they encounter a different, but comparable robot again). From stereotype research, we have learned that people often maintain general beliefs about a social category even when acquiring stereotypedisconfirming information about specific individuals [83]. Likewise, people may update their beliefs about a specific robot they encounter repeatedly without changing their beliefs about robots in general [82].

16

Chapter 1

Conclusion Trust is one agent’s expectation about the other’s actions. Trust is broken when the other does not act as one expected—is not as reliable or competent as one expected, or is dishonest or unethical. In all these cases, a discrepancy emerges between what one agent expected and the other agent delivered. Human-robot interactions, we suggest, often exemplify such cases: people expect more of their robots than the robots can deliver. Such discrepancies have many sources, from misleading and deceptive information to the seemingly innocuous but powerful presence of deepseated social signals. This range of sources demands a range of remedies, and we explored several of them, from patience to legibility, from incremental design to longitudinal research. Because of people’s complex responses to artificial agents, there is no optimal recipe for minimizing discrepancies and maximizing trust. We can only advance our understanding of those complex human responses to robots, use this understanding to guide robot design, and monitor how improved design and human adaptation, over time, foster more calibrated and trust-building human-robot interactions.

References [1]

[2]

[3]

[4] [5] [6]

D. Ullman and B. F. Malle, “What does it mean to trust a robot? Steps toward a multidimensional measure of trust,” in Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, New York, NY, USA: ACM, 2018, pp. 263–264. D. Ullman and B. F. Malle, “Measuring gains and losses in human-robot trust: Evidence for differentiable components of trust.,” in Companion to the 2019 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’19., New York, NY: ACM, 2019, pp. 618–619. L. Lewis and S. Shrikanth, “Japan lays bare the limitations of robots in unpredictable work,” Financial Times, 25-Apr-2019. [Online]. Available: https://www.ft.com/content/beece6b8-4b1a-11e9-bde6-79eaea5acb64. [Accessed: 04-Jan-2020]. R. K. Moore, “Spoken language processing: Where do we go from here,” in Your Virtual Butler: The Making-of, R. Trappl, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 119–133. M.-A. Williams, “Robot social intelligence,” in Social Robotics, S. S. Ge, O. Khatib, J.-J. Cabibihan, R. Simmons, and M.-A. Williams, Eds. Springer Berlin Heidelberg, 2012, pp. 45–55. K. Fischer, H. M. Weigelin, and L. Bodenhagen, “Increasing trust in human–robot medical interactions: effects of transparency and

Trust and Discrepancy

[7]

[8] [9] [10]

[11]

[12] [13]

[14] [15]

[16] [17]

[18]

17

adaptability,” Paladyn, Journal of Behavioral Robotics, vol. 9, no. 1, pp. 95–109, 2018, doi: 10.1515/pjbr-2018-0007. T. L. Sanders, T. Wixon, K. E. Schafer, J. Y. Chen, and P. Hancock, “The influence of modality and transparency on trust in human-robot interaction,” presented at the Cognitive Methods in Situation Awareness and Decision Support (CogSIMA), 2014 IEEE International InterDisciplinary Conference on, 2014, pp. 156–159. K. Fischer, “Tracking anthromorphizing behavior in human-robot interaction,” Manuscript submitted for publication, 2020. N. Eilan, C. Hoerl, T. McCormack, and J. Roessler, Eds., Joint attention: Communication and other minds. New York, NY: Oxford University Press, 2005. B. Mutlu, T. Kanda, J. Forlizzi, J. Hodgins, and H. Ishiguro, “Conversational gaze mechanisms for humanlike robots,” ACM Trans. Interact. Intell. Syst., vol. 1, no. 2, pp. 12:1–12:33, Jan. 2012, doi: 10.1145/2070719.2070725. K. Fischer, K. Lohan, J. Saunders, C. Nehaniv, B. Wrede, and K. Rohlfing, “The impact of the contingency of robot feedback on HRI,” in 2013 International Conference on Collaboration Technologies and Systems (CTS), IEEE, 2013, pp. 210–217. K. Fischer, K. Foth, K. Rohlfing, and B. Wrede, “Mindful tutors – linguistic choice and action demonstration in speech to infants and to a simulated robot,” Interaction Studies, vol. 12, no. 1, pp. 134–161, 2011. M. Kwon, M. F. Jung, and R. A. Knepper, “Human expectations of social robots,” in Proceedings of the Eleventh ACM/IEEE International Conference on Human Robot Interaction, HRI’16, Piscataway, NJ, 2016, pp. 463–464. R. Mermelshtine, “Parent–child learning interactions: A review of the literature on scaffolding,” British Journal of Educational Psychology, vol. 87, no. 2, pp. 241–254, Jun. 2017, doi: 10.1111/bjep.12147. L. Jussim, S. L. Robustelli, and T. R. Cain, “Teacher expectations and selffulfilling prophecies,” in Handbook of motivation at school., K. R. Wenzel and A. Wigfield, Eds. New York, NY: Routledge/Taylor & Francis Group, 2009, pp. 349–380. R. E. Kraut, “Effects of social labeling on giving to charity,” Journal of Experimental Social Psychology, vol. 9, no. 6, pp. 551–562, Nov. 1973, doi: 10.1016/0022-1031(73)90037-1. M. de Graaf, S. B. Allouch, and J. van Dijk, “Why do they refuse to use my robot?: Reasons for non-use derived from a long-term home study,” in Proceedings of the 2017 ACM/IEEE International Conference on HumanRobot Interaction, 2017, pp. 224–233. M. A. Bobes, M. Valdessosa, and E. Olivares, “An ERP study of expectancy violation in face perception,” Brain and Cognition, vol. 26, no. 1, pp. 1–22, Sep. 1994, doi: 10.1006/brcg.1994.1039.

18 [19] [20]

[21]

[22] [23] [24] [25]

[26]

[27] [28] [29]

[30]

[31]

Chapter 1 J. K. Burgoon, D. A. Newton, J. B. Walther, and E. J. Baesler, “Non-verbal expectancy violations,” Journal of Nonverbal Behavior, vol. 55, no. 1, pp. 58–79, 1989. U. Bruckenberger, A. Weiss, N. Mirnig, E. Strasser, S. Stadler, and M. Tscheligi, “The good, the bad, the weird: Audience evaluation of a ‘real’ robot in relation to science fiction and mass media,” ICSR 2013, vol. 8239 LNAI, pp. 301–310, 2013, doi: 10.1007/978-3-319-02675-6_30. T. Komatsu, R. Kurosawa, and S. Yamada, “How does the difference between users’ expectations and perceptions about a robotic agent affect their behavior?,” Int J of Soc Robotics, vol. 4, no. 2, pp. 109–116, Apr. 2012, doi: 10.1007/s12369-011-0122-y. Vitsoe, “The power of good design,” 2018. [Online]. Available: https://www.vitsoe.com/us/about/good-design. [Accessed: 22-Oct-2018]. G. Donelli, “Good design is honest,” 13-Mar-2015. [Online]. Available: https://blog.astropad.com/good-design-is-honest/. [Accessed: 22-Oct-2018]. C. de Jong, Ed., Ten principles for good design: Dieter Rams. New York, NY: Prestel Publishing, 2017. D. J. Rea, D. Geiskkovitch, and J. E. Young, “Wizard of awwws: Exploring psychological impact on the researchers in social HRI experiments,” in Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, New York, NY, USA: Association for Computing Machinery, 2017, pp. 21–29. W. C. Redding, “Ethics and the study of organizational communication: When will we wake up? In Jaksa, J.A., Pritchard, M.S. (eds.) (pp. ). Hampton, Cresskill,” in Responsible Communication: Ethical Issues in Business, Industry, and the Professions, J. A. Jaksa and M. S. Pritchard, Eds. Cresskill, NJ: Hampton Press, 1996, pp. 17–40. E. C. Collins, “Vulnerable users: deceptive robotics,” Connection Science, vol. 29, no. 3, pp. 223–229, Jul. 2017, doi: 10.1080/09540091.2016.1274959. A. Matthias, “Robot lies in health care: When is deception morally permissible?,” Kennedy Inst Ethics J, vol. 25, no. 2, pp. 169–192, Jun. 2015, doi: 10.1353/ken.2015.0007. K. Wada, T. Shibata, T. Saito, and K. Tanie, “Effects of robot-assisted activity for elderly people and nurses at a day service center,” Proceedings of the IEEE, vol. 92, no. 11, pp. 1780–1788, Nov. 2004, doi: 10.1109/JPROC.2004.835378. E. Karakosta, K. Dautenhahn, D. S. Syrdal, L. J. Wood, and B. Robins, “Using the humanoid robot Kaspar in a Greek school environment to support children with Autism Spectrum Condition,” Paladyn, Journal of Behavioral Robotics, vol. 10, no. 1, pp. 298–317, Jan. 2019, doi: 10.1515/pjbr-2019-0021. H. H. Clark, “How do real people communicate with virtual partners?,” presented at the Proceedings of AAAI-99 Fall Symposium, Psychological Models of Communication in Collaborative Systems, November 5-7th, 1999, North Falmouth, MA., 1999.

Trust and Discrepancy [32] [33]

[34] [35]

[36] [37] [38] [39] [40]

[41] [42]

[43]

[44]

19

K. Fischer, Designing speech for a recipient. partner modeling, alignment and feedback in so-called “simplified registers.” Amsterdam: John Benjamins, 2016. J. Goetz, S. Kiesler, and A. Powers, “Matching robot appearance and behavior to tasks to improve human-robot cooperation,” in The 12th IEEE International Workshop on Robot and Human Interactive Communication, vol. 19, New York, NY: Association for Computing Machinery, 2003, pp. 55–60. B. F. Malle, P. Bello, and M. Scheutz, “Requirements for an artificial agent with norm competence,” in Proceedings of 2nd ACM conference on AI and Ethics (AIES’19), New York, NY: ACM, 2019. E. Sanoubari, S. H. Seo, D. Garcha, J. E. Young, and V. LoureiroRodríguez, “Good robot design or Machiavellian? An in-the-wild robot leveraging minimal knowledge of passersby’s culture,” in Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), New York, NY, USA: Association for Computing Machinery, 2019, pp. 382–391. B. F. Malle and D. Ullman, “A multi-dimensional conception and measure of human-robot trust,” in Trust in human-robot interaction: Research and applications, C. S. Nam and J. B. Lyons, Eds. Elsevier, 2020. K. Fischer and R. Moratz, “From communicative strategies to cognitive modelling,” presented at the First International Workshop on `Epigenetic Robotics’, September 17-18, 2001, Lund, Sweden, 2001. S. Payr, “Towards human-robot interaction ethics.,” in A Construction Manual for Robots’ Ethical Systems: Requirements, Methods, Implementations, R. Trappl, Ed. Cham, Switzerland: Springer International, 2015, pp. 31–62. K. Fischer, What computer talk is and isn’t: Human-computer conversation as intercultural communication. Saarbrücken: AQ-Verlag, 2006. K. R. Fleischmann and W. A. Wallace, “A covenant with transparency: Opening the black box of models,” Communications of the ACM - Adaptive complex enterprises, vol. 48, no. 5, pp. 93–97, 2005, doi: 10.1145/1060710.1060715. M. Hind et al., “Increasing trust in AI services through supplier’s declarations of conformity,” ArXiv e-prints, Aug. 2018. S. R. Fussell, S. Kiesler, L. D. Setlock, and V. Yew, “How people anthropomorphize robots,” in Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction, HRI ’08, New York, NY, USA: Association for Computing Machinery, 2008, pp. 145– 152. J. =áRWRZVNL H. Sumioka, S. Nishio, D. F. Glas, C. Bartneck, and H. Ishiguro, “Appearance of a robot affects the impact of its behaviour on perceived trustworthiness and empathy,” Paladyn, Journal of Behavioral Robotics, vol. 7, no. 1, 2016, doi: 10.1515/pjbr-2016-0005. C. Nass and Y. Moon, “Machines and mindlessness: Social responses to computers,” Journal of Social Issues, vol. 56, no. 1, pp. 81–103, Jan. 2000, doi: 10.1111/0022-4537.00153.

20 [45] [46]

[47]

[48]

[49]

[50]

[51]

[52]

[53] [54] [55] [56]

Chapter 1 H. Admoni and B. Scassellati, “Social eye gaze in human-robot interaction,” Journal of Human-Robot Interaction, vol. 6, no. 1, pp. 25–63, May 2017, doi: 10.5898/JHRI.6.1.Admoni. X. Zhao, C. Cusimano, and B. F. Malle, “Do people spontaneously take a robot’s visual perspective?,” in Proceedings of the Eleventh ACM/IEEE International Conference on Human Robot Interaction, HRI’16, Piscataway, NJ: IEEE Press, 2016, pp. 335–342. C. Bartneck, T. Kanda, O. Mubin, and A. Al Mahmud, “Does the design of a robot influence its animacy and perceived intelligence?,” International Journal of Social Robotics, vol. 1, no. 2, pp. 195–204, Feb. 2009, doi: 10.1007/s12369-009-0013-7. E. Broadbent et al., “Robots with display screens: A robot with a more humanlike face display is perceived to have more mind and a better personality,” PLoS ONE, vol. 8, no. 8, p. e72589, Aug. 2013, doi: 10.1371/journal.pone.0072589. F. Eyssel, D. Kuchenbrandt, S. Bobinger, L. de Ruiter, and F. Hegel, “‘If you sound like me, you must be more human’: On the interplay of robot and user features on human-robot acceptance and anthropomorphism,” in Proceedings of the 7th ACM/IEEE International Conference on HumanRobot Interaction, HRI’12, New York, NY: Association for Computing Machinery, 2012, pp. 125–126. B. F. Malle, “How many dimensions of mind perception really are there?,” in Proceedings of the 41st Annual Meeting of the Cognitive Science Society, E. K. Goel, C. M. Seifert, and C. Freksa, Eds. Montreal, Canada: Cognitive Science Society, 2019, pp. 2268–2274. E. Phillips, D. Ullman, M. de Graaf, and B. F. Malle, “What does a robot look like?: A multi-site examination of user expectations about robot appearance.,” in Proceedings of the Human Factors and Ergonomics Society Annual Meeting., 2017. E. Phillips, X. Zhao, D. Ullman, and B. F. Malle, “What is human-like? Decomposing robots’ human-like appearance using the Anthropomorphic roBOT (ABOT) database,” in Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, New York, NY, USA: ACM, 2018, pp. 105–113. X. Zhao, E. Phillips, and B. F. Malle, “How people infer a humanlike mind from a robot body,” PsyArXiv, preprint, Nov. 2019. K. Fischer, “Interpersonal variation in understanding robots as social actors,” in Proceedings of \em HRI’11, March 6-9th, 2011. Lausanne, Switzerland, 2011, pp. 53–60. S. Payr, “Virtual butlers and real people: Styles and practices in long-term use of a companion,” in Your virtual butler: The making-of, R. Trappl, Ed. Berlin, Heidelberg: Springer, 2013, pp. 134–178. K. M. Lee, Y. Jung, J. Kim, and S. R. Kim, “Are physically embodied social agents better than disembodied social agents?: The effects of physical embodiment, tactile interaction, and people’s loneliness in human–robot

Trust and Discrepancy

[57]

[58] [59]

[60]

[61]

[62]

[63]

[64]

[65]

[66]

21

interaction,” International Journal of Human-Computer Studies, vol. 64, no. 10, pp. 962–973, Oct. 2006, doi: 10.1016/j.ijhcs.2006.05.002. K. S. Haring, K. Watanabe, M. Velonaki, C. C. Tossell, and V. Finomore, “FFAB—The form function attribution bias in human–robot interaction,” IEEE Transactions on Cognitive and Developmental Systems, vol. 10, no. 4, pp. 843–851, Dec. 2018, doi: 10.1109/TCDS.2018.2851569. G. M. Lucas, J. Gratch, A. King, and L.-P. Morency, “It’s only a computer: Virtual humans increase willingness to disclose,” Computers in Human Behavior, vol. 37, pp. 94–100, Aug. 2014, doi: 10.1016/j.chb.2014.04.043. T. Uchida, H. Takahashi, M. Ban, J. Shimaya, Y. Yoshikawa, and H. Ishiguro, “A robot counseling system — What kinds of topics do we prefer to disclose to robots?,” in 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2017, pp. 207– 212. R. Pak, N. Fink, M. Price, B. Bass, and L. Sturre, “Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults,” Ergonomics, vol. 55, no. 9, pp. 1059–1072, Sep. 2012, doi: 10.1080/00140139.2012.691554. L. D. Riek, T. Rabinowitch, B. Chakrabarti, and P. Robinson, “Empathizing with robots: Fellow feeling along the anthropomorphic spectrum,” in 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, 2009, pp. 1–6. S. H. Seo, D. Geiskkovitch, M. Nakane, C. King, and J. E. Young, “Poor thing! Would you feel sorry for a simulated robot? A comparison of empathy toward a physical and a simulated robot,” in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, New York, NY, USA: Association for Computing Machinery, 2015, pp. 125–132. D. Y. Geiskkovitch, D. Cormier, S. H. Seo, and J. E. Young, “Please continue, we need more data: An exploration of obedience to robots,” Journal of Human-Robot Interaction, vol. 5, no. 1, pp. 82–99, 2016, doi: 10.5898/10.5898/JHRI.5.1.Geiskkovitch. M. Salem, G. Lakatos, F. Amirabdollahian, and K. Dautenhahn, “Would you trust a (faulty) robot? Effects of error, task type and personality on human-robot cooperation and trust,” in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI ’15, New York: ACM, 2015, pp. 141–148. A. Powers and S. Kiesler, “The advisor robot: Tracing people’s mental model from a robot’s physical attributes,” in Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-robot Interaction, New York, NY, USA: ACM, 2006, pp. 218–225. V. Chidambaram, Y.-H. Chiang, and B. Mutlu, “Designing persuasive robots: How robots might persuade people using vocal and nonverbal cues,” in Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI ’12), New York, NY, USA: Association for Computing Machinery, 2012, pp. 293–300.

22 [67]

[68] [69]

[70] [71] [72] [73] [74]

[75] [76]

[77]

[78]

[79] [80]

Chapter 1 S. Turkle, C. Breazeal, O. Dasté, and B. Scassellati, “First encounters with Kismet and Cog: Children respond to relational artifacts,” in Digital media: Transformations in human communication, P. Messaris and L. Humphreys, Eds. New York, NY: Peter Lang, 2006, pp. 313–330. C. Lichtenthäler and A. Kirsch, Legibility of robot behavior: A literature review. https://hal.archives-ouvertes.fr/hal-01306977, 2016. M. Kwon, S. H. Huang, and A. D. Dragan, “Expressing robot incapability,” in Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’18), New York, NY, USA: Association for Computing Machinery, 2018, pp. 87–95. G. Briggs and M. Scheutz, “‘Sorry, I can’t do that:’ Developing mechanisms to appropriately reject directives in human-robot interactions,” in Proceedings of the 2015 AAAI Fall Symposium on AI and HRI, 2015. M. de Graaf and B. F. Malle, “How people explain action (and autonomous intelligent systems should too),” in 2017 AAAI Fall Symposium Series Technical Reports, Palo Alto, CA: AAAI Press, 2017, pp. 19–26. H. H. Clark, “Communal lexicons,” in Context in Language Learning and Language Understanding, v, 198 vols., K. Malmkjær and J. Williams, Eds. Cambridge University Press, 1998, pp. 63–87. E. A. Schegloff, “Notes on a conversational practise: formulating place,” in Studies in Social Interaction, D. Sudnow, Ed. New York: Free Press, 1972, pp. 75–119. S. Paepcke and L. Takayama, “Judging a bot by its cover: An experiment on expectation setting for personal robots,” in 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), New York, NY: Association for Computing Machinery, 2010, pp. 45–52. E. R. Teoh, “What’s in a name? Drivers’ perceptions of the use of five SAE Level 2 driving automation systems,” Journal of Safety Research, 2020. J. Bhuiyan, “A federal agency says an overreliance on Tesla’s Autopilot contributed to a fatal crash,” Vox, 12-Sep-2017. [Online]. Available: https://www.vox.com/2017/9/12/16294510/fatal-tesla-crash-self-drivingelon-musk-autopilot. [Accessed: 05-Jan-2020]. F. Lambert, “Tesla driver was eating and drinking during publicized Autopilot crash, NTSB reports,” Electrek, 03-Sep-2019. [Online]. Available: https://electrek.co/2019/09/03/tesla-driver-autopilot-crash-eating -ntsb-report/. [Accessed: 05-Jan-2020]. M. A. Regan, C. Hallett, and C. P. Gordon, “Driver distraction and driver inattention: Definition, relationship and taxonomy,” ACCIDENT ANALYSIS AND PREVENTION, vol. 43, no. 5, pp. 1771–1781, Sep. 2011, doi: 10.1016/j.aap.2011.04.008. M. Rothbart and B. Park, “On the confirmability and disconfirmability of trait concepts,” Journal of Personality and Social Psychology, vol. 50, pp. 131–142, 1986. C. V. Smedegaard, “Reframing the Role of Novelty within Social HRI: from Noise to Information,” 2019 14th ACM/IEEE International

Trust and Discrepancy

[81] [82]

[83]

23

Conference on Human-Robot Interaction (HRI), pp. 411–420, 2019, doi: 10.1109/HRI.2019.8673219. W. J. McGuire, “Inducing resistance to persuasion: Some contemporary approaches,” in Advances in Experimental Social Psychology, vol. 1, L. Berkowitz, Ed. Academic Press, 1964, pp. 191–229. M. J. Ferguson, M. Kwon, T. Mann, and R. A. Knepper, “The formation and updating of implicit impressions of robots.,” presented at the The Annual Meeting of the Society for Experimental Social Psychology, Toronto, Canada, 2019. M. Rothbart and M. Taylor, “Category labels and social reality: Do we view social categories as natural kinds?,” in Language, interaction and social cognition, Thousand Oaks, CA, US: Sage Publications, Inc, 1992, pp. 11– 36.

CHAPTER 2 TALKING TO ROBOTS AT DEPTH ROBERT CODD-DOWNEY, ANDREW SPEERS, MICHAEL JENKIN Electrical Engineering and Computer Science Lassonde School of Engineering York University, Canada

Effective human-robot interaction can be complex at the best of times and under the best of situations but the problem becomes even more complex underwater. Here both the robot and the human operator must be shielded from the effects of water. Furthermore, the nature of water itself complicates both the available technologies and the way in which they can be used to support communication. Small-scale robots working in close proximity to divers underwater are further constrained in their communication choices by power, mass and safety concerns, yet it is in this domain that effective human-robot interaction is perhaps most critical. Failure in this scenario can result in vehicle loss as well as vehicle operation that could pose a threat to local operators. Here we describe a range of approaches that have been used successfully to provide this essential communication. Tethered and tetherless approaches are reviewed along with design considerations for human input and display/interaction devices that can be controlled by divers operating at depth.

Introduction Effective human-robot communication is essential everywhere, but perhaps nowhere is that more the case than when a human is communicating with a robot that is operating underwater. Consider the scenarios shown in Figure 2.1. Here two different robots are shown operating in close proximity to an underwater operator. Effective operation of the robot requires a mechanism

Talking to Robots at Depth

(a) AQUA [13]

25

(b) Milton [11]

Figure 2.1: Divers operating in close proximity with robots underwater. Divers require an effective means to communicate with a robot when operating at depth. A range of potential solutions exist but any such solution must take into account the realities of the operating medium and the cognitive load placed on the diver. (a) shows AQUA [13] a six legged amphibious hexapod being operated in a pool. (b) shows Milton [11] a more traditional thruster-based Unmanned Underwater Vehicle (UUV) being operated in the open ocean. In both cases the robots are shown operating with divers in close proximity.

for the operator (a diver) to communicate instructions to the robot and to have the robot communicate acknowledgment of those instructions and provide other information to the diver. Failure in this communication can lead to mission failure, injury to the diver and damage to, and even loss of, the vehicle. The development of effective communication strategies for Unmanned Underwater Vehicles (UUVs) is critical. Unfortunately not only does the underwater environment require effective communication between a robot and its operator(s), it also places substantive constraints on the ways in which this communication can take place. The water column restricts many common terrestrial communication approaches and even systems that might be appropriate for underwater use tend to offer only low data bandwidth and require high power consumption. Communication underwater is further complicated by the limitations that the underwater environment place on the ways in which the human can utilize a given technology to communicate with the robot and the robot communicate with the human user. For example, recreational SCUBA equipment typically requires the diver to hold a SCUBA regulator in their mouth eliminating voice-based command options. Normal touch input devices (e.g., keyboards, mice, etc.) are difficult to make work underwater. Although such devices can be made waterproof the pressure of water at depth renders many touch-sensitive devices ineffective as the surrounding water pressure is mistaken as user input. Finally, the refractive nature of transparent housings for display can

26

Chapter 2

complicate the readability of displays designed for humans to view, whether located on the robot itself or on some diver-carried display panel. A further issue in robot-diver communication involves cognitive task loading. Tasks that may appear simple at the surface can be difficult for a diver to perform at depth. Cognitive loading as a consequence of diving is well documented [30]. The effects on the cognitive abilities of divers utilizing various gas mixtures, including oxygen-enriched air [2], have also been documented. Task loading is a known risk factor in SCUBA diving [36], and alerting divers to this risk is a component of recreational and commercial dive training. Given these constraints, developing effective interaction mechanisms for divers and robots operating at depth is a complex and challenging task.

Some realities of working underwater Communication between a human operator and a robot typically relies on some medium such as a physical tether (e.g., wire), electromagnetic waves (e.g., radio, light), or acoustic energy (e.g., sound) for communication. The same is true underwater, however the physical properties of the communications medium (e.g., water versus air) places complex restrictions on such options. Water is denser than air. The density of water varies with temperature but for normal operational conditions a density of 1gm/cm3 is a reasonable approximation. Air has a density of approximately 0.001225g/cm3. This difference will cause objects (such as communication cables) that would normally fall to the floor in terrestrial operation to be buoyant or sink depending on their density. Furthermore the high density of water will cause cables or tethers to introduce considerable drag on the vehicle even when suspended in the water column. Buoyant cables will be subject to surface wave and wind action while cables that are denser than water will encounter the normal drag problems associated with terrestrial cables. Depending on the location of the UUV and the operator a cable may be partially buoyant and partially sunk. Terrestrial wireless communication between an operator and a robot is typically straightforward. Standard communication technologies based on radio communication including WiFi and BlueTooth are pervasive and standard technologies exist to support the development of communication protocols based on these and other infrastructures. Water, unfortunately, is not an effective medium for radio wave-based communication. Radio waves are attenuated by water, and by salt water in particular [25]. This typically limits their use to very short distances [43]. Different radio frequencies are

Talking to Robots at Depth

27

attenuated differently by water. Through an appropriate choice of frequencies it is possible to use radio to communicate over short distances underwater (see [4]), but given the constraints with such technologies there is considerable interest in the development of other underwater communication technologies. See [21] for a review.

(a) Shore-based robot control (b) Tether management at the surface Figure 2.2: Tethered communication. Here surface-based operators shown in (a) communicate with a submerged device through the tether shown in (b). Note that the operators do not have direct view of the device, nor do they have direct view of any divers who might be accompanying the device at depth. For this deployment also observe the large number of cable handlers required to service the cable as it travels through the surf zone.

The poor transmission of electromagnetic energy through water is also found with visible light. However the effects are not as significant as compared with the rest of the electromagnetic spectrum. The transmission of light through water is impacted both by the water’s turbidity as well as the nature of the light being transmitted. The absorption of light through one meter of sea water can run from a low of around 2% for blue-green portion of the visible light spectrum through transparent ocean water to over 74% for the red portion in coastal sea water[20]. For a given level of turbidity, the red light band is absorbed much more quickly than the blue-green band. Thus, under natural sunlight, objects at depth that naturally appear red end up appearing more blue-green than they do at the surface. From a communications point of view, such colour loss is often not critical as over short distances (5m or less) the increased absorption of the red band over the blue-green band will not have a significant impact on computer displays. High levels of turbidity, on the other hand, can easily obscure displays associated with the diver-operator and the robot itself.

28

Chapter 2

One final complication with light underwater for human-robot communication is the nature of light refraction. Displays mounted within air-tight housings are typically viewed through flat clear ports. Light travelling from some display through the air within the device, through the port and into the water pass through three different materials and the light is refracted at each boundary according to Snell’s Law. This refraction will introduce a distortion in displays and for each boundary a critical viewing angle exists beyond which the port will act as a reflector rather than allowing a view of the display within. The net effect of this is to require displays to be viewed straight on as much as possible.

(a) Android-based underwater (b) PC-based underwater tablet tablet [41] Figure 2.3: Tethered communication underwater. Tether-based communication can also be accomplished completely underwater. Here the operator utilizes a properly protected interaction device tethered to the vehicle. (a) shows a mobile operator following the robot with a small blue optical tether. (b) shows a larger underwater display and interaction device being operated by a diver. Such devices provide considerably less flexibility than the computer monitor and keyboard input shown in Figure 2.2(b) but allow for direct line-of-sight to the vehicle aiding in the operators situational awareness.

Finally, sound-based communications technology is common underwater (see [29]). Such systems can communicate extremely long distances. Unfortunately such systems can be quite bulky and have considerable power requirements, reducing their potential application involving small scale devices such as those shown in Figure 2.1.

Talking to Robots at Depth

29

Using a physical tether Given the complexities of the underwater environment perhaps the most straightforward mechanism for structuring human-robot communication for autonomous underwater vehicles is from a surface controller or an underwater operator via a physical tether to the UUV (see [26, 22, 1, 37, 38] for examples). While such an approach can provide for excellent communication between the operator and the device as well as providing a conduit for power and vehicle recovery if necessary, a tether, and in particular a surface-based tether also presents several problems. An above surface operator is typically located in some safe, dry location (as shown in Figure 2.2(a)). Here the operator has no direct view of the autonomous vehicle. Furthermore it is typically the case that the operator’s only “view” of the operational environment is via sensors mounted on-board the platform. As a consequence the operator tends to have very poor situational awareness. The tether being managed in Figure 2.2(b) provides both power and data to the robotic sensor operating at the other end of the tether. This particular tether is buoyant which has implications for the control of the sensor package as well as the nature of the drag on the vehicle which is impacted by surface wave action. When using small robots, such as AQUA and Milton, the tether can be fragile and require special care in handling. When working in the field, the operator’s controlling computer, being in close proximity to water, may be unintentionally exposed to environmental contaminants such as water from ocean spray or rain. Reducing this risk requires that the operator be placed at a safe distance from water and thus from UUV operation. This implies longer cables between the robot and semi-dry operator locations, increasing cable management issues and handler communication concerns. The actual UUV operator is, of course, not the only human involved in controlling an underwater vehicle. Although a tether provides a number of advantages, at the end of the day it is a tether that must be properly managed. Different deployments necessitate different tether management strategies but personnel must be deployed in order to deal with problems that arise in standard operation of the tethered vehicle. Figure 2.2 illustrates the complexity of this problem for the shore deployment of an underwater sensor package. A number of personnel are engaged in the task and the ability of the various personnel to communicate among each other effectively is key to successful UUV deployment. This problem becomes even more acute underwater where personnel (divers) must be deployed to manage the tether between the UUV and the operator. The problems with

Chapter 2

30

tethered operations become even more severe at depth. Communication with submerged divers can be problematic. Divers are limited in their ability to assess the state of the robot, relying instead on confirmation from the operator passed through surface-based cable wranglers.

(a) Li-Fi Modems

(b) Li-Fi Modems underwater

Figure 2.4: Li-Fi modems for operation underwater. (a) shows the modems in their housings with an array of LEDs and photo diodes for light generation and capture. (b) shows the same modems deployed underwater. As Manchester coding is used for encoding the message in the light, the source light does not appear to flicker but rather appears as a dim constant light source.

An alternative to having the UUV teleoperated from the surface involves placing the operator in close proximity to the UUV and then operating the vehicle from underwater. This is shown in Figure 2.3. Teleoperation at depth allows the operator to interact directly with the robot. This enables a number of different operational modes not possible with a ship- or shorebased operator. For example, a diver operating at a relatively safe depth (say 80’) can teleoperate a vehicle operating 30’-40’ deeper, without exposing the diver to the increased dangers associated with the lower dive profile. An underwater tether can also be used to enable a diver to remain outside of potentially dangerous environments while the robot operates within them. For example, a robot could be sent to investigate the inside of a wreck, while allowing the diver to remain outside. Unfortunately the nature of the underwater environment limits the kinds of interaction that the operator can engage in, and the remote interaction device and cognitive loading is also a serious issue. The remote interaction device used by the diver-operator needs to be as neutrally buoyant as possible so as to minimize the effect of the device on the diver-operator’s ability to maneuver underwater. The device shown in Figure 2.3(a), for example, has a very small form factor in

Talking to Robots at Depth

31

part to reduce the effect of the buoyancy of the device on the diver. The device shown in Figure 2.3(b) is negatively buoyant, which makes operating the robot from the seabed more comfortable than operating the robot from the middle of the water column.

(a) Static marker

(b) Dynamic marker

Figure 2.5: Visual fiducial markers. Both static (a) and dynamic (b) fiducial markers can be used to communicate command information through a visual channel.

Given the complexities of tether-based operation technologies, there is a desire for communication approaches that can replace the physical tether with some form of wireless technology that is suitable for underwater operation. The requirement that a diver operates in close proximity to the robot and the small form factor of the robot limits some of the potential technologies that might be deployed. Sound-based technologies require a power budget that is unlikely to be available on a device that could be carried by the diver or small scale form-factor vehicle. Sound itself might pose health risks to the diver and other marine life at certain decibel levels. RF-based technology requires considerable power to operate over the distances that would be required. Given these constraints visible light-based communication is an appropriate choice. Modern LED-based lighting systems utilize very little power, and by limiting the power of any light sources used we can ensure that the light is safe for any diver.

Encoded light-based communication Given that light travels long distances underwater, at least outside of the red portion of the visible light spectrum, visible light would seem to be an appropriate medium for underwater communication. Underwater wireless optical communication (UWOC) can either be based on LASER-based light

32

Chapter 2

sources or on the use of regular light, typically generated through a array of low power LEDs or a single high powered LED. See [28] for a recent review of both approaches. Regardless of the technology used to generate the light, the basic approach is to encode the message through modulating the light source and then observing this modulation at the receiving end of the communication. Given the frequency-dependency of light absorption in water, typically light sources in the white or blue-green spectrum are used. Light also works as a conduit upon which communication can be built terrestrially. Light-Fidelity communication (Li-Fi) aims to use visible light as the communication medium for digital communication. (See [17] for a review of the technology.) Although still in its infancy, Li-Fi has shown substantive promise. There have, however, been few large-field tests of the technology. Beyond the terrestrial domain there have also been a number of efforts to deploy Li-Fi technology underwater. For example, the transmission properties of different light sources for Li-Fi have been studied underwater, leading to the observation that LED-based communication has advantages while underwater when line of sight cannot be guaranteed [24]. At a systems level, a long distance (100m) light-based communication system has been demonstrated that utilizes optics to concentrate the emitter and a single photon avalanche diode to enhance detection [42]. The IEEE 802.15.7 standard for visible light communication (VLC) [32] utilizes on-off keying (OOK) to encode the format of the data stream from the transmitter to the receiver. The basic idea here is that by turning a light on and off at the transmitter using a message-driven encoding the receiver can decode this sequence into the transmitted message. A popular approach for this OOK process is Manchester encoding [40], which is a recommended OOK approach in the IEEE VLC standard. Essentially this approach modulates the data stream using a clock signal. One downside of this mechanism is its relatively high overhead in terms of the communication signal, consuming 100% more bandwidth than a raw encoding scheme. Figure 2.4 shows the experimental Li-Fi underwater modem described in [8, 9]. A key problem in the deployment of Li-Fi underwater is the construction of an appropriate light emission/collection device that can operate underwater and that is more or less agnostic to misalignment errors between the emitter and receiver. The Light-Byte modems shown in Figure 2.4 utilize a ring of emitter/receivers that provide a 360ƕ range of light emission/detection in a reasonably wide vertical band. All processing of the incoming and outgoing light signals is performed within the underwater housings themselves allowing the units to appear as USB modems to the external computers or robots.

Talking to Robots at Depth

33

One problem with deploying Li-Fi-based systems outdoors is that the technology must compete with other light sources present in the environment. In essence, the receiver must be able to extract the encoded light message from the ambient light. Utilizing a brighter emitter source can help, but it is it difficult to compete with the Sun, especially on cloudless days. Underwater this means that performance of Li-Fi modems is actually worse near the surface and that performance improves markedly with depth. Short pass light filters may be a reliable mechanism to overcome this limitation.

Visual target-based communication Fiducial markers are two-dimensional binary tags that convey information to the observer. Technologies such as ARTags [15], April-Tags [27] and Fourier Tags [34] can be used to determine the pose of an observer with respect to the tag. Such tags can also be used to communicate messages along with pose information. The amount of information can be very limited or can encode a large number of bytes. For example, QRCodes [12] can encode a large amount of custom binary data. In this type of communication a two-dimensional visual target is presented to the robot which captures the image using an on-board camera and processes the image stream to obtain the intended message. One benefit of target-based communication for human to robot communication is that in an underwater environment the small amount of processing power required to localize and recognize the target within an image can be very beneficial. A collection of unique visual targets allows for the development of a simple command language, where each tag corresponds to a different command. Even a simple set of visual command targets can provide effective vehicle control given a controlled operational environment. Sequences of tags such as those found in RoboChat [14] can be strung together to describe sophisticated tasks. Figure 2.5 illustrates this process in action for both static pre-printed targets as well as dynamically generated targets on an underwater display. The use of static fiducial target-based communication is effective but cumbersome, as operators need to carry a library of tags. Finding the specific tag needed for the next part of a command sequence is arduous and it is possible to accidentally show the wrong card to the robot while searching for the correct tag. The use of custom or dynamic tags has also been explored [41]. This technique utilizes an underwater tablet-like device that allows the user to select and present a series of control commands to the robot (Figure 2.5(b)). The robot captures these images and once verified a compact sequence of tags that encode the command sequence is generated

34

Chapter 2

and can be shown to the robot. This approach can reduce the complexity of carrying a large collection of tags but requires the development of a suitable underwater display and interaction box.

(a) Marker identification

(b) Refraction error

(c) Reflection error Figure 2.6: Complexities of dynamic marker identification. Although the basic task of marker identification and recognition can be straight- forward underwater the nature of the protective housing introduces a range of complexities. (b) shows refraction-based error where the incidence angle is sufficiently large that the marker housing acts as a mirror obscuring the view of the target. (c) shows reflection in the port of the underwater housing. Even though the internal target is visible, a clear view of the target is obscured by the reflection of the camera (here held by a diver).

Figure 2.6 illustrates the normal process of dynamic marker identification along with some potential pitfalls associated with the approach. Figure 2.6(a) illustrates the desired outcome. This figure is from the output of the target identification process. The view of the target and its housing has been overlayed by a red rectangle, illustrating the localization of the target, and the target identity is printed over the target itself. The process of actually capturing this target can be frustrated in a number of ways. The first is due to refraction effects (Figure 2.6(b). Here the oblique viewing angle of the target past the critical viewing angle causes the surface of the display to act as a mirror and reflect light rather than pass light through the port. Figure 2.6(c) shows a further failure mode where the robot or some other participant – here the robot operator – is reflected in the port. Notwithstanding these types of errors, reasonably high recognition rates (57%) with no false positives at

Talking to Robots at Depth

35

30 frames-per-second are reported (resulting in more than 17 measurements of the target being made each second) [39].

Interaction hardware and software Many of the technologies used to communicate messages to the robot from an underwater diver-operator require some mechanism for input and display. Terrestrial robot control can exploit standard input devices including keyboards, joysticks, and phone/tablet interfaces to communicate with a robot. Underwater the choices are more limited. A critical requirement is that the input device be protected from water and pressure and be operable by a diver working at depth. In general this means constructing some waterproof enclosure that provides both access to any display and input devices. It is also possible to use the entire waterproof enclosure as a joystick by augmenting the device with an appropriate tilt sensor and to use the entire device as a pointing device through the use of a compass or IMU. Although the waterproof container can be machined to be relatively light when empty, it is critical that this container be (approximately) neutrally buoyant when deployed underwater. If it is not neutrally buoyant then the diver-operator will have to compensate for this when operating the device which complicates the diver-operator’s task. In order for the entire tablet to be neutrally buoyant it must weigh the same as the weight of the water it displaces. In essence this limits the usability of large volume underwater housings. It is certainly possible to build such housings but the large volume of the housing will require that the housing be weighted through either the inclusion of very heavy components or through the addition of external mass. The resulting device may be “weightless” underwater, but will difficult to move when underwater and to deploy from the surface. Beyond mass and buoyancy there are a number of other issues in terms of the design of any housing that must be taken into account. The surface area of the device is a concern. The housing acts as a reaction surface underwater, acts as a drag when the diver is swimming, and acts as a sail in strong current or swells. Each penetration into the underwater housing is a potential failure point, so although more inputs may be desirable each one increases the risk of flood-failure to the device. Deploying a robot and its support team typically means entry through the surf or off of a boat and thus switches must be positioned so as to minimize the potential for accidental operation during deployment. Cognitive loading of the diver-operator is an issue, and although it is possible to provide large numbers of controls it can be difficult for the operator to make effective use of them. Operators will view displays through their dive goggles, the water column and a transparent

Chapter 2

36

port on the interaction device. Detailed displays may be difficult to view, especially in turbid water conditions. Figure 2.7 shows a lightweight interaction device for tethered control of an underwater vehicle [38]. The control device is housed in a custom fabricated housing as shown in Figure 2.7(b). An Android Nexus 7 tablet provides both a display and on board computation to process switch and other inputs and to condition the data for transmission to the robot.

(a) Interaction

(b) Housing

Figure 2.7: Devices for underwater interaction with a robot must be designed to be operated at depth. This requires displays and interaction mechanisms that are appropriate for divers. (a) shows a diver operating AQUA. The tablet acts as a joystick and provides simple interaction mechanisms controlled by two, three-state switches. (b) shows the diver’s view of the sensor.

The availability of small form factor displays and interaction devices with general purpose computing, such as Android tablets, provides a number of options for input and display. Sensors within the tablet itself, including a compass and IMUs, can be exploited for robot control, they have internal batteries for power, they typically support WiFi, Bluetooth and USB communication, and provide a rich software library for control. One disadvantage of such devices is that they do not support standard robot control software middlewares such as ROS [31]. Being able to have the interaction device communicate via ROS considerably simplifies communication between the interaction device and the robot itself. Even for devices that are not physically wired to the robot, using ROS as a common communication framework between the interaction device and the robot has benefits. ROS provides an effective logging mechanism, and there exist visualization tools that can be used post deployment to help in understanding any issues that may have arisen during deployment.

Talking to Robots at Depth

37

Within ROS overall robot control is modeled as a collection of asynchronous processes that communicate by message passing. Although a very limited level of support does exist for ROS on commodity computer tablets, environments such as Android or iOS are not a fully supported ROS environment. In order to avoid any potential inconsistencies between these environments and supported ROS environments, one option is to not build the software structures in ROS directly, but rather to exploit the RosBridge mechanism instead. RosBridge provides a mechanism within which ROS messages are exposed to an external agent and within which an external agent can inject ROS messages into the ROS environment. This injection process uses the standard WebSocket protocol. The process of developing specific interaction devices for robot-operator control can be simplified by automating much of the specialized software required to map interaction devices to ROS commands for interaction. Software toolkits that can be used to semi-automatically generate display and interaction tools using the RosBridge communication structure have previously been developed for Android [37] and iOS [7] mobile platforms.

Gesture-based communication Rather than augmenting the diver with some equipment that assists in diver-robot communication it is possible to deploy novel gesture-based communication languages (e.g., [19, 18, 3, 5, 6]). Although such approaches can be effective, they require divers to learn a novel language for humanrobot communication while retaining their existing gesture-based language for diver-diver communication. In addition to the increased cognitive load on divers, such an approach also has the potential for accidental miscommunication among divers and between divers and robots, given the common symbols used in the two gesture languages. Rather than developing a novel gesture-based language for human-diver communication, another alternative is to leverage existing diver-diver gesture-based communication. Divers have developed a set of effective strategies for implicit and explicit communication with other divers. A standard set of hand gestures (signals) have been developed with special commands for specific tasks and environments. National and international recreational diving organizations such as PADI teach these languages and help to establish and maintain a standard set of gesture symbols and grammar. These gestures – which include actions such as pointing with the index finger and obvious motions of the hand while it is held in some configuration – are strung together in a simple language. For example, to indicate that there is something wrong with one’s ears, one would indicate “unwell” by holding the hand flat and

38

Chapter 2

rotating the wrist and then pointing at the effected ear. One observation about these signals is that their semantics are dependent on their location relative to the body. Thus a critical step in having a robot understand normal diver gestures involves identifying the relative position of the hands to the body and the configuration and motion of the hands. (See Figure 2.8.) Diverdiver gesture-based communication relies on both an explicit and implicit communication strategy. For example, given the lack of a straightforward mechanism to name locations underwater, many commands conveying coordinated motion are phrased as “you follow me” or “we swim together” and rely on the implicit communication of aspects of the diver’s state.

(a) Diver pointing at their ear

(b) Tracked hand position

Figure 2.8: Divers utilize a standard set of gestures to communicate with other divers underwater. (a) To indicate that there is an issue with their ear – for example some problem with pressure equalization – the diver would indicate that something is wrong and then point at their ear. Understanding this communication involves tracking the diver’s hand relative to their head. (b) Results of this tracking plotted in head-centric coordinates.

Implicit understanding of a diver’s state requires some mechanism to track and monitor the diver as they move. Autonomous following of an underwater diver has been explored in the past. Perhaps the simplest diverfollowing technique involves augmenting the diver in some manner to simplify diver tracking. For example, [35] adopts the use of an atypically coloured ball and simple computer vision techniques to track the diver. This requires the diver to hold onto an inflated ball of some recognizable colour, which will affect their buoyancy control negatively. Other methods (e.g., [33, 18]) track flipper oscillations of a certain colour in the frequency domain to determine the location within the scene to track. Another

Talking to Robots at Depth

39

approach is to just recognize and localize the diver as part of a more general diver communication mechanism. A range of different technologies exist that could be used to identify and track diver body parts in order to localize the diver and to understand the actions being performed. Both traditional image processing techniques as well as data driven (e.g., neural network-based) approaches could be utilized. Previous work in this area includes an approach based on transfer learning using a pre-trained Convolutional Neural Network (CNN) to identify parts of divers, which are then tracked through time [10]. Deploying a CNN requires an appropriately trained dataset and an appropriate CNN that can be applied to the problem. This is addressed in part through the SCUBANet dataset [10]. The SCUBANet dataset contains underwater images of divers taken from both freshwater and saltwater environments. The freshwater portion of the dataset was collected in Lake Seneca in King City, Ontario, Canada. The saltwater portion of the dataset was collected just off the west coast of Barbados. The SCUBANet dataset was collected using the Milton robot [11] and consists of over 120,000 images of divers. CNNs require that the dataset be properly labelled. SCUBANet’s dataset was labelled using a crowd-sourcing tool and as of 2019 over 3200 image annotations had been performed. Work performed in our lab utilizes a transfer-learning approach to recognize diver parts. Transfer learning involves taking some pretrained CNN from a related task, using a larger dataset for the initial training, then training the final level (or levels) using a smaller task-specific dataset. For this task, a CNN trained on the COCO dataset for object detection, segmentation and captioning [23] was used with the final level being retrained on the SCUBANet dataset [10]. This process is typically much faster than training a new CNN from scratch. Performance of the resulting networks can be very good. For example, [10] reports that the retrained faster rcnn inception v2 architecture demonstrated an average recognition rate for divers, heads and hands of 71.6% mean average precision at 0.5 intersection over union [10]. Figure 2.8 shows the first steps in diver-robot communication using diver gestures. Individual frames from a video of the diver are captured by the robot and processed to identify and localize body parts important for both implicit and explicit communication. Here the diver’s head and hand positions are tracked while the diver points to their ear. When plotted in a head-centric frame of reference the motion of the diver’s hand as it is raised to and then lowered from the side of the diver’s head is clear (Figure 2.8(b)). Ongoing work is investigating different techniques for labelling the specific hand/finger gestures being used during these motions.

40

Chapter 2

Summary Communication between humans and robots is transitioning from keyboard inputs performed by machine specialists to interactions with the general public and with specialists who have been trained for particular tasks. Many of these tasks have developed multi-modal communication structures that are designed to meet the specifics of the task as hand. As robots move out of the lab and into application domains it is critical that the communications strategies used are appropriate for the task at hand. This is true for many terrestrial tasks but is especially true underwater. Here the environment places constraints on the technology available for the human to talk to the robot and for the robot to talk to the human. Although it is certainly possible for the human to remain warm and dry above the surface of the water and to communicate either wirelessly (e.g., through sound) or through a physical tether, the lack of a direct view of the operation being undertaken reduces considerably the situational awareness of the operator. Placing the operator in the water with the robot creates its own set of problems. Underwater tethers can certainly be used, but this then requires careful tether management, because a tangled tether underwater is a threat to both the operator and the robot. Wireless communication must be safe for the diver and not place undue power requirements on the diver, thus limiting the use of RF and sound-based technologies. Visible light-based technologies would seem appropriate although here again it is critical to not place undue constraints on the diver operator. Carried interaction devices must work well at depth neither upsetting the diver’s buoyancy nor placing undue load on the diver’s cognitive abilities. Perhaps the most desirable diver to robot communication strategy is to exploit the normal diver to diver gesture-based communication strategy. Recent work suggests that such an approach has promise and ongoing research is exploring how best to understand complex statements and commands based on gesture underwater. Table 2.1 provides a summary of the various technologies presented in this chapter. This chapter has concentrated on diver to robot communication. Communication from the robot to the diver, especially when approaches such as gesture-based are used that do not augment the diver, is also an issue. Given that robots are often augmented with lights and displays then clearly these devices can be leveraged for robot to diver communications. But other options are possible. For example, it is possible for the robot to choose specific motions to encode simple yes/no responses to queries and even more complex motion sequences can be exploited to communicate more sophisticated messages to the diver [16].

Talking to Robots at Depth

Strategy Physical tether

Acoustic Static visual target Dynamic visual target

UWOC Specialized gesture language

Diver gesture language

41

Properties Advantages: High data bandwidth, reasonably low cost, bi-directional, support for standard communication protocols. Disadvantages: Tether management, tether drag. Advantages: Good range, not impacted by turbidity. Disadvantages: Power requirements, potential damage to divers and marine life, low bandwidth. Advantages: Inexpensive, easy to deploy. Disadvantages: Restrictive communication set, potential for accidental target viewing by robot, low bandwidth. Advantages: Large symbol set, easy to structure the display so as to avoid accidental target display to the robot. Disadvantages: Requirement of a display device, complexity of viewing the target through the display port, bandwidth can be improved by ganging together multiple symbols. Advantages: Low-power, relatively high bandwidth. Disadvantages: Short-range, impacted by turbidity. Advantages: Can be tuned to the task at hand, can be easily learned, can be designed to be easily recognized by the robot. Disadvantages: Increased cognitive loading on the diver, possibility of confusion between symbol set used in diver to diver communication and diver to robot communication, low bandwidth. Advantages: Well known by divers. Disadvantages: Complex gestures to recognize, low bandwidth.

Table 2.1: The communication strategies described in this chapter along with their advantages and disadvantages.

Bibliography [1]

T. Aoki, T. Maruashima, Y. Asao, T. Nakae, and M. Yamaguchi, “Development of high-speed data transmission equipment for the full-depth remotely operated vehicle – KAIKO,” in OCEANS, vol. 1, Halifax, Canada, 1997, pp. 87–92.

42 [2]

[3]

[4] [5] [6] [7]

[8] [9] [10] [11] [12] [13]

[14] [15]

Chapter 2 A.-K. Brebeck, A. Deussen, H. Schmitz-Peiffer, U. Range, C. Balestra, and S. C. J. D. Schipke, “Effects of oxygen-enriched air on cognitive performance during scuba-diving – an open-water study,” Research in Sports Med., vol. 24, pp. 1–12, 2017. A. G. Chavez, C. A. Mueller, T. Doernbach, D. Chiarella, and A. Birk, “Robust gesture-based communication for underwater human-robot interaction in the context of search and rescue diver missions,” in IROS Workshop on Human-Aiding Robotics, Madrid, Spain, 2018, held in conjunction with IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). X. Che, I. Wells, G. Dickers, P. Kear, and X. Gong, “Re-evaluation of rf electromagnetic communication in underwater sensor networks,” IEEE Communications Magainze, vol. 48, pp. 143–151, 2011. D. Chiarella, M. Bibuli, G. Bruzzone, M. Caccia, A. Ranieri, E. Zereik, L. Marconi, and P. Cutugno, “Gesture-based language for diver-robot underwater interaction,” in OCEANS, Genoa, Italy, 2015. D. Chiarella, “A novel gesture-based language for underwater human-robot interaction,” J. of Marine Science and Engineering, vol. 6, 2018. R. Codd-Downey and M. Jenkin, “RCON: dynamic mobile interfaces for command and control of ROS-enabled robots,” in International Conference on Informatics in Control, Automation and Robotics (ICAR, Colmar, France, 2015. R. Codd-Downey, “LightByte: Communicating wirelessly with an underwater robot using light,” in International Conference on Informatics in Control, Automation and Robotics (ICINO), Porto, Portugal, 2018. R. Codd-Downey, “Wireless teleoperation of an underwater robot using lifi,” in Proc. IEEE International Conference on Information and Automation (ICIA), Wuyishan, China, 2018. R. Codd-Downey, “Finding divers with SCUBANet,” in IEEE International Conference on Robotics and Automation (ICRA), Montreal, Canada, 2019. R. Codd-Downey, M. Jenkin, and K. Allison, “Milton: An open hardware underwater autonomous vehicle,” in Proc. IEEE International Conference on Information and Automation (ICIA), Macau, China, 2017. CSA: Standards Council of Canada, “Information Technology - Automated Identification and Capture Techniques - QR Code 2005 Bar Code Symbol Specification,” 2016, Canadian Government Report. G. Dudek, P. Giguere, C. Prahacs, S. Saunderson, J. Sattar, L.-A. TorresMendez, M. Jenkin, A. German, A. Hogue, A. Ripsman, J. Zacher, E. Milios, H. Liu, and P. Zhang, “AQUA: An amphibious autonomous robot,” IEEE Computer, vol. Jan., pp. 46–53, 2007. G. Dudek, J. Sattar, and A. Xu, “A visual language for robot control and programming: A human-interface study,” in IEEE International Conference on Robotics and Automation (ICRA), Rome, Italy, 2007, pp. 2507–2513. M. Fiala, “Artag, a fiducial marker system using digital techniques,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, San Diego, CA, 2005, pp. 590–596.

Talking to Robots at Depth [16]

[17] [18] [19]

[20] [21] [22] [23] [24]

[25] [26] [27] [28] [29] [30] [31]

43

M. Fulton, C. Edge, and J. Sattar, “Robot communication via motion: Closing the underwater human-robot interaction loop,” in International Conference on Robotics and Automation (ICRA), Montreal, Canada, 2019, pp. 4660– 4666. H. Haas, L. Yin, Y. Wang, and C. Chen, “What is LIFI?” J. of Lightwave Technology, vol. 34, pp. 1533–1544, 2016. M. J. Islam, “Understanding human motion and gestures for underwater human-robot collaboration,” J. of Field Robotics, vol. 36, 2018. M. J. Islam, M. Ho, and J. Sattar, “Dynamic reconfiguration of mission parameters in underwater human-robot collaboration,” in IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 2018, pp. 1–8. N. G. Jerlov, Optical Oceanography. Amsterdam, The Netherlands: Elsevier, 1968. H. Kaushal and G. Kaddoum, “Underwater optical wireless communication,” IEEE Access, vol. 4, pp. 1518–1547, 2016. P. Lee, B. Jeon, S. Hong, Y. Lim, C. Lee, J. Park, and C. Lee, “System design of an ROV with manipulators and adaptive control if it,” in 2000 International Symposium on Underwater Technology, Tokyo, Japan, 2000, pp. 431–436. T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, “Microsoft COCO: common objects in context,” CoRR, vol. abs/1405.0312, 2014. P. Medhekar, S. Mungekar, V. Marathe, and V. Meharwade, “Visible light underwater communciation using different light sources,” International Journal of Modern Trends in Engineering and Research, vol. 3, pp. 635–638, 2016. R. K. Moore, “Radio communciation in the sea,” Spectrum, vol. 4, pp. 42– 51, 1967. M. Nokin, “ROV 6000 – objectives and description,” in OCEANS, vol. 2, Brest, France, 1994, pp. 505–509. E. Olson, “AprilTag: a robust and flexible visual fiducial system,” in IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, 2011. H. Oubei, C. Shen, A. Kammoun, E. Zedini, K. Park, X. Sun, G. Liu, C. H. Kang, T. K. Ng, M. S. Alouini, and B. S. Ooi, “Light based underwater wireless communications,” Japanese J. of Applied Physics, vol. 57, 2018. D. Pompili and I. F. Akyildiz, “Overview of networking protocols for underwater wireless communications,” IEEE Commun. Mag., vol. Jan., pp. 97–102, 2009. S. F. Pourhashemi, H. Sahraei, G. H. Meftahi, B. Hatef, and B. Gholipour, “The effect of 20 minutes SCUBA diving on cognitive function of professional SCUBA divers,” Asian J. of Sports Med., vol. 7, 2016. M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs, E. Berger, R. Wheeler, and A. Y. Ng, “ROS: an open-source robot operating system,” in Open-Source Software workshop at the International Conference on Robotics and Automation (ICRA), Kobe, Japan, 2009.

44 [32] [33]

[34]

[35]

[36] [37] [38] [39]

[40] [41]

[42] [43]

Chapter 2 S. Rajagopal, R. D. Roberts, and S. K. Lim, “IEEE 802.15.7 visible light communication: modulation schemes and dimming support,” IEEE Communications Magazine, vol. 50, p. 72–82, 2012. J. Sattar and G. Dudek, “Where is your dive buddy: tracking humans underwater using spatio-temporal features,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sand Diego, CA, 2007, pp. 3654–3659. J. Sattar, E. E. Bourque, P. Giguere, and G. Dudek, “Fourier tags: Smoothly degradable fiducial markers for use in human-robot interaction,” in Canadian Conference on Computer and Robot Vision (CRV), Montreal, Canada, 2007, pp. 165–174. J. Sattar, P. Giguere, G. Dudek, and C. Prahacs, “A visual servoing system for an aquatic swimming robot,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Edmonton, Canada, 2005, pp. 1483– 1488. R. W. Smith, “Application of a medical model to psychopathology in diving,” in 6th International Conference on Underwater Education, San Diego, CA, 1975, pp. 377–385. A. Speers, P. Forooshani, M. Dicke, and M. Jenkin, “Lightweight tablet devices for command and control of ROS-enabled robots,” in International Conference on Advanced Robotics (ICAR), Montevideo, Uruguay, 2013. A. Speers and M. Jenkin, “Diver-based control of a tethered unmanned underwater vehicle,” in International Conference on Informatics in Control, Automation and Robotics (ICINCO), Rekjavik, Iceland, 2013. A. Speers, A. Topol, J. Zacher, R. Codd-Downey, B. Verzijlenberg, and M. Jenkin, “Monitoring underwater sensors with an amphibious robot,” in Canadian Conference on Computer and Robot Vision (CRV), St. John’s, Canada, 2011. A. S. Tanenbaum and D. J. Wetherall, Computer Networks. Pearson, 2011. B. Verzijlenberg and M. Jenkin, “Swimming with robots: Human robot communication at depth,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Taipei, Taiwan, 2010, pp. 4023– 4028. C. Wang, H.-Y. Yu, and H.-J. Zhu, “A long distance underwater visible light communication system with single photo avalanche diode,” IEEE Photonics J., vol. 8, pp. 1–11, 2016. A. Zoksimovski, C. M. Rappaport, D. Sexton, and M. Stojanovic, “Underwater electromagnetic communications using conduction – channel characterization,” in Proc. of the 7th ACM International Conference on Underwater Networks and Systems, Los Angeles, CA, 2012.

CHAPTER 3 TOWARDS THE IDEAL HAPTIC DEVICE: REVIEW OF ACTUATION TECHNIQUES FOR HUMAN-MACHINE INTERFACES MACIEJ àĄ&.,, CARLOS ROSSA Ontario Tech University, Faculty of Engineering and Applied Science Oshawa, Ontario Canada

Abstract The ideal haptic device has no friction, mass, or inertia, and it can render an infinite range of impedances. As a result of delays and signal discretization in computer-controlled systems, conventional haptic devices, that use active actuators, are fundamentally limited in the range of impedance they can generate. It is well-known that there exists an intrinsic trade-off between stability and transparency. In the pursuit of the ideal haptic device, different actuation methods can be employed. Passive actuators, like brakes, render a wide range of impedance as they are not subjected to the aforementioned trade-off. However, passive actuators are prone to stiction, cannot generate force in arbitrary directions, nor can they restore energy to the user. These characteristics limit their use to a narrow range of haptic applications. Hybrid actuators, comprising both active and passive actuators, combine the characteristics of its components. As a result, these actuators are not theoretically limited in the range of impedance they generate. Some hybrid configurations increase the impedance range by improving their ability to render high impedance, while other ones prioritize minimization of the lowest impedance. In this chapter, we review recent advancements in the field of passive and hybrid haptic actuation. We highlight the design considerations and trade-offs associated with these actuation methods and provide guidelines

Chapter 3

46

on how their use can help with the development of the ultimate haptic device.

1. Introduction The kinesthetic sense of touch plays a key role in humans’ capability to perform even basic tasks [1]. In some situations, like in teleoperated surgery [2–4], fly-by-wire [5], drive-by-wire systems [6, 7], or surgical simulation [8–12] the mechanical linkage between the user and the tool is broken or inexistent. In such situations, haptic force-feedback devices can restore the connection between the user and the tool, virtually. Haptic devices are robotic manipulators designed to provide forcefeedback to a human operator. A haptic device acts like a universal coupling between a virtual or distant tool and the user by conveying the dynamics and the forces acting on the tool to the operator, like in Fig.1.1(a). From a control perspective, the interaction between the user and the virtual tool can also be represented as a network diagram as in Fig.1.1(c). During a haptic simulation, the device captures the motion of the user șr and transmits it to the virtual environment. Based on the transmitted motion, the virtual environment simulates the motion of a virtual tool șv. As the user moves the device, the virtual tool can encounter an object which may impede its motion. The virtual environment determines the desired torques IJv required to render the mechanical impedance acting on the tool ܼ௩ = ߬௩ /ߠሶ௩ [13–15]. This signal is then sent to the haptic device which generates a torque IJr. An ideal haptic device should accurately transmit the position and forces between the virtual environment and the user i.e., it should act as a universal coupling such that ߬௩ ݄ ቂߠ ቃ = ൤ ଵଵ ݄ଶଵ ௥

݄ଵଶ ߠ௩ ൨൤ ൨ ݄ଶଶ ߬௥

(1.1)

where h21 = h12 = 1 and h11 = h22 = 0, resulting in IJr = IJv and șr = șv. To this end, the device should render objects with an infinite range of impedance, infinite bandwidth, experience no delays, all the while having no losses, and no discernible mass m or inertia J.

Towards the Ideal Haptic Device

(a)

47

(b)

(c) Figure 1.1. Universal coupling shown in (a) conveys the motion and forces acting on the virtual tool (dashed) to the user. A haptic device functions according to (c) where Zv represents the virtual environment, ߠሶ௥ and ߠሶ௩ the user and the virtual tool motion, while IJr and IJv are the torques acting on the user and on the virtual tool. In reality, the coupling between the user and the tool reassembles the one shown in (b), where the spring-damper system represents the viscous friction and the rigidity of the device.

Being physical objects, haptic devices have a mass and they are subject to losses such as viscous friction causing ݄ଵଵ ് 0 and as such ߬௥ ് ߬௩ . As digital systems, their position measurements are subject to delays resulting from quantization and discretization of continuous signals, meaning that ݄ଶଵ ് 1 and ߠ௥ ് ߠ௩ . The performance of a haptic device can be assessed against the ideal device using mainly three metrics. First, the mechanical transparency defined as, ߟ௧ =

௓೏ ௓ೌ

(1.2)

where Zd represents the desired impedance and Za the output impedance, quantifies the disparity between the generated impedance and the desired impedance. In other words, (1.2) measures the distortion of the output force resulting from mechanical imperfections and limitations of the physical mechanism. An ideal device should be transparent, meaning that there should be no difference between the desired and the output

Chapter 3

48

impedance, i.e, Za = Zd, results in Șt = 1. However, the impedance of the device, represented by ZL, results in Zd/(Za + ZL) < 1. Second, the Z-width is a measure of haptic device’s impedance range defined as ߟ௭ =

௓೘ೌೣ ௓೘೔೙

(1.3)

where Zmax and Zmin respectively represent the maximum and the minimum impedance generated by the device. An ideal device should render impedances varying from zero to infinity, thus, ideally Șz ĺ ’ [16,17]. The device’s inertia and friction, however, mean that ܼ௠௜௡ = ܼ௅ ് 0and the physical limitations of the actuators mean that Zmax is finite, resulting in Șz with a finite value. Lastly, the bandwidth is the frequency range of force generated by the device. Low-frequency, high-amplitude forces render the geometry of a virtual object while minute details of the shapes, like their texture, are rendered using low-impedance at high-frequencies. An ideal device with an infinite bandwidth can render both shape and its texture. As a result of inertia and delays in the actuators, haptic devices are typically able to render only a finite band of frequencies. This band may be situated either in the lower or higher end of the frequency spectrum constraining the device to rendering either low or high frequency forces. Consider the model of a haptic system shown in Fig.1.2. A user moves the haptic device, which has an inertia J and a viscous damping b, to interact with a virtual environment H(z) modelled as a stiff virtual wall with some damping characteristics. The virtual wall cannot exert energy on the user in excess of what is provided to it, i.e., the environment acts as a passive impedance H(z) = Z(v) [17]. The user is the only source of energy in the haptic loop. From the perspective of a haptic device, however, the user impedes motion and, in fact, stabilizes the haptic device. The user is, therefore, modelled as a passive impedance Zu [13,14,18–21]. Since all components in Fig.1.2 are passive, the entire system should be passive as well [22]. Note, however, that the virtual environment is a discrete system that measures the position of the user ș at a sampling period of T. The actuator acts like a zero-order-hold function, íeíTs)/s, converting the discrete desired torque into a continuous signal. The delay induced by the discretization of a continuous signal injects energy into an otherwise passive system possibly leading to instability. The response of the haptic loop to the injected energy depends on the type of actuator used. Conventional haptic devices, for instance, use electric motors to generate forces. As we will show in Section 1.2, these

Towards the Ideal Haptic Device

49

devices need to dissipate energy, through viscous damping, proportionally to the impedance of the virtual environment to maintain stability. Due to the reliance on the damping, active haptic devices have a limited upper and lower bound of the Z-width so they cannot be perfectly transparent. Passive actuators, on the other hand, only dissipate energy making them intrinsically stable. As a result, passive devices have higher Z-width and transparency than active devices. Their passivity, however, limits the forces they can generate as they cannot restore energy to the user, as discussed in Section 1.3. Haptic devices equipped with hybrid actuators, ones comprising motors, brakes, and other components, can generate a wider Z-width than active devices and they do not suffer from the same limitations as purely passive or active devices.

Figure 1.2. The user, modelled as impedance Zu, applies IJu(s) to the device with inertia J and damping b, resulting in angular velocity ߠሶ(‫)ݏ‬. The position ș(s) is sampled at a sampling period T , resulting in a discrete position signal ș(z), which is conveyed to the virtual environment H(z). The virtual environment calculates the desired torque IJd(z) and directs it to the motor M, which acts like a zero-orderhold function (1 íeíTs)/s, converting it into a continuous signal IJd(s).

In Section 1.4, we discuss various implementations of hybrid actuators as well as their associated control schemes. The limitations, challenges, and considerations associated with active, passive, and hybrid actuation methods are addressed in Section 1.5. Finally, Section 1.6 provides recommendations and applications for each type of actuator.

50

Chapter 3

2. Active Actuators Conventional haptic devices, which includes all devices on today’s market, use active actuators, most commonly DC motors [7,17,23–25], to render impedances. These actuators rely on an external source of energy to generate forces allowing them to provide energy to the system. These actuators are relatively inexpensive and exhibit a relatively linear current response making them easy to control [26]. Electric motors, however, tend to generate relatively small torque and often rely on gearboxes or capstan transmissions to amplify the torque. The gearboxes increase the inertia, amplify viscous friction, and introduce nonlinearities [27–29]. Specialized haptic interfaces can use other types of active actuators. A planar device presented in [30] uses linear induction motors to render forces in the plane of the desk. The actuators in this type of system do not move but, instead, generate a magnetic field which results in a force applied directly to the tool. Since there is no mechanical linkage between the tool and the actuator, these devices can be very transparent, but their force range is relatively small [31]. Air muscles are another type of active actuator. They generate force by expanding or contracting like human muscles using compressed air. Air muscles generate high forces while having relatively low mass, but due to the compressibility of air they experience delays and a nonlinear force response making them difficult to control [32, 33]. Irrespective of the actuator, however, there is an inherent trade-off between the stability and transparency of an active haptic device.

2.1 Trade-off Between Stability and Transparency For an illustrative example, consider a haptic loop composed of a haptic device with a typical sampling rate T = 1 kHz [34] rendering a virtual environment composed of a relatively soft spring with a stiffness of K = 500 kN/m. The user approaches the virtual wall at a steady speed xÚ = 100 mm/s. Assume that at sample k = n the distance between the user and the virtual wall ¨x = 0 mm and no force is applied to impede the user. At k = n + 1 the user moved ¨x = í100 μm into the wall which should require an energy input of 50 J. Even though the user has not provided any significant energy into the system, the virtual environment exerts 10 N of force on the user. The excessive force generated by the device pushes the user out of the wall at a rapid velocity and leads to instability [35,36]. To minimize the amount of energy injected into the system, the resolution of position sensors must be maximized, the sampling delay minimized, and the velocity signal must be filtered [16]. Reducing the

Towards the Ideal Haptic Device

51

amount of injected energy, however, will not stabilize the haptic system as stability requires dissipation of all the excess energy. Haptic devices can only dissipate energy through mechanical losses in the form of viscous damping b. The amount of friction required to stabilize the device depends on the sampling period of the control loop and the impedance of the environment. For a haptic device to be stable it must satisfy the stability condition, defined as ܾ>

்௄ ଶ

+‫ܤ‬

(1.4)

where B and K respectively represent the viscous damping and stiffness of the virtual wall [17]. In other words, the impedance range of an active haptic device depends on its ability to dissipate energy. Since energy is only dissipated through viscous damping, an active haptic device cannot be perfectly transparent while maintaining stability.

2.2 Stabilizing Active Haptic Devices The stable impedance range of an active haptic device can be expanded beyond the limit imposed by the stability condition (1.4) by using specifically developed control schemes. There are two methods of extending the impedance range: rejecting the excess energy resulting from delays or filtering the force output of the virtual environment using a virtual coupling. Virtual Coupling: Let us replace the rigid universal coupling in Fig.1.1(a) with a coupling comprised of a spring with stiffness Kc and a damper with a viscous friction coefficient Bc connected in parallel like in Fig.1.1(b). As a result, a user in contact with an infinitely stiff virtual wall will not sense the impedance of the virtual wall but instead the impedance of the virtual coupling. In other words, the desired force IJd is not directly determined by the virtual environment, but instead by the coupling such that where ߬௖ and ߠሶௗ are the torque and velocity, respectively, between the coupling and the virtual environment [15,37,38].

Chapter 3

52 ௄ ்௭

െ ೎ െ ‫ܤ‬௖ ߬ௗ (‫)ݖ‬ ൨ = ቎ ௄௭ିଵ ൤ ்௭ ߬௖ (‫)ݖ‬ െ ೎ െ‫ܤ‬ ௭ିଵ

௖

‫ܭ‬௖ + ‫ܭ‬௖ +

஻೎ (௭ିଵ)

ߠሶ (‫)ݖ‬ ்௭ ቏൤ ௗ ൨ ஻೎ (௭ିଵ) ߠ(‫)ݖ‬

(1.5)

்௭

Figure 1.3. The virtual coupling C acts as an intermediary between the main haptic loop and the virtual environment H(z) in Fig.1.2. The torque applied to the user IJd(z) is a function of the user position ș(z) and the velocity of the virtual coupling ߠሶௗ (‫)ݖ‬, while IJc(z) is the force between virtual environment.

The coupling effectively acts like a filter that limits the maximum impedance displayed to the user guaranteeing system stability [15]. However, it limits the maximum impedance of the haptic device, while also introducing a discrepancy between the position of the real and the virtual tools [39]. These limitations can be avoided all together if the output of the motor is restricted from providing energy in excess to that provided by the user. Passivity Observation and Control: In an interaction with a virtual wall the energy provided by the user to the system is calculated using a passivity observer, ‫ܧ‬௢௕௦ (݊) = ȟܶ σ௡௞ୀ଴ ߬௜௡ (݇) ߠሶ (݇)

(1.6)

where IJin is the force input of the user and k represents some sampling period between zero and n. For the device to be passive Eobs < 0 ‫׊‬k. Given an estimate of system energy and the desired torque IJd, the torque output of the device becomes, where

߬௔ = ߬ௗ + ߙߠሶ ିா೚್ೞ

ߙ = ቊ ்ఏሶ 0

(1.7) ݂݅‫ܧ‬௢௕௦ > 0

(1.8)

‫݁ݏ݅ݓݎ݄݁ݐ݋‬

which virtually dissipates the energy injected by the delay [39,40]. The time-domain passivity observer was improved in [41–44] and adapted for multi-DOF devices in [45]. Eliminating excess energy in the

Towards the Ideal Haptic Device

53

haptic device using this scheme is limited by the accuracy of the energy estimation. Thus, the scheme stabilizes a haptic device for a wide, but finite, range of operating conditions [42,46]. Absolute stability of a haptic device can be achieved by replacing the motor with a device that is physically incapable of providing energy.

Figure 1.4. The user, modeled as impedance Zu, applies IJu(s) to the device with inertia J and damping b, resulting in angular velocity ߠሶ(‫)ݏ‬. The position ș(s) is sampled at a sampling period T, resulting in a discrete position signal ș(z), which is conveyed to the virtual environment H(z). The virtual environment calculates the desired torque IJd(z) and directs it to the brake B which acts like a zero-orderhold function (1 í eíTs)/s, converting it into a continuous signal IJd(s). The brake torque depends on the velocity, thus only the magnitude of torque is controllable, as indicated by the diagonal arrow.

3. Passive Actuators A brake is an actuator that generates torque by dissipating kinetic energy making it intrinsically stable. As a result, only the magnitude of a brake’s torque can be controlled while the direction depends on its velocity, as indicated in Fig.1.4 by the diagonal line crossing the brake B. The choice of brake type for a passive haptic device is a key consideration. Friction brakes, commonly used in automotive applications, are not suitable as their response is nonlinear, prone to high hysteresis, and they are difficult to control. Haptic devices, instead, employ electromechanical actuators such as electrorheological (ER), magnetorheological (MR), eddy current, or particle brakes. MR and ER brakes use a liquid substance that changes its viscosity under the influence of a magnetic or an electric field, respectively. These brakes generate a wide range of impedance, but they suffer from response time greater than that of a motor. ER brakes require high voltages, on the

54

Chapter 3

scale of kilovolts, to function, further complicating their control [47, 48]. MR brakes, on the other hand, operate at much lower voltages, but they have a higher off-state torque resulting from the remnant magnetic flux in the core [49]. Both MR and ER brakes have a higher torque-to-mass ratio than DC motors resulting in devices that have a higher Z-width [48,50– 52]. Instead of using a fluid that changes viscosity depending on the magnetic flux, particle brakes use a fine magnetic powder that binds together when subjected to a magnetic field. Moving the rotor through these particles applies a resistive force on the rotor, resulting in torque at the output shaft. Particle brakes are neither as strong nor as efficient as MR or ER brakes, but they have a relatively linear currenttorque response [53,54]. Unlike other actuators employed in haptic applications, eddy current brakes act like a controllable viscous damper; their torque output depends on the relative velocity of the input and output shafts. An eddy current brake is composed of a rotor made from an electric conductor and a stator generating magnetic field. As the rotor moves through the magnetic field an eddy current is induced, creating a magnetic field that opposes that of the stator [55]. These actuators are easy to control due to their linear current-torque response, fast response time, and minimal losses resulting from the lack of physical connection between the rotor and the stator [56]. Passive actuators offer fast response times, linear current-torque responses, and low energy consumption. As most of these actuators can also generate higher torque than a similarly sized motor, they do not require a gearbox, minimizing losses of the resulting device [57,58]. There are two major issues that affect passive haptic devices. The first one, stiction, is a result of the difference between the static and the dynamic friction coefficient of physical objects. When a brake stops moving, its torque output increases leading to the rendered objects feeling sticky. Additionally, brake’s inability to add energy into the system inhibits them from rendering a force in arbitrary directions.

3.1 Force Output of a Passive Haptic Devices A brake has two modes of operation. When moving, the brake can only generate torque opposing its velocity. While stationary, the torque of the brake opposes any torque input i.e.,

Towards the Ideal Haptic Device

െ‫ߠ(݊݃ݏ‬ሶ )|߬ௗ | ݂݅ ߠሶ ߬ௗ < 0, ߠሶ ് 0 ߬௔ = ቐെ‫߬(݊݃ݏ‬௜௡ )|߬ௗ | ݂݅ ߬ௗ ߬௜௡ < 0, ߠሶ = 0 0 ‫݁ݏ݅ݓݎ݄݁ݐ݋‬

55

(1.9)

where IJin is the torque input and IJa the torque output [59]. Clearly, only the magnitude of the brake’s torque is controllable. As a result, passive devices cannot generate a force in an arbitrary direction making them incapable of rendering some virtual environments; even ones that are totally passive. For instance, consider a simple passive environment composed of a linear spring. As the user compresses the spring, the brake opposes the compression by dissipating energy. Assuming that eventually the user stops compressing the spring, the stored energy should be released by applying the force to the user. With a passive device, however, this is impossible as the brake cannot restore energy. The issues with force displayability are further complicated by the nonlinear relationship between the velocity and force at the end-effector, and the angular velocity and torque at the joints. Consider a 2-DOF device attempting to render a stiff virtual wall with no friction shown in Fig.1.5(a). The end-effector of the device moves with a velocity ‫ ܄‬towards the wall, with which it collides. To render the wall, the device must apply a force ۴ௗ , which acts perpendicular to the wall. Through the manipulator kinematics, ‫ ܄‬results in ߠሶଵ > 0 and ߠሶଶ > 0. As a consequence of (1.9), the device can only generate torques such that IJ1 < 0 and IJ2 < 0. Applying only IJ1 results in a force R1 and applying IJ2 results in R2, while applying both torques simultaneously results in forces contained in region Ÿ2 in Fig.1.5(b). Since Fd is not located in Ÿ2, the force cannot be rendered even though the desired force opposes velocity i.e., Fd · V < 0 [60].

Chapter 3

56

(a)

(b)

Figure 1.5. The 2-DOF haptic device in (a) moves with a velocity V and attempts to render a virtual wall, represented as a grey line, by generating force Fd using combination IJ1 < 0 and IJ2 < 0 resulting in forces R1 and R2, respectively. The device can generate forces in the region Ÿ2, compared to the range of an active device represented by Ÿ1. Lastly, Ÿ3 represents the regions where the passive device can approximate forces.

3.2 Improving Force Output of a Passive Haptic Devices The controllers intended for passive haptic devices must address the two key issues: limited range of displayable forces and stiction. To date, two types of controllers have been developed to address these two issues. Alternate Brake Locking: The first class of control methods for passive haptic devices locks all but one brake to induce motion along a single DOF path to approximate the motion of the virtual object [61]. The control scheme was refined in [62] where a narrow band above the virtual surface was introduced. Inside of the band, the controller switches the actuated and the un-actuated brake, creating a zig-zag motion approximating the geometry of the surface. These controllers eliminate stiction, as there is always at least one brake allowing free motion, by approximating the motion along the virtual surface; a trait that motivated the development of other controllers for passive haptic devices. Force Approximation: As an alternative, it is possible to approximate a desired force by imitating its effects using a displayable force, increasing the displayable force range to Ÿ3 in Fig.1.5(b). Originally developed for underactuated wire-based devices [63], the force approximation scheme was refined and adapted for 2-DOF passive device in [64], and later for 3DOF in [65]. Consider Fig.1.6, where the user applies a force Fin into the endeffector which is in contact with a stiff, frictionless virtual wall. By

Towards the Ideal Haptic Device

57

applying the desired force, the user is stopped from penetrating the virtual environment resulting in the net force on the end-effector Nd acting parallel to the surface and preventing penetration.

Figure 1.6. When rendering a firm virtual wall, the force input Fin and the desired force Fd should result in a net force at the end-effector Nd, parallel to the surface. If Fd cannot be rendered, one of the actuators, R1 in this case, can be used to apply a force that will result in the net force on the end-effector N parallel to Nd.

The device cannot render the desired force, however, it is possible to generate a force F such that the resulting net force is also parallel to the virtual surface. This is achieved by determining the magnitude of a scaling factor ߙ=െ

۴೏ ‫ڄ‬۴೔೙ ۴೏ ‫ ܀ڄ‬భ

(1.10)

such that F = R1Į, like in Fig.1.6 [64]. Apart from increasing the displayable range of forces, this control scheme eliminates stiction since the net force at the end-effector induces sliding along the virtual surface. For the force approximation scheme to work, the controller must have an accurate estimate of the user input force, thus, requiring a heavy and costly force sensor. The reliance on the force sensor can be reduced using a modified force approximation scheme, presented in [66], where the energy of the system is used to calculate Į. Force approximation control of the passive device increases the range of displayable forces and eliminates stiction; addressing the main issues of a passive haptic device.

58

Chapter 3

4. Hybrid Actuators Passive haptic devices are intrinsically stable but they cannot restore energy to the user or generate forces in arbitrary directions. Active devices, in contrast, generate forces in arbitrary directions but the magnitude of the impedance they can render is constrained to a narrow stability range. A hybrid transducer combines the advantages of both motors and brakes. Hybrid actuators can be designed with combinations of active and passive actuators as well as using dampers [67, 68], springs [54], and transmissions [69]. These actuators increase the Z-width of a haptic device in one of two ways: increase the maximum impedance Zmax by improving the stability and force range of the actuators, or by minimizing inertia and mechanical losses to lower the minimum impedance Zmin.

Figure 1.7. The user, modeled as impedance Zu, applies IJu(s) to the device with inertia J and damping b, resulting in angular velocity ߠሶ(‫)ݏ‬. The position ș(s) is sampled at a sampling period T, resting in a discrete position signal ș(z), which is conveyed to the virtual environment H(z). The virtual environment calculates the desired torque IJd(z) and sends it to the controller, which divides the desired torque into IJdb(z) and IJdm(z) directed to the brake B and the motor M, respectively. The two actuators convert the discrete signal into a continuous one using the zero-order-hold function íeíTs)/s, resulting in IJdb(s) and IJdm(s). The brake torque depends on the velocity, thus only the magnitude of its torque is controllable, as indicated by the diagonal arrow.

The cross-section view of several hybrid actuator configurations are shown in Fig.1.8. The user interacts with the end-effector represented by a black dot connected to the actuator shaft, shown as an empty narrow rectangle. Actuators, represented by pairs of blocks labelled ’B’ for brakes and ’M’ for motors, apply the torque to the shaft.

Towards the Ideal Haptic Device

59

4.1 Increasing the High End of the Z-width (Zmax) The actuator shown in Fig.1.8(a) is the most common hybrid configuration for haptic devices [25, 70–75], where a motor and a brake are connected to the same shaft in parallel. The fusion of the two actuators allows the system to render high impedance in many directions while maintaining stability. Note, however, that the use of two actuators increases the total inertia Jtotal = Jm + Jb. Additionally, this actuator is prone to stiction just like the brake alone. One way of addressing the issue of stiction is the same as with a passive device; adding a force sensor and releasing the brake when needed. Another method of eliminating stiction was proposed in [76], where two unidirectional brakes were used in parallel in a passive device, as shown in (b). Each unidirectional brake, represented by an ’L’ shaped block ’B’, can generate torque in only one direction. Consequently, impeding the user’s motion in one direction does not affect his ability to move in the other direction. This design was modified to include an electric motor in [77], resulting in configuration (c).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Chapter 3

60

(m)

Figure 1.8. Schematic cross-section view of hybrid actuators. The user interacts with the end-effector represented by a black circle connected to the actuator shaft, shown as a hollow narrow rectangle. Brakes, labelled ’B’, and motors, ’M’, may be attached to the housing, represented by dashed lines, like in (a), or to the shaft of another actuator, represented by a ’U’ shaped bracket like in (d). A geared serial connection that rotates in the direction opposite to the shaft, like the one in (g) is indicated by dashed lines. Other components like springs, viscous dampers, unidirectional brakes, and differential are represented using symbols shown above. .

The maximum torque IJmax, of parallel hybrid actuators varies from the peak torque of the motor IJm to the peak torque of both the motor and the brake IJb, depending on the control scheme. For instance, one way to increase the maximum impedance of the device is to use a brake as a programmable damper bb so that the stability condition becomes [25,56,73], ܾ + ܾ௕ >

்௄ ଶ

+ ‫ܤ‬.

(1.11)

Assuming that bb is sufficiently high to satisfy the stability condition, the maximum torque of the actuator is limited only by the torque of the motor. Considering that brakes can often generate higher torque than a similar-sized motor, a controller capable of utilizing the brake alongside the motor should further increase Zmax of this hybrid configuration. Another class of control scheme for parallel hybrid actuators divides the torque output between the motor and the brake. The energy-bounding approach [77] employs an energy observer to determine the total energy of the system. If the energy is negative, the system is passive and the motor generates the desired torque. If, however, the energy becomes positive, the torque is directed to the brake which dissipates the energy. The controller guarantees stability but it still underutilizes the brake. This issue was addressed in the same paper by stiffness-bounding algorithm [77]. The

Towards the Ideal Haptic Device

61

controller estimates the stiffness and damping of the virtual environment, compares the estimated stiffness to the limit imposed by (1.4) and uses the motor to render the maximum stable stiffness while the remainder is rendered by the brake. In [78, 79], a low-pass filter feeds the desired position to the brake, while the motor renders high-frequency, lowamplitude forces. As a result, the brake generates most of the torque output improving stability, increasing Zmax, and reducing the actuator response time. Lastly, the controller presented in [80] uses an artificial neural network to determine the optimal division of torque among the two actuators by considering motor stability, brake’s ability to generate forces, and model-based friction compensation. As a result, this controller not only increases Zmax but it also lowers Zmin.

4.2 Lowering the Low End of the Z-width (Zmin) Using the parallel configurations in Fig.1.8(a) or (c) with a large brake for rendering impedance and a small motor for friction [81–83], gravity [84], inertia [85] compensation, can improve the lower end of the Z-width Zmin. The motor can also compensate for the response time of the brake increasing the bandwidth of the device [75]. These compensation schemes will reduce the apparent inertia and friction allowing the user to better sense low impedances. Parallel hybrid actuators are intrinsically limited in their ability to render the lower boundary of the Z-width and high-frequency torques. As the frequency of the torque increases, the compensation of friction and inertia becomes more challenging due to the limited response time of the motors. There are, however, hybrid actuator arrangements designed specifically for rendering low impedances at high frequencies. The configuration (d) referred to as the micro-macro or course-fine manipulator, introduced in [86, 87] and adapted for haptic applications in [88, 89], uses a large grounded motor (the macro manipulator) to move a smaller motor (micro manipulator) coupled directly to the end-effector. Since there is no direct connection between the larger motor and the endeffector, the user senses the inertia and friction of the smaller motor. The smaller motor responds quickly and, thus, is able to render a highfrequency impedance. Note that the torque developed by the smaller motor is also the maximum torque of the actuator, thus it cannot render a high impedance. To increase the impedance range, the smaller motor can be replaced by a small brake, like in (e) [90], or a controllable damper similar to (f) [67]. The brake, acting as a clutch or a viscous damper controls the torque

62

Chapter 3

transmission between the motor and the user. Much like configuration (d), it can generate a high-frequency impedance, but the upper torque limit is bounded by the brake [67, 90]. As opposed to the micro-macro manipulator, this configuration struggles to render forces in arbitrary directions. The brake can rapidly change the amount of torque provided to the user but only if the direction of motion stays constant. Once the direction changes, the macro actuator must change its direction of motion, introducing additional response latency. Adding a second brake connected to a shaft spinning in the opposite direction, like in (g), eliminates this issue [91]. In configuration (h), a large motor is connected to an eddy current brake, acting as viscous coupling, and to a smaller motor [68]. Like in the previous examples, the large motor provides high torques, while the eddy current brake controls the amount of torque transmitted to the end-effector. The smaller motor, on the other hand, compensates for the delay introduced by the eddy current brake. When interacting with the device, the user only senses the inertia and friction of the smaller motor.

4.3 Other Hybrid Configurations Series Elastic Actuators (SEA) form another class of hybrid actuators characterized by inclusion of elastic elements in their design. The simplest forms of SEAs, in Fig.1.8(i) and (j) combine, respectively, a motor or a brake connected in series with a spring. The presence of the spring decouples the dynamics of the actuator from the end-effector. As a result, the apparent inertia is reduced and the torque output is controlled through deflection of the spring. The compliance introduced by the spring makes it unsuitable for haptic applications. A refined design, presented in (k), eliminates the issue of deflection by adding a second motor into the design. The first motor can apply a torque directly to the user, while the second motor controls the energy stored in the spring. The torque of the spring and the torque of the first motor sum together exposing the user to the inertia of just the first motor. By replacing one of the motors with a brake, like in (l), the device can also collect energy from the user while minimizing energy consumption [54]. The device presented in [54] uses a large brake to control the energy stored in the spring enabling it to both dissipate and restore energy. The motor, on the other hand, activates only when the torque in the spring acts opposite to the desired torque. In such a situation, the motor provides the torque to the user, while the brake dissipates the energy stored in the spring.

Towards the Ideal Haptic Device

63

Lastly, a design proposed in [69] uses a motor connected to a high reduction gearbox connected in series with a spring, like in (i), with a brake and a differential, as shown in (m). This configuration provides multiple operating modes for the actuator. When the brake is unlocked and the motor is stationary, the user can move the end-effector freely while perceiving only the inertia of brake and the differential. By locking the brake, all the energy provided by the user is stored by the spring. Both the brake and the motor can control the amount of energy stored by the spring and the torque exerted on the user. Clearly, there are many hybrid actuator designs, many of which have properties desirable in haptic devices. Let us now discuss the design considerations, challenges, and opportunities related to the design of haptic devices using active, passive, and hybrid actuators.

5. Discussion Most, if not all, haptic devices available today use only active actuators despite their limitations [7, 92–96]. Passive and hybrid actuation techniques are an alternative that can improve stability, impedance range, and bandwidth of a haptic interface. To develop devices using these actuation techniques, however, a designer must consider a number of challenges and fundamental limitations of each type of actuator.

5.1 Passive Haptic Devices Due to their high torque-to-mass ratio and relatively low response time, MR or ER brakes are the most suitable passive actuators for haptic applications. These actuators, however, must be custom designed and built due to their limited market availability [50, 51]. Particle brakes, in contrast, have a lower torque-to-mass ratio but they are widely available for purchase making them far easier to implement [53,54]. Passive haptic devices are difficult to design and control due to their restricted range of displayable forces as well as the fact they are prone to stiction. To date, most of the passive haptic devices are constrained to planar 2-DOF [57,61,62,64, 66,97–99]. There have been only a handful of attempts at developing higher DOF devices, which included four 3-DOF devices [52, 65, 100, 101], one 4-DOF [102], and one 6-DOF device [103]. For non-redundant passive haptic devices, the ability to generate force in arbitrary directions diminishes exponentially as the number of DOF increases [104]. As a result, higher DOF passive haptic devices must rely on force approximation schemes to generate the majority of the forces.

64

Chapter 3

Apart from force approximation, the range of displayable forces a passive haptic device generates can be increased by redundantly actuating the device [97,98,105, 106]. Note, however, that even a redundantly actuated device will not be able to restore energy to the user, therefore, it will never render forces such that F·V > 0, as it would require the brake to add energy into the system. Adding additional actuators also increases the friction and inertia of the device. A passive device must always be equipped with a force sensor further increasing its mass. The reliance on the force sensor varies from requiring accurate measurement of the force at all times in [64, 65] to only requiring the force measurement when the device sticks [66], but the sensor is an indispensable component of the device. The mass of a planar passive device working perpendicular to the force of gravity does not affect transparency. For higher DOF devices, however, gravity becomes a significant concern that can be compensated using one of two passive methods. The first method attempts to statically balance each link using a counterweight, such that the sum of moments due to gravity on any link is zero [107]. Adding counterweights increases the inertia of the device leading to a poorer display of low impedances. Alternatively, the moment at each link can be balanced by springs attached to the links of the device [108]. This method is not as effective at canceling the forces of gravity, but it does not increase the inertia of the device.

5.2 Hybrid Haptic Devices There are few documented haptic devices using hybrid actuators. The devices developed thus far spanned from 2-DOF [75, 109–111], 3-DOF [32, 54, 112], and one partially hybrid 6-DOF device [113]. The lack of multi-DOF hybrid devices suggests that the design of such devices poses a considerable challenge. The first challenge is the selection of a hybrid actuator configuration. There are numerous configurations with varying characteristics and no single configuration can simultaneously expand the top and bottom ends of the device’s Z-width; the two objectives are in conflict. Parallel hybrid actuators are the most suitable for the majority of haptic devices as they can easily replace electric motors. Serial actuators, on the other hand, have limited utility due to the relatively low forces they generate. The second challenge pertains to the use of hybrid actuators in multiDOF devices. Hybrid actuators have a much higher mass and volume than either a brake or a motor, which can increase the inertia of the resulting haptic devices. For a higher-DOF device to remain transparent, they must

Towards the Ideal Haptic Device

65

use active inertia compensation schemes [85] or hardware designs that minimize the inertia caused by the actuators. To this end, the motion of the actuator relative to the base of the device must be eliminated using one of the following two methods. Tendon drives convey torque of a motor to a distant joint through tension of the ’tendon’ connecting the two [114]. This method increases the complexity of the mechanism and introduces nonlinearities making the resulting device more difficult to control [115]. Parallel kinematic structures, on the other hand, connect multiple kinematic chains together, such that only the first joint in each link requires actuation [116]. The reduced inertia of the device comes at the expense of the workspace size, as parallel manipulators have smaller workspace than comparable serial configurations. The next challenge relates to control. There is no single universal control scheme addressing all the issues of a hybrid haptic device. The control schemes developed thus far focus on addressing a subset of associated issues and limitations. An optimal control scheme for hybrid actuators should use both the motor and the brake to render high impedances. The brake should always stabilize the motor, while the motor should compensate for the friction, inertia, and gravity of the haptic system. Preferably, the majority of the torque should be generated using the brake to minimize energy consumption. The optimal control scheme should not require high computational power, such that it can be implemented on a standard microcontroller.

6. Conclusions An ideal haptic device has no mass, no inertia, no friction, infinite bandwidth, and it can render a wide range of impedance. Currently, active actuators are the predominant actuation method for haptic devices, but they are known to have a limited impedance range due to stability concerns. Passive actuators, on the other hand, are intrinsically stable so they can generate higher impedance ranges. Their inability to generate energy, however, promotes stiction and limits the direction of forces they can generate. Hybrid actuators generate force in arbitrary directions while maintaining stability when rendering high impedance. Each type of actuator has key characteristics that make it uniquely suitable for specific haptic applications. For instance, active haptic devices are the simplest type to implement due to their relatively simple construction and control. This makes them ideal for low-cost interfaces, where the stability and accuracy of the force reproduction are of lesser importance. Active devices will most likely find use in applications like

66

Chapter 3

gaming, computer control, or computer-based visual design, among many others. The key attribute of passive haptic devices, on the other hand, is their unparalleled stability. This makes them ideal for mission-critical applications, where stability is paramount to haptic fidelity. These applications include automotive driver’s assist, guiding systems for surgical applications, and master devices for teleoperation robots. Finally, hybrid devices have the widest Z-width that is, theoretically, only bounded by the physical actuator limitations. Depending on the configuration, these hybrid actuators can improve the low or the high end of the impedance range. The low impedance configurations are well suited for rendering textures and the subtle forces such as the ones involved in neurosurgery. Simulators and teleoperated systems, on the other hand, benefit from the high impedance variants of hybrid actuators, since they can accurately render impedance acting on the virtual or teleoperated tool. Of various actuation techniques, the hybrid actuators are the most robust solution for haptic applications, even though they present several challenges; they are difficult to construct and challenging to control. However, these do not appear to be fundamental and insurmountable issues compromise the theoretical performance of the haptic device. Further work on improving the physical design, such as miniaturization, and development of optimal control laws for hybrid devices may eventually lead to the development of an ultimate haptic device.

Bibliography [1] [2] [3] [4] [5] [6] [7]

G. Robles-De-La-Torre, “The importance of the sense of touch in virtual and real environments,” Ieee Multimedia, vol. 13, no. 3, pp. 24–30, 2006. G. Hubens et al., “What have we learnt after two years working with the da vinci robot system in digestive surgery?” Acta Chirurgica Belgica, vol. 104, no. 6, pp. 609–614, 2004. M. .LWDJDZD et al., “Effect of sensory substitution on suture-manipulation forces for robotic surgical systems,” The Journal of thoracic and cardiovascular surgery, vol. 129, no. 1, pp. 151–158, 2005. S. M. Prasad et al., “The effect of robotic assistance on learning curves for basic laparoscopic skills,” The American journal of surgery, vol. 183, no. 6, pp. 702–707, 2002. G. Jayaraman and Z. Szulyk, “Active sidestick design using impedance control,” in 2009 IEEE International Conference on Systems, Man and Cybernetics. IEEE, 2009, pp. 4432–4437. J. J. Gil et al., “New driving control system with haptic feedback: Design and preliminary validation tests,” Transportation research part C: emerging technologies, vol. 33, pp. 22–36, 2013. V. Hayward et al., “Haptic interfaces and devices,” Sensor review, 2004.

Towards the Ideal Haptic Device [8] [9] [10] [11]

[12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22]

67

B. Sainsbury et al., “Evaluation of a virtual reality percutaneous nephrolithotomy (PCNL) surgical simulator,” Frontiers in Robotics and AI, vol. 6, p. 145, 2019. E. M. Overtoom et al., “Haptic feedback, force feedback, and force-sensing in simulation training for laparoscopy: a systematic overview,” Journal of surgical education, vol. 76, no. 1, pp. 242–261, 2019. M. Bajka et al., “Evaluation of a new virtual-reality training simulator for hysteroscopy,” Surgical endoscopy, vol. 23, no. 9, p. 2026, 2009. G. Ahlberg et al., “Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies,” The American journal of surgery, vol. 193, no. 6, pp. 797–804, 2007. B. T. Bethea et al., “Application of haptic feedback to robotic surgery,” Journal of Laparoendoscopic & Advanced Surgical Techniques, vol. 14, no. 3, pp. 191–195, 2004. N. Hogan, “Impedance Control: An Approach to Manipulation: Part I— Theory,” Journal of Dynamic Systems, Measurement, and Control, vol. 107, no. 1, pp. 1–7, 03 1985. ——, “Impedance control: An approach to manipulation: Part ii— implementation,” Journal of dynamic systems, measurement, and control, vol. 107, no. 1, pp. 8–16, 1985. R. J. Adams and B. Hannaford, “Stable haptic interaction with virtual environments,” IEEE Transactions on robotics and Automation, vol. 15, no. 3, pp. 465–474, 1999. J. E. Colgate and J. M. Brown, “Factors affecting the z-width of a haptic display,” in Robotics and Automation, 1994. Proceedings, 1994 IEEE International Conference on. IEEE, 1994, pp. 3205–3210. J. E. Colgate and G. G. Schenkel, “Passivity of a class of sampled-data systems: Application to haptic interfaces,” Journal of robotic systems, vol. 14, no. 1, pp. 37–47, 1997. N. Hogan, “Controlling impedance at the man/machine interface,” in Robotics and Automation, 1989. Proceedings., 1989 IEEE International Conference on. IEEE, 1989, pp. 1626–1631. T. Hulin, C. Preusche, and G. Hirzinger, “Stability boundary for haptic rendering: Influence of human operator,” in 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2008, pp. 3483–3488. J. E. Colgate et al., “Implementation of stiff virtual walls in force-reflecting interfaces,” in Virtual Reality Annual International Symposium, 1993., 1993 IEEE. IEEE, 1993, pp. 202–208. N. Miandashti and M. Tavakoli, “Stability of sampled-data, delayed haptic interaction passive or active operator,” IET Control Theory & Applications, vol. 8, no. 17, pp. 1769–1780, 2014. J. M. Brown and J. E. Colgate, Passive implementation of multibody simulations for haptic display. Northwestern University, 1998.

68 [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37]

Chapter 3 M. Sreelakshmi and T. Subash, “Haptic technology: A comprehensive review on its applications and future prospects,” Materials Today: Proceedings, vol. 4, no. 2, pp. 4182–4187, 2017. V. Hayward and O. R. Astley, “Performance measures for haptic interfaces,” in Robotics research. Springer, 1996, pp. 195–206. J. An and D.-S. .ZRQ “Stability and performance of haptic interfaces with active/passive actuators—theory and experiments,” The International Journal of Robotics Research, vol. 25, no. 11, pp. 1121–1136, 2006. N. . Sinha, C. D. Dicenzo, and B. Szabados, “Modeling of dc motors for control applications,” IEEE Transactions on Industrial Electronics and Control Instrumentation, no. 2, pp. 84–88, 1974. T. Verstraten et al., “Modeling and design of geared dc motors for energy efficiency: Comparison between theory and experiments,” Mechatronics, vol. 30, pp. 198–213, 2015. F. Roos, H. Johansson, and J. Wikander, “Optimal selection of motor and gearhead in mechatronic applications,” Mechatronics, vol. 16, no. 1, pp. 63– 72, 2006. R. Pastorino et al., “Geared pm coreless motor modelling for driver’s force feedback in steer-by-wire systems,” Mechatronics, vol. 21, no. 6, pp. 1043– 1054, 2011. H. Noma et al., “The proactive desk: A new haptic display system for a digital desk using a 2-dof linear induction motor,” Presence: Teleoperators & Virtual Environments, vol. 13, no. 2, pp. 146–163, 2004. A. Ortega et al., “Linear induction actuators for a haptic interface: a quasiperfect transparent mechanism,” in 2017 IEEE World Haptics Conference (WHC). IEEE, 2017, pp. 575–580. D. Senkal and H. Gurocak, “Haptic joystick with hybrid actuator using air muscles and spherical MR-brake,” Mechatronics, vol. 21, no. 6, pp. 951–960, 2011. H. Li et al., “Achieving haptic perception in forceps’ manipulator using pneumatic artificial muscle,” IEEE/ASME Transactions on Mechatronics, vol. 18, no. 1, pp. 74–85, 2011. O. R. Astley and V. Hayward, “Design constraints for haptic surgery simulation,” in Robotics and Automation, 2000. Proceedings. ICRA’00. IEEE International Conference on, vol. 3. IEEE, 2000, pp. 2446–2451. R. B. Gillespie and M. R. Cutkosky, “Stable user-specific haptic rendering of the virtual wall,” in Proceedings of the ASME International Mechanical Engineering Congress and Exhibition, vol. 58, 1996, pp. 397–406. V. Hayward and . E. MacLean, “Do it yourself haptics: part I,” IEEE Robotics & Automation Magazine, vol. 14, no. 4, 2007. J. E. Colgate, M. C. Stanley, and J. M. Brown, “Issues in the haptic display of tool use,” in Intelligent Robots and Systems 95.’Human Robot Interaction and Cooperative Robots’, Proceedings. 1995 IEEE/RSJ International Conference on, vol. 3. IEEE, 1995, pp. 140–145.

Towards the Ideal Haptic Device [38] [39] [40] [41] [42] [43]

[44] [45]

[46] [47] [48]

[49]

[50] [51]

69

B. E. Miller, J. E. Colgate, and R. A. Freeman, “Guaranteed stability of haptic systems with nonlinear virtual environments,” IEEE Transactions on Robotics and Automation, vol. 16, no. 6, pp. 712–719, 2000. D. Weir and J. Colgate, “Stability of haptic displays,” Haptic Rendering: Foundations, Algorithms and Applications, pp. 123–156, 2008. B. Hannaford and J.-H. Ryu, “Time-domain passivity control of haptic interfaces,” IEEE Transactions on Robotics and Automation, vol. 18, no. 1, pp. 1–10, 2002. J.-H. Ryu, Y. S. .LP and B. Hannaford, “Sampled-and continuous-time passivity and stability of virtual environments,” IEEE Transactions on Robotics, vol. 20, no. 4, pp. 772–776, 2004. J.-H. Ryu et al., “Time domain passivity control with reference energy following,” IEEE Transactions on Control Systems Technology, vol. 13, no. 5, pp. 737–742, 2005. Y. Ye, Y.-J. Pan, and Y. Gupta, “A power based time domain passivity control for haptic interfaces,” in Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference. IEEE, 2009, pp. 7521–7526. J.-P. .LP S.-Y. Baek, and J. Ryu, “A force bounding approach for multidegree-of-freedom haptic interaction,” IEEE/ASME Transactions on Mechatronics, vol. 20, no. 3, pp. 1193–1203, 2014. C. Preusche et al., “Time domain passivity control for 6 degrees of freedom haptic displays,” in Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453), vol. 3. IEEE, 2003, pp. 2944–2949. J.-H. Ryu, D.-S. .ZRQ and B. Hannaford, “Stable teleoperation with timedomain passivity control,” IEEE Transactions on robotics and automation, vol. 20, no. 2, pp. 365–373, 2004. T. D. C. Thanh and . . Ahn, “Intelligent phase plane switching control of pneumatic artificial muscle manipulators with magneto-rheological brake,” Mechatronics, vol. 16, no. 2, pp. 85–95, 2006. W. Bullough et al., “The electro-rheological clutch: design, performance characteristics and operation,” Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, vol. 207, no. 2, pp. 87–95, 1993. O. Erol et al., “Magnetic induction control with embedded sensor for elimination of hysteresis in magnetorheological brakes,” Journal of Intelligent Material Systems and Structures, vol. 23, no. 4, pp. 427–440, 2012. C. Rossa et al., “Development of a multilayered wide-ranged torque magnetorheological brake,” Smart Materials and Structures, vol. 23, no. 2, p. 025028, 2014. O. Erol and H. Gurocak, “Interactive design optimization of magnetorheological-brake actuators using the taguchi method,” Smart Materials and Structures, vol. 20, no. 10, p. 105027, 2011.

70 [52] [53]

[54] [55] [56] [57] [58] [59] [60] [61]

[62] [63]

[64]

[65]

Chapter 3 Y.-M. Han and S.-B. Choi, “Control of an ER haptic master in a virtual slave environment for minimally invasive surgery applications,” Smart Materials and Structures, vol. 17, no. 6, p. 065012, 2008. D. Shin, X. Yeh, and O. .KDWLE “A new hybrid actuation scheme with artificial pneumatic muscles and a magnetic particle brake for safe human– robot collaboration,” The International Journal of Robotics Research, vol. 33, no. 4, pp. 507–518, 2014. F. Conti and O. .KDWLE “A new actuation approach for haptic interface design,” The International Journal of Robotics Research, vol. 28, no. 6, pp. 834–848, 2009. E. Simeu and D. Georges, “Modeling and control of an eddy current brake,” Control Engineering Practice, vol. 4, no. 1, pp. 19–26, 1996. A. H. Gosline and V. Hayward, “Eddy current brakes for haptic interfaces: Design, identification, and control,” IEEE/ASME Transactions On Mechatronics, vol. 13, no. 6, pp. 669–677, 2008. M. R. Reed and W. J. Book, “Modeling and control of an improved dissipative passive haptic display,” in Proceedings of IEEE International Conference on Robotics and Automation, vol. 1. IEEE, 2004, pp. 311–318. O. Topçu et al., “Experimental investigation of a magneto-rheological fluid damper with permanent magnet for haptic finger grasping,” in Machine Vision and Mechatronics in Practice. Springer, 2015, pp. 105–112. D. .DUQRSS “Computer simulation of stick-slip friction in mechanical dynamic systems,” Journal of dynamic systems, measurement, and control, vol. 107, no. 1, pp. 100–103, 1985. M. àDÞFNL B. DeBoon, and C. Rossa, “Impact of kinematic structure on the force displayability of planar passive haptic devices,” Transaction on Haptics, 2020. D. . Swanson and W. J. Book, “Obstacle avoidance methods for a passive haptic display,” in Advanced Intelligent Mechatronics, 2001. Proceedings. 2001 IEEE/ASME International Conference on, vol. 2. IEEE, 2001, pp. 1187–1192. J. Furusho et al., “Development of ER brake and its application to passive force display,” Journal of Intelligent Material Systems and Structures, vol. 13, no. 7-8, pp. 425–429, 2002. C. Melchiorri, M. Montanari, and G. Vassura, “Control strategies for a defective, wire-based, haptic interface,” in Intelligent Robots and Systems, 1997. IROS’97., Proceedings of the 1997 IEEE/RSJ International Conference on, vol. 1. IEEE, 1997, pp. 181–187. C. Cho, M. .LP and J.-B. Song, “Direct control of a passive haptic device based on passive force manipulability ellipsoid analysis,” International Journal of Control, Automation, and Systems, vol. 2, no. 2, pp. 238–246, 2004. M. àDÞFNL and C. Rossa, “Design and control of a 3 degree-of-freedom parallel passive haptic device for surgical applications,” Transactions on Haptics, 2020.

Towards the Ideal Haptic Device [66] [67] [68] [69] [70]

[71] [72]

[73] [74] [75] [76] [77] [78] [79]

71

C. Cho, J.-B. Song, and M. .LP “Energy-based control of a haptic device using brakes,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 37, no. 2, pp. 341–349, 2007. C.-M. Chew, G.-S. Hong, and W. Zhou, “Series damper actuator: a novel force/torque control actuator,” in 4th IEEE/RAS International Conference on Humanoid Robots, 2004., vol. 2. IEEE, 2004, pp. 533–546. A. Mohand-Ousaid et al., “Haptic interface transparency achieved through viscous coupling,” The International Journal of Robotics Research, vol. 31, no. 3, pp. 319–329, 2012. B. DeBoon et al., “Differentially-clutched series elastic actuator for robotaided musculoskeletal rehabilitation,” in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 1507–1513. M. Badescu, C. Wampler, and C. Mavroidis, “Rotary haptic knob for vehicular instrument controls,” in Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2002. HAPTICS 2002. Proceedings. 10th Symposium on. ieee, 2002, pp. 342–343. J. Chen and W.-H. Liao, “Design and testing of assistive knee brace with magnetorheological actuator,” in Robotics and Biomimetics, 2008. ROBIO 2008. IEEE International Conference on. IEEE, 2009, pp. 512–517. J. An and D.-S. .ZRQ “Haptic experimentation on a hybrid active/passive force feedback device,” in Robotics and Automation, 2002. Proceedings. ICRA’02. IEEE International Conference on, vol. 4. IEEE, 2002, pp. 4217– 4222. O. Baser, E. ilhan .RQXNVHYHQ and H. Gurocak, “Stability and transparency improvement in haptic device employing both MR-brake and active actuator,” in RO-MAN, 2012 IEEE. IEEE, 2012, pp. 12–18. Y.-J. Nam and M.-. Park, “A hybrid haptic device for wide-ranged force reflection and improved transparency,” in Control, Automation and Systems, 2007. ICCAS’07. International Conference on. IEEE, 2007, pp. 1015–1020. T.-B. .ZRQ and J.-B. Song, “Force display using a hybrid haptic device composed of motors and brakes,” Mechatronics, vol. 16, no. 5, pp. 249–257, 2006. J. Troccaz, M. Peshkin, and B. Davies, “Guiding systems for computerassisted surgery: introducing synergistic devices and discussing the different approaches,” Medical Image Analysis, vol. 2, no. 2, pp. 101–119, 1998. C. Rossa, J. Lozada, and A. Micaelli, “Design and control of a dual unidirectional brake hybrid actuation system for haptic devices,” IEEE Transactions on Haptics, vol. 7, no. 4, pp. 442–453, 2014. B. Gonenc and H. Gurocak, “Virtual needle insertion with haptic feedback using a hybrid actuator with DC servomotor and MR-brake with hall-effect sensor,” Mechatronics, vol. 22, no. 8, pp. 1161–1176, 2012. B. Gonenc and H. Gurocak, “Virtual tissue cutting with haptic feedback using a hybrid actuator with DC servomotor and magnetorheological brake,” Journal of Computing and Information Science in Engineering, vol. 16, no. 3, p. 030902, 2016.

72 [80] [81] [82]

[83] [84]

[85] [86] [87]

[88] [89] [90]

[91] [92] [93]

Chapter 3 O. Baser, H. Gurocak, and E. I. .RQXNVHYHQ “Hybrid control algorithm to improve both stable impedance range and transparency in haptic devices,” Mechatronics, vol. 23, no. 1, pp. 121–134, 2013. Y.-J. Nam and M.-. Park, “A hybrid haptic device for wide-ranged force reflection and improved transparency,” in 2007 International Conference on Control, Automation and Systems. IEEE, 2007, pp. 1015–1020. Y.-J. Nam, Y.-J. Moon, and M.-. Park, “Performance improvement of a rotary MR fluid actuator based on electromagnetic design,” Journal of Intelligent Material Systems and Structures, vol. 19, no. 6, pp. 695–705, 2008. D. Bi et al., “Friction modeling and compensation for haptic display based on support vector machine,” IEEE Transactions on industrial electronics, vol. 51, no. 2, pp. 491–500, 2004. Y.-Q. Yu and J. Lin, “Active balancing of a flexible linkage with redundant drives,” TRANSACTIONS-AMERICAN SOCIETY OF MECHANICAL ENGINEERS JOURNAL OF MECHANICAL DESIGN, vol. 125, no. 1, pp. 119– 123, 2003. J. J. Gil, A. Rubio, and J. Savall, “Decreasing the apparent inertia of an impedance haptic device by using force feedforward,” IEEE Transactions on Control Systems Technology, vol. 17, no. 4, pp. 833–838, 2009. A. Sharon and D. Hardt, “Enhancement of robot accuracy using endpoint feedback and a macro-micro manipulator system,” in 1984 American Control Conference. IEEE, 1984, pp. 1836–1845. A. Sharon, N. Hogan, and D. E. Hardt, “High bandwidth force regulation and inertia reduction using a macro/micro manipulator system,” in Proceedings. 1988 IEEE International Conference on Robotics and Automation. IEEE, 1988, pp. 126–132. L. Stocco and S. E. Salcudean, “A coarse-fine approach to force-reflecting hand controller design,” in Proceedings of IEEE International Conference on Robotics and Automation, vol. 1. IEEE, 1996, pp. 404–410. S. A. Wall and W. Harwin, “A high bandwidth interface for haptic human computer interaction,” Mechatronics, vol. 11, no. 4, pp. 371–387, 2001. D. S. Walker, D. J. Thoma, and G. Niemeyer, “Variable impedance magnetorheological clutch actuator and telerobotic implementation,” in Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on. IEEE, 2009, pp. 2885–2891. P. Fauteux et al., “Dual-differential rheological actuator for highperformance physical robotic interaction,” IEEE Transactions on Robotics, vol. 26, no. 4, pp. 607–618, 2010. J.-S. Oh and S.-B. Choi, “State of the art of medical devices featuring smart electro-rheological and magnetorheological fluids,” Journal of King Saud University-Science, vol. 29, no. 4, pp. 390–400, 2017. A. Bolopion and S. Régnier, “A review of haptic feedback teleoperation systems for micromanipulation and microassembly,” IEEE Transactions on automation science and engineering, vol. 10, no. 3, pp. 496–502, 2013.

Towards the Ideal Haptic Device [94] [95] [96] [97]

[98] [99] [100] [101]

[102] [103] [104] [105] [106]

[107] [108]

73

D. Escobar-Castillejos et al., “A review of simulators with haptic devices for medical training,” Journal of medical systems, vol. 40, no. 4, p. 104, 2016. P. Xia, “New advances for haptic rendering: state of the art,” The Visual Computer, vol. 34, no. 2, pp. 271–287, 2018. E. Samur, Performance metrics for haptic interfaces. Springer Science & Business Media, 2012. C. Cho, J.-B. Song, and M. .LP “Design and control of a planar haptic device with passive actuators based on passive force manipulability ellipsoid (FME) analysis,” Journal of Robotic Systems, vol. 22, no. 9, pp. 475–486, 2005. H. Davis and W. Book, “Torque control of a redundantly actuated passive manipulator,” in Proceedings of the 1997 American Control Conference (Cat. No. 97CH36041), vol. 2. IEEE, 1997, pp. 959–963. S. Munir, L. Tognetti, and W. Book, “Experimental evaluation of a new braking system for use in passive haptic displays,” in Proceedings of the 1999 American Control Conference, vol. 6. IEEE, 1999, pp. 4456–4460. D. Senkal, H. Gurocak, and E. I. .RQXNVHYHQ “Passive haptic interface with mr-brakes for dental implant surgery,” Presence: Teleoperators and Virtual Environments, vol. 20, no. 3, pp. 207–222, 2011. S.-R. Lee et al., “Repulsive force control of minimally invasive surgery robot associated with three degrees of freedom electrorheological fluid-based haptic master,” Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, vol. 228, no. 9, pp. 1606– 1621, 2014. J.-S. Oh et al., “A 4-DOF haptic master using er fluid for minimally invasive surgery system application,” Smart Materials and Structures, vol. 22, no. 4, p. 045004, 2013. S.-R. .DQJ et al., “Controllable magnetorheological fluid based actuators for 6-degree-of-freedom haptic master applicable to robot-assisted surgery,” Sensors and Actuators A: Physical, vol. 279, pp. 649–662, 2018. M. àDÞFNL and C. Rossa, “On the feasibility of multi-degree-of-freedom haptic devices using passive actuators,” in IEEE International Conference on Intelligent Robots and Systems, Nov 2019, pp. 7282–7287. W. J. Book et al., “The concept and implementation of a passive trajectory enhancing robot,” in Intelligent Machine Dynamics Laboratory Publications. Georgia Institute of Technology, 1996. C. Cho, M. .LP and J.-B. Song, “Performance analysis of a 2-link haptic device with electric brakes,” in Proceedings of 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE, 2003, pp. 47–53. J. Wang and C. M. Gosselin, “Static balancing of spatial four-degree-offreedom parallel mechanisms,” Mechanism and machine theory, vol. 35, no. 4, pp. 563–592, 2000. A. Mashayekhi et al., “Virsense: A novel haptic device with fixed-base motors and a gravity compensation system,” Industrial Robot: An International Journal, 2014.

74

Chapter 3

[109] T.-B. .ZRQ and J.-B. Song, “Force display using a hybrid haptic device composed of motors and brakes,” Mechatronics, vol. 16, no. 5, pp. 249–257, 2006. [110] J. An and D.-S. .ZRQ “Five-bar linkage haptic device with DC motors and MR brakes,” Journal of Intelligent Material Systems and Structures, vol. 20, no. 1, pp. 97–107, 2009. [111] M. Zinn et al., “A new actuation approach for human friendly robot design,” The international journal of robotics research, vol. 23, no. 4-5, pp. 379–398, 2004. [112] M. Zinn, “Large workspace haptic devices-a new actuation approach,” in 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.IEEE, 2008, pp. 185–192. [113] O. Schneider and J. Troccaz, “A six-degree-of-freedom passive arm with dynamic constraints (padyc) for cardiac surgery application: Preliminary experiments,” Computer aided surgery, vol. 6, no. 6, pp. 340–351, 2001. [114] S. Hirose and S. Ma, “Coupled tendon-driven multijoint manipulator,” in Proceedings. 1991 IEEE International Conference on Robotics and Automation. IEEE, 1991, pp. 1268–1275. [115] M. .DQHNR T. Yamashita, and . Tanie, “Basic considerations on transmission characteristics for tendon drive robots,” in Fifth International Conference on Advanced Robotics’ Robots in Unstructured Environments. IEEE, 1991, pp. 827–832. [116] J.-P. Merlet, Parallel Robots, 2nd ed.Springer Science & Business Media, 2006, vol. 128.

CHAPTER 4 NEW RESEARCH AVENUES IN HUMAN-ROBOT INTERACTION FRAUKE ZELLER Associate Professor Dr. Frauke Zeller Ryerson University Faculty of Communication & Design [email protected]

Abstract “Technology is a very human activity – and so is the history of technology” (Kranzberg, 1986, p. 545). Kranzberg’s laws of technology have always served as a fitting point of departure to discuss human interactions and relationships with technology. As such, they are especially relevant to the research field of Human-Robot Interaction (HRI). HRI mostly conceives and builds technology (i.e. robots) to enhance, complement, support human activity. Robots represent the only kind of technology that derives its form from human (i.e. anthropomorphic) features. This mainly holds true for social robots, the main focus of attention in this chapter. This chapter introduces an extended HRI research model adapted from communication and mass communication studies to explain and conceptually define the intertwined nature of robots and humans. The model provides a useful guide to the analyses and understanding of the social dimension of social robots. Whereas the discussion of this model will be on a conceptual level, the chapter connects the different areas of the model to existing research studies and experiments in order to provide concrete examples. These examples are studies from the social sciences, humanities, and behavioural and cognitive sciences, focusing on the different forms and kinds of communication that are inherent in HRI.

76

Chapter 4

Introduction Robots have fascinated humankind for thousands of years, appearing in ancient Egyptian and Greek mythology (see Ichbiah, 2005; Reichardt, 1978). Central to this fascination is the human desire to ‘create’ something living or life-like out of inert matter. One of the most famous examples is the myth of the golem, an anthropomorphic living being that is created by humans (usually) from clay or mud. One version describes how a golem was created in Prague in the 16th century to protect the Jewish population (Cohen, 1966). Another, a female golem, was created by the Jewish poet and philosopher Solomon ibn Gabirol. The ailing Gabirol “was said to have created the woman golem to keep house for him […]” (Bashevis Singer, 1984, para. 4). While the idea of creating an animated protector out of nonliving materials has its appeal, it has also come with misgivings. In fact, most robot narratives have at their core the notion of a human ‘master’ and a robot ‘servant’, and the need to maintain this relationship. Hence, Asimov’s famous laws of robotics (1950) were designed to prevent a robot from turning against its master. In recent years, however, popular perceptions of robots appear to have changed and are very much in flux. Movie robot characters such as the Terminator, for example, might have started to introduce this shift. The Terminator (2004), a sophisticated artificial intelligence machine with (more or less) anthropomorphic features, can transform into human doppelgangers resembling human characters, albeit with much stronger (and usually evil) powers. More recently, there is a trend in popular media and science communication to merge different technological advancements – from robots to AI and autonomous systems like self-driving cars – into one idea, that is robots. In fact, even chatbot dialogue systems, such as those hosted on smart technologies like Apple’s Siri, or Amazon’s Alexa, are often referred to as robots in public discourse (Madrigal, 2011; Nowak, 2011; Petri, 2011; Weaver, 2013). Moreover, news stories about AI or autonomous systems are often accompanied by an image of a robot 1. 1

See, for example, a news media item from the Associated Press on the White House proposing guidelines for the regulation of AI usage: https://apnews.com/cf2ef1681c65139a55623f5f5df3709f The item shows the image of a metal head made out of motor parts, thus resembling an anthropomorphic robot. Another example is an article by The Conversation Canada, a popular science communication media platform. The article talks about AI and Quantum theory, yet showing a robot holding a human skull https://theconversation.com/will-ai-take-over-quantum-theory-suggests-otherwise126567

New Research Avenues in Human-Robot Interaction

77

Obviously, not all robots are based on AI technology, nor can we define every robot as an autonomous system comparable to self-driving cars 2. However, the trend to use robots as a cover term for relevant futuristic inventions represents at least two important shifts: Firstly, the field of human-robot interaction (HRI) and related disciplines have become subject to unprecedented public attention. Secondly, the expansion of the concept of a robot to include an amalgamation of robots, AI, autonomous systems, and spoken dialogue systems is likely to have an effect on how end-users interact with robots, and subsequently, have an impact HRI research designs. In short, more people have familiarity and experience of interaction with machines or automata that they would call a ‘robot’ – be it a spoken dialogue system on their smart phones, or a vacuum cleaner at home. The expanded concept of robots and their heightened public exposure calls for a revision of theory and analysis of human-robot interaction, especially our interaction with social robots. This chapter provides a discussion of the different dimensions and factors that form and shape particularly the ‘social’ interface of robots. It will introduce an extended HRI research model, which is adapted from communication and mass communication studies, and discuss three core dimensions of social robotics, which are key in building socially interactive robots.

Background HRI can be defined as a multi-disciplinary “research field at the intersection of psychology, cognitive science, social sciences, artificial intelligence, computer science, robotics, engineering and human-computer interaction” (Dautenhahn, 2007, p. 103). This chapter aims to contribute to this multi-disciplinary outlook by focusing on the social dimension of social robots. The social dimension includes all cultural and social factors that shape the interaction between humans and robots. The social dimension of HRI draws upon the social sciences, humanities, and behavioural and cognitive sciences. In the following sections, a research model will be introduced, together with existing research approaches. The forms and kinds of communication inherent in HRI are introduced, ranging from end-user

2

The mixing of AI with robots makes insofar sense as a robot indeed appears to be the most plausible in-between of humans and AI. Given that it is difficult to connect a unique image with AI, using well-known robots to communicate artificial intelligence appears to be the next-best option.

78

Chapter 4

interactions with robots to cultural discourses relevant to the design and development of social robots. The impact of technological advancements upon the individual and society is among the key interests of social sciences and humanities. In the past ten or more years, these disciplines have contributed discussion and research around topics such as big data, social media, algorithms and artificial intelligence, and, more recently, robots. Fortunati (2018) provides an overview of the numerous recent publications from the social sciences and communication studies on social robotics. Among those are special issues (Böhlen & Karppi, 2017; Fortunati, Esposito, & Lugano, 2015; Pfadenhauer, Sugiyama, & Ess, 2015; Sugiyama & Vincent, 2013), edited books (Vincent, et al., 2015; Guzman, 2018), and monographs, such as Gunkel’s Communication and Artificial Intelligence (2020), which includes a major chapter on social robotics (see Zeller, 2005, for early work on this topic). Some media and communication scholars discuss the robot as a medium or apply classic theories and debates from their field, such as technology affordances and agency (Pentzold & Bischof, 2019). The coherence of these scholarly contributions is complicated by an uneven application of an overly broad concept of robot, i.e. associating the term ‘robot’ with the varieties of technological advancements listed above. What is lacking is the notion of the unique symbolic role that robots have played historically and continue to play in contemporary culture. The key question is ‘What is a social robot – a medium/machine, a cultural sign, or a friend?’. This question also concerns the kind of communication and conversation humans prefer with robots. As Baron points out: “While social robots are the creation of human beings, it is not obvious what kinds of conversation people desire to have with computer-based devices” (Baron, 2013, p. 257). Baron argues that human communication and its linguistic functions do more than just convey information. For example, “informational and empathetic functions are likely to be more welcomed than those involving social control or critique” (Baron, 2013, p. 257). In addition to acknowledging the variety of functions in human communication, there are good reasons to question the applicability of human-human communication for the design of human-robot communication. Dautenhahn says that “it is important to realize that the nature of human-robot interaction is related to but different from human-human or human computer interaction” (Dautenhahn, 2007, p. 103). Whereas this statement holds true until today, another prediction made more than ten years ago might need to be revised: “After all, as long as robots and humans are distinguishable, […], people are likely to not treat them identically to human beings. Once people can no longer distinguish a robot from a person, […], then people treat them like humans” (Dautenhahn,

New Research Avenues in Human-Robot Interaction

79

2007, p. 104). The revision is needed, given recent references to spoken dialogue systems as robots, which appear to question the necessity and role of a robot’s physical form. Calling spoken dialogue systems, which aim to support humans, a robot, simply because they are “helpful” and “intelligent”, appears to bring the communicative function and potential of a robot to the forefront. The reduction of a (physical) robot to a voice, is interesting and relevant for the field of HRI, as it impacts users’ perception of robots and their experiences. Moreover, given the increasing success of spoken dialogue systems and particularly so-called smart speaker systems like Siri, Alexa, and google home 3, we are facing more than simply an ephemeral idea of our zeitgeist. The relevant point is that when HRI researchers conduct experiments, asking about the participants’ prior experiences with robots, they now might get very different answers. For example, participants might evaluate themselves as highly experienced when it comes to interacting with robots, simply on the basis that they are using a smart speaker system. Being then asked to interact with physical, anthropomorphic robots, for example, clearly is a very different situation and might result in confusion on the participant’s side. HRI has always been confronted with the difficulty of covering a field where the research object comes in a multitude of forms and functions, and can be used in a similarly plentiful set of situations. These variables have made reproduction of results and representative studies difficult. Given the present expansion of the concept of robot, this situation appears to be even more extreme. Participants will interact with robots on the basis of their preconceptions and it may not be possible nor desirable to control for each person’s detailed ideas about robots before each experiment

The Encoding/Decoding Model Figure 1 shows an adaption of Stuart Hall’s (1993) encoding/decoding model. Hall, a cultural theorist, designed this model in the 1990s as a way to understand and discuss mass communication (e.g. television broadcasting). Hall rejected the prevailing idea of effects research that mass communication consumers (audiences) can be influenced by top-down messaging (i.e. television programmes). Instead, audience members play an active role in the creation of messages, given that they bring their own social context into the decoding process of mass communication messages. Applying the 3

The penetration rate of smart speaker systems among households with internet access was in 2018 already at 20 % in the United States and 18% in the United Kingdom, and this only a few years after their introduction (Liu, 2019).

80

Chapter 4

encoding/decoding model to HRI, we may think of social robot users and members of the public who have some exposure to robots in any form as an active audience.

Figure 1 Encoding/decoding model of human-robot communication, adapted from Hall (1993).

For this chapter, the main idea of the model – the active audience – can be adapted in order to visualize the different social dimensions that are relevant to the design of social robots, and to underscore the idea of the ‘active end-user’ who brings their own experiences, knowledge, biases, etc. to any human-robot interaction. More concretely, this adaptation aims to: (1) Stress the importance of the users’ experiences, preconceptions, attitudes that can influence the human-robot interaction or communication process; (2) Show the different socio-cultural dimensions that influence the communication process, starting with the design of the robot; (3) Apply Hall’s notion of a circular process to interaction design, where the design of user interfaces, for example, consists of iterative design and development stages enriched by user feedback. The model shows how human-robot communication messages are grounded in a “shared field of social institutions, knowledge, and culture”

New Research Avenues in Human-Robot Interaction

81

(Gasher, Skinner, & Lorimer, 2016, p. 94). Here, we can differentiate between macro, meso, and micro level dimensions, which can provide different points of departure into the study of social robotics. The macro level refers to the overall socio-cultural settings in which robots are built, including economical and political aspects. The meso level stands for the organizational level, as depicted in the model on the left with ‘Technical Infrastructure’, ‘Relations of Production’, and ‘Frameworks of Knowledge’, for example. Hall writes here: “From this general perspective, we may crudely characterize the television communicative process as follows. The institutional structures of broadcasting, with their practices and networks of production, their organized relations and technical infrastructures, are required to produce a programme” (Hall, 1993, p. 123). We can use this definition by looking into how laboratories and factories – the sites and contexts of robot constructions – and their institutional norms, policies, but also cultures, influence the design and development of robots. The micro level depicts what Hall calls the ‘programme’, or in our case the ‘robotic programme’, which entails how the end-user engages in ‘meaningful discourse’ with the robot. The meaning dimension – in the model depicted with meaning structures 1 and 2 as well as ‘meaningful discourse’ – underscores the notion that encoder and decoder do not necessarily need to share the same meaning, i.e. an encoded message can be decoded differently or misunderstood: The codes of encoding and decoding may not be perfectly symmetrical. The degrees of symmetry – that is the degrees of ‘understanding’ and ‘misunderstanding’ in the communicative exchange – depend on the degrees of symmetry/asymmetry (relations of equivalence) established between the positions of the ‘personifications’, encoder-producer and decoder-receiver. (Hall, 1993, p. 123)

Understanding and misunderstanding the social robot as a ‘meaningful discourse’ can be interpreted as the actual communicative messages uttered by the robot. But it also depicts the overall appearance of the robot (design), it’s communicated intentions, personality, etc. What is central here is the idea that the robot, with its design and its back-end and front-end programming, is a product of the “discursive rules” to be found in the “institution-societal relations of production” (Hall, 1993, p. 123). These discursive rules have a role in defining norms, policies, guidelines, as well as ideas and descriptions of robots. All these can differ from the user’s ideas, norms, and experiences. Thus, the ‘robotic programme’ can only turn into a ‘meaningful discourse’ if the user decoding is successful, which in return depends upon finding a common ground between producers and audiences.

82

Chapter 4

Hall writes: “The lack of fit between the codes has a great deal to do with the structural differences or relation and position between broadcasters [robot producers] and audiences [end-users] […] (Hall, 1993, p. 124, square braces added). The field of HRI and related disciplines has been focusing on the lack of fit between codes by studying human-human interactions as well as human-robot interactions. What is often lacking, though, is a closer understanding of the left-hand side of the model, particularly the meso and macro levels: We need to understand the underlying assumptions and preferences inherent to the production process as well as the cultural norms and understandings of social robots held by users. The following sections discuss how the macro, meso and micro levels are relevant in HRI.

Macro level The macro level refers to the social institutions, knowledge, and culture (visualized in the bottom of the model/Figure 1) that are shared by all participants in the human-robot communication process: politicians, decision-makers, business and industry practitioners, engineers, designers, computer programmers, and end-users. Decision makers like politicians or regulators come up with policies, specific laws and regulations relating to the construction and manufacturing of robots. They usually have advisors from research and industry who bring specific business and political objectives in addition to personal notions, objectives, and ideas. All these can influence the formation of regulations, laws, and the allocation of resources. Politicians also have to acknowledge economical aspects, such as business and industry practices. These aspects relate to the robotics industry, for example determining what market regulations can impact the production of robots, such as patents, trade and labor conditions, and safety standards. Moreover, macro-social conditions are relevant, such as the aging population in many Western societies, which contributes to the prioritization of research on how social robots can support an increasingly aging society. At present, social robots are a smaller category of mass produced robots than industrial robots, which mostly account for the replacement of physical human labour 4. However, the market for social robots, which are part of the overall category of service robots, is predicted to grow on average between 20 and 25 percent annually (Anandan, 2018). Even

4

A 2018 report by the International Federation of Robotics (IFR) estimated a total of 3 million robots working in the industrial workforce (IFR, 2018), which represents an annual growth rate of 14 percent. Another source claims that almost 40% of all jobs in America will be held by robots by the 2030s (Matthews, 2018).

New Research Avenues in Human-Robot Interaction

83

the comparatively smaller category of personal/domestic service robots is estimated to quintuple in the next years. These are the robots we will find closest and most personal to humans (Zeller, 2020). Recent advancements in the field of AI and autonomous systems have the potential to put politicians and policymakers under pressure to consider ethical and socio-cultural implications. There are now broad discussions and prolific research on this aspect, and initiatives ranging from single (industry) sectors to pan-national activities, such as the planned Centre on Artificial Intelligence and Robotics founded by the United Nations. Interestingly, this centre is to be organized under the UN Interregional Crime and Justice Research Institute (UNICRI), and its projected activities are: ƒ Performance of a risk assessment and stakeholder mapping and analysis ƒ Implementation of training and mentoring programmes ƒ Contributing to the UN Sustainable Development Goals through facilitation of technology exchange and by orienting policies to promote security and development ƒ Convening of expert meetings ƒ Organization of policy makers’ awareness-raising workshops ƒ Organization of international conferences (UNICRI, nd). How influential this centre will be remains to be seen, but its inclusion within an institute focussing on crime and justice research already produces certain expectations and preconceptions. The public’s notion of the current and future state of robotics is often influenced by literature and movies (Oestreicher & Eklundh, 2006). Even before the introduction of voice assistants, there have been critical voices that anthropomorphic (social) robots in fiction and popular culture mislead the general public to assume that “AI has advanced further than it has in reality” (=áRWRZVNL et al., 2015, p. 352). Today, these exaggerated impressions are further instilled through so-called influencers’ talks about future developments. For example, Elon Musk’s ideas about self-driving cars are being quoted extensively in the media and add to the often-dystopian notion of robots and society in the future. According to Musk, his electric ‘robotaxis’ will be replacing other taxi services like Uber and Lyft (Durkee, 2019), which are still dependent on human drivers. Uber and Lyft business models are based upon the ‘sharing economy’ in which humans are central to the transaction, turning private goods or property (e.g. AirBnB) into common resources. The business model of proposed robotaxis eliminates the human driver, magnifying concerns that robots will be taking over our

84

Chapter 4

jobs and livelihoods. Another of Musk’s widely disseminated ideas is the self-driving car with different modes or personalities: standard, aggressive and ‘Mad Max’ (E&T, 2018), which resulted in viral social media and extensive news media coverage. It is surely debatable whether any self-driving car could also be termed a robot, specifically when looking into the standardized norms and definitions of the different kinds of robots. However, the term appears to be used synonymously for those autonomous systems, even though self-driving cars do differ significantly from those mobile vehicle-like robots from the past, such as the Mars Rover robot. Whereas the Mars Rover was controlled and maneuvered by human beings, Musk’s self-driving cars (and others) are controlled and maneuvered by AI systems, which use large databases and learning algorithms for decision making. The dynamic nature of these macro level cultural trends and factors are reflected in the increasing attention to the topics of AI and robots in our public news media discourse. From a research perspective, those descriptions and discussions about robots in the news media impact our perceptions, ideas, and expectations. In fact, the way that technology and science related issues are reported on in news media is assumed to play a decisive role on the public’s understanding and impression of science and technology. Scholars in this area who focus on mediatization theories (Deacon & Stanyer, 2014; Jensen, 2013; Krotz, 2008), detect three dimensions of change in the media’s coverage of science and technology topics: 1) Extensiveness: Science is said to be increasingly represented in the mass media. 2) Pluralisation: Media coverage on science is said to be increasingly diverse in terms of actors and topics. 3) Controversy: Media coverage on science is seen as increasingly controversial. (Schäfer, 2009, p. 478). Other studies have shown that mass media reporting on technological and digital issues has not only increased but also has become inherent in all societal dimensions, ranging from social and political to economic issues (Zeller, Porten Chee, & Wolling, 2010). Thus, conducting framing studies, for example, on mass communication content (e.g. news media reporting, movies, books) can provide important insights and links to changes of people’s perceptions of robots, potential critique or trust issues, and expectations. Another macro level research approach would be to conduct analyses of social media for how both social and news media impact people’s perceptions

New Research Avenues in Human-Robot Interaction

85

and interactions with robots. A study on hitchBOT, a popular robot that hitchhiked across different countries, shows that even though the robot did not meet many people in a direct, physical interaction, it still garnered remarkable popularity through media (Smith & Zeller, 2017; 2018). People felt attached to the robot, even though their experience of hitchBOT consisted entirely of online exposure via social media. These online interactions were limited to consuming updates on the objectives and progress of the robot and interactions with the community of social media followers of the robot (Zeller, Smith, Duong, & Mager, 2019). In the case of hitchBOT, the hitchhiking robot, we have a new type of reductive definition for robots, in this case a robot-personality that convenes interaction without any physical and direct contact. This example provides an illustration of factors in HRI that influence interaction and communication with robots, which go beyond physical interaction. Audience and reception studies of how people interact via different communication channels, such as social media platforms, can thus provide additional insights about social robots and their communicative potentials as perceived by the end-users.

Meso level As mentioned above, the meso level refers to the organizational level, visualized in the left side of the model as ‘Technical Infrastructure’, ‘Relations of Production’, and ‘Frameworks of Knowledge’. The meso level can also be referred to as the “institutional organizational context of production” (Gasher, Skinner, & Lorimer, 2016, p. 94) and accounts for organizational policies, strategies, institutional ethical guidelines, etc. Technical infrastructure relates to an organization’s technological setup and planned direction. It also relates to how people use such technical infrastructures to build social robots. Frameworks of knowledge refer to professional values, which describe the cultural norms in an organization or profession that influence the building of robots. Thus, the training of engineers, planners, designers, and even communication personnel of companies or organizations involved is important. What do they know about human end-users and their preferences regarding social robots? How does their knowledge, culturally formed in their social context, influence the robots they build? Meso level factors may be researched through science and technology studies (STS) methods (Felt et al., 2017; Fuller, 2006), for example, analyzing laboratory cultures (Latour & Woolgar, 1979; Pickering, 1992; Woolgar, et al., 2014). Developing an understanding of cultures of collaboration and co-creation, or cultures of knowledge, i.e. how knowledge is defined and formed (Knorr-Cetina, 1981; Knorr-Cetina & Mulkay, 1983;

86

Chapter 4

Knorr-Cetina, 1999), can be helpful to gain insight into the environment where robots are conceived, planned, designed, tested, and built. Examples range from organizational guidelines, objectives, definitions, to experimental designs and methods. The choice of research methods, for example, to test a robot’s interaction and impression, will shape the results obtained. Dautenhahn (2007) demands in this case fewer methodological battles and more innovation: “instead of simply following e.g. social sciences textbook knowledge, adaptations and new developments of methods are asked for” (p. 103). Related to this, it is not surprising that a meta study of personality measurement and design in HRI found that only 20 percent of the studies conducted personality measurements for both – humans and robots. In fact, 37 percent focussed solely upon human personality and 43 percent assessed robot personality (Santamaria & Nathan-Roberts, 2017). Adding to this diverse field, “only three of the seven studies that measured both used assessments based on the same personality models for humans and robots” (Santamaria & Nathan-Roberts, 2017, p. 855). On the side of the end-user of a social robot, the meso level should also be accounted for in the case of social robots situated in organizational contexts: For example, social robots as co-workers in office environments, as caregivers in elderly homes or hospitals, as receptionists in hotels, as retail assistants or as educators in learning environments. In each of these cases, regulations at the organizational level regarding safety, libel, and privacy can influence how we approach such robots. One example is the question of surveillance and privacy regarding robot co-workers: an organization’s transparency about whether or not they also use robots as surveillance tools is an important factor regarding the forming of trust and general acceptance.

Micro level The micro level addresses the individual, personal interaction between humans and robots and is visualized in the right side of the model, as well as the top. Arguably, the micro level is the most researched area of HRI, and thus numerous studies exist relating to all kinds of different robots, user impressions, and interaction preferences. Whereas these studies must and will be continued, the encoding/decoding model also suggests an additional research avenue at the micro level. One of the core ideas in HRI and particularly social robotics, has been the concept of anthropomorphism. One of the assumptions behind this concept is that in order to build a robot that humans can accept as peers or friends, they need to resemble us. However, this does not mean that human-human interactions are the same or follow

New Research Avenues in Human-Robot Interaction

87

the same prerequisites, cultural norms, linguistic patterns, as human-robot interactions: “It is a misconception that results from human-human interactions can be directly applied to human-robot interaction” (Dautenhahn, 2007, p. 104). And yet, we still find numerous studies where this connection is made. For example, in the field of cognitive robotics, researchers discuss examples, of “how robots can be used to test hypotheses pertaining to human social neuroscience” and how social cognitive neuroscience can provide “a better understanding of social interactions of human-human and human-machines” (Chaminade & Cheng, 2009, p. 286). Whereas it is not intended to say that this approach is faulty, rather, this chapter argues to also focus on human-robot communication and interaction as unique categories, instead of mere reflections of human-human and human-computer interaction. Human-robot interaction should be defined as a unique form of social interaction, which would provide an opportunity for theoretical extensions, methodological innovation, and pragmatic modeling. One possible form of theorizing in this direction is to define robots from a semiotic and linguistic perspective. As such, robots can be defined as object and symbol processing systems based on formal languages (i.e. computer programming languages) and ‘numerical’ objectives (Zeller, 2005). However, they are also object and symbol processing systems that are meant to interact with natural language-based, non-numerical object and symbol processing systems, that is humans. A linguistic framework facilitates the direct connection of the object and symbol processing levels, albeit with different foundations (numerical and non-numerical). Furthermore, the notion of both robots and humans as object and symbol processing systems allows researchers to: 1. Combine their numeric research questions with non-numeric questions. 2. Juxtaposition a robot in its formal-language-habitat, ranging from its algorithms and circuits with the natural language-based discourses of laboratories and industrial production sites. 3. Translate robots into discursive contexts from communication studies, linguistics, and neurocognitive studies. 4. Integrate recent changes in our popular discourses about robots that represent bodily reductions and focus on the communicative/ linguistic dimension. Potential research questions regarding this new research avenue, could be: What linguistic patterns and preferences exist in human-robot communication? What are notions of trust or affect in human-machine

88

Chapter 4

communication? How can human-machine communication be measured, analysed? Those questions can be answered by focusing on the linguistic (verbal and non-verbal) interactions, using methods such as sentiment analyses, discourse and conversation analyses, statistical language studies, language acquisition studies, etc. Basically, the linguistic framework of human-robot interaction could provide multiple quantitative and qualitative research results to inform our understanding of human-robot interaction as a cultural, situated phenomenon. Linguistic studies are decisive in this regard, given that the language we use defines who we are and with whom we communicate. Additional input can be provided by the “cognitive/behavioural structures” in the encoding/decoding model (Figure 1), framing HRI studies with cognitive and behavioural research questions to inform our understanding of micro level interactions. There has been an increased interest in merging neural, cognitive and behavioural studies with robotics in recent years (Chaminade & Cheng, 2009; Chiao, 2009; Kaplan, F., 2008; Kitayama & Park, 2010; Mergner & Tahboub, 2009; Urgen et al., 2013), and the results are promising insofar as they can add important insights into the human user. The cognitive/behavioral disciplines have the additional advantage of sharing the roboticist’s interest in neural networks that, like the object and symbol processing approach, constitute another direct link between humans and robots,

Conclusion The encoding/decoding model of human-robot interaction and communication provides a first step in an extended discussion around recent changes in HRI and public discourse. The chapter does not intend to suggest that extant research in HRI is poorly conceived. Instead, it identifies and describes levels of activity that have direct bearing on HRI and that can be included in the scope of HRI research. Most aspects discussed here are not new ideas, rather the overall discussion provides a new HRI framing approach via encoding/decoding analyses of the meta, mesa, and micro levels. This approach would achieve a more comprehensive understanding of the complex factors influencing how human beings communicate and interact with robots. Much research has been conducted within the different disciplines on humans and their interactions and relations with robots. However, all partners need to find a common starting point for interdisciplinary research to be successful. The adapted encoding/decoding model could represent

New Research Avenues in Human-Robot Interaction

89

such a starting point, inviting discussions about each disciplines’ definitions or understandings of the key terms – meaning, programme, discourse, infrastructure, knowledge – in order to negotiate a common conceptual ground. Finally, the extension of the research objective appears to be a necessary means to approach the aforementioned contemporary expanded category of robots, including instances subtracting the robot’s physical form and presence. This means applying audience and reception studies to our public discussions of robots and AI. With social media, we have ample communication channels providing content, together with traditional news media, which we can use to look into how people talk about robots, their relationships, and their needs and preferences. hitchBOT, the hitchhiking robot illustrated how people’s creativity in their human-robot interactions could be sparked by taking the robot out of the laboratory context and stripping it of all instructions, guidelines, and prescribed meanings (or meaningful discourses): The robot simply asked for a ride, aiming to cross the country, and people responded with creative ideas, passion, and emotional attachment.

References Anandan, T.M. (2018). Service robots on the world stage. Retrieved from https://www.robotics.org/content-detail.cfm?content_id=7061 Asimov, I. (1950). I, Robot. Gnome Press. Baron, N. S. (2015). Shall we talk? Conversing with humans and robots. The Information Society, 31(3), 257–264. http://doi.org/10.1080/01972243.2015.1020211 Bashevis Singer, I. (1984, August 12). The golem is a myth for our time. The New York Times, p. 25. Böhlen, M., & Karppi. T. (2017). Social robots: human-machine configurations, special issue of transformations. Journal of Media, Culture & Technology 29: 1–22. Cameron, J., Hurd, G. A., Schwarzenegger, A., Biehn, M., Hamilton, L., Winfield, P., Henriksen, L., ... MGM Home Entertainment Inc. (2004). The Terminator. Santa Monica, CA: MGM DVD. Chaminade, T., & Cheng, G. (2009). Social cognitive neuroscience and humanoid robotics. Journal of Physiology - Paris, 103(3-5), 286–295. http://doi.org/10.1016/j.jphysparis.2009.08.011 Chiao, J. Y. (2009). Cultural neuroscience: A once and future discipline. Progress in Brain Research (Vol. 178, pp. 287–304). Elsevier. http://doi.org/10.1016/S0079-6123(09)17821-4 Cohen, J. (1966). Human robots in myth and science. London: George Allen & Unwin LTD.

90

Chapter 4

Dautenhahn, K. (2007). Methodology & themes of Human-Robot Interaction: A growing research field. International Journal of Advanced Robotic Systems, 4(1), 15. http://doi.org/10.5772/5702 Deacon, D., & Stanyer, J. (2014). Mediatization: Key concept or conceptual bandwagon? Media, Culture & Society, 36(7), 1032–1044. http://doi.org/10.1177/0163443714542218 Durkee, A. (2019, April 23). Elon Musk wants his own “robot taxis” to replace Uber and Lyft. Vanity Fair-Hive. Retrieved from https://www.vanityfair.com/news/2019/04/elon-musk-robot-taxis-replace-uberand-lyft E&T (2018, June 25). Tesla to offer ‘aggressive’ and ‘Mad Max’ Autopilot modes, Elon Musk says. Engineering and Technology. Retrieved from https://eandt.theiet.org/content/articles/2018/06/tesla-to-offer-aggressive-andmad-max-autopilot-modes-elon-musk-says/ Felt, U., Fouché, R., Miller, C. A., & Smith-Doerr, L. (eds.) (2017). The handbook of science and technology studies (4th ed). Cambridge and London: The MIT Press. Fortunati, L. (2018). Robotization and the domestic sphere. New Media & Society, 20(8), 2673–2690. http://doi.org/10.1177/1461444817729366 Fortunati, L., Esposito, A., & Lugano, G. (2015). Introduction to the Special Issue “Beyond Industrial Robotics: Social Robots Entering Public and Domestic Spheres.” The Information Society, 31(3), 229–236. http://doi.org/10.1080/01972243.2015.1020195 Fuller, S. (2006). The philosophy of science and technology studies. New York: Taylor and Francis. Gasher, M., Skinner, D., & Lorimer, R. (2016). Mass communication in Canada (8. Ed). Oxford University Press. Gunkel, D. J. (2020). An introduction to communication and artificial intelligence. Cambridge: Polity. Guzman, A. L. (ed) (2018). Human-Machine Communication. Peter Lang. Hall, S. (1993). Encoding, decoding. In S. During (ed), The Cultural Studies Reader, pp. 122-128. London, New York: Routledge. Ichbiah, D. (2005). Roboter. Geschichte, Technik, Entwicklung. München: Knesebeck. IFR. (2018). Executive Summary World Robotics 2018 Industrial Robots (pp. 1–10). Jensen, K. B. (2013). Definitive and sensitizing conceptualizations of mediatization. Communication Theory, 23(3), 203–222. http://doi.org/10.1111/comt.12014 Kaplan, F. (2008). Neurorobotics: an experimental science of embodiment. Frontiers in Neuroscience, 2(1), 22–23. http://doi.org/10.3389/neuro.01.023.2008 Kitayama, S., & Park, J. (2010). Cultural neuroscience of the self: understanding the social grounding of the brain. Social Cognitive and Affective Neuroscience, 5(23), 111–129. http://doi.org/10.1093/scan/nsq052 Knorr-Cetina, K. (1981). The manufacture of knowledge: An essay on the constructivist and contextual nature of science. Oxford New York: Pergamon Press.

New Research Avenues in Human-Robot Interaction

91

Knorr-Cetina, K., & Mulkay, M. (1983). Science observed: Perspectives on the social study of science. London Beverly Hills: Sage Publications. Knorr-Cetina, K, (1999). Epistemic cultures: How the sciences make knowledge. Cambridge, Massachusetts: Harvard University Press. Kranzberg, M. (1986). Technology and History: "Kranzberg's Laws". Technology and Culture, 27(3), 544-560. Krotz, F. (2008). Media connectivity: Concepts, conditions, and consequences. In A. Hepp, F. Krotz, & S. Moores (eds), Network, Connectivity and flow: Key concepts for media and cultural studies. New York: Hampton Press. Latour, B., & Woolgar, S. (1979). Laboratory life: The construction of scientific facts. Princeton University Press. Liu, S. (2019). Penetration rate of smart speakers in internet households 2018, by country. Retrieved from https://www.statista.com/statistics/974927/smartspeaker-penetration-rate-by-country/ Madrigal, A. (2011, October 12). Siri: The perfect robot for our time. The Atlantic. Retrieved from https://www.theatlantic.com/technology/archive/2011/10/sirithe-perfect-robot-for-our-time/246516/ Matthews, K. (2018). 10 key robotics statistics you need to know. Retrieved from https://blog.robotiq.com/10-key-statistics-about-robotics-you-need-to-know Mergner, T., & Tahboub, K. (2009). Neurorobotics approaches to human and humanoid sensorimotor control. Journal of Physiology - Paris, 103(3-5), 115– 118. http://doi.org/10.1016/j.jphysparis.2009.07.006 Nowak, P. (2011, October 17). Is Siri artificially intelligent or just a robot? MacLean’s. Retrieved from https://www.macleans.ca/society/technology/issiri-artificially-intelligent-or-just-a-robot/ Oestreicher, L., Eklundh, K. S. (2006). User expectations on human-robot cooperation. In: Robot and human interactive communication, 2006. The 15th IEEE international symposium on ROMAN 2006. IEEE, pp 91–96. Pentzold, C., & Bischof, A. (2019). Making affordances real: Socio-material prefiguration, performed agency, and coordinated activities in Human–Robot Communication. Social Media + Society, 5(3). http://doi.org/10.1177/2056305119865472 Petri, A. (2011, December 1). Apple’s Siri, the uncomfortably pro-life robot. The Washington Post. Retrieved from https://www.washingtonpost.com/blogs/compost/post/apples-siri-theuncomfortably-pro-life-robot/2011/12/01/gIQAOeWfHO_blog.html Pfadenhauer, M., Sugiyama, S., & Ess, C. M. (ed) (2015). Special Issue of IJSR on Social Robots: Form, Content, Critique. International Journal of Social Robotics, 7(3), 1–2. http://doi.org/10.1007/s12369-015-0291-1 Pickering, A. (ed) (1992). Science as practice and culture. University of Chicago Press. Reichardt, J. (1978). Robots: Fact, fiction and prediction. London: Thames & Hudson. Santamaria, T., & Nathan-Roberts, D. (2017). Personality Measurement and Design in Human-Robot Interaction: A Systematic and Critical Review. Proceedings of

92

Chapter 4

the Human Factors and Ergonomics Society Annual Meeting, 61(1), 853–857. http://doi.org/10.1177/1541931213601686 Schäfer, M. S. (2009). From public understanding to public engagement: An empirical assessment of changes in science coverage. Science Communication, 30(4), 475–505. doi:10.1177/1075547008326943 Smith, D. H., & Zeller, F. (2017). The death and lives of hitchBOT: The design and implementation of a hitchhiking robot. Leonardo, 50(1), 77-78. Smith, D. H., & Zeller, F. (2018). hitchBOT: The risks and rewards of a hitchhiking robot. Suomen Antropologi: Journal of the Finnish Anthropological Society, 42(3), 63-65. Sugiyama, S., & Vincent, J. (eds) (2013). Introduction to the Special Issue „Social robots and emotion: Transcending the boundary between humans and ICTs“. intervalla, 1, 1-6. UNICRI (nd). UNICRI Centre for Artificial Intelligence and Robotics. Retrieved from http://www.unicri.it/in_focus/on/UNICRI_Centre_Artificial_Robotics Urgen, B. A., Plank, M., Ishiguro, H., Poizner, H., & Saygin, A. P. (2013). EEG theta and Mu oscillations during perception of human and robot actions. Frontiers in Neurorobotics, 13, 7-19. http://doi.org/10.3389/fnbot.2013.00019/abstract Vincent, J., Taipale, S., Sapio, B, Lugano, G., & Fortunati, L. (eds) (2015). Social robots from a human perspective. Springer. Weaver, J. F. (2013). Robots are people too: How Siri, Google Car, and artificial intelligence will force us to change our laws. Praeger. Woolgar, S., Coopmans, C., Vertesi, J. (2014). Representation in scientific practice revisited. Cambridge, Massachusetts: MIT Press. Zeller, F. (2005). Mensch-Roboter Interaktion: Eine sprachwissenschaftliche Perspektive. Kassel: Kassel University Press. Zeller, F. (in press). Algorithmic Machines: From Binary Communication Designs to Human-Robot Interactions. In Special Issue “Algorithms and Communication”, Digital Communication Research. Zeller, F., Wolling, J., & Porten-Chee, P. (2010). Framing 0/1. Wie die Medien über die “Digitalisierung der Gesellschaft” berichten. Medien und Kommunikationswissenschaft, 58(4), 503–524. Zeller, F., Smith, D. H., Duong, J. A., & Mager, A. (2019). Social media in Human– Robot Interaction. International Journal of Social Robotics, 29(3), 1–14. =áRWRZVNL J., Proudfoot, D., Yogeeswaran, K., & Bartneck, C. (2015). Anthropomorphism: Opportunities and Challenges in Human–Robot Interaction. International Journal of Social Robotics, 7(3), 1–14. http://doi.org/10.1007/s12369-014-0267-6

CHAPTER 5 INTERPRETING BIOELECTRICAL SIGNALS FOR CONTROL OF WEARABLE MECHATRONIC DEVICES TYLER DESPLENTER, JACOB TRYON, EMMA FARAGO, TAYLOR STANBURY, ANA LUISA TREJOS Western University, 1151 Richmond St, London, ON, N6A 3K7 Canada

1. Introduction Facilitating intelligent human–robot interactions (HRIs), as required in wearable mechatronic devices, relies on the exchange of motion intent between the two parties, the human and the robot. Motion intent is the description of the motion that a system desires to produce. In order to derive the motion intent of a system, signals must be collected and interpreted. Determining the motion intent of humans is typically done by collecting and interpreting biological signals, such as bioelectrical signals, which form the actuation commands for human muscles. The motion intent that is derived from these signals can be used by the robotic system to make decisions about how to behave. The intent of the robotic system can also be relayed to the human, using interfaces such as numbers, text, or graphics. Without an appropriate interface, neither humans or robots are unable to detect the motion intent of their counterpart and must rely on predicting intent or reacting to the produced motion. Motion intent detection only becomes a useful tool for planning interactions between two systems if it is possible to determine the intent before the motion is produced. Humans’ motion intent can be determined before they produce motion. As a result, this allows for an appropriate

94

Chapter 5

response of a device to be generated prior to the motion being produced by the human. For a two-system interaction, such as in an HRI, there are four motion intent configurations that are possible, as shown in Figure 1. The first configuration is the most common, where the human’s motion intent is known to the robot, but the human does not know the robot’s motion intent. In this case, the robot will adapt to the human’s motion to complete the desired task. This configuration is commonly used with rehabilitative or assistive robotics, where the human’s motion abilities may vary greatly from their intent and therefore must be estimated by the mechatronic system [1–3].

Figure 1: Four basic motion intent configurations of human–robot interactions. The arrows dictate the direction of motion intent knowledge transfer from one system to the other.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic Devices

95

The second configuration is the opposite of the first configuration and is useful when a robot must perform highly repeatable tasks. This configuration has been well-established for manufacturing where humans and robots are working together to mass produce material goods and the intended robot motion must be highly predictable. It is important to note that the human’s motion intent cannot be known fully with current knowledge of the human body, only estimated. Whereas, in most cases, a robot’s motion intent can be fully described since it is programmed to perform specific motions. The third configuration is the most difficult and least common configuration for HRIs. In this case, both systems are aware of each other’s motion intent and must both adapt to achieve the desired motion task. Researchers are working on co-adaptive interaction framework and control models [4–6]. However, the limitation of existing human motion models makes it difficult to design a robot that can adapt to the ill-defined and constantly changing motion intent of the human. Furthermore, the human will need time to learn the adaptation dynamics of the robot. Large discrepancies between adaptation rates of the human and the robot can lead to interaction errors. The fourth configuration is where neither system has any information about the motion intent of the other system. Since motion intent is so difficult to determine and interpret, this configuration has become a standard for HRIs. In this configuration, the interaction must be based on the resultant motion of the systems, instead of on the motion intent. The most common model that is used to describe this configuration is through impedance and admittance interaction models [7–9]. These models describe the resistance to force or position changes that either the human or robot want to impose on their counterpart. Ideally, there would be a global solution for all HRIs that is modelled after the third configuration. If this were the case, then humans and robots could complete a variety of tasks with ever changing internal and external factors. A major limitation in achieving this scenario is in the determination of the human’s motion intent. If a model of the human body is to be developed, it would be a complex system, for which the parameters are difficult to fully determine using external measurement techniques. Even in situations where internal measurement is possible, the current technology is too limited to monitor this complexity. For most applications outside of a clinical setting, developers of robotic systems are left to build control systems that rely on data about the human that can be collected from measurements external to the body. One focus of existing HRI research is on the interpretation of motion intent from the bioelectrical signals of humans. Improvements in this domain will aid in applications that use either the first or third motion intent

96

Chapter 5

configuration. There are two main sites for the collection of the bioelectrical signals that contain motion intent: the brain and the muscles. The bioelectrical signals stemming from the brain are known as electroencephalography (EEG) signals, while those from the muscles are called electromyography (EMG) signals. The are two main reasons for using these signals in motion estimation and control of HRIs. First, muscles are actuated through the electrical stimuli that are generated within the brain or other portions of the central nervous system [10]. As a result, the motion intent of the human is encoded within these signals. Second, there is a delay between the occurrence of the bioelectrical stimulus and the production of motion, which means that there is an opportunity to react to intended motion before it occurs. These two types of bioelectrical signals contain the convolution of electrical stimuli that are produced underneath the electrodes and in the environment surrounding them. This may include electrical noise from the environment, motion artifacts from skin motion, and electrical signals from both desired and undesired electrical generators within the human body. Considering that the desired motion intent is captured within these signals, the challenge becomes extracting this motion intent from the signal. This is a major problem, as there are many sources of data captured within the same signals and the current models of biological components that generate electrical stimuli are very limited with respect to the complexity of human motor control systems. Despite these major limitations, researchers have been able to discover different ways of estimating motion intent and successfully use these signals to both predict human motion and control robotic systems during HRIs. The objective of this chapter is to highlight some existing methods for interpreting EEG and EMG signals that are useful for the control of wearable mechatronic devices. These methods are focused on modelling motion for the purpose of controlling wearable mechatronic devices that target musculoskeletal rehabilitation of the upper limb. The remainder of this chapter will detail bioelectrical signal processing methods for various types of models that are used in the estimation of motion parameters and the control of devices. The challenges of using these bioelectrical signals for HRIs will also be discussed.

2. Bioelectrical Motion Signals Bioelectrical signals are generated in the brain and are used to coordinate many of the functions within the human body, including the production of motion [11]. Capturing these signals from the skin over the skull is done

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic Devices

97

through the process of EEG [12]. Many neurons are firing simultaneously during the capture of EEG signals, making it difficult to separate out the neurons that are controlling a specific muscle or motor pattern. In order to produce skeletal motion, some of the bioelectrical signals that are generated in the brain must propagate out to the muscle tissues. Bioelectrical signals at the muscle level are captured through the process of EMG [13]. When these signals are captured from the skin, this process is called surface electromyography [11]. These signals are used to command each of the motor units that make up the muscles, and together, these motor units drive the force production of the muscles. EMG signals can be directly correlated to the desire of the individual to produce force within the muscles. An advantage of EMG signals is that they are affected by fewer sources of noise compared to EEG signals. In the case of motion generation, it takes time for the biological circuitry to transmit signals and produce a force. In control, these durations of time are known as delays. At a cellular level, there are delays between the transmission of action potentials between each cell. However, modelling of human motion, especially from using externally derived signals, considers only the aggregation of these transmission delays. When developing models that consider bioelectrical signals, there are three major delays: generation delay, propagation delay, and activation delay. Generation delay is the amount of time it takes to generate the action potential in the nervous system. Propagation delay is the duration of time it takes an activation potential to travel from the source to the muscle tissues. This delay varies based on the distance the signal must travel. Once the signal reaches the muscle, there is a duration of time before muscular tension is produced and this is known as the activation delay. Commonly, these three delays are aggregated together under the term electromechanical delay (EMD). This is due to the difficulty in measuring the generation, propagation, and activation delays separately and in a noninvasive manner. Therefore, EMD represents the entire duration between the generation of the action potential and actual production of the desired force. This delay varies between muscles and based on other factors, such as the level of fatigue [11] and chemical availability [14]. The EMD is one of the characteristics of motion production that makes EMG and EEG signals useful for motion estimation. Due to this delay, EMG and EEG signals allow for the determination of an individual’s desired motion before that motion occurs. However, determining the motion intent requires first processing these bioelectrical signals.

98

Chapter 5

3. EMG Signal Processing A general EMG signal processing procedure may consist of DC offset removal, band-pass filtering, rectification, and normalization. However, not all of these steps are performed in all signal processing procedures. DC offset removal involves shifting the entire signal, such that the EMG signal oscillates around 0 volts. The offset value is determined by finding the oscillation point and subtracting this offset from the collected EMG signal. A band-pass filter is used to remove high-frequency noise and lowfrequency motion artifacts from the EMG signal. The community is not unified on the cut-off frequencies to use for band-pass filters. De Luca states that the main power of the signal is situated in the 0–500 Hz band and that filtering the signal in the 20–500 Hz band increases the signal to noise ratio [15]. In the literature, high-pass cut off frequencies range between 5–100 Hz and low-pass frequencies between 300 and 10 000 Hz [10, 15–20]. Figure 2 shows a comparison of collected EMG signals before and after the various band-pass filters have been applied.

Figure 2: Band-pass filters, such as the fourth-order Butterworth with 20–300 Hz bandwidth used here, is applied to EMG signals to remove unwanted noise. The differences between the raw EMG signals (top) of the biceps brachii (blue) and triceps brachii (red) muscles and the filtered EMG signals (bottom) are subtle and difficult to detect visually.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic Devices

99

Once the EMG signals have been filtered, the next step is to scale the signal. Many of the models for estimating human motion rely on a signal representing the muscle activation as a percentage of the full force generation capacity of the muscles, such as on a range from 0 to 1 [10]. However, EMG signals are a voltage signal that is both positive and negative. Scaling the EMG signal involves a two-step process of rectification and normalization. Rectification is a process that transforms a signal oscillating about a point to be bounded by only oscillating above or below that point. To remove the negative voltages of the collected EMG signal, an absolute-value function can be applied to the signal. This ensures that the signal oscillates only between 0 and some positive maximum voltage. The rectification process makes the EMG signal easier to interpret and has been used by many researchers in the processing of EMG signals for control and motion estimation [10, 21–27]. After rectification, the signal has a lower bound of 0, but the maximum value of the signal is the maximum voltage detected from the individual who produced the EMG signal. Normalization is a method to provide an upper bound of 1 for the signal and has been used by many researchers when processing EMG signals [21, 23, 26, 28–30]. Commonly, the subject or user would perform a maximum voluntary contraction (MVC) of the desired muscle and the maximum voltage of that signal would be used as the value from which to scale the collected EMG signals. However, this does not always ensure that the signal remains within the bounds due to variability of the EMG signals. Figure 3 shows the transformation of the filtered EMG into a rectified and normalized EMG signal. The end result of this signal processing pipeline is a signal containing the information that is thought to have a high correlation to the activation of the muscle, and is bounded between the values 0 and 1, which can be used as input to the motion estimation models.

3.1 EMG Signal Features Upon visual inspection, the processed EMG signal still varies significantly in its shape compared to the corresponding motion signals. As a result, many researchers have developed metrics that can be extracted from the processed EMG signal. These are also known as features and correlate with time-domain or frequency-domain changes in the EMG signals. It is understood that EMG signals contain information that relate to the activation of the muscle tissues [22]. However, due to the difficulty of measurement and technological limitations, the EMG signal cannot

100

Chapter 5

currently be fully decomposed into the individual sources of muscle action potentials.

Figure 3: An example of the filtered (top), rectified (middle), and normalized (bottom) EMG signals of the biceps brachii (blue) and triceps brachii (red) muscles. The scales of the graphs are different in order to highlight the small changes in the signals.

The EMG signal features attempt to address this limitation by capturing the relationships between changes in the EMG signal and the corresponding motion characteristics, such as joint force or position changes. In general, these features have been developed under the basic assumption that muscle force or joint torque is proportional to the features. Using this assumption, these features are able to inform motion estimation and control models about the desired motion intent of the individual. A summary of common features used in EMG-driven motion estimation and control is presented in Table 1.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 101 Devices

Table 1: Mathematical descriptions of common EMG signal features. Feature [Reference] Slope Sign Change (SSC) [31]

Description ேିଵ

ܵܵ‫ = ܥ‬෍ ݃ሾሺ‫ݔ‬௜ െ ‫ݔ‬௜ିଵ ሻ × ሺ‫ݔ‬௜ െ ‫ݔ‬௜ାଵ ሻሿ ௜ୀଶ

݃(‫ = )ݖ‬൜ Mean Absolute Value (MAV) [32] Mean Absolute Value Slope (MAVS) [33] Zero Crossings (ZC) [32]

Autoregressive Coefficients (AR) [31] Root Mean Square (RMS) [32]

1,

if ‫ ݖ‬൒ ܵܵ‫ܥ‬௧௛௥௘௦௛௢௟ௗ otherwise

0,

(1) (2)

ே

1 ෍|‫ݔ‬௜ | ܰ

‫= ܸܣܯ‬

(3)

௜ୀଵ

‫ܸܣܯ = ܸܵܣܯ‬௝ାଵ െ ‫ܸܣܯ‬௝

(4)

ேିଵ

ܼ‫ = ܥ‬෍ ߮(‫ݔ‬௜ ) 1, ߮(‫ݔ‬௜ ) = ൜

(5)

௜ୀଵ

if ‫ݔ‬௜ × ‫ݔ‬௜ାଵ < 0 and |‫ݔ‬௜ െ ‫ݔ‬௜ାଵ | ൒ ܼ‫ܥ‬௧௛௥௘௦௛௢௟ௗ (6) 0, otherwise ை

‫ = ܴܣ‬෍ ܽை ‫ݔ‬௜ିை + ‫ݓ‬௜

(7)

௜ୀଵ

ே

1 ܴ‫ = ܵܯ‬ඩ ෍ ‫ݔ‬௜ ଶ ܰ

(8)

௜ୀଵ

Wave Length (WL) [32] Mean Frequency (MNF) [31]

ே

ܹ‫ = ܮ‬෍(‫ݔ‬௜ െ ‫ݔ‬௜ିଵ )

(9)

௜ୀଵ ெ

ெ

‫ = ܨܰܯ‬෍ ݂௝ ܲ௝ ൙෍ ܲ௝ ௝ୀଵ

௝ୀଵ

(10)

102 Median Frequency (MDF) [31]

Chapter 5 ெ

‫= ܨܦܯ‬

1 ෍ ܲ௝ 2

(11)

௝ୀଵ

* ‫ – ݔ‬EMG signal, ܰ – number of samples, ݅, ݆ – indices, ܵܵ‫ܥ‬௧௛௥௘௦௛௢௟ௗ – slope sign change threshold, ܼ‫ܥ‬௧௛௥௘௦௛௢௟ௗ – zero crossing threshold, ܽ – polynomial coefficients, ܱ – polynomial order, ‫ – ݓ‬white noise error term, ݂ – frequency, ܲ – power spectrum, ‫ – ܯ‬length of the frequency window.

Most of these EMG features are calculated from the processed EMG signals, as described in the previous section. However, features can also be determined from the raw EMG signals or any other combination of processing operations. For example, the zero crossings feature requires that the EMG signal is shifted to oscillate around 0 V and that it has not been rectified [31]. Generally, EMG features are calculated over a finite number of samples, known as a window. Sliding window feature determination involves sliding this window along the data set and recalculating the feature, as shown in Figure 4. Due to the variability of the EMG signals, it has been recommended that features are calculated over windows containing 250– 500 data points for EMG signals sampled at 1000 Hz [34]. However, there has been a wide variety of window sizes and overlaps used when calculating features from EMG signals. In fact, some researchers have even compared window sizes and overlaps, showing that these parameters have an effect on the calculated value [35]. For control systems that use an EMG-driven model, many features, window sizes, and overlaps should be considered. The choice of these parameters, along with other signal processing methods, can have a significant effect on the performance of the control systems. Controlling wearable mechatronic devices comes with computational limitations, which means that the number of features chosen should be constrained to reduce the computational burden. Since there is no global set of ideal EMG features, it is important to consider the requirements of the application and the resources available to the control system when choosing features. In many cases, determining the appropriate set of EMG features will require experimentation using different sets of features.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 103 Devices

Figure 4: Features are calculated using a sliding window technique. The window has a specific size and slides along the data set. Windows can overlap previous windows (top graph), have no overlap (middle graph), or have spaces between subsequent windows (bottom graph), where data points between windows are not used for feature calculation. In this example, the window size is 250 data points and the overlaps are 125, 0, and -50 data points, respectively.

104

Chapter 5

4. EMG-driven Models for Control A major advantage of using bioelectrical signals for the control of wearable mechatronic devices is that they can provide information to the control systems to enable more intelligent decisions to be made, which enhances the performance of the devices. Specifically, EMG signals provide the motion intention information of a particular muscle segment, muscle, or group of muscles, allowing the device to intuitively respond to the user’s needs. Due the difficulty in fully characterizing the EMG signal, the information contained within the signal has been decoded in many ways. Some control applications require the motion intention be decoded into a binary state (ON/OFF). Other control applications rely on the motion intention to be decoded in a more continuous nature, such that control system parameters can be estimated at each time step. In the following sections, four different EMG signal interpretation methods for control will be presented and discussed.

4.1 Thresholding Control There are many applications for wearable mechatronic devices, where simply knowing the user’s motion intention as a binary state is sufficient to complete the desired motion task. Thresholding control is formed on this basis and involves performing a pre-defined motion when the user’s motion intention is an ‘ON’ state. To facilitate this binary motion intention state, a particular value for each EMG feature is chosen to distinguish the ‘ON’ and ‘OFF’ states, known as the threshold. At values above this threshold, the user is thought to be in a state of intending to move, while below this threshold, the user is considered at rest. Essentially, one or more EMG features can be used to inform the control system that the user has crossed the threshold(s) and is or will be attempting to perform a motion. Thresholding control rules are commonly paired with other control models to estimate motion parameters. For example, the estimated angular velocity (߱௘ ) to drive the elbow joint of a wearable mechatronic device can be determined using a proportional model in combination with a threshold function, as follows: ߱௘ = ‫ܭ‬஻஻ ή ݁݉݃஻஻ ή ݃(݁݉݃஻஻ ) 1, if ݁݉݃ > ܶ஻஻ ݃(݁݉݃) = ൜ 0, otherwise

(12) (13)

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 105 Devices

where ‫ܭ‬஻஻ is the proportionality constant, ݁݉݃஻஻ is the EMG signal of the biceps brachii muscle, ݃(݁݉݃) is the threshold function, and ܶ஻஻ is the threshold voltage. Using this model, the command elbow joint velocity will only be non-zero if the EMG signal is above the pre-determined threshold. At any point below the threshold, the motion of the device will stop, which builds a safety mechanism into the model. An experiment was conducted in which a wearable mechatronic elbow brace was controlled using the model described in Equations (12) and (13), with the goal of tracking subject position using a velocity-based control scheme [36]. In this experiment, three subjects completed elbow flexion motions, which were recorded using an EMG sensor on the biceps brachii muscle and an accelerometer attached to the subject’s wrist. In the experiment, the EMG signal processing only involved normalization. The EMG signals were sampled at a rate of 1000 Hz and split into nonoverlapping windows of 250 ms. Based on the analysis of the recorded data, the proportionality constant, ‫ܭ‬஻஻ , and the threshold voltage, ܶ஻஻ , were determined by manually examining the subject’s data and choosing these values. The estimated velocity from Equation (12) was then commanded to the device at a frequency of 4 Hz. The average root mean square (RMS) position tracking error was 14.57 ± 7.48° across all subjects. A comparison of the position tracking performance is shown in Figure 5.

Figure 5: Comparison of elbow joint trajectories captured from the subjects (blue line) and a wearable mechatronic elbow brace (red line) controlled during an elbow flexion motion task. The device was controlled using the model described in Equations (12) and (13). Each graph represents one of the three different subjects that participated in the experiment.

Using this simplistic model, elbow flexion motion tasks were completed, but the accuracy of the control system to track subject motion was limited. One limitation with this model is seen when the EMG signal is being used as a continuous input at each time step. Due to the nature of the EMG signals, the EMG signal may drop below the threshold for one time-step,

106

Chapter 5

which causes the velocity to drop to zero even though the individual was continuing to produce motion during this time-step. One solution to this problem is to command device motion at much lower frequency than that at which the EMG signal is sampled. This will force the control system to take an average of the motion parameter across a window of data, which will produce a non-zero value if at least one of the data points in the window is non-zero. Another limitation is that the proportionality constant and threshold voltage need to adapt to the subject over time, which was not the case in this experiment. As a result, this thresholding control may be better suited as a method for beginning and stopping pre-defined motion tasks, as opposed to motion tasks that require adaptive control performances.

4.2 Motion Parameter Classification A second method for decoding motion intention to improve control system intelligence is through the use of classifiers. Classifiers are a set of models that use one or more features to distinguish between a set of classes [37]. There are cases in which it is not enough to decode the motion intention into a binary state, such as when a device needs to track human motion accurately at each time step. Classifiers can be used to decode the motion intention to a higher number of classes, which lends more power to control systems that need to make decisions based on these motion intention states. It is important to note that classifiers can also be used for binary state classification, as will be discussed in Section 4.4. The reason threshold controllers may not be suitable for motion tracking or motion assistance applications is due to the fact that a variety of motion factors, such as limb position, external forces, and motion velocity, affect the EMG signals during dynamic movements. Motion classifiers can be used to recognize patterns in the EMG signal that are associated with different levels of these motion factors. This allows the control system to adapt to changes in the desired motion task, the user, or the environment. For example, a motion classifier used in an assistive device could identify the direction of an external force applied to the system and provide this information to the control system so that it can compensate for these forces appropriately. However, the accuracy of classification models to classify motion greatly decreases for dynamic unconstrained motion when compared to simple constrained motions in a laboratory environment [38, 39]. An experiment was conducted in order to explore the ability to use EMG for the classification of three important motion factors: limb position, interaction force, and motion velocity [30]. A KUKA robot was utilized to

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 107 Devices

perform a repeated measures motion trial, which involved collecting dynamic data during elbow flexion–extension (FE) and activities of daily living (ADLs) with the limb in different orientations, varying motion speeds, and specified interactions forces between the subjects and the robot. EMG signals from 15 upper limb muscles were sampled at a frequency of 2000 Hz, while motion data from the KUKA robot were sampled at a frequency of 1000 Hz. A set of 11 EMG features were extracted from the filtered and normalized EMG signals, consisting of the MAV, SSC, WL, ZC, RMS, MNF, MDF features and a fourth-order AR model, where each coefficient represents a single feature (Table 1). This process resulted in a total of 165 features calculated for each window of data. Window size and overlap were held constant at values of 250 ms and 125 ms, respectively. The EMG features were predictors fed into linear discriminant analysis (LDA) and support vector machine (SVM) classification models, which are common models used for classification of EMG signals [40–42]. Both LDA and SVM models are types of machine learning algorithms that map one or more features to a set of one or more classes. In this work, these models were used to detect classes of arm position, force levels, or movement velocities. The results of this classification experiment are shown in Table 2. For elbow flexion–extension motions, both the LDA and SVM had difficulty classifying velocity (stationary, slow, or fast), but were able to predict limb position and interaction force with a higher degree of accuracy (between 73 and 79% with chance being 33%). The ability of both the LDA and SVM classification models to classify both force and velocity during ADLs was not much higher than chance (50%). These results align with previous studies that show less accurate classification for unconstrained dynamic motions versus more constrained motions, such as the elbow FE motion performed in this experiment [38, 39]. Through the development of appropriate classification models working in conjunction with complex control strategies, EMG signals can act as an intuitive interface between a human and a device. Intelligent systems can relate a user’s EMG signals to their intended motion in order to control an assistive device. Based on the experimental results, motion classifiers can be used to classify motion parameters, such as limb position or interaction force, in a small finite set of classes. This information could allow control systems to change their behaviour based on these classes. For example, the stiffness term of an impedance-based control model could be adjusted based on the level of interaction forces that the user and the device are experiencing. However, the benefits of classification approaches are still limited by a small finite set of classes from which to classify motion parameters. As a result, there may be significant variations in the motion

Chapter 5

108

parameters while the classifier outputs the same motion parameter class. For HRIs that require smooth and continuous control of the device, motion classification may still be too limited to provide a large benefit. However, in applications where there is a small finite set of desired tasks or task variations, the motion classification models can be used effectively to adapt control system behaviour or switch between control system models. Table 2: Classification accuracy comparison between the linear discriminant analysis (LDA) and support vector machine (SVM) classification models. Motion Type

Characteristic

Elbow FE Elbow FE

Position Force

Classes

LDA Accuracy (%) 78.70 73.77

SVM Accuracy (%) 83.02 74.54

3 (P1, P2, P3) 3 (0 N, +22 N, -22 N) Elbow FE Velocity 3 (stationary, 43.98 47.22 slow, fast) ADLs Force 2 (11 N, 54.69 60.42 22N) ADLs Velocity 2 (slow, fast) 52.78 54.86 * P1 – Arm down along the torso, P2 – Arm horizontal, stretched forward, P3 – Arm horizontal, stretched to the side, FE – flexion–extension

4.3 Musculotendon and Joint Dynamics Models Musculoskeletal motion models attempt to define some or all of the biological components and relationships that contribute to the production of joint motion. At minimum, they attempt to define relationships between biological parameters, such as bioelectrical signals, and the resultant motion of the body segments. However, some researchers have proposed motion models that include aspects that are impossible to measure precisely and externally, such as bone deformation dynamics [Moody2009]. For these models, their parameters become best approximations and may never represent the exact dynamics of the musculoskeletal system. Musculoskeletal motion models can be compared along a complexity spectrum. At one end of the spectrum, there are models that treat the entire complex relationship between bioelectrical signals and motion as a blackbox, such as neural network models [25]. These types of models use mathematical relationships that do not require knowledge of the underlying musculoskeletal system to map inputs to outputs. At the other end of the

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 109 Devices

spectrum, there are models that attempt to decompose the complexity of the human motion systems as much as possible. For example, Fuglevand et al. developed a model to define the electrical pulses of individual motor unit action potentials [43]. Much of the control research for using musculoskeletal motion models falls within the middle of the spectrum. This is because the black-box models are not accurate or robust enough, and the highly detailed models often require the determination of parameters that cannot be measured externally to the human body, or at all, given current technologies available to wearable mechatronic devices. In order to examine these musculoskeletal motion models, four different models were developed for the control of a wearable mechatronic elbow brace. Three of these models, a Kalman filter motion model (KFMM), a proportional motion model (PMM), and a nonlinear polynomial motion model (NPMM), have much of the elbow motion dynamics aggregated. These models defined the relationships between EMG signals of the biceps and triceps brachii muscles and the joint position with mathematical models that aggregate the dynamic components of the elbow muscles and joint. The output of these three models was the estimated elbow joint position. The fourth model was developed based on the Hill-type motion model (HTMM). This model represents a level of complexity decomposition where there are biological representations for whole muscles and tendons, as well as some passive joint forces. However, many of the parameters corresponding to these tissues cannot be measured externally and must be determined through optimization procedures. This model was used to determine the estimated joint torque. A comparison of these models is presented in Figure 6.

110

Chapter 5

Figure 6: A visual comparison of the inputs, outputs, and components of the Kalman filter motion model (KFMM), proportional motion model (PMM), nonlinear polynomial motion model (NPMM), and Hill-type motion model (HTMM). ݁݉݃ is the processed EMG signal, ‫ ݑ‬is the neural activation signal, a is the muscle activation signal, ‫ܨ‬ெ is the force of the muscle, ‫ܯ‬ெ is the moment generated by the muscle about the elbow, ߠ௠ is the measured elbow joint position, and ‫ܯ‬௘ , ߠሷ௘ , ߠሶ௘ , ߠ௘ are the estimated joint moment, acceleration, velocity, and position, respectively. Although the PMM and NPMM contain the same modules in this figure, the implementations within the musculotendon dynamics modules are different.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 111 Devices

4.3.1 Elbow Joint Position Control Neural inputs to the muscle tissues are used to activate the desired contraction of muscles, however, the resulting forces are directly coupled to the position of the joints through limb dynamics. Furthermore, the brain must regulate both force and position in order to coordinate safe and effective movement of the limbs. As a result, some models, such as the KFMM, are developed to directly link between the EMG signals and joint motion. The PMM and NPMM assume an intermediary step that links the EMG signals to produced muscle forces through musculotendon dynamics, and then aggregates those forces to describe the joint position changes. These three models are described briefly in the following sections, while further details of the models can be found in [27]. 4.3.1.1 Kalman Filter Musculotendon Model The KFMM was developed by Kyrylova et al. as a simplified Kalman Filter model for correlating EMG signals from the biceps brachii and triceps brachii muscle directly to changes in elbow joint position [23]. Using this model to control a wearable mechatronic elbow brace, Kyrylova achieved an average position tracking error ranging from 2.65° to 5.62°. From this original version, the KFMM prediction equation was modified to the following: ߠ௣ (‫ ݐ‬+ 2) =

‫ܩ‬௡௔ ‫ ݐ( ் ݑ‬+ 1) + ߠ௠ (‫)ݐ‬ 2

(14)

where ߠ௣ is the predicted joint angle, ߠ௠ is the measure joint angle, ‫ ் ݑ‬is the difference between the neural activation signals derived from the biceps brachii and triceps brachii EMG signals, and ‫ܩ‬௡௔ is a gain for amplifications of the combined neural activation signals. The KFMM has a two-step ahead prediction of the elbow joint position, as shown in Equation (14). For this model, the neural activations are defined using the linear envelope feature (see Table 1) of the processed EMG signals. The Kalman filter then corrects the predicted joint angle for both measurement and process noise through the following equations: ܲ(‫ܲ = )ݐ‬ᇱ (‫ ݐ‬െ 1) + ܳ ܲ(‫)ݐ‬ ‫= )ݐ(ܩ‬ ൫ܴ + ܲ(‫)ݐ‬൯ ᇱ (‫)ݐ‬ = ܲ(‫)ݐ‬൫1 െ ‫)ݐ(ܩ‬൯ ܲ

(15) (16) (17)

112

Chapter 5

ߠ௖ (‫ߠ = )ݐ‬௣ (‫ )ݐ‬+ ‫ )ݐ(ܩ‬ቀ‫ )ݐ( ் ݑ‬െ ߠ௣ (‫)ݐ‬ቁ

(18)

where P(t), Q(t), and P’(t) are functions of noise and ߠ௖ is the corrected joint angle estimate. The KFMM can be configured through optimizing the ‫ܩ‬௡௔ , ܴ, and ܳ parameters of the model. Using this version of the KFMM, a series of control experiments were conducted in which the model was responsible for predicting subjects’ elbow position based on their EMG signals and current elbow position. EMG signals were sampled at rates of either 1000, 2000, or 4000 Hz, depending on the experiment. Across all experiments, EMG signals were processed by filtering, removing DC offset, rectification and normalization. The MAV feature was used as input to the KFMM and was calculated over windows with lengths of 250 ms and a variety of window overlaps (0.5–4 ms). Across a variety of elbow motions and experimental conditions, the KFMM was able to achieve average position tracking errors of 0.03–8.1° and 0.44–12.46° during simulations and offline remote-control experiments, respectively [27]. The results suggest that the KFMM has the potential to be a highly accurate model for predicting elbow motion. However, experimental analysis has shown that the KFMM performance varies between subjects performing the same motion. Furthermore, there are cases where the KFMM produces very similar estimation errors using only position input (‫= )ݐ( ் ݑ‬ 0, ‫ )ݐ׊‬or only EMG input (ߠ௠ (‫ = )ݐ‬0, ‫ )ݐ׊‬when compared to using both EMG and position input. This should be explored further by varying the weighting of the contributions of both EMG and position inputs and creating alternative forms of Equation (14). 4.3.1.2 Proportional Musculotendon Model The creation of the PMM was based on the idea that the contribution of each muscle to the joint torque is directly proportional to the EMG signals, as follows: ‫ߠܫ‬ሷ = ‫ܭ‬஻஻ ‫ݑ‬஻஻ െ ‫்ܭ‬஻ ‫் ݑ‬஻

(19)

where ‫ ܫ‬is the inertia of the lower arm, ‫ݑ‬஻஻ is the neural activation of the biceps brachii muscle, ‫் ݑ‬஻ is the neural activation of the triceps brachii muscle, and ‫ܭ‬஻஻ and ‫்ܭ‬஻ are the muscle torque constants of the biceps brachii and triceps brachii, respectively. The joint position is derived from taking the double integral of the acceleration in Equation (19). The decades of study and remaining mysteries of human motion control have shown that the complexity of human motion cannot fully be described

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 113 Devices

by a simple proportional model. However, this model was developed as a method for quantifying a very simplistic form of joint position estimation from the linear envelope of the processed EMG signals. As a result, it provides a baseline estimation that is useful for comparing against other models. The PMM was used in an experiment to evaluate its ability to track human elbow position [27]. EMG signals were collected from eight healthy subjects at a sampling frequency of 2000 Hz, which were filtered, adjusted for DC offsets, rectified, and normalized. The MAV feature was calculated over a 250 ms windows with a 1 ms overlap. During this experiment, the PMM was able to estimate dynamic elbow motion with position estimation errors of 4.32 ± 1.09°. The PMM was also able to track elbow motion through remote control of a wearable mechatronic elbow brace with position tracking errors of 12.64 ± 4.99°. These results suggest that this simple model can produce elbow joint position estimates that align with other studies in the literature [23, 29]. However, the addition of the PMM when controlling the wearable mechatronic elbow brace produced much higher tracking errors than during the estimation. The results from the experiments in which this model was used also showed a high variability in tracking errors from subject to subject. Future experiments with the PMM may help to uncover the causes of this variability. 4.3.1.3 Nonlinear Polynomial Musculotendon Model Development of the NPMM was based on work by Clancy et al. and Liu et al., which used nonlinear polynomial models to define the relationship between EMG and torque during static postures [44, 45]. Based on the structure of these models, a new model was defined to provide a relationship between the EMG signals and joint position, as follows: ஽

ோ

‫ߠܫ‬ሷ = ෍ ‫݌‬ௗ ‫ݑ‬஻஻ ௗ െ ෍ ‫݌‬௥ ‫் ݑ‬஻ ௥ ௗୀ଴

(20)

௥ୀ଴

where ‫݌‬ௗ and ‫݌‬௥ are the constants of the polynomials, and D and R are the degree of the polynomial. The first polynomial in Equation (20) represents the contributions of the biceps brachii, while the second represents the triceps brachii muscle torque. The joint position estimate is derived from the acceleration defined in Equation (20). In this work, the NPMM was configured to a degree-3 (D, R = 3) polynomial with two EMG signals being used as the inputs, one for each of the biceps brachii and triceps brachii

114

Chapter 5

muscles. These inputs were defined as the linear envelope of the processed EMG signals. An elbow motion tracking experiment was conducted to evaluate the NPMM [27]. Like the previous experiments, this one involved both estimation of human motion and tracking of human elbow motion with a wearable mechatronic elbow brace. EMG signal collection and processing methods were the same as those used for the PMM. The NPMM was able to estimate subject motion with an average error of 30.08°. It was also used to track subject motion through control of the device with an average error of 16.68°. The high average estimation error is due to large variability in this model’s performance across individuals. For some individuals, the optimization of this model resulted in a few degrees of error, while for others it reached the range of 100 degrees of error. However, when controlling the brace, the average tracking error decreased with respect to the optimized model estimations. This was due to the fact that the brace could not respond quickly to those large errors, which essentially filtered them out of the performance. As with the KFMM and PMM, further exploration is required to determine the factors that influence how this model will behave for each individual. 4.3.2 Elbow Joint Torque Control Elbow joint torque can be determined by accounting for the forces and torque acting upon the elbow joint through models. A series of EMG signals can be used to represent the activation of the muscles that produce these forces. The most commonly used models that define these relationships in terms of musculotendon dynamics are variations of the Hill-type musculotendon model (HTMM) [17, 28, 46]. The HTMM breaks the different musculotendon dynamics into separate components, such as the active and passive forces of the musculotendon complex. In the generalized HTMM, the active forces of the muscle, the passive forces of the muscle, and the passive forces of the tendon are represented by a contractile element (CE), a parallel passive element (PPE), and a series passive element (SPE), respectively. Typically, the PPE and SPE are modelled as springs, as shown in Figure 7.A, however, Millard et al. found that modelling the SPE as a rigid tendon increased simulation speeds and produced similar estimation accuracy when using sub-maximal activations [47]. Based on this finding, the HTMM shown in Figure 7.B, in which the SPE is chosen to be a rigid tendon element, was used for elbow joint torque estimation.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 115 Devices

The CE takes three inputs, the muscle activation, muscle fiber length, and muscle fiber velocity, to produce the active force of the muscle (‫ܨ‬஼ா ), as follows: ‫ܨ‬஼ா = ܽ(‫ܨ)ݐ‬௠௔௫ ‫ܨ‬ி௅ (‫ܮ‬ெ )‫ܨ‬ி௏ ൫‫ܮ‬ሶெ ൯

(21)

where ܽ(‫ )ݐ‬is the muscle activation signal, ‫ܨ‬௠௔௫ is the maximum isometric force of the muscle, ‫ܨ‬ி௅ is the normalized force–length relationship of the muscle, ‫ܨ‬ி௏ is the normalized force–velocity relationship of the muscle, and ‫ܮ‬ெ and ‫ܮ‬ሶெ are the muscle fiber length and velocity, respectively. It is important to note that the normalized force–length and force– velocity curves are used to generalize the musculotendon properties, but these curves will vary for each individual and are based on many individual factors. Furthermore, the force–length and force–velocity relationships of muscles are not decoupled from the muscular activation inside the human body [48]. In Equation (21), they are decoupled to simplify the dynamics.

Figure 7: The generalized Hill-type musculotendon model (A) consists of a contractile element (CE), parallel passive element (PPE), and series passive element (SPE), which represent the active force of the muscle, passive force of the muscle, and passive force of the tendon, respectively. A variation on this model (B) considers the SPE as a rigid tendon. In this figure, LMT is the length of the musculotendon unit, LM is the length of the muscle fiber, and ‫ ׋‬is the pennation angle between the tendon and muscle fibers. This figure was modified from [46].

The PPE of the HTMM is often modelled as a nonlinear spring, such as the one presented by Chadwick et al. [46]: ‫ܮ‬ெ ൑ ‫ܮ‬ௌ ݇ଵ (‫ܮ‬ெ െ ‫ܮ‬ௌ ), ‫ܨ‬௉௉ா = ൜ ݇ଵ (‫ܮ‬ெ െ ‫ܮ‬ௌ ) + ݇ଶ (‫ܮ‬ெ െ ‫ܮ‬ௌ )ଶ , ‫ܮ‬ெ > ‫ܮ‬ௌ

(22)

Chapter 5

116

where ݇ଵ and ݇ଶ are stiffness coefficients and ‫ܮ‬ௌ is the muscle slack length. Together Equations (20) and (21) are combined using an equilibrium model of the connection point between the muscle and the tendon, as follows: ‫ܨ‬ௌ௉ா = (‫ܨ‬஼ா + ‫ܨ‬௉௉ா ) cos ߶

(23)

where ‫ܨ‬ௌ௉ா is the force of the series passive element and ߶ is the pennation angle of the muscle. Since a rigid-tendon element was used for the SPE, ‫ܨ‬ௌ௉ா becomes the output force of this HTMM. A simulation experiment was conducted to determine the ability of a simplified HTMM to estimate elbow joint torque based on data collected from healthy individuals [26]. The elbow joint torque estimation model used in this experiment consists of two HTMM, one to represent elbow flexor muscles and another to represent elbow extensor muscles, which is a very simplified view of the musculature that drives human elbow motion. EMG signals are collected from the biceps brachii and triceps brachii muscles at a sampling frequency of 4000 Hz. These signals are then processed by filtering, DC offset removal, rectification and normalization, before being transformed through two other models to account for the neural activation and muscle activation dynamics, as shown in Figure 6. A window size of 250 ms was used with an overlap of 0.5 ms. Elbow joint position was also measured from the subjects. Using OpenSim (National Center for Simulation in Rehabilitation Research, California, U.S.A.), equations were derived that define the relationships between elbow joint position and muscle fiber length and between elbow joint position and musculotendon length (‫ܮ‬ெ் ). Muscle fiber velocities were calculated based on changes in the muscle fiber length. The moment arm (‫ )ݎ‬for each muscular force was determined by differentiating the musculotendon length with respect to elbow joint position, as follows: ‫=ݎ‬

݀ ‫)ߠ( ܮ‬ ݀ߠ ெ்

(24)

The elbow joint position, muscle fiber length, muscle fiber velocity and moment arm data were then used as input to an HTMM. Two HTMMs were used for generating the estimated muscle forces, where each one represents the aggregated force production of the elbow flexors and elbow extensors, respectively. Finally, these two estimated forces were combined with other forces acting on the elbow joint to produce an estimated joint moment (‫ܯ‬௝௢௜௡௧ ), as follows:

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 117 Devices ே

‫ܯ‬௝௢௜௡௧ = ‫ߠܫ‬ሷ + ‫ܯ‬௉ + ‫ ீܯ‬+ ෍ ‫ܨ‬ௌ௉ா ‫ݎ‬௜

(25)

௜ୀ଴

where ‫ܯ‬௉ is the moment due to passive joint forces, such as damping, and ‫ ீܯ‬is the moment due to gravitational forces. Average elbow joint torque estimation errors of 2.09 ± 1.39 Nm resulted from the experiment. Compared to existing studies that employed HTMMs (3.4–4.2 Nm [28, 49]), these results are an improvement. Although this simplified HTMM was able to produce estimation errors that were an improvement on existing studies, the time it took to optimize this model was long. The HTMM attempts to represent the dynamics of musculotendon complexity with individual components, however, this introduces many parameters, most of which cannot be measured. As a result, model optimization techniques are required, but become time consuming due to the number of unknown parameters. In the elbow torque estimation model, only two HTMMs were used to represent the entirety of the musculature that drives the elbow. Optimizations took between 1 and 4 hours to complete on a desktop computer system. Since there still remains significant work to be conducted on mapping raw EMG signals to the muscular activations, it is likely that the optimizations conducted on the model will not be valid for different motion tasks. In addition to this, the EMG signals and the underlying biological state are highly variable, which complicates this issue further. The elbow torque estimation model was only tested during a motion tracking simulation. However, it would be simple to extend this to control a motion tracking task using a wearable mechatronic device. Through the addition of actuator and transmission system models, the estimated elbow torque can be converted to an input signal for the actuator. This process was used for testing the KFMM, PMM, and NPMM during motion tracking control of a wearable mechatronic elbow brace. Most studies involving HTMMs were only concerned with motion simulation experiments using data collected from healthy individuals. As a result, there are many opportunities to explore both the control of wearable mechatronic devices using HTMMs and variations in the model that result from using data collected from muscle at different health levels.

4.4 Muscle Health Model Intelligent wearable devices have been proposed not only to provide movement assistance, but also to assist with patient monitoring and health

118

Chapter 5

assessment. Since EMG provides a direct measurement of muscle and nerve signals, there is the possibility of using EMG sensing systems or wearable mechatronic devices to interpret, monitor, and assess muscle health. Collecting data in this manner can be used to develop muscle health models. In this case, a muscle health model will be able to classify or estimate parameters that represent the health of the muscle, such as the maximum force production, based on the EMG signals collected from the individual. Muscle health models have two major benefits for wearable mechatronic devices, especially those used in musculoskeletal rehabilitation. First, muscle health can be assessed objectively and changes in the muscle health can be tracked over time. If muscle health is assessed using a wearable device, it could enable diagnostic assessments to occur outside of clinical settings, while a subject is performing movements in variety of conditions. Second, muscle health models can be used to improve the intelligence of the control systems of wearable mechatronic devices. For instance, the maximum torque output of the device could be scaled to a level that is safe for each individual’s needs. Both of these usages of muscle health models have merit, but developing a model for improved control performance is the focus of this discussion. Surface EMG signal features have been found to exhibit differences between healthy subjects and patients with a variety of neuromuscular or muscle disorders, including Duchenne muscular dystrophy [50], nonspecific arm pain [51], and stiff elbow [52]. Haddara et al. identified statistically significant differences in EMG signals recorded from elbow trauma patients and a group of healthy subjects [53]. In particular, the RMS and MAV features exhibited significantly higher differences in injured subjects compared to healthy subjects. Promisingly, the RMS and MAV features collected from the patients near the end of their therapy program were found to more closely resemble the healthy population. This work can be extended further, in order to generate a muscle health model for elbow trauma patients. For this purpose, an experiment was conducted using EMG data recorded from thirty elbow trauma patients [54]. Elbow traumas included elbow fractures, elbow dislocations, arthroscopic releases, and bicep tendon repairs. For this experiment, the uninjured limb from the same subject was used as the healthy control. The patients were instructed to perform 10 upper limb motions, including elbow flexion, elbow extension, forearm pronation, forearm supination, wrist flexion, wrist extension, ulnar deviation, radial deviation, hand open, and hand close motions. Surface EMG signals were sampled at a frequency of 2000 Hz from 7 elbow-related muscles, which included the biceps brachii, triceps brachii lateral head, triceps brachii long head, pronator teres, brachioradialis,

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 119 Devices

extensor carpi ulnaris, and flexor carpi ulnaris. Each of the signals was band-pass filtered within a frequency range of 20–450 Hz. A muscle health model was developed using machine learning models to classify whether the EMG signals indicated that the data set was taken from a healthy or injured limb. Forty-two EMG features were extracted, including all of the features in Table 1, using a window size and window overlap of 250 and 125 ms, respectively. Different numbers of features were combined into feature sets and used as inputs for the classification algorithms. Three machine learning algorithms, LDA, SVM and Random Forest (RF), were used to perform this binary classification of muscle health. A depiction of the muscle health classification process is shown in Figure 8. Feature sets consisting of 3–5 features were developed based on individual feature performances, and the type of information (i.e., muscle fatigue, motor unit contraction rate) thought to be provided by each feature. The developed muscle health models achieved classification accuracies of 45.9–82.1%. These results suggest that the set of features chosen for classification has a major impact on the classification accuracy. However, in some cases, it may be possible to produce a high level of classification accuracy, which would help to improve the control system performance for HRIs. There was also some evidence, based on three returning patients that repeated the experiment after further rehabilitation, that the EMG signal features of the injured limbs began to more closely resemble the behaviour of the healthy limb. However, due to the variability of the subjects and their traumas, the muscle health models will need to be individualized to maintain a high level of accuracy. Furthermore, a simple binary classification of muscle health may not provide much benefit for control systems due to the variation in patient factors, such as type of injury, patient health prior to injury, and response to rehabilitation. Although muscle health models have the potential to individualize HRIs, further exploration of the relationships between muscle health and EMG signals is still needed.

120

Chapter 5

Figure 8: Flowchart of muscle health classification process. EMG signals features are extracted, combined into feature sets, and used to train the classifiers. The trained classifiers are then tested to determine their classification accuracy.

5. EEG Signal Processing While control of many wearable mechatronic devices is done using EMG, it is not the only bioelectrical signal that has been used for this purpose. Many systems base their control scheme on changes in brain

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 121 Devices

activity, measured with EEG. As with all bioelectric signals, it is hard to use EEG in its raw form and processing is required to refine the signal to a useable state. In general, EEG signals are processed similarly to EMG: filter noise, segment the signal using windowing techniques, and calculate features. The main difference comes in the complexity of the signal and its susceptibility to noise. The amplitude of EEG is several orders of magnitude smaller than EMG (0.5–100 μV for EEG [12] compared to 0–100 mV for EMG [42]), meaning that other signals can easily contaminate it, ruining the signal-to-noise ratio. Even other bioelectrical signals within the subject’s own body, for example electrocardiogram signals generated by the heart (known as ECG) or EMG signals generated by the neck/jaw muscles, can easily dominate the EEG signal [12], as shown in Figure 9. For this reason, filtering is a crucial step of EEG processing, and many different approaches are used, depending on the application. Understanding the fundamental meaning behind EEG signals has been a large barrier to its use as a control signal for wearable devices. While EMG analysis has traditionally focused on time-domain information, much of the key information encoded in EEG has been studied in the frequency-domain [55]. This can make it harder to intuitively understand what is happening in the EEG signal via direct observation, without significant signal processing. The function of the brain itself also contributes greatly to this complexity. It is constantly active, processing different information and controlling various bodily functions. Therefore, it is more difficult to correlate EEG signals to motion, than it is for EMG. As shown in Figure 10, the times at which the person is either in motion or at rest, is clearer in the EMG signal than it is in the EEG signal. In general, the EEG signal shows no distinct visual pattern to help differentiate between periods of motion and rest. This makes the selection and calculation of features an important aspect of using EEG within a mechatronic device. As with EMG, the first step of EEG processing is to remove unwanted noise using a bandpass filter. The frequency range used for the bandpass filter is highly variable and can change depending on the application. It is understood that the frequency range for EEG can fall between 1 and 100 Hz [56]; however, as the frequency of the EEG components increases, the amplitude of these components decreases. This makes the higher frequencies hard to measure, since it becomes difficult to distinguish these components from noise in the signal. For this reason, a smaller range of frequencies can be used that will exclude the higher frequencies. For example, this can be performed by using a low-pass filter with a cut-off frequency that is lower than the expected high frequency range of the EEG, such as 40 Hz [57].

122

Chapter 5

Figure 9: An example EEG signal showing contamination from EMG signals generated in the neck muscles (top) and ECG signals (bottom). EMG contamination is recognized by the presence of higher frequency, higher amplitude components. ECG contamination is recognized by larger amplitude periodic spikes present in the EEG signal. In both cases, an example of the contamination is outlined in red.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 123 Devices

Figure 10: A sample EEG (top) and biceps EMG (bottom) signal recorded during three repetitions of elbow flexion–extension. The queues given to start and stop motion are indicated by the green and red lines, respectively. By observing the EMG signal, a distinction between motion and rest can be observed by looking at the change in overall muscle activity. However, it cannot be as clearly observed in the EEG signal, as it does not present a distinct visual change.

124

Chapter 5

Beyond being used for noise removal, bandpass filtering is also used to separate the EEG signal into different frequency components. It is common in many EEG applications to decompose the signal further into different frequency ranges called EEG bands, or sometimes called rhythms, with each band labelled using a letter of the Greek alphabet. The frequency ranges for these bands are defined based on observations made that link signal activity in certain frequencies to different biological actions. It can be useful in some applications to isolate certain frequency bands depending on what action the EEG signal is trying to detect. It should be noted that the cut-offs for each frequency band are not an exact number and different researchers may vary the numbers slightly. However, the general range of each band is typically agreed upon. Table 3 lists the various EEG bands with commonly used frequency cut-off ranges, as well as corresponding actions often associated with each band [58]. After bandpass filtering, spatial filtering can also be applied in an attempt to improve the performance of EEG-driven devices. Spatial filtering is used to reduce the number of EEG signal channels by combining nearby channels or selecting optimal channels using various methods. The reason for combining channels is that information generated in the brain does not perfectly correlate to one specific channel. Neighbouring electrodes can pick up portions of the EEG signal generated over an area because of how the electrical signal propagates through the skull [55, 59]. Some spatial filters attempt to reconstruct the proper EEG signal, by removing other EEG information picked up by nearby electrodes. There are multiple spatial filtering algorithms used to achieve this, with a popular method being the Laplacian filter [59]. Another use of spatial filtering is to perform channel selection, choosing an optimal subset of channels for each subject to improve performance [60]. A commonly used method to do this is the Common Spatial Pattern (CSP) algorithm, which weights the channels to maximize variance between different mental tasks being detected [57, 61].

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 125 Devices

Table 3: Frequency ranges and physiological action assigned to each EEG bands [58]. Name

Frequency Range [Hz]

Delta (į) Theta (ș) Alpha (Į) Beta (ȕ) Gamma (Ȗ)

30

Mu (μ)

8–12

Physiological Significance Deep sleep Drowsiness, meditation Relaxation, closed eyes Active concentration Sensory processing, short-term memory Like Į, but recorded over motor cortex, motion rest state

Finally, for some EEG applications, further noise removal is necessary to use the signal properly. Noise caused by other bioelectrical signals or external sources can greatly affect the EEG signal, and sometimes bandpass filtering is not sufficient to completely remove all noise from the signal. In this case, other more advanced filtering methods can be used to try and remove unwanted contributions to the EEG signal [62]. One method used for this is called Independent Component Analysis (ICA). ICA breaks the EEG signal into multiple base components and attempts to remove unwanted components contained within the EEG signal before reconstructing it. A common example of this technique is to use ICA to remove ECG signal contamination from the EEG signal [63]. Since ECG is a very recognizable waveform, it is possible to identify its components in the EEG signal and remove it using ICA. Unfortunately, more advanced filtering techniques, like ICA, are often very computationally expensive and are usually performed during offline analysis where computation power/time is not as limited. In a wearable application, it may not be feasible to make use of these methods, in which case the system will need to be robust enough to use the EEG signals containing noise from other sources.

5.1 EEG Signal Features Many different types of brain activity can be detected using EEG signals, such as Steady State Visually Evoked Potentials, which are an oscillatory response to a flashing visual stimulus, or the P300 wave, which indicates a change in EEG amplitude that occurs roughly 300 ms after an unexpected stimulus [64]. One of the most commonly used types of brain activity sensing for control of wearable mechatronic devices is Motor

126

Chapter 5

Imagery (MI). MI activity is detected in the motor cortex during the presence of actual or imagined motion, and can be used by a mechatronic device to know when the person wants to actuate the system [64, 65]. By using the appropriate EEG features, MI brain activity can be measured and used as an input signal to the system. Within the EEG signal, MI activity is recognized as a change in the signal power of certain EEG bands, namely the μ and ȕ bands [55, 66]. During a real or imagined motor action the power in these bands will decrease, which is referred to as an Event Related Desynchronization (ERD). Likewise, the power will increase in the absence of an MI action, known as Event Related Synchronization (ERS) [55, 66]. Detecting ERD or ERS is the primary focus of MI-based systems and is done commonly using band power as the main feature [55]. Band power calculates the average signal power over a specific frequency range, which in this case would be ranges for the μ and ȕ bands (Table 3), to capture any increase or decrease over time. Band power can be calculated using various methods [67]. Another common feature used for MI detection are the Hjorth parameters [68]. The Hjorth parameters are time domain features that provide frequency information from the EEG signal. This means that they can be calculated directly from the time domain EEG signal without requiring any transformation to the frequency domain. This is a significant benefit, as it allows the Hjorth parameters to provide frequency information in a less computationally expensive way. This can be of great use for systems with limited computational resources, such as those used in wearable mechatronic devices. There are three Hjorth parameters that provide different information on the EEG signal. They are expressed, for the EEG signal, ‫)ݐ(ݔ‬, as follows [69]: ே

‫݁ܿ݊ܽ݅ݎܽݒ = ݕݐ݅ݒ݅ݐܿܣ‬൫‫)ݐ(ݔ‬൯ = ෍ ௜ୀଵ

(‫ )݅(ݔ‬െ ߤ)ଶ ܰ

(26)

݀‫)ݐ(ݔ‬ ൰ ‫ ݕݐ݅ݒ݅ݐܿܣ‬൬ ݀‫ݐ‬ ඩ ‫= ݕݐ݈ܾ݅݅݋ܯ‬ ‫ݕݐ݅ݒ݅ݐܿܣ‬൫‫)ݐ(ݔ‬൯

(27)

݀‫)ݐ(ݔ‬ ൰ ݀‫ݐ‬ ‫= ݕݐ݅ݔ݈݁݌݉݋ܥ‬ ‫ݕݐ݈ܾ݅݅݋ܯ‬൫‫)ݐ(ݔ‬൯

(28)

‫ ݕݐ݈ܾ݅݅݋ܯ‬൬

where ߤ is the mean of ‫ )ݐ(ݔ‬and ܰ is the number of samples in ‫)ݐ(ݔ‬.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 127 Devices

Activity provides a measurement of the average power of the EEG signal. A high value for Activity indicates that a large number of high frequency components are present and visa versa for low activity [70]. Mobility provides an estimate of the mean frequency of the EEG signal [69]. Complexity indicates an estimate of bandwidth [69], which provides information on the change in frequency by the EEG signal. For scale, the lowest complexity score possible is 1, which would indicate that the EEG signal is a pure sine wave [70]. These three parameters, while simple to calculate, have shown promise as efficient EEG features that can be used to extract MI information from the EEG signal. While these features are commonly used in EEG control task based on MI changes, they are by no means the only features available. Other features, such as Discrete Wavelet Coefficients [66] and Power Spectral Density [70, 71], have shown potential for EEG applications. There are countless features that use EEG, and indeed one could likely write a chapter solely on EEG features given how many are present in the literature. The complex nature of EEG means that no universally optimal features exist, and it can be very application and subject specific. Care should be taken when using EEG to develop a feature set that is optimal for the specific scenario in question, to ensure that the correct information is being captured from the EEG signal. As with EMG, EEG signals are usually broken into segments using windows to increase the number of feature samples and to see how the signal varies over time. Since the response of the EEG signal is slower, when compared to EMG, longer window lengths are sometimes used. However, there is no general consensus on optimal window lengths, and they can vary greatly for different applications. Some researchers may user shorter window lengths, such as 250 ms [72], while others may use window lengths of 1 second [73] or longer. For wearable mechatronic devices, these longer window lengths can present a significant challenge. For real-time applications, a study showed that a delay greater than 300 ms can have a major impact on a subject’s ability to control a system [74]. This means that EEG windows should be kept below 300 ms if the data can influence the real-time performance of the system. EEG systems need to be optimized to provide suitable MI detection accuracy, even with window sizes smaller than what is often used in MI applications.

128

Chapter 5

6. EEG and EMG Fusion Both EEG and EMG have shown potential to be used for control of wearable devices, however, both are hindered by unique challenges. It can be difficult to decode EEG signals, as they do not provide a direct measurement of muscle activity [75] and can require complex computation to be useable. On the other hand, EMG does offer a more direct measurement of muscle activity, but it is affected by changes in the body, such as muscle fatigue, which can drastically alter the expected behaviour of the signal [76, 77]. One method that has been proposed to address these challenges is the simultaneous use of both EEG and EMG to control a wearable device, referred to as EEG–EMG fusion [77]. In EEG–EMG fusion, information is taken from both bioelectrical signals and combined using various fusion algorithms into a single control signal used by the mechatronic system. Obtaining motion intention information from multiple bioelectrical sources can allow the system to leverage the strengths of each signal to try and address their drawbacks, providing a more robust control scheme to the user. A common method of preforming EEG–EMG fusion is through the use of machine learning classifiers and the following two approaches: featurelevel fusion and decision-level fusion. In feature-level fusion, the EEG and EMG signals are processed separately until feature extraction is performed. At this point the EEG and EMG features are combined into one feature set, which is then used to train one classifier that outputs motion intention decisions (Figure 11). The fusion of the signals is done inside the classifier, which uses the information from both feature sets to train the model. This fusion technique is referred to as the Single-Classifier (SC) method and has been presented by several research groups [78–81]. In decision-level fusion, both signals are processed separately and used to train separate classifiers (one for EEG and one for EMG). Both classifiers will output a prediction of motion intention, and this output is what is fused using different fusion algorithms, as shown in Figure 11. Many different methods of combining EEG–EMG classifier decisions have been demonstrated by various research groups. A summary of the different fusion methods can be seen in Table 4.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 129 Devices

Feature-Level Fusion

Decision-Level Fusion

Figure 11: The processes used for both fusion types. Feature-level fusion is where the information from both bioelectrical signals is combined before classifier training and only one model is trained using features from both EEG and EMG. Decisionlevel fusion is where information from each bioelectrical signal is used to train their own classifier and the output of these classifiers is combined into one decision, using various methods of fusion.

Table 4: EEG–EMG fusion methods used for decision-level fusion. Name Weighted Average AND OR EEG Gating EMG Gating Majority Vote

Method of Fusion Outputs are combined using weighted average of class probabilities [76, 81–83]. Activates if both EEG and EMG detect motion [84–88]. Activates if either EEG or EEG detect motion [87, 88]. Checks EEG first for motion. If it detects motion, then checks EMG [75, 89]. Checks EMG first for motion. If it detects motion, then checks EEG [90]. Multiple classifiers are trained for EEG and EMG. The result is the most common output [91].

130

Chapter 5

6.1 Evaluating EEG–EMG Fusion Methods While many methods of EEG–EMG fusion have been proposed, the efficacy of this control method still requires further evaluation [92]. Testing was often done using a small number of subjects [75–77, 83, 87] and usually focused on only evaluating one fusion method. Few experiments have been done to compare the performance of different fusion methods against each other. Two such studies have been performed on a small subset of EEG– EMG fusion methods, but the motions performed by the study subjects during data collection were limited to simple button presses, which is not representative of the types of dynamic motions seen during control of wearable mechatronic devices [87, 88]. Changes in motion parameters, such as motion speed, amount of applied force, or muscle fatigue, can have a large effect on bioelectrical signals, particularly the EMG signals, and how these factors affect the performance of EEG–EMG fusion needs to be determined. To further assess the effectiveness of EEG–EMG fusion methods, an experiment was completed where EEG–EMG signals were collected during elbow–flexion extension motions, with varied motion types, and used as input to different EEG–EMG fusion methods for motion classification [91]. This allowed for a more extensive comparison of the algorithms and observation of their performance in more realistic motion scenarios. Data collection was performed on 18 healthy subjects who had their EEG and EMG signals sampled at a frequency of 4 kHz. Two sets of motions were performed: speed/load varying and muscle fatiguing motions. During the speed/load varying motions, the subjects performed elbow flexion– extension motions with different combinations of two speeds: slow (10°/second), and fast (150°/second) and three weights: 0 lbs, 3 lbs, and 5 lbs. During muscle fatiguing motions, the subjects held a higher weight (10 lbs) and performed more repetitions. EEG signals were recorded from the C3, C4, and Cz locations specified by the International 10–20 System used to standardize EEG electrode placement (Figure 12.A) and they were referenced using ear clip electrodes attached to the subject’s ear lobe. These locations were chosen because they correlate with motor cortex of the brain, the area responsible for motor control. EMG signals were recorded using bipolar electrodes placed on the center of the biceps brachii and triceps brachii muscles, placed according the SENIAM Project guidelines used to standardize EMG electrode placement (Figure 12.B) [93]. These muscles were chosen for signal collection as they are the primary muscles used during elbow flexion– extension.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 131 Devices

A

B

Figure 12: The placements of the EEG (A) and EMG (B) electrodes used. In (A), the EEG electrodes are indicated in red on the International 10–20 System diagram showing possible EEG electrode locations. In (B), a picture of an example study participant showing the bipolar EMG electrode locations for the biceps (left) and triceps (right) is shown.

After EEG–EMG signals were collected from the study participants, an offline analysis was completed to evaluate EEG–EMG fusion, as shown in Figure 13. Both signals were filtered and processed to remove any unwanted noise. EEG was filtered using filtered using a 0.5–40 Hz band pass filter (3rd order Butterworth) and EMG had the DC offset removed and was filtered using a using a 20–500 Hz band pass filter (4th order Butterworth). Following this, the signals were segmented into regions of motion (when the subject was performing elbow flexion–extension) and regions of rest (when the subject was not moving their arm), which was done using markers placed in the data via an external trigger system, while the trials were being conducted. Using a window size of 250 ms and a 125 ms overlap, features were extracted from both EEG and EMG signals. For EEG, the features calculated were the Hjorth Parameters and band power of the ȝ and ȕ frequency bands. The EMG features calculated were the MAV, MAVS, WL, ZC and fourth-order AR coefficients.

132

Chapter 5

Figure 13: A summary of the data analysis method used during the offline evaluation of EEG–EMG fusion methods. Note that the Single-Classifier fusion method differs from this protocol slightly, since in this case, the EEG–EMG features are combined into one feature set and used to train one classifier.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 133 Devices

Following their extraction, the EEG and EMG features were used to train SVM binary classifiers to determine the intention to move or remain at rest. An SVM model was trained for both EEG and EMG each, as well as a third model trained using combined EEG–EMG features to provide an example of Single Classifier fusion. The decisions of the separate EEG– EMG classifiers were combined using the AND, OR, and Weighted Average fusion methods. Weights of 50%/50%, 25%/75%, and 75%/25% for EEG/EMG, respectively, were used for the Weighted Average fusion method. The gating and majority vote fusion methods were not used in this experiment. The results of the experiment showed an overall classification accuracy of 86.81%, 70.98%, and 86.77% for the EMG-only, EEG-only, and top performing EEG–EMG fusion method (Weighted Average 25%/75%), respectively. A summary of key results of each classification model is presented in Table 5. This experiment provided a robust evaluation of EEG– EMG fusion used for motion classification. It was able to show that an advantage of using EEG–EMG fusion was not necessarily an improvement in classification accuracy, but in the ability of the system to remain more stable during changes in motion parameters. Fusion methods were able to achieve the same overall accuracy as EMG, but did not show the same accuracy drop and increase in variability present in EMG alone when moving at the slow speed with no weight (Table 5, Row 1). Likewise, when the performance of EEG dropped during the fast/no weight motions, the fusion methods once again were not as affected (Table 5, Row 2). Even though both EEG and EMG showed performance drops for certain motions, many fusion methods remained consistent throughout, demonstrating the ability of fusion to provide a more stable performance. It is important to note that this work was an initial analysis on EEG–EMG fusion and further research must be done to explore more experimental factors. For instance, more combinations of weights and speeds, fatigue-specific motion protocols and advanced signal processing methods should be explored. The improvement in robustness of classification through the fusion of EEG and EMG signals is a promising result from this experiment that can be used to improve the performance of decision-making models used for the control of wearable mechatronic devices.

Chapter 5

134

Table 5: Accuracy results of each classification model during two example motion types, as well as the overall performance averaged across all motion types. Data are presented as the mean ± standard deviation across subjects. Motion Type slow, 0 lbs fast, 0 lbs Overall

Model Accuracy [Percentage of Correct Classifications] EEG

EMG

SC

89.4 ± 2.8 48.8 ± 6 .7 71.0 ± 2.9

87.3 ± 9.9 89.0 ± 9.4 86.8 ± 3.9

83.3 ± 12.9 89.6 ± 6.6 86.6 ± 3.7

AN D 85.3 ± 10.9 87.8 ± 9.7 85.2 ± 3.4

OR 91.4 ± 1.9 50.1 ± 6.6 72.6 ± 3.1

WA50/50 90.1 ± 6.4 86.7 ± 11.9 86.5 ± 4.1

WA75/25 90.9 ± 2.8 55.7 ± 6.0 74.4 ± 2.4

WA25/75 88.2 ± 8.8 88.4 ± 9.9 86.8 ± 3.9

7. Challenges for Interpreting Bioelectrical Signals The use of bioelectrical signals to improve control system performance during HRIs has resulted in many opportunities to enhance the quality of life through the use of assistive mechatronic devices. However, the interpretation of these signals is still limited, and many challenges must be overcome to provide the community with devices that can be controlled in a continuously adaptive manner. For example, EMG signal decomposition and its use in the control of devices has been studied for many decades. Although many important pieces of knowledge have been discovered, the result of these efforts still leaves many developers trimming much of the information from these bioelectrical signals or using models that condense the dynamic properties of human motion into a few classes. This mainly due to the fact that achieving ubiquitous HRIs is an extremely difficult problem. The difficulty of this problem stems from the inherent complexity and variation of humans, robots, and their interactions. Humans are composed of a highly adaptive set of synergistic systems. There are many factors that affect the ability of humans to produce forces on both devices and environments, such as motion task training, genetic factors, nutrition, sleep, hydration, hormone profiles, and mental state. On the other hand, mechatronic devices can be designed to be much less variable and complex. However, the design choices, such as the sensors, signals processing methods, control system models, and actuation systems, can have a major

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 135 Devices

impact on the ability of the device to contribute in HRIs. For example, misplacement of EEG and EMG sensors can have a significant impact on the ability of the control system to determine the specific motion intent of the user. Being able to perform useful interactions between humans and mechatronic devices is also heavily dependent on aspects of the interaction, including the desired motion tasks, mechanical interfaces, data feedback systems, motion intention detection capabilities, and properties of the external environment. Accounting or controlling for all of these factors makes it an extremely difficult endeavour. One method for dealing with this complexity is to program the devices to have a highly predictable behaviour, create well-defined motion interactions, and relay the motion intention of the device to the user. However, even in these cases, the human is a highly adaptive and unpredictable system, which makes the entirety of the HRI difficult to control in a productive and safe manner. The desired goal is to be able to control and direct HRIs in such a way that the devices can be used in many environmental conditions and are also able to adapt their motion intention to improve the performance of the HRI (Fig. 1, Configuration 3). However, there are still many more efforts required to realize these types of unconstrained interactions. In addition to the natural complexity and variation of the human, there are limitations with existing technologies that make it impossible to fully determine certain aspects of the human for use in a real-time adaptive control system. For example, the ability of the human to perform repetitive motion tasks without resting is highly dependent on the level of various chemicals in the muscle. Having information about the level of these chemicals could allow the device to adapt its contribution to the motion task with much more accuracy, but there are currently no methods for determining these aspects of the muscle in an accurate and non-invasive manner. Together, the complexity, variability, and technological limitations present a set of problems that will require many future efforts to address. Given current technology and understanding, bioelectrical signals cannot be fully decoded. As a result, bioelectrical signals contain more information than the existing models can handle. To deal with this issue, many signal processing methods are used in an attempt to target specific pieces of information that have been shown to correlate with either the internal components of the musculoskeletal system, such as muscle health models, or the motion of the human, such as musculoskeletal motion models. Decomposing EEG and EMG signals presents its own difficulty as these signals are known to contain data not relevant to the particular motion intent of interest.

136

Chapter 5

EEG signals may contain potentials that are generated for the control of many functions that the brain is responsible for regulating, while EMG signals may contain both muscle potentials and nerve potentials. Furthermore, both signal types may also contain electrical potentials from other internal sources, such as the heart (see Fig. 9.B), and external sources, such as power line interference on the sensing system. As a result, the models that act upon these signals are acting not only on the muscle action potentials, but upon other types of potentials in the signals. This issue is compounded by the fact that these signals occur at very small amplitudes, making it difficult to distinguish genuine motion data from noise in the signal. Although bioelectrical signals present new opportunities for HRIs, the control of the devices to perform these behaviours is limited by the ability to decode motion intention information from these signals. The inherent variability in bioelectrical signals forces developers to implement case-specific models. This is due, at least in part, to the fact that human biology is affected by a multitude of factors. For example, the literature shows that there are clear differences in muscle fatigue patterns [94] and in EMG characteristics between males and females [95]. It is likely that physiological differences in muscle activity, neurological mechanisms of action potential propagation, subcutaneous fat, and muscular function will cause differences in bioelectrical signals between males and females. Furthermore, each of these physiological characteristics are influenced by genetics, sleep, nutrition, hydration, and motion training, among other factors. This diversity of biology will require accommodation within the software of wearable mechatronic devices. To date, no work has examined how algorithms that rely on EMG signals perform differently in males and females, let alone the many other factors that affect these signals. Hence, it is paramount to investigate whether there are additional differences to consider depending on these factors. These considerations must be accounted for in participant recruitment strategies (to ensure adequate representation), in the analysis of the data (exploring potential interaction effects in planned discriminative analyses), and in the development and translation of the technology (how the algorithms consider the sensed data). In general, this means that control system models will need to be optimized for each user and each motion task. For example, it was found that elbow motion models used to control a wearable mechatronic elbow brace had statistically different performances when used for different elbow motions and subjects [27]. It is important to note that these differences were found using EMG signals collected from healthy individuals. For individuals suffering from musculoskeletal disorders, the need for individualization goes further. Both Haddara et al. and Farago et al. found that the EMG signal

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 137 Devices

amplitudes of individuals that suffered elbow traumas were changed as a result of their injury [53, 54]. They found that muscle activity was lower for some muscles and higher in other muscles, when compared to healthy baselines. Features that are extracted from bioelectrical signals are often chosen based on either developer experience or using feature selection methods that assess the correlation of the features with a particular motion parameter. For example, the frequency cut-offs of the EEG bands are not consistent across studies and no consensus to their exact values have been made by the community. These frequency band values are a convention developed by researchers for convenience and should not be completely accepted as physiological truth. It is likely that the brain itself does not use hard cut-offs for how information is grouped. Even in the case that this was true of the brain, the physiological variability between people would likely mean that everyone has slightly different ranges of values [3]. Again, this idea supports the use of control models that can be optimized for each particular case. However, one drawback of this approach is that it takes time and resources to optimize models for many different cases. Control system intelligence comes at a cost of computational resources. All modern control systems of mechatronic devices are housed and executed on computer systems. Therefore, the computer systems constrain the type and amount of data that can be processed, as well as the kind of behaviours that can be performed by the device. For wearable mechatronic devices, this presents a whole new set of challenges. The computer systems of these devices must be small, wireless, and battery-powered, while accounting for the aforementioned complexity and facilitating the interaction of the user and itself. To meet these criteria, the processing power of these computer systems is often limited, when compared to the system used to design and simulate the control models and HRIs. The limited computational resources often negate the use of more accurate signal processing techniques, such as the spatial filtering method used for EEG signals. Spatial filtering requires a larger number of channels to be effective, since for each desired electrode location there needs to be enough neighbouring electrodes to properly apply the spatial filter. For a device with limited computational resources, this can be a challenge, since each channel that is added increases the overall complexity and processing time. Additional EMG sensors abide by the same computational limitation as well. It is intuitive to place one or more EMG sensors on each of the muscles that contribute to a joint’s motion. However, measuring the entirety of the upper limb muscles could result in dozens of EMG sensors. Currently, there are no computer system technologies that could simultaneously meet

138

Chapter 5

this processing demand and other important requirements of a wearable system, such as the size and power requirements. One method to address this issue is to use multiple sensor types, such as with the fusion of EEG and EMG signals. Leveraging information from both signal types at the same time could make the system more robust to the natural variability of each signal type, as was shown in the experiment discussed in Section 6.1. This could allow for the simplification of the control system, which is critical in wearable applications where processing power and device complexity is severely limited. However, sampling and processing multiple signal types comes with its own computational cost. Therefore, developers must consider the computational cost of their design choices and perform trade-off analyses between the computational demand and other important control system factors, such as accuracy and robustness. Technological advances in computer system will provide new opportunities to improve control system intelligence and performance, but the complexity of humans, devices, and their interactions will continue to be a major challenge.

8. Conclusion Bioelectrical signals have provided opportunities to develop and control mechatronic devices that are capable of enhancing our interactions with the world around us. This is due to the fact that human motion intention is encoded within these signals. Thus far, the two most common bioelectrical signals used for the control of wearable mechatronic devices are EEG and EMG signals. EEG signals provide a point of motion intention as developed in the human brain, while EMG signals collect that motion intention as it is produced within human muscle. The extraction of motion intention can be used to provide data to control systems of mechatronic devices that enable them to interact in a useful and safe manner while worn by the user. Due to an ongoing quest to understand EEG and EMG signals, many different signal processing methods, signal features, and motion models have been proposed. Control system models that use EMG signals range in complexity from simple thresholding controllers to continuous musculoskeletal motion estimation models. More recently, classification algorithms have been used to fuse EEG and EMG signal features together in an attempt to address the limitations of the individual signal types. The use of these models for motion parameter estimation and control has been successful, but further development is still required to facilitate the control of more complex and unconstrained HRIs. Understanding the dynamic relationships

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 139 Devices

between these bioelectrical signals and internal systems of the human body will enable more intelligent control of wearable mechatronic devices.

Acknowledgements The authors would like to thank Shrikant Chinchalkar, Abelardo Escoto, Evan Friedman, Anastasiia Kyrylova, Daniel Lizotte, Joan Lobo-Prat, and Arno Stienen for their contributions to the development, testing, and analysis of the models and control systems presented in this chapter.

References [1]

[2]

[3] [4] [5] [6] [7] [8]

[9] [10]

J. Huang, W. Huo, W. Xu, S. Mohammed, and Y. Amirat, “Control of UpperLimb Power-Assist Exoskeleton Using a Human-Robot Interface Based on Motion Intention Recognition,” IEEE Trans. Autom. Sci. Eng., vol. 12, no. 4, pp. 1257–1270, 2015. A. M. Khan, F. Khan, and C. Han, “Estimation of desired motion intention using extreme learning machine for upper limb assist exoskeleton,” in IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 2016, pp. 919–923. T. Kawase, T. Sakurada, Y. Koike, and K. Kansaku, “A hybrid BMI-based exoskeleton for paresis: EMG control for assisting arm movements,” J. Neural Eng., vol. 14, no. 1, pp. 1–12, 2017. N. Jarrassé, T. Charalambous, and E. Burdet, “A Framework to Describe, Analyze and Generate Interactive Motor Behaviors,” PLoS One, vol. 7, no. 11, pp. 1–13, 2012. J. C. Landes, “Intrinsic and Extrinsic Biomechanical Factors in a Co-adaptive ECoG-based Brain Computer Interface,” Ph. D. Thesis, Washington University, Washington, Missouri, U.S.A., 2017. K. J. Wang, M. Sun, R. Xia, and Z. H. Mao, “Human-robot symbiosis framework on exoskeleton devices,” in Proc. IEEE Int. Conf. Ind. Technol., 2016, pp. 1500–1506. N. Hogan, “Impedance Control: An Approach to Manipulation: Part I— Theory,” J. Dyn. Syst. Meas. Control, vol. 107, no. 1, pp. 1–7, 1985. D. W. Franklin, R. Osu, E. Burdet, M. Kawato, and T. E. Milner, “Adaptation to Stable and Unstable Dynamics Achieved by Combined Impedance Control and Inverse Dynamics Model,” J. Neurophysiol., vol. 90, no. 5, pp. 3270– 3282, 2003. A. Q. L. Keemink, H. van der Kooij, and A. H. A. Stienen, “Admittance control for physical human–robot interaction,” Int. J. Rob. Res., vol. 37, no. 11, pp. 1421–1444, 2018. T. S. Buchanan, D. G. Lloyd, K. Manal, and T. F. Besier, “Neuromusculoskeletal modeling: Estimation of muscle forces and joint

140

[11]

[12] [13] [14] [15] [16] [17] [18] [19] [20] [21]

[22]

[23]

[24]

Chapter 5 moments and movements from measurements of neural command,” Journal of Applied Biomechanics, vol. 20, no. 4. pp. 367–395, 2004. P. A. Karthick, D. M. Ghosh, and S. Ramakrishnan, “Surface electromyography based muscle fatigue detection using high-resolution timefrequency methods and machine learning algorithms,” Comput. Methods Programs Biomed., vol. 154, pp. 45–56, 2018. M. Teplan, “Fundamentals of EEG measurement,” Meas. Sci. Rev., vol. 2, no. 2, pp. 1–11, 2002. K. Kiguchi and Y. Hayashi, “Motion estimation based on EMG and EEG signals to control wearable robots,” in IEEE International Conference on Systems, Man, and Cybernetics, 2013, pp. 4213–4218. J. M. Winters, “An improved muscle-reflex actuator for use in large-scale neuromusculoskeletal models,” Ann. Biomed. Eng., vol. 23, no. 4, pp. 359– 374, 1995. C. J. De Luca, “Surface electromyography: Detection and recording," DelSys Incorporated, Tech. Rep., 2002. D. Novak and R. Riener, “A survey of sensor fusion methods in wearable robotics,” Rob. Auton. Syst., vol. 73, pp. 155–170, 2015. A. L. C. Ruiz, C. Pontonnier, J. Levy, and G. Dumont, “Motion control via muscle synergies: Application to throwing,” in ACM SIGGRAPH Conference on Motion in Games, 2015, pp. 65–72. S. Wang, Y. Gao, J. Zhao, T. Yang, and Y. Zhu, “Prediction of sEMG-based tremor joint angle using the RBF neural network,” in IEEE Int. Conf. Mechatronics Autom., 2012, pp. 2103–2108. R. Song and K. Y. Tong, “Using recurrent artificial neural network model to estimate voluntary elbow torque in dynamic situations,” Med. Biol. Eng. Comput., vol. 43, no. 4, pp. 473–480, 2005. M. Hakonen, H. Piitulainen, and A. Visala, “Current state of digital signal processing in myoelectric interfaces and related applications,” Biomed. Signal Process. Control, vol. 18, pp. 334–359, 2015. J. Lobo-Prat, A. Q. L. Keemink, A. H. a Stienen, A. C. Schouten, P. H. Veltink, and B. F. J. M. Koopman, “Evaluation of EMG, force and joystick as control interfaces for active arm supports.,” J. Neuroeng. Rehabil., vol. 11, p. 68, Jan. 2014. N. Nazmi, M. A. A. Rahman, S. A. Mazlan, H. Zamzuri, and M. Mizukawa, “Electromyography (EMG) based Signal Analysis for Physiological Device Application in Lower Limb Rehabilitation,” in International Conference on Biomedical Engineering, 2015. A. Kyrylova, T. Desplenter, A. Escoto, S. Chinchalkar, and A. L. Trejos, “Simplified EMG-driven Model for Active-Assisted Therapy,” in IEEE International Conference on Intelligent Robots and Systems, Workshop on Rehabilitation and Assistive Robotics, Chicago, U.S.A., 2014. L. M. Vaca Benitez, M. Tabie, N. Will, S. Schmidt, M. Jordan, and E. A. Kirchner, “Exoskeleton technology in rehabilitation: Towards an EMGbased orthosis system for upper limb neuromotor rehabilitation,” J. Robot., 2013.

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 141 Devices [25]

[26] [27]

[28] [29] [30] [31] [32] [33] [34] [35]

[36]

[37] [38]

L. Peng, Z. G. Hou, and W. Wang, “A dynamic EMG-torque model of elbow based on neural networks,” in Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2015, pp. 2852–2855. T. Desplenter and A. L. Trejos, “Evaluating muscle activation models for elbow motion estimation,” Sensors (Switzerland), vol. 18, no. 4, pp. 1–18, 2018. T. Desplenter, “Development of Digital Control Systems for Wearable Mechatronic Devices: Applications in Musculoskeletal Rehabilitation of the Upper Limb,” Ph.D. Thesis, University of Western Ontario, London, Canada, 2018. J. Rosen, M. B. Fuchs, and M. Arcan, “Performances of hill-type and neural network muscle models-toward a myosignal-based exoskeleton.” Computers and Biomedical Research, vol. 32, no. 5, pp. 415–439, 1999. J. Bai, Y. Gao, S. Wang, and J. Zhao, “An elbow biomechanical model and its coefficients adjustment,” in IEEE International Conference on Software Engineering and Service Sciences, 2014, pp. 954–957. T. Stanbury, “Dynamic Calibration of EMG Signals for Control of a Wearable Elbow Brace,” M. E. Sc. Thesis, University of Western Ontario, London, Canada, 2018. A. Phinyomark, P. Phukpattaranont, and C. Limsakul, “Feature reduction and selection for EMG signal classification,” Expert Syst. Appl., vol. 39, no. 8, pp. 7420–7431, 2012. Z. Tang, H. Yang, L. Zhang, and P. Liu, “Effect of shoulder angle variation on sEMG-based elbow joint angle estimation,” Int. J. Ind. Ergon., vol. 68, no. July, pp. 280–289, 2018. B. Hudgins, P. Parker, and R. N. Scott, “A New Strategy for Multifunction Myoelectric Control,” IEEE Trans. Biomed. Eng., vol. 40, no. 1, pp. 82–94, 1993. G. Kamen and D.A. Gabriel, Essentials of Electromyography. Champaign, IL: Human Kinetics, 2010. L. H. Smith, L. J. Hargrove, B. A. Lock, and T. A. Kuiken, “Determining the optimal window length for pattern recognition-based myoelectric control: Balancing the competing effects of classification error and controller delay,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 19, no. 2, pp. 186–192, 2011. T. Desplenter, A. Kyrylova, T. K. Stanbury, A. Escoto, S. Chinchalkar, and A. L. Trejos, “A wearable mechatronic brace for arm rehabilitation,” in IEEE RAS/EMBS International Conference on Biomedial Robotics and Biomechatronics, 2014, pp. 491–496. F. Lotte, M. Congedo, A. Lécuyer, F. Lamarche, and B. Arnaldi, “A review of classification algorithms for EEG-based brain-computer interfaces,” J. Neural Eng., vol. 4, no. 2, pp. 1–13, 2007. D. Yang, Y. Gu, L. Jiang, L. Osborn, and H. Liu, \Dynamic training protocol improves the robustness of PR-based myoelectric control," Biomedical Signal Processing and Control, vol. 31, pp. 249–256, 2017.

142 [39] [40]

[41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53]

Chapter 5 B. Lock, K. B. Englehart, and B. Hudgins, \Real-time myoelectric control in a virtual environment to relate usability vs. accuracy," in Proceedings of the MyoElectric Controls/Powered Prosthetics Symposium, 2005, pp. 17–20. A. Phinyomark, F. Quaine, S. Charbonnier, C. Serviere, F. Tarpin-Bernard, and Y. Laurillau, “EMG feature evaluation for improving myoelectric pattern recognition robustness,” Expert Syst. Appl., vol. 40, no. 12, pp. 4832–4840, 2013. K.-H. Park, H.-I. Suk, and S.-W. Lee, “Position-Independent Decoding of Movement Intention for Proportional Myoelectric Interfaces,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 24, no. 9, pp. 928–939, 2016. N. Nazmi, M. A. A. Rahman, S. I. Yamamoto, S. A. Ahmad, H. Zamzuri, and S. A. Mazlan, “A review of classification techniques of EMG signals during isotonic and isometric contractions,” Sensors, vol. 16, no. 8, pp. 1–28, 2016. A. J. Fuglevand, D. A. Winter, and A. E. Patla, “Models of recruitment and rate coding organization in motor-unit pools,” J. Neurophysiol., vol. 70, no. 6, pp. 2470–2488, 1993. E. A. Clancy, L. Liu, P. Liu, and D. V. Z. Moyer, “Identification of constantposture EMG-torque relationship about the elbow using nonlinear dynamic models,” IEEE Trans. Biomed. Eng., vol. 59, no. 1, pp. 205–212, 2012. P. Liu, L. Liu, and E. A. Clancy, “Influence of Joint Angle on EMG-Torque Model During Constant-Posture, Torque-Varying Contractions,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 23, no. 6, pp. 1039–1046, 2015. E. K. Chadwick, D. Blana, R. F. Kirsch, and A. J. van den Bogert, “RealTime Simulation of Three-Dimensional Shoulder Girdle and Arm Dynamics,” IEEE Trans. Biomed. Eng., vol. 61, no. 7, pp. 1947–1956, 2014. M. Millard, T. Uchida, A. Seth, and S. L. Delp, “Flexing Computational Muscle: Modeling and Simulation of Musculotendon Dynamics,” J. Biomech. Eng., vol. 135, no. 2, 2013. E. J. Perreault, C. J. Heckman, and T. G. Sandercock, “Hill muscle model errors during movement are greatest within the physiologically relevant range of motor unit firing rates,” J. Biomech., vol. 36, no. 2, pp. 211–218, 2003. E. E. Cavallaro, J. Rosen, J. C. Perry, and S. Burns, “Real-time myoprocessors for a neural controlled powered exoskeleton arm.,” IEEE Trans. Biomed. Eng., vol. 53, no. 11, pp. 2387–2396, 2006. J. Y. Hogrel, “Clinical applications of surface electromyography in neuromuscular disorders," Neurophysiologie Clinique, vol. 35, no. 2–3, pp. 59–71, 2005. K. M. Calder, D. A. Gabriel, and L. McLean, “Differences in EMG spike shape between individuals with and without non-specific arm pain," Journal of Neuroscience Methods, vol. 178, pp. 148–156, 2009. C. Page, S. I. Backus, and M. W. Lenho_, “Electromyographic activity in stiff and normal elbows during elbow flexion and extension," J. Hand Ther., vol. 16, pp. 5–11, 2003. R. Haddara, Y. Zhou, S. Chinchalkar, and A. L. Trejos, “Postoperative healing patterns in elbow using electromyography: Towards the development

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 143 Devices

[54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67]

of a wearable mechatronic elbow brace,” In Proceedings of the IEEE International Conference on Rehabilitation Robotics, 2017, pp. 1395–1400. E. Farago, S. Chinchalkar, D. J. Lizotte, and A. L. Trejos, “Development of an EMG-Based Muscle Health Model for Elbow Trauma Patients,” Sensors, vol. 19, no. 15, p. 3309, 2019. F. Lotte, “A Tutorial on EEG Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces,” in Guide to Brain-Computer Music Interfacing, Springer London, 2014, pp. 131–161. S. Vaid, P. Singh, and C. Kaur, “EEG signal analysis for BCI interface: A review,” in International Conference on Advanced Computing and Communication Technologies, 2015, pp. 143–147. D. Planelles, E. Hortal, Á. Costa, A. Úbeda, E. Iáñez, and J. M. Azorín, “Evaluating classifiers to detect arm movement intention from EEG signals,” Sensors, vol. 14, no. 10, pp. 18172–18186, 2014. M. Z. Baig, N. Aslam, and H. P. H. Shum, “Filtering techniques for channel selection in motor imagery EEG applications: a survey,” Artif. Intell. Rev., pp. 1–26, 2019. M. M. Togha, M. R. Salehi, and E. Abiri, “Improving the performance of the motor imagery-based brain-computer interfaces using local activities estimation,” Biomed. Signal Process. Control, vol. 50, pp. 52–61, 2019. M. Arvaneh, S. Member, C. Guan, K. K. Ang, and C. Quek, “Optimizing the channel selection and classification accuracy in EEG-based BCI,” IEEE Trans. Biomed. Eng., vol. 58, no. 6, pp. 1865–1873, 2012. T. Alotaiby, F. E. A. El-Samie, S. A. Alshebeili, and I. Ahmad, “A review of channel selection algorithms for EEG signal processing,” EURASIP J. Adv. Signal Process., vol. 2015, no. 1, pp. 1–21, 2015. J. Cheng et al., “Remove diverse artifacts simultaneously from a singlechannel EEG based on SSA and ICA: A semi-simulated study,” IEEE Access, vol. 7, pp. 60276–60289, 2019. N. N. Kumar and A. G. Reddy, “Removal of ECG artifact from EEG data using Independent Component Analysis and S-Transform,” Int. J. Sci. Eng. Technol. Res., vol. 5, no. 3, pp. 712–716, 2016. K. S. Hong and M. J. Khan, “Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands: A review,” Front. Neurorobot., vol. 11, pp. 1–27, 2017. B. Gudiño-Mendoza, B. Gudi, G. Sanchez-Ante, and J. M. Antelis, “Detecting the intention to move upper limbs from Electroencephalographic brain signals,” Comput. Math. Methods Med., vol. 2016, pp. 1–9, 2016. S. Bhattacharyya, A. Konar, and D. N. Tibarewala, “Motor imagery and error related potential induced position control of a robotic arm,” IEEE/CAA J. Autom. Sin., vol. 4, no. 4, pp. 639–650, 2017. N. Brodu, F. Lotte, and A. Lécuyer, “Comparative study of band-power extraction techniques for Motor Imagery classification,” in IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain, 2011, pp. 1–6.

144 [68]

[69]

[70] [71]

[72]

[73] [74] [75]

[76] [77]

[78] [79] [80]

Chapter 5 P. Gaur, R. B. Pachori, H. Wang, and G. Prasad, “A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian geometry,” Expert Syst. Appl., vol. 95, pp. 201–211, 2018. P. Gaur, R. B. Pachori, Hui Wang, and G. Prasad, “An empirical mode decomposition based filtering method for classification of motor-imagery EEG signals for enhancing brain-computer interface,” in 2015 International Joint Conference on Neural Networks, 2015, pp. 1–7. R. Lahiri, P. Rakshit, and A. Konar, “Evolutionary perspective for optimal selection of EEG electrodes and features,” Biomed. Signal Process. Control, vol. 36, pp. 113–137, 2017. C. Kim, J. Sun, D. Liu, Q. Wang, and S. Paek, “An effective feature extraction method by power spectral density of EEG signal for 2-class motor imagery-based BCI,” Med. Biol. Eng. Comput., vol. 56, no. 9, pp. 1645–1658, 2018. E. Abdalsalam M, M. Z. Yusoff, D. Mahmoud, A. S. Malik, and M. R. Bahloul, “Discrimination of four class simple limb motor imagery movements for brain–computer interface,” Biomed. Signal Process. Control, vol. 44, pp. 181–190, 2018. M. Tavakolan, Z. Frehlick, X. Yong, and C. Menon, “Classifying three imaginary states of the same upper extremity using time-domain features,” PLoS One, vol. 12, no. 3, pp. 1–18, 2017. Z. Tang, K. Zhang, S. Sun, Z. Gao, L. Zhang, and Z. Yang, “An upper-limb power-assist exoskeleton using proportional myoelectric control.,” Sensors, vol. 14, no. 4, pp. 6677–6694, Jan. 2014. A. Sarasola-Sanz, A., Irastorza-Landa, N., López-Larraz, E., Bibian, C., Helmhold, F., Broetz, D., Birbaumer, N., & Ramos-Murguialday, “A hybridBMI based on EEG and EMG activity for the motor rehabilitation of stroke patients,” in IEEE International Conference on Rehabilitation Robotics, 2017, pp. 895–900. R. Leeb, H. Sagha, R. Chavarriaga, and J. D. R. Millán, “A hybrid braincomputer interface based on the fusion of electroencephalographic and electromyographic activities,” J. Neural Eng., vol. 8, no. 2, pp. 1–5, 2011. T. Dulantha Lalitharatne, K. Teramoto, Y. Hayashi, and K. Kiguchi, “Towards hybrid EEG-EMG-based control approaches to be used in biorobotics applications: current status, challenges and future directions,” PALADYN J. Behav. Robot., vol. 4, no. 2, pp. 147–154, 2013. P. Xie, X. Chen, P. Ma, X. Li, and Y. Su, “Identification method of human movement intention based on the fusion feature of EEG and EMG,” in World Congress on Engineering - WCE 2013, 2013, pp. 1–5. X. Li, O. W. Samuel, X. Zhang, H. Wang, P. Fang, and G. Li, “A motionclassification strategy based on sEMG-EEG signal combination for upperlimb amputees,” J. Neuroeng. Rehabil., vol. 14, no. 1, pp. 1–13, 2017. E. Loopez-Larraz, N. Birbaumer, and A. Ramos-Murguialday, “A hybrid EEG-EMG BMI improves the detection of movement intention in cortical stroke patients with complete hand paralysis,” in Proceedings of the Annual

Interpreting Bioelectrical Signals for Control of Wearable Mechatronic 145 Devices

[81]

[82]

[83]

[84] [85]

[86]

[87] [88] [89]

[90] [91]

[92]

International Conference of the IEEE Engineering in Medicine and Biology Society, 2018, pp. 2000–2003. R. Leeb, H. Sagha, R. Chavarriaga, and J. D. R. Millán, “Multimodal fusion of muscle and brain signals for a Hybrid-BCI,” in Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2010, pp. 4343–4346. V. F. Annese and D. De Venuto, “FPGA based architecture for fall-risk assessment during gait monitoring by synchronous EEG–EMG,” in IEEE International Workshop on Advances in Sensors and Interfaces, 2015, pp. 116–121. A. Manolova, G. Tsenov, V. Lazarova, and N. Neshov, “Combined EEG and EMG fatigue measurement framework with application to hybrid braincomputer interface,” in IEEE International Black Sea Conference on Communications and Networking, 2016, pp. 1–5. P. Aricò et al., “FES controlled by a hybrid BCI system for neurorehabilitation – driven after stroke,” in GNB2012, 2012, vol. 2, pp. 3–4. F. Cincotti et al., “EEG-based brain-computer interface to support poststroke motor rehabilitation of the upper limb,” in Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2012, pp. 4112–4115. F. Grimm, A. Walter, M. Spüler, G. Naros, W. Rosenstiel, and A. Gharabaghi, “Hybrid neuroprosthesis for the upper limb: combining braincontrolled neuromuscular stimulation with a multi-joint arm exoskeleton,” Front. Neurosci., vol. 10, pp. 1–11, 2016. E. A. Kirchner, M. Tabie, and A. Seeland, “Multimodal movement prediction - Towards an individual assistance of patients,” PLoS One, vol. 9, no. 1, pp. 1–9, 2014. H. Wöhrle, M. Tabie, S. K. Kim, F. Kirchner, and E. A. Kirchner, “A hybrid FPGA-based system for EEG- and EMG-based online movement prediction,” Sensors, vol. 17, no. 7, pp. 1–41, 2017. D. De Venuto, V. F. Annese, M. De Tommaso, and E. Vecchio, “Combining EEG and EMG Signals in a Wireless System for Preventing Fall in Neurodegenerative Diseases,” in Ambient Assisted Living: Italian Forum 2014, 2015, pp. 317–327. P. E. Gaillardon et al., “A digital processor architecture for combined EEG– EMG falling risk prediction,” in Design, Automation & Test in Europe Conference & Exhibition, 2016, pp. 427–432. T. D. Lalitharatne, K. Teramoto, Y. Hayashi, and K. Kiguchi, “Evaluation of perception-assist with an upper-limb power-assist exoskeleton using EMG and EEG signals,” in IEEE International Conference on Networking, Sensing and Control, 2014, pp. 524–529. J. Tryon, E. Friedman, and A. L. Trejos, “Performance evaluation of EEG– EMG fusion methods for motion classification,” in IEEE International Conference on Rehabilitation Robotics, 2019, pp. 971–976.

146 [93] [94] [95]

Chapter 5 D.F. Stegeman and H.J. Hermens, “Standards for surface electromyography: the european project (SENIAM),” In Proceedings of the Third General SENIAM Workshop on surface electromyography, 1998, pp. 108–112. B. C. Clark, et al., “Gender differences in skeletal muscle fatigability are related to contraction type and EMG spectral compression,” Journal of Applied Physiology, vol. 95, pp. 2263–2272, 2003. E. C. Hill, et al., “Effect of sex on torque, recovery, EMG, and MMG responses to fatigue,” Journal of Musculoskeletal and Neuronal Interactions, vol. 16, no. 4, pp. 310–317, 2016.

CHAPTER 6 HUMAN-ROBOT INTERACTION STRATEGY IN ROBOTIC-ASSISTED BALANCE REHABILITATION TRAINING JIANCHENG JI, SHUAI GUO, JEFF XI, JIN LIU Jiancheng (Charles) Ji Department of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Shanghai, 200444, China [email protected] Shuai Guo* Department of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Shanghai, 200444, China [email protected] Fengfeng (Jeff) Xi Department of Aerospace Engineering, Ryerson University, 350 Victoria Street, Toronto, ON M5B 2K3, Canada [email protected] Jin Liu Department of Mechatronic Engineering and Automation, Shanghai University, 99 Shangda Road, Shanghai, 200444, China [email protected] Sources of Funding: 1) National Natural Science Foundation of China under Grant 61573234. 2) National Key Research and Development Program of China (Grant No. 2018YFC2001601) Corresponding author: Shuai Guo Tel +86 13818243678 e-mail: [email protected]

148

Chapter 6

Subject Terms: Rehabilitation robot, Human-robot interaction, Passive force field, Walking training.

Abstract Discussed in this chapter is a training method for patient balance rehabilitation based on human-robot interaction. Compared to existing training methods, a passive force method is protective and preventive to avoid secondary damage due to exceeded force. The proposed robot has four passive degrees of freedom (DoF) for the position and orientation of pelvis; meanwhile, one DoF is actively controlled to support the body weight. The passive force field generated by flexible joints provides the force perturbation for a patient to regain his/her balance whilst walking. For proper training, a walking path is planned to obtain an appropriate force perturbation in two approaches. First, the Mobile Pelvic Support Robot (MPSR) is modeled to create a passive force field. Second, a walking path is determined based on given input force from the patient by searching over the passive force field. In addition, several implementation issues are discussed including the determination of a given force and a reference path. Key words: Rehabilitation robot, Human-robot interaction, Passive force field, Walking training.

I. Introduction PELVIC movement is important to normal walking because i) pelvic rotation, pelvic obliquity and lateral displacement of the pelvis are key factors of gait motion [1], and ii) the human body’s center of mass (CoM) is located close to the pelvic center [2]. The primary function of walking is to transfer the body’s CoM from the initial position to the target position [3]. The movement of the pelvis is highly related to energy efficiency of the gait [4]. In addition, the mastery of appropriate weight shifting requires a good dynamic balance during walking to resist falling down [5]. Especially, the small vertical fluctuation of the pelvis plays an important role to decrease mechanical work while walking [6]. The lateral movement of the CoM induces a smooth weight shift between the two legs [7]. Moreover, pelvic rotation and obliquity may influence the upper body [8] and play an important role in generating desired gait patterns and normal locomotion [9, 10]. Multifarious pelvic robot interventions have been researched to correct the pathological movements of the pelvis for paralytic patients through an

Human-Robot Interaction Strategy in Robotic-assisted Balance Rehabilitation Training

149

external force field. The reason is that stroke patients usually show exaggerated lateral displacements of the pelvis, which increases the risk of falling. Olenšek et al. developed a balance assessment robot (BAR) to study balance responses during over-ground walking. The BAR was able to deliver a desired force field to the pelvis by using four linear actuators, and the balance strategies were discussed [11]. The Robotic Gait Rehabilitation Trainer (RGR) was another design by a linear electro-magnetic actuator to target gait deviations for post-stroke patients. Aoyagi et al. designed a five DoFs pneumatic robot to train spinal cord injury (SCI) patients to walk on a treadmill [12]. These robots were capable of generating proper perturbation force through the motors, but the rigid connection may lead to secondary training injury. Kang et al. developed a Tethered Pelvic Assist Device (TPAD) to teach patients how to walk with a specified pelvic trajectory on a treadmill by a force field controller [13]. The perturbation forces implemented at the hip and the knee were generated using active devices. However, existing path track training methods were usually based on active robots, and their effectiveness in passive robots has not been studied in-depth. Passive pelvic support robots have also been designed to study adaptation in human walking with externally applied forces. In [14], the Ego system was designed with a compliant pelvis support mechanism to assist the dynamic balance training during over-ground walking. In [15], a Smart Walking Assistance Device with passive elastic joints was designed to train dynamic balance. Furthermore, a number of robots including the RAGT [16], the robotic walker [17] and the Kineassist [18] were developed with passive devices to study the pelvic robot intervention. Unfortunately, they were not engaged in the path-related pursuit. In the literature, the mechanism of the perturbation-based balance training (PBBT) to improve the balance ability has been explored. It has been observed that without perturbation force training, patients will have abnormal pelvic movements and increase the chance of falling [19]. The research on Assist-as-Needed strategy discussed the influence of force field intervention [20,21]. A few studies investigated dynamic balance response in relation to proximal perturbations in the frontal plane for a neurologically healthy population walking on a treadmill [22,23]. In [24], a computercontrolled pneumatic device was used to study the responses of human hip abductor muscles with lateral balance perturbations. In [25], Vlutters used a two-motor instrumented treadmill to perturb a person’s pelvis. In [26], the Active Pelvis Orthosis was used to supply the assistive strategy for stability recovery after sudden and unexpected slipping-like perturbations. Tania et al. used the Lokomat-applied resistance to improve the performance for

Chapter 6

150

over-ground walking [27]. However, the external perturbations were mostly determined by experiments and the interaction forces were implemented by actuators. Though pelvic support robots have been developed and the mechanism of force generation has been studied, there is little research done in terms of generating a training path under a force field. In this chapter, a method for training path planning is proposed to guide the patient to regain the dynamic balance ability under force. Our robot is designed with springs that can generate a passive force field to provide resistance during balance training. Compared to active force methods, passive force methods are protective and preventive to avoid secondary injury due to the application of force. Our training strategy is to generate a target path for the pelvic center based on a reference path. During training, the system will display the target pelvic path as well as the current pelvic center on the screen to guide the patient to make an effort to follow the target path. Doing so, the system will generate a force to the patient to regain the sense of dynamic balancing. In what follows, the details of this research are provided. Table I List of abbreviations Abbreviation PRY DoF PBBT JAIST BAR ROM s.d. OMP BWS PAM RAGT MPSR COM RGR SCI TPAD BL EF VF GA PT RMS

Description pitch, roll and yaw degree of freedom perturbation-based balance training Japan Advanced Institute of Science and Technology balance assessment robot range of motion Mean and standard deviation omni-directional mobile platform body weight support system pelvic assistive mechanism Robot-assisted gait training Mobile Pelvic Support Robot center of mass Robotic Gait Rehabilitation spinal cord injury Tethered Pelvic Assist Device baseline session elastic force visual feedback Genetic Algorithms post-training session root-mean-square

Human-Robot Interaction Strategy in Robotic-assisted Balance Rehabilitation Training

151

II. System Description Our research is focused on the realization of a pelvic training method with a human-robot intention system. The robot consists of a mobile platform, a body weight support system and a pelvic mechanism. The mobile platform has two passive caster wheels and two omni-directional driving wheels. The body weight support system provides mass-offloading via a synchronous belt and a linear actuator. The pelvic mechanism has four compliant joints to satisfy the pelvic motions.

A. System Description Prior to describing the training path planning method, the design of our Mobile Pelvic Support Robot (MPSR) is elaborated. As shown in Fig.1, the MPSR consists of three main parts: i) an omni-directional mobile platform (OMP); ii) a body weight support system (BWS) and iii) a pelvic assist mechanism (PAM) [28, 31]. The purpose of the OMP is to provide the over-ground mobility. It is designed as a horizontally U-shaped rigid steel frame to provide the patient with a free space of 0.9 m in the lateral or medio direction, and a free space of 1.2 m in the longitudinal or anterior direction. The OMP is supported by two passive castor wheels in the front to enable the rotation of the MPSR and two active omni-directional wheels at the back to provide a pushing force for walking assistance. The entire OMP can rotate 360° during the gait training. The purpose of the BWS is to provide a body weight support. It is designed to have a vertical displacement of 0.6 m for body weight support via a synchronous belt and a linear actuator, which is labeled by Ro in Fig. 1. The PAM is designed to be installed on the BWS. It is of four degreesof-freedom (DoFs). The first DoF labeled by R1 allows the pelvis to have a lateral displacement of 0.15 m. The other three DoFs labeled by R2, R3 and R4 provide three pelvis rotations. While the OMP and the BWS are driven by actuators, the PAM is completely passive with flexibility embedded in joint R1, R2 and R3. All joint flexibility is realized by springs as shown in Fig. 2. As shown in Fig. 2, during training the patient’s pelvis is connected to the MPSR through a harness. For force measurement, two ATI force/torque sensors (F/T Sensor Mini 45) are installed at the connection points on both sides of the pelvis. For motion measurement, a motion capture system called OQUS-700 from QUALISYS Co. Ltd. is used to track the pelvic movements with respect to gait cycles.

152

Chapter 6

To summarize, the force generated by the OMP actuators is employed to assist the over-ground walking. The force generated by the BWS is for the body weight support. The force field that we study for training path planning is generated by the joint flexibility in the PAM. The underlying problem is defined as to finding a path for the pelvic center that would exert a preset force during a specified period of walk training. The described walking training under this study is for treadmill walking, i.e. without the consideration of the OMP.

B. Control method The control strategy of the robot is to assist a patient with the body weight support, and then the patient can move his/her pelvic center to follow the target path. For this purpose, a human-robot intention recognition system, consists of two ATI force/torque sensors and three encoders, as shown in Fig. 3, is designed to obtain the force/torque and encoder signals to monitor the patient’s moving directions and then reveal the position of pelvic center on the screen.

III. Modeling A. Statics Modeling The force field generated by the PAM is modeled using the robot statics. Figure 2 shows the kinematic scheme of the MPSR including joint flexibility. Joint 1 (R1) is a parallelogram loaded with a circular leaf spring of stiffness k1 that is installed in the middle of the first joint. The axis of joint 1 is parallel to the global Z axis. When the parallelogram rotates, it deforms the leaf spring to generate a spring force against the rotation of R1, thereby contributing a force to the pelvis. Joint 2 (R2) is a pendulum mechanism with two linear springs of stiffness k2 on both sides. The axis of joint 2 is parallel to the global Y axis. When R2 rotates, the two springs deform to generate spring forces against the rotation of R2, thereby contributing a force to the pelvis. Joint 3 (R3) is a parallelogram with a linear spring of stiffness k3 on one side. The axis of joint 3 is parallel to the global Z axis. Joint R1, R2 and R3 form a typical Euler angle rotation matrix as RZRYRZ. Encoders are installed in joints 1 to 3 to measure the joint angles under deformation which in turn are used to determine the joint forces under given joint stiffness. Furthermore, two pelvis connection points are modeled as two spherical joints, labeled R4 for both sides.

Human-Robot Interaction Strategy in Robotic-assisted Balance Rehabilitation Training

153

Figure 3 shows the kinematic structure of the PAM, which is equivalent to a serial robot. Considering the traditional robot kinematics, the position vector P of the pelvic center and the rotation matrix R of the pelvis frame are expressed with the global frame as [29] ࡼ = σ௡௜ୀ଴ ࡾ௜ ࡼᇱ௜ ࡾ = ςni=0 ࡾ௜

(1) (2)

where ࡼ'௜ is the position vector expressed in the ith local frame from the ith joint to the (i+1) joint, ࡾ௜ is the rotation matrix from the ith frame to the (i1)th frame. Note that here ࡾ௜ is a function of joint deflection angles, not active joint angles as no actuator is used, that is ࡾ௜ (qi), where qi is the deflection angle of joint i. In general, kinematic equations of a serial robot can be written as X = f(q)

(3)

where q represents a vector of joint deflection angles, function f can be determined from Eq. (1) and Eq. (2), and ‫ = ܆‬ሾx, y, z, Ƚ, Ⱦ, ɀሿ୘ is the vector representing the position (first three elements) of the pelvic center along the global X, Y and Z axis and the orientation (last three elements) of the frame attached to the pelvis about the global X, Y and Z axis. In medical term, three translations are named as: along the X axis called lateral displacement, along the Y axis called walking displacement and along the Z axis called vertical displacement. Three rotations are named as: Ƚ around the X axis called pelvic bending, Ⱦ about the ܻ axis called pelvic obliquity, and ɀ about the Z axis called pelvic rotation. The global coordinate frame is set according to the human body frame. As shown in Fig. 2, the X axis is in the lateral direction, the Y axis is in the forward walking direction and the Z axis is in the upward direction. The XY plane is referred to as the Transverse plane, X-Z plane as the Coronal plane, and Y-Z plane as the Sagittal plane. Following the traditional robot kinematics, the movement increment of the pelvic center, i.e. the end-effector, can be related to joint deformation (increment) in terms of Jacobian J as ο‫ = ܆‬Jο‫ܙ‬

(4)

where ο‫ = ܆‬ሾɁx, Ɂy, Ɂz, ɁȽ, ɁȾ, Ɂɀሿ୘ is the vector representing the position increment (first three elements) of the pelvic center along the global X, Y and Z axis and the orientation increment (last three elements) of the frame

Chapter 6

154

attached to the pelvis about the global X, Y and Z axis. Note that though the initial joint setting mentioned before forms a Euler angle rotation matrix but it can be readily converted a PRY (pitch, roll and yaw) rotation matrix. ο‫ܙ‬ is the vector representing the joint deflections. It should be noted that for the robot with passively flexible joints, if the initial joint angles are set zero, then ο‫ ܙ‬will represent the joint angles q. Therefore, q will be used instead as joint deflection angles, i.e. q = [q1,…,qn]T. Considering duality, the joint forces can be related to the wrenches on the pelvis (end-effector) as ૌ = J் ۴

(5)

୘

where ۴ = ൣf୶ , f୷ , f୸ , m୶ , m୷ , m୸ ൧ is the vector representing the wrenches acting on the pelvis with the first elements for the force and the last elements for the moment, W=[W,…Wn]T is the vector representing the joint torques. For given joint stiffness, joint torques are determined as ૌ = K‫ܙ‬

(6)

where K is the joint stiffness matrix. Combination of Eq. (4) and Eq. (5) yields ۴ = ۹ ீ ο‫܆‬

(7)

where KG is the system stiffness matrix expressed as KG = (JT)-1KJ-1

(8)

Since the robot Jacobian is a function of angles q which are caused by the pelvis movement, KG is essentially a function of ο‫܆‬. In other words, the force field of the PAM is determined by the pelvis movement for given joint stiffness K. Therefore, Eq. (7) represents a passive force model for the PAM with passive flexible joints. A. Inverse Kinematics Prior to proceeding with the path planning, it is also of interest to look at inverse kinematics. For a pure rigid serial robot, if it is required to move from the current configuration X0=f(To) to a required pose X = f(T), Eq. (3) can be expanded using a Taylor series as X = f(To) + JοT + O

(9)

Human-Robot Interaction Strategy in Robotic-assisted Balance Rehabilitation Training

155

where T represents a vector of active joint angles instead of joint deflection angles q, and O represents higher order terms which can be omitted. Then Eq. (9) can be re-arranged as ο‫ =܆‬JοT

(10)

where οX = X - Xo and Xo= f(To). Considering Eq. (10), οT can be solved through matrix inversion as οT=J-1ο‫ ܆‬for square J or generalized matrix inversion οT =J+ο‫ ܆‬for non-square J. The final joint angles can be obtained as Tf =To+οT, after a number of iterations under a specified threshold. This is commonly referred to as the Newton-Raphson method. For serial robots with active joints containing flexibility, i.e. flexible joint robots, kinematic equations can be expressed as X = f(T, q)

(11)

where q as defined before is the vector representing the joint deflection angles. Then, Eq. (11) is expanded as X = f(To) + JοT+ J

(12)

By omitting the higher order terms, Eq. (12) can be re-arranged to the one similar to Eq. (10), but ο‫ =܆‬X – Xo – Jq. For given tip wrench F, the joint torques can be expressed as W =JTF = Kq. This gives ο‫ =܆‬X – Xo – JK-1JTF. In other words, οT can be still solved through matrix inversion as ο‫=ܙ‬J-1 ο‫ ܆‬or οT =J+ ο‫܆‬, but with a correction term pertaining to joint deflections. The final joint angles can still be obtained as Tf = To+ οT, after a number of iterations that meet a specified threshold. Now for serial robots with solely passive joints, kinematic equations are expressed as X = f(q)

(13)

Expansion of Eq. (13) and omission of higher order terms yield οX = Jq

(14a)

where οX as defined before represents the tip movement. In this case, for given tip movement ο‫܆‬, q cannot be directly determined by Eq. (14a) using the Newton-Raphson method. It is related to tip wrench F as W =JTF = Kq. Therefore, Eq. (14a) can be rewritten as

156

Chapter 6

οX = KG-1F

(14b)

where KG is defined in Eq. (8). To this end, one should realize that for a passive flexible joint robot, the tip movement is coupled with the tip force as shown in Eq. (14b). A tip movement will generate a tip force as shown in Eq. (7). In other words, the system under study is a passive compliance system.

IV. Training Method and Experimental Studies A. Training Path Planning After introduction to the compliance model, now the problem of training path planning for the MPSR can be defined. As shown in Fig. 4, let point P (black dot) be the pelvic center on a path to be planned and point C (red dot) be the corresponding point on a reference path. A reference path denoted by P(Xo) is defined as a path for normal people walking without the insertion of force on the pelvis. The path to be planned denoted by P(X), called target path, will generate a force on the pelvis to assist walking. Hence, the displacement between P and C is ο‫܆‬, which is used to generate a force as defined in Eq. (7). Now the problem of training path planning is defined as: given a desired training force Fo and a reference path P(Xo), it is required to find a target path P(X) that will generate a force F as close to Fo as possible. Mathematically, this problem is formulated as an optimization problem given below min |ࡲ െ ࡲ௢ | ௉‫א‬௑

(15)

Subject to: P(X) ‫ܨ‬௧௛௥௘_ீோி ‫ݐ( א ݐ‬ଵ ‫ݐ‬ଶ )

(9) (10) (11)

In the ‘right leg swing with half step’ state, the GRF applied to the right foot is zero. When the swing leg contacts the ground, the wearer will shift from the ‘right leg swing with half step’ state to the ‘double legs stance with right leg leading’ state. In this situation, the GRF applied to the right foot, FGRF_rigth_foot, is larger than zero, as indicated by Eq. (12). ‫ீܨ‬ோி_௥௜௚௛௧_௙௢௢௧ > 0

(12)

During the ‘double legs stance with right leg leading’ state, the wearer moves the two crutches forward one by one, and he/she will also transfer the bodyweight forward to move the system COG forward with the help of the crutches. When the conditions in Eq. (9), Eq. (10), and Eq. (11) are met, the wearer shifts to the ‘left leg swing’ state. Similarly, in the ‘left leg swing’ state, the GRF applied to the left foot is zero. When the left foot contacts the ground, the wearer shifts to the ‘double legs stance with left leg leading’ state, and the GRF applied to the left foot, FGRF_left_foot, is larger than zero, as indicated by Eq. (13). ‫ீܨ‬ோி_௟௘௙௧_௙௢௢௧ > 0

(13)

Similarly, the wearer moves the two crutches forward one by one during the ‘double legs stance with left leg leading’ state. In addition, the wearer transfers the system COG forward. When the conditions in Eq. (9), Eq. (10), and Eq. (11) are met, the wearer shifts to the ‘right leg swing’ state. The walking state shown in Fig. 7 is a continuous periodic motion. During the walking assistance of the paraplegic patients, a shift from one leg swing to the other leg swing without a double legs stance phase (i.e. both feet off the ground) should be prevented. During the double-leg stance phase, there are four supporting points for the human–exoskeleton system, which correspond to a very stable condition. In this situation, the wearer can stop walking and shift to the

Development of a Wearable Exoskeleton Suit for Paraplegic Parents

189

standing state from the walking state. If the wearer does not swing the crutches forward for a period of time, tstanding, larger than the time threshold, tthreshold, the exoskeleton recognises the wearer’s motion intention and stops the continuous walking motion. Then, the human–exoskeleton system shifts to the ‘right leg swing with half step’ state or the ‘left leg swing with half step’ state to terminate the walking motion and finally reach the standing state. Therefore, the following condition should be met if the wearer wants to stop walking and shift to the standing state during the double-leg stance phase. ‫ݐ‬௦௧௔௡ௗ௜௡௚ > ‫ݐ‬௧௛௥௘௦௛௢௟ௗ

(14)

4.4. Step-length real-time adjustment Humans can perform various gait movements at different walking speeds and step-lengths with high ef¿ciency [27]. For the human– exoskeleton system, the step-length is mainly determined by the joint angles in the single-leg stance phase. Thus, the reference trajectories of the robotic exoskeleton during this phase will be modified according to the distance d2 to generate the desired step-length. In this study, two thresholds are predetermined for d2, and three values are predetermined for d1. During every single-step cycle, d1 can be determined using Eq. (15) to make the robotic exoskeleton easier to use and more comfortable for the wearer. ‫݀ כ ܮ‬ଵଵ (݀ଶ < ݀௧௛௥௘௦௛௢௟ௗଵ ) ݀ଵ = ቐ‫݀ כ ܮ‬ଵଶ (݀௧௛௥௘௦௛௢௟ௗଵ ൑ ݀ଶ < ݀௧௛௥௘௦௛௢௟ௗଶ ) ‫݀ כ ܮ‬ଵଷ (݀ଶ ൒ ݀௧௛௥௘௦௛௢௟ௗଶ )

(15)

where L is the length of the wearer’s leg, d11, d12 and d13 are the normalised values of the step-length based on the wearer’s leg length, dthreshold1 and dthreshold2 are the predefined thresholds for the distance d2, and they are determined from preliminary experiments. The angles ș1, ș2, ș31, and ș32 are related by Eq. (16). According to Eq. (1), we know that both the hip and knee joint angles can affect the steplength of the human–exoskeleton system. ߠଶ = ߠଵ + ߠଷଶ െ ߠଷଵ

(16)

A preliminary test was conducted with an optical motion capture system (Vicon Motion Systems Ltd, Oxford, United Kingdom) to obtain the influence of the hip and knee joint angles on the step-length, as shown in Fig. 10. In the test, the subject was a 26-year-old male, with a height of 1.68

190

Chapter 7

m and a weight of 61 kg. The platform was marked at specific distance intervals so that the subject was able to walk with a specific step-length easily. The subject walked with step-lengths of 20, 30, 40, 50, and 60 cm in a normal gait pattern in the test.

Fig. 10. Preliminary test of walking with different step-lengths.

The test results are shown in Fig. 11. It can be seen that with the increase in the step-length, the hip joint angles of the stance leg (at 50% of the gait cycle) and swing leg (at 100% of the gait cycle) increase at the end of the single-leg stance phase, especially for the stance leg while the knee joint angles of the stance leg and swing leg do not change significantly. For the human–exoskeleton system, the effects of the hip and knee joints on the step-length should be similar. As described in the previous section, the steplength can be obtained at the beginning of the double-leg stance phase using Eq. (1). Therefore, to generate the desired step-length for the human– exoskeleton system, the reference joint trajectories for the hip joints of the swing and stance legs will be modified in real-time by a smooth interpolation.

Development of a Wearable Exoskeleton Suit for Paraplegic Parents

191

20

Hip Joint Angle (deg)

15 10 5 0 -5 -10

20 30 40 50 60

-15 -20 -25 0

10

20

30

40

50

60

70

80

90

100

Percentage of Gait Cycle (%)

(a) 60 20 30 40 50 60

Knee Joint Angle (deg)

50 40 30 20 10 0 -10 0

10

20

30

40

50

60

70

80

90

100

Percentage of Gait Cycle (%)

(b) Fig. 11. Hip and knee joint angles during walking with different step-lengths. (a) Hip joint angles. (b) Knee joint angles.

5. Performance evaluation and testing results To evaluate the performance of the wearable exoskeleton suit, paraplegic patients were recruited, and pilot clinical trials were conducted. The control strategy of the robotic exoskeleton is briefly described in this section. Before conducting the trials with paraplegic patients, preliminary training of the paraplegic patients recruited was performed. Then, pilot clinical trials with the robotic exoskeleton were conducted.

Chapter 7

192

5.1. Control strategy of wearable exoskeleton suit When an individual wears a robotic exoskeleton, physical human–robot interactions exist in the human–exoskeleton system. In general, the human– robot interaction can be divided into two types, namely physical and cognitive human–robot interactions [28,29], as shown in Fig. 12. In a robotic exoskeleton, the physical human–robot interaction is related to the interaction force/torque between the exoskeleton and the wearer, which is generated from the exoskeleton actuators and the human musculoskeletal system. On the other hand, the cognitive human–robot interaction is related to the bi-directional cognitions, such as the inference and planning of the exoskeleton and the wearer. Signal Force/torque

Brian

Cognitive human-robot interaction

Exoskeleton controller

Nerve

Exoskeleton Muscle

Sensors

Joints

Human leg Physical human-robot interaction

Fig. 12. Human–exoskeleton interactions.

In this study, the wearable exoskeleton suit is developed to provide motion assistance to paraplegic patients who have lost the motor and sensory functions in the lower extremities. Thus, the control strategies based on the voluntary force/torque applied to the wearable exoskeleton suit from the patient’s lower extremities are not suitable. For the wearable exoskeleton suit, the position-based trajectory tracking control is adopted, and the robotic exoskeleton is controlled to track the reference joint trajectories. The controller architecture designed for the walking assistance

Development of a Wearable Exoskeleton Suit for Paraplegic Parents

193

of the robotic exoskeleton is shown in Fig. 13. It is mainly composed of a high-level controller and a low-level controller.

High-level Control șrh șrk

Trajectory generation

d1

Step-length determination

F d2

Low-level Control +

ǻș

-

PD control

Exoskeleton actuators

IJ

Walking motion intention recognition

Robotic exoskeleton Patient

xCOP ș

Kinematic model

Multiple sensors

F ș

Fig. 13. Controller architecture of the wearable exoskeleton suit. PD is the abbreviation of proportional-derivative.

The function of the high-level controller, which is implemented with a small PC, is to recognise the wearer’s motion intention based on the motion data collected by the multiple sensors. Then, the distance d2 is calculated based on the kinematic model, and the desired step-length d1 for the human– exoskeleton system is determined. Finally, the predetermined reference joint trajectories will be updated in real-time accordingly. The low-level controller is implemented with the microcontrollers. The proportionalderivative control is used to regulate the actuators of the robotic exoskeleton with the motor controllers (Maxon, ESCON 50/5) to follow the references. Then, the assistive force/torque can be generated for the robotic exoskeleton and the wearer, which are well connected through the braces and belts, to perform the desired motions. The kinematic and kinetic data of the human– exoskeleton system are also measured by the multiple sensors through the low-level controller.

5.2. Preliminary training for paraplegic patients In this study, five paraplegic patients were recruited for the pilot clinical trials under the clinical ethical approval granted by the Joint CUHK-NTEC CREC, as shown in Fig. 14. The first (Fig. 14(a)) and second (Fig. 14(b)) patients were SCI patients with a paraplegic level of T4–T5. The third patient (Fig. 14(c)) also suffered a SCI due to a traffic accident; he had a paraplegic level of T5–T6 and was eight years post-injury. The three

Chapter 7

194

subjects had no strength and sensory functions in their lower extremities. The fourth patient (Fig. 14(d)) was also an SCI patient and had a paraplegic level of T6–T7. He suffered from Ewing Sarcoma on his spine when he was 24 years old, which made his lower body paraplegic. He had no strength and a very limited sensory function in his lower extremities. The fifth patient (Fig. 14(e)) was a poliomyelitis patient that contracted the disease when he was two years old. His lower extremities have no strength and a limited sensory function. All the patients recruited had to rely on a wheelchair for mobility in their daily life. The clinical characteristics of the paraplegic patients recruited are listed in Table 2.

Fig. 14. Paraplegic patients recruited in this research.

Table 2. Clinical characteristics of the paraplegic patients Case

Height

Weight

(m)

(kg)

Male

1.70

67

26

Male

1.76

51

Male

1.80

4

24

Male

5

29

Male

Age

Sex

1

43

2 3

No.

Diagnosis

Paralysis

Assistive

Type

Device

SCI

Paraplegia

Wheelchair

63

SCI

Paraplegia

Wheelchair

85

SCI

Paraplegia

Wheelchair

1.72

62

SCI

Paraplegia

Wheelchair

1.65

66

Polio

Paraplegia

Wheelchair

Because the patients were not familiar with the wearable exoskeleton suit, it was necessary to train the patients before conducting the pilot clinical trials. The preliminary training for the subject involved three main parts. Firstly, the patients were trained to be able to keep in the standing posture with external assistance continuously over 20 min so that they could complete the tests, and their cardiovascular system was used to standing. The duration of this part depended on the patients’ physical conditions: it would last longer for patients that had sat on the wheelchair for many years without standing. This preliminary training phase was performed by physical therapists.

Development of a Wearable Exoskeleton Suit for Paraplegic Parents

195

Secondly, the patients were given information on the wearable exoskeleton suit. This stage included two main steps. In the first step, we introduced the robotic exoskeleton to the patients in detail. We told the patients the function of the robotic exoskeleton and how to operate it. Then, we showed them a video describing the pilot study performed with healthy individuals. Finally, we wore the exoskeleton and demonstrated its mechanisms to the patients. In the second step, we measured the patients’ body dimensions and adjusted the exoskeleton dimensions accordingly. Thirdly, we asked patients to use the smart crutches while wearing the robotic exoskeleton to teach them when and how to move the crutches and transfer the system COG with the upper body strength. During the use of the exoskeleton, the wearer needed to exert forces to place the crutches in suitable positions to keep balance and transfer his/her COG position. To stand up or sit down, the wearer should place the crutches in different positions (front and back of the wearer). Thus, the patients were first trained to move the crutches to the back and front while sitting. Then, they attempted to stand up with the assistance of the robotic exoskeleton and a therapist, and they were trained to move the crutches to the back and front in the standing posture. In addition, they were trained to transfer their COG to the right and left, and front and back while standing. Different training sessions were scheduled for each paraplegic patient. After one training session, the first patient did not participate in the following training sessions owing to personal reasons. The second patient also participated in only one training session with the wearable exoskeleton suit because his blood pressure increased abruptly when he stood up with the assistance of the robotic exoskeleton. He needed more preliminary training and was not suitable for the current tests. Patient 3 was busy in his daily life and was not always available for the tests. To date, we have performed two training sessions for him. The fourth patient experienced a contracture and consequently, large tensions in his lower extremities. So far, he has participated in five training sessions. The fifth patient was suitable for the clinical tests and was very willing to participate in the tests. He completed the preliminary training sessions and participated in the pilot clinical trials, as described in the following subsections. Two physical therapists were engaged in all stages of training.

5.3. Pilot clinical trials and results After the preliminary training, pilot clinical trials were conducted to evaluate the effectiveness of the motion assistance of the wearable

196

Chapter 7

exoskeleton suit. The human trials were conducted under the clinical ethical approval that was reviewed and approved by the Joint CUHK-NTEC, and informed consent was obtained from the paraplegic patient before conducting the tests. The pilot clinical trials were divided into three stages depending on the devices used to help the patient to maintain his balance.

Fig. 15. Devices used to help the patients maintain his balance: (a) standard set of parallel bars; (b) hoist; (c) walker; (d) pair of crutches.

During the ¿rst stage, the patient was trained to stand up/sit down and walk with a set of standard parallel bars. In this situation, the patient could hold the parallel bars (Fig. 15(a)) to help support and balance his body weight instead of the crutches. The height of the parallel bars could be adjusted according to the patient’s height. In addition, a hoist (Fig. 15(b)) was connected to the waist of the patient through a slack sling to prevent him from falling down in case of system failure. During the second stage, the patient was trained to use a walker to stand up/sit down and walk with the assistance of the robotic exoskeleton. With the walker (Fig. 15(c)), the patient quickly learned how to maintain balance. After the second stage, the patient was trained to stand up/sit down and walk with a pair of crutches (Fig. 15(d)) to maintain his balance. After ten sessions of training, the patient could walk independently and smoothly with the assistance of the wearable exoskeleton suit using the crutches to maintain balance. Each training session lasted approximately two hours. Then, walking tests with the paraplegic patient were conducted, in which the robotic exoskeleton was operated with the motion intention recognition and real-time step-length adjustment algorithms. In the test, the patient was first in the standing posture with the crutches. After moving the crutches forward, the motion intention was automatically detected based on the motion data measured by the multiple sensors, and the walking motion was initiated by the patient. Then, the human–exoskeleton system turned to walking mode. One physical therapist was engaged in the walking tests. Snapshots of a gait cycle of the human trials are shown in Fig. 16.

Development of a Wearable Exoskeleton Suit for Paraplegic Parents

197

Fig. 16. Snapshots of a gait cycle of the walking tests.

The testing results of the pilot clinical trials in two gait cycles are shown in Fig. 17. The reference and measured joint angles of the exoskeleton knee and hip joints are shown in Fig. 17(a) and Fig. 17(b), respectively. The state transition of the walking tests is shown in Fig. 17(c). The measured steplength of the four single-step cycles is shown in Fig. 17(d). The first and second single-step cycles lasted from 0 to 6 s and from 6 to 12 s, respectively. The third and fourth single-step cycles lasted from 12 to 18.45 s and from 18.45 to 24.75 s, respectively. The first and second single-step periods were both 6 s, and the third and fourth single-step periods were 6.45 and 6.3 s, respectively. The differences were due to the different time used to move the crutches forward by the patient during the different single-step cycles. From Fig. 17(a) and Fig. 17(b), it can be seen that the knee and hip joints of the exoskeleton could follow the references well to realise effective and smooth walking assistance. As shown in Fig. 17(c), with the satisfaction of the state transition conditions, the human–exoskeleton system could shift from one state to the next state smoothly in the walking FSM. In the walking tests, the patient had different step-length during different single-step cycles (Fig. 17(d)). From the first to the fourth single-step cycles, the step-length were 0.25, 0.25, 0.20, and 0.30 m, respectively, which were adjusted according to the patient’s walking conditions. The desired step-length was achieved by modifying the reference trajectories of the hip joints in real-time (Fig. 17(b)). Therefore, with the developed walking assistance of the wearable exoskeleton suit, the patient’s walking motion intention could be estimated automatically, and the patient could walk smoothly with the desired steplength.

Chapter 7

198

Knee Joint Angle (deg)

105 100

Right Ref.

Right Actual

Left Ref.

a

Left Actual

80

60

40

20

0

0

3

6

9

12

15

18

21

24

Time (s) Right Ref.

50

Right Actual

Left Ref.

b

Left Actual

Hip Joint Angle (deg)

40 30 20 10 0 -10 -20

0

3

6

9

12

15

18

21

24

Time (s) 7

c

Walking States

6

5

4

3

0

3

6

9

12

15

18

21

24

Time (s)

0.32

d

Step-Length (m)

0.3

0.28

0.26 0.24

0.22

0.2 0

3

6

9

12

15

18

21

24

Time (s)

Fig. 17. Testing results of the human trials in two gait cycles: (a) knee joint angles; (b) hip joint angles; (c) states in the walking FSM; (d) step-length in the walking tests.

6. Discussion and Conclusion In this Chapter, a wearable exoskeleton suit for the motion assistance of paraplegic patients was developed. The hardware design of the robotic exoskeleton was presented, including the design of the mechanical structure, electronic system, and human–machine interface. In addition, the kinematic model of the human–exoskeleton system was established. With the developed motion intention recognition and real-time step-length adjustment algorithms,

Development of a Wearable Exoskeleton Suit for Paraplegic Parents

199

the wearer’s motion intention could be estimated automatically based on the motion data collected by the multiple sensors. Furthermore, the step-length of the human–exoskeleton system could be adjusted in real-time by modifying the reference joint trajectories. After the preliminary training of the paraplegic patient, pilot clinical trials were conducted. The testing results validated the effectiveness of the wearable exoskeleton suit. During the preliminary training and pilot clinical trials, the patient did not report adverse effects. However, there remain parts of the wearable exoskeleton suit that can be improved and developed. The total weight of the robotic exoskeleton system is too large for paraplegic patients. Accordingly, in future studies, the mechanical structure of the wearable exoskeleton suit will be optimised to reduce the weight. Materials with low density and high stiffness/strength (such as carbon fibre and titanium alloys) as well as some smart materials (such as shape memory alloys and shape memory polymers) will be used for the mechanical structure. In this study, the ankle joints of the wearable exoskeleton suit were passive, and to use them the users should have sufficient upper body strength to maintain balance with a pair of crutches. In the future, active ankle joints will be developed for the robotic exoskeleton to combine ankle joints with the control algorithms and improve the exoskeleton capability to maintain balance. If this is achieved, the wearer’s efforts can be reduced during the use of the wearable exoskeleton suit. Another limitation of this study is that the number of patients involved was small. In the future, more paraplegic patients will be recruited, and more clinical trials will be conducted to further evaluate the performance of the wearable exoskeleton suit through statistical analyses. In addition, safety tests for the long-term wearing of the robotic exoskeleton will be conducted.

Acknowledgements This work was supported by the National Natural Science Foundation of China (Project No. 51805132), the Fundamental Research Funds for the Central Universities (Project No. JZ2019HGTB0084, JZ2018HGBZ0166), and the Innovation and Technology Commission (Project No. ITS/296/14) of the Hong Kong Special Administrative Region, China.

200

Chapter 7

References [1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

United Nations, Department of Economic and Social Affairs, Population Division. World Population Prospects: The 2015 Revision, 2015: http://esa.un.org/unpd/wpp/ (Accessed 15 Nov., 2019) Lee W. S., Cheung W. H., Qin L., Tang N., and Leung K. S., “Age-associated decrease of type IIA/B human skeletal muscle fibers,” Clinical Orthopaedics and Related Research, vol. 450, pp. 231-237, 2006. Cheung W. H., Lee W. S., Qin L., Tang N., Huang W. Y., and Leung K. S., “Type IIB human skeletal muscle fibers positively correlate with bone mineral density irrespective to age,” Chinese Medical Journal (English Edition), vol. 123, no. 21, pp. 3009-3014, 2010. World Health Organization, Media Center, Spinal cord injury, 2013: http://www.who.int/mediacentre/factsheets/fs384/en/ (Accessed 15 Dec., 2019) Kubota S., Nakata Y., Eguchi K., Kawamoto H., Kamibayashi K., Sakane M., Sankai Y., and Ochiai N., “Feasibility of rehabilitation training with a newly developed wearable robot for patients with limited mobility,” Archives of Physical Medicine and Rehabilitation, vol. 94, no. 6, pp. 1080-1087, 2013. Ueba T, Hamada O., Ogata T., Inoue T., Shiota E., and Sankai Y., “Feasibility and safety of acute phase rehabilitation after stroke using the hybrid assistive limb robot suit,” Neurologia Medico-Chirurgica, vol. 53, no. 5, pp. 287-290, 2013. McKinley W. O., Jackson A. B., Cardenas D. D., and DeVivo M. J., “Longterm medical complications after traumatic spinal cord injury: a regional model systems analysis,” Archives of Physical Medicine and Rehabilitation, vol. 80, no. 11, pp. 1402-1410, 1999. Brown-Triolo D. L., Roach M. J., Nelson K., and Triolo R. J., “Consumer perspectives on mobility: implications for neuroprosthesis design,” Journal of Rehabilitation Research and Development, vol. 39, no. 6, pp. 659-669, 2002. Scivoletto G., Mancini M., Fiorelli E., Morganti B., and Molinari M., “A prototype of an adjustable advanced reciprocating gait orthosis (ARGO) for spinal cord injury (SCI),” Spinal Cord, vol. 41, no. 3, pp. 187-191, 2003. Shamaei K., Napolitano P. C., and Dollar A. M., “Design and functional evaluation of a quasi-passive compliant stance control knee-ankle-foot orthosis,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 22, no. 2, pp. 258-268, 2014. Chen B., Ma H., Qin L. Y., Gao F., Chan K. M., Law S. W., Qin L., and Liao W. H., “Recent developments and challenges of lower extremity exoskeletons,” Journal of Orthopaedic Translation, vol. 5, pp. 26-37, 2016.

Development of a Wearable Exoskeleton Suit for Paraplegic Parents [12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

201

Chen B., Zhong C. H., Zhao X., Ma H., Qin L., and Liao W. H., “Reference joint trajectories generation of CUHK-EXO Exoskeleton for system balance in walking assistance,” IEEE Access, vol. 7, no. 1, pp. 33809-33821, 2019. Chen Q., Zi B., Sun Z., Li Y., Xu Q. S., “Design and development of a new cable-driven parallel robot for waist rehabilitation,” IEEE/ASME Transactions on Mechatronics, DOI: 10.1109/TMECH.2019.2917294, 2020 Farris R. J., Quintero H. A., Murray S. A., Ha K. H., Hartigan C, and Goldfarb M., “A preliminary assessment of legged mobility provided by a lower limb exoskeleton for persons with paraplegia,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 22, no. 3, pp. 482-490, 2014. Onen U., Botsali F. M., Kalyoncu M., TÕnkÕr M., YÕlmaz N., and Sahin Y., “Design and actuator selection of a lower extremity exoskeleton,” IEEE/ASME Transactions on Mechatronics, vol. 19, no. 2, pp. 623-632, 2014. Wang S., Wang L., Meijneke C., van Asseldonk E., Hoellinger, T., Cheron G., Ivanenko Y., Scaleia V. L., Sylos-Labini F., Molinari M., Tamburella, F., Pisotta I., Thorsteinsson F., Ilzkovitz M., Gancet J., Nevatia Y., Hauffe R., Zanow F., and van der Kooij H., “Design and control of the MINDWALKER exoskeleton,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 23, no. 2, pp. 277-286, 2015. Talaty M., Esquenazi A., and Briceño J. E., “Differentiating ability in users of the ReWalkTM powered exoskeleton: An analysis of walking kinematics,” In: Proceedings of IEEE International Conference on Rehabilitation Robotics (ICORR), Seattle, USA, 2013, pp. 1-5. Tsukahara A., Hasegawa Y., Eguchi K., and Sankai Y., “Restoration of gait for spinal cord injury patients using HAL with intention estimator for preferable swing speed,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 23, no. 2, pp. 308-318, 2015. Tsukahara A., Hasegawa Y., and Sankai Y., “Gait support for complete spinal cord injury patient by synchronized leg-swing with HAL,” In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA, 2011, pp. 1737-1742. Li L., Hoon K. H., Tow A., Lim P. H., and Low K. H., “Design and control of robotic exoskeleton with balance stabilizer mechanism,” In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Hamburg, Germany, 2015, pp. 3817-3823. Collinger J. L., Boninger M. L., Bruns T. M., Curley K., Wang W., and Weber D. J., “Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury,” The Journal of Rehabilitation Research and Development, vol. 50, no. 2, pp. 145-160, 2013. Chen B., Zhao X., Ma H., Qin L., and Liao W. H., “Design and characterization of a magneto-rheological series elastic actuator for a lower extremity exoskeleton,” Smart Materials and Structures, vol. 26, no. 10, pp. 1-17, 2017.

202 [23] [24]

[25] [26]

[27]

[28]

[29]

Chapter 7 Pons J. L., “Wearable robots: biomechatronic exoskeletons,” New Jersey: John Wiley & Sons, Inc., ch. 3, pp. 47-85, 2008. Onen U., Botsali F., M, Kalyoncu M., TÕnkÕr M., YÕlmaz N., and Sahin Y., “Design and actuator selection of a lower extremity exoskeleton,” IEEE/ASME Transactions on Mechatronics, vol. 19, no. 2, pp. 623-632, 2014. Burnfield M., “Gait analysis: normal and pathological function,” Journal of Sports Science and Medicine, vol. 9, no. 2, pp. 353, 2010. Chen B., Zi B., Wang Z. Y., Qin L., and Liao W. H., “Knee exoskeletons for gait rehabilitation and human performance augmentation: A state-of-the-art,” Mechanism and Machine Theory, vol. 134, pp. 499-511, 2019. Huang Y., Vanderborght B., Van Ham R., Wang Q. N., Van Damme M., Xie G. M., and Lefeber D., “Step length and velocity control of a dynamic bipedal walking robot with adaptable compliant joints,” IEEE/ASME Transactions on Mechatronics, vol. 18, no. 2, pp. 598-611, 2013. Bueno L., Brunetti F., Frizera A., and Pons J. L., “Human-robot cognitive interaction, in: Pons J. L. (Ed.), Wearable robots: biomechatronic exoskeletons,” John Wiley & Sons, pp. 87-125, 2008. Rocon E., Ruiz A. F., Raya R., Schiele A., and Pons J. L., “Human-robot physical interaction, in: Pons, J. L. (Ed.), Wearable robots: biomechatronic exoskeletons,” John Wiley & Sons, pp. 127-163, 2008.