Tactile Perception by Electrovibration [1st ed.] 9783030522513, 9783030522520

Citation preview

Springer Series on Touch and Haptic Systems

Yasemin Vardar

Tactile Perception by Electrovibration

Springer Series on Touch and Haptic Systems Series Editors Manuel Ferre, Universidad Politécnica de Madrid, Madrid, Spain Marc Ernst, Ulm University, Ulm, Germany Alan Wing, University of Birmingham, Birmingham, UK Editorial Board Members Carlo A. Avizzano, Scuola Superiore Sant’Anna, Pisa, Italy Massimo Bergamasco, Scuola Superiore Sant’Anna, Pisa, Italy Antonio Bicchi, University of Pisa, Pisa, Italy Jan van Erp, University of Twente, Enschede, The Netherlands Matthias Harders, University of Innsbruck, Innsbruck, Austria William S. Harwin, University of Reading, Reading, UK Vincent Hayward, Sorbonne Université, Paris, France Juan M. Ibarra, Cinvestav, Mexico City, Mexico Astrid M. L. Kappers, Eindhoven University of Technology, Eindhoven, The Netherlands Miguel A. Otaduy, Universidad Rey Juan Carlos, Madrid, Spain Angelika Peer, Libera Università di Bolzano, Bolzano, Italy Jerome Perret, Haption, Soulgé-sur-Ouette, France Domenico Prattichizzo, University of Siena, Siena, Italy Jee-Hwan Ryu, Korea Advanced Institute of Science and Technology, Daejeon, Korea (Republic of) Jean-Louis Thonnard, Université Catholique de Louvain, Ottignies-Louvain-la-Neuve, Belgium Yoshihiro Tanaka, Nagoya Institute of Technology, Nagoya, Japan Dangxiao Wang, Beihang University, Beijing, China Yuru Zhang, Beihang University, Beijing, China

More information about this series at http://www.springer.com/series/8786

Yasemin Vardar

Tactile Perception by Electrovibration

Yasemin Vardar Mechanical Engineering Koç University Istanbul, Turkey

ISSN 2192-2977 ISSN 2192-2985 (electronic) Springer Series on Touch and Haptic Systems ISBN 978-3-030-52251-3 ISBN 978-3-030-52252-0 (eBook) https://doi.org/10.1007/978-3-030-52252-0 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Series Editors’ Foreword

This volume is the 17th in the ‘Springer Series on Touch and Haptic Systems’, which is published in collaboration with Springer and the EuroHaptics Society. Touch screens with electrovibration modulate the friction between the finger and a capacitive screen by electrostatic forces, which are generated by applying an alternating voltage to the conductive layer of the screen. Tactile Perception by Electrovibration extends our understanding of human tactile perception by electrovibration and provides useful information for improving the design of tactile feedback on touch screens. The chapters of this book include relevant advances on tactile interaction. Two fundamental problems are addressed: the first relates to how frequency of input voltage affects finger tactile perception, with highly relevant results obtained at frequencies above 60 Hz, and the second defines how interference between signals affects the perceived quality of the tactile stimulus. Such interference is defined as tactile masking. The author shows how specific perceptual deficits can be produced, including effects on detection thresholds, and the localization or identification of tactile stimuli. This book originated from the thesis of Dr Yasemin Vardar, who received the EuroHaptics award for the Best PhD Thesis in 2018. This award recognizes the relevance of this work and provides reliable qualitative research tools that benefit the whole haptics community. June 2020

Manuel Ferre Marc Ernst Alan Wing

v

Preface

Our sense of touch enables us to grasp objects, manipulate them, and determine their geometric and material properties. For example, when we hold a baby’s hand, we immediately feel its small size, cute shape, and warm, smooth, and soft skin. Now, imagine you are touching a photo of the same baby hand on the screen of your smartphone; every depicted sensation feels cold and hard as glass. However, you can easily take a photo of this baby’s hand and display it on your smartphone. Or, you can record its laughter and play the sound. Unfortunately, the technology of delivering haptic cues on current electronic devices is still primitive and limited to simple vibrations. One promising approach to generate compelling tactile sensations on touchscreens is electrovibration. Although this technique has been around for 50 years, displaying realistic tactile effects using this technique is still an open research problem. The main challenge is the limited knowledge of the underlying physical mechanism of electrovibration and its tactile perception. In this book, we tackle this problem by simultaneously investigating both physical and perceptual aspects of electrovibration. Based on our results, we also propose new methods and insights to display realistic textures using electrovibration. This book presents my contributions to the field of haptics during my doctoral studies. Chapters 3 and 4 present the studies I conducted under the guidance of my doctoral advisors Cagatay Basdogan and Burak Güçlü. Chapter 5 covers the study that Tamara Fiedler and I conducted, while she was an intern student with me. Chapter 6 presents a collaborative study that I conducted together with Aykut Isleyen and M. Khurram Saleem under the guidance of Cagatay Basdogan. Hence, due to the collaborative nature of this work, I use the pronoun “we” throughout the book. Stuttgart, Germany May, 2020

Yasemin Vardar

vii

Acknowledgements

I would like to thank, first and foremost, my advisors Prof. Dr. Cagatay Basdogan and Prof. Dr. Burak Güçlü for their guidance throughout my Ph.D. study. They provided me a great research environment that motivated me to become an independent, curious, analytical, and creative researcher. I have been nourished by their different approaches and had an opportunity to develop myself both as an engineer and a scientist. They spent an enormous amount of time and effort to raise me. I will never forget the things they have done for me. I would like to thank Cagatay for teaching me to seek always the best. When I think about my past mistakes, I admire the patience and faith he showed for me. He always encouraged me to develop myself beyond my boundaries. Besides, he showed great patience to my emotional breakdowns and high level of stress. I would like to thank Burak for teaching me how to be a good scientist. Thanks to him, I tasted the joy of working on fundamental science. I would like to thank Prof. Dr. Edward Colgate, Prof. Dr. Hong Tan, Dr. Ipek Basdogan, and Dr. Evren Samur for agreeing to be part of my thesis committee. I would like to thank them for spending their precious time to listen to my presentation and read my thesis. I am very fortunate to benefit from their suggestions and perspectives. Also, I would like to thank Prof. Dr. Ozgur Birer for his valuable comments and discussions in the early phase of this study. I would like to thank Dr. Katherine J. Kuchenbecker for her support during the editing process of this monograph. Also, I feel lucky to be on her team as a postdoctoral researcher. I am amazed by her brilliance, energy, vision, and compassion. Words are not enough to express my gratitude for her. I have never had such a role model in my life before; I will do my best to inspire my future students as she inspired me. I would like to thank Gokhan Serhat for his endless support in this adventure. He helped me find my path, happiness, joy, and love again. He has always held my hand whenever I fell down and cheered me up whenever I was sad. He always motivated me to enjoy life, nature, sports, and science. Also, I gained a lot from his extraordinary intelligence during our discussions about my research.

ix

x

Acknowledgements

I would like to thank Ozan Caldiran for his valuable comments and fruitful discussions during my study. His comments about my very first experiments led me to dig in more and understand the physical mechanism behind electrovibration. He was always the first volunteer for my experiments without any complaints. I wonder if I would be at this stage without his companionship. I would like to thank Utku Boz, M. Khurram Saleem, Aykut Isleyen, Amir Reza Aghakhani, Yusuf Aydin, and Omer Sirin for their valuable comments and technical assistance during the preparation of hardware and software needs of my experimental apparatus. Thanks to them, I got critical answers quickly whenever I was stuck, and moved on. They always answered my questions, even if my questions were ridiculous, with great patience. I was very lucky to have such good colleagues whom I could ask for help anytime. I would like to thank Senem Ezgi Emgin and Enes Selman Ege for their initial help and support. They introduced electrovibration to me and provided a quick start for my Ph.D. study. I learned a lot from them and developed my study based on the foundation they provided me. I would like to thank Tamara Fiedler and Aykut Isleyen for being collaborators in my research. When we first met, they were like the seeds I planted. Now, they are like young trees with lots of fruits and flowers. I gained a lot from working together in terms of developing research and management skills. I would like to thank all subjects who participated in my experiments. They spent enormous time and showed great patience to complete my study voluntarily. I will never forget their self-sacrifice. I would like to thank all teachers who educated me since preliminary school. I would not be at this stage without their foundation and efforts. I will never forget any of their contribution to my personality, perspective, and scientific knowledge. I would like to thank my family. Mom and dad have raised me as a healthy, strong, ambitious, curious, and independent woman. They always supported me to follow my dreams and taught me never to give up. My sisters Yesim and Ozlem always listened to me and showed the best path when I lost my way. My grandfather and grandmother always wished the best things for me and made me feel proud of myself during my long studentship period. Although there are kilometers between us, I have always felt my family standing right next to me. It was always such a relief to have people on my back regardless of the difficulty of any situation. I would like to thank my friends Buket Baylan, Utku Boz, Mehmet Murat Gozum, Ozan Caldiran, Serena Muratcioglu, Yusuf Aydin, Sinemis Temel, Amir Reza Aghakhani, Isil Koyuncu, Bugra Bayik, Tagra Bayik, Omer Sirin, Mehmet Ayyildiz, Bilgesu Erdogan, Efe Elbeyli, Ipek Karakus, and Yavuzer Karakus for the great friendship. We shared laughter, joy, pain, tears, and frustration together. They were ears when I needed to talk, shoulders when I needed to cry, hands when I needed help, tongues when I needed to be criticized. I feel very lucky to have you in my life. Moreover, I would like to thank my other colleagues in RML lab:

Acknowledgements

xi

Cigil Ece Madan, Mohammad Ansarin, Soner Cinoglu, Bushra Sadia, Zaid Rassim Mohammed al-saadi, Yahya Mohey Hamad Al-qaysi, Ayberk Sadic, Utku Erdem, and Milad Jamalzadeh. The Scientific and Technological Research Council of Turkey (TUBITAK) supported this work under the Student Fellowship Program BIDEB-2211.

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 3 4 5

2

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Human Tactile Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Fingerpad and Skin Anatomy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Sensory Receptors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3 Tactile Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Overview of Surface Haptic Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Mechanical Vibration Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Electrotactile Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Surface Shape Changing Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4 Thermal Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.5 Friction Modulation Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Electrovibration for Tactile Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Foundation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.4 Tactile Rendering and User Interface Design . . . . . . . . . . . . . . . . . 2.3.5 Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.6 Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.7 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7 7 7 9 10 18 18 19 20 20 21 21 21 24 24 28 29 32 33 36

3

Effect of Waveform on Tactile Perception by Electrovibration . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Waveform Analysis of Electrovibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Materials and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Experiment 1: Psychophysical Experiments . . . . . . . . . . . . . . . . . . 3.3.2 Experiment 2: Force & Acceleration Measurements . . . . . . . . .

43 43 45 49 49 52 xiii

xiv

Contents

3.4

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Results of Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Results of Experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

56 56 57 63 65 66

4

Effect of Masking on Tactile Perception by Electrovibration . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Threshold Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 Sharpness Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Results of Threshold Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Results of Sharpness Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

69 69 72 72 72 72 78 81 81 83 85 89 90

5

Texture Rendering by Electrovibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Texture Data Recording . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Texture Reproduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Texture Rendering on Electrostatic Displays . . . . . . . . . . . . . . . . . 5.2.4 Texture Compression. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.5 Psychophysical Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

93 93 94 94 95 96 99 101 104 105 106 106

6

Roughness Perception of Virtual Gratings by Electrovibration . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Stimuli . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.3 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.4 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.5 Data Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

109 109 110 110 110 113 113 114 114 119 120 121

Contents

7

Conclusion and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 Human Tactile Perception of Electrovibration . . . . . . . . . . . . . . . . 7.2.2 Texture Rendering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 Design of Tactile Displays and Applications . . . . . . . . . . . . . . . . . 7.3 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 Masking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.3 Multi-finger Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.4 Multi-modal Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.5 Device Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.6 Texture and Shape Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xv

123 123 124 124 125 125 127 127 128 128 128 128 129 129

A Supplementary Materials for Chap. 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 A.1 Non-averaged Results Across Different Scan Speeds . . . . . . . . . . . . . . . . 131 B Supplementary Materials for Chap. 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 B.1 Subject-Wise Threshold Shifts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Chapter 1

Introduction

Abstract This chapter introduces the topic, scope, and goal of this monograph, and it presents an overview of the book chapters. Keywords Tactile · Feedback · Surface haptics · Electrovibration · Fingertip

1.1 Introduction Touch screens are an indispensable part of our lives. They are used in several electronic devices such as smartphones, tablet computers, smart TVs, kiosks, and digital information panels. The usage of touch screens simplifies the design of the electronic devices into one piece of equipment and ease the tailoring of their user interfaces. However, our interactions with current touchscreens mainly involve auditory and visual channels and lack tactile feedback. Tactile feedback can, for example, improve user performance during gesture interactions with digital controls such as keyboards, sliders, and knobs. Receiving a tactile confirmation when you press a digital key or feeling the detents of a digital knob while rotating it may help to user focus on the task rather than the controller itself. Moreover, providing realistic tactile feedback can enhance user experience and human perception in interactive applications such as online shopping, digital games, and education. For example, feeling the simulated texture of a jean before purchasing it from the Internet would certainly be more motivating for shoppers. Furthermore, designing user interfaces for visually impaired so that they can feel the shapes of digital objects and appreciate graphical information on touch screens is another motivating and exciting application (see Fig. 1.1 for more possible applications of tactile feedback on touchscreens). Due to the popularity of touchscreens and the possibility of designing exciting tactile applications on them, surface haptics has recently gained a growing interest by researchers, tech companies, and startups. The current studies on surface haptics have mainly focused on displaying efficient tactile feedback to a user as the user moves her/his finger on the screen. One approach to generating tactile feedback on © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 Y. Vardar, Tactile Perception by Electrovibration, Springer Series on Touch and Haptic Systems, https://doi.org/10.1007/978-3-030-52252-0_1

1

2

Fig. 1.1 Possible applications of tactile feedback on touchscreens

1 Introduction

1.2 Scope

3

Fig. 1.2 Our interactions with current and future touchscreens. Now, when we move our finger on an image displayed on a touchscreen, we only feel glass. In the future, we will be able to feel images displayed on the screen by modulating the friction forces between our fingers and the screen

touchscreens is electrovibration. In this technique, tactile sensations are created by modulating the friction between user fingertip and touchscreen (see Fig. 1.2). The friction force is altered via electrostatic forces generated by applying an alternating voltage signal to the conductive layer of a capacitive touch screen [1]. By changing the amplitude [1], frequency [3, 8] and waveform [5, 6] of the input voltage, it is possible to render textures [2, 7] and even 3D shapes [4] on touch screens.

1.2 Scope Although electrovibration can potentially provide substantial tactile feedback on touchscreens, realistically generating these effects is challenging and still a major research question. There is still not any comprehensive model or method that links to input voltage signal parameters to a particular tactile sensation. The goal of this work is to better understand the underlying physics behind the electrostatic forces developed between a finger and a touchscreen first and then investigate the effects of input voltage signal and the electromechanical properties of finger-touchscreen interaction on the perceived sensation. It aims to develop knowledge on how to design input signals to create realistic tactile sensations on touch screens. This book simultaneously investigates both the physical and perceptual aspects of the electrovibration to achieve this goal. It blends the available knowledge on

4

1 Introduction

electrical and mechanical properties of finger-touchscreen interaction and human tactile perception with the results of new psychophysical experiments conducted in tandem with physical measurements. Based on this combined theoretical and experimental knowledge, it proposes new methods and insights on generating realistic haptic effects such as textures and edges on the electrovibration displays. Besides, it presents the state of the art research on electrovibration for tactile displays and discusses future work on this field.

1.3 Outline This book is presented in seven chapters including this introduction and organized as follows. Chapter 2 reviews the current literature about the human touch, which falls into the scope of this monograph. First, it gives a brief introduction of the finger anatomy and human tactile perception. Then, it presents an overview of surface haptics displays. Finally, it summarizes the state of the art research on electrovibration for surface haptic displays. Chapter 3 investigates the effect of input voltage properties on our tactile perception of electrovibration on touch screens. First, a theoretical model that explains the detection mechanism of electrovibration stimuli is hypothesized. Then this hypothesis is supported by presenting the results of a simulation based on the electrical model of finger-touchscreen interaction and two experiments. The first experiment focuses on obtaining human psychophysical detection thresholds of electrovibration stimuli generated by sinusoidal and square voltages at various fundamental frequencies. The second experiment, on the other hand, focuses on measuring contact force and accelerations acting on the index fingers of the subjects, when the touch screen is actuated at the threshold voltages estimated in the first experiment. Chapter 4 discusses the effect of masking on the tactile perception of electrovibration displayed on touch screens by presenting two psychophysical experiments. The first experiment aims to determine the influence of masking amplitude and type on the detection thresholds of electrovibration stimuli generated by a sinusoidal voltage. The second experiment, on the other hand, investigates the effect of tactile masking on our sharpness perception of the virtual edges generated by electrovibration. Chapter 5 proposes a new data-based texture rendering approach based on the methodologies presented in Chaps. 3 and 4. The main novelty of this method is its ability to compress the tactile data. This chapter first explains how to collect tactile data from real textures and how to analyze this data for rendering. Then it explains the details and of the rendering method. Finally, it discusses the method’s compression feasibility by presenting a psychophysical experiment. Chapter 6 focuses on human roughness perception of virtual gratings displayed by electrovibration. It first explains how to generate programmable virtual gratings by electrovibration. Then, it presents a psychophysical experiment conducted in

References

5

tandem with force measurements. Finally, it compares the roughness perception of virtual textures with the real ones. Chapter 7 concludes the book, summarizes the outcomes and contributions, and suggests possible future directions.

References 1. Bau, O., Poupyrev, I., Israr, A., Harrison, C.: Teslatouch: electrovibration for touch surfaces. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (UIST), pp. 283–292 (2010) 2. Ilkhani, G., Aziziaghdam, M., Samur, E.: Data-driven texture rendering on an electrostatic tactile display. Int. J. Hum. Comput. Interact. 33(9), 756–770 (2017) 3. Meyer, D.J., Peshkin, M.A., Colgate, J.E.: Fingertip friction modulation due to electrostatic attraction. In: Proceedings of the IEEE World Haptics Conference 2013, pp. 43–48 (2013) 4. Osgouei, R.H., Kim, J.R., Choi, S.: Improving 3D shape recognition with electrostatic friction display. IEEE Trans. Haptic 10(4): 533–544 (2017) 5. Vardar, Y., Güçlü, B., Basdogan, C.: Effect of waveform in haptic perception of electrovibration on touchscreens. In: Haptics: Perception, Devices, Control, and Applications: 10th International Conference, EuroHaptics 2016, London, 4–7 July 2016, Proceedings, Part I, pp. 190–203 (2016) 6. Vardar, Y., Güçlü, B., Basdogan, C.: Effect of waveform on tactile perception by electrovibration displayed on touch screens. IEEE Trans. Haptics 10(4), 488–499 (2017) ˙sleyen, A., Saleem, M., Basdogan, C.: Roughness perception of virtual textures 7. Vardar, Y., I¸ displayed by electrovibration on touch screens. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 263–268 (2017) 8. Vezzoli, E., Amberg, M., Giraud, F., Lemaire-Semail, B.: Electrovibration modeling analysis. In: Haptics: Neuroscience, Devices, Modeling, and Applications, pp. 369–376. Springer, Berlin/Heidelberg (2014)

Chapter 2

Background

Abstract Understanding the characteristics of human tactile sensing is essential to design effective tactile displays. Hence, this chapter first reviews the current literature about the human touch, which falls into the scope of this monograph. It gives a brief introduction of the fingerpad and skin anatomy, sensory receptors, and human tactile perception. Afterward, it presents an overview of surface haptics displays explaining different actuator types. Finally, it summarizes the state of the art research on electrovibration for surface haptic displays. Keywords Human tactile sensing · Fingerpad · Human skin · Sensory receptors · Surface haptic displays · Electrovibration

2.1 Human Tactile Sensing Our sense of touch is enabled through our sensory organ skin. On our body, there are two types of skin hairy and glabrous. These two skin types have different anatomy and mechanical properties. As this book focuses on surface haptics with electrovibration, this section will cover the anatomy of fingerpad and glabrous skin and its sensory receptors. The anatomy of hairy skin and tactile perception by other parts of the body is out of the scope of this book.

2.1.1 Fingerpad and Skin Anatomy The human finger pad consists of glabrous skin and bone and muscles attached to that by connective fibers (see Fig. 2.1). The glabrous skin has a multi-layer structure, which is aligned as the epidermis, dermis, and subcutaneous tissue, from outside to inside [11, 63, 107]. The epidermis protects the inner tissues from infection, chemical and mechanical stresses. The outermost layer of the epidermis is called stratum corneum, which © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 Y. Vardar, Tactile Perception by Electrovibration, Springer Series on Touch and Haptic Systems, https://doi.org/10.1007/978-3-030-52252-0_2

7

8

2 Background

Tendons

Bone

Subcutaneous tissue

Dermis

Epidermis Stratum Corneum

Epidermis Stratum corneum

Epidermis

Dermis

Merkel cells

Sweat gland

Meissener corpuscle

Ruffini endings

Free nerve ending Pacinian corpuscle

Fig. 2.1 Human finger anatomy

consists of approximately 15–20 layers of dead cells. The fingerprints are also in this layer. The inner part of the epidermis includes keratinocytes and keratin. The dermis is composed of blood vessels, sensory nerves, and sweat glands, and lipid glands. Below the epidermis and dermis, there is the subcutaneous tissue that contains blood

2.1 Human Tactile Sensing

9

Table 2.1 Mechanical properties of human finger skin Elastic modulus (MPa) Poisson’s ratio

Stratum corneum 1.00 0.3

Epidermis 0.136 0.3

Dermis 0.080 0.48

Subcutaneous tissue 0.034 0.48

vessels, fat tissues, connective tissues, and the axons of the sensory neurons. The mechanical properties of these skin layers are nonlinear and vary among people (see Table 2.1 for nominal values taken from the literature [118]).

2.1.2 Sensory Receptors This section summarizes the sensory receptors on the human glabrous skin.

2.1.2.1

Mechanoreceptors

These receptors perceive mechanical stimuli such as pressure, vibration, and texture. There are four different mechanoreceptors in glabrous skin: Merkel cells, Ruffini endings, Meissener corpuscles, and Pacinian corpuscle (see Fig. 2.1). These receptors are categorized based on the nerve fibers that they are connected: (fastadapting (FA) or slowly-adapting (SA)) and the size of the receptive fields (small (I) or large (II)) of these fibers [35, 102]. The fast-adapting nerve fibers produce neural spikes only at the beginning and the end of the stimuli. On the contrary, the slowly-adapting nerve fibers produce neural spikes during the whole stimulation period. The receptive fields are related to spatial acuity: the fibers which have small receptive fields respond with a high spatial acuity. The categorization of the mechanoreceptors based on their adaptation rate and receptive fields is shown in Table 2.2. Each receptor is sensitive to different type of stimuli, for example, Pacinian receptors are most sensitive to vibrations, whereas Merkel receptors are most sensitive to pressure.

2.1.2.2

Thermoreception

Thermoreceptors perceive temperature stimuli and have two different categories: cold and hot receptors. Krause end bulbs and free nerve endings are known as cold receptors, whereas Ruffini endings and free nerve endings can detect warmth. The process of temperature sensation is still investigated.

10

2 Background

Table 2.2 The four mechanoreceptors and their response sensitivity [102] Mechanoreceptor

Adaptation rate

Receptive field

Merkel disks

Slow (SA)

Small (I)

Ruffini end organ

Slow (SA)

Large (II)

Meissener corpuscle

Fast (FA)

Small (I)

Pacinian corpuscle

Fast (FA)

Large (II)

2.1.2.3

Sensitivity Pressure: deformation in spatial structure (2–16 Hz) Stretch: lateral deformation (100–500 Hz) Flutter: light touch, movement or deformation changes (2–40 Hz) Vibration: fine textures, movement or deformation changes (40–500 Hz)

Pain Receptors

These receptors detect mechanical, thermal, or chemical stimuli that can cause damage to the skin. They are called nociceptors and usually in the form of free nerve endings. Some of them cause a dull pain to avoid touching that body part until it is healed.

2.1.3 Tactile Perception Our skin can sense mechanical stimuli (vibrotactile), thermal stimuli, and pain through sensory receptors, as explained in Sect. 2.1.2. This section will summarize the topics for human vibrotactile (mechanical) stimuli for glabrous skin relevant to the scope of this book. It is important to note that the findings presented here except for the texture perception part are obtained by passive touch, where the hand and finger is stationary.

2.1.3.1

Detection of a Vibrotactile Stimulus

The current theories on vibrotactile perception are based on the four-channel theory [27, 28, 31, 34, 43, 48]. According to this theory, the mechanoreceptors in the glabrous skin are stimulated by mechanical deformation. Then, each fires an electrical pulse, which is transmitted to the celebral cortex through a nerve [6]. Each type of mechanoreceptor mediates one psychophysical channel. Each channel is sensitive to different input frequencies, which partially overlap (see Fig. 2.2a). The P (Pacinian) channel is mediated by Pacinian receptors and is most sensitive in the range of 40–500 Hz. Its sensitivity follows a U-shaped trend with the lowest value approximately at 250 Hz. NPII channel shows a similar sensitivity region with P channel, however, its sensitivity is much lower than P channel if the area of the

Threshold Displacement (dB re 1.0 micrometer peak )

2.1 Human Tactile Sensing

11

2

2.9 cm contactor 60

RA I

SA II

30

SA I 0

P -30 10

0.1

1000

Stimulus Frequency (Hz)

Threshold Displacement (dB re 1.0 micrometer peak )

a) 2

0.008 cm contactor 60

RA I

SA II P

30

SA I

0

-30 0.1

10

1000

Stimulus Frequency (Hz) b) Fig. 2.2 The sensitive regions of four psychophysical channels that were determined by detection threshold experiments when the mechanical stimuli were applied to the glabrous skin of the hands of the participants through (a) a large (2.9 cm2 ) and (b) a small (0.008 cm2 ) contactor [33, 35]

12

2 Background

Mechanical Shaker

Contactor

Participant’s hand

Mold

Fig. 2.3 Example of an experimental setup for threshold experiments. For those experiments, the stimuli is in general displacement delivered by a mechanical shaker [33, 35, 43]

stimulation is large (compare Fig. 2.2a, b). The NPI channel is mediated by Meissner receptors, and it is most sensitive in the range of 2–40 Hz. Finally, the NPIII channel is mediated by Merkel receptors with a sensitivity region of 2–16 Hz. The sensitivity regions of four channels are determined by detection thresholds experiments [34, 35]. These experiments are designed to obtain the minimum stimulus amplitude that can be detected. The stimulus is, in general, a displacement that is delivered by a mechanical shaker (Fig. 2.3). For example, Gescheider et al. conducted detection threshold experiments at various frequencies (0.4–500 Hz) applied to the glabrous skin of the hand through a large (2.9 cm2 ) and a small (0.008 cm2 ) contactors [33] (Fig. 2.3). They found that the detection thresholds of P channel vary with the contactor size which is due to the spatial summation property of P channel. Among all psychophysical channels, the P channel is the only one which has the spatial and temporal summation property. The sensitivity of the P channel increases a function of stimulation area (spatial summation) and duration (temporal summation). A complex vibrotactile stimulus (i.e. mechanical displacement) can be considered as a weighted sum of sinusoidal vibrations with different frequencies based on Fourier decomposition [43]. The tactile detection occurs when the energy content of one of these sinusoidal vibrations exceeds the detection threshold (energy) at that frequency. In other words, the detection thresholds of a complex vibrotactile

2.1 Human Tactile Sensing

13

stimulus are determined by the spectral component which has the highest energy weighted by the human psychophysical sensitivity.

2.1.3.2

Spatial Resolution, Amplitude and Frequency Discrimination

Spatial acuity of skin is evaluated by two main methods: two-point touch threshold and point-localization threshold [74]. The two-point touch threshold represents the smallest spatial gap between two stimuli applied to the skin that can be detected. This value has been found approximately as 2–4 mm for fingertip [61]. The point localization threshold is the smallest spatial gap between two stimuli applied with a time gap; it has been found as 1–2 mm for fingertip [122]. The discrimination thresholds are represented by Weber fraction, which the ratio of the just noticeable difference to the intensity or frequency of a stimulus. The value of the Weber fraction varies as a function of stimulus condition. Gescheider et al. found the Weber fraction for amplitude discrimination above the perceptual threshold approximately as 50% at 10 dB SL, which decreased to 5% at 40 dB SL [30]. They also found similar results for 25 and 250 Hz sinusoidal signal and broadband noise. Verrillo et al. [114] determined the equal subjective magnitudes for a range of vibrotactile frequencies. They found that these curves scale in proportion to incremental amplitude in decibels, meaning that it is logarithmic with the power of the signal. The exponent of the power function differs for the applied frequency. Different groups found conflicting results in terms of frequency discrimination thresholds. The frequency discrimination thresholds of glabrous skin were measured by Mahns et al. [81] at 20, 50, 100, and 200 Hz. They found that Weber fraction vary from 30% to 13% for 20 and 200 Hz, respectively. Conversely, Rothenberg et al. [96] found that Weber fractions increased from 18% at 20 Hz and approximately 30% at 300 Hz. Mobray and Gebhard found that the thresholds were around 2% at low frequencies and increased to 6% at higher frequencies [86].

2.1.3.3

Masking

Previous vibrotactile studies have shown that presenting one stimulus may interfere with the perception of another one. This interference is called tactile masking and can cause certain deficits in perception such as increasing detection thresholds and hindering localization or identification [22, 43, 113]. This phenomenon has been investigated extensively via detection and identification experiments. In detection experiments, the threshold amplitude for detecting a vibrotactile stimulus is measured separately in the absence of and presence of a masking stimulus. The difference in amplitude is defined as the threshold shift (i.e. amount of masking). In identification experiments, identification performance of a target stimuli in a presence of masking stimuli is determined. The identification of the target stimuli decreases as amount of masking increases. For both detection and identification experiments, the most commonly used masking techniques are forward (masking

14

2 Background

Pedestal Masking

Forward Masking

Mask Mask Test

a)

Test

Mask

b)

Mask

Simultaneous Masking

Backward Masking

Mask Test

Test

Mask

c)

d)

Sandwich Masking

Common-onset Masking Mask

Mask

Test

e)

Mask

Test

f)

Fig. 2.4 Stimulus timing diagrams for (a) pedestal, (b) forward, (c) simultaneous, (d) backward, (e) sandwich, (f) common-onset masking techniques

stimulus precedes test stimulus), backward (masking stimulus follows test stimulus), simultaneous (masking and test stimulus starts and ends at the same time), pedestal (test stimulus occurs during a continuous masking stimulus), sandwich masking (test stimulus is sandwiched between two masking stimuli), and common-onset masking (masking and test stimulus starts simultaneously, but latter one ends earlier). The stimulus timing diagrams of these masking techniques is illustrated in Fig. 2.4. Researchers have studied vibrotactile masking to understand neural and psychophysical mechanisms behind our touch sensation. The majority of these works were performed by Verrillo and his colleagues [27, 28, 31, 34, 48]. They conducted series of detection experiments using pedestal and forward masking techniques. In their experiments, they used test and masking stimuli in wide range of frequencies (0.4–500 Hz) applied by contactors in different sizes. The results of these experiments led them put forward the four channel theory explained previously. They found that each channel is sensitive to different input frequencies, which partially overlap. And, tactile masking only occurs when mask and test stimuli excite the same psychophysical channel. Based on these results, they suggested that

2.1 Human Tactile Sensing

15

the perceptual qualities of touch might be determined by the combined inputs from four channels. Most of these findings were validated by future works in different laboratories [40, 43, 82], and used in computational modelling of the sense of touch [38, 39, 41, 42, 44]. Recently, [71] applied sinusoidal vibrotactile stimuli in different frequencies to the neighbouring fingers and the different hands of the subjects. When the subjects judged the frequency of one vibration, the perceived frequency shifted towards the other. Moreover, when they judged the frequency of the pair as a whole, they reported the intensity-based interpolation of these two vibrations. These results suggested that perception of frequency is functionally enriched by signal integration across different mechanoreceptor channels and separate skin locations. Several factors influence the amount of masking. These factors are related to both mask and test stimuli such as their magnitude and duration, as well as the time between them, known as interstimulus interval (ISI). Many studies observed that increasing the duration of test stimulus and ISI decreases the amount of masking, whereas increasing mask duration and magnitude affect oppositely (see Table 2.3 for summary of these studies). Also, mask site (i.e. applied location on body) is another important factor that affects the resultant masking. The amount of masking increases if the test and mask stimuli applied to the same location [36, 115].

2.1.3.4

Texture Perception

Multidimensional scaling studies revealed that there are two main perceptual dimensions in texture perception: roughness/smoothness and hardness/softness [49, 50, 90, 129]. These dimensions are followed by stickiness/slipperiness and temperature. Perceptual stickiness is also often related to friction [90]. Since it is not possible to change the softness and temperature of the touchscreen using electrovibration yet, only roughness and friction perception will be summarized in this book. • Roughness: Human roughness perception of real textures has been investigated extensively in the literature [12, 14, 15, 51, 68, 72, 75, 76, 76, 77, 104]. In the literature, several types of stimuli have been used to investigate roughness perception of real textures; raised dots with controlled height and density [15, 77], dithered cylindrical raised elements [51, 68, 104], and metal plates with linear gratings [72, 73, 75]. These studies showed that size of the tactile elements (i.e gratings, dots, cones) and the spacing between them are critical parameters in roughness perception. Moreover, Hollins et al. [51] and Klatzky and Lederman [69] found that the underlying mechanism behind roughness perception is different for microtextures (textures having inter-element spacing ≈0.2 mm). At macro-textural scale, Lederman and collegues [68, 72, 73, 75] observed that groove width has a greater effect on perceived roughness than ridge width. This observation has been supported by other studies later [12] suggesting that the perceived roughness increases monotonically with the groove width, but

Backward pedestal forward

Forward

Pedestal

250 Hz sinusoidal

250 Hz sinusoidal

250 Hz sinusoidal & band limited noise (250–1000 Hz) 500 Hz sinusoidal & centered noise at 27 Hz 20 and 250 Hz sinusoidal 250 Hz sinusoidal

250 Hz sinusoidal & band limited noise (250–1000 Hz) 500 Hz sinusoidal & centered noise at 27 Hz

20 and 250 Hz sinusoidal 250 Hz sinusoidal

Test stimuli

Mask stimuli

Table 2.3 Summary of earlier studies investigating vibrotactile masking

20 dB SL

Variable 10– 30 dB SL 20 dB SL

Variable 5–25 dB SL

Variable 10– 50 dB SL

Mask level

700 ms

Variable 10– 1000 ms 700 ms

20.5 & 10 ms

Mask duration 1500 ms

Variable 10– [26] 660 ms Variable [29] 0–2000 ms

Variable 30– 660 ms 50 ms

[32]

[83]

[31]

Source

25 ms

Variable 5–595 ms

–

ISI

50 ms

20.5 & 10 ms

Variable 15– 1000 ms

Test duration

16 2 Background

2.1 Human Tactile Sensing

17

decreases modestly with the ridge width. On the other hand, further increasing the spacing between the tactile elements (more than 3.5 mm) causes the subjects perceive the surface as smooth rather than rough. Hence, roughness perception follows an inverted U-shaped function of groove width, reaching a maximum value at approximately 3.5 mm of bump separation. Based on these observations, Lederman and Taylor developed a mechanical model that estimated the perceived roughness as a power function of the total area of the skin that was instantaneously deformed through contact with the surface texture [68, 75]. Later, Connor et al. [15] developed a neural model which stated that roughness perception of macro textures is achieved by spatial cues through pressure change and finger deformation. These arguments were later supported by showing that speed has little effect on perceived roughness [77]. In contrast to these studies, Cascio and Sathian [12] showed that roughness perception is affected by temporal frequency, which is defined as the ratio of finger speed to wavelength of texture. They conducted experiments with different finger speeds on real textures. Their results showed that roughness perception of the subjects increased as a function of temporal frequency for textures with varying ridge widths. Moreover, Smith et al. [104] reported that roughness perception is positively correlated with the rate of change in the lateral force. This result also implies that the temporal cues play a role in roughness perception of macrotextures. At micro-textural scale, Hollins and colleagues [51] reported that the roughness perception is mainly achieved by vibratory cues. Moreover, they found direct evidence that roughness perception of microtextures is mediated by Pacinian Corpuscles. They used a Hall effect transducer to record the vibrations generated on the skin when a set of micro-textured surfaces is passively presented to the index finger of subjects. They weighted the power of the measured vibrations according to the spectral sensitivity of Pacinian mechanoreceptors to show that roughness estimates of the subjects were correlated with these weighted skin vibrations [10]. • Friction: Both tangential and normal forces play a role in friction perception, as there should be relative sliding motion between the finger and surface to perceive it. Most of the studies in this area investigated the friction perception of a smooth glass surface under different experimental conditions. These studies observe the changes in the friction coefficient, which is the ratio of the tangential force to the applied normal force. It has been found that increasing normal forces increases tangential force, while the coefficient of friction decreases until a steady-state value [6]. This behavior is common for viscoelastic materials, such as a fingertip. Moreover, the friction coefficient is affected by other factors [1, 17, 18, 94, 109] such as velocity, surface roughness, moisture, fingerprints, age, and gender.

18

2 Background

2.2 Overview of Surface Haptic Displays Surface haptic displays are flat surfaces that can deliver programmable haptic feedback to a user’s finger [6, 62, 70]. Different actuator technologies, such as eccentric rotating mass motors, linear resonant actuators, electroactive polymers, voice coils, piezoelectric actuators, shape memory alloys, have been used to create haptic feedback on these devices. These actuators vary in their bandwidth, intensity, response times, power requirements, actuation directionality, and modality types [70, 120]. The properties of the selected actuator type often determine the domain of application of the display. Surface haptics displays can be classified into five groups based on their actuation methods: mechanical vibrations, electrotactile, surface shape-changing, thermal, and friction modulation (see Fig. 2.5).

2.2.1 Mechanical Vibration Displays The devices in this category deliver high-frequency mechanical vibration signals to the user. Eccentric mass motors, voice coils, linear resonant actuators, and piezoelectric devices are the most common actuator types used in these displays.

2.2.1.1

Eccentric Rotating Mass Motors (ERM)

An ERM is a DC motor with an eccentric mass attached to its shaft. When a DC voltage or current is applied, this eccentricity generates radial forces. Although these actuators are cheap and simple to use, both the frequency and amplitude of the generated vibration are coupled with the motor’s rotational speed [13, 123]. Hence, they cannot render vibrations at arbitrary combinations of frequency and amplitude. Moreover, they have high internal static friction, which causes a delay in delivering the tactile cue. Nonetheless, they are still one of the most common actuators used for surface haptic displays, and they can be found in most mobile phones and game controllers [4]. Surface Haptic Displays

Mechanical Vibration ERM Voice Coil LRA Piezoelectric

Electrotactile

Surface shape changing Pneumatic Fluidic Electroactive Polymers

Thermal

Friction Modulation Electrovibration Ultrasonic

Fig. 2.5 Overview of surface haptic displays. Based on their actuation types, they can be classified into five groups: mechanical vibration, electrotactile, surface shape changing, thermal, and friction modulation

2.2 Overview of Surface Haptic Displays

2.2.1.2

19

Voice Coils

These actuators are consist of a movable permanent magnet is enclosed by a coil of wire [105]. When an alternating current is applied to the coil, an oscillating magnetic field is generated due to the Lorentz principle. This time-varying magnetic field moves the magnet and as a result, produces vibrations. Voice coils are easy to integrate to different devices, and they can produce large bandwidth accelerations (typically 50 Hz to several kHz) [123].

2.2.1.3

Linear Resonant Actuators (LRA)

These actuators have a similar working principle of the voice coils. However, in an LRA, a voice coil is pressed against a moving mass, which is attached to a spring. When the voice coil is at the same frequency as the spring, the entire actuator vibrates with a perceptible force. Although the amplitude and frequency of the resulting vibration can be controlled by the input current, the actuator should be driven at its resonant frequency to generate large enough vibrations. Therefore, their bandwidth is lower than the voice coils. However, they are more advantageous than voice coils in terms of power consumption [13].

2.2.1.4

Piezoelectric Actuators

Piezoelectric actuators are ceramic materials that change shape when an electrical voltage is applied to them. Inversely, when they are elongated, they can produce electrical voltage; hence they can be used as sensors. These actuators are generally used as multi-layer disks and beams to deliver haptic feedback. They are most effective when they are actuated at their resonant frequencies. Therefore, they are generally used by modulating their vibration amplitude at various frequencies. They are fast, low-powered, and they can deliver high-resolution outputs [11]. However, they require high input voltages, and they are hard to integrate on a device.

2.2.2 Electrotactile Displays Electrotactile displays directly activate nerve fibers within the skin by passing an electrical current through surface electrodes. They are simple to control and maintain, as they do not contain any moving parts. They are also compact and require low power requirements [62]. However, they have a limited dynamic range compared to electromechanical actuators. They are generally used in sensory substitution systems.

20

2 Background

2.2.3 Surface Shape Changing Displays These devices deliver tactile feedback by changing the shape of the surface through pneumatic and hydraulic elements, shape memory alloys, and electroactive polymers [13].

2.2.3.1

Pneumatic and Hydraulic Actuators

These actuators are controlled by gas or liquid pressure, and they move, expand, or contract mechanical elements in contact with the user’s skin. They typically require a high-pressure air or liquid source. Pneumatic actuators are lightweight, and they have a high power-to-weight ratio, but low bandwidth. Hydraulic actuators are fast, and they deliver high actuation forces. However, they are generally bulky and heavy [11].

2.2.3.2

Electroactive Polymer Actuators (EAP)

These actuators are elastomers which change shape and size when activated by an electric field. They are mainly classified into two categories: ionic and electric. Ionic actuators need low operational voltages, but their response times and actuation forces are small. Nonetheless, electric actuators require high voltages, but they can deliver fast and high actuation forces [11]. Electric EPAs are further classified as piezoelectric polymers, electrostrictive polymers, dielectric elastomers, liquid crystal elastomers, and ferroelectric polymers.

2.2.3.3

Shape Memory Alloys (SMA)

SMA actuators are metals that can change their mechanical properties in response to temperature changes and can go back to their original shapes. These actuators can be small and have a high power to weight ratio. However, they suffer from slow response times, larger hysteresis, and high energy consumption [11].

2.2.4 Thermal Displays Thermal displays change the skin temperature by a thermal actuator to provide thermal cues to the user. The most commonly used thermal actuators are Peltier devices. These devices create a temperature difference at the junctions of two dissimilar conductors in contact when a DC current passes through the circuit. In

2.3 Electrovibration for Tactile Displays

21

addition to Peltier devices, infrared lamps and fans are also used to display thermal information [62].

2.2.5 Friction Modulation Displays Friction modulation displays change the friction between the fingertip of the user and the touchscreen to generate tactile feedback. Currently, there are two main friction modulation techniques: ultrasonic actuation and electrovibration [6].

2.2.5.1

Ultrasonic Vibration Displays

Ultrasonic vibration displays modulate the friction force between the finger and the touchscreen by mechanically actuating the touchscreen at high frequencies [121]. In general, ceramic piezoelectric actuators are vibrated at their resonance frequency at an ultrasonic frequency, and the amplitude of the resulting vibration vary friction. These displays can deliver fast and high actuation forces, but they require high driving voltages [123].

2.2.5.2

Electrovibration Displays

Electrovibration displays modulate the friction force between the finger and the touchscreen by electrostatic forces, which are generated by applying an alternating voltage signal to the conductive layer of a capacitive touch screen. These displays require low power, and they can deliver fast, high-bandwidth, and dynamic tactile feedback. The state of the art research on electrovibration is summarized in Sect. 2.3.

2.3 Electrovibration for Tactile Displays 2.3.1 Foundation The electrical attraction between human skin and a charged surface was first reported by Johnsen and Rahbeck [60]. Around thirty years later, Mallinckrodt discovered that applying alternating voltages to an insulated aluminum plate can increase friction during touch and create a strange resin-like feeling [84]. He explained this phenomenon based on the well-known principle of the parallelplate capacitor. Later, Grimnes named this phenomenon as “electrovibration” and reported that surface roughness and dryness of finger skin could affect the perceived haptic effects [37].

22

2 Background

The first tactile display using electrostatic actuation was developed by Strong and Troxel [106]. This display consisted of an array of electrodes insulated with a thin layer of polyvinylidene chloride. They conducted various experiments and found that the intensity of touch sensation was primarily due to the applied voltage rather than the current density and effected from the active electrode area. Based on these results, they developed a physical model between the input voltage and generated electrostatic force. Later, Beebe et al. [9], developed a polyimide-onsilicon electrostatic fingertip tactile display using lithographic microfabrication. They were able to generate tactile sensations on this thin and durable display using 200–600 V voltage pulses and reported the perception at the fingertip as sticky. Later, Tang et al. [108] performed experiments of detection threshold, line separation, and pattern recognition with visually impaired subjects. Although they encountered problems such as dielectric breakdown and sensor degradation, the subjects were able to differentiate simple tactile patterns by haptic exploration. In all of the above studies, electrovibration was obtained using opaque patterns of electrodes on small scale surfaces (see Fig. 2.6a for an illustration). However, in the recent work of Bau et al. [8], electrovibration was delivered via a transparent electrode on a large commercial touch surface, which demonstrates the viability of this technology on mobile applications. This touch surface is a glass-coated with a conductive indium tin oxide (ITO) layer that is insulated with a silicon dioxide (SiO2 ) layer. When an alternating voltage signal is sent through the conductive ITO layer, an electrostatic attraction force is built between the display and the finger, as the finger is also electrically conductive. Because the ITO layer on those surfaces is a single electrode, this causes a uniform feeling all over the touch surface (see Fig. 2.6b). Reversely, it is also possible to send the input voltage to the finger and create an attraction force on various surfaces [7] (see Fig. 2.6c). This method makes possible to use electrovibration for augmented reality systems independent from a touchscreen. Another possible way to generate tactile feedback using electrostatic forces is to touch the display with a conductive pad [128]. This method can augment the generated forces and eliminate irregular perceptual effects due to moisture. Similarly, one can send the input voltage to the conductive pad between the finger and the display. Using this approach, Nakamura et al. [88] designed a multiuser surface visuo-haptic display. In that study, they delivered the haptic actuation signal to the multiple contact pads instead of the touch surface itself. They applied low-frequency haptic voltage and high-frequency sensing voltage to each pad. In addition to delivering different sensations to multi-fingers, their technique also allowed sensing the finger position without additional hardware. Similarly, a penbased electrostatic system (EV-Pen) was introduced by [119]. In their system, the input voltage signal is applied to a capacitive pen, and the electrostatic force is generated between the moving pen and the touch screen.

2.3 Electrovibration for Tactile Displays

23

(a)

(b)

(c)

(d)

(e) Fig. 2.6 Different types of electrostatic displays. (a) Each electrode is independent and can be actuated separately to deliver localized sensation [9, 106, 108]. (b) The input voltage signal is delivered to the conductive layer of the touchscreen [8, 80]. (c) The input voltage is delivered to the user [7]. (d) The user touches the display through a conductive pad and perceives the propagated forces [128]. (e) The input voltage is sent directly to the conductive pad, which enables multi-user applications [88]

24

2 Background

2.3.2 Applications Electrovibration has great potential for a wide range of applications for design tools, virtual controls and widgets, education, gaming, and communication purposes [6]. Bau et al. proposed several examples of these applications, such as painting, slider design, non-visual information layers, drag and drop operations, simulating texture feelings [8]. Electrovibration can also be an effective way of developing assistive and sensory substitution tools for visually impaired people. Xu et al. [126] investigated delivering 2D tactile information such as dots, Braille letters, and shapes using electrovibration. They conducted a user study, where subjects explored these patterns without visual cues. The subjects recognized the dots easily but had difficulties to recognize the letters, without visual cues. Also, they were moderately successful in recognizing the shapes. Similarly, Israr et al. [56] used electrovibration to deliver assistive tactile feedback for visually impaired users. Recently, Lim et al. [79] designed an application that allows visually impaired people to perceive the main features of the persons’ photos.

2.3.3 Modelling 2.3.3.1

Electrical Modelling

The electrostatic force developed between a sliding finger and the electrode can be explained by the well-known parallel plate capacitor principle, [64, 106]. According to this principle, if two charged conducting parallel plates are separated by an insulator with a thickness d, an electrostatic force, F , occurs across the insulator: F =

Ap Vg2 2d 2

,

(2.1)

where  is the permittivity of the insulator, Ap is the area of the conductors, and Vg is the voltage difference between the two conducting layers. Regarding finger-surface interaction, this model should be modified. A touchscreen used for electrovibration consists of two thin layers deposited on a glass substrate. The first layer on top of the glass is a thin layer of transparent conductive material – mostly indium tin oxide (ITO). On top of this layer, a thin layer of insulator material appears. The fingertip has approximately 200 microns thick outermost skin called stratum corneum. This layer acts as an additional dielectric which enables a potential drop from ITO to the conducting tissue under the stratum corneum when a voltage applies, [64, 85, 116]. Earlier studies claimed that electrostatic forces are developed at the boundaries of the two dielectrics: stratum corneum and insulator. According to these studies, if a human finger on a touchscreen surface is represented in Fig. 2.7a, the electrostatic force which effects the fingertip can be expressed as

2.3 Electrovibration for Tactile Displays

25

Fig. 2.7 Equivalent circuit model of human finger on a tactile display surface: (a) neglecting the air gap between finger ridges and touch screen, (b) considering the air gap between finger ridges and touch screen

(a)

(b)

26

2 Background

2  0 sc A Vsc , 2 dsc

Fe =

(2.2)

where sc is the relative permittivity of the stratum corneum, 0 is the permittivity of vacuum, A is the area of the fingerpad, dsc is the thickness of the stratum corneum. Vsc is the voltage across the stratum corneum, which can be expressed as a function of the voltage applied to the conductive layer of the touch screen, V , as Vsc = V

Zbody

Zsc , + Zsc + Zi

(2.3)

where, Zbody , Zsc , and Zi represent the impedances of the human body, stratum corneum, and touch surface respectively. The reader may refer to [16] for more information related to the derivation of the electrostatic force generated at the boundaries of two parallel or series dielectrics. Recent studies suggest that the air gap between fingertip ridges and touch screen has also great influence on the generated electrostatic. These studies [89, 99] explain that the effective electrostatic force is developed across the thin gap of air at the interference: Fe =

2  0 air A Vair , 2 dair

(2.4)

where air is the relative permittivity of the air, dair is the thickness of the air gap. Vair is the voltage across the air gap, which can be expressed as a function of the voltage applied to the conductive layer of the touch screen, V , as Vair = V

Zbody

Zair , + Zsc + Zair + Zi

(2.5)

where, Zair , represent the impedance of the air (see Fig. 2.7b). Vodlak et al. [118] conducted finite element analysis simulations and showed that the presence of the air gap could increase the electrostatic force up to 10– 20%. Supporting these simulations, Guo et al. [45] measured finger contact forces at different applied normal forces by keeping the apparent finger contact area constant. Then they estimated the air gap thickness by using the Equation 2.4. Their results showed that the relationship between the air gap thickness and the normal force follows a power function. Moreover, Shultz et al. [100] measured electrical impedance of the sliding and stationary fingers under electrostatic force. Their result showed a significant magnitude difference between those two cases. They explained that this difference could be caused by the electrical impedance of the air gap between finger and touchscreen, which was shorted out when the finger does not move.

2.3 Electrovibration for Tactile Displays

2.3.3.2

27

Contact Mechanics

The earlier studies assumed the contribution of the electrostatic force to the total frictional force Ff follows Columb model of friction: Ff = μ(Fn + Fe ),

(2.6)

where Fn is normal force applied by the fingertip, and μ is the friction coefficient [64, 85, 116]. To understand how mechanical forces develop at the fingertip-surface interface, Meyer et al. [85] developed a tribometer and measured the lateral force of a fingertip actuated by electrostatic force by keeping the applied normal force constant. Their results indicated a similar relationship with Equation 2.6. They also showed that the measured electrostatic forces changed as a function of frequency due to the frequency-dependent electrical properties of stratum corneum. By conducting detailed simulations and experiments, Vardar et al. [110, 111] further proved that this nonuniform electrical property of human skin also affects the perception of electrovibration stimuli generated by different input voltage waveforms. The results of this study are one of the contributions of this book, and they will be explained in detail in Chap. 3. The Coulomb friction model gives valuable insight into the relationship between the resulting tangential force due to electrostatic force at constant normal forces. However, the friction of human skin against smooth surfaces is governed by the adhesion model of friction [1]: Ff = τ Areal ,

(2.7)

where τ is interfacial shear strength and Areal is the real contact area of the finger. The relationship with the Areal and the normal force, Fn is nonlinear. In the case of a finger, the real contact area is the sum of the contacting asperities of the fingerprints with the surface of the touchscreen, and it is much smaller than the apparent contact area. Following this adhesion model, Persson [95] proposed a mean-field theory based on the multi-scale contact mechanics to analyze the effect of electrostatic forces on sliding friction. Later, Ayyildiz et al. [3] validated this theory by conducting full-scale contact mechanics simulations and experiments. They measured the friction forces of a finger sliding on a touchscreen actuated by electrostatic forces by applying different normal forces. They found that the friction coefficient decreased as a function of the applied normal force. Their results showed that electroadhesion increases the real contact area, increasing frictional force. Later, Sirin et al. [101] measured the fingerpad contact area evolution under electrostatic forces and showed that the effect of electrovibration is only present during full slip but not before the slip. Also, the apparent contact area of the fingertip was smaller under electrovibration. Recently, Basdogan et al. [5] developed a model for estimating voltage-induced frictional forces between human finger and a touchscreen as a function of the applied normal force. They estimated

28

2 Background

the unknown parameters via optimization by minimizing the error between the measured tangential forces and the ones generated by the model.

2.3.4 Tactile Rendering and User Interface Design Tactile rendering of textures, shapes, surface features, and user interfaces is substantial for the future usage of electrovibration displays. This section will summarize the state of the art studies related to tactile rendering and user interface design with electrovibration.

2.3.4.1

Texture Rendering

The most common approach for rendering textures on electrostatic displays is using data-driven methods. Yamamoto et al. [128] developed the first data-driven texture rendering system. Their experimental setup was consist of a linear stage with a built-in tactile sensor-driven that moved in synchronization with a slider on the tactile display. The vibration data collected by a tactile sensor was processed and regenerated on tactile display simultaneously via electrostatic forces. Their experimental results showed that subjects discriminated against different textures with a correct response of 79%. Later, Ilkhani et al. [52] collected surface data from real textures using an accelerometer and then replayed on the touchscreen directly. However, they did not compensate for the nonlinear dynamics of the electrostatic force generation. The results of their psychophysical experiments demonstrated that virtual textures generated by the data-driven approach show higher similarity to realistic textures in comparison to the ones generated by periodic square waves at different frequencies. Later, Jiao et al. [59] collected force and position data while a finger was sliding on 10 different fabric textures. Then, they modulated the input voltage based on the calculated friction coefficients and evaluated their method by conducting psychophysical experiments. Their results showed that the virtual textures were perceived similarly to the corresponding real textures. Different from these approaches, Osguei et al. proposed an inverse neural network model to compensate for the nonlinear dynamics of electrovibration [92]. Recently, Fiedler et al. [23] proposed a data-based rendering method that can significantly compress the tactile data. This study is one of the topics of this book, and it will be explained in detail in Chap. 5. Another approach to generate texture feelings on electrovibration displays is generating virtual gratings by modulating the friction based on the finger motion. Vardar et al. [112] designed virtual textures by using low frequency unipolar pulse waves in different shape (sinusoidal, square, sawtooth, triangle), and spacing (e.g. groove width). They modulated these waves with a 3 kHz high frequency sinusoidal carrier signal. This study is another topic of this book, and the details will be explained in Chap. 6.

2.3 Electrovibration for Tactile Displays

29

Researchers also generated textures by image-based features. Wu et al. [125] estimated the local gradients of images and mapped the frequency (texture granularity) and amplitude (texture height) of the voltage signals. Recently, Mun et al. [87] represented 32 textures using regular tessellations of polygons.

2.3.4.2

Shape and Surface Feature Rendering

Xu et al. [126] displayed raised dots for Braille cells, and simple geometric shapes by electrovibration. However, the recognition success rate was around 56%. Later, Kim et al. [66] developed an algorithm to render a 3D geometrical surface in the form of a heightmap. For that purpose, they modulated the friction force based on the local gradient of the surface. Following this study, [91], generalized this algorithm to estimate the surface gradient for any 3D mesh and added an edge detection algorithm to render sharp edges. They tested their algorithm and found that their approach can improve the performance of 3D shape recognition when visual information is limited.

2.3.4.3

User Interface Design

There are only a few studies on user interface design with electrovibration. Zhang and Harrison [130] showed that increasing friction on a target improves target acquisition performance. Emgin et al. [21] rendered a virtual knob by rendering its detents by electrovibration. Although the user performance was not affected by the presence of the tactile feedback, the subjective experience of the participants was increased.

2.3.5 Perception In the surface haptics domain, understanding the human perception of tactile stimuli generated by different displays is crucial to the design of the applications, illusions, and new devices. For example, a device can deliver high bandwidth stimuli over a large range of frequencies, but if these frequencies are outside of human haptic perception, this device is useless regardless of how sophisticated it is. In this section, the state of the art research, which investigates the factors affecting electrovibration perception, are summarized.

2.3.5.1

Input Signal Properties

As explained in the previous paragraphs, electrovibration stimuli are generated by sending an input voltage signal to the conductive layer of one of the contacting

30

2 Background

bodies. The studies on the electrovibration perception have investigated the case where the input voltage is sent to the touchscreen (Fig. 2.6b). As any haptic device, the generated stimuli are coupled with the input signal properties for electrovibration. In this regard, Kaczmarek et al. [64] explored the differences in detection thresholds of electrovibration stimuli generated by positive, negative, and biphasic input voltages. They found that the subjects perceived negative or biphasic pulses better than positive ones. They claimed that this disparity could be due to the asymmetric electrical properties of human skin. Later, Bau et al. measured the sensory thresholds of electrovibration using sinusoidal inputs applied at different frequencies [8]. The results showed that the change in threshold voltage as a function of frequency followed a U-shaped curve similar to the one observed in vibrotactile studies. Wijekoon et al. [124] investigated the perceived intensity of electrovibration stimuli and found that it was logarithmically proportional to the amplitude of the applied voltage signal. Later, Vardar et al. [110, 111] investigated the effect of input voltage waveform on the tactile perception of electrovibration by conducting psychophysical experiments and measuring contact force and accelerations occurred due to various stimuli. Their results showed that low-frequency square wave voltage signals are perceived more strongly than sinusoidal ones at the same frequency. They explained that even slow square waves have high-frequency components that stimulate the sensitive Pacinian psychophysical channel. These two studies are included in the topics of this book, and they will be explained in detail in Chap. 3. Later, Kang et al. [65] investigated the methods that can provide high-intensity electrovibration perception with a lower voltage input. Their measurements showed that applying input voltage with a DC-offset can provide a larger electrostatic force than that of without a DC-offset when the peak-to-peak amplitudes of both signals are equal. Moreover, their psychophysical experimental results validated that this method can provide a high-intensity electrovibration perception with less voltage.

2.3.5.2

Interference of Multiple Stimuli

The future touchscreen applications will probably require multiple and complex tactile stimuli displayed simultaneously or consecutively. For example, a gaming application may require different stimuli to deliver the change of the scene or effects. Hence, the interference of one stimulus on the perception of another one should be investigated for designing effective interfaces. In this regard, Vardar et al. [113] investigated the interference of multiple tactile stimuli generated by electrovibration. Their results indicated that the presence of an electrostatic stimulus reduces the perception of another one. Moreover, they showed that the perceived sharpness of virtual edges explored by sliding depends on the local haptic contrast between the background texture and the foreground item. This study is also another topic of this book, and its details will be explained in Chap. 4. Other studies investigated whether the presence of a mechanical stimulus affects the perception of an electrovibration stimulus. Ryu et al. [97] designed a hybrid display by attaching piezo actuators to a capacitive touchscreen, and they measured

2.3 Electrovibration for Tactile Displays

31

the detection thresholds of electrovibration stimulus by applying mechanical stimuli with the same frequency at different levels. Their results showed that the detection thresholds of electrovibration increased as a function of the intensity of the mechanical vibration. Later, Jamalzadeh et al. [57] applied vibrotactile stimulus on the proximal phalanx of the index finger of participants and measured the detection thresholds of electrovibration stimuli applied to their fingertips. Although the vibrotactile stimulus did not interfere with the electrovibration stimulus, they observed a clear psychophysical masking effect due to central neural processes.

2.3.5.3

Texture Perception

As explained in the previous section, the tactile rendering of textures and shapes is very important for the future applications of electrovibration. Hereof, Vardar et al. [112] investigated the roughness perception of periodic gratings of four different waveforms (sine, square, triangular, and saw-tooth) displayed by electrovibration. Their experimental results showed that the roughness perception of the gratings followed an inverted U-shaped trend along groove width. The subjects perceived square wave as the roughest, while they perceived other waveforms similar. This study is also a contribution of this book, and its details will be explained in Chap. 6. Later, Isleyen et al. [55] compared the roughness perception of these virtual gratings with the real ones and found that the roughness perception of real and virtual gratings are different. They argued that this difference can be explained by the amount of fingerpad penetration into the real gratings, which is not possible for the virtual case. They also found that increasing normal force increases the perceived roughness of real gratings while it causes an opposite effect for the virtual gratings. By following a similar texture rendering approach, Jiao et al. measured the detection and discrimination thresholds of virtual gratings in different waveforms [58]. They found that JND of the gratings increases with the increase of voltage amplitudes and triangle and sawtooth waveforms had the highest JNDs. Ozdamar et al. [93] investigated the tactile perception of a step change in friction. Their experimental results showed that the participants perceived rising friction stronger than falling friction, and both the normal force and sliding velocity significantly influenced their perception. Recently, Mun et al. [87] visualized three-dimensional perceptual space of virtual textures expressed by regular tessellations of polygons by conducting pairwise similarity experiments. They showed that these dimensions are most related to the perceptual attributes of rough-smooth, dense-sparse, and bumpy-even.

2.3.5.4

Finger Moisture

Although there are not any detailed study on the relation of finger moisture and electrovibration perception, many studies reported that moisture decreases the strength of electrovibration perception [37, 84, 108]. When a sweat layer

32

2 Background

accumulates between a fingertip and a touchscreen, the electric field is formed between the touchscreen and the sweat. This situation may decrease the perceived force on the skin. In addition to this, the physical characteristics of sweat layer may also prevent the formation of shear force between fingertip and touchscreen, and degrade the sensation [84, 108].

2.3.5.5

Insulator Properties

As explained in Sect. 2.3.3, the insulator thickness and dielectric properties affect the generated electrostatic force. Agarwal et al. [2] investigated the effect of dielectric thickness on haptic perception during electrovibration stimulation. Their results showed that variations in dielectric thickness had little effect on the threshold voltage. By conducting simulations, Ayyildiz et al. [3] showed that using thinner insulator thickness increases the generated electrostatic force, which may improve electrovibration perception.

2.3.6 Advantages Each surface haptic display listed in Sect. 2.2 has its advantages and disadvantages which determines their primary usage area. The advances in materials and actuator technologies and also developments in haptic research will help surface haptics displays be available in the mass-market in the future. The advantages of electrovibration displays compared to other surface haptic devices are discussed in [8] and summarized below.

2.3.6.1

Easiness to Scale

One of the main advantages of electrovibration is its easiness to scale. If the coating of the surface can be done evenly, the electrovibration can be delivered without a substantial perceptual difference to large [21] or small [98] surfaces. The input voltage is delivered to an ITO electrode coated with an insulator, which can be manufactured in different sizes. On the other hand, generating haptic feedback with mechanically actuated devices is rather complicated, as it requires design optimization for each size as the mechanical modes and resonances will change depending on display size.

2.3.6.2

Compactness

Although the actuator industry has been developed substantially, it is still hard to fit many actuators into mobile devices and produce sufficient force to displace the

2.3 Electrovibration for Tactile Displays

33

touchscreen. As the actuation occurs between the finger and the display itself, the electrovibration provides a compact design solution.

2.3.6.3

Absence of Mechanical Motion

Electrovibration occurs due to the electrostatic attraction forces developed between a finger and the touchscreen; the actuation does not require mechanical motion. The absence of moving parts helps to generate uniform and tactile feedback over the surface. For the displays actuated with mechanical forces, the generated feedback can vary in different parts of the same surface due to its mechanical modes. Similarly, the intensity of the haptic feedback may suddenly attenuate or increase for different frequencies, due to the resonances of the screen. Compensating those effects needs precise mechanical modeling and control of the display. For electrovibration, although human skin acts as a high-pass filter electrically and attenuates the amplitude of the generated electrostatic forces for frequencies lower than 100 Hz, this problem can easily be solved by simple filtering or amplitude modulation. Moreover, the absence of mechanical motion causes a silent and reliable actuation, which is crucial for using these displays in the mass-market.

2.3.6.4

Possibility to Design for Localized Feedback, Multi-touch, and Flexible Displays

Another advantage of electrovibration is the possibility to apply it for future touchscreens which may need localized haptic feedback, multi-touch, or flexible displays. Although the current technology is not at this level yet, the rapid developments in material technology can open new possibilities to use electrovibration for future displays. As explained in the previous paragraphs, electrovibration can deliver uniform feedback regardless of the size of the display, which makes it perfect to use it for localized or multi-touch displays. However, for these types of displays, multiple electrodes should be actuated. There are some recent attempts from different research labs to design such displays; interested readers can refer to [19, 47, 53]. Moreover, electrovibration can be easily delivered on flexible or curved surfaces [25, 54].

2.3.7 Challenges Despite their advantages, electrovibration displays have many challenges that should be overcome for their usage in future electronic devices. Some of these challenges are also reported in [6].

34

2.3.7.1

2 Background

Stable Tactile Feedback Regardless of the Environmental Conditions

One of the main challenges of generating tactile feedback by electrostatic actuation is maintaining the intensity of the tactile feedback for different environmental conditions. When the electrovibration displays are explored by a bare finger, the electrostatic force generation will be based on the electrical circuit between the touchscreen and the human body. Therefore, all factors affecting this circuit, such as clothing, grounding [67], temperature [46], moisture [101], electrowetting [78], normal force [45], and contact area [3], influence the resulting tactile feedback. Hence, this situation will cause frustration if one wants to use the same display both on a beach or a snowing mountain. One solution to prevent the grounding and clothing problem is controlling current input instead of voltage [67, 100].

2.3.7.2

Consistent Tactile Feedback Among Different Users

The studies investigated the tactile perception of electrovibration often reported high variability among subjects [8, 23, 110–113]. Hence, depending on the tactile sensitivity or the skin property of the user, the perceived feedback can be nonsatisfactory. Especially, naive users need some warm-up time to feel the tactile effect [84]. During our demonstrations to different guests, we have solved this issue by using a conductive glove [20, 24]. Conversely, the tactile feedback can be annoyingly high to the user. For example, one person was extremely sensitive in our demonstration and commented that it left a lingering effect. To eliminate such problems, there should be a personalized calibration process for each user.

2.3.7.3

Tactile Feedback Displayed in Different Directions

The electrostatic forces between the finger and the touchscreen using electrovibration occur in the normal direction. These electrostatic forces are rather small compared to vibrations generated by mechanical actuators. The perceived electrostatic force is due to the enhancement of tangential force when the finger moves [8, 110]. Hence, it is not possible to generate perceivable feedback in different directions using current technology. For example, one cannot deliver a button click effect on the current electrostatic displays. The solution to this problem could be the optimization of the touchscreen for electrovibration purposes. Another solution could be designing hybrid systems combining both mechanical or electrotactile actuation with electrovibration [21, 97, 103, 117, 127].

2.3 Electrovibration for Tactile Displays

2.3.7.4

35

High-Voltage Consumption

Another challenge related to electrovibration displays is high-voltage consumption. However, this problem can be easily solved by optimizing the touchscreen by reducing the thickness of the dielectric layer of the touchscreen and maintaining a good grounding condition. More effective tactile feedback with lower voltages can be obtained by using a conductive pad, pen, or a glove instead of interacting with the surface with a bare finger.

2.3.7.5

Dependence on Contact Conditions

The electrostatic forces are developed between two bodies, and they are prone to the contact conditions such as surface roughness or friction [3, 37, 100]. Hence, the future touchscreens should be produced by considering these properties to deliver efficient tactile feedback.

2.3.7.6

Localized Feedback and Multi-touch Systems

As explained in Sect. 2.3.1, there are different approaches to deliver electrostatic feedback to the users. For more realistic and efficient tactile effects, delivering localized sensations to the same of multiple fingers is preferred. These can be achieved by actuating multiple electrodes simultaneously or sending the voltage signal to different conductive pads [88]. Although the first is generally desired, but it is rather challenging and requires material design optimization.

2.3.7.7

Realistic Tactile Effects

Human tactile perception of surface properties is dependent on many perceptual dimensions, including roughness, hardness, warmth, and stickiness [49]. Although one can change the perceived friction and roughness using electrovibration, it is not possible to change the perceived hardness or temperature unless it is used with another actuator. Also, it is challenging to deliver topographical information (macrolevel details) simultaneously with micro-level roughness with electrovibration causing different perceived feelings compared to real surfaces [55, 112].

36

2 Background

References 1. Adams, M., Johnson, S.A., Lefevre, P., Levesque, V., Hayward, V., Andre, T., Thonnard, J.L.: Fingerpad friction and its role in grip and touch. J. R. Soc. Interface 10(80) (2013) 2. Agarwal, A.K., Namni, K., Kaczmarek, K.A., Tyler, M.E., Beebe, D.J.: A hybrid natural/artificial electrostatic actuator for tactile stimulation. In: Proceedings of the 2nd Annual Conference on Microtechnologies in Medicine and Biology, pp. 341–345 (2002) 3. Ayyildiz, M., Scaraggi, M., Sirin, O., Basdogan, C., Persson, B.N.J.: Contact mechanics between the human finger and a touchscreen under electroadhesion. Proc. Natl. Acad. Sci. U. S. A. 115(50), 12668–12673 (2018) 4. Banter, B.: Touch screens and touch surfaces are enriched by haptic force-feedback. Inform. Display 3(10) (2008). https://doi.org/10.1002/j.2637-496X.2010.tb00231.x 5. Basdogan, C., Alipour, M.R., Sirin, O.: Modeling sliding friction between human finger and touchscreen under electroadhesion. IEEE Trans. Haptics (2020). https://doi.org/10.1109/ TOH.2020.2989221 6. Basdogan, C., Giraud, F., Levesque, V., Choi, S.: A review of surface haptics: enabling tactile effects on touch surfaces. IEEE Trans. Haptics (2020). https://doi.org/10.1109/TOH.2020. 2990712 7. Bau, O., Poupyrev, I.: Revel: Tactile feedback technology for augmented reality. ACM Trans. Graph. 31(4) (2012). https://doi.org/10.1145/2185520.2185585 8. Bau, O., Poupyrev, I., Israr, A., Harrison, C.: Teslatouch: electrovibration for touch surfaces. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (UIST), pp. 283–292 (2010) 9. Beebe, D., Heymel, C., Kaczmarek, K., Tyler, M.: A polyimide-on-silicon electrostatic fingertip tactile display. In: Proceedings of the IEEE 17th Annual Conference on Engineering in Medicine and Biology Society, pp. 1545–1546 (1995) 10. Bensmaia, S., Hollins, M.: Pacinian representations of fine surface texture. Percept. Pscyhophys. 67, 842–854 (2005) 11. Biswas, S., Visell, Y.: Emerging material technologies for haptics. Adv. Mater. Technol. 4(1900042) (2019). https://doi.org/10.1002/admt.201900042 12. Cascio, C., Sathian, K.: Temporal cues contribute to tactile perception of roughness. J. Neurosci. 21(14), 5289–5296 (2001) 13. Choi, S., Kuchenbecker, K.: Vibrotactile display: perception, technology, and applica-tions. Proc. IEEE 101(9), 2093–2104 (2013) 14. Connor, C., Johnson, K.: Neural coding of tactile texture: comparison of spatial and temporal mechanisms for roughness perception. J. Neurosci. 12, 3414–3426 (1972) 15. Connor, C., Hsiao, S., Philips, J., Johnson, K.: Tactile roughness: neural codes that account for pscyhophysical magnitude estimates. J. Neurosci. 10, 3823–3836 (1990) 16. Demarest, K.R.: Engineering Electromagnetics. Prentice Hall, Upper Saddle River (1998) 17. Derler, S., Gerhardt, L.: Tribology of skin: review and analysis of experimental results for the friction coefficient of human skin. Tribol. Lett. 45, 1–27 (2012) 18. Derler, S., Gerhardt, L.C., Lenz, A., Bertaux, E., Hadad, M.: Friction of human skin against smooth and rough glass as a function of the contact pressure. Tribol. Int. 42(11–12), 1565– 1574 (2009) 19. Emgin, S.E., Ege, E., Birer, O., Basdogan, C.: Localized multifinger electrostatic haptic display. In: Hands-on Demonstration in IEEE World Haptics (2015) 20. Emgin, S.E., Sadia, B., Vardar, Y., Basdogan, C.: Enhancing human-computer interaction via electrovibration. In: Demo in IEEE World Haptics (2017) 21. Emgin, S.E., Aghakhani, A., Sezgin, T.M., Basdogan, C.: Haptable: an interactive tabletop providing online haptic feedback for touch gestures. IEEE Trans. Visual. Comput. Graph. 25(9), 2749–2762 (2019) 22. Enriquez, M., MacLean, K.: Backward and common-onset masking of vibrotactile stimuli. Brain Res. Bull. 75, 761–769 (2008)

References

37

23. Fiedler, T., Vardar, Y.: A novel texture rendering approach for electrostatic displays. In: Proceedings of the International Workshop on Haptic and Audio Interaction Design (HAID) (2019) 24. Fiedler, T., Vardar, Y., Strese, M., Steinbach, E., Basdogan, C.: Reproduction of textures based on electrovibration. In: Hands-on Demonstration in IEEE World Haptics (2017) 25. Fukuda, T., Shimazu, K., Hashimoto, Y.: Basic characteristics of printable large-area electrostatic tactile display*. In: 2019 IEEE World Haptics Conference (WHC), pp. 497–502 (2019) 26. Gescheider, G., Migel, N.: Vibrotactile forward masking 2: some temporal parameters in vibrotactile forward masking. J. Acoust. Soc. Am. 98, 3195–3199 (1995) 27. Gescheider, G., Verrillo, R., Doren, C.V.: Prediction of vibrotactile masking functions. J. Acoust. Soc. Am. 72, 1421–1426 (1982) 28. Gescheider, G., Malley, J., Verrillo, R.: Vibrotactile forward masking: evidence for channel independence. J. Acoust. Soc. Am. 74, 474–485 (1983) 29. Gescheider, G., Bolanowski, S., Verrillo, R.: Vibrotactile masking: effects of stimulus onset asynchrony and stimulus frequency. J. Acoust. Soc. Am. 85, 2059–2064 (1989) 30. Gescheider, G.A., Bolanowski, S.J., Verrillo, R.T., Arpajian, D.J., Ryan, T.F.: Vibrotactile intensity discrimination measured by three methods. J. Acoust. Soc. Am. 87(330), 330–338 (1990) 31. Gescheider, G., Hoffman, E., Harrison, M., Travis, M., Bolanowski, S.: The effects of masking on vibrotactile temporal summation in the detection of sinusoidal and noise signals. J. Acoust. Soc. Am. 95, 1006–1016 (1994) 32. Gescheider, G., Santoro, E., Makous, J., Bolanowski, S.: Vibrotactile forward masking 1: effects of the amplitude and duration of the masking stimulus. J. Acoust. Soc. Am. 98, 3188– 3194 (1995) 33. Gescheider, G., Bolanowski, S., Hardick, K.: The frequency selectivity of informationprocessing channels in the tactile sensory systems. Somatosens. Mot. Res. 18, 191–201 (2001) 34. Gescheider, G.A., Bolanowski, S.J., Pope, J.V., Verrillo, R.T.: A four-channel analysis of the tactile sensitivity of the fingertip: frequency selectivity, spatial summation, and temporal summation. Somatosens. Mot. Res. 19(2), 114–124 (2002) 35. Gescheider, G.A., Wright, J., Verrillo, R.: Information-Processing Channels in the Tactile Sensory System a Psychophysical and Psysiological Analysis. Psychology Press (2009) 36. Gilson, R.D.: Vibrotactile masking: some spatial and temporal aspects. Percept. Psychophys. 5(3), 176–180 (1969) 37. Grimnes, S.: Electrovibration, cutaneous sensation of microampere current. Acta Physiol. Scand. 118, 19–25 (1983) 38. Güçlü, B.: Deviation from Weber’s law in the non-pacinian i tactile channel: a psychophysical and simulation study of intensity discrimination. Neural Comput. 19(10), 2638–2664 (2007) 39. Güçlü, B., Bolanowski, S.: Probability of stimulus detection in a model population of rapidly adapting fibers. Neural Comput. 16(1), 39–58 (2004) 40. Güçlü, B., Bolanowski, S.: Vibrotactile thresholds of the non-pacinian I channel: I. methodological issues. Somatosens. Mot. Res. 22, 49–56 (2005) 41. Güçlü, B., Bolanowski, S.: Vibrotactile thresholds of the non-pacinian I channel: II. predicting the effects of contactor location on the phalanx. Somatosens. Mot. Res. 22(1/2), 57–68 (2005) 42. Güçlü, B., Bolanowski, S., Istefanopulos, Y.: Population-response model for vibrotactile spatial summation. Somatosens. Mot. Res. 22(4), 239–253 (2005) 43. Güçlü, B., Öztek, C.: Tactile sensitivity of children: effects of frequency, masking, and the non-pacinian I psychophysical channel. J. Exp. Child Psychol. 98, 113–130 (2007) 44. Güçlü, B., S.M. ¸ Dinçer: Neural coding in the non-pacinian I tactile channel: a psychophysical and simulation study of magnitude estimation. Somatosens. Mot. Res. 30(1), 1–15 (2013) 45. Guo, X., Zhang, Y., Wang, D., Lu, L., Xu, W.: The effect of applied normal force on the electrovibration. IEEE Trans. Haptics 12(4), 571–580 (2019)

38

2 Background

46. Guo, X., Zhang, Y., Wei, W., Xu, W., Wang, D.: Effect of temperature on the absolute and discrimination thresholds of voltage on electrovibration tactile display. IEEE Trans. Haptics (2019). https://doi.org/10.1109/TOH.2019.2962111 47. Haga, H.: Visual display with multi-finger tactile interaction. Spie (2016). https://doi.org/10. 1117/2.1201611.006686 48. Hamer, R.D., Verrillo, R., Zwislocki, J.: Vibrotactile masking of pacinian and non-pacinian channels. J. Acoust. Soc. Am. 73, 1293–1303 (1983) 49. Hollins, M., Faldowski, R., Rao, R., Young, F.: Perceptual dimensions of tactile surfaced texture: a multidimensional scaling analysis. Percept. Psychophys. 54(6), 697–705 (1993) 50. Hollins, M., Bensmaïa, S., Karlof, K., Young, F.: Individual differences in perceptual space for tactile textures: evidence from multidimensional scaling. Percept. Psychophys. 62(8), 1534–1544 (2000) 51. Hollins, M., Bensmaia, S., Washburn, S.: Vibrotactile adaptation impairs discrimination of fine, but not coarse, textures. Somatosens. Mot. Res. 18(4), 253–262 (2001) 52. Ilkhani, G., Aziziaghdam, M., Samur, E.: Data-driven texture rendering on an electrostatic tactile display. Int. J. Hum.–Comput. Interact. 33(9), 756–770 (2017) 53. Ilkhani, G., Samur, E.: Creating multi-touch haptic feedback on an electrostatic tactile display. In: Proceedings of the IEEE Haptics Symposium, pp. 163–168 (2018) 54. Ishizuka, H., Hatada, R., Cortes, C., Miki, N.: Development of a fully flexible sheet-type tactile display based on electrovibration stimulus. Micromachines 9(5) (2018). https://doi. org/10.3390/mi9050230 55. Isleyen, A., Vardar, Y., Basdogan, C.: Tactile roughness perception of virtual gratings by electrovibration. IEEE Trans. Haptics (2019) 56. Israr, A., Bau, O., Kim, S.C., Poupyrev, I.: Tactile feedback on flat surfaces for the visually impaired. In: Proceedings of the Extended Abstracts on Human Factors in Computing Systems (CHI’12), pp. 1571–1576 (2012) 57. Jamalzadeh, M., Çaˇgatay, B., Vardar, Y., Basdogan, C.: Effect of remote masking on detection of electrovibration. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 229– 234 (2019) 58. Jiao, J., Wang, D., Zhang, Y., Cao, D., Visell, Y., Guo, X., Sun, X.: Detection and discrimination thresholds for haptic gratings on electrostatic tactile displays. IEEE Trans. Haptics 12(1) (2018) 59. Jiao, J., Zhang, Y., Wang, D., Wisell, Y., Cao, D., Guo, X., Sun, X.: Data-driven rendering of fabric textures on electrostatic tactile displays. In: Proceedings of the IEEE Haptics Symposium, pp. 169–174 (2018) 60. Johnsen, A., Rahbek, K.: A physical phenomenon and its applications to telegraphy, telephony, etc. IEEE J. Inst. Elect. Eng. 61(320), 713–725 (1923) 61. Johnson, K.: The roles and functions of cutenous mechanoreceptors. Curr. Opin. Neurobiol. 11, 455–461 (2001) 62. Jones, L.A.: Perspectives on the evolution of tactile, haptic, and thermal displays. Presence 25(3), 247–252 (2016) 63. Jones, L., Sarter, N.: Tactile displays: guidance for their design and application. Hum. Factors J. Hum. Factors Ergon. Soc. 50(90), 90–111 (2008) 64. Kaczmarek, K., Nammi, K., Agarwal, A., Tyler, M., Haase, S., Beebe, D.: Polarity effect in electrovibration for tactile display. IEEE Trans. Biomed. Eng. 53(10), 2047–2054 (2006) 65. Kang, J., Kim, H., Choi, S., Kim, K.D., Ryu, J.: Investigation on low voltage operation of electrovibration display. IEEE Trans. Haptics 10(3), 371–381 (2017) 66. Kim, S., Israr, A., Poupyrev, I.: Tactile rendering of 3D features on touch surfaces. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology UIST, pp. 531–538 (2013). St. Andrews 67. Kim, H., Kang, J., Lim, K., Ryu, J.: Method for providing electrovibration with uniformed density. IEEE Trans. Haptics 8(4), 492–496 (2015) 68. Klatzky, R., Lederman, S., Hamilton, C., Grindley, M., Swendsen, R.: Feeling textures through a probe. Percept. Psychophys. 65(4), 613–631 (2003)

References

39

69. Klatzky, R.L., Lederman, S.J.: Multisensory texture perception. In: Multisensory Object Perception in the Primate Brain, pp. 211–230. Springer, New York (2010) 70. Kuchenbecker, K.J.: Haptics and Haptic Interfaces, pp. 1–9. Springer, Berlin/Heidelberg (2018) 71. Kuroki, S., Watabene, J., Nishida, S.: Integration of vibrotactile frequency information beyond the mechanoreceptor channel and somatotopy. Sci. Reports 7(2778) (2017). https://doi.org/ 10.1038/s41598-017-02922-7 72. Lawrance, M., Ryo, K., Klatzky, R., Lederman, S.: Haptic roughness perception of linear gratings via bare finger or rigid probe. Perception 36, 547–557 (2007) 73. Lederman, S.: Tactual roughness perception: spatial and temporal determinants. Can. J. Psychol. 37(4), 498–511 (1983) 74. Lederman, S.J., Klatzky, R.L.: Haptic perception: a tutorial. Atten. Percept. Psychophys. 71(7), 1439–1459 (2009) 75. Lederman, S., Taylor, M.: Fingertip force, surface geometry, and the perception of roughness by active touch. Percept. Psychophys. 12, 401–408 (1972) 76. Lederman, S., Thorne, G., Jones, B.: Perception of texture by vision and touch: multidimensionality and intersensory integration. J. Exp. Psychol. Hum. Percept. Perfor. 12, 169–180 (1972) 77. Lederman, S., Klatzky, R., Hamilton, C., Ramsay, G.: Perceiving roughness via a rigid probe: psychophysical effects of exploration speed and mode of touch. Haptics-e Electron. J. Haptics Res. 1, 1–20 (1999) 78. Li, X., Choi, C., Ma, Y., Boonpuek, P., Felts, J.R., Mullenbach, J., Shultz, C., Colgate, J.E., Hipwell, M.C.: Electrowetting: a consideration in electroadhesion. IEEE Trans. Haptics (2020). https://doi.org/10.1109/TOH.2020.2979439 79. Lim, J., Yoo, Y., Cho, H., Choi, S.: Touchphoto: enabling independent picture taking and understanding for visually-impaired users. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 124–134 (2019) 80. Linjama, J., Mäkinen, V.: E-sense screen: novel haptic display with capacitive electrosensory interface. In: Proceedings of the 4th Workshop for Haptic and Audio Interaction Design (HAID’09). Dresden (2009) 81. Mahns, D.A., Perkins, N.M., Sahai, V., Robinson, L., Rowe, M.J.: Vibrotactile frequency discrimination in human hairy skin. J. Neurophysiol. 95, 1442–1450 (2006) 82. Makous, J., Friedman, R., Vierck, C.J.: A critical band filter in touch. J. Neurosci. 15(4), 2808–2818 (1995) 83. Makous, J., Gescheider, G., Bolanowski, S.: Decay in the effect of vibrotactile masking. J. Acoust. Soc. Am. 99(2), 1124–1129 (1995) 84. Mallinckrodt, E., Hughes, A., Sleator, W.J.: Perception by the skin of electrically induced vibrations. Science 118(3062), 277–278 (1953) 85. Meyer, D.J., Peshkin, M.A., Colgate, J.E.: Fingertip friction modulation due to electrostatic attraction. In: Proceedings of the IEEE World Haptics Conference 2013, pp. 43–48 (2013) 86. Mowbray, G.H., Gebhard, J.W.: Sensitivity of the skin to changes in rate of intermittent mechanical stimuli. Science 125, 1297–1298 (1957) 87. Mun, S., Lee, H., Choi, S.: Perceptual space of regular homogeneous haptic textures rendered using electrovibration. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 7–12 (2019) 88. Nakamura, T., Yamamoto, A.: A multi-user surface visuo-haptic display using electrostatic friction modulation and capacitive-type position sensing. IEEE Trans. Haptics 9(3), 311–322 (2016) 89. Nakamura, T., Yamamoto, A.: Modeling and control of electroadhesion force in DC voltage. Robomech J. 4(18) (2017). https://doi.org/10.1186/s40648-017-0085-3 90. Okamoto, S., Nagano, H., Yamada, Y.: Psychophysical dimensions of tactile perception of textures. IEEE Trans. Haptics 6(1), 81–93 (2012) 91. Osgouei, R.H., Kim, J.R., Choi, S.: Improving 3d shape recognition with electrostatic friction display. IEEE Trans. Haptics 10(4) (2017)

40

2 Background

92. Osgouei, R.H., Shin, S., Kim, J.R., Choi, S.: An inverse neural network model for datadriven texture rendering on electrovibration display. In: Proceedings of the IEEE Haptics Symposium, pp. 270–277 (2018) 93. Ozdamar, I., Alipour, S.M., Delhaye, B.P., Basdogan, C.: Step-change in friction under electrovibration. IEEE Trans. Haptics 15 (2019) 94. Pasumarty, S.M., Johnson, S.A., Watson, S.A., Adams, M.J.: Friction of the human fingerpad: influence of moisture, occlusion, and velocity. Tribol. Lett. 44, 1–27 (2011) 95. Persson, B.: The dependency of adhesion and friction on electrostatic attraction. J. Chem. Phys. 148(14), 144701 (2018) 96. Rothenberg, M., Verrillo, R.T., Zahorian, S.A., Brachman, M.L., Bolanowski, S.J.: Vibrotactile frequency for encoding a speech parameter. J. Acoust. Soc. Am. 62, 1003–1012 (1977) 97. Ryu, S., Pyo, D., Lim, S., Kwon, D.: Mechanical vibration influences the perception of electrovibration. Sci. Reports 4(4555) (2018). https://doi.org/10.1038/s41598-01822865-x 98. Shizuka, H., Komurasaki, S., Kato, K., Kajimato, H.: Evaluation of electrovibration stimulation with a narrow electrode. Micromachines (2018). https://doi.org/10.3390/mi9100483 99. Shultz, C.D., Peshkin, M.A., Colgate, J.E.: Surface haptics via electroadhesion: expanding electrovibration by Johnsen and Rahbek. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 57–62 (2015) 100. Shultz, C.D., Peshkin, M.A., Colgate, J.E.: On the electrical characterization of electroadhesive displays and the prominent interfacial gap impedance associated with sliding fingertips. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 151–157 (2018) 101. Sirin, O., Barrea, A., Lefevre, P., Thonnard, J.L., Basdogan, C.: Fingerpad contact evolution under electrocvibration. J. R. Soc. 16(156) (2019) 102. Skedung, L.: Tactile perception: role of friction and texture. Ph.D. thesis, KTH Royal Institute of Technology (2012) 103. Smith, T.A., Gorlewicz, J.L.: Hue: a hybrid ultrasonic and electrostatic variable friction touchscreen. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 635–640 (2017) 104. Smith, A.M., Chapman, C.E., Deslandes, M., Langlais, J.S., Thibodeau, M.P.: Role of friction and tangential force variation in the subjective scaling of tactile roughness. Exp. Brain Res. 144(2), 211–223 (2002) 105. Spirkovska, L.: Summary of tactile user interfaces techniques and systems. Technical report, NASA Ames Research Center (2005) 106. Strong, R.M., Troxel, D.E.: An electrotactile display. IEEE Trans. Man-Mach. Syst. 11(1), 72–79 (1970) 107. Tactile Feedback Devices: Friction Control and Texture Generation. Ph.D. thesis, University of Lille (2016) 108. Tang, H., Beebe, D.: A microfabricated electrostatic haptic display for persons with visual impairments. IEEE Trans. Rehabil. Eng. 6(3), 241–248 (1998) 109. Tomlinson, S., Lewis, R., Carre, M.: Review of the frictional properties of finger-object contact when gripping. Proc. Inst. Mech. Eng. Part J J. Eng. Tribol. 221(8), 841–850 (2007) 110. Vardar, Y., Güçlü, B., Basdogan, C.: Effect of waveform in haptic perception of electrovibration on touchscreens. In: Haptics: Perception, Devices, Control, and Applications: 10th International Conference, EuroHaptics 2016, London, 4–7 July 2016, Proceedings, Part I, pp. 190–203 (2016) 111. Vardar, Y., Güçlü, B., Basdogan, C.: Effect of waveform on tactile perception by electrovibration displayed on touch screens. IEEE Trans. Haptics 10(4), 488–499 (2017) ˙sleyen, A., Saleem, M., Basdogan, C.: Roughness perception of virtual textures 112. Vardar, Y., I¸ displayed by electrovibration on touch screens. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 263–268 (2017) 113. Vardar, Y., Güçlü, B., Basdogan, C.: Tactile masking by electrovibration. IEEE Trans. Haptics 11(4), 623–635 (2018)

References

41

114. Verrilo, R.T., Fraioli, A.J., Smith, R.L.: Sensation magnitude of vibrotactile stimuli. Percept. Psychophys. 6(6A), 366–372 (1969) 115. Verrillo, R.T., Geschieder, G.A., Calman, B.G., van Doren, C.L.: Vibrotactile masking: effects of one-and two-site stimulation. Percept. Psychophys. 33(4), 379–387 (1983) 116. Vezzoli, E., Amberg, M., Giraud, F., Lemaire-Semail, B.: Electrovibration modeling analysis. In: Haptics: Neuroscience, Devices, Modeling, and Applications, pp. 369–376. Springer, Berlin/Heidelberg (2014) 117. Vezzoli, E., Messaoud, W.B., Amberg, M., Giraud, F., Lemaire-Semail, B., Bueno, M.A.: Physical and perceptual independence of ultrasonic vibration and electrovibration for friction modulation. IEEE Trans. Haptics 8(2), 235–239 (2015) 118. Vodlak, T., Vidrih, Z., Vezzoli, E., Lemaire-Semail, B., Peric, D.: Multi-physics modelling and experimental validation of electrovibration based haptic devices. Biotribology 8, 12–25 (2016) 119. Wang, Q., Ren, X., Sarcar, X., Sun, X.: Ev-pen: leveraging electrovibration haptic feedback in pen interaction. In: ISS’16, Proceedings of the 2016 ACM on Interactive Surfaces and Spaces, pp. 57–66 (2016) 120. Wang, D., Guo, Y., Liu, S., Zhang, Y., Xu, W., Xiao, J.: Haptic display for virtual reality: progress and challenges. Virtual Real. Intell. Hardware 1(2), 136–162 (2019) 121. Watabene, T., Fukui, S.: Method for controlling tactile sensation of surface roughness using ultrasonic vibration. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 1134–1139 (1995) 122. Weinstein, S.: Intensive and Extensive Aspects of Tactile Sensitivity as a Function of Body Part, Sex, and Laterality, pp. 195–222. Thomas, Springfield (2018) 123. Wiertlewski, M.: Reproduction of tactual textures: transducers, mechanics, and signal encoding. Ph.D. thesis, Université Pierre et Marie Curie Paris VI (2012) 124. Wijekoon, D., Cecchinato, M., Hoggan, E., Linjama, J.: Electrostatic modulated friction as tactile feedback: intensity perception. In: Haptics: Perception, Devices, Mobility, and Communication Lecture Notes in Computer Science, vol. 7282, pp. 613–624 (2012) 125. Wu, S., Sun, X., Wang, Q., Chen, J.: Tactile modeling and rendering image-textures based on electrovibration. Vis. Comput. 1–10 (2016) 126. Xu, C., Israr, A., Bau, O., Kim, S., Poupyrev, I.: Tactile feedback on flat surfaces for the visually impaired. In: CHI’12 Extended Abstracts on Human Factors in Computing Systems, pp. 1571–1576 (2012) 127. Xu, H., Klatzky, R.L., Peshkin, M.A., Colgate, J.E.: Localized rendering of button click sensation via active lateral force feedback. In: Proceedings of the IEEE World Haptics Conference (WHC), pp. 509–514 (2019) 128. Yamamato, A., Nagasawa, S., Yamamato, H., Higuchi, T.: Electrostatic tactile display with thin film slider and its application to tactile tele-presentation system. IEEE Trans. Visual. Comput. Graph. 12(2), 168–176 (2006) 129. Yoshioka, T., Bensmaia, S.J., Craig, J.C., Hsiao, S.S.: Texture perception through direct and indirect touch: an analysis of perceptual space for tactile textures in two modes of exploration. Somatosens. Mot. Res. 24(1–2), 53–70 (2007) 130. Zhang, Y., Harrison, C.: Quantifying the targeting performance benefit of electrostatic haptic feedback on touchscreens. In: Proceedings of the 2015 International Conference on Interactive Tabletops and Surface, ITS’15, pp. 43–46 (2015)

Chapter 3

Effect of Waveform on Tactile Perception by Electrovibration

Abstract In this chapter, we investigated the effect of input voltage waveform on our tactile perception of electrovibration on touch screens. Through psychophysical experiments performed with eight subjects, we first measured the detection thresholds of electrovibration stimuli generated by sinusoidal and square voltages at various fundamental frequencies. We observed that the subjects were more sensitive to stimuli generated by square wave voltage than sinusoidal one for frequencies lower than 60 Hz. Using Matlab simulations, we showed that the sensation difference of waveforms in low fundamental frequencies occurred due to the frequency-dependent electrical properties of human skin and human tactile sensitivity. To validate our simulations, we conducted a second experiment with another group of eight subjects. We first actuated the touch screen at the threshold voltages estimated in the first experiment and then measured the contact force and acceleration acting on the index fingers of the subjects moving on the screen with a constant speed. We analyzed the collected data in the frequency domain using the human vibrotactile sensitivity curve. The results suggested that Pacinian channel was the primary psychophysical channel in the detection of the electrovibration stimuli caused by all the square-wave inputs tested in this study. We also observed that the measured force and acceleration data were affected by finger speed in a complex manner suggesting that it may also affect our haptic perception accordingly. Keywords Human tactile sensing · Fingerpad · Human skin · Sensory receptors · Surface haptic displays · Electrovibration

3.1 Introduction As we discussed in Chap. 2, electrovibration is a promising technology for providing rich tactile sensations on future electronic devices. However, the number of applications of this technology is limited yet due to our poor understanding of the electrical and mechanical properties of human finger and its interaction with a touch surface. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 Y. Vardar, Tactile Perception by Electrovibration, Springer Series on Touch and Haptic Systems, https://doi.org/10.1007/978-3-030-52252-0_3

43

44

3 Effect of Waveform on Tactile Perception by Electrovibration

For example, both the electrical and mechanical impedance of the human finger are frequency-dependent, and the coupling between them has not been well understood yet [4, 27, 30, 38]. Moreover, human to human variability of these properties and the influence of the environmental factors on these properties further complicate the problem. In addition to the physical factors mentioned above, it is known that human tactile (mechanical) perception varies with stimulation amplitude and frequency [16]. Even though the effects of amplitude and frequency on the human tactile perception of electrovibration have already been investigated using pure sine waves [4], there is no earlier study on how our perception changes when another waveform is used. In this chapter, we investigate how input voltage waveform alters human haptic perception of electrovibration. This work is mainly motivated by our initial observation that square-wave excitation causes stronger vibratory sensation than sine-wave excitation. According to the parallel-plate capacitor principle, the electrostatic force is proportional to the square of the input voltage signal, hence the electrostatic force generated by a square-wave is supposed to be constant [8, 37]. Since DC (constant) excitation voltages do not cause vibration sensation (though it causes adhesion sensation as reported in [24, 34]), the square wave excitation is expected to be filtered electrically by the stratum corneum. This filtering suppresses the lowfrequency components in the excitation voltage and generates an electrostatic force with a distorted waveform. We hypothesise that the stronger vibratory sensation caused by a square wave is due to the high-frequency components in the resulting force signal. Since this waveform is rather complex (contains many frequency components), it can activate different psychophysical channels at different threshold levels [6, 16]. These four psychophysical channels (NPI, NPII, NPIII, and P) are mediated by four corresponding mechanoreceptors and enable tactile perception. To predict tactile sensitivity, the Fourier components of the waveform should be analyzed by considering human sensitivity curve [16]. Here, using a simulation model developed in Matlab-Simulink, we first show that the forces displayed to human finger by electrovibration are very different for square and sinusoidal input voltages at low fundamental frequencies due to electrical filtering. Then, we show that the force waveform generated by square-wave excitation contains high-frequency components to which human tactile sensation is more sensitive. We support this claim by presenting the results of two experiments conducted with eight subjects. In the first experiment, we measure the detection threshold voltages for sinusoidal and square signals at various frequencies. In the second experiment, we actuate the touch screen at those threshold voltages and measure the contact force and acceleration acting on the index finger of subjects moving on the touch screen with a constant speed. We analyze the collected data in frequency domain by taking into account the human sensitivity curve and show that the square wave excites mainly Pacinian channel [22, 42]. Our results also suggest that scan speed has a significant effect on measured acceleration and force data and potentially on our haptic perception.

3.2 Waveform Analysis of Electrovibration

45

3.2 Waveform Analysis of Electrovibration To investigate the effect of waveform in electrovibration, we developed an equivalent circuit model of human finger in Matlab-Simulink environment. In this model, we neglected the capacitance of the human body and air gap, as its effect on the electrostatic force mostly magnitude-wise. and also the internal resistance of the touch screen (see Fig. 3.1). The capacitance of the touch screen was calculated based on the properties of a commercial touch screen1 (3M Inc.), which was also used in our experiments. Previous studies showed that the human skin (especially sweat ducts and the stratum corneum) is not a perfect dielectric and has frequencydependent resistive properties [18, 25, 26, 40]. Therefore, we modelled stratum corneum as a resistance and a capacitance in parallel. In [38], Vezzoli et al. used frequency-dependent values of resistivity, ρsc , and dielectric constant, sc , of human stratum corneum reported by [40]. Their simulations showed that intensity of electrovibration was highly frequency-dependent. Similarly, we fitted polynomial functions to the experimental data reported by [40] and used those functions in our Matlab simulations (see Fig. 3.2). (s) Figure 3.3 represents the Bode plot of the transfer function VVsc(s) , estimated by using the values tabulated in Table 3.1. The system displays the behavior of a

Fig. 3.1 The simplified equivalent circuit model of human finger on a touch surface

1 The

touch screen (3M Inc.) that we used in our simulations and later in our experiments is a commercial product. It is originally designed for capacitive-based touch sensing and composed of a transparent conductive sheet coated with an insulator layer on top of a glass plate. To generate haptic effects via electrovibration, the conductive sheet is excited by applying a voltage signal through the connectors designed for position sensing.

46

3 Effect of Waveform on Tactile Perception by Electrovibration 10 5

ρ sc [Ω m], sc

10 4

10 3

ρsc literature sc literature

10 2

ρsc fit function sc fit function

10 1

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Frequency [Hz] Fig. 3.2 The experimental values of resistivity and dielectric constant of stratum corneum as reported in [40] and the polynomial functions fitted to them

Magnitude[dB]

0 -20 -40 -60 10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 1

10 2

10 3

10 4

10 5

10 6

Angle [Degree]

40 20 0 -20 -40 -60 10 0

Frequency [Hz] Fig. 3.3 The transfer function between Vsc and V

3.2 Waveform Analysis of Electrovibration

47

Table 3.1 The description of the parameters used in the circuit model and the corresponding values used in the Matlab simulations Parameter A 0 Rbody Ci

Explanation Area of the human fingertip Permittivity of vacuum Resistance of human body [27] Capacitance of the 3M MicroTouch

Value 1 8.854 × 10−12 1 Ci = 0dii A

Unit cm2 F/m k F

i di Rsc Csc

Relative permittivity of the insulator Thickness of the insulator Resistance of stratum corneum Capacitance of stratum corneum

3.9 1 Rsc = Csc =

– μm  F

ρsc sc

Resistivity of stratum corneum Relative permittivity of the stratum corneum

Fig. 3.2 Fig. 3.2

ρsc dsc A 0 sc A dsc

m –

bandpass filter with cut-off frequencies, flow , and, fhigh , at approximately 1 and 20 kHz respectively. Hence, it shows a first order high pass filter behaviour up to 1 kHz, which can cause distortions on the voltage that is transmitted to stratum corneum at low frequencies. To test the effects of this electrical filtering, we performed simulations with two different input waveforms (sinusoidal and square) at two fundamental frequencies (15 and 480 Hz). Figure 3.4 shows the input voltage signal, the voltage across stratum corneum (filtered signals), and the resultant electrostatic force transmitted to mechanoreceptors for both waveforms at low and high frequencies (Fig. 3.4a, b). In low-frequency case (15 Hz), when the input is a sinusoidal signal, the output force signal is phase-shifted, and its amplitude drops significantly. Whereas, for a square wave signal, the output contains exponentially decaying relatively higher amplitude transients. In the high-frequency case (480 Hz), the decline in the output amplitude of the sinusoidal signal is much less, as expected from high pass filtering. Also, the output of the square signal resembles the input signal more because the signal alternates faster than the discharge rate of the capacitor formed by the human skin and touch screen insulator. The results depict that the stimuli on the mechanoreceptors have different waveform and amplitude than those of the input voltage signal. If a complex waveform (containing many frequency components) arrives at mechanoreceptors, it can activate different psychophysical channels at different threshold levels [2, 6, 16]. These four psychophysical channels (NPI, NPII, NPIII, P) are mediated by four corresponding mechanoreceptor populations, which enable the tactile perception[6, 16, 20, 22, 41]. For this reason, the Fourier components of the stimulus should be weighted with the inverse of the human sensitivity curve to predict tactile sensitivity to complex stimuli [16]. The stimulus detection occurs at the channel where the maximum of this weighted function is located in the

48

3 Effect of Waveform on Tactile Perception by Electrovibration LOW FREQUENCY CASE (15 Hz)

Voltage on stratum corneum 100

50

50

0 -50

0

0.02

-50

0.1 sec

100

50

50 Vsc [V]

100

0 -50

0.1 sec

0.05

F [N] e

0.1 sec

V [V]

0.05

-100

-100

Square Input

Force on mechanoreceptors

F [N] e

100

Vsc [V]

V [V]

Sinusoidal Input

Input voltage to touchscreen

0

0.02

-50 -100

-100

0.1 sec

0.1 sec

0.1 sec

(a) HIGH FREQUENCY CASE (480 Hz)

Voltage on stratum corneum 100

50

50

0 -50

0

0.02

-50

0.01 sec

100

50

50 Vsc [V]

100

0 -50

0.01 sec

0.05

F [N] e

0.01 sec

V [V]

0.05

-100

-100

Square Input

Force on mechanoreceptors

F [N] e

100

Vsc [V]

V [V]

Sinusoidal Input

Input voltage to touchscreen

0

0.02

-50 -100

-100 0.01 sec

0.01 sec

(b) Fig. 3.4 Simulation results: (a) low frequency case, (b) high frequency case

0.01 sec

3.3 Materials and Methods

49

frequency domain. For example, a sinusoidal signal contains a single frequency component. To be able to detect this signal, its energy level must be higher than the human sensation threshold at that frequency. However, a square signal contains many frequency components. Detection occurs as soon as the energy level of one frequency component is higher than the human sensation threshold at that frequency. The tactile detection process for electrovibration is illustrated in Fig. 3.5. Here, a sinusoidal and a square voltage signals at the same fundamental frequency but different amplitude are applied to the touch screen. Due to electrical filtering of human finger, they generate electrostatic forces on the mechanoreceptors with the same amplitude. Therefore, the energy in 30 Hz component is the same for both force signals shown in Fig. 3.5c. However, the square wave input has higher frequency components, which are weighted more with respect to the human sensitivity curve (Fig. 3.5d). As a result, the weighted force signal contains a relatively high frequency component of 180 Hz (Fig. 3.5e). Therefore, in this illustration, the square wave is detected, but the sinusoidal wave is not.

3.3 Materials and Methods 3.3.1 Experiment 1: Psychophysical Experiments To investigate how our detection threshold changes with input waveform, we conducted absolute detection experiments. These experiments enable us to determine the minimum voltage amplitude that the observer can barely detect [13, 21, 22, 41]. We aim to compare detection thresholds for sinusoidal and square wave voltage inputs at different frequencies to support our arguments made in Sect. 3.2.

3.3.1.1

Participants

We performed experiments with eight subjects (four female, four male) having an average age of 27.5 (SD: 1.19). All of the subjects were right-handed except one. All of them were engineering Ph.D. students. The subjects used the index finger of their dominant hand during the experiments. They washed their hands with soap and rinsed with water before the experiment. Also, their fingers and the touch screen were cleaned by alcohol before each measurement. The subjects read and signed the consent form before the experiments. The form was approved by Ethical Committee for Human Participants of Koç University.

Sinusoidal Input at 15 Hz

b)

a)

0.1 sec

0.1 sec

a

a

Force signal after filtering

TOUCH SCREEN

10

10

c)

600 Hz

30 Hz

600 Hz

30 Hz

Energy of force signal

FINGERPAD

ELECTRICAL FILTERING + ELECTROSTATIC FORCE GENERATION

0.1 sec

0.1 sec

INPUT VOLTAGE

(Meissner)

(Merkel)

d)

NP I

1

P

50

600 Hz

(Pacinian)

Human sensitivity curve

NP III

0.1

SENSORY PROCESSING IN THE BRAIN

Frequency-dependent pscyhophysical channels

MECHANORECEPTORS

HUMAN SENSING

10

10

e)

180 Hz

600 Hz

600 Hz

30 Hz

Weighted energy of force signal

DECISION NETWORK

DECISION PROCESS

f)

DETECTED!

NOT DETECTED!

PROBABILITY OF DETECTION

Fig. 3.5 An illustration of how tactile detection occurs. (a) Input sinusoidal and square voltage signals at 15 Hz applied to touch screen at different amplitudes. (b) These input signals are filtered electrically by human finger (see Fig. 3 for filtering process) before generating electrostatic forces with the same amplitude on the mechanoreceptors. (c) The energy of the force signal originated from the sinusoidal wave contains only one frequency component (30 Hz due to squaring in Equation 2.2) while the one from the square wave contains many frequency components. (d) The frequency-dependent human sensitivity curve; the most sensitive frequency regions of three psychophysical channels are color-coded. The fourth channel (NPII) does not appear in this illustration. (e) When the Fourier components of the force signals are weighted by the inverse of the human sensitivity curve, the resulting signals from the sinusoidal and square waves have their maximum peaks at 30 and 180 Hz, respectively. Moreover, the energy of the frequency component for the square wave case is larger than that of the sinusoidal one at those frequencies. (f) Therefore, the square signal is detected, but the sinusoidal signal is not

Square Input at 15 Hz

50 3 Effect of Waveform on Tactile Perception by Electrovibration

3.3 Materials and Methods

51

Fig. 3.6 Experimental setup used in our psychophysical experiments

3.3.1.2

Stimuli

We estimated absolute detection thresholds for seven input frequencies (15, 30, 60, 120, 240, 480 and, 1920 Hz) and two waveforms (sinusoidal and square).

3.3.1.3

Experimental Setup

The experimental setup used for the psychophysical experiment is shown in Fig. 3.6. A touch screen (SCT3250, 3M Inc.) was placed on top of an LCD screen. An IR frame was placed above the touch screen to detect the finger location. The touch screen was excited with a voltage signal generated by a DAQ card (PCI6025E, National Instruments Inc.) and augmented by an amplifier (E-413, PI Inc.). Subjects entered their responses through a computer monitor. An arm rest supported the subjects’ arms during the experiments. For isolation of the background noises, subjects were asked to wear headphones displaying white noise during experiments.

52

3.3.1.4

3 Effect of Waveform on Tactile Perception by Electrovibration

Procedure

We used the two-alternative-forced-choice method to determine the detection thresholds. This method enables criterion-free experimental results [22]. We displayed two regions (A and B) on the LCD screen (Fig. 3.6). Tactile stimulus was displayed in only one of the regions, and its location was randomized. The finger position of the subjects was detected via the IR frame. The subjects were asked to explore both areas consecutively and choose the one displaying a tactile stimulus. We changed the amplitude of the tactile stimulus via one-up/two-down adaptive staircase method. This procedure decreases the duration of the experimentation by reducing the number of trials [19, 20, 28, 29, 41]. We started each session with the stimulus amplitude of 100 V. This initial voltage amplitude provided sufficiently high-intensity stimulus for all the subjects. The voltage amplitude of the new stimulus was adjusted adaptively based on the past responses of each subject. If the subject gave two consecutive correct answers, the voltage amplitude was decreased by 10 V. If the subject had one incorrect response, the stimulus intensity was increased by 10 V. The change of the response from correct to incorrect or the vice versa was counted as one reversal. After four reversals, the step size was decreased by 2 V to obtain a more precise threshold value, as suggested in [4]. We stopped the experiment after 18 reversals and estimated the absolute detection threshold as the average of the last 15 reversals (Fig. 3.7). The subjects completed the experiments in 14 sessions, executed in 7 separate days (two sessions per day). The duration of each session was about 15–20 min.

3.3.2 Experiment 2: Force & Acceleration Measurements We measured the contact forces and accelerations acting on subjects’ finger moving on the surface of the touch screen, which was actuated at the threshold voltages estimated in Experiment 1. Our main goal was to determine the frequency components of these recorded signals in order to validate our theoretical model and simulation results. We calculated the signal energies and weighted them with human sensitivity curve to estimate which components enabled the tactile detection. We also investigated the effect of scan speed on measured signals.

3.3.2.1

Participants

We conducted experiments with eight (four female and four male) subjects having the average age of 27.8 (SD: 2.1). The subjects read and signed the consent form before the experiments. The form was approved by Ethical Committee for Human Participants of Koç University. The subjects washed their hands with commercial soap and rinsed with water before each measurement. Then, they dried their hands

3.3 Materials and Methods

53

100

Correct Incorrect 90

Amplitude [V]

80

70

Threshold value

Large step (10V) 60

50

Small step (2V)

40

0

10

20

30

40

50

60

Number of Trials Fig. 3.7 An example data set collected by one up-two down adaptive staircase method

in the room temperature and ambient pressure. Also, the touch screen was cleaned by alcohol before each measurement.

3.3.2.2

Stimuli

We measured accelerations and forces under 48 different conditions; there were 2 waveforms (sinusoidal, square), 6 frequencies (15, 30, 60, 120, 240, 480 Hz), and 4 finger scan speeds (10, 20, 50, 100 mm/s), which are tabulated in Table 3.2. In each measurement, one parameter was changed while fixing the others. We selected the finger scan speeds based on the values used in the earlier studies [1, 14, 39, 43]. The amplitude of the input signals was chosen 8 dB SL (sensation level: 8 dB higher than the threshold) more than the averaged threshold values measured in Experiment 1 (see Sect. 3.3.1). Initially, we performed two separate control measurements to test the reliability of the collected data.2 First, the forces and accelerations were measured when the

2 In

the first set of control measurements, we checked the signal to noise ratio (SNR). If the SNR value of a measurement was lower than 5 dB, that measurement was repeated. In the second set of control measurements, we checked the signal energies due to finger motion without

54

3 Effect of Waveform on Tactile Perception by Electrovibration

Table 3.2 Experimental parameters Type Test

Control 1 (EMI effect) Control 2 (No excitation)

Parameter Frequency Waveform Speed Frequency Waveform Speed

Value 15, 30, 60, 120, 240, 480 Sinusoidal, Square 10, 20, 50, 100 15, 30, 60, 120, 240, 480 Sinusoidal, Square 10, 20, 50, 100

Unit Hz – mm/s Hz – mm/s

finger was stationary in 12 conditions to observe the electromagnetic interference (EMI) effect on the sensors (Table 3.2). Second, the forces and accelerations were measured without any electrostatic excitation in 4 conditions (Table 3.2). Therefore, 64 different (48 test, 16 control) measurements were performed in total for each subject.

3.3.2.3

Experimental Setup

The experimental setup was similar to the one used in our psychophysical experiments (Fig. 3.6). For this experiment, the touch screen (SCT3250, 3M Inc.) was placed on top of a force sensor (Nano17, ATI Inc.). The sensor was attached to the screen and an aluminium base using double-sided adhesive tapes (3M Inc.). The aluminum base was also attached to a stationary table by the same adhesive tape. The touch screen was excited with a voltage signal generated by a signal generator (33220A, Agilent Technologies Inc.). The voltage signal from the generator was amplified by an amplifier (E-413, PI Inc.) before transmitted the touch screen. An IR frame was placed on top of the touch screen to measure the finger scan speed during experiments. An accelerometer (ADXL 335, Analog Devices Inc.) was glued on the fingernail of the subjects. The accelerometer and force data were acquired by two separate DAQ cards (USB-6251 and PCI-6025E, NI Inc.). The cables of the accelerometer were taped on the finger and arm of the subjects as shown in Fig. 3.8. Both accelerometer and force data were acquired using LabView (NI, Inc.). An arm rest was used to support the subjects’ arm during the experiments. The subjects were asked to wear a ground strap on their stationary wrist. The subjects were also asked to synchronize their scan speeds with the speed of a visual cursor displayed on the computer screen.

any electrostatic excitation. These energies were compared to those obtained from the test measurements to investigate the effect of electrostatic excitation (see Sect. 3.4.2).

3.3 Materials and Methods

55

ACCELEROMETER

IR FRAME

m

10 c

TOUCH SCREEN

FORCE SENSOR

Fig. 3.8 Illustration for the attachment of force sensor and accelerometer

3.3.2.4

Procedure

The subjects were instructed to sit on a chair in front of the experimental setup and move their index fingers back and forth in the horizontal direction on the touch screen. They were asked to move their finger only in a 10 × 3 cm rectangular region on the touch screen. They were asked to synchronize their fingers with the motion of a moving cursor on the computer screen. Also, they received visual feedback about the magnitude of the normal force that they applied to the touch screen. For this purpose, two led lights were displayed on the computer screen and used to keep the normal force between 0.1 and 0.6 N. We selected this range based on the normal forces reported in the literature as relevant to tactile exploration [1, 10]. If the user applied less than 0.1 N to the touch screen, the led labelled as “press more” turned to green. However, if the user applied more than 0.6 N, the led labelled as “press less” turned to red. The subjects were instructed to complete four strokes (two forward, two backward) under each experimental condition. Before starting the experiment, the subjects were given instructions about the experiment, and asked to complete a training session. This training session enabled subjects to adjust their finger scan speed and normal force before the actual experimentation. The experiments were performed in two blocks. The first and second blocks had six and seven sessions respectively. The experimental blocks were formed based on the input voltage waveform whereas the sessions were based on the input voltage frequency. The second block also contained one session without any input voltage. It took approximately 1.5 h to complete all the measurements for a subject, including the time for attaching the accelerometer to the subjects’ finger and the training session.

56

3.3.2.5

3 Effect of Waveform on Tactile Perception by Electrovibration

Data Analysis

The force and acceleration data were analyzed in Matlab. An example data collected during one session is shown in Fig. 3.9. The figure shows force and acceleration data recorded at different scan speeds. We calculated the displacement values by integrating the acceleration data twice as suggested in [17]. The collected force, acceleration and displacement data were segmented according to the finger scan speed (see coloured regions in Fig. 3.9). Then, DC offset was removed from each segment by subtracting the mean values. To remove the lowfrequency noise due to finger motion, data in each stroke was filtered by a high-pass filter having a cut-off frequency of 10 Hz. Afterwards, the RMS of each stroke was calculated and an average RMS was obtained for each finger speed using the data of 4 strokes. For detection analysis, power spectrum of each stroke was calculated for the signals in the normal direction. Then, an average power spectrum was obtained for each finger speed using the power spectrum of 4 strokes. The peak frequencies were determined using this spectrum. The energy (in unit time) of each peak frequency was calculated by integrating its power spectrum data for the peak interval. Finally, the calculated raw energies were multiplied by the inverse of the normalized human sensitivity function to obtain the weighted ones (Fig. 3.10). We used the human sensitivity functions reported in [23, 31] for the force, acceleration and displacement data, respectively. Moreover, we calculated the corresponding electrostatic forces generated by the same waveforms and amplitudes via Matlab simulations. We also calculated the weighted energies of those simulated forces using the same data analysis approach discussed above. In addition, the average friction coefficient was calculated by dividing the unfiltered lateral force of each stroke to those of normal force. Then, an average friction coefficient of each condition was obtained using the data of 4 strokes.

3.4 Results 3.4.1 Results of Experiment 1 Figure 3.11 depicts the measured threshold voltages for seven fundamental frequencies (15, 30, 60, 120, 240, 480, 1920 Hz) and two different waveforms (sinusoidal and square). We analyzed the results using two-way analysis of variance (ANOVA) with repeated measures. Both main effects (frequency and waveform) were statistically significant on the threshold levels (p