Self-Powered Internet of Things: How Energy Harvesters Can Enable Energy-Positive Sensing, Processing, and Communication 3031276841, 9783031276842

Citation preview

Green Energy and Technology

Muhammad Moid Sandhu Sara Khalifa Marius Portmann Raja Jurdak

Self-Powered Internet of Things How Energy Harvesters Can Enable Energy-Positive Sensing, Processing, and Communication

Green Energy and Technology

Climate change, environmental impact and the limited natural resources urge scientific research and novel technical solutions. The monograph series Green Energy and Technology serves as a publishing platform for scientific and technological approaches to “green”—i.e. environmentally friendly and sustainable—technologies. While a focus lies on energy and power supply, it also covers “green” solutions in industrial engineering and engineering design. Green Energy and Technology addresses researchers, advanced students, technical consultants as well as decision makers in industries and politics. Hence, the level of presentation spans from instructional to highly technical. **Indexed in Scopus**. **Indexed in Ei Compendex**.

Muhammad Moid Sandhu · Sara Khalifa · Marius Portmann · Raja Jurdak

Self-Powered Internet of Things How Energy Harvesters Can Enable Energy-Positive Sensing, Processing, and Communication

Muhammad Moid Sandhu Australian e-Health Research Centre (AEHRC) Commonwealth Scientific and Industrial Research Organisation (CSIRO) Herston, QLD, Australia Marius Portmann School of Information Technology and Electrical Engineering University of Queensland St. Lucia, QLD, Australia

Sara Khalifa Commonwealth Scientific and Industrial Research Organisation (CSIRO) Data61 Pullenvale, QLD, Australia Raja Jurdak School of Computer Science Queensland University of Technology Brisbane, QLD, Australia

ISSN 1865-3529 ISSN 1865-3537 (electronic) Green Energy and Technology ISBN 978-3-031-27684-2 ISBN 978-3-031-27685-9 (eBook) https://doi.org/10.1007/978-3-031-27685-9 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

This book is dedicated to my beloved parents, teachers, family, friends, and to all the people who provided me with endless love, support, appreciation, and encouragement M. M. Sandhu To my family R. Jurdak

Foreword by Sajal K. Das

With the advancement of sensing and communications technology, the Internet of Things (IoT) has evolved to play vital roles in our daily lives in the connected world. Sensor-enabled IoT devices have the capability to monitor our daily context (mobility, activity, mood, emotions) and our environments (e.g., homes, buildings, cities, workplaces, critical infrastructures, etc.). The processing and analysis of such collected data help make context-aware decisions and actionable inferences to improve our quality of life or the environment. The ever-increasing number of connected IoT devices and the scale at which data are being collected pose significant challenges in sensing, processing, and communication. The success and widespread adoption of IoT devices, particularly for continuous activity recognition, will be hindered without reliable sources of energy to power these devices. This book titled Self-Powered Internet of Things: How Energy Harvesters Can Enable Energy-Positive Sensing, Processing, and Communication is timely and a comprehensive guide to understanding and implementing self-powered IoT devices for human activity recognition (HAR). The book divided into three parts covers a wide range of topics from energy harvesting technologies to energy-positive sensing techniques. Given the myriad of potential applications of self-powered activity recognition using IoT devices, from improving healthcare monitoring to optimizing energy consumption in smart buildings, this book is a valuable resource for anyone interested in exploring this exciting field. The authors have done an excellent job in delving into various issues and challenges, namely multi-source energy harvesting, real-time operation, and managing power consumption that must be overcome to realize truly self-powered IoT devices. With expertise in electrical engineering, computer science, and data science, they have provided an in-depth exposition of the latest research and development in

vii

viii

Foreword by Sajal K. Das

“Self-Powered Internet of Things”. This book will undoubtedly prove useful to the engineers, researchers, developers, and entrepreneurs alike. February 2023

Sajal K. Das IEEE Fellow Curators’ Distinguished Professor, Daniel St. Clair Endowed Chair Missouri University of Science and Technology Rolla, USA

Foreword by Mahbub Hassan

The Internet of Things (IoT) is revolutionizing the way we interact with the world around us, providing new opportunities for monitoring and tracking activities, environments, and devices. However, as the number of IoT devices continues to grow, so too does the challenge of powering these devices. In “Self-Powered Activity Recognition in Internet of Things: How Energy Harvesters Can Enable EnergyPositive Sensing, Processing, and Communication,” the authors explore how energy harvesting technology can be used to power IoT devices for activity recognition, and how this can lead to energy-positive sensing, processing, and communication. In Part I of the book, the authors provide an in-depth overview of IoT and activity recognition, including the types of IoT devices that are used for human activity recognition (HAR) and the energy challenges that must be overcome. They also discuss the motivation behind the use of energy harvesting technology and the organization of the book. They also provide a detailed discussion of the various types of IoT devices used for human activity recognition, including implantable, wearable, and environmental devices, and the challenges faced by these devices. In Part II, the authors delve into the details of energy harvesting, including the use of ambient energy to power IoT sensors and the different energy harvesting modes that are available. They also discuss the use of solar, kinetic, and thermal energy harvesting, as well as the challenges and opportunities presented by these technologies. They also provide a thorough examination of the various energy harvesting modes, including photovoltaics, thermoelectrics, piezoelectrics, and electromagnetic, and the advantages and disadvantages of each mode. In Part III, the authors discuss a very interesting concept whereby the energy harvesters are used as a proxy for sensors. The idea is to analyse the energy harvesting patterns of a device to gain information about the context that produces the energy in the first place. This idea, also coined as Simultaneous Sensing and Energy Harvesting, promises a new generation of IoT devices that can save energy and cost by reusing the same hardware for both energy production and sensing. Many example applications that use kinetic and solar energy harvesters as sensors are covered. The authors also provide a detailed discussion of the various machine learning algorithms and techniques used in activity recognition, including data acquisition ix

x

Foreword by Mahbub Hassan

and preprocessing, segmentation, feature extraction, model training, model testing, and evaluation metrics. They also provide a comprehensive analysis of the various datasets used for developing and evaluating activity recognition algorithms, and the challenges faced by current activity recognition mechanisms. Overall, this book is a valuable resource for anyone interested in IoT and activity recognition, as well as energy harvesting technology. With the authors’ expertise and the wealth of information provided, readers will gain a deeper understanding of the potential of energy harvesting to power IoT devices, and how this can lead to more sustainable and efficient sensing, processing, and communication. The book is well written and easy to understand, making it accessible to a wide range of readers, from students and researchers to engineers and practitioners in the field. The book is a must-read for anyone interested in the future of IoT and the role of energy harvesting in powering IoT devices. February 2023

Prof. Mahbub Hassan University of New South Wales Sydney, Australia

Preface

This book covers a journey that we started in 2018 to explore novel mechanisms of activity recognition that ensure the perpetual operation of sensor nodes. In October 2018, Moid started his Ph.D. to explore the use of energy harvesters as activity sensors under the supervision of Sara, Marius, and Raja. Initially, we reviewed previous studies that employ energy harvesters as activity sensors to explore research gaps. We discovered that previous studies employ energy harvesters either as an activity sensor or an energy source. In order to solve this issue, we thoroughly investigated the energy harvesting mechanism and explored various sensing points in the energy harvesting circuit. Through this research, we designed and demonstrated the first system that uses kinetic energy harvesting transducer as a simultaneous source of energy and context information. We then moved to the exploration of solar energy harvesting as an information and energy source. This led us to design a novel mechanism of selfpowered activity recognition using a solar energy harvester as an activity sensor due to the rich context information embedded in its harvesting signal. Finally, we explored signal fusion using kinetic and solar energy harvesting signals to maximise the harvested energy and context detection performance resulting in the perpetual operation of wearable devices. This book is written keeping in mind the graduate-level students, professors, researchers, and industrial professionals working in the domain of Internet of Things (IoT), activity recognition, wearable devices, and energy harvesting. It covers most of the topics related to the use of energy harvesters as a simultaneous source of energy and context information. This will help the research community to understand the emerging area of energy-positive sensing and propose new solutions to enable autonomous, uninterrupted, and perpetual operation of wearable devices. This book is the pinnacle of our four-year journey in developing energy-positive sensing systems with the aim to share the steps of our journey and the lessons learnt with the readers. This book is orchestrated into three main parts. Part I provides an overview of IoT, activity recognition mechanisms, and the challenges related to limited energy availability in IoT. Part II discusses energy harvesting mechanisms to power the IoT devices to ensure their perpetual operation and the use of energy xi

xii

Preface

harvesters as activity/context sensors. Part III presents detailed mechanisms of selfpowered human activity recognition which employs energy harvesting transducers as activity sensors and sources of energy. We have had a wonderful time travelling in this area thus far. We anticipate that the reader will like our journey and enjoy reading this book. Brisbane, Australia February 2023

Muhammad Moid Sandhu Sara Khalifa Marius Portmann Raja Jurdak

Acknowledgements We would like to thank Dr. Brano Kusy and Dr. Kai Geissdoerfer for their contribution to our team’s research work over the past several years. We must also acknowledge, with great gratitude, the financial support extended to us by the Australian Government, the University of Queensland, Data61 and the Australian e-Health Research Center (AEHRC), Commonwealth Scientific and Industrial Research Organisation (CSIRO), Queensland University of Technology (QUT) during some of the works that have underpinned this book.

Contents

Part I

Overview of IoT and Activity Recognition

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Types of IoT Devices for HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Implantable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Wearable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.3 Environmental . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Energy Challenges in the use of IoT for HAR . . . . . . . . . . . . . . . . . 1.3 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Book Organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 5 5 6 6 7 8 9 10

2 Activity Recognition in IoT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Activity Recognition Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Wearable Sensors for HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Activity Recognition Using Machine Learning . . . . . . . . . . . . . . . . . 2.3.1 Data Acquisition and Preprocessing . . . . . . . . . . . . . . . . . . . 2.3.2 Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.4 Model Training (Learning) . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.5 Model Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.6 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Datasets for Developing and Evaluating HAR Algorithms . . . . . . . 2.5 Challenges in Current Activity Recognition Mechanisms . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13 14 15 15 15 16 17 17 18 19 20 20 21

Part II

Energy Harvesting

3 Using Ambient Energy to Power IoT Sensors . . . . . . . . . . . . . . . . . . . . . 3.1 Energy Harvesting Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Solar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Kinetic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29 31 33 33

xiii

xiv

Contents

3.4 3.5 3.6 3.7 3.8

Thermal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RF Waves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kinetic Energy Harvesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kinetic Energy Harvesting Circuits . . . . . . . . . . . . . . . . . . . . . . . . . . Operation of KEH Transducer at MPP . . . . . . . . . . . . . . . . . . . . . . . . 3.8.1 MPP of the KEH Transducer . . . . . . . . . . . . . . . . . . . . . . . . . 3.8.2 Harvested Power Stored in the Capacitor . . . . . . . . . . . . . . . 3.8.3 Impact of Threshold Voltage of DC-DC Converter on the Harvested Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8.4 Power Consumption of DC-DC Converter . . . . . . . . . . . . . . 3.9 Solar Energy Harvesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.10 Solar Energy Harvesting Circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.11 Operation of SEH Transducer at MPP . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

34 34 35 37 40 41 43

4 Energy Harvester as an Information Source . . . . . . . . . . . . . . . . . . . . . . 4.1 KEH as a Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Step Count . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Audio Signal Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 Transport Mode Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.5 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 SEH as a Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 TEH as a Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 RFEH as a Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

53 54 55 56 56 57 57 58 59 59 59 65

44 45 46 47 48 49 49

Part III Self-Powered IoT 5 Simultaneous Sensing and Energy Harvesting . . . . . . . . . . . . . . . . . . . . . 5.1 Challenges in Simultaneous Sensing and Energy Harvesting . . . . . 5.2 System Architecture for Simultaneous Sensing and Energy Harvesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Sensing and Energy Harvesting . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Energy-Positive Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Exploring Multiple Sensing Points . . . . . . . . . . . . . . . . . . . . 5.3 System Design for Simultaneous Sensing and Energy Harvesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Hardware Designs for KEH Sensing and Energy Harvesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 The Interference Problem at Different Sensing Points . . . .

71 71 73 74 74 75 75 76 77 79

Contents

5.4

xv

Transport Mode Detection: A Case Study . . . . . . . . . . . . . . . . . . . . . 5.4.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 Detection Accuracy of KEH-Based Sensing Signals . . . . . . 5.5.2 Energy Harvesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.3 Energy Consumption and System Costs . . . . . . . . . . . . . . . . 5.5.4 Energy-Positive Sensing: Discussion and Analysis . . . . . . . 5.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

82 82 83 85 85 88 88 90 91 92

6 Solar Cell Based Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.1 Previous HAR Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Human Activity Recognition Using Solar Cell . . . . . . . . . . . . . . . . . 6.3 SolAR: System Model and Implementation . . . . . . . . . . . . . . . . . . . 6.3.1 Measurement Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Solar Cell as a Novel Human Activity Sensor . . . . . . . . . . . 6.3.3 Implementation of SolAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Classification Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Variability Analysis of Human Activities . . . . . . . . . . . . . . . 6.4.3 Varying Window Sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.4 Varying Signal Sampling Frequency . . . . . . . . . . . . . . . . . . . 6.4.5 Robustness to User Variance . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.6 Environment-Agnostic Analysis . . . . . . . . . . . . . . . . . . . . . . . 6.5 Energy-Positive HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5.1 SolAR Harvested Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5.2 SolAR Power Consumption . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5.3 Energy-Positive HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95 96 96 98 99 99 101 102 104 104 104 107 108 109 109 110 110 111 113 115 115

7 Fusion-Based Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Accelerometer-Based HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.2 KEH-Based HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.3 SEH-Based HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.4 Limitations and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Fusing Solar and Kinetic Energy Signals . . . . . . . . . . . . . . . . . . . . . . 7.2.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Measurement Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 Human Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Classification Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 Varying Window Sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

119 119 120 120 121 122 122 123 124 126 128 128 130

xvi

Contents

7.3.3 7.3.4 7.3.5 7.3.6

Varying Sampling Frequency of the Signal . . . . . . . . . . . . . . Robustness to User Variance . . . . . . . . . . . . . . . . . . . . . . . . . . Robustness to Diverse Lighting Conditions . . . . . . . . . . . . . Robustness to Environment-Agnostic and Environment-Preserving Scenarios . . . . . . . . . . . . . . . . . 7.4 Analysis of Harvested and Consumed Power . . . . . . . . . . . . . . . . . . 7.4.1 Harvested Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Power Consumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.3 Energy-Positive HAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

132 134 134

8 Energy-Positive Activity Recognition: Future Directions . . . . . . . . . . . 8.1 Energy-Efficient Communication Using Energy Harvesters . . . . . . 8.2 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Federated Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Personalised AI Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Real-Time Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Multi-source Energy Harvesters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Hardware Implementation on the Edge Device . . . . . . . . . . . . . . . . . 8.8 Batteryless Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 Security and Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.10 Reducing the System Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11 Exploring Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

145 146 146 147 147 148 148 148 149 149 149 150 150

135 136 136 138 140 142 142

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

About the Authors

Muhammad Moid Sandhu is currently a postdoctoral research fellow at Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia. He is working on the design and implementation of sensing technologies for human centric applications. Previously, he completed his Ph.D. in Information Technology and Electrical Engineering from the University of Queensland. During his Ph.D., Moid worked on batteryless wearable IoT devices for human activity, fitness, and health monitoring applications. He has published several research papers in various well-reputed peer-reviewed international journals and conferences. He is a reviewer of various journals including ACM Transactions on Internet Technology, Scientific Reports, IEEE Internet-of-Things Journal, Journal of Network and Computer Applications, and IEEE Journal on Selected Areas in Communication. His research interests include eHealth, digital health, wearables, IoT, pervasive computing, and embedded machine learning. He is an active member of IEEE and ACM. Moid was awarded a Gold Medal and Academic Roll of Honor for his outstanding performance during his B.Sc. degree in Electrical Engineering, in 2012. Sara Khalifa is currently a senior research scientist at the Distributed Sensing Systems research group, Data61-CSIRO. She is also an honorary adjunct senior lecturer at Queensland University of Technology, conjoint senior lecturer at the University of New South Wales, and adjunct lecturer at the University of Queensland. Her research interests rotate around the broad aspects of mobile and ubiquitous computing, mobile sensing, and the Internet of Things (IoT). She obtained a Ph.D. in Computer Science and Engineering from UNSW (Sydney, Australia). Her Ph.D. dissertation received the 2017 John Makepeace Bennett Award which is awarded by CORE (the Computing Research and Education Association of Australasia) to the best Ph.D. dissertation of the year within Australia and New Zealand in the field of Computer Science. Her research has been recognised by multiple iAwards including 2017 NSW Mobility Innovation of the year, 2017 NSW R&D Innovation of the year, National Merit R&D Innovation of the year, and the Merit R&D award at the Asia Pacific ICT Alliance (APICTA) Awards, commonly known as the ‘Oscar’ of the ICT industry in the Asia Pacific, among others. xvii

xviii

About the Authors

Marius Portmann is an Associate Professor in the School of Electrical Engineering and Computer Science at the University of Queensland, Australia. He received a Ph.D. in Electrical Engineering from the Swiss Federal Institute of Technology (ETH), Zürich. His research interests include Computer Networks with a focus on Software Defined Networks (SDN), Cybersecurity, applied Machine Learning, and IoT (Internet of Things) technology. He serves as a member of the Editorial Board of Computer Communications (Elsevier). Raja Jurdak is a Professor of Distributed Systems and Chair in Applied Data Sciences at Queensland University of Technology, and Director of the Trusted Networks Lab. He holds a PhD degree from University of California, Irvine. He previously established and led the Distributed Sensing Systems Group at CSIRO’s Data61. He also spent time as visiting academic at MIT and Oxford University in 2011 and 2017. His research interests include trust, mobility and energy-efficiency in networks. Prof. Jurdak has published over 250 peer-reviewed publications, including three authored books on blockchain, cyberphysical systems and IoT. His publications have attracted over 12500 citations, with an h-index of 49. He serves on the editorial board of Ad Hoc Networks, Nature Scientific Reports, and on the organising and technical program committees of top international conferences, including Percom, ICBC, IPSN, WoWMoM, and ICDCS. He was TPC co-chair of ICBC in 2021. He is an Adjunct Professor with the University of New South Wales, a Senior Member of the IEEE and a Distinguished Visitor of the IEEE Computer Society.

Acronyms

AccAR ADC ADL ALAP APR ASAP BLE CPS CV DT DVFS EDF EEH EH-IoT EM ENO ESU EWMA FusedAR GB GPIO HAR HVAC IoT ISM IV KEH KEHAR KNN LED MAC

Accelerometer-based Activity Recognition Analog-to-Digital Converter Activities of Daily Living As Late As Possible Acquisition Power Ratio As Soon As Possible Bluetooth Low Energy Cyber-Physical System Cross Validation Decision Tree Dynamic Voltage and Frequency Scaling Earliest Deadline First Electromagnetic Energy Harvester Energy Harvesting-based IoT Electromagnetic Energy Neutral Operation Energy Storage Unit Exponentially Weighted Moving Average Fusion-based Activity Recognition Gradient Boosting General Purpose Input/Output Human Activity Recognition Heating, Ventilation, and Air Conditioning Internet of Things Industrial, Scientific, and Medical Current-Voltage Characteristic Curve Kinetic Energy Harvesting KEH-based Activity Recognition K-Nearest Neighbor Light Emitting Diode Medium Access Control xix

xx

MCU MDP MEMS MLP MOSFET MPP NB NC PCA PEH PMU QoS RF RFE RFEH RFID SEH SMOTE SRC SVM TDMA TEG TEH TMD WSN

Acronyms

Micro-controller Unit Markov Decision Process Micro-Electromechanical Systems Multilayer Perceptron Metal Oxide Semiconductor Field Effect Transistor Maximum Power Point Naive Bayes Nearest Centroid Principle Component Analysis Piezoelectric Energy Harvester Power Management Unit Quality-of-Service Random Forest Recursive Feature Elimination RF Energy Harvester Radio Frequency Identification Solar Energy Harvesting Synthetic Minority Over-sampling Technique Sparse Representation-based Classification Support Vector Machine Time Division Multiple Access Thermoelectric Power Generator Thermoelectric Energy Harvester Transport Mode Detection Wireless Sensor Network

Part I

Overview of IoT and Activity Recognition

Chapter 1

Introduction

Pervasive, miniaturised and smart sensing devices are deployed in the physical world that work collectively to gather the required data/information, and this mechanism is often referred as ubiquitous sensing, smart dust, and Internet of Things (IoT) [1]. These sensing devices translate physical signals into a digital data that can be processed by a computing device/machine for further analysis and to extract a useful information. The collected data from the senors can be processed on the device or transmitted to a server for further analysis. With the significant progress in research and advancement in technology, low-power tiny IoT sensors are becoming popular to sense physical parameters in various applications including smart home, smart city, connected health, supply chain and smart farming as depicted in Fig. 1.1. Other applications include surveillance, transportation, exploration of mines and battle field monitoring [1]. A common application of IoT is Human Activity Recognition (HAR), which has gained significant interest from industry and researchers for its broad range of potential applications [2]. Automatic recognition of user’s physical activities through HAR [3] enables the continuous and potentially remote monitoring of the current status of a user, as depicted in Fig. 1.2. Such HAR mechanisms employ various onbody sensors to measure different physiological parameters [4] such as temperature, glucose level, blood oxygen, heart rate, sleep pattern, or respiration rate, to name a few. Figure 1.2 shows a typical architecture of a HAR system, applied in a health context. Various types of on-body sensors collect the physiological parameters and wirelessly transmit the collected data to a remote server using a gateway e.g., a smartphone. Later, this data can be shared with the hospital, clinician, emergency services and family members to provide timely assistance to the user, if needed. This also maintains a 24×7 history of the user’s activities and health and helps the carers to track the user’s condition. Figure 1.2 also shows that information flows from the user towards the server and that, in accordance with the readings of the vital parameters, a carer can intervene and provide real-time advice/assistance. For example, in case of abnormal glucose level, an automated insulin delivery [5] can be triggered from a remote station. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9_1

3

4

1 Introduction

Fig. 1.1 Applications of IoT

The proliferation of HAR has been underpinned by recent technological advances in pervasive IoT devices, for applications in human health [6], fitness [7], and activity [8] monitoring. Existing commercial wearable products can be divided into three categories [9], namely (i) Accessories, (ii) E-Textiles, and (iii) E-Patches. Accessories are wearable products that monitor human fitness/health and include smart watches [10], wrist bands [11], head-mounted devices [12], smart jewellery [13] and straps [14]. E-Textiles are mainly clothing items that also serve as wearables, such as smart clothes/garments [15] and smart shoes/socks [16]. In the final category of wearable devices, E-patches can be attached to or tattooed on the skin. E-patches includes sensor patches [17] and E-tattooes [18] with flexible and stretchable electronic circuits. In the next section, we discuss the types of IoT devices for HAR in more detail while outlining the key challenges for the use of such device in HAR contexts. Finally, we map out the road map for the rest of this book.

1.1 Types of IoT Devices for HAR

5

Fig. 1.2 Wearable IoT devices are employed to monitor human activity and health in various applications

1.1 Types of IoT Devices for HAR There are various types of IoT sensors including implantable, wearable and environmental as described in the following subsections.

1.1.1 Implantable Implantable sensors can provide useful data of a body part being diagnosed, such as gastrointestinal (temperature, pH, pressure) parameter values, blood glucose and pressure levels and electrocardiogram readings [19]. For example, a tiny implantable wireless sensor [20] can be used to monitor the internal nerves, muscles or organs in real-time. Furthermore, an accelerometer or piezoelectric sensor can be used as a cochlear implant for hearing aids [21]. Another study [22] proposes the use of an implanted sensor for monitoring exercise-induced or disease-induced muscle strain and ligament injuries. In addition to arm and eye, glucose sensor can be implanted on the fat layer between skin and the muscle for tissue glucose monitoring [23]. However, biocompatibility of implantable devices and safety of the user must be considered before using these devices on living objects [24].

6

1 Introduction

1.1.2 Wearable Wearable devices [25] which include smart watches, wrist bands and rings are commonly used to detect and monitor vital human physical and biological parameters. For example, a commercial wearable Fitbit device [26] can monitor various parameters pertaining to health, fitness and sleep. Wearable sensors attached to ankle, arm, wrist, finger, and car seat belt can be used to monitor human activities, glucose level, heart rate/rhythms and body temperature [23]. In the application of physical therapy and rehabilitation [27], pedometers, gyroscopes and accelerometers can be attached to the leg or ankle to monitor the walking pattern and number of steps taken in a specific time slot. Blood glucose is an important biochemical marker which is linked to various chronic diseases [28] and can be measured using a wearable sensor. In addition, an actuator in the form of insulin injector can be employed to automatically inject insulin if sugar level falls below a certain threshold. Another important body parameter is heart rate and rhythm which can provide information about the cardiopulmonary and vascular health [29, 30]. Other types of sensors include smart jewellery, straps [9] and necklace [31] to monitor the daily food intake for healthy and independent life. In addition to the implementable and wearable sensors, the field of smart dust [32] is also emerging which includes body dust and neural dust for monitoring the human body. It employs millimeter sized motes to sense, process and communicate the body parameters and can safely be injected in the human body for a detailed diagnosis.

1.1.3 Environmental In addition to implantable and wearable sensors, environmental sensors can also provide useful information about the human context and interaction with the surrounding environment. Environmental or ambient sensors can be employed to monitor various environmental parameters such as temperature, pressure and humidity [33]. For example, a commonly used Passive Infrared (PIR) sensor [34] measures infrared light radiating from objects for context, interaction and motion detection. Similarly, various no-body or ambient home sensors such as pressure sensors, cameras and accelerometers can be employed to detect fall events and sudden changes in activity that can represent an abnormality. Environmental, ambient home and phone sensors are also employed for monitoring social-isolation, stress and loneliness [35, 36] as these conditions can give birth to various chronic diseases and other co-morbidities.

1.2 Energy Challenges in the use of IoT for HAR

7

1.2 Energy Challenges in the use of IoT for HAR Wearable IoT devices have numerous applications, including activity detection, fitness tracking, health monitoring, travel route planning, etc. Energy is an important and one of the major challenges for the perpetual operation of IoT-based HAR. These devices have a limited lifetime due to the finite energy capacity of their batteries [25]. This impedes their pervasive deployment and widespread adoptability. One of the solutions is to regularly recharge and replace the batteries of wearable devices. However, this is cumbersome as one has to continuously track the battery state-of-charge. Furthermore, one may not be able to recharge a wearable device when s/he is at a remote place with no power supply. In addition, batteries are bulky, expensive, hazardous and wear out after a few years [37]. Although battery technology is evolving with time, still the battery lifetime is a major bottleneck in the wide adoption of wearable devices [25]. A promising direction to perpetually operate the wearable IoT devices is to employ the ambient energy in the environment to power these miniaturised sensors. In our environment, a plethora of energy is available in the form of sunlight, heat, vibrations and radio frequency (RF) waves. This ambient energy can be converted into electrical energy using various types of energy harvesting transducers. For example, solar cells convert the abundantly available light energy into electrical energy. Electromagnetic and piezoelectric transducers convert vibration/motion energy into electrical energy. Thermoelectric energy harvesters convert heat/thermal energy into electrical energy. On the other hand, the RF Energy Harvester (RFEH) captures RF waves in the environment and generates electrical energy that can be used to power wearable IoT devices. One goal for harvesting energy to power IoT sensors is to move away from use of batteries, to avoid their drawbacks outlined above. Recently, capacitors that offer longer life compared to batteries, have been employed to store energy harvested for powering the wearable IoT devices. Figure 1.3 shows that capacitors offer higher power density and lower energy density compared to batteries (such as lead-acid, lithium-ion (Li-ion), nickel–cadmium (Ni-Cd) and nickel metal hydride (Ni-MH)). This means that capacitors cannot store energy for a long time and offer higher leakage compared to batteries [38]. Therefore, capacitors can only be used to power isolated tasks which run intermittently using the available harvested energy. However, the harvested power using wearable-sized energy harvesters is typically not sufficient to ensure the perpetual operation of wearable IoT devices. For example, the harvested power from human movements/vibrations is significantly lower than the required power to run the wearable devices [39]. Therefore, there is a need to devise alternative HAR mechanisms that can lower the energy consumption of wearable devices to match the harvested power. A recent survey about wearable devices [9] also shows that although significant research has been carried out for battery-powered wearables, there exists very limited literature on energy-harvesting wearables for HAR. The aim of this book is to study the limitations of conventional

8

1 Introduction

Fig. 1.3 Power density vs energy density of various batteries and capacitors [38]

energy harvesting wearables and propose novel solutions to overcome these limitations and thus enable autonomous and self-powered operation of IoT wearable devices for HAR.

1.3 Motivation Figure 1.4 shows a typical energy harvesting mechanism to power wearable IoT devices. An energy harvesting transducer converts the ambient energy into electrical energy and a power management circuit is used to charge an energy storage device (battery/capacitor). This stored harvested energy is used to power various components of wearable devices, including sensor, signal processing unit (i.e., microcontroller) and transceiver. However, in real-world scenarios, the harvested power from the tiny

Fig. 1.4 Ambient energy harvesting to power the wearable devices [40]

1.4 Book Organisation

9

transducers is not sufficient to power the wearable devices in human-centric applications [41]. Previous work [40] shows that IoT sensor nodes consume significant energy to process the collected signal and transmit the data to a remote server for real-time activity recognition. On the other hand, tiny and wearable-sized energy harvesters do not generate sufficient power to allow uninterrupted operation of the IoT sensing device [9]. Although solar energy harvesters generate higher power compared to kinetic, thermal and RF energy harvesters [9], they need a light source and bright light to generate sufficient energy and thus may not harvest sufficient power in low light and dark environments (such as at night) perpetually to power the wearable devices. This means that alternative HAR mechanisms should be devised to reduce the energy consumption of wearable device so that it can match with the average harvested power (in different environments) and thus can lead towards autonomous and real-time activity recognition. Recently, energy harvesters have been used as activity sensors to replace conventional power-hungry inertial sensors. For example, the Kinetic Energy Harvesting (KEH) signal from human movements contains the signatures of the underlying movements/vibrations and thus can be used to recognise the activities causing the movements [25]. This reduces the sensor-related energy consumption [25] that would otherwise be used to power conventional activity sensors. However, the harvested energy is still not sufficient to enable end-to-end activity recognition including signal sampling, processing and communication [41]. Most of the previous research has explored how to infer activities from the harvesting signal, yet without exploiting the harvested energy for powering the IoT devices [25, 42, 43]. In practical applications, the energy harvesting transducers can be employed to generate energy in addition to providing activity recognition. Thus previous works [25, 42, 43] did not realise the full benefits of the energy harvesters and employed them in isolation as activity sensors without generating energy. In this book, instead, we discuss the use of energy harvesters as a simultaneous source of energy and activity information. Furthermore, we explore novel source(s) of activity recognition which generate sufficient energy to autonomously power wearable IoT devices.

1.4 Book Organisation This book is organised as follows. Chapter 2 provides state-of-the-art related to human activity recognition algorithms in IoT. In Chap. 3, we describe energy harvesting mechanisms to power IoT sensing devices with the focus on the use of kinetic and solar energy harvesting to power IoT sensor nodes. Chapter 4 presents the use of energy harvesters as an information source for context/activity detection/recognition applications and Chap. 5 explores the use of kinetic energy harvester as a simultaneous source of energy and context information. We present the novel mechanism of using solar cells as activity sensors and source of energy simultaneously in Chap. 6. Chapter 7 describes the signal fusion using solar and kinetic energy harvesting

10

1 Introduction

signals to maximise the harvested energy and activity recognition performance, thereby enabling energy-positive HAR. Finally, Chap. 8 concludes this book and provides suggestions, pathways and directions for future work.

References 1. Sandhu MM, Khalifa S, Jurdak R, Portmann M (2021) Task scheduling for energy-harvestingbased iot: a survey and critical analysis. IEEE Internet Things J 8(18):13825–13848 2. Hassan MM, Uddin MZ, Mohamed A, Almogren A (2018) A robust human activity recognition system using smartphone sensors and deep learning. Future Gener Comput Syst 81:307–313 3. Dunn J, Runge R, Snyder M (2018) Wearables and the medical revolution. Personal Med 15(5):429–448 4. Seshadri DR, Li RT, Voos JE, Rowbottom JR, Alfes CM, Zorman CA, Drummond CK (2019) Wearable sensors for monitoring the physiological and biochemical profile of the athlete. NPJ Digit Med 2(1):1–16 5. Joshi AM, Jain P, Mohanty SP (2020) Secure-iglu: a secure device for noninvasive glucose measurement and automatic insulin delivery in iomt framework. In: IEEE computer society annual symposium on VLSI (ISVLSI), vol 2020. IEEE, pp 440–445 6. Jeong IC, Bychkov D, Searson PC (2018) Wearable devices for precision medicine and health state monitoring. IEEE Trans Biomedi Eng 66(5):1242–1258 7. Scalise L, Cosoli G (2018) Wearables for health and fitness: measurement characteristics and accuracy. In: IEEE international instrumentation and measurement technology conference (I2MTC), vol 2018. IEEE, pp 1–6 8. Hegde N, Bries M, Swibas T, Melanson E, Sazonov E (2017) Automatic recognition of activities of daily living utilizing insole-based and wrist-worn wearable sensors. IEEE J Biomed Health Inf 22(4):979–988 9. Seneviratne S, Hu Y, Nguyen T, Lan G, Khalifa S, Thilakarathna K, Hassan M, Seneviratne A (2017) A survey of wearable devices and challenges. IEEE Commun Surv Tutor 19(4):2573– 2620 10. Gruenerbl A, Pirkl G, Monger E, Gobbi M, Lukowicz P (2015) Smart-watch life saver: smartwatch interactive-feedback system for improving bystander cpr. In: Proceedings of the 2015 ACM international symposium on wearable computers, 2015, pp 19–26 11. Cornelius C, Peterson R, Skinner J, Halter R, Kotz D (2014) A wearable system that knows who wears it. In: Proceedings of the 12th annual international conference on Mobile systems, applications, and services, 2014, pp 55–67 12. Ugulino W, Fuks H (2015) Landmark identification with wearables for supporting spatial awareness by blind persons. In: Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing, 2015, pp 63–74 13. Gummeson J, Priyantha B, Liu J (2014) An energy harvesting wearable ring platform for gestureinput on surfaces. In: Proceedings of the 12th annual international conference on Mobile systems, applications, and services, 2014, pp 162–175 14. He L, Xu C, Xu D, Brill R (2015) Pneuhaptic: delivering haptic cues with a pneumatic armband. In: Proceedings of the 2015 ACM international symposium on wearable computers, 2015, pp 47–48 15. Mokaya F, Lucas R, Noh HY, Zhang P (2015) Myovibe: vibration based wearable muscle activation detection in high mobility exercises. In: Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing, 2015, pp 27–38 16. Mariani B, Rochat S, Büla CJ, Aminian K (2012) Heel and toe clearance estimation for gait analysis using wireless inertial sensors. IEEE Trans Biomed Eng 59(11):3162–3168

References

11

17. Kao H-L, Holz C, Roseway A, Calvo A, Schmandt C (2016) Duoskin: rapidly prototyping on-skin user interfaces using skin-friendly materials. In: Proceedings of the 2016 ACM international symposium on wearable computers, 2016, pp 16–23 18. Alberth Jr WP (2013) Coupling an electronic skin tattoo to a mobile communication device, US Patent App 13/931,883. (7 Nov 2013) 19. Umay I, Fidan B, Barshan B (2017) Localization and tracking of implantable biomedical sensors. Sensors 17(3):583 20. Xpress M (2022) Wireless, implantable sensors the size of a grain of sand could have wide use in body monitoring. https://medicalxpress.com/news/2016-08-wireless-implantable-sensorssize-grain.html. Accessed 29 Sept 2022 21. Calero D, Paul S, Gesing A, Alves F, Cordioli JA (2018) A technical review and evaluation of implantable sensors for hearing devices. Biomed Eng Online 17(1):1–26 22. Sheng F, Zhang B, Zhang Y, Li Y, Cheng R, Wei C, Ning C, Dong K, Wang ZL (2022) Ultrastretchable organogel/silicone fiber-helical sensors for self-powered implantable ligament strain monitoring. ACS Nano 16(7):10958–10967 23. Appelboom G, Camacho E, Abraham ME, Bruce SS, Dumont EL, Zacharia BE, D’Amico R, Slomian J, Reginster JY, Bruyère O et al (2014) Smart wearable body sensors for patient self-assessment and monitoring. Arch Public Health 72(1):1–9 24. Damaj AW, El Misilmani HM, Abou Chahine S (2018) Implantable antennas for biomedical applications: an overview on alternative antenna design methods and challenges. In: 2018 international conference on high performance computing & simulation (HPCS). IEEE, pp 31–37 25. Khalifa S, Lan G, Hassan M, Seneviratne A, Das SK (2017) Harke: human activity recognition from kinetic energy harvesting data in wearable devices. IEEE Trans Mob Comput 17(6):1353– 1368 26. Fitbit, Trackers (fitbit inspire 2). https://www.fitbit.com/global/au/products/trackers. Accessed 29 Sept 2022 27. Cook DJ, Thompson JE, Prinsen SK, Dearani JA, Deschamps C (2013) Functional recovery in the elderly after major surgery: assessment of mobility recovery using wireless technology. Ann Thorac Surg 96(3):1057–1061 28. Zanon M, Sparacino G, Facchinetti A, Riz M, Talary MS, Suri RE, Caduff A, Cobelli C (2012) Non-invasive continuous glucose monitoring: improved accuracy of point and trend estimates of the multisensor system. Med Biol Eng Comput 50(10):1047–1057 29. Lukowicz P, Anliker U, Ward J, Troster G, Hirt E, Neufelt C (2002) Amon: a wearable medical computer for high risk patients. In: Proceedings sixth international symposium on wearable computers. IEEE, pp 133–134 30. Weber S, Scharfschwerdt P, Schauer T, Seel T, Kertzscher U, Affeld K (2013) Continuous wrist blood pressure measurement with ultrasound, Biomedical Engineering/Biomedizinische Technik 58 (SI-1-Track-E) (2013) 000010151520134124 31. Alshurafa N, Kalantarian H, Pourhomayoun M, Liu JJ, Sarin S, Shahbazi B, Sarrafzadeh M (2015) Recognition of nutrition intake using time-frequency decomposition in a wearable necklace using a piezoelectric sensor. IEEE Sens J 15(7):3909–3916 32. Carrara S (2020) Body dust: well beyond wearable and implantable sensors. IEEE Sens J 21(11):12398–12406 33. Bouchabou D, Nguyen SM, Lohr C, LeDuc B, Kanellos I (2021) A survey of human activity recognition in smart homes based on iot sensors algorithms: taxonomies, challenges, and opportunities with deep learning. Sensors 21(18):6037 34. A express, 220v pir motion sensor. https://www.aliexpress.com/item/33002808775.html. Accessed 29 Sept 2022 35. Wu C, Barczyk AN, Craddock RC, Harari GM, Thomaz E, Shumake JD, Beevers CG, Gosling SD, Schnyer DM (2021) Improving prediction of real-time loneliness and companionship type using geosocial features of personal smartphone data. Smart Health 20:100180 36. Qirtas MM, Zafeiridi E, Pesch D, White EB (2022) Loneliness and social isolation detection using passive sensing techniques: scoping review. JMIR mHealth and uHealth 10(4):e34638

12

1 Introduction

37. Hester J, Sorber J (2017) The future of sensing is batteryless, intermittent, and awesome. In: Proceedings of the 15th ACM conference on embedded network sensor systems. ACM, p 21 38. Kansal A, Hsu J, Srivastava M, Raghunathan V (2006) Harvesting aware power management for sensor networks. In: Proceedings of the 43rd annual Design Automation Conference, 2006, pp 651–656 39. Sandhu MM, Khalifa S, Geissdoerfer K, Jurdak R, Portmann M (2021) SolAR: energy positive human activity recognition using solar cells. In: 2021 IEEE international conference on pervasive computing and communications (PerCom), IEEE, pp 1–10 40. Liu Y, Khanbareh H, Halim MA, Feeney A, Zhang X, Heidari H, Ghannam R (2021) Piezoelectric energy harvesting for self-powered wearable upper limb applications. Nano Select 41. Sandhu MM, Geissdoerfer K, Khalifa S, Jurdak R, Portmann M, Kusy B (2020) Towards energy positive sensing using kinetic energy harvesters. In: 2020 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 42. Lan G, Xu W, Khalifa S, Hassan M, Hu W (2016) Transportation mode detection using kinetic energy harvesting wearables. In: IEEE international conference on pervasive computing and communication (PerCom) workshops, Sydney, Australia, pp 1–4 43. Lin Q, Xu W, Lan G, Cui Y, Jia H, Hu W, Hassan M, Seneviratne A (2020) Kehkey: kinetic energy harvester-based authentication and key generation for body area network. In: Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, vol 4, no 1, pp 1–26

Chapter 2

Activity Recognition in IoT

Due to the advancements in technology and microelectromechanical systems, there is an exceptional development in the capabilities of sensors and smart devices. Nowadays people interact with these devices regularly in their daily lives due to the enhanced computational power, compact size, user-friendly interface and reduced cost of these devices. This ultimately leads to employing these sensors to extract knowledge about the surrounding environment and the underlying physical phenomenon–also known as Ubiquitous Sensing [1, 2]. In particular, recognising and extracting information about human activities has become an area of interest with many applications including gaming, e-commerce, security, healthcare, sports, military and surveillance. For example, regular exercise can reduce all-cause mortality, disability, and cardiovascular diseases in older adults [3] and people with chronic health conditions. In addition, daily activity pattern, sleep duration and diet intake affect the fitness level [4] of people. Therefore, recognising human activities such as walking, running, jumping, sitting, standing, etc., has paramount importance to monitor the daily patterns of users and to provide feedback to the user, caregiver(s) and healthcare professional. Similarly, people with sleep apnea, dementia and autism spectrum disorder can be monitored in real-time to provide timely assistance to avoid undesirable consequences [5–7]. Likewise, fall events related to elderly people are of concern and, therefore, monitoring their physical activities is important to minimise the risk of injury or illness [8]. Finally, precise information about the human activities not only helps healthcare professionals in health/fitness monitoring but also acts as a stepping stone towards providing real-time assistance in case of an unusual, abnormal or critical event.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9_2

13

14

2 Activity Recognition in IoT

2.1 Activity Recognition Mechanisms HAR systems automatically recognise activities performed by individuals using raw data collected from various sensors. HAR is performed using two broad methods namely external and on-body sensors. In the former case, sensors are deployed in the ambient environment to infer activities when a user interacts with them during their daily activities. In the latter case, various sensors are placed on the body (e.g., wearable devices) to accurately detect the activities performed by the individuals in various environments. Ambient sensors deployed in a home environment [9, 10] are an example of activity recognition using external sensors. These in-home sensors are an indirect source to recognise activities when the user makes an interaction with them. These interactions include touching, passing by them or generating a sound signal which is captured by these sensors to infer the activity. External sensors are generally employed in applications where users are expected to interact with them regularly. In other words, these sensors may not perform well when users do not interact with them or when activities are performed outside their vicinity and coverage zone. Moreover, these sensors involve high installation and maintenance cost which makes them less attractive for pervasive deployment. Cameras [11, 12], as external sensors, have also been widely employed for activity recognition in various applications. The captured videos from cameras are used to develop a machine learning algorithm which is later used to infer the activities. For example, a commercially available device [13] uses cameras for 3D sensing and gesture recognition. Although camera-based activity recognition offers promising results, it has some inherent disadvantages. The first one is related to security and privacy as a lot of people do not want to be continuously monitored using videos. Secondly, camera-based activity recognition can only be performed in a built environment where cameras are deployed in fixed positions and thus lack the pervasive use in natural outdoor environments. Thirdly, the processing of video data is a complex task and it requires intensive resources that are rarely available on smart user devices. Finally, cameras are generally expensive and can cover only a limited range. Thus, activities performed outside their range are ignored and can not be recognised. These limitations make camera-based activity recognition a less attractive choice for the general population. Due to the aforementioned limitations of ambient sensors and cameras, wearable (on-body) sensors are the most attractive choice for human activity recognition applications. These wearable sensors offer the advantages of small form factor, user-friendliness, and low cost. The wearable sensors include motion sensors (e.g., accelerometer, magnetometer, gyroscope, GPS), physiological sensors (e.g., heart rate, skin temperature, blood pressure) and environmental sensors (e.g., humidity, temperature, pressure).

2.3 Activity Recognition Using Machine Learning

15

2.2 Wearable Sensors for HAR Due to the burgeoning advancements in technology, smart wearable devices [14] such as smartwatches, smart bands and smart rings are being widely used by the people. These devices contain an activity sensor that can collect data continuously which can also be employed to detect and monitor the physical activities. In addition to activity sensing, modern wearable devices [15] include sensors to measure hear rate, blood pressure, skin temperature, breathing rate and sleep pattern. Most of the smart devices include inertial sensors, physical health sensors, environmental sensors, camera, and microphones. These sensors are employed to collect data continuously which can later be used to infer the corresponding activities and health conditions. We discuss the data processing and activity recognition model development process in the following subsections.

2.3 Activity Recognition Using Machine Learning Similar to other domains, machine learning models can be developed using the collected data from wearable IoT devices to infer the human activities [16, 17]. Figure 2.1 shows a general workflow from data collection to the development of machine learning models for HAR. We elaborate various steps of model development in the following subsections.

2.3.1 Data Acquisition and Preprocessing First of all, the data is acquired from various sensors and stored in a machine/computer to develop a machine learning model. As this is the raw data extracted directly from sensors, it may contain noise and unwanted components. For example, if a sensor is imprecise, the collected data can be noisy, and if a sensor fails, there might be missing values [18]. Therefore, the collected data is preprocessed to remove abnormal and missing values [19]. Depending upon the type of signal, a filter can also be employed to remove a certain frequency component from the collected data [20]. For example, the authors in [21] employ a moving average filter of order three to remove random noise components in the accelerometer signal. The resulting high fidelity signal may offer improved performance in recognising human activities.

16

2 Activity Recognition in IoT

Fig. 2.1 A general work flow for a developing machine learning model for HAR

2.3.2 Segmentation Segmentation is performed to group the data sequences that share the same characteristics. In sensor-based HAR, the collected data is segmented into sequences of fixed length so that each of them is analysed separately and sequentially. This mechanism is also known as time windows or sliding windows. Table 2.1 shows that Table 2.1 Data window size in the literature

Window size (seconds)

Articles

80%) for energy harvesting based sensing using well-known machine learning classification algorithms. This means that energy harvesting based sensing can be used in place of conventional power hungry activity sensors (such as accelerometers and magnetometers) to save the energy while attaining reasonable context detection performance. In order to further enhance the context detection accuracy, deep learning and neural network based models can be

References

65

employed [40], which have shown promising results in various applications such as speech recognition [41], face recognition [42] and natural language processing [43]. In Part II of this book, we have explored how energy harvesting can support more sustainable operation of IoT devices. We have also discussed how energy harvesters can be employed as sources of energy or information. Part III of this book focuses on the how energy harvesters can be used simultaneously as sources of energy and information to enable self-powered HAR in IoT.

References 1. Khalifa S, Lan G, Hassan M, Seneviratne A, Das SK (2017) Harke: Human activity recognition from kinetic energy harvesting data in wearable devices. IEEE Trans Mobile Comput 17(6):1353–1368 2. Sandhu MM, Geissdoerfer K, Khalifa S, Jurdak R, Portmann M, Kusy B (2020) Towards optimal kinetic energy harvesting for the batteryless iot. In: 2020 IEEE international conference on pervasive computing and communications workshops (PerCom Workshops). IEEE, pp 1–6 3. Martin P, Charbiwala Z, Srivastava M (2012) DoubleDip: leveraging thermoelectric harvesting for low power monitoring of sporadic water use. In: Proceedings of the 10th ACM conference on embedded network sensor systems (Sensys). Toronto, Canada, pp 225–238 4. Umetsu Y, Nakamura Y, Arakawa Y, Fujimoto M, Suwa H (2019) Ehaas: energy harvesters as a sensor for place recognition on wearables. In: Proceedings of the 2019 IEEE international conference on pervasive computing communications (PerCom). IEEE, pp 1–10 5. Kalantarian H, Sarrafzadeh M (2016) Pedometers without batteries: an energy harvesting shoe. IEEE Sens J 16(23):8314–8321 6. Khalifa S, Hassan M, Seneviratne A (2015) Step detection from power generation pattern in energy-harvesting wearable devices. In: IEEE international conference on data science and data intensive systems. IEEE, pp 604–610 7. Lan G, Xu W, Khalifa S, Hassan M, Hu W (2017) Veh-com: demodulating vibration energy harvesting for short range communication. IEEE international conference on pervasive computing and communications (PerCom). Hawaii, USA, pp 170–179 8. Khalifa S, Hassan M, Seneviratne A (2016) Feasibility and accuracy of hotword detection using vibration energy harvester. IEEE 17th international symposium on a world of wireless, mobile and multimedia networks (WoWMoM). Coimbra, Portugal, pp 1–9 9. Khalifa S, Hassan M, Seneviratne A, Das SK (2015) Energy-harvesting wearables for activityaware services. IEEE Internet Comput 19(5):8–16 10. Khalifa S, Lan G, Hassan M, Hu W (2016) A bayesian framework for energy-neutral activity monitoring with self-powered wearable sensors. In: 2016 IEEE international conference on pervasive computing and communication workshops (PerCom Workshops). IEEE, pp 1–6 11. Manjarrés J, Lan G, Gorlatova M, Hassan M, Pardo M (2021) Deep learning for detecting human activities from piezoelectric-based kinetic energy signals. IEEE Internet Things J 9(10):7545– 7558 12. Lin Q, Peng S, Wu Y, Liu J, Hu W, Hassan M, Seneviratne A, Wang CH (2020) E-jacket: posture detection with loose-fitting garment using a novel strain sensor. In: 2020 19th ACM/IEEE international conference on information processing in sensor networks (IPSN). IEEE, pp 49– 60 13. Lan G, Ma D, Xu W, Hassan M, Hu W (2017) Capsense: capacitor-based activity sensing for kinetic energy harvesting powered wearable devices. In: Proceedings of the 14th EAI international conference on mobile and ubiquitous systems: computing, networking and services. ACM, pp 106–115

66

4 Energy Harvester as an Information Source

14. Lan G, Ma D, Xu W, Hassan M, Hu W (2020) Capacitor-based activity sensing for kineticpowered wearable iots. ACM Trans Internet Things 1(1):1–26 15. Xu W, Lan G, Lin Q, Khalifa S, Bergmann N, Hassan M, Hu W (2017) Keh-gait: towards a mobile healthcare user authentication system by kinetic energy harvesting. In: NDSS 16. Ma D, Lan G, Xu W, Hassan M, Hu W (2018) SEHS: simultaneous energy harvesting and sensing using piezoelectric energy harvester. In: 2018 IEEE/ACM third international conference on internet-of-things design and implementation (IoTDI). IEEE, pp 201–212 17. Ma D, Lan G, Xu W, Hassan M, Hu W (2020) Simultaneous energy harvesting and gait recognition using piezoelectric energy harvester. IEEE Trans Mobile Comput 18. Abadi MJ, Khalifa S, Kanhere SS, Hassan M (2016) Energy harvesting wearables can tell which train route you have taken. In: 2016 IEEE 41st conference on local computer networks workshops (LCN workshops). IEEE, pp 199–204 19. Lan G, Xu W, Khalifa S, Hassan M, Hu W (2016) Transportation mode detection using kinetic energy harvesting wearables. IEEE international conference on pervasive computing and communication (PerCom) workshops. Australia, Sydney, pp 1–4 20. Lan G, Xu W, Ma D, Khalifa S, Hassan M, Hu W (2019) Entrans: leveraging kinetic energy harvesting signal for transportation mode detection. IEEE Trans Intell Transp Syst 21. Sandhu MM, Geissdoerfer K, Khalifa S, Jurdak R, Portmann M, Kusy B (2020) Towards energy positive sensing using kinetic energy harvesters. In: 2020 IEEE international conference on pervasive computing and communications (PerCom), IEEE, pp 1–10 22. Xiang T, Chi Z, Li F, Luo J, Tang L, Zhao L, Yang Y (2013) Powering indoor sensing with airflows: a trinity of energy harvesting, synchronous duty-cycling, and sensing. In: Proceedings of the 11th ACM conference on embedded networked sensor systems. ACM, p 16 23. Lan G, Khalifa S, Hassan M, Hu W (2015) Estimating calorie expenditure from output voltage of piezoelectric energy harvester: an experimental feasibility study. In: Proceedings of the 10th EAI international conference on body area networks, ICST. Sydney, Australia, pp 179–185 24. Kalantarian H, Alshurafa N, Le T, Sarrafzadeh M (2015) Monitoring eating habits using a piezoelectric sensor-based necklace. Comput Biol Med 58:46–55 25. Safaei M, Meneghini RM, Anton SR (2018) Energy harvesting and sensing with embedded piezoelectric ceramics in knee implants. IEEE/ASME Trans Mechatron 23(2):864–874 26. Lin Q, Xu W, Liu J, Khamis A, Hu W, Hassan M, Seneviratne A (2019) H2b: heartbeat-based secret key generation using piezo vibration sensors. In: Proceedings of the 18th international conference on information processing in sensor networks. ACM, pp 265–276 27. Lin Q, Xu W, Lan G, Cui Y, Jia H, Hu W, Hassan M, Seneviratne A (2020) Kehkey: Kinetic energy harvester-based authentication and key generation for body area network. Proc ACM Interact, Mobile, Wearable Ubiquitous Technol 4(1):1–26 28. Varshney A, Soleiman A, Mottola L, Voigt T (2017) Battery-free visible light sensing. In: Proceedings of the 4th ACM workshop on visible light communication systems, pp 3–8 29. Ma D, Lan G, Hassan M, Hu W, Upama MB, Uddin A, Youssef M (2019) Solargest: ubiquitous and battery-free gesture recognition using solar cells. In: The 25th annual international conference on mobile computing and networking. ACM, pp 1–15 30. Sandhu MM, Khalifa S, Geissdoerfer K, Jurdak R, Portmann M (2021) SolAR: energy positive human activity recognition using solar cells. In: 2021 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 31. Campbell B, Ghena B, Dutta P (2014) Energy-harvesting thermoelectric sensing for unobtrusive water and appliance metering. In: Proceedings of the 2nd international workshop on energy neutral sensing systems. ACM, Memphis, pp 7–12 32. Zarepour E, Hassan M, Chou CT, Adesina AA (2015) Remote detection of chemical reactions using nanoscale terahertz communication powered by pyroelectric energy harvesting. In: Proceedings of the second annual international conference on nanoscale computing and communication. ACM, p 8 33. Zarepour E, Hassan M, Chou CT, Adesina AA (2017) Semon: sensorless event monitoring in self-powered wireless nanosensor networks. ACM Trans Sens Netw (TOSN) 13(2):15

References

67

34. Pradhan S, Chai E, Sundaresan K, Qiu L, Khojastepour MA, Rangarajan S (2017) Rio: a pervasive rfid-based touch gesture interface. In: Proceedings of the 23rd annual international conference on mobile computing and networking (MobiCom). ACM, Snowbird, pp 261–274 35. Wang J, Abari O, Keshav S (2018) Challenge: Rfid hacking for fun and profit. In: Proceedings of the 24th annual international conference on mobile computing and networking (MobiCom). ACM, New York, pp 461–470 36. Khamis A, Kusy B, Chou CT, McLaws ML, Hu W (2020) Rfwash: a weakly supervised tracking of hand hygiene technique. In: Proceedings of the 18th conference on embedded networked sensor systems, pp 572–584 37. Blank P, Kautz T, Eskofier BM (2016) Ball impact localization on table tennis rackets using piezo-electric sensors. In: Proceedings of the ACM international symposium on wearable computers (ISWC). Heidelberg, Germany, pp 72–79 38. Lan G, Xu W, Ma D, Khalifa S, Hassan M, Hu W, Entrans: leveraging kinetic energy harvesting signal for transportation mode detection. IEEE Trans Intell Transp Syst 39. Xu W, Lan G, Lin Q, Khalifa S, Hassan M, Bergmann N, Hu W (2019) KEH-Gait: using kinetic energy harvesting for gait-based user authentication systems. IEEE Trans Mobile Comput 18(1):139–152 40. Xu W, Feng X, Wang J, Luo C, Li J, Ming Z (2019) Energy harvesting-based smart transportation mode detection system via attention-based lstm. IEEE Access 7:66423–66434 41. Zhao J, Mao X, Chen L (2019) Speech emotion recognition using deep 1d and 2d cnn lstm networks. Biomed Signal Process Control 47:312–323 42. Li Y, Zeng J, Shan S, Chen X (2018) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450 43. Young T, Hazarika D, Poria S, Cambria E (2018) Recent trends in deep learning based natural language processing. IEEE Comput Intell Mag 13(3):55–75

Part III

Self-Powered IoT

Chapter 5

Simultaneous Sensing and Energy Harvesting

In contrast to conventional activity sensors, energy harvesters provide both energy as well as context information. We have discussed in the previous chapter that kinetic, solar, thermal and RF energy harvesters can be used to detect the underlying activity in various applications. Among the available options, KEH is the most common harvesting mechanism, which is used to detect the activity in various applications involving mobility. Most of the previous works employ KEH open circuit voltage for context detection which is inefficient in practical scenarios, as energy harvesters can also be used to generate energy, but energy harvesting is not possible with open circuit, in the absence of a load. Therefore, simultaneous sensing and energy harvesting is a promising concept for maximising the benefit of the energy harvesters. This chapter describes the mechanism of simultaneously sensing and harvesting energy using KEH, to detect the underlying activity and to power the sensor nodes in IoT for their perpetual, uninterrupted and reliable operation.

5.1 Challenges in Simultaneous Sensing and Energy Harvesting Energy harvesters can be used as activity sensors as well as sources of energy for detecting underlying activity and powering the sensor nodes. The generated energy from the energy harvesting transducer is stored in an Energy Storage Unit (ESU), which is later used to run a system load. Figure 3.2 highlights the general energy harvesting mechanism using a DC-DC boost converter [1] in the Power Management © [2020] IEEE. Reprinted, with permission, from M. M. Sandhu, K. Geissdoerfer, S. Khalifa, R. Jurdak, M. Portmann and B. Kusy, “Towards Energy Positive Sensing using Kinetic Energy Harvesters,” 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom), Austin, TX, USA, 2020, pp. 1–10, doi: 10.1109/PerCom45495.2020.9127356.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9_5

71

72

5 Simultaneous Sensing and Energy Harvesting

Unit (PMU).1 The harvested energy is first stored in a capacitor and then it is used to run a hardware device. It is evident from Table 4.1 that most of the previous works focus on employing open circuit AC or capacitor voltages for extracting the activity information from the energy harvester. Figure 3.2 depicts the basic building blocks of an energy harvesting circuit, which includes a transducer, PMU and an ESU. When a load is connected to the energy harvesting circuit, it modifies the shape of the input AC as well as capacitor voltages because the AC voltage (V AC ) of the transducer depends on the capacitor voltage (Vcap ) as: V AC = Vcap + V pmu

(5.1)

where, V pmu is typically equal to the voltage drop across two diodes in the bridge rectifier. The shape of the harvesting AC voltage depends upon the capacitor voltage, which may impact its embedded activity information content, compared to the original open circuit AC voltage. Similarly, when a load is connected to the capacitor, it discharges the capacitor after irregular time intervals, instead of by using fixed manual discharging [2]. The reason for this lies in the distinct vibration pattern during various activities using KEH, which produces different voltage levels at the output of the transducer; thus charging the capacitor at a different rate. Therefore, the harvested energy (stored in the capacitor) distorts the generated AC signal according to Eq. 5.1, if energy harvesting and sensing are performed simultaneously using the same KEH. Therefore, the authors in [3] employ two KEH transducers to procure accurate context information in the presence of a distorted AC signal. They design a hardware prototype which is embedded in a shoe and contains two KEH transducers mounted in the front and rear of the shoe. They observe that the harvested AC voltage increases in amplitude due to the rise in the capacitor voltage, as depicted in Fig. 5.1.

Fig. 5.1 The harvesting AC voltage increases [3] with the rise in KEH capacitor voltage 1

In the literature, PMU is also called as an Energy Management Unit (EMU).

5.2 System Architecture for Simultaneous Sensing and Energy Harvesting

73

This is because the charging current decreases when the capacitor is being charged to a higher voltage level due to the decrease in potential difference between the transducer and capacitor. Furthermore, KEH has a significantly high internal resistance of the order of M [4, 5] and the output voltage is determined by the load resistance as well as the internal resistance of Piezoelectric Energy Harvester (PEH) [4, 6]. As the current flow is decreased, the voltage on the internal resistance also decreases, enhancing the corresponding output voltage. This modified (and distorted) AC signal may have less information content as compared to the original AC signal. The authors in [3] propose a filtering algorithm to remove the effect of capacitor voltage on the harvesting AC voltage to enhance the gait recognition accuracy. However, this filtering algorithm is difficult to implement on the miniaturised and resource constrained sensor nodes in real-world practical environments with a limited and time-varying supply of harvested energy. The authors in [3] do not employ the harvested energy to power a system load. Instead, they discharge the capacitors manually and then recharge them again with the harvested energy from the transducers. As a result of the above, more sophisticated algorithms are needed to extract useful information from the harvesting signals in the presence of an energy harvesting circuit while powering a realistic intermittently-powered load. This intermittent operation is due to the use of capacitors as ESU, which store a small amount of energy (also called energy burst) to run at most one atomic task, in contrast to batteries which can power nodes continuously for a longer duration. One drawback of employing capacitor voltage for activity detection is that the harvested energy, in some applications, is not sufficient to quickly charge the capacitor. In other words, sometimes, it takes longer to charge the capacitor up to a certain required level of load voltage (especially under lower vibrations), which may introduce uncontrolled delay in activity detection. One option is to employ a smaller capacitor that can be charged quickly thereby having a smaller energy burst that may not be sufficient to run the node for executing at least one atomic task. In summary, it is important to devise a sensing mechanism which provides activity information using the altered/distorted harvested signal in the presence of a capacitor as well as system load. This configuration will eventually let the energy harvester work as both source of energy and provider of context information simultaneously, leading towards the perpetual and autonomous operation of sensors in wearable Energy Harvesting based IoT (EH-IoT).

5.2 System Architecture for Simultaneous Sensing and Energy Harvesting The principal building blocks of the system architecture are shown in Fig. 5.2. A transducer converts the kinetic energy into electrical energy and a rectifier is used to rectify the harvested AC voltage. The rectified voltage charges a capacitor either directly or through a DC-DC converter. The energy from the capacitor is used to power a load, for example, the signal acquisition circuit, microcontroller, or a transceiver. We detail the main characteristics of the system architecture in the following subsections.

74

5 Simultaneous Sensing and Energy Harvesting

Fig. 5.2 Architecture of simultaneous sensing and energy harvesting with the availability of multiple sensing points

5.2.1 Sensing and Energy Harvesting Due to high source impedance [7], the voltage across a transducer changes dramatically when current flows, i.e., when closing the circuit to extract energy. When connecting a capacitor to the output of the rectifier, the capacitor voltage envelops the harvesting voltage and changes its pattern compared to the open circuit configuration. This has been described as the interference problem and a previous study proposes a filtering algorithm based on the capacitor voltage to reduce the effect of the problem on sensing signal quality [3]. Our architecture includes not only a capacitor, but also an intermittently powered load that uses the energy stored in the capacitor, as shown in Fig. 5.2. The load reflects the behaviour of a typical batteryless sensor node that switches on when enough charge has been accumulated in a small capacitor to, for example, sample, store, process or transmit the KEH data. The load discharges the capacitor and its dynamic behaviour thus also distorts the harvesting voltage waveform, as discussed in Sect. 5.3.3. In contrast to previous work [3], which uses custom filters to mitigate the effect of the capacitor on the harvesting AC voltage, we explore the potential of other sensing signals that do not suffer from the interference problem, allowing us to (1) reduce cost in terms of energy, delay and computational complexity by omitting these additional filter stages; and (2) exclude the hard-to-predict effects of dynamic load behaviour on sensing signal quality.

5.2.2 Energy-Positive Sensing In order to highlight the difference between conventional sensing and KEH-based sensing, we categorise sensing devices into two classes based on their energy profile: energy negative sensors and energy-positive sensors. The key difference between a conventional motion sensor and a KEH transducer is that the former consumes energy to convert the kinetic energy to an analog signal, whereas the latter generates energy, as shown in Fig. 5.3. Both types of sensors require an Analog-to-Digital Converter (ADC) to convert the analog signal to its digital form. Conventional motion sensors

5.3 System Design for Simultaneous Sensing and Energy Harvesting

75

Fig. 5.3 Conventional (energy-negative) versus energy-positive sensing

(such as accelerometers) are thus always energy negative and need to replenish energy regularly for their uninterrupted operation. KEH-based sensors, on the other hand, can be energy negative or energy-positive depending on the amount of harvested energy relative to the energy consumed for acquiring the KEH signal. If the harvested energy is higher than the energy required for signal acquisition, it is called energy-positive sensing.

5.2.3 Exploring Multiple Sensing Points Previous works employ open circuit AC voltage [8–10] from the energy harvester or capacitor voltage [11] for extracting context information. There are various sensing points in the energy harvesting circuit that offer two types of sensing signals i.e., voltage and current, which contain context information. Sensing points 1, 2 and 3 in Fig. 5.2 capture the current and voltage signals at the transducer, rectifier and energy storage unit, respectively. We evaluate various KEH signals by comparing the information content of each, when using different designs of the energy harvesting circuit.

5.3 System Design for Simultaneous Sensing and Energy Harvesting In this section, we discuss design options for simultaneous sensing and energy harvesting, present our hardware prototypes and analyse KEH signals.

76

5 Simultaneous Sensing and Energy Harvesting

5.3.1 Hardware Designs for KEH Sensing and Energy Harvesting We employ a piezoelectric transducer to convert ambient kinetic energy into electrical energy. Under mechanical stress, it generates an electric field with alternating polarity [8]. Most of the previous studies on KEH sensing [8–10, 12] directly use this open circuit AC voltage as a sensing signal. To extract usable electrical energy from the transducer, the circuit has to be closed, such that current can flow. The harvested current is typically alternating and thus needs to be rectified by, for example, a full bridge rectifier, consisting of four diodes. Batteryless design Traditional energy harvesting systems use large rechargeable batteries in order to compensate for variations in the harvested energy [13]. Batteryless devices instead use only a tiny capacitor to accumulate just enough energy to support the largest atomic operation [14], for example, transmitting a packet over a wireless link. Instead of running continuously as do battery-based devices, batteryless devices work opportunistically which means they operate only when energy is available in the capacitor. This works well when the temporal application requirements are well aligned with temporal energy availability, as in energy-positive sensors. These sensors only provide useful data when they also provide energy, which allows to dramatically reduce system cost and size by omitting large batteries. As the amount of harvested energy is usually too small to support perpetual operation of the device, it resorts to what is known as intermittent execution (shown in Fig. 5.4). The device remains completely off, while the capacitor accumulates charge from the harvester. on threshold is reached, the device switches on and executes, quickly Once the Vthr of f draining the capacitor until the Vthr threshold is reached and the device is powered off again. In contrast to previous work, we discuss the effect of this dynamic load behaviour in the context of the interference problem in Sect. 5.3.3. There are two main design options for charging the capacitor which are described in detail below:

Fig. 5.4 Capacitor voltage in intermittently powered sensors

5.3 System Design for Simultaneous Sensing and Energy Harvesting

77

Converter-less design In a converter-less design, the capacitor is placed in parallel to the rectifier; therefore, the voltage across the transducer is given by: v AC = vcap + 2 · vd

(5.2)

where, vcap is the capacitor voltage and vd is the voltage drop across the corresponding diode in the full wave rectifier. If the open circuit voltage is higher than the voltage on the capacitor, vd is approximately constant (one diode drop, typically 350 mV) and thus, the voltage across the transducer equals the capacitor voltage plus two diode drops. This may result in low energy yield. For example, if the capacitor voltage is 3 V and the input vibration is low, then the open circuit voltage of the transducer may be less than 3 V. In this case, no current can flow and thus energy that could have potentially been harvested is wasted. The dependence between capacitor voltage and transducer voltage has important implications for the sensing signal quality that will be discussed in Sect. 5.3.3 in detail. Converter-based design In a converter-based design, a DC-DC boost-converter is placed between the rectifier and the capacitor. This makes it feasible to optimise the operating point (i.e., harvesting voltage) of the transducer independent of the voltage on the capacitor. For example, the converter can be configured to regulate the voltage at its input to 100 mV by dynamically controlling the current flow from the transducer. This allows energy to be extracted for charging the capacitor from the transducer even under very low motion or vibrations. The decoupling of the transducer from the capacitor and load has another important effect: while the harvesting voltage is kept constant by the regulator and thus does not contain context information, the current changes approximately linearly with the kinetic energy input, yielding a high quality sensing signal that is not affected by the interference problem [3]. In this document, we refer to DC-DC boost converters, DC-DC converters and converters interchangeably.

5.3.2 Experimental Setup In this study, we design four different KEH hardware prototypes for collecting data from three types of KEH circuits: open circuit (Fig. 5.5a), converter-less (Fig. 5.5b) and converter-based (Fig. 5.5c). For the converter-less design, we use two prototypes; one measures the voltage signals, the other measures the current signals. Considering these different designs, the three sensing points in our architecture in Fig. 5.2 offer numerous sensing signals, ten of which we record and analyse as shown in Fig. 5.5. There is no current flowing in the open circuit design, so we only record the voltage before and after the rectifier. The voltage at sensing point 2b is the same as at 3b, therefore we do not sample it twice in Fig. 5.5b. Lastly, we can not sample the AC current in converter-less and converter-based designs and the AC transducer voltage in the converter-based design due to hardware limitations.

78

5 Simultaneous Sensing and Energy Harvesting

Fig. 5.5 Illustration of hardware prototypes developed for KEH data collection a Open circuit design that collects AC and rectified voltages, and b Converter-less and c Converter-based designs that sample the current and voltage signals at various sensing points in the circuit

We use a S230-J1FR-1808XB two-layered piezoelectric bending transducer from MIDÉ technology2 in all hardware designs. All signals are sampled with a 12-bit ADC at a sampling frequency of 100 Hz. The first design (see Fig. 5.5a) represents the open-circuit configuration and serves as a benchmark for comparison to the state-of-the-art [8–10, 12] Transport Mode Detection (TMD) systems. The other two designs use a 220 µF capacitor to temporarily store the harvested energy and an intermittently powered load3 consisting of two LEDs, mimicking the behaviour of a batteryless device. The third design (in Fig. 5.5c) uses a TI BQ25504 DC-DC boost-converter between the rectifier and the capacitor. The converter is configured to regulate its input voltage to an empirically calculated value of around 1 V. The prototypes are based on a portable testbed [15] and designed as dataloggers with a focus on enabling accurate measurements of signals at all sensing points rather than on optimizing harvesting efficiency. In order to analyse the characteristics of the available signals, in an initial study, we collect data from a full-sized adult tricycle using the four hardware prototypes as shown in Fig. 5.6. For data collection, the prototypes are placed in a plastic box sized 39 cm × 29 cm × 17 cm and the piezoelectric transducers are mounted on a 23 cm long metallic bar mounted inside the box. We use a block tip mass of 24.62 g ± 0.5% with each transducer to make it more sensitive to lower frequency vibrations. 2 3

https://www.mide.com/. on = 3.38V and V o f f = 2.18V . In this experiment, we set Vthr thr

5.3 System Design for Simultaneous Sensing and Energy Harvesting

79

Fig. 5.6 Experimental setup for data collection using tricycle

During the experiment using the tricycle, the box containing the prototypes is placed in the wire basket behind the saddle, as shown in Fig. 5.6.

5.3.3 The Interference Problem at Different Sensing Points Most previous researchers into KEH-based sensing use the open circuit AC voltage of the transducer as a sensing signal [8–10, 12]. However, Fig. 5.7 depicts example data traces from all signals collected from our hardware prototypes (including accelerometer and KEH). The AC voltage shown in Fig. 5.7b is proportional to the displacement of the tip and thus accurately reflects the excitation of the transducer. Figure 5.7c depicts the voltage after rectification, corresponding to the absolute values of the AC voltage. When closing the circuit by connecting a capacitor to the output of the rectifier, the voltage on the transducer is enveloped by the voltage on the capacitor, as discussed in Sect. 5.3.1. This has previously been described as the interference problem [3]. In this section, we extend the discussion of the interference problem to the effect of a transiently powered load that consumes energy from the capacitor. Furthermore, we explore various other sensing signals in different energy harvesting designs and discuss how they are affected by the capacitor and load. Effect of capacitor and load The effect of the combination of capacitor and load on harvested AC voltage is illustrated in Fig. 5.8. As expected, the AC voltage is enveloped by the capacitor voltage minus two diode drops (see Eq. 5.2). The dashed green graph shows the mirrored capacitor voltage, highlighting that, due to the rectifier, the signal is enveloped in either polarity. In a converter-less design, the capacitor is connected directly to the output of the rectifier. Thus, the converterless rectified voltage shown in Fig. 5.7e equals the capacitor voltage. The shape of the envelope is determined by the behaviour of the transiently powered load that is illustrated in Fig. 5.4. The gradual rise reflects the charging process and the sharp drop in voltage is due to the load being switched on. Nevertheless, the affected signals may still contain enough information to distinguish between various contexts with reasonable accuracy, as described in Sect. 5.5 in detail. The previous approach of applying filters to compensate for the capacitor charging curve [3] can not easily be applied when considering the hard-to-predict load behaviour. Instead, we seek

80

5 Simultaneous Sensing and Energy Harvesting

Fig. 5.7 Different types of signals from KEH on a tricycle; the labels indicate the sensing points in Fig. 5.5

5.3 System Design for Simultaneous Sensing and Energy Harvesting

81

Fig. 5.8 KEH AC voltage is enveloped in the capacitor voltage

to explore different sensing signals in the energy harvesting circuit that may be less affected by the interference problem. Current signals Figure 5.9 shows the harvesting current for converter-less and converter-based designs compared to the rectified harvested voltage. The rectified voltage equals the capacitor voltage and thus exhibits the expected envelope distortion. The current signals, in contrast, exhibit the typical waveform of a damped spring mass oscillator, which is a widely used model for a tip-mass loaded piezoelectric transducer [8]. These signals are approximately proportional to the displacement of the tip mass and thus incorporate details of the underlying physical process. However, current can only flow once the voltage across the transducer exceeds the voltage on the output of the rectifier plus two diode drops. Thus, the current is zero when the voltage on the transducer is lower than the threshold voltage and the corresponding information is lost. We call this threshold distortion. In a converter-less system, the threshold voltage is the varying capacitor voltage. In a converter-based system, it is the constant, configurable input voltage of the DC-DC converter. For example,

Fig. 5.9 Effect of capacitor voltage on the harvesting current

82

5 Simultaneous Sensing and Energy Harvesting

in our measurement campaign, we empirically set the input voltage to 1 V. As a result, current starts to flow at lower voltages, resulting in higher energy yield and more context information in the converter-based current signal (Fig. 5.9i) than in the converter-less signal (Fig. 5.9f). Threshold distortion is less critical than envelope distortion as it only affects the part of the signal when the amplitude of the signal is too low for harvesting. For energy-positive, batteryless sensing, the transducer must be dimensioned to provide sufficient energy when we want to sample the signal. Envelope distortion instead, affects the parts of the voltage signal when energy can be harvested. In summary, the harvested current signal is an attractive signal whose sensing potential has not been previously explored in sufficient detail.

5.4 Transport Mode Detection: A Case Study We employ TMD as a case study to compare the sensing performance of the considered KEH signals and elaborate results in the following subsections.

5.4.1 Data Collection With respect to the TMD application, our objective is to collect data from multiple transport modes, instead of multiple subjects. Therefore, a volunteer4 is asked to carry a box (containing the hardware prototypes) while travelling, and using multiple transport modes including ferry, train, bus, car, tricycle and pedestrian movement. During transitions between the vehicular transport modes, the volunteer walks as a pedestrian, varying between slow walking, brisk walking, moving upstairs/downstairs and having some stop periods. The data is collected from various transport modes in Brisbane city with variations in seating location, time of the day and transport routes, with an average duration of 78 min for each transport mode, as shown in Table 5.1. For each trial, we use a different vehicle or choose another route. We also vary the location of the box of prototypes within each vehicle (i.e., front, middle or rear section). This ensures that the collected data is representative of all types of vibrations experienced in the vehicles. In order to compare the performance of the proposed KEH-based architecture with the state-of-the-art TMD architectures, we also collect data using an MPU9250 3-axis accelerometer. Finally, the volunteer manually records each different mode of transport s/he uses across the course of the experiments, to serve as ground truth for classification. We do not use load voltage and current (at points 3b and 2c in Fig. 5.5) for sensing as the load is not turned on frequently, especially when vibrations are low such as on the ferry or train. Similarly, the rectified voltage in Fig. 5.5d (at point 1c) is not used for extracting information as it is fixed to a specific level by the converter and does not contain information. The remaining five KEH signals are analysed in terms of their sensing potential in Sect. 5.5. 4 Ethical approval has been granted from the University of Queensland [2019001916/106/19] and CSIRO [106/19] for carrying out this experiment.

5.4 Transport Mode Detection: A Case Study

83

Table 5.1 Detail of the collected data using KEH Transport mode Duration with stops Duration without stops Number of trials (min) (min) Ferry Train Bus Car Tricycle Pedestrian Total

156 125 144 98 72 216 811

93 96 76 83 59 66 473

4 4 6 2 4 8 –

5.4.2 System Model The proposed system model for TMD is depicted in Fig. 5.10. A wearable device collects real-time data from various sensing points in the KEH circuit while the wearer travels on various modes of transport. A server processes the collected data, removes stops/pauses, extracts the dominant feature set for TMD, and implements the classification techniques to classify the current transport mode. A computational unit located within the vehicle along the travel route or a personal digital assistant can serve the purpose of a server. Below, we explain each component in detail.

5.4.2.1

Pre-processing

First, we convert the ADC readings into actual voltage and current signals. When the vehicle is stationary, for example at traffic lights, there are lower vibrations as compared to when it is moving, making TMD difficult [10]. Therefore, we detect

Fig. 5.10 The model architecture for transport mode detection

84

5 Simultaneous Sensing and Energy Harvesting

and remove these stops/pauses based on thresholds which are calculated from the average value of the energy harvesting signals [16].

5.4.2.2

Feature Selection

We divide the collected data into equal sized windows with a 50% overlap and extract multiple time and frequency domain features as described in [8, 10, 17]. As individual features can embed varying levels of information content about the transport mode, we employ Recursive Feature Elimination (RFE) to find the minimal and most significant feature set using Cross Validation (CV) [18]. Table 5.2 shows the total number of features selected in various types of sensing signals with the corresponding CV score. It also depicts that only 9 features are selected out of 42 time and frequency domain features from the converter-based current signal. These are the common features selected from all types of signals (which include maximum value, minimum value, amplitude range, coefficient of variation, skewness, kurtosis, interquartile range, absolute area, and root mean square value). Note that the converterbased current signal offers comparable CV scores to the accelerometer and open circuit AC voltage signals using a smaller feature set, indicating the rich information content embedded in it.

5.4.2.3

Classification

Five well-known machine learning classifiers are implemented including Random Forest (RF), Decision Tree (DT), Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and Naive Bayes (NB). For each classifier, we perform 10-fold cross validation [10] and plot all results averaged with 95% confidence interval. Prior to the implementation of classification algorithms, we use Synthetic Minority Oversampling Technique (SMOTE) [19] to handle imbalanced data from the various transport modes and normalise the selected features with zero mean and standard deviation of one.

Table 5.2 The number of features selected using RFE Initial features Using RFE Signal Accelerometer (Acc) OC-AC-V OC-REC-V CL-AC-V CL-REC-V CL-C CB-C

128 42 42 42 42 42 42

36 26 31 26 36 29 09

CV score 0.95 0.93 0.86 0.92 0.84 0.72 0.93

5.5 Performance Evaluation

85

5.5 Performance Evaluation In this section, we evaluate the performance of the proposed architecture using TMD as a case study.

5.5.1 Detection Accuracy of KEH-Based Sensing Signals 5.5.1.1

Different Classifiers

We compare TMD accuracies of an accelerometer and five KEH-based sensing signals using the classification schemes considered in Fig. 5.11. The results indicate that the RF classifier outperforms other classifiers for all signals, including the accelerometer. Therefore, in the remainder of the document, we present the results from the RF classifier. For all classifiers, the open circuit rectified voltage achieves lower detection accuracy than the open circuit AC voltage due to the loss of information by inverting the negative part of the signal. Also, the converter-less current signal achieves lower detection accuracy than the converter-based current signal due to the interference problem of the capacitor voltage in the former, as discussed in Sect. 5.3.3. Intuitively, although the pattern of converter-less AC voltage is affected due to the interference problem shown in Fig. 5.8, we find that its context detection performance (93.95%) is not highly affected compared to the open circuit AC voltage (94.56%); however, it provides lower detection accuracy than the accelerometer signal (98.35%), as depicted in Fig. 5.11. The converter-based current signal offers the highest detection accuracy (96.85%) from all the KEH-based signals, which is close to the detection accuracy of the accelerometer signal (98.35%) for a window size of one second. This shows that using the converter to decouple the transducer and capacitor, allows the flow of current during smaller vibrations, ultimately

Fig. 5.11 TMD accuracy of accelerometer and KEH signals using five classification algorithms (window size = 1 s)

86

5 Simultaneous Sensing and Energy Harvesting

yielding context-rich KEH signals. It is worth mentioning that any single axis signal of accelerometer offers lower detection accuracy (Accx : 92.10%, Acc y : 93.12%, Accz : 90.70%) than the combined 3-axis signal (98.35%), for a window size of one second, due to the higher information content across spatial dimensions embedded in the 3-axis signal. Therefore, all results presented in this document employ a 3-axis accelerometer signal as the key benchmark, while KEH signals are inherently single axis in nature.

5.5.1.2

Varying Window Size

We now study the impact of window size on the TMD accuracy of the accelerometer, open circuit AC voltage, converter-less AC voltage and converter-based current, these being the four signals that provide highest detection accuracy. We plot the accuracy for each signal with varying window sizes from 1 s to 10 s in Fig. 5.12. It is clearly shown that the TMD accuracy increases with the increasing window sizes for all, including the accelerometer and KEH signals. Although the open circuit AC voltage signal offers lower accuracy than the accelerometer signal for smaller window sizes, it provides comparable accuracy to the accelerometer signal for window sizes greater than 7 s. Similarly, with converter-less AC voltage, the detection accuracy increases with the increasing window size, as depicted in Fig. 5.12. However, it is slightly lower than the open circuit AC voltage for all window sizes due to the interference problem experienced with the inclusion of an energy harvesting circuit. Nevertheless, the converter-based current signal achieves comparable accuracy to the accelerometer even with a smaller window size (starting from 1 s) as shown in Fig. 5.12. This is at least 75% higher than that achieved in previous work [10]

Fig. 5.12 Accuracy of TMD using accelerometer and KEH signals with increasing window sizes

5.5 Performance Evaluation

87

(for a window size of 1 s), that employs open circuit AC voltage for detecting the transport mode. Furthermore, for larger window sizes (10 s), the converter-based current signal provides the same accuracy as the accelerometer signal (98.5%). It is worth noting that the KEH converter-based current signal offers high detection accuracy using fewer features than the conventional 3-axis accelerometer signal, as depicted in Table 5.2.

5.5.1.3

Current Signal Versus AC Voltage

We now focus on comparing the KEH current signal with the converter-less AC voltage signal in order to distinguish between the transport modes considered. Figure 5.13 shows the confusion matrices of both signals, highlighting that converter-less AC voltage signal provides detection accuracy of more than 94% for most of the transport modes. However, the detection accuracy for trains (87.63%) and ferries (90.76%) is the lowest as both of these modes of transport have dedicated non-road transport surfaces (i.e., river water and tracks respectively), resulting in similar, low vibration amplitudes. Furthermore, the lower signal amplitude for these modes of transport is more susceptible to noise, which makes it harder to differentiate between them. Similarly, for converter-based current signal, an accuracy of higher than 97% is achieved for most of the modes of transport except for ferries and trains, where 92.55% and 91.94% accuracies are achieved, respectively. The reason behind the high detection accuracy of the converter-based current lies in the lower threshold distortion as compared to the converter-less signals (voltage and current), as depicted in Fig. 5.9. Other types of signals (like converter-less AC voltage and current) are affected by

Fig. 5.13 Confusion matrices for TMD using a KEH converter-less AC voltage and b KEH converter-based current signals (window size = 1 s)

88

5 Simultaneous Sensing and Energy Harvesting

the charged capacitor as it hinders the flow of current from the transducer, especially during lower vibrations. On the other hand, in the converter-based design, the capacitor voltage is decoupled from the transducer voltage and current flows even during lower vibrations, which reflects the detailed physical phenomenon and enhances the detection accuracy.

5.5.2 Energy Harvesting In order to compare the energy harvesting potential between transport modes and circuit designs, we calculate the average harvesting power by first calculating the stored energy on the capacitor for any point in time, then adding up all positive changes in the stored energy and finally dividing by the recording duration. The results are presented in Table 5.3. The highest power is harvested in the car and on the tricycle, where we observe strong vibrations close to the resonant frequency of the transducer (≈25 Hz). On the other hand, due to smooth pathways and lower vibration amplitudes, we record the lowest power on the ferry and the train. In most cases, we observe significantly higher energy yield with the converter-based design than with the converter-less energy harvesting design.

5.5.3 Energy Consumption and System Costs In Sect. 5.5.1, we discovered that the current into the DC-DC converter can be used as a sensing signal for TMD, achieving detection accuracy comparable to a 3-axis accelerometer signal. Previous work found that sampling the voltage of a KEH transducer consumes orders of magnitude less energy than sampling a state-of-the-art accelerometer [8]. In this section, we show how the harvesting current signal can

Table 5.3 Harvested power for various transport modes using converter-less and converter-based KEH circuits Transport mode Harvested Power [µW] Converter-less Converter-based Ferry Train Bus Car Tricycle Pedestrian Average

0.1 0.06 3.2 4.0 5.3 3.7 2.7

0.2 0.1 20.4 27.8 20.4 10.5 13.2

5.5 Performance Evaluation

89

Fig. 5.14 Current sensing mechanism using an amplifier

be accessed in a real system and how this solution compares to accelerometers in terms of power requirements and costs. We propose to use a shunt ampere-meter to convert the current into a voltage that can easily be sampled by the sensor node using an ADC, as shown in Fig. 5.14. The current over resistor R S causes a negative voltage drop that is inverted and amplified by the inverting operational amplifier A. By choosing the corresponding resistor values, the amplifier output can be adjusted to match the input range of the ADC. For all power calculations, we assume a supply voltage of 3 V. Using a low power operational amplifier (e.g. TI LPV521, 350 nA) and low value for the shunt resistor, the sum of power consumption and losses is around 2 µW under typical harvesting conditions. This is more than two orders of magnitude less than the power consumption of the lowest power analog accelerometer (ADXL356, 450 µW) that we could find. When adding the current for an external, low power ADC (e.g., ADS7042, ≈700 nW @ 100 Hz), the power consumption of the proposed system is still less than 3 µW. Highly integrated, digital accelerometers achieve significantly lower power consumption than their analog counterparts (e.g., ADXL363, 72 µW@ 100 Hz) by co-design of the sensor and the signal acquisition chain and duty-cycling according to the configured sampling rate. This is still more than twice as much as our proposed KEH current sensor consumes based on discrete, off the shelf components. We expect that integrating our circuit with an optimized signal acquisition chain would further reduce the power consumption. Figure 5.15 compares the harvested and consumed power for accelerometer, open circuit voltage, converter-less voltage and converter-based current. It shows that the KEH circuits consume significantly lower energy than the accelerometer for providing the sensing signals. The accelerometer and KEH open circuit design consume power from an external source without generating energy. On the other hand, on average, converter-less and converter-based designs harvest more energy than that required for signal acquisition, leading towards energy-positive sensing.

90

5 Simultaneous Sensing and Energy Harvesting

Fig. 5.15 Comparison between the consumed power in signal acquisition and the harvested power using KEH

Costs The component costs for the proposed KEH current sensing circuit, including the amplifier (0.49 USD) and three resistors ( 1. The boundary between the two classes at APR = 1 represents energy neutral sensing. In Fig. 5.16, we plot the APR over the achieved TMD accuracy for all combinations of transport modes and sensing signals. Although the accelerometer signal provides the highest detection accuracy, it has zero APR due to zero harvested power. Similarly, KEH-based sensing in an open circuit configuration offers lower accuracy than the accelerometer signal and provides zero APR as no energy is being harvested. Instead both accelerometer and KEH open circuit designs consume energy from an external source for signal acquisition. Therefore, both of these devices are energy negative for all transport modes. The transducer voltage in the converter-less design (CL-AC-V) offers energy-positive sensing with APR of four to eight for four out of six considered transport modes with reasonable detection accuracy. However, the converter-based

5.6 Discussion

91

Fig. 5.16 Comparison of various signals in terms of APR and TMD accuracy (window size = 1 s)

current signal (CB-C) outperforms all KEH signals and offers TMD accuracy comparable to the accelerometer signal, with an APR of four to ten for four out of six transport modes considered. In summary, the results show that energy-positive sensing is possible with both converter-less and converter-based energy harvesting designs for at least two-thirds of the transport modes considered. Depending on the transport mode, KEH-based sensing circuit provides up to ten times as much power as required for the signal acquisition while offering detection accuracy close to the 3-axis accelerometer.

5.6 Discussion In this chapter, we present a scheme to use KEH simultaneously as a source of energy and information, thus enabling energy-positive sensing. The novelty of this work lies in the exploration of voltage and current signals at various sensing points in the energy harvesting circuit. In addition to sensing, we utilise the harvested energy from KEH to power a realistic load. We present transport mode detection as a case study, design four different KEH prototypes and collect data from six transport systems. Five classification techniques are implemented on five types of KEH signals and it is concluded that the harvesting current in a converter-based energy harvesting circuit achieves detection accuracy close to the accelerometer signal, with two-fold lower

92

5 Simultaneous Sensing and Energy Harvesting

energy consumption. The results show that energy-positive sensing is possible for at least two-thirds of the transport modes considered. The next chapter takes a further step towards self-powered Human Activity Recognition (HAR) in IoT. While this chapter has demonstrated how energy harvesters can be a source of both information and energy leading to energy-positive sensing, the KEH energy (particularly from human-centric applications) is not sufficient to fully power the other steps in the HAR pipeline, including processing and communication. The next chapter uses SEH as a novel context sensor to achieve fully self-powered HAR.

References 1. Geissdoerfer K, Chwalisz M, Zimmerling M (2019) Shepherd: a portable testbed for the batteryless iot. In: Proceedings of the 17th conference on embedded networked sensor systems, pp 83–95 2. Lan G, Ma D, Xu W, Hassan M, Hu W (2020) Capacitor-based activity sensing for kineticpowered wearable iots. ACM Trans Internet Things 1(1):1–26 3. Ma D, Lan G, Xu W, Hassan M, Hu W (2018) SEHS: simultaneous energy harvesting and sensing using piezoelectric energy harvester. In: 2018 IEEE/ACM third international conference on internet-of-things design and implementation (IoTDI). IEEE, pp 201–212 4. Huang Q, Mei Y, Wang W, Zhang Q (2016) Battery-free sensing platform for wearable devices: the synergy between two feet. In: IEEE INFOCOM 2016-The 35th annual IEEE international conference on computer communications. IEEE, pp 1–9 5. Xiang T, Chi Z, Li F, Luo J, Tang L, Zhao L, Yang Y (2013) Powering indoor sensing with airflows: a trinity of energy harvesting, synchronous duty-cycling, and sensing. In: Proceedings of the 11th ACM conference on embedded networked sensor systems. ACM, p 16 6. Sodano HA, Park G, Inman D (2004) Estimation of electric charge output for piezoelectric energy harvesting. Strain 40(2):49–58 7. Kalantarian H, Alshurafa N, Le T, Sarrafzadeh M (2015) Monitoring eating habits using a piezoelectric sensor-based necklace. Comput Biol Med 58:46–55 8. Khalifa S, Lan G, Hassan M, Seneviratne A, Das SK (2017) Harke: Human activity recognition from kinetic energy harvesting data in wearable devices. IEEE Trans Mob Comput 17(6):1353– 1368 9. Umetsu Y, Nakamura Y, Arakawa Y, Fujimoto M, Suwa H (2019) Ehaas: energy harvesters as a sensor for place recognition on wearables. In: Proceedings of the 2019 IEEE international conference on pervasive computing communications (PerCom). IEEE, pp 1–10 10. Lan G, Xu W, Ma D, Khalifa S, Hassan M, Hu W, Entrans: leveraging kinetic energy harvesting signal for transportation mode detection. IEEE Trans Intell Transp Syst 11. Lan G, Ma D, Xu W, Hassan M, Hu W (2017) Capsense: capacitor-based activity sensing for kinetic energy harvesting powered wearable devices. In: Proceedings of the 14th EAI international conference on mobile and ubiquitous systems: computing, networking and services. ACM, pp 106–115 12. Khalifa S, Hassan M, Seneviratne A, Das SK (2015) Energy-harvesting wearables for activityaware services. IEEE Internet Comput 19(5):8–16 13. Geissdoerfer K, Kusy B, Jurdak R, Zimmerling M (2019) Getting more out of energy-harvesting systems: energy management under time-varying utility with preact. In: 2019 18th ACM/IEEE international conference on information processing in sensor networks (IPSN). IEEE, pp 109– 120

References

93

14. Gomez A, Sigrist L, Magno M, Benini L, Thiele L (2016) Dynamic energy burst scaling for transiently powered systems. In: Proceedings of the 2016 conference on design, automation and test in Europe. EDA Consortium, pp 349–354 15. Geissdoerfer K, Chwalisz M, Zimmerling M (2019) Shepherd: a portable testbed for the batteryless iot. In: Proceedings of the 17th ACM conference on embedded networked sensor systems (SenSys), pp 83–95 16. Stockx T, Hecht B, Schöning J (2014) Subwayps: towards smartphone positioning in underground public transportation systems. In: Proceedings of the 22nd ACM SIGSPATIAL international conference on advances in geographic information systems. ACM, pp 93–102 17. Hemminki S, Nurmi P, Tarkoma S (2013) Accelerometer-based transportation mode detection on smartphones. In: Proceedings of the 11th ACM conference on embedded networked sensor systems. ACM, p 13 18. Guyon I, Weston J, Barnhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46(1–3):389–422 19. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority oversampling technique. J Artif Intell Res 16:321–357

Chapter 6

Solar Cell Based Activity Recognition

In the previous chapter, we employed a KEH transducer as a simultaneous source of energy and context information and showed that the harvested power from the transducer can be employed for signal acquisition leading towards energy-positive sensing. However, we observe that, in human-centric applications, the harvested energy from a tiny, single, untuned KEH is not sufficient to power all components of a Human Activity Recognition (HAR) system [1, 2]. Therefore, our objective in this chapter is to enable end-to-end energy-positive HAR, where the harvested energy exceeds the energy required for signal acquisition, classification, and activity transmission. To this end, we explore Solar-based Activity Recognition (SolAR), which employs solar cell as a sensor for activity recognition and a source of energy simultaneously. As human activities interfere with the ambient light differently, the output signal from the wearable solar cell embeds a signature of the underlying activity. In addition, the harvested power from the solar cell can be sufficient to run the end-to-end HAR algorithm (including signal acquisition, classification and real-time wireless activity transmission) and thus enables energy-positive HAR, as depicted in Fig. 6.1. Using well-known machine learning algorithms, we discover that, compared to conventional KEH-based HAR systems [3, 4], the proposed SolAR system delivers an order of magnitude higher harvested power indoors, and up to 8.3% higher HAR accuracy. In outdoor settings, SolAR offers comparable HAR accuracy and more than two orders of magnitude higher harvested power compared to KEH-based HAR. The significant increase in the harvested power enables real-time and energy-positive HAR. © [2021] IEEE. Reprinted, with permission, from M. M. Sandhu, S. Khalifa, K. Geissdoerfer, R. Jurdak and M. Portmann, “SolAR: Energy Positive Human Activity Recognition using Solar Cells,” 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom), Kassel, Germany, 2021, pp. 1–10, doi: 10.1109/PERCOM50583.2021.9439128. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9_6

95

96

6 Solar Cell Based Activity Recognition

Fig. 6.1 Using a solar cell for activity recognition as well as to power the sensor node, leading towards energy-positive HAR

6.1 Background This section describes the previous HAR mechanisms and preliminary study related to SEH-based HAR.

6.1.1 Previous HAR Mechanisms Previously developed HAR techniques rely mainly on conventional activity sensors [5] such as accelerometers and magnetometers which consume significant energy and require an external energy source for their perpetual operation [6]. In order to allow uninterrupted operation and to reduce the energy consumption of IoT sensor nodes, recently, KEH transducers are also being used as activity sensors for HAR. Khalifa et al. [3] show that KEH-based sensing can offer reasonable HAR accuracy with significantly reduced energy consumption compared to conventional activity sensors. Kalantarian et al. [7] design a KEH-based necklace for monitoring food intake and eating habits. Lan et al. [8] use a capacitor to store the harvested energy from KEH and then use the capacitor voltage signal for HAR. This reduces energy consumption due to the reduced sampling rate for acquiring the slowly varying capacitor voltage. Instead of using a single KEH transducer, Ma et al. [9] employ two transducers in a shoe to identify the underlying human activities. They use both transducers for HAR and for extracting energy (stored in the capacitors), but do not

6.1 Background

97

consider the effect of a realistic, dynamic load. Sandhu et al. [1], instead, use a KEH transducer as an activity sensor and source of energy simultaneously to power a dynamic load. They show that KEH offers energy-positive signal acquisition, which means that the harvested energy is higher than the energy required for acquiring the activity signal. However, in human-centric applications, the limited harvested energy from a KEH transducer may not be sufficient to run all tasks on the sensor node, including signal acquisition, classification and transmission, without the need for any external energy source [1]. This limits its application to scenarios with offline, cloud-based classification. Therefore, there is still a significant need for alternative HAR mechanisms that can ensure the autonomous and perpetual operation of sensor nodes for running the HAR algorithm, leading towards a truly pervasive energy harvesting-based IoT. Previous studies illustrate that ambient light contains information about the human context and, for example, can be used to analyse human eating habits [10] when combined with other sensory data. The authors in [11] deploy light sensors on the floor and use the output signals to detect human gestures. Zhang et al. [12] develop a photodiode array and use it in various applications including for detection of door opening/closing, liquid level detection, step count and touch detection, etc. Instead of using conventional light sensors, the authors of [13] employ solar cells to recognise various hand gestures under a fixed lamp. Furthermore, Li et al. [14] use arrays of photodiodes for finger gesture recognition and employ the harvested energy to power the gesture recognition module. However, they perform gestures by touching the arrays of photodiodes and thus their system may not recognise finger gestures performed in the vicinity of (without touching) the photodiode array. In another study [15], a combination of solar and kinetic energy harvesters (placed on the human chest) is used for room-level place recognition in buildings. These works employ SEH either in a controlled environment [13] (indoor only) or in combination with KEH transducers [15]. Furthermore, they (except [14]) employ SEH merely as an activity sensor without harvesting energy to power a dynamic load. Moreover, to the best of our knowledge, the potential of using solar cells to detect human physical activities (indoors and outdoors) in HAR applications remains unexplored. The harvested energy from a solar cell varies according to the intensity of incidental light and the orientation of the solar surface relative to the light source [13]. When solar cells are worn on the human body, the harvested energy changes during various human activities thanks to the different type of mobility relative to the source(s) of light as well as to the shadowing, which contains a unique signature of the underlying activity. Figure 6.2 plots the generated power from a wrist-wearable small-sized solar cell during three common indoor human activities: running, walking, and standing. As various human activities interfere with the ambient light differently, resulting in distinct harvesting patterns, we can use the harvesting signal as a sensing signal in order to classify the current activity. Thus, solar cells offer an attractive combination of activity information and harvested energy for realizing pervasive energy harvesting-based HAR.

98

6 Solar Cell Based Activity Recognition

Fig. 6.2 Solar harvested power during various human activities

6.2 Human Activity Recognition Using Solar Cell This section describes the architecture of our proposed SolAR system whereas the specific details of the implementation are provided in Sect. 6.3. We employ a wearable solar cell simultaneously as an activity sensor for HAR as well as an energy source to power the system load for the autonomous and perpetual operation of wearable IoT devices. Figure 6.3 depicts the architecture of the SolAR model, showing both the energy and data flows. We use a DC-DC boost converter to optimise harvested energy [16] and to decouple the harvesting signal from the energy storage and load behaviour [1]. The harvested energy is stored in an energy storage (a capacitor/battery) and is finally used to power the system. The information about the underlying activity is only encoded in the harvesting current, because the DCDC boost converter regulates the voltage of the solar cell to a constant, optimised value [1]. We use an Micro-controller Unit (MCU) to sample and process this current signal and to infer the underlying activity, as shown in Fig. 6.3. Firstly, various time and frequency domain features are extracted from the acquired SEH signal [1, 3]. Then, the extracted features are used as input to a classifier to detect the underlying activity. Finally, the result of the inferred activity is transmitted to a receiver (e.g., a smartphone) where it is exploited, e.g., by health or fitness applications. Note that, in contrast to [1] which samples the KEH signal locally and streams the raw data to a server, SolAR implements signal acquisition, feature extraction, classification and activity transmission on the wearable device, powered only by the harvested energy from the wearable solar cell. Implementing the complete HAR pipeline on the sensor node not only reduces the power consumption [17, 18], but also improves application latency and privacy [19, 20]. Omitting conventional activity sensors, rectification circuits (required for KEH) and external energy sources, SolAR minimises the cost, complexity, form factor, and environmental impact of the

6.3 SolAR: System Model and Implementation

99

Fig. 6.3 Proposed SolAR model using the solar cell as an activity sensor as well as an energy source, simultaneously

wearable IoT system. This finally realises the vision of energy-positive HAR in which end-to-end HAR is performed in real-time on wearable devices using only harvested energy.

6.3 SolAR: System Model and Implementation This section explains the measurement setup and implementation process of SolAR.

6.3.1 Measurement Setup We use the tool from [21] to sample the solar current from an off-the-shelf IXYS SLMD121H10L solar module during five human activities. The solar cell measures 4.2 cm × 3.5 cm and weighs 4.5 g, which is suitable for wearable devices and smart watches [22]. As a baseline, we simultaneously record the harvesting current from a 7.1 cm × 2.54 cm MIDÉ technology S230-J1FR-1808XB two-layered piezoelectric bending transducer. We use a tip mass of 24.62 g ± 0.5% to tune the resonance fre-

100

6 Solar Cell Based Activity Recognition

Fig. 6.4 Experimental setup for data collection using SEH and KEH transducers during various human activities

quency of the KEH transducer to the low frequency vibrations typically observed in human-centric applications [3], resulting in a total mass of 30.46 g. Both energy harvesting signals are sampled with an 18-bit ADC at 100 kHz. Finally, we also record acceleration data from an InvenSense MPU9250 3-axis accelerometer with a 12-bit ADC at 100 Hz. Before processing, the energy harvesting signals are down-sampled to the corresponding target frequency (see Sect. 6.4.4). The solar cell, the piezoelectric transducer and the accelerometer are mounted on the wrist of the participants during data collection, as shown in Fig. 6.4. The recording devices are placed on the waist of the participants, as depicted in Fig. 6.4. We collect SEH, KEH and accelerometer data from five common human1 activities, i.e., sitting (while putting hands on the table), standing, walking, running and going up/downstairs. In order to be able to explore the performance of SolAR under different light conditions, we conduct two separate sets of experiments: the first set of experiments is conducted indoors in a mostly carpeted room of size 9 m × 22.5 m with 13 healthy adults (age: 34 ± 9.2 years, mass: 78.4 ± 12.5 kg), and the second set of experiments is conducted outdoors with 8 healthy adults (age: 32.4 ± 4.4 years, mass: 79.7 ± 8.8 kg). In order to collect a representative SEH dataset which reflects diverse lighting intensity conditions that can be observed at different locations, we conduct the experiments on different days under different weather conditions (i.e., sunny, cloudy and partially cloudy) and at different times (i.e., morning, noon and evening). The participants are asked to perform each of the five activities for three minutes with a break of one minute after each activity. The participants are advised to perform the activities according to their daily routine and as naturally as possible. In total, we collect 390 min of data from five human activities and 21 participants.

1

Ethical approval has been granted from CSIRO [106/19] for carrying out this experiment.

6.3 SolAR: System Model and Implementation

101

6.3.2 Solar Cell as a Novel Human Activity Sensor We present the sample data traces collected from a wrist-wearable solar cell during various human activities in indoor as well outdoor environments in Fig. 6.5a and b respectively. The figure depicts that the harvested power from SEH during various human activities is significantly higher outdoors than indoors, thanks to the higher power density of the sunlight compared to artificial indoor light. Figure 6.5 also shows that the harvested power during sitting is higher than during standing due to the direct incidence of light on the solar cell, for example, when placing the hands on the table in a sitting position. In addition, dynamic activities such as walking, running and up/downstairs cause a dynamically changing orientation of the

Fig. 6.5 A wrist-wearable solar cell generates distinctive patterns of harvested power during various human activities in a indoor and b outdoor environments

102

6 Solar Cell Based Activity Recognition

wrist-wearable solar cell relative to the light source, which results in a distinct pattern of harvested power. During walking, for example, the human body produces a distinct shadowing effect on the moving wrist-worn solar cell, generating a unique pattern of harvested power. We also observe that an indoor environment has multiple sources of light which may complement one another whereas an outdoor environment contains only one source of light i.e., sun. In addition, outdoors there are more obstacles (such as trees and buildings) between the single light source and the solar cell, which results in more shadowing than in an indoor environment. Therefore, although the harvested energy outdoors is higher than for indoors, we expect higher activity detection accuracy indoors due to multiple light sources which complement each other and reduce shadowing effects.

6.3.3 Implementation of SolAR Below, we describe the implementation of SolAR in detail.

6.3.3.1

Pre-processing

The collected energy harvesting data from SEH contains stop periods (after each activity) which are removed from the data. Then, we segment the collected energy harvesting data into equal sized windows of 2 s [23] which is the typical time required to cover one stride length during walking [24, 25]. In order to retain the context information at the edges of windows, and to enhance the data points, we overlap [3] the consecutive windows before feature extraction. Analysing the effect of varying degree of window overlap on HAR accuracy, we observe that activity recognition accuracy increases increase of degree of window overlap. However, increasing the overlap also increases cost in terms of complexity and energy consumption, which is particularly relevant in the case of a limited energy budget. Therefore, in line with previous works [1, 3], we choose a window overlap of 50 % as a trade-off between HAR accuracy and energy consumption.

6.3.3.2

Feature Extraction

We extract various time and frequency domain features [1, 3, 5] from the energy harvesting data as presented in Table 6.1. In addition to time and frequency-domain features, we consider various peak-based features, such as peak-to-peak value, maximum distance between peaks, mean distance between peaks, maximum peak value, etc. These peak-based features have proven to be useful to improve human context detection from KEH signals [3]. In order to discover the minimum set of features that offers highest HAR accuracy, we employ various supervised and unsupervised feature selection algorithms such as mutual information [26], principal component

6.3 SolAR: System Model and Implementation

103

Table 6.1 Selected features from the SEH signal Features Signal SEH-indoor

SEH-outdoor

Peak-to-peak value, Coefficient-of-variation, Absolute area, Max. distance between peaks, 1st Quartile, 2nd Quartile, Frequency domain entropy, Median, Spectral peak, Min. value, Mean distance between peaks, Range, Max. value, Root-mean-square value, Absolute mean, Dominant frequency ratio, Kurtosis Peak-to-peak value, Coefficient-of-variation, Absolute area, Max. distance between peaks, 1st Quartile, 2nd Quartile, Frequency domain entropy, Median, Spectral peak, Min. value, Mean distance between peaks, Range, Max. value, Min. peaks, Standard deviation, Median absolute deviation, Frequency domain energy, Mean, 3rd Quartile, Max. peak, Autocorrelation

analysis [27], univariate [28], and correlation based feature selection [29]. After extensive analysis, we find that the mutual information based feature selection scheme achieves the highest HAR accuracy with the lowest number of features. The resulting, reduced feature set contains 17 and 21 features for SEH (as shown in Table 6.1), and 25 and 13 features for KEH, in indoor and outdoor environments respectively.

6.3.3.3

Activity Classification and Transmission

Prior to implementing classification algorithms, we employ Borderline-Synthetic Minority Oversampling Technique (SMOTE) [30] to handle imbalanced data from various human activities. Then, we apply seven well-known supervised machine learning classification algorithms including RF, KNN, SVM, DT, NB, Nearest Centroid (NC) and Gradient Boosting (GB) on the energy harvesting datasets. The classification algorithms are trained offline and the trained model is then imported to the embedded device for activity recognition from the real-time SEH signals. After implementing the classification algorithm, the inferred activity is transmitted using the Bluetooth Low Energy (BLE) wireless communication protocol. Thus, the proposed work not only acquires the activity signals [1], but also implements the classification algorithm on the node, using the harvested energy without the need of any external energy source. In order to enable comparability of our results to the state-of-the-art [13, 31] HAR mechanisms, all the results in this chapter are obtained using 10-fold CV (except Sect. 6.4.5 which presents results using leave-one-user-out CV to analyse the robustness of our system with user variance), and are presented with 95% confidence level. In order to ensure the robust performance of the classifier, the folds are selected randomly from the available data. Prior to invoking the classification algorithms, we augment the data from various human activities and normalise the features with zero mean and standard deviation of one. Unless stated otherwise, the results are obtained using activity signals sampled at 100 Hz.

104

6 Solar Cell Based Activity Recognition

6.4 Performance Evaluation SolAR relies on ambient light to generate energy and uses the harvested solar current signal for HAR. Since the ambient light differs significantly between indoor and outdoor environments, we initially evaluate the performance of SolAR using SEH data from indoor and outdoor experiments separately, with an extensive comparison made between accelerometer and KEH-based HAR. We present the classification results from well-known classifiers, analyse the variability in the chosen human activities as well as the effect of varying window sizes and sampling frequency on the accuracy of activity recognition. Then, we examine the robustness of SolAR against new/unseen users. Finally, we evaluate the performance of SolAR using the combined data from both indoor and outdoor experiments.

6.4.1 Classification Accuracy Figure 6.6 depicts the HAR accuracy offered by SEH, KEH and 3-axis accelerometer signals using various classification algorithms for indoor (Fig. 6.6a) and outdoor (Fig. 6.6b) environments. We find that, from among all the classification algorithms, the RF classifier offers the highest HAR accuracy of all the types of signals. Therefore, in the remainder of the chapter, all results are obtained using the RF classification algorithm. Furthermore, among the activity signals, the accelerometer signal offers highest HAR accuracy due to the rich context information embedded in its 3-axis signal. Figure 6.6 shows that of the two energy harvesting signals, SEH indoor and outdoor offers higher HAR accuracy than KEH thanks to the higher signal amplitude and unique harvested energy pattern during various human activities. In addition, SEH indoor offers higher HAR accuracy than SEH outdoor due to the uniform light availability, less shadowing and more light sources in the former which complement each other. In other words, the diversity of light sources indoors edges its performance closer to the 3-axis accelerometer, which captures three orthogonal signals for activity classification. Thus, SEH-based HAR outperforms conventional KEH-based HAR systems [3, 8] in terms of activity recognition accuracy.

6.4.2 Variability Analysis of Human Activities In order to analyse the classification results for individual human activities and to explore their variability, we show the confusion matrices using SEH and KEH signals (using the RF algorithm) in indoor as well as outdoor environments in Fig. 6.7. The figure shows that SEH offers higher recognition accuracy for static activities i.e., standing and sitting. The incident angle of the ambient light changes with the

6.4 Performance Evaluation

105

Fig. 6.6 HAR accuracy of 3-axis accelerometer (ACC), SEH and KEH signals in a indoor and b outdoor environments using various classification algorithms (window size = 2 s)

orientation of the solar cell, generating a significantly different amount and pattern of energy during static positions, which can be used to identify the underlying human activities. KEH, instead, generates a negligible amount of energy during static positions, which may not provide significantly different patterns and thus KEH offers lower HAR accuracy. Furthermore, SEH provides higher recognition accuracy during up/downstairs activity due to the different distribution of light compared to other activities. In contrast, using KEH, up/downstairs activity is confused with walking due to the similarity in arm movements between both, which generate identical signals from the KEH transducer. Finally, KEH provides higher recognition accuracy for dynamic activities including walking and running, due to the motion-specificbreak harvesting signal and significantly different type of mobility compared to other activities. On the other hand, due to the similar distribution of light sources, the harvesting

106

6 Solar Cell Based Activity Recognition

Fig. 6.7 Confusion matrices for HAR using a SEH and b KEH signals in indoors and outdoors (window size = 2 s)

6.4 Performance Evaluation

107

signal pattern from SEH may have a certain degree of similarity during dynamic activities and thus offers lower HAR accuracy compared to the KEH signal.

6.4.3 Varying Window Sizes Next, we explore the impact of larger window sizes on activity recognition accuracy. Figure 6.8 shows the activity recognition accuracy when using various window sizes from 2 s to 12 s for SEH and KEH signals (using the RF algorithm) in indoor as well as outdoor environments. We find that indoor SEH offers relatively stable HAR accuracy with a slight increase of about 2.8% when increasing the window size from 2 s to 12 s. The outdoor SEH signal instead is less sensitive to increase in window size and offers constant HAR accuracy from 2 s to 9 s and for 12 s. The HAR accuracy of the KEH signals (indoors and outdoors), on the other hand, increases by about 6% with an increase in window size from 2 s to 12 s. However, increasing the window size also results in increased latency, computational complexity and memory requirements. Therefore, the window size should be selected keeping in mind the required HAR accuracy, responsiveness of the system as well as the processing complexity due to the miniaturised and resource constrained target (wearable) device. Based on the previous discussion, we observe in Fig. 6.8 that 8 s is the minimum size of the window that offers best results in terms of HAR accuracy for all types of signals. Therefore, the remainder of the results in this chapter are presented using a window size of 8 s.

Fig. 6.8 HAR accuracy using SEH and KEH signals with increasing window sizes in indoor and outdoor environments

108

6 Solar Cell Based Activity Recognition

6.4.4 Varying Signal Sampling Frequency Figure 6.9 shows HAR performance over varying sampling frequencies ranging from 10 to 100 Hz (using the RF algorithm). The figure shows that the SEH signal indoors offers stable HAR accuracy and does not exhibit a perceptible change when sampling rate is increased from 10 to 100 Hz. In contrast, the SEH signal outdoors offers a small increase of about 3.7% in HAR accuracy with the increase in sampling rate from 10 to 100 Hz. The HAR accuracy from indoor KEH signal increases by about 3.8% at a sampling frequency of 100 Hz compared to 10 Hz. The KEH signal outdoors is not sensitive to the increase in sampling frequency and provides relatively stable HAR accuracy at all sampling frequencies. The improvement in HAR accuracy at higher sampling rates stems from the higher resolution of the signal and its ability to capture the more fine grained details of the activity pattern. However, as depicted in Fig. 6.9, the energy consumption increases with the increase in sampling frequency due to the acquisition of more samples in a fixed time interval. Therefore, depending on the type of application and the amount of harvested energy, the SEH sampling rate can be chosen to be as low as 10 Hz to minimise energy consumption, while still offering activity recognition accuracy of above 93% and 83% in indoor and outdoor environments, respectively.

Fig. 6.9 Average HAR accuracy and required power with varying sampling frequencies of SEH and KEH signals in indoor and outdoor environments (window size = 8 s)

6.4 Performance Evaluation

109

6.4.5 Robustness to User Variance In this subsection, we analyse the robustness of the SolAR system against new/unseen users. To this end, we perform leave-one-user-out CV on the collected data (using the RF algorithm) and present the averaged results in Table 6.2. The table shows that SEH indoor, accelerometer and KEH indoor signals are least sensitive to variation in the user and offer 5–9% decreased HAR accuracy for new and unseen users. On the other hand, SEH and KEH signals outdoors are significantly affected by user variance and offer decreased HAR accuracy by 19–25% for new users due to the significant variation in the activity pattern. However, the SEH transducer still offers more than 84% HAR accuracy in indoor environments for new and unseen users, which shows its applicability in practical and real-world scenarios. We expect that a larger training sample will further reduce this sensitivity.

6.4.6 Environment-Agnostic Analysis Instead of training the classification model separately in indoor and outdoor environments, we combine the energy harvesting data from both environments in an environment-agnostic scenario. We train the classification algorithm using all features listed in Table 6.1 and plot the classification results (using the RF algorithm) from individual as well as combined datasets in Fig. 6.10. The figure shows that SolAR can recognise human activities in an environment-agnostic scenario with a 7% higher accuracy than conventional KEH-based HAR. Furthermore, the performance of SolAR in the environment-agnostic scenario is higher than that for an outdoor environment, and it is close to the performance in the indoor environment. This demonstrates the general applicability of our proposed approach in different environments.

Table 6.2 Average HAR accuracy from the robustness experiment Type of signal 10-fold CV Leave-one-user-out CV SEH-indoor SEH-outdoor KEH-indoor KEH-outdoor Accelerometer

93.47 86.87 88.89 86.14 99.3

84.62 61.69 83.25 66.43 92.27

110

6 Solar Cell Based Activity Recognition

Fig. 6.10 HAR accuracy using separate as well as combined data from indoor and outdoor environments (window size = 8 s)

6.5 Energy-Positive HAR In this section, we analyse the harvested power from SEH and KEH transducers during various human activities as well as the power required for running SolAR on a wearable device.

6.5.1 SolAR Harvested Power We calculate the average harvested power from the collected SEH and KEH data during various human activities and present the results in Table 6.3. The last row of Table 6.3 describes the power density, i.e., the harvested power per area. We find that, on average, SEH generates more than one order of magnitude higher power indoors, and more than two orders of magnitude higher power outdoors compared to KEH

Table 6.3 Average harvested power during various activities Human activity Harvested power [µW] Outdoor Indoor Kinetic Solar Kinetic Running Walking Using stairs Standing Sitting Average power Power density [µW/cm2 ]

11.7 3.16 3.57 0.49 0.47 3.88 0.218

3800 2800 2100 840 1900 2288 163.429

6.5 2.4 2.8 0.45 0.21 2.47 0.139

Solar 29.7 29.6 13 26 54.7 30 2.143

6.5 Energy-Positive HAR

111

due to the high power density of visible light as well as high conversion efficiency of solar cells [32]. The results also show that harvested power from SEH is less dependant on physical human movements compared to KEH; there is a minimum power level that can be harvested from SEH even during static activities such as standing and sitting. KEH, on the other hand, harvests only a small amount of power (i.e., 0.21–0.49 µW) during these static activities due to negligible movement of the human body. The harvested power outdoors is higher than indoors for both SEH and KEH. For SEH, this is because natural sunlight has a higher power density compared to the artificial indoor lights. For KEH, we suspect a combination of two effects: (1) walking on a paved surface generates a higher degree of vibrations than walking on a carpeted floor indoors [33] and, (2) people tend to move faster outdoors, which results in higher vibrations and, as a result, higher KEH power.

6.5.2 SolAR Power Consumption We implement the SEH- and KEH-based HAR models on an ultra low power Nordic Semiconductor nRF52840 wireless MCU (shown in Fig. 6.11) to measure and compare the power consumption of the individual tasks, i.e., sampling, feature extraction, classification and transmission. Based on the results from Sect. 6.4, we choose a window size of 8 s, a sampling frequency of 10 Hz and the RF classification algorithm. The result of the classification is transmitted as a BLE packet (after every 8 s), consisting of 6 bytes header, 3 bytes checksum and 1 byte payload, encoding the result

Fig. 6.11 Experimental setup for measuring the power consumption in implementing the HAR model

112

6 Solar Cell Based Activity Recognition

of the classification. To measure the current, we place a 10 Ω shunt resistor in series with the 2 V supply voltage and measure the voltage drop with an Agilent Technologies MSO4104B oscilloscope, as depicted in Fig. 6.11. The firmware is configured to set a dedicated General Purpose Input/Output (GPIO) pin high while executing a task. By simultaneously recording the current and GPIO pins, we can precisely measure the execution time and power consumption of each task. Table 6.4 shows the average, per-task power requirements of running SolAR with the indoor feature set. Sampling takes 1.594 µW, including 0.294 µW for the ADC and 1.3 µW for converting the ADC data into floating point values. Feature extraction and classification take 1.637 µW and 0.061 µW, respectively. Transmitting the result over the wireless channel takes 0.125 µW, including 0.067 µW to power up the high frequency clock and 0.058 µW for transmitting the packet. For 99.93 % of the time, the MCU remains in deep sleep mode, consuming only 4.5 µW. Thus, the total average power consumption of our implementation is 7.92 µW. Table 6.5 compares the power requirements of SEH- and KEH-based HAR with the indoor and outdoor feature sets, respectively. We find that, because SEH- and KEH-based HAR have different features, they have a different power consumption. Interestingly, the average power for the classifier (0.061–0.078 µW) is more than one order of magnitude lower than the power required in feature extraction across the board.

Table 6.4 Average required power to implement the end-to-end HAR algorithm on the sensor node using the SEH signal in an indoor environment (window size = 8 s) Task Power [mW] Time [µs] Avg. power [µW] Sampling (@10 Hz) Feature extraction Classification Data transmission Sleep mode Total

8.6 6.6 6.6 11.2 0.0045 33.004

2752 1985 74 491 7.995 × 106 8 × 106

1.59 1.64 0.06 0.12 4.5 7.92

Table 6.5 Required power to implement the embedded machine learning HAR (window size = 8 s) No. of features Required power [µW] Signal Feature ext. Classification Total SEH-indoor SEH-outdoor KEH-indoor KEH-outdoor

17 21 25 13

1.637 2.236 2.557 1.567

0.061 0.073 0.078 0.072

1.698 2.309 2.635 1.639

6.5 Energy-Positive HAR

113

6.5.3 Energy-Positive HAR In this subsection, we examine whether SolAR achieves energy-positive HAR by comparing the harvested power to the power required to run SolAR on the wearable r ), similar to the signal device. To this end, we define the HAR power ratio (Phar acquisition power ratio in [1]: r = Phar

H ar vested_ power H A R_ power

(6.1)

When the harvested power from a wearable-sized transducer is less than the power r < 1), the system is energy negative. On required for running the HAR model (Phar the other hand, if the harvested power is greater than the power required for running r > 1), the system is energy-positive. Figure 6.12 compares the HAR model (Phar the harvested power from SEH and KEH with the corresponding power to run the HAR model, averaged over all activities. The average SEH power is higher than the power required to run the HAR model on the sensor node both indoors and outdoors. Thus, SolAR is energy-positive and enables autonomous and perpetual operation of the sensor node without the need of any external energy source. Although the KEH transducer that we used in our experiments is larger and heavier than the solar cell (18.03 cm2 , 30.46 g vs. 14.7 cm2 , 4.5 g), the average KEH power is not sufficient to run the HAR model on the node and thus provides energy negative HAR. Figure 6.13 plots the HAR accuracy and HAR power ratio of individual activities for SEH- and KEH-based HAR indoors and outdoors. We find that SolAR offers higher HAR accuracy and is energy-positive across all activities indoors and outdoors. KEH-based HAR instead is mostly energy negative due to significantly lower harvested power (see Table 6.3). The only exception, where KEH also delivers sufficient power for end-to-end HAR is the running activity outdoors. Note that a custom-designed energy harvesting circuit with a perfectly tuned KEH transducer may deliver higher power [34] than observed in this study. While this could

Fig. 6.12 Average harvested and consumed power in implementing the end-to-end HAR model using SEH and KEH signals indoors and outdoors

114

6 Solar Cell Based Activity Recognition

Fig. 6.13 HAR accuracy versus HAR power ratio for various human activities using a small-sized and lightweight SEH (14.7 cm2 , 4.5 g) compared to the KEH transducer (18.03 cm2 , 30.46 g)

potentially result in energy-positive HAR for some dynamic activities, it is impractical to tune the KEH transducer to specific scenarios. Furthermore, KEH transducers are fundamentally unable to deliver energy during mostly static activities. Because humans generally spend a great proportion of their time performing such activities, the average harvested energy from KEH transducers may not be sufficient to ensure the perpetual operation of the wearable devices. Thus, SEH offers clear advantages over KEH in terms of higher harvested energy, higher HAR accuracy, as well as the fact that it does not require hardware customization for different application scenarios. In contrast to the previous work [1] that achieves energy-positive sensing only for signal acquisition using KEH, SolAR ensures end-to-end energy-positive HAR. We find that SEH harvests 22.08 µW and 2.28 mW higher power than required for running the end-to-end HAR model indoors and outdoors respectively, as shown in Tables 6.3 and 6.4. This means that the size of the solar cell can be reduced significantly or additional harvested power can be used to run other body sensors which ensures real-time, continuous and perpetual monitoring of human health, fitness and activity without the need for any external depletable energy source, leading towards truly pervasive IoT. Moreover, in order to achieve a better trade-off between energy and HAR accuracy, a scheduling technique [35, 36] can be devised which makes the best use of the accelerometer, SEH and KEH signals for activity recognition, depending on the energy budget.

References

115

6.6 Discussion In order to run wearable IoT devices perpetually, recently, KEH transducers have been used as activity sensors as well as sources of energy simultaneously. However, the harvested energy from human movements using miniaturised KEH transducers is not enough to run the wearable devices perpetually. In this chapter, we propose SolAR, a novel HAR mechanism which employs solar cells for recognizing human activities as well as for powering the wearable device. As human activities interact and interfere with the ambient light differently, the harvesting signal from the wearable solar cell contains information about the underlying activities. In order to explore the sensing potential of solar cells, we collect SEH data from 21 participants performing five common human activities in indoor as well as outdoor environments. After rigorous analysis, we find that SolAR offers up to 8.3% higher HAR accuracy compared to the previous KEH-based HAR mechanisms. In addition, the harvested power from wearable-sized solar cells is higher than required for running the end-to-end HAR model on the sensor node and thus ensures energy-positive HAR. This chapter has demonstrated how SEH can achieve self-powered HAR in IoT. In the next chapter, we will explore the benefits of fusing KEH and SEH signals for enhancing the accuracy of self-powered HAR in IoT.

References 1. Sandhu MM, Geissdoerfer K, Khalifa S, Jurdak R, Portmann M, Kusy B (2020) Towards energy positive sensing using kinetic energy harvesters. In: 2020 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 2. Sandhu MM, Geissdoerfer K, Khalifa S, Jurdak R, Portmann M, Kusy B (2020) Towards optimal kinetic energy harvesting for the batteryless iot. In: 2020 IEEE international conference on pervasive computing and communications workshops (PerCom Workshops). IEEE, pp 1–6 3. Khalifa S, Lan G, Hassan M, Seneviratne A, Das SK (2017) Harke: human activity recognition from kinetic energy harvesting data in wearable devices. IEEE Trans Mob Comput 17(6):1353– 1368 4. Lan G, Ma D, Xu W, Hassan M, Hu W (2020) Capacitor-based activity sensing for kineticpowered wearable iots. ACM Trans Internet of Things 1(1):1–26 5. Hemminki S, Nurmi P, Tarkoma S (2013) Accelerometer-based transportation mode detection on smartphones. In: Proceedings of the 11th ACM conference on embedded networked sensor systems. ACM, p 13 6. Khalifa S, Hassan M, Seneviratne A, Das SK (2015) Energy-harvesting wearables for activityaware services. IEEE Internet Comput 19(5):8–16 7. Kalantarian H, Alshurafa N, Le T, Sarrafzadeh M (2015) Monitoring eating habits using a piezoelectric sensor-based necklace. Comput Biol Med 58:46–55 8. Lan G, Ma D, Xu W, Hassan M, Hu W (2017) Capsense: capacitor-based activity sensing for kinetic energy harvesting powered wearable devices. In: Proceedings of the 14th EAI international conference on mobile and ubiquitous systems: computing, networking and services. ACM, pp 106–115 9. Ma D, Lan G, Xu W, Hassan M, Hu W (2018) SEHS: simultaneous energy harvesting and sensing using piezoelectric energy harvester. In: 2018 IEEE/ACM third international conference on internet-of-things design and implementation (IoTDI). IEEE, pp 201–212

116

6 Solar Cell Based Activity Recognition

10. Zhang S, Zhao Y, Nguyen DT, Xu R, Sen S, Hester J, Alshurafa N (2020) Necksense: A multisensor necklace for detecting eating activities in free-living conditions. In: Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, vol 4, no 2, pp 1–26 11. Venkatnarayan RH, Shahzad M (2018) Gesture recognition using ambient light. In: Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, vol 2, no 1, pp 1–28 12. Zhang D, Park JW, Zhang Y, Zhao Y, Wang Y, Li Y, Bhagwat T, Chou W-F, Jia X, Kippelen B et al (2020) Optosense: Towards ubiquitous self-powered ambient light sensing surfaces. In: Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, vol 4, no 3, pp 1–27 13. Ma D, Lan G, Hassan M, Hu W, Upama MB, Uddin A, Youssef M (2019) Solargest: ubiquitous and battery-free gesture recognition using solar cells. In: The 25th annual international conference on mobile computing and networking. ACM, pp 1–15 14. Li Y, Li T, Patel RA, Yang X-D, Zhou X (2018) Self-powered gesture recognition with ambient light. In: Proceedings of the 31st annual ACM symposium on user interface software and technology, 2018, pp 595–608 15. Umetsu Y, Nakamura Y, Arakawa Y, Fujimoto M, Suwa H (2019) Ehaas: energy harvesters as a sensor for place recognition on wearables. In: Proceedings of the 2019 IEEE international conference on pervasive computing communications (PerCom). IEEE, pp 1–10 16. Sharma H, Haque A, Jaffery ZA (2018) Modeling and optimisation of a solar energy harvesting system for wireless sensor network nodes. J Sens Actuator Netw 7(3):40 17. Rezaie H, Ghassemian M (2017) An adaptive algorithm to improve energy efficiency in wearable activity recognition systems. IEEE Sens J 17(16):5315–5323 18. Fafoutis X, Marchegiani L, Elsts A, Pope J, Piechocki R, Craddock I (2018) Extending the battery lifetime of wearable sensors with embedded machine learning. In: IEEE 4th world forum on internet of things (WF-IoT). IEEE, pp 269–274 19. Wang A, Chen L, Xu W (2017) Xpro: a cross-end processing architecture for data analytics in wearables. ACM SIGARCH Comput Archit News 45(2):69–80 20. Verhelst M, Moons B (2017) Embedded deep neural network processing: algorithmic and processor techniques bring deep learning to iot and edge devices. IEEE Solid-State Circuits Mag 9(4):55–65 21. Geissdoerfer K, Chwalisz M, Zimmerling M (2019) Shepherd: a portable testbed for the batteryless iot. In Proceedings of the 17th ACM conference on embedded networked sensor systems (SenSys), 2019, pp 83–95 22. Seneviratne S, Hu Y, Nguyen T, Lan G, Khalifa S, Thilakarathna K, Hassan M, Seneviratne A (2017) A survey of wearable devices and challenges. IEEE Commun Surv Tutor 19(4):2573– 2620 23. Stisen A, Blunck H, Bhattacharya S, Prentow TS, Kjærgaard MB, Dey A, Sonne T, Jensen MM (2015) Smart devices are different: assessing and mitigating mobile sensing heterogeneities for activity recognition. In: Proceedings of the 13th ACM conference on embedded networked sensor systems, 2015, pp 127–140 24. Lim YP, Lin Y-C, Pandy MG (2017) Effects of step length and step frequency on lower-limb muscle function in human gait. J Biomech 57:1–7 25. Russell DM, Apatoczky DT (2016) Walking at the preferred stride frequency minimizes muscle activity. Gait & Posture 45:181–186 26. Ross BC (2014) Mutual information between discrete and continuous data sets. PloS one 9(2):e87357 27. Minka TP (2001) Automatic choice of dimensionality for pca. In: Advances in neural information processing systems, 2001, pp 598–604 28. Emura T, Matsui S, Chen H-Y (2019) Compound. cox: univariate feature selection and compound covariate for predicting survival. Comput Methods Programs Biomed 168:21–37 29. Mursalin M, Zhang Y, Chen Y, Chawla NV (2017) Automated epileptic seizure detection using improved correlation-based feature selection with random forest classifier. Neurocomputing 241:204–214

References

117

30. Han H, Wang W-Y, Mao B-H (2005) Borderline-smote: a new over-sampling method in imbalanced data sets learning. In: International conference on intelligent computing. Springer, pp 878–887 31. Lan G, Xu W, Ma D, Khalifa S, Hassan M, Hu W (2019) Entrans: leveraging kinetic energy harvesting signal for transportation mode detection. IEEE Trans Intell Transp Syst 32. Bhatti NA, Alizai MH, Syed AA, Mottola L (2016) Energy harvesting and wireless transfer in sensor network applications: concepts and experiences. ACM Trans Sens Netw (TOSN) 12(3):1–40 33. Menz HB, Lord SR, Fitzpatrick RC (2003) Acceleration patterns of the head and pelvis when walking on level and irregular surfaces. Gait & Posture 18(1):35–46 34. Kuang Y, Ruan T, Chew ZJ, Zhu M (2017) Energy harvesting during human walking to power a wireless sensor node. Sens Actuators A Phys 254:69–77 35. Sandhu MM (2020) Phd forum abstract: energy harvesting based sensing for the batteryless iot. In: 2020 19th ACM/IEEE international conference on information processing in sensor networks (IPSN). IEEE, pp 373–374 36. Sommer P, Geissdoerfer K, Jurdak R, Kusy B, Liu J, Zhao K, McKeown A, Westcott D (2019) Energy-and mobility-aware scheduling for perpetual trajectory tracking. IEEE Trans Mob Comput 19(3):566–580

Chapter 7

Fusion-Based Activity Recognition

In the previous chapter, we have discussed that SEH can be employed as source of context information and energy simultaneously. However, it may face problems during low light conditions such as at night to harvest sufficient energy to power a sensor node. Therefore, in order to enhance the harvested energy and context recognition performance, a fused signal which employs both solar and kinetic energy harvesting signals can be explored. In this chapter, we explore Fused signal-based human Activity Recognition (FusedAR) which employs both solar and kinetic energy harvesters simultaneously as sensors for activity recognition as well as sources of energy as depicted in Fig. 7.1. We have presented the initial evaluation of Solar-based human Activity Recognition (SolAR) in [1], whereas in this chapter, we rigorously evaluate SolAR in diverse lighting conditions and discover that it can recognise activities not only during day times but also during night times when the ambient light is significantly limited. After implementing the classification algorithm on an ultra low-power Micro-controller Unit (MCU), we find that FusedAR harvests sufficient power to run the end-to-end HAR algorithm (including feature extraction, classification and activity transmission) and thus ensures energy-positive HAR. Finally, in order to evaluate our proposed mechanism, we define a metric of HAR power ratio which is the ratio of harvested power and required power for implementing the HAR algorithm (see Sect. 7.4.3).

7.1 Background This section describes previous wearable-based HAR mechanisms in detail.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9_7

119

120

7 Fusion-Based Activity Recognition

Fig. 7.1 Employing solar and kinetic energy harvesting transducers for activity recognition as well as to power the wearable sensor node simultaneously, leading towards end-to-end energy-positive HAR

7.1.1 Accelerometer-Based HAR Previous research studies have shown that wearable IoT devices, which contain inertial sensors such as accelerometers and magnetometers [4], can be attached to different parts of the human body to achieve reliable HAR [5]. These wearable devices have numerous applications including health/fitness monitoring, activity detection, tracking and localization [6]. The authors in [7, 8] propose that wearable devices, which monitor human physiological parameters, can be employed for the early recognition of asymptomatic and pre-symptomatic cases of COVID-19. In another work [9], a wrist wearable device is designed to detect the obstacles for providing independent mobility to visually impaired people. Sztyler et al. [10] employ a wearable device for HAR and suggest that it offers best results when placed on waist as it is centre of the body. The major bottleneck of these wearable devices is their limited lifetime which hinders their widespread adaptability [11] and pervasive deployment. Although battery technology is evolving over time, their limited lifetime is still one of the biggest impediments to advancing wearable technology [4]. This opens the door to exploring energy harvesting sources as viable alternatives for batteries which can ultimately result in the uninterrupted and autonomous operation of wearable IoT sensing devices.

7.1.2 KEH-Based HAR Recently, KEH transducers have been used as an energy-efficient activity sensor. Khalifa et al. [11] present that the harvesting signal from KEH can be used for HAR which minimises the energy consumption and allows perpetual activity recognition compared to conventional systems. Lan et al. [12] store the kinetic harvested energy

7.1 Background

121

Fig. 7.2 Difference of light sources, shadowing and variation in light intensity in (a) indoors and (b) outdoors

in a capacitor and use the capacitor voltage signal for activity recognition. KEH transducers have been also used for monitoring food intake [13], recognising transport modes [14], and generating security key for wearables [15]. In order to increase the energy generated from KEH, Ma et al. [16] employ two separate transducers in a shoe and use the harvesting signal for gait recognition. On the other hand, Sandhu et al. [17] employ a single KEH transducer simultaneously to harvest energy and to recognise the daily activities, and coin the term of energy-positive signal acquisition. However, the harvested energy from human movements/vibrations is not enough to ensure the autonomous operation of wearable devices [17] which opens door to explore alternate HAR mechanisms which can harvested sufficient energy to ensure the perpetual and autonomous operation of wearable IoT devices (Fig. 7.2).

7.1.3 SEH-Based HAR SEH offers significant benefits in terms of power density, energy conversion efficiency, and robustness compared to KEH, as shown in Table 7.1. Therefore, SEH can be considered as an attractive source to power IoT sensor nodes [18]. Solar cells have been also used as a proxy for recognising hand gestures and room-level localisation. Ma et al. [19] use a solar cell to identify and recognise various hand gestures under a lamp light in an indoor environment. As the movement of the hand obstructs the light falling on the solar surface, the resulting harvesting signal contains information about the gestures performed. On the other hand, Umetsu et al. [20] employ multi-source energy harvesters including solar and kinetic, for place recognition in an indoor environment. However, these works [19, 20] employ SEH merely as a context sensor without extracting energy simultaneously and thus do not fully reap the benefits of the energy harvesters. Furthermore, to the best of our knowledge, the use of solar cells to recognise human physical activities in HAR applications remains largely uninvestigated. In an initial study called SolAR [1], we show the potential use of solar cell as a simul-

122

7 Fusion-Based Activity Recognition

Table 7.1 SEH and KEH properties [2, 3] Photovoltaic Property [µW/cm2 ]

Power density Energy conversion efficiency Robustness

10 µW to 15 mW Up to 40% High

Piezoelectric Upto 330 µW Up to 30% Low

taneous source of energy and context information for recognising human physical activities. In the present chapter (Sect. 7.3), we provide an extensive evaluation of SolAR in diverse lighting conditions and using a large cohort of participants.

7.1.4 Limitations and Challenges Previous studies [11, 17] have shown that KEH can be employed as a simultaneous source of energy and context information. However, the limited harvested energy from a miniaturised KEH transducer is not sufficient to perpetually run the wearable IoT device [17]. Table 7.1 shows that SEH offers higher power density and thus can offer higher harvested energy compared to a KEH transducer. Sandhu et al. [1] proposed the use of SEH as a simultaneous source of energy and context information and showed that SEH offers better results in terms of harvested energy and activity recognition compared to KEH-based sensing. However, SEHs are unable to harvested sufficient power during night (and low light conditions) [1] and thus can not capture activities performed in low-light conditions. This chapter is motivated by the fact that both kinetic and solar energy harvesters can be employed simultaneously for activity recognition and energy harvesting. By fusing the context information and combining the harvested energy from both KEH and SEH transducers, we can get higher harvested energy and enhanced sensing performance. Note that both KEH and SEH transducers embed the context information differently i.e., from the changing vibration signals and the varying light intensity respectively. As activity signals from both transducers complement each other, the resultant high-fidelity fused signal can offer fine-grained information about the underlying activity. In addition, we can obtain energy even during static activities (from SEH) and low light conditions (from KEH), resulting in the near-perpetual operation of the wearable IoT devices.

7.2 Fusing Solar and Kinetic Energy Signals We describe the architecture of Fusion-based Activity Recognition (FusedAR) system as well as the implementation specific details in the following subsections.

7.2 Fusing Solar and Kinetic Energy Signals

123

7.2.1 Architecture We employ a wearable kinetic energy harvester and a solar cell simultaneously as activity sensors for activity recognition and energy sources to run the system load for the self-powered operation of wearable IoT sensing devices. Figure 7.3 shows the architecture of our proposed FusedAR model. We employ an ultra low power DC-DC boost converter with maximum power point tracking, with each transducer, to optimise the harvested power [21] as well as to decouple the harvesting signal from the energy storage and load behaviour [17]. The converters are configured to provide a fixed output voltage when the input (harvested) voltage exceeds a certain threshold. The combined stored harvested energy (in a capacitor/battery) is used to power the system load using a converter to match the capacitor voltage with the system specifications. Using the proposed architecture, the context information is only embedded in the harvesting current signal, because the converter fixes the transducer voltage to a constant, optimised value [17]. We employ an MCU to acquire and collect the harvesting current signals from both transducers and to recognise the underlying activity using a machine learning model, as portrayed in Fig. 7.3.

Fig. 7.3 Proposed FusedAR architecture using the kinetic and solar energy harvesters as activity sensors as well as energy sources simultaneously

124

7 Fusion-Based Activity Recognition

We extract various time and frequency-domain features from the collected KEH and SEH signals [11, 17]. Later, the most informative features from both energy harvesting signals are selected using well-known feature selection techniques. These selected features are then fused together to combine the context information from both distinct sources to obtain enhanced activity recognition performance. Then, the resultant most dominant features from the fused signal, are employed to train the classifier. Finally, the inferred activity is transmitted to an accessible receiver where it is used by health, activity or fitness monitoring applications. In the proposed work, the complete HAR pipeline is implemented on the sensor node, which minimises power consumption [22, 23] while simultaneously improving application latency and privacy [24, 25].

7.2.2 Measurement Setup We use Shepherd [26], a portable testbed for the batteryless IoT that allows to record harvested signal traces with high resolution, to sample the SEH current signals from a tiny solar cell. In addition to solar cell, we simultaneously collect the harvesting current signal from a KEH transducer with a tip mass of 24.62 g±0.5% as described in [1] in detail. In order to compare our results with the state-of-the-art, we also collect data from a 3-axis acclereometer. As depicted in Fig. 7.4, we place the the solar cell, the KEH transducer and the accelerometer modules on the wrist and recording devices on the waist of the participants. We record SEH, KEH and 3-axis accelerometer data1 from five common daily human activities including sitting, standing, walking, running and going up/downstairs. Table 7.2 elaborates that data is collected in two different environments i.e., indoors and outdoors. The first set of experiments is conducted indoors in various lighting conditions with 18 healthy adults (age: 35±10.3 years, mass: 79.7±13.6 kg), and the second set of experiments is performed outdoors with 22 healthy adults (age: 34.2±6.8 years, mass: 78.3±9.6 kg). In order to collect a representative energy harvesting dataset which reflects diverse light intensity conditions, we conduct a series of experiments at different locations on different days with different weather conditions. Overall, 600 minutes of data is recorded from five human activities and 40 volunteers. The sample data traces collected from a wrist-wearable solar cell, a KEH transducer and an accelerometer during various human activities are presented in Fig. 7.5. It shows distinct SEH patterns with unique variations during different human activities in indoor and outdoor environments. It is interesting to notice that the SEH power does not show significant variation during sitting and standing activities due to the lower mobility compared to other activities. Furthermore, the amount of harvested power is significantly different during sitting and standing activities indoors. It is due to the different orientation of the wearable solar cell relative to the indoor light 1

Ethical approval has been granted from CSIRO [106/19] for carrying out this experiment.

7.2 Fusing Solar and Kinetic Energy Signals

125

Fig. 7.4 Data collection setup using SEH and KEH transducers during different daily human activities Table 7.2 Detail of data collection experiments in indoors and outdoors Environment No. of users Light source Surface type Indoors

Outdoors

9 4 5 8 9 5

LED lights LED lights Fluorescent lights Sunlight Sunlight Street lights

Carpeted room Carpeted room Tiled hallway Public street Private street Public street

Day/Night Day Night Day Day Day Night

Fig. 7.5 A wrist-wearable energy harvester/sensor generates distinct pattern of signals during various human activities using a 3-axis accelerometer b KEH indoors c KEH outdoors d SEH indoors and e SEH outdoors

126

7 Fusion-Based Activity Recognition

source (resulting in different light intensity) compared to outdoors where light intensity is mostly uniform in daylight conditions. In addition to SEH, we plot the KEH signals from various human activities in Fig. 7.5. It depicts that the KEH patterns are distinct not only for each activity but also for different environments (i.e., indoors and outdoors). As the result, the output KEH signals can be employed to distinguish the activities and environments in which activities are performed. We also present 3-axis accelerometer traces in Fig. 7.5 for comparison purposes which clearly show unique patterns during various activities.

7.2.3 Human Activity Recognition 7.2.3.1

Pre-processing

After removing the stop periods, the acquired energy harvesting data is segmented into equal sized 2 s [27] segments which is a typical time duration required to cover one stride length during walking [28, 29]. We overlap [11] the consecutive windows before feature extraction to retain the context information at both edges of windows and to increase the number of data points. When we examine the influence of different degrees of window overlap on HAR accuracy, we find that activity recognition accuracy improves as the degree of window overlap increases. However, increasing the window overlap raises the cost of complexity and energy consumption, which is especially important when working with a limited energy budget. Consequently, in accordance with previous works [11, 17], and as a trade-off between energy usage and HAR accuracy, we use a window overlap of 50 %.

7.2.3.2

Feature Extraction

Initially, various time-domain, frequency-domain and peak-based features [11, 17, 30] are extracted from the energy harvesting data (both SEH and KEH signals) as shown in Table 7.3. We employ various feature selection algorithms such as mutual information [31], principal component analysis [32], univariate [33], and correlation based feature selection [34], to get the smallest set of features that provides the best HAR accuracy. After detailed analysis, we discover that the mutual information based feature selection scheme offers best performance with the smallest number of features. As a result, as presented in Table 7.3, SEH offers 17 and 21 features, and KEH offers 25 and 13 features, in indoors and outdoors respectively. On the other hand, as shown in Table 7.4, FusedAR provides activity information using a significantly reduced feature set which contains only 8–12 features depending on the environment.

7.2 Fusing Solar and Kinetic Energy Signals

127

Table 7.3 SEH signal features in indoors and outdoors Signal Features SEH-indoor

SEH-outdoor

Root-mean-square value, Absolute area, Peak-to-peak value, Coefficient-of-variation, Frequency domain entropy, Max. distance between peaks, Median, Range, 1st Quartile, 2nd Quartile, Spectral peak, Min. value, Max. value, Absolute mean, Mean distance between peaks, Dominant frequency ratio, Kurtosis Mean, Absolute area, Peak-to-peak value, Median, Range, Coefficient-of-variation, Frequency domain entropy, Max. distance between peaks, 1st Quartile, 2nd Quartile, 3rd Quartile, Standard deviation, Spectral peak, Min. value, Max. value, Min. peaks, Max. peak, Mean distance between peaks, Median absolute deviation, Frequency domain energy, Autocorrelation

Table 7.4 Selected features from the fused signal in indoors and outdoors Signal Features Fused-indoors

Fused-outdoors

7.2.3.3

SEH: Max. distance between peaks, Number of peaks, Coefficient-of-variation, Range, Absolute area KEH: Peak-to-peak value, Number of peaks, Absolute area SEH: Minimum peak value, Max. distance between peaks, Coefficient-of-variation, Minimum value KEH: Absolute area, peak-to-peak value, Minimum peak value, 3rd quartile, RMS value, Peak value, Number of peaks, Coefficient-of-variation

Signal Fusion

In FusedAR, we explore the potential of integrating the high fidelity context information content present in both KEH and SEH signals to enhance the activity recognition performance. We perform a feature-level fusion where we identify the most informative features from the fused signals using mutual information [31], and present the selected features in Table 7.4. It shows that the fused-indoor signal offers more features from the SEH signal compared to KEH signal due to the increased capability of SEH signal to correctly identify the activities as depicted in Fig. 7.6a. It is also worth noting that inferring the activity from the fused-outdoor signal requires more features from the KEH signal (compared to SEH) to accommodate for the shadowing and interference from the surroundings which may affect the SEH signal in outdoor environments. Comparing Table 7.4 and Table 7.3, we observe that the FusedAR provides detailed activity information using a smaller feature set (i.e., 8–12) compared to using only SolAR (i.e., 17–21 features) and thus requires reduced energy in executing the task of feature extraction on the node, as elaborated in Sect. 7.4.2.

128

7.2.3.4

7 Fusion-Based Activity Recognition

Activity Classification and Transmission

Before training a classification algorithm using the collected data, BorderlineSynthetic Minority Oversampling Technique (SMOTE) [35] is used to augment the data and handle imbalanced data from various human activities. Later, we implement seven widely used supervised machine learning classifiers including RF, DT, KNN, SVM, NC, NB and GB on the energy harvesting datasets. Firstly, these classification algorithms are trained offline and then imported to the embedded IoT device for real-time activity recognition. The result of the output activity is then transmitted using BLE protocol. Thus, our work not only captures the context/activity signals [17] but also implements the classification algorithm and end-to-end HAR pipeline on the node using the acquired energy signals, eliminating the requirement for an external energy source. We perform extensive evaluation of FusedAR and SolAR with extended datasets, and present the detailed results and insights in the following section.

7.3 Performance Evaluation FusedAR relies on the ambient light and motion/vibration to generate energy and uses the fused energy signal for activity recognition. Because the ambient light in indoor and outdoor locations changes greatly, we initially evaluate the performance of FusedAR and SolAR using the data separately from indoor and outdoor experiments, with an extensive comparison to conventional Accelerometer-based Activity Recognition (AccAR) and KEH-based Activity Recognition (KEHAR) mechanisms. We next accumulate the data and examine the performance of FusedAR and SolAR in environment-agnostic and environment-preserving scenarios in Sect. 7.3.6. All of the results in this research were achieved offline using 10-fold CV to allow for comparison to the state-of-the-art (except Sect. 7.3.4 which describes the results using leave-one-user-out CV), and are reported with 95% confidence level. The folds are chosen at random from the supplied data to ensure the classifier’s robust performance. Prior to implementing the classification algorithms, we augment the collected data from diverse human activities and normalise the features. If not mentioned otherwise, all of the results in this chapter are acquired using 100 Hz activity signals.

7.3.1 Classification Accuracy Figure 7.6 contains the results in terms of HAR accuracy offered by FusedAR and SolAR compared to AccAR and KEHAR using different classification algorithms in indoor (Fig. 7.6a) and outdoor (Fig. 7.6b) environments. The RF classifier, out of all the classification algorithms, has the highest accuracy for all HAR mechanisms. As

7.3 Performance Evaluation

129

Fig. 7.6 Average accuracy of AccAR, KEHAR, SolAR and FusedAR in a indoors b outdoors using various classification algorithms (sampling frequency = 100 Hz, window size = 2 s)

a result, the RF classification technique is used to obtain all of the results in the rest of the chapter. Furthermore, due to the extensive context information encoded in its 3-axis accelerometer signal, AccAR provides the best accuracy (more than 99 % for all five activities) among the considered mechanisms. For the two individual energy harvesting-based sensing mechanisms, SolAR offers higher accuracy than KEHAR in both indoor and outdoor environments and SolAR-indoors offers higher accuracy than SolAR-outdoors. The advantage of SolAR-indoors over SolAR-outdoors is due to the uniform light availability, lower shadowing effect and more light sources which complement each other in indoor environment. On average, SolAR offers up to 4 % higher accuracy than KEHAR and at least 10.2 % lower accuracy than AccAR for a window size of 2 s. Figure 7.6 also presents the average activity recognition accuracy of FusedAR compared to individual energy harvesting mechanisms (KEHAR

130

7 Fusion-Based Activity Recognition

and SolAR) and conventional AccAR. It demonstrates that the FusedAR offers significantly higher HAR accuracy compared to SolAR and KEHAR in both indoor and outdoor environments and further reduces the accuracy gap to AccAR. This is due to the accumulated context information present in the fused signal compared to the individual energy harvesting signals, which leads towards enhanced activity recognition performance. To get more insights on the performance of FusedAR and SolAR compared to KEHAR, we present the confusion matrices for FusedAR, SolAR and KEHAR in Fig. 7.7. It depicts that outdoor scenarios for both SolAR and KEHAR offer lower HAR accuracy than indoors, in particular during the stairs activity, which can be due to more shadowing and uneven outdoor surface, respectively. We notice that SolAR offers significantly higher HAR accuracy than KEHAR for sitting and stairs activities. This is due to the distinct orientation of SEH transducer relative to the light source that helps SolAR to accurately distinguish sitting from standing and stairs from walking. Whereas, KEHAR struggles to distinguish theses activities which are very similar in their physical motion. In contrast, during walking and running activities, SolAR offers lower accuracy compared to KEHAR. It is due to potentially similar orientation of the solar cell in these activities compared to the significantly distinct movement pattern which helps KEHAR to take advantage over SolAR in distinguishing those two mobile activities. FusedAR employs signal fusion using both SEH and KEH signals to take advantage of both energy harvesting signals and enhance the performance closer to the conventional AccAR. The confusion matrices in Fig. 7.7 depict that the fused signal offers improved results for all activities compared to KEHAR and SolAR in both indoor and outdoor environments. This is because the fused signal extracts context information from both energy harvesting signals, which encompasses rich activity information in different dimensions i.e., changing ambient light and vibrations due to mobility. Thus both activity signals may complement each other in the fused signal and offer up to 10.2 % higher HAR performance compared to individual energy harvesting mechanisms (KEHAR and SolAR) and lower the accuracy gap (only 4.6 %) compared to AccAR. In the remaining of Sect. 7.3, we further evaluate the performance of SolAR and FusedAR compared to conventional AccAR and KEHAR for varying window sizes, signal sampling frequency, robustness to user variance, robustness to diverse lighting conditions and environment-agnostic scenarios.

7.3.2 Varying Window Sizes Figure 7.8 compares the HAR accuracy of FusedAR, SolAR, KEHAR and AccAR when varying the window size from 2 s to 12 s. As AccAR offers similar performance in both indoor and outdoor environments (see Fig. 7.6), we plot results only for the indoor scenario in the rest of the chapter. Figure 7.8 shows that AccAR offers a stable accuracy regardless of the window size due to the detailed activity information

7.3 Performance Evaluation

131

Fig. 7.7 Confusion matrices using a KEHAR, b SolAR and c FusedAR in indoors and outdoors (sampling frequency = 100 Hz, window size = 2 s, and RF classifier)

132

7 Fusion-Based Activity Recognition

Fig. 7.8 Average accuracy of AccAR, KEHAR, SolAR and FusedAR with increasing window sizes in indoors and outdoors (sampling frequency = 100 Hz, and RF classifier)

embedded in its 3-axis signal even for smaller window sizes. On the other hand, the accuracy of KEHAR and SolAR (both indoors and outdoors) improves when increasing the window size from 2 s to 12 s which means the window size is an important factor to consider for both KEHAR and SolAR. Figure 7.8 also confirms the improved accuracy of FusedAR over individual energy harvesting mechanisms (both indoors and outdoors) which bridges the accuracy gap to AccAR. It is also evident from the figure that FusedAR (both indoors and outdoors) offers a relatively stable accuracy regardless of the window size which is a similar behaviour to AccAR. It is important to note that increasing the window size results in increased computational complexity, latency and memory requirements. As a result, the window size should be chosen with the required HAR accuracy, system responsiveness, and processing complexity in mind due to the miniaturised and resource restricted wearable as our target device. Based on the previous explanation, we can see in Fig. 7.8 that 8 s is the smallest window size that provides the best HAR accuracy for all types of activity signals. As a result, the remaining of the results in this document are provided with an 8 s window.

7.3.3 Varying Sampling Frequency of the Signal Figure 7.9 compares the HAR accuracy of FusedAR, SolAR, KEHAR and AccAR when varying the sampling frequency of the activity signals from 10 Hz to 100 Hz. It also presents the power consumption for sampling the activity signals in FusedAR, SolAR and AccAR using our measurement setup presented in Sect. 7.4.2. Figure 7.9 depicts that AccAR provides a stable HAR accuracy regardless of the sampling

7.3 Performance Evaluation

133

Fig. 7.9 Average accuracy and required sampling power with varying sampling frequencies of AccAR, KEHAR, SolAR and FusedAR in indoors and outdoors (window size = 8 s and RF classifier)

frequency due to the rich activity information present in its 3-axis activity signal. On the other hand, KEHAR (both indoors and outdoor) shows an increase in HAR accuracy when varying the sampling rate similar to SolAR outdoors with an increment of about 4 % when increasing the sampling rate from 10 to 100 Hz. In contrast, SolAR indoors offers a stable accuracy with very small variation when increasing the sampling rate. It is also evident from Fig. 7.9 that FusedAR (both indoors and outdoors) offers higher HAR accuracy than individual energy harvesting-based mechanisms (i.e., SolAR and KEHAR) over various sampling frequencies and offers a stable performance when varying the sampling frequency. The higher sampling rates improve HAR accuracy because the activity signal has a higher resolution and can capture more fine-grained characteristics of the activity pattern. However, as seen in Fig. 7.9, energy consumption goes up as sampling frequency rises, owing to the collection of more samples in a given time interval. Figure 7.9 shows that FusedAR and SolAR requires up to 27 % and 36 % lower power than AccAR respectively, in sampling the activity signal, saving the sensor-related power consumption [11]. Additionally, keeping in mind the type of application and the energy budget, the sampling rate in SolAR and FusedAR can be set as low as

134

7 Fusion-Based Activity Recognition

10 Hz to save energy while maintaining activity detection accuracy of over 93 % and 96 %, respectively.

7.3.4 Robustness to User Variance The robustness of the FusedAR, SolAR, and KEHAR mechanisms against new/unknown users is examined in this section. To this purpose, we employ the RF algorithm, perform a leave-one-user-out CV and report the averaged findings in Table 7.5. The table shows that FusedAR indoors is the least sensitive to user variation and offers only 4.5 % lower average HAR accuracy for unseen users. Furthermore, SolAR indoors and AccAR are slightly more sensitive to the variation in the user and offer 9–10 % lower average accuracy for the unseen users. On the contrary, SolAR outdoors, FusedAR outdoors and KEHAR indoors/outdoors are significantly affected by the user variance and offer significantly lower average accuracy (17– 25.6 %) for new users. This is due to the significant variation in the kinetic harvested energy patterns from different users as well as the more shadowing and interference from the outdoor environment which affects SolAR outdoors. We expect that a larger training sample from diverse users will further improve the accuracy of the different mechanisms.

7.3.5 Robustness to Diverse Lighting Conditions As indicated in Table 7.2, we collected energy harvesting data from diverse environments during day and night in indoor and outdoor environments. In order to explore the performance of FusedAR and SolAR in different lighting conditions, we trained and evaluated these HAR algorithms using individual as well as combined datasets from day and night as listed in Table 7.2. Figure 7.10 shows that SolAR performs 2-4 % better than KEHAR indoors during both day and night due to high fidelity Table 7.5 Average accuracy from the user robustness experiment (sampling frequency = 100 Hz, window size = 8 s, and RF classifier) Environment HAR mechanism 10-fold CV Leave-one-user-out CV Indoors

Outdoors

Overall

KEHAR SolAR FusedAR KEHAR SolAR FusedAR AccAR

91.33 93.57 95.82 89.28 88.10 90.31 99.85

68.89 83.48 91.30 69.40 62.53 72.89 90.35

7.3 Performance Evaluation

135

Fig. 7.10 Average accuracy of KEHAR, SolAR and FusedAR in diverse lighting conditions (see Table 7.2) in indoors and outdoors (sampling frequency = 100 Hz, window size = 8 s, and RF classifier)

SEH signal which captures detailed activity information. In outdoor environment, due to shadowing, SolAR offers similar HAR accuracy as that of KEHAR during daylight as depicted in Fig. 7.10. KEHAR offers about 6.7 % higher HAR accuracy than SolAR during night time due to insufficient ambient light to embed the activity signature in the harvested solar power to accurately recognise the underlying activities. Finally, Fig. 7.10 depicts that FusedAR offers higher HAR accuracy than SolAR and KEHAR in various environments (indoors and outdoors) due to the integrated embedded information present in the fused signal as discussed in Sect. 7.3.1.

7.3.6 Robustness to Environment-Agnostic and Environment-Preserving Scenarios All of the previous results are obtained using the data collected from indoor and outdoor environments separately. This subsection explores the performance of FusedAR and SolAR compared to KEHAR when combined data from indoor and outdoor environments are employed. For this analysis, we consider the following two scenarios: • Environment-agnostic: In this scenario, the data from indoor and outdoor experiments are combined without information about the environment (e.g., walkingindoor and walking-outdoor are combined as one walking activity). • Environment-preserving: In this scenario, the data from indoor and outdoor experiments are combined while preserving the condition of the environment (e.g., walking-indoor, walking-outdoor, running-indoor, running-outdoor, etc). Figure 7.11 shows that, in environment-agnostic scenario, KEHAR, SolAR and FusedAR offer comparable performance with a small difference of 1.5-3 %. This can be due to the higher incongruity in the solar harvesting signal in indoor and outdoor environments which may affect the performance of both SolAR and FusedAR, bringing it closer to that of KEHAR. Figure 7.11 also shows that, in environmentpreserving scenario, SolAR offers 87.8 % HAR accuracy in contrast to its counterpart

136

7 Fusion-Based Activity Recognition

Fig. 7.11 Average accuracy of KEHAR, SolAR and FusedAR in environment-agnostic and environment-preserving scenarios (sampling frequency = 100 Hz, window size = 8 s, and RF classifier)

KEHAR which faces difficulty in recognising contexts and offers lower accuracy of 79.8 %. This reveals an advantage of SolAR compared to its counterpart KEHAR, which lacks rich information about the contexts in which activities are performed. Figure 7.11 also shows that, due to the fusion of both SEH and KEH signals, FusedAR offers higher accuracy (94.8 %) than SolAR and KEHAR in environment-preserving scenario. Thus, SolAR and FusedAR can recognise not only human activities but also environment (also called context such as indoors and outdoors) in which the activities are performed, in contrast to KEHAR which relies only on vibration/stress of physical human movements.

7.4 Analysis of Harvested and Consumed Power This section analyses the harvested power from SEH and KEH during various human activities as well as the consumed power in implementing the end-to-end HAR algorithm. We find that SolAR and FusedAR offer energy-positive HAR as the harvested power exceeds the required power for running the end-to-end HAR algorithm by more than one order of magnitude indoors and two orders of magnitude outdoors.

7.4.1 Harvested Power Using the tool from [26], we sample both voltage and current signals from the energy harvesting transducers, and then compute the average harvested power over a time interval τ as: H ar vested_ power =

τ 1 voltage(t) × curr ent (t) τ t=1

(7.1)

7.4 Analysis of Harvested and Consumed Power

137

Table 7.6 Average harvested power during various human activities using KEHAR, SolAR and FusedAR in indoors and outdoors Human activity Harvested Power [µW] Outdoors Indoors KEHAR SolAR FusedAR KEHAR SolAR FusedAR Running Walking Using stairs Standing Sitting Average power Power density [µW/cm2 ]

9.71 3.14 3.63 0.43 0.47 3.48 0.20

2839 2263 1802 867 1360 1826 130.44

2848.71 2266.14 1805.63 867.43 1360.47 1829.48 130.64

6.83 2.50 2.77 0.49 0.31 2.60 0.15

31.88 31.53 11.38 24.54 49.92 29.85 2.13

38.71 34.03 14.15 25.03 50.23 32.45 2.28

Table 7.6 presents the average harvested power from KEHAR, SolAR and FusedAR for the considered human activities in both indoor and outdoor environments. The power density, or harvested power per area, is described in the last row of Table 7.6. SolAR harvests more than one order of magnitude higher power indoors and more than two orders of magnitude higher power outdoors than KEHAR, due to the higher power density of visible light and superior energy conversion efficiency of solar cells [3]. The results in Table 7.6 also illustrate that the harvested power outdoors is higher than indoors for both SolAR and KEHAR. This is due to the reason that, in SolAR, natural outdoor sunlight has a higher power density compared to the artificial lights indoors. For KEHAR, this can be the result of following two effects: (1) walking on a paved outside surface produces more vibrations than walking on a carpeted floor indoors [36]; (2) individuals move quicker outdoors, resulting in stronger vibrations and, as a result, higher KEH power. Furthermore, FusedAR offers higher harvested power than SolAR and KEHAR due to the accumulated harvested energy from both transducers. It is also interesting to notice that the harvested power from SolAR is not tightly dependant on the human physical movements compared to KEHAR. Since KEHAR relies on the kinetic energy from the physical movements, it harvests a very small amount of power (0.31–0.49 µW) during static activities such as standing and sitting. While Shepherd [26] can sample signals with the resolution of 3 µA and 50 µV, the actual amount of harvested power can vary slightly, in particular, during sedentary activities due to significantly lower harvesting voltage and/or current. On the other hand, SolAR can harvest a certain non-zero amount of power during these static activities due to the availability of ambient light. Our experiments show that SolAR generates very small amount of energy at night due to the insignificant light intensity. For example, the average harvested power using SolAR at night is 0.21 µW for running, 0.18 µW for walking, 0.03 µW for using stairs, 0.07 µW for standing, and 0.06 µW for sitting. However, it is worth noting that even with this low level of harvested power during night, SolAR preserves its ability to distinguish the activities

138

7 Fusion-Based Activity Recognition

using distinct harvested power patterns. Finally, as shown in Table 7.6, FusedAR provides higher power compared to KEHAR and SolAR due to the accumulation of harvested energy from both transducers. This makes FusedAR most favourable HAR mechanism due to improved performance in both dimensions, i.e., activity recognition and harvested power.

7.4.2 Power Consumption Figure 7.12 depicts the hardware setup to measure the power consumption in implementing the complete end-to-end achar algorithm. We use an ultra low power Nordic Semiconductor nRF52840 wireless MCU and a Fluke 8845A multimeter to measure the average current draw. We place a 1 Ω shunt resistor and a TI AD8421 precision amplifier in series with the 3 V supply voltage to measure the dynamic current draw with a Rigol MSO5072 as depicted in Fig. 7.12. We also use the ultra low power digital accelerometer Bosch BMA400 as a baseline for comparison to state of the art. The firmware is configured to set a dedicated General Purpose Input/Output (GPIO) pin high while performing a task. The power consumption and execution time of a task can be measured by toggling a GPIO pin and recording the current consumption while the task is being executed. Table 7.7 provides the average, per-task power consumption of running SolAR and FusedAR as compared to AccAR and KEHAR. Based on the results from Sect. 7.3, we chose a sampling frequency of 10 Hz, a window size of 8 s where we extracted the indoor feature set, and use the RF classification algorithm. The result of the of the inferred activity is then communicated as a BLE packet (periodically after every 8 s), consisting of 6 bytes header, 3 bytes checksum and 1 byte payload. Our measurements from Table 7.7 show that AccAR requires higher power to implement the end-to-end HAR algorithm compared to SolAR and FusedAR mainly due to higher sampling power. Sampling SolAR and FusedAR signals consume 43.45 % and 12.41 % lower than the power required for sampling the 3-axis accelerometer (i.e., 4.35 µW) in AccAR. We also notice slight differences in the average required power for feature extraction and classification tasks per each mechanism. This is due to the different number and type of features to be used as input to the classification algorithm as for each mechanism. Interestingly, the average required power for implementing the classifier (0.049–0.089 µW) is one order of magnitude lower than the power required for feature extraction (1.567–2.557 µW) across the board including AccAR, KEHAR, SolAR and FusedAR. Overall power required to transmit the activity result is 0.125 µW, including 0.067 µW to power up the high frequency clock and 0.058 µW for transmitting the data packet. For rest of the 99.93 % time, the MCU stays in deep sleep mode consuming only 3.15 µW. Thus, as shown in Table 7.7, the total average power consumption of our implementation of SolAR and FusedAR is 22.36 % and 8.78 % lower than the 9.57 µW required power for AccAR which employs an ultra low power digital 3-axis accelerometer as the most efficient state-of-the-art baseline.

7.4 Analysis of Harvested and Consumed Power

139

Fig. 7.12 Experimental setup for measuring the power consumption in sampling the signal and implementing the end-to-end HAR algorithm using AccAR, KEHAR, SolAR and FusedAR Table 7.7 Average power consumption when implementing the end-to-end HAR algorithm on the sensor node using AccAR, KEHAR, SolAR, and FusedAR in an indoor environment (window size = 8 s) Task Average required power [µW] AccAR KEHAR SolAR FusedAR Sampling (@10 Hz) Feature extraction Classification Data transmission Sleep mode Total

4.35 1.90 0.05 0.12 3.15 9.57

2.46 2.56 0.08 0.12 3.15 8.37

2.46 1.64 0.06 0.12 3.15 7.43

3.81 1.58 0.07 0.12 3.15 8.73

It is also worth mentioning that our prototypical implementation is based on discrete off-the-shelf components and we compare it with the lowest power accelerometers which have been optimised for decades. For example, the power required for sampling the highly integrated Bosch BMA400 digital accelerometer is two orders of magnitude lower than the power required for sampling an analog accelerometer that is not integrated with the adc (e.g., Analog Devices ADXL356, ∼450 µW). By tightly integrating the amplifier with an optimised, low-power adc, we expect a further reduction of the power consumption in both SolAR and FusedAR in the same order that has been achieved by manufacturers of accelerometers. Moreover, the component cost for our circuit that measures solar current [17], including the amplifier (0.49 USD) and three resistors ( 1). The harvested power from the power required to run the HAR algorithm (Phar AccAR, KEHAR, SolAR and FusedAR is compared to the power required to run the HAR algorithm, averaged across all activities in Fig. 7.13. It shows that the average SolAR harvested power is greater than the power needed to operate the HAR algorithm on the sensor node both indoors and outdoors. SolAR is thus energypositive, allowing the sensor node to operate autonomously and perpetually without the use of external energy source. Figure 7.13 shows that FusedAR also provides higher harvested power (due to high power from SolAR) compared to the required power for running HAR algorithm and thus offers energy-positive HAR. Despite the fact that the KEH transducer we utilised in our study is larger and heavier than

Fig. 7.13 Average harvested and consumed power in implementing end-to-end HAR algorithm using AccAR, KEHAR, SolAR and FusedAR in indoors and outdoors

7.4 Analysis of Harvested and Consumed Power

141

Fig. 7.14 Average HAR accuracy vs average HAR power ratio using AccAR, KEHAR (KEH size: 18.03 cm2 , 30.46 g), SolAR (SEH size: 14.7 cm2 , 4.5 g) and FusedAR in indoors and outdoors

the solar cell (18.03 cm2 , 30.46 g vs. 14.7 cm2 , 4.5 g), the average KEH power is insufficient to execute the HAR algorithm on the sensor node, resulting in energynegative HAR. In contrast to SolAR and KEHAR, AccAR does not harvest any power and thus offers energy-negative HAR according to Eq. 7.2. We plot the average HAR accuracy and average HAR power ratio using AccAR, KEHAR, SolAR and FusedAR in Fig. 7.14. It shows that AccAR and KEHAR indoor/outdoor reside in the energy-negative region due to the lower harvested power than the consumed power in implementing the end-to-end HAR algorithm. On the other hand, SolAR and FusedAR offer energy-positive HAR in both indoor and outdoor environments due to the higher harvested power compared to the required power in running the HAR algorithm. One of the limitation of SolAR is that it needs a light source for its operation which means that it may face difficulties in recognising activities in low light conditions such as at night. In addition, it may not harvest sufficient power at night (and during low light conditions) to allow the perpetual operation of the wearable device. However, these challenges can be addressed by employing FusedAR that provides energy and context information concurrently using kinetic and solar energy harvesting transducers. In order to further enhance the performance, thermal and RF energy harvesters can also be employed in the future. It is interesting to mention that manually tuned KEH transducers can generate higher power [37] from human movements. However, it is impractical to tune the KEH transducer to specific scenarios. Furthermore, KEH transducers are fundamentally unable to deliver sufficient power during mostly static and sedentary activities. As humans generally spend a great proportion of their time performing such activities (e.g., office work), the average harvested power from KEH transducers may not be adequate to ensure

142

7 Fusion-Based Activity Recognition

the continuous operation of the wearable sensing devices. Thus, both SolAR and FusedAR offer energy-positive HAR for all activities (indoors and outdoors) and the remaining harvested energy can be used to power other body sensors leading towards truly pervasive wearable IoT.

7.5 Discussion This chapter presents FusedAR, a novel HAR mechanism which employs miniaturised wearable solar and kinetic energy harvesters for recognising human activities as well as energy sources. In contrast to conventional accelerometer-based HAR and recently developed KEH-based HAR, FusedAR generates sufficient energy to offer energy-positive HAR in which end-to-end activity recognition algorithm is implemented on the wearable device using only the harvested energy. After rigorous experiments, we found that FusedAR which combines the respective embedded context information from both solar and kinetic energy harvesters, recognises not only activities but also the environment/context in which activities are performed (i.e., indoors/outdoors and day/night). As both energy harvesting signals complement each other, the FusedAR offers significantly higher HAR accuracy compared to the individual energy harvesting signals. An interesting future direction is to evaluate Solar-based Activity Recognition (solar) and FusedAR using an extended dataset. This may involve more participants (male and female) to collect representative data from a diverse cohort varying in age, weight, and height. In addition, instead of employing only five activities, more activities of daily living (such as cooking, watching TV, cleaning, etc.) can be considered to develop a generalisable model for recognising common activities of daily living. Furthermore, data can be collected in diverse environments such as indoors, outdoors and in different locations to capture various solar energy harvesting scenarios for solar and FusedAR.

References 1. Sandhu MM, Khalifa S, Geissdoerfer K, Jurdak R, Portmann M (2021) SolAR: energy positive human activity recognition using solar cells. In: 2021 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 2. Pozo B, Garate JI, Araujo JÁ, Ferreiro S (2019) Energy harvesting technologies and equivalent electronic structural models. Electronics 8(5):486 3. Bhatti NA, Alizai MH, Syed AA, Mottola L (2016) Energy harvesting and wireless transfer in sensor network applications: concepts and experiences. ACM Trans Sens Netw (TOSN) 12(3):1–40 4. Seneviratne S, Hu Y, Nguyen T, Lan G, Khalifa S, Thilakarathna K, Hassan M, Seneviratne A (2017) A survey of wearable devices and challenges. IEEE Commun Surv Tutor 19(4):2573– 2620

References

143

5. Altun K, Barshan B, Tunçel O (2010) Comparative study on classifying human activities with miniature inertial and magnetic sensors. Pattern Recognit 43(10):3605–3620 6. Dian FJ, Vahidnia R, Rahmati A (2020) Wearables and the internet of things (IoT), applications, opportunities, and challenges: a survey. IEEE Access 8:69200–69211 7. Smarr BL, Aschbacher K, Fisher SM, Chowdhary A, Dilchert S, Puldon K, Rao A, Hecht FM, Mason AE (2020) Feasibility of continuous fever monitoring using wearable devices. Sci Rep 10(1):1–11 8. Ates HC, Yetisen AK, Güder F, Dincer C (2021) Wearable devices for the detection of COVID19. Nat Electron 4(1):13–14 9. Ramadhan AJ (2018) Wearable smart system for visually impaired people. Sensors 18(3):843 10. Sztyler T, Stuckenschmidt H, Petrich W (2017) Position-aware activity recognition with wearable devices. Pervasive Mob Comput 38:281–295 11. Khalifa S, Lan G, Hassan M, Seneviratne A, Das SK (2017) Harke: human activity recognition from kinetic energy harvesting data in wearable devices. IEEE Trans Mob Comput 17(6):1353– 1368 12. Lan G, Ma D, Xu W, Hassan M, Hu W (2017) Capsense: capacitor-based activity sensing for kinetic energy harvesting powered wearable devices. In: Proceedings of the 14th EAI international conference on mobile and ubiquitous systems: computing, networking and services. ACM, pp 106–115 13. Kalantarian H, Alshurafa N, Le T, Sarrafzadeh M (2015) Monitoring eating habits using a piezoelectric sensor-based necklace. Comput Biol Med 58:46–55 14. Lan G, Xu W, Ma D, Khalifa S, Hassan M, Hu W (2019) Entrans: leveraging kinetic energy harvesting signal for transportation mode detection. IEEE Trans Intell Transp Syst 15. Lin Q, Xu W, Liu J, Khamis A, Hu W, Hassan M, Seneviratne A (2019) H2b: heartbeat-based secret key generation using piezo vibration sensors. In: Proceedings of the 18th international conference on information processing in sensor networks. ACM, pp 265–276 16. Ma D, Lan G, Xu W, Hassan M, Hu W (2018) SEHS: simultaneous energy harvesting and sensing using piezoelectric energy harvester. In: 2018 IEEE/ACM third international conference on internet-of-things design and implementation (IoTDI). IEEE, pp 201–212 17. Sandhu MM, Geissdoerfer K, Khalifa S, Jurdak R, Portmann M, Kusy B (2020) Towards energy positive sensing using kinetic energy harvesters. In: 2020 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 18. Kansal A, Hsu J, Srivastava M, Raghunathan V (2006) Harvesting aware power management for sensor networks. In: Proceedings of the 43rd annual design automation conference, pp 651–656 19. Ma D, Lan G, Hassan M, Hu W, Upama MB, Uddin A, Youssef M (2019) Solargest: ubiquitous and battery-free gesture recognition using solar cells. In: The 25th annual international conference on mobile computing and networking. ACM, pp 1–15 20. Umetsu Y, Nakamura Y, Arakawa Y, Fujimoto M, Suwa H (2019) Ehaas: energy harvesters as a sensor for place recognition on wearables. In: Proceedings of the 2019 IEEE international conference on pervasive computing communications (PerCom). IEEE, pp 1–10 21. Sharma H, Haque A, Jaffery ZA (2018) Modeling and optimisation of a solar energy harvesting system for wireless sensor network nodes. J Sens Actuator Netw 7(3):40 22. Rezaie H, Ghassemian M (2017) An adaptive algorithm to improve energy efficiency in wearable activity recognition systems. IEEE Sens J 17(16):5315–5323 23. Fafoutis X, Marchegiani L, Elsts A, Pope J, Piechocki R, Craddock I (2018) Extending the battery lifetime of wearable sensors with embedded machine learning. In: IEEE 4th world forum on internet of things (WF-IoT). IEEE, pp 269–274 24. Wang A, Chen L, Xu W (2017) Xpro: a cross-end processing architecture for data analytics in wearables. ACM SIGARCH Comput Archit News 45(2):69–80 25. Verhelst M, Moons B (2017) Embedded deep neural network processing: algorithmic and processor techniques bring deep learning to iot and edge devices. IEEE Solid-State Circuits Mag 9(4):55–65

144

7 Fusion-Based Activity Recognition

26. Geissdoerfer K, Chwalisz M, Zimmerling M (2019) Shepherd: a portable testbed for the batteryless iot. In: Proceedings of the 17th ACM conference on embedded networked sensor systems (SenSys), 2019, pp 83–95 27. Stisen A, Blunck H, Bhattacharya S, Prentow TS, Kjærgaard MB, Dey A, Sonne T., Jensen MM (2015) Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In: Proceedings of the 13th ACM conference on embedded networked sensor systems, 2015, pp 127–140 28. Lim YP, Lin Y-C, Pandy MG (2017) Effects of step length and step frequency on lower-limb muscle function in human gait. J Biomech 57:1–7 29. Russell DM, Apatoczky DT (2016) Walking at the preferred stride frequency minimizes muscle activity. Gait & Posture 45:181–186 30. Hemminki S, Nurmi P, Tarkoma S (2013) Accelerometer-based transportation mode detection on smartphones. In: Proceedings of the 11th ACM conference on embedded networked sensor systems. ACM, p 13 31. Ross BC (2014) Mutual information between discrete and continuous data sets. PloS one 9(2):e87357 32. Minka TP (2001) Automatic choice of dimensionality for pca. In: Advances in neural information processing systems, 2001, pp 598–604 33. Emura T, Matsui S, Chen H-Y (2019) Compound. Cox: univariate feature selection and compound covariate for predicting survival. Comput Methods Programs Biomed 168:21–37 34. Mursalin M, Zhang Y, Chen Y, Chawla NV (2017) Automated epileptic seizure detection using improved correlation-based feature selection with random forest classifier. Neurocomputing 241:204–214 35. Han H, Wang W-Y, Mao B-H (2005) Borderline-smote: a new over-sampling method in imbalanced data sets learning. In: International conference on intelligent computing. Springer, pp 878–887 36. Menz HB, Lord SR, Fitzpatrick RC (2003) Acceleration patterns of the head and pelvis when walking on level and irregular surfaces. Gait & Posture 18(1):35–46 37. Kuang Y, Ruan T, Chew ZJ, Zhu M (2017) Energy harvesting during human walking to power a wireless sensor node. Sens Actuators A Phys 254:69–77

Chapter 8

Energy-Positive Activity Recognition: Future Directions

This book has presented various mechanisms for self-powered activity recognition in IoT. Conventional activity recognition systems use various activity sensors such as accelerometers, magnetometers and gyroscopes for wearable-based activity recognition. However, these systems consume significant energy and thus deplete the batteries of the wearable very quickly, resulting in their limited active lifetime. This impedes the pervasive adaptability of wearable devices as their batteries need to be replaced or recharged regularly. Although energy can be harvested from the environment (such as solar, kinetic, thermal and RF) to power these wearable IoT devices, it is insufficient to perpetually operate the sensing devices, particularly in human-centric applications while using tiny and miniaturised transducers. Recently, energy harvesters have also been employed as a source of context information in a plethora of applications including human activity recognition [1–3], transport mode detection [4] and room-level place recognition [5]. This book has demonstrated the use of KEH transducer as an activity senors for HAR. As the vibration pattern changes during various activities, a wearable KEH transducer generates distinct pattern of energy harvesting signal that embeds unique information about the underlying activity. A machine learning algorithm can be developed using the collected data from a wearable KEH transducer which can later be used to infer and predict the activities in real-time. This book has also explored the use of KEH transducer as a simultaneous source of energy and context information for HAR as well as to power the signal acquisition in the wearable IoT devices. However, due to limited harvested energy, particularly during static and sedentary human activities, a tiny and miniaturised wearable KEH transducer may not offer sufficient power to perpetually operate the wearable IoT devices. Therefore, a SEH transducer can be employed as a simultaneous source of energy and context information resulting in energy-positive HAR. As SEH transducers (such as solar cells) offer higher power density and energy conversion efficiency than their counterpart KEH transducers, they can harvest sufficient power for the perpetual and uninterrupted operation of wearable IoT devices. Finally, fusing the KEH and SEH signals in FusedAR can provide improved HAR accuracy as well higher harvested power for the reliable, © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9_8

145

146

8 Energy-Positive Activity Recognition: Future Directions

perpetual and uninterrupted operation of wearable IoT devices. Thus, SolAR and FusedAR offer end-to-end energy-positive HAR where the harvested energy can not only power the signal acquisition but also the the implementation of complete HAR pipeline on the embedded IoT sensor node. Some of the future directions for employing energy harvesters as activity sensors as well as source of energy to enable autonomous operation of wearable IoT devices for real-time HAR are described in the following subsections.

8.1 Energy-Efficient Communication Using Energy Harvesters Lan et al. [6] explore the use of KEH transducer as a communication receiver. The main advantage of KEH-based communication receiver is that the sound signal is decoded without using power consuming digital signal processing module, which makes it an energy-efficient solution compared to conventional microphone-based decoder. In another study [7], the authors hide the audio signals in a background music signal and employ sound masking theory to maximize signal to noise ratio of data communication without being audible to the music listener. Ma et al. [8] employ KEH transducer for vibration-based Multiple-Input Multiple-Output (MIMO) communication over human skin. Using motors as transmitters and KEH transducers as receivers, the authors in [8] show that Skin-MIMO can improve MIMO capacity by a factor of 2.3 compared to single-input-single-output or open-loop MIMO. In the future, energy harvesters can be explored as energy-efficient data receivers that can work in long range communication, provide high data rate and offer robust performance in the presence of ambient noise.

8.2 Deep Learning Instead of using conventional machine learning models, emerging deep learning models [9] can be employed to achieve improved performance depending upon the type of sensors, amount of generated data and the type of application. In contrast to classical machine learning algorithms that typically rely on manual hand crafted features which may not extract rich embedded information from the contextual data, deep learning models automatically extract the hidden context information in the raw data and thus can lead to superior performance [10, 11]. In addition, due to higher complexity [10], deep learning models generally offer better performance when the volume of the generated data is significantly large to train the model. Given the nature of activity recognition application, the variety of on-body, off-body and ambient sensors continuously generate high volume of data. This leverages deep learning models to learn the historical patterns and predict the future activity and context with

8.4 Personalised AI Models

147

high accuracy. Although compared to classical machine learning algorithms, deep learning models take longer to train the model [10], the difference in training time becomes insignificant when the model is implemented on a resource-sufficient server. In addition, the enhanced performance of deep learning models is sometimes more desirable in certain applications such as elderly monitoring. However, importing deep learning models on resource-constrained edge IoT devices can be expensive in terms of processing time, energy consumption and resource usage. This is another open direction for future research.

8.3 Federated Learning Previous studies [2, 3] implement machine learning algorithms at the cloud or at the edge depending on the type of application. Federated learning [12] combines the benefits of both edge and cloud computing and employs both types of resources to enhance the overall performance. In federated learning, sensor nodes implement a local light-weight machine learning or deep learning model and share the model parameters (instead of user’s raw data) to the cloud/server which aggregates the locally trained models from all sensor nodes and sends back the aggregated global model to the nodes for further training and prediction, and the cycle goes on. This mechanism addresses the data/user privacy issue as sensor nodes do not need to share the private and sensitive user-specific raw data to any other device. In addition, the model is not bounded by the limited resources of the edge IoT device, instead, the (abundantly available) cloud/server resources are used to train the overall global model. This results in highly accurate machine learning or deep learning models without the risk of data privacy and security issues. It also results in lower latency due to the implementation of the model on the same node and alleviating the need of transmitting large volume of raw data to a remote server.

8.4 Personalised AI Models Wearable IoT devices collect data continuously [13] which can embed detailed information about the user’s activity and health. Comparing the physiological data of a person with its own previous history can provide useful information—detecting deviation from one’s usual baseline—instead of comparing it with the population statistics which is a fundamentally different approach [13]. Therefore, personalised models can be developed for individual users to accurately detect the events of interest instead of using heterogeneous data from a group of different users [14] that may possess non independent and identical data distributions. Thus the personalised AI model can accurately detect and identify any abnormality (from the same user) and thus can offer enhanced performance.

148

8 Energy-Positive Activity Recognition: Future Directions

8.5 Real-Time Activity Recognition Most of the previous HAR systems [2, 3, 5, 15–19] focus on collecting data from wearable sensors and store it on a cloud/server which is later used to evaluate physical activities of the user. Instead of using this reactive approach, the data can be monitored and analysed proactively in real-time to provide timely assistance/advice to the user. It can also reduce the risk of serious injury as the abnormal activity can be detected at an early stage—minimising the health and safety risks. This mechanism of real-time activity monitoring takes full advantage of the deployed IoT solutions to significantly enhance the quality of life of individuals and ensure their healthy, active and safe living. Therefore, an optimised hardware that can implement energy harvesting-based HAR online for end-to-end real-time activity recognition is an important future step.

8.6 Multi-source Energy Harvesters Most of the previous works employ either kinetic [2, 15] or solar energy harvesters [20] for recognising human activities. Therefore, in the future, other energy harvesters such as TEH and RFEH can be employed as activity sensors and source of energy simultaneously. An interesting future direction is to combine multi-source energy harvesters and explore their overall performance as activity sensor and source of energy. As multi-source energy harvesters can complement each other, they may offer improved activity recognition performance compared to individual energy harvesting transducers.

8.7 Hardware Implementation on the Edge Device In the future, energy harvesting-based sensing can be implemented in a real-world environment to infer the activities/events in real-time. This can include increased number of classes to capture most of the activities of daily living. Implementation on the edge device may involve exploring light weight machine learning and deep learning models to implement on resource-constrained edge IoT devices. It is also important to explore suitable edge IoT deices for HAR applications. The harvested energy from the transducers can be employed to power the device without the need of any other external source—resulting in its autonomous and perpetual operation. Thus, implementing and complete HAR system that can run energy harvesting-based HAR online solely using the harvested energy is a promising future direction to enable the self-powered and autonomous operation of wearable devices for HAR applications.

8.10 Reducing the System Cost

149

8.8 Batteryless Operation Batteries are not only bulky, expensive, and hazardous [21] but also incompatible with a sustainable IoT. Therefore, instead of batteries, capacitors can be employed to store the harvested energy to later power the IoT device. However, capacitors possess lower energy density compared to batteries—resulting in the intermittent operation of the sensor node instead of its continuous operation. In the future, HAR systems can be implemented on a batteryless IoT device that can operate intermittently using the harvested energy in the capacitor. This will result in the sustainable and self-powered operation of wearable IoT device for HAR without the need of human intervention.

8.9 Security and Privacy The human activity data collected from sensor nodes is subject to security and privacy risks. There are various methods that can be employed to ensure data confidentiality and integrity both during communication and at the end device. Encryption is a de-facto method to ensure confidentiality in communication where both sender and receiver use a secret key to decrypt the data [22–25]. In addition to encryption, authentication protocols [26, 27] can be employed to insert an additional layer of security in situations where malicious nodes can impersonate servers to get unauthorised access to the data. Moreover, in order to increase user discretion while maintaining a balance between the security of personal information and the effectiveness of the system in terms of fulfilling its objectives, researchers should consider building contextual privacy-protection interfaces and tools [28].

8.10 Reducing the System Cost The cost of a technological system plays an important and critical role in its widespread deployment and worldwide adoption [29]. In addition to being robust, user-friendly and reliable, the cost of the HAR system should be within the reach of individuals. Furthermore, the system should have backward compatibility to incorporate the existing healthcare and communication infrastructure instead of re-deploying the basic sensing and communication hardware components ab initio. The HAR system should also be designed keeping in mind the current infrastructure [30].

150

8 Energy-Positive Activity Recognition: Future Directions

8.11 Exploring Other Applications Most of the previous works employ energy harvesters as context sensors in recognising human activities [1, 2, 15, 17], transport modes [4], and room-level places [5]. However, the activity recognition mechanism presented in this book can be employed in other applications including cattle monitoring, travel route planning, animal location/position tracking, etc. In healthcare applications, energy harvesting-based HAR can be employed to detect Parkinson disease, Ataxia, Tic disorders, Huntington’s disease, essential tremor and Dystonia. In addition, energy harvesting-based sensors can be employed for monitoring emotional dysregulation and agitation in people living with dementia and on autism spectrum. Therefore, in the future, energy harvesting-based HAR can be employed in various applications to explore its performance compared to conventional methods. Later, energy harvesting-based sensing can replace the conventional methods due to improved performance, low cost, pervasive deployment, and autonomous and perpetual operation.

References 1. Khalifa S, Hassan M, Seneviratne A, Das SK (2015) Energy-harvesting wearables for activityaware services. IEEE Internet Comput 19(5):8–16 2. Khalifa S, Lan G, Hassan M, Seneviratne A, Das SK (2017) Harke: human activity recognition from kinetic energy harvesting data in wearable devices. IEEE Trans Mobile Comput 17(6):1353–1368 3. Sandhu MM, Khalifa S, Geissdoerfer K, Jurdak R, Portmann M (2021) SolAR: energy positive human activity recognition using solar cells. In: 2021 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 4. Lan G, Xu W, Ma D, Khalifa S, Hassan M, Hu W (2019) Entrans: leveraging kinetic energy harvesting signal for transportation mode detection. IEEE Trans Intell Transp Syst 5. Umetsu Y, Nakamura Y, Arakawa Y, Fujimoto M, Suwa H (2019) Ehaas: energy harvesters as a sensor for place recognition on wearables. In: Proceedings of the 2019 IEEE international conference on pervasive computing communications (PerCom). IEEE, pp 1–10 6. Lan G, Xu W, Khalifa S, Hassan M, Hu W (2017) Veh-com: demodulating vibration energy harvesting for short range communication. In: IEEE International conference on pervasive computing and communications (PerCom), Hawaii, USA, 2017, pp 170–179 7. Lan G, Ma D, Hassan M, Hu W (2018) Hiddencode: hidden acoustic signal capture with vibration energy harvesting. In: 2018 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 8. Ma D, Wu Y, Ding M, Hassan M, Hu W (2020) Skin-mimo: vibration-based mimo communication over human skin. In: IEEE INFOCOM 2020-IEEE conference on computer communications. IEEE, pp 784–793 9. Baldominos A, Cervantes A, Saez Y, Isasi P (2019) A comparison of machine learning and deep learning techniques for activity recognition using mobile devices. Sensors 19(3):521 10. Moradi B, Aghapour M, Shirbandi A (2022) Compare of machine learning and deep learning approaches for human activity recognition. In: 2022 30th international conference on electrical engineering (ICEE). IEEE, pp 592–596 11. Shakya SR, Zhang C, Zhou Z (2018) Comparative study of machine learning and deep learning architecture for human activity recognition using accelerometer data. Int J Mach Learn Comput 8(6):577–582

References

151

12. Li T, Sahu AK, Talwalkar A, Smith V (2020) Federated learning: challenges, methods, and future directions. IEEE Signal Process Mag 37(3):50–60 13. Ates HC, Yetisen AK, Güder F, Dincer C (2021) Wearable devices for the detection of COVID19. Nat Electron 4(1):13–14 14. Bianchi V, Bassoli M, Lombardo G, Fornacciari P, Mordonini M, De Munari I (2019) Iot wearable sensor and deep learning: an integrated approach for personalized human activity recognition in a smart home environment. IEEE Internet of Things J 6(5):8553–8562 15. Lan G, Ma D, Xu W, Hassan M, Hu W (2017) Capsense: capacitor-based activity sensing for kinetic energy harvesting powered wearable devices. In: Proceedings of the 14th EAI international conference on mobile and ubiquitous systems: computing, networking and services. ACM, pp 106–115 16. Lin Q, Xu W, Lan G, Cui Y, Jia H, Hu W, Hassan M, Seneviratne A (2020) Kehkey: kinetic energy harvester-based authentication and key generation for body area network. In: Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, vol 4, no 1, pp 1–26 17. Lan G, Ma D, Xu W, Hassan M, Hu W (2020) Capacitor-based activity sensing for kineticpowered wearable iots. ACM Trans Internet of Things 1(1):1–26 18. Khalifa S, Hassan M, Seneviratne A (2015) Step detection from power generation pattern in energy-harvesting wearable devices. In: IEEE international conference on data science and data intensive systems. IEEE, pp 604–610 19. Khalifa S, Lan G, Hassan M, Hu W (2016) A Bayesian framework for energy-neutral activity monitoring with self-powered wearable sensors. In: 2016 IEEE international conference on pervasive computing and communication workshops (PerCom Workshops). IEEE, pp 1–6 20. Sandhu MM, Geissdoerfer K, Khalifa S, Jurdak R, Portmann M, Kusy B (2020) Towards energy positive sensing using kinetic energy harvesters. In: 2020 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10 21. Hester J, Sorber J (2017) The future of sensing is batteryless, intermittent, and awesome. In: Proceedings of the 15th ACM conference on embedded network sensor systems. ACM, p 21 22. Snader R, Kravets R, Harris III AF (2016) Cryptocop: lightweight, energy-efficient encryption and privacy for wearable devices. In: Proceedings of the 2016 workshop on wearable systems and applications, 2016, pp 7–12 23. Venkatasubramanian KK, Banerjee A, Gupta SKS (2009) Pska: usable and secure key agreement scheme for body area networks. IEEE Trans Inf Technol Biomed 14(1):60–68 24. Revadigar G, Javali C, Xu W, Hu W, Jha S (2016) Secure key generation and distribution protocol for wearable devices. In: 2016 IEEE international conference on pervasive computing and communication workshops (PerCom Workshops). IEEE, pp 1–4 25. Venkatasubramanian KK, Banerjee A, Gupta SK (2008) Plethysmogram-based secure intersensor communication in body area networks. In: MILCOM 2008-2008 IEEE military communications conference. IEEE, pp 1–7 26. Rahman M, Carbunar B, Banik M (2013) Fit and vulnerable: attacks and defenses for a health monitoring device. arXiv:1304.5672 27. Diez FP, Touceda DS, Camara JMS, Zeadally S (2015) Toward self-authenticable wearable devices. IEEE Wirel Commun 22(1):36–43 28. Nathan V, Paul S, Prioleau T, Niu L, Mortazavi BJ, Cambone SA, Veeraraghavan A, Sabharwal A, Jafari R (2018) A survey on smart homes for aging in place: toward solutions to the specific needs of the elderly. IEEE Signal Process Mag 35(5):111–119 29. Christensen H, Amato N, Yanco H, Mataric M, Choset H, Drobnis A, Goldberg K, Grizzle J, Hager G, Hollerbach J et al (2021) A roadmap for us robotics–from internet to robotics 2020 edition. Found Trends® Robot 8(4):307–424 30. Robinson H, MacDonald B, Broadbent E (2014) The role of healthcare robots for older people at home: a review. Int J Soc Robot 6(4):575–591

Author Index

A Abadi, M.J., 57 Abari, O., 59 Aberer, K., 16 Abou Chahine, S., 5 Abraham, M.E., 5, 6 Abuellil, A., 35 Adesina, A.A., 59 Affeld, K., 6 Aghapour, M., 146 Agostini, V., 18 Ahmad, F.F., 47 Ahmed, N., 18 Alberth Jr, W.P., 4 Alemdar, K., 34 Alfes, C.M., 3 Alhaddad, D., 18 Ali, A., 18 Ali, A.H., 18 Alizai, M.H., 35, 46, 111, 137 Aljarrah, A.A., 18 Almogren, A., 3 Alsharif, M.H., 18 Alshurafa, N., 6, 58, 74, 96, 97, 121 Altun, K., 120 Alves, F., 5 Amato, N., 149 Aminian, K., 4 Amirat, Y., 18 Anguita, D., 20 Anjum, A., 18 Anliker, U., 6 Anton, S.R., 58 Apatoczky, D.T., 102, 126 Appelboom, G., 5, 6

Arakawa, Y., 54, 58, 63, 64, 75, 76, 78, 97, 121, 145, 148, 150 Araujo, J.A., 35 Arefin, M.S., 33 Aschbacher, K., 120 Ates, H.C., 120, 147 Attal, F., 18 Atzmueller, M., 13 Azzopardi, G., 13

B Bahr, R., 33, 46 Balachander, B., 47 Baldominos, A., 146 Balestra, G., 18 Banerjee, A., 149 Banik, M., 149 Banos, O., 20 Barczyk, A.N., 6 Barnhill, S., 84 Barshan, B., 5, 120 Bassoli, M., 147 Beevers, C.G., 6 Beigl, M., 16 Benini, L., 33, 76 Berchtold, M., 16 Bergmann, N., 57 Bernardos, A.M., 16, 18 Bettayeb, M., 47 Bhagwat, T., 97 Bhattacharya, S., 102, 126 Bhatti, N.A., 35, 46, 111, 137 Bianchi, V., 147 Bibbo, D., 34

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9

153

154 Bieber, G., 16 Bito, J., 29–31, 33, 46 Blaabjerg, F., 18 Blank, P., 60 Blaževi´c, D., 33 Blunck, H., 102, 126 Bobek, S., 13 Borges, L.M., 18 Bosch, S., 16, 20 Bouchabou, D., 6 Bouendeu, E., 30 Bowyer, K.W., 84 Bozic, B., 18 Bries, M., 4 Brill, R., 4 Broadbent, E., 149 Bruce, S.S., 5, 6 Bruyère, O., 5, 6 Budde, M., 16 Büla, C.J., 4 Burke, J., 16 Bychkov, D., 4

C Caduf, A., 6 Calatroni, A., 20 Calero, D., 5 Calvo, A., 4 Camacho, E., 5, 6 Camara, J.M.S., 149 Cambone, S.A., 149 Cambria, E., 64 Campbell, A.T., 16, 18 Campbell, B., 59 Cang, S., 15 Cao, H., 18 Carbunar, B., 149 Carrara, S., 6 Casar, J.R., 16, 18 Cerny, M., 34 Cervantes, A., 146 Chai, D., 18 Chai, E., 59 Chakraborty, D., 16 Chamroukhi, F., 18 Chang, N., 31, 33 Charbiwala, Z., 54, 59 Chaudhry, S.A., 18 Chavarriaga, R., 20 Chawla, N.V., 84, 102, 126 Cheng, R., 5 Chen, H.Y., 102, 126

Author Index Chen, L., 65, 98, 124 Chen, X., 65 Chen, Y., 102, 126 Chew, Z.J., 34, 113, 141 Chien, C., 18 Chi, Z., 39, 63, 64, 73 Chng, H.B., 34 Choi, D., 16 Chong, Y.W., 47 Choset, H., 149 Chou, C.T., 59 Choudhury, T., 16, 18 Chou, W.F., 97 Chowdhary, A., 120 Chowdhury, K.R., 34 Chrisman, B., 14 Christensen, H., 149 Chuang, J.-S., 18 Chwalisz, M., 31, 71 Cobelli, C., 6 Collado, A., 29–31 Conforto, S., 34 Cook, D.J., 6, 15, 16, 18 Cordioli, J.A., 5 Cornelius, C., 4 Cosoli, G., 4 Craddock, I., 98, 124 Craddock, R.C., 6 Cui, Y., 9, 58, 148

D Damaj, A.W., 5 Damas, M., 20 D’Amico, R., 5, 6 Das, B., 16, 18 Das, S.K., 6, 7, 9, 17, 19–21, 29, 36, 53, 54, 56, 64, 75, 76, 78, 79, 81, 84, 95, 96, 100, 102, 120, 145, 148, 150 Dearani, J.A., 6 Dedabrishvili, M., 18 De Leonardis, G., 18 De Munari, I., 147 De Oliveira, M.C.F., 13 Dernbach, S., 16 Deschamps, C., 6 De Toledo, P., 14 Devarakonda, P.G., 18 Dey, A., 102, 126 Dian, F.J., 120 Díez-Fernández, A., 13 Diez, F.P., 149 Digumarti, S.T., 20

Author Index Dilchert, S., 120 Dincer, C., 120, 147 Ding, M., 146 Divakaran, S.K., 34, 35 Dong, K., 5 Drobnis, A., 149 Drummond, C.K., 3 Duan, J., 34 Duan, S., 18 Dumont, E.L., 5, 6 Dunn, J., 3 Durmaz Incel, O., 18 Dutta, P., 59

E El Misilmani, H.M., 5 Eisenman, S.B., 18 Elkaseer, A., 18 Ellis, K., 20 Ellul, J., 13 Elsts, A., 98, 124 Emura, T., 102, 126 Ersoy, C., 16, 18 Eskofier, B.M., 60 Estrada-López, J.J., 35 Estrin, D., 16

F Facchinetti, A., 6 Fafoutis, X., 98, 124 Fan, L., 18 Fan, P., 34 Feeney, A., 8, 9 Feng, X., 64 Ferng, H.-W., 18 Ferreiro, S., 35 Fidan, B., 5 Firdaus, H., 33 Fisher, S.M., 120 Fitzpatrick, R.C., 111, 137 Flynn, B.O., 13 Fodor, K., 18 Fogarty, J., 16 Fornacciari, P., 147 Froehlich, J., 16 Fujimoto, M., 54, 58, 63, 64, 75, 76, 78, 97, 121, 145, 148, 150 Fuks, H., 4

G Gaber, M.M., 18

155 Galvin, P., 13 Gangasri, A., 18 Gao, B., 15 Garate, J.I., 35 Garcia, R., 20 Gastaldi, L., 18 Geissdoerfer, K., 31, 71, 114 Georgiadis, A., 29–31, 33, 46 Gesing, A., 5 Ghani, N.A., 18 Ghannam, R., 8, 9 Ghassemian, M., 98, 124 Ghena, B., 59 Ghenai, C., 47 Ghio, A., 20 Gholamiangonabadi, D., 18 Gljuš´ci´c, P., 33 Gobbi, M., 4 Goldberg, K., 149 Gomes, J.B., 18 Gomez, A., 76 Gordon, D., 16 Gorlatova, M., 56 Gosling, S.D., 6 Green, T.C., 33 Greiner, A., 30 Grizzle, J., 149 Grolinger, K., 18 Gruenerbl, A., 4 Güder, F., 120, 147 Gummeson, J., 4 Gupta, S.K.S., 149 Guyon, I., 84

H Hager, G., 149 Hakem, N., 34 Halim, M.A., 8, 9 Hall, L.O., 84 Halter, R., 4 Hamid, S., 18 Han, H., 103, 128 Hanif, M.A., 15 Hansen, M., 16 Haque, A., 18, 29, 98, 123 Harari, G.M., 6 Harris III, A.F., 149 Hassan, M., 6, 7, 9, 17, 19–21, 29, 36, 46, 53–59, 63, 64, 72–79, 81, 84, 95–97, 100, 102, 103, 120, 121, 145, 146, 148, 150 Hassan, M.M., 3

156 Havinga, P.J., 16, 20 Hazarika, D., 64 Hecht, B., 84 Hecht, F.M., 120 Hegde, N., 4 Heidari, H., 8, 9 He, J., 18 He, L., 4 Hemminki, S., 40, 84, 96, 102, 126 Henpraserttae, A., 16, 18 Hester, J., 7, 97, 121, 149 Hester, J.G., 33, 46 Hirt, E., 6 Hoang, D.C., 34 Hoang, M.T., 16 Hodges, R., 34 Holder, L.B., 18 Holgado-Terriza, J.A., 20 Hollerbach, J., 149 Holmes, A.S., 33 Holz, C., 4 Hou, C., 14 Hsu, J., 7, 8, 46, 121 Huang, L., 34 Huang, Q., 73 Hu, B., 34 Hu, W., 9, 46, 56–59, 63, 64, 72–75, 77, 79, 95–97, 103, 121, 145, 146, 148–150 Hu, Y., 4, 6, 7, 9, 15, 20, 99, 120 I Iglesias, J., 16, 18 Ignatov, A., 18 Ilyas, M.U., 18 Incel, O.D., 16, 18, 20 Inman, D., 73 Intille, S.S., 14 Isasi, P., 146 Islam, M.R., 18 Ismail, W., 47 Iso, T., 16 J Jafari, R., 149 Jaffery, Z.A., 29, 98, 123 Jain, P., 3 James, D.A., 18 Javali,W, 149 Jensen, M.M., 102, 126 Jeong, I.C., 4 Jeung, H., 16 Jha, S., 149

Author Index Jia, H., 9, 58, 148 Jia, X., 97 Jibukumar, M., 34 Jokic, P., 33 Joshi, A.M., 3 Joshitha, K.L., 18 Jurdak, R., 3, 21, 114 Jur, J.S., 34

K Kalantarian, H., 6, 55, 58, 64, 74, 96, 121 Kamenar, E., 33 Kanellos, I., 6 Kanhere, S.S., 57 Kansal, A., 7, 8, 46, 121 Kantarci, B., 18 Kao, H.L., 4 Kasik, V., 34 Kautz, T., 60 Kegelmeyer, W.P., 84 Kelechi, A.H., 18 Kertzscher, U., 6 Keshav, S., 59 Khalid, M., 15 Khalifa, S., 3, 4, 6, 7, 9, 15, 17, 19–21, 29, 36, 53–58, 64, 75, 76, 78, 79, 81, 84, 95, 96, 99, 100, 102, 103, 120, 145, 146, 148, 150 Khamis, A., 58, 59, 121 Khan, A.M., 15, 16, 18 Khan, A.U., 18 Khanbareh, H., 8, 9 Khan, G., 18 Khan, M.U.G., 18 Khan, S.S., 13 Khattak, A.M., 16 Khojastepour, M.A., 59 Kim, S., 29–31 Kim, T.-S., 15 Kim, Y., 33 Kippelen, B., 97 Kjærgaard, M.B., 102, 126 Kline, A., 14 Knaflitz, M., 18 Kneubühler, D., 33 Ko, K., 47 Koldrack, P., 16 Korvink, J.G., 30 Kose, M., 16, 18 Kotz, D., 4 Kravets, R., 149 Krishna, D.D., 34, 35

Author Index Krishnan, N.C., 16 Krishnaswamy, S., 18 Kuang, Y., 34, 113, 141 Kulkarni, P., 46 Kusy, B., 59 Kutt, K., 13 Kwapisz, J.R., 18, 20

L Labrador, M.A., 18 Laine, T.H., 16 Lakshmi, P.S., 34 Lanckriet, G., 20 Landay, J., 16 Lane, N.D., 16, 18 Lan, G., 4, 6, 7, 9, 15, 17, 19, 20, 29, 36, 46, 53, 54, 56–58, 63, 64, 72–79, 81, 84, 95–97, 99, 100, 102, 103, 120, 121, 145, 146, 148, 150 Lara, O.D., 18 Larson, K., 14 Leblanc, E., 14 Lebres, A.S., 18 LeDuc, B., 6 Lee, C.Y., 47 Lee, H.G., 31, 33 Lee, J.B., 18 Lee, S.-W., 18 Lee, S.-Y., 15 Lee, Y.-K., 15 Leonov, V., 34 Lester, J., 16 Le, T., 58, 74, 96, 121 Li, F., 39, 63, 64, 73 Li, G., 18 Li, J., 64 Lim, Y.P., 102, 126 Lin, Q., 9, 56–58, 121, 148 Lin, S., 34 Lin, Y.C., 102, 126 Li, R.T., 3 Li, T., 97, 147 Liu, J., 4, 34, 56, 58, 114, 121 Liu, J.J., 6 Liu, Y., 8, 9, 33 Liu, Z., 18 Li, X.-L., 18 Li, Y., 5, 65, 97 Lohr, C., 6 Lombardo, G., 147 Lord, S.R., 111, 137 Lucas, R., 4

157 Lu, H., 16, 18 Lukowicz, M., 4 Lukowicz, P., 6 Luo, C., 64 Luo, J., 39, 63, 64, 73 Luo, R., 33

M MacDonald, B., 149 Ma, D., 46, 57, 58, 63, 64, 72–75, 77, 79, 95–97, 103, 121, 145, 146, 148, 150 Magno, M., 33, 76 Mahapatra, K., 47 Manjarres, J., 56 Mao, B.H., 103, 128 Mao, X., 65 Marchegiani, L., 98, 124 Mariani, B., 4 Martínez-Vizcaíno, V., 13 Martín, H., 16, 18 Martin, P., 54, 59 Marukatat, S., 16, 18 Mason, A.E., 120 Maswadi, K., 18 Mataric, M., 149 Matsui, S., 102, 126 Mayer, P., 33 McCarthy, M.W., 18 McKeown, A., 114 McLaws, M.L., 59 Mei, Y., 73 Melanson, E., 4 Menasalvas, E., 18 Mendonca, F., 13 Meneghini, M., 58 Menz, H.B., 111, 137 Micucci, D., 20 Mihailidis, A., 13 Millán, J.d.R., 20 Miluzzo, E., 18 Ming, Z., 64 Minka, T.P., 103, 126 Misra, A., 16 Mitcheson, P.D., 33 Mobilio, M., 20 Mohamed, A., 3 Mohammed, S., 18 Mohanty, S.P., 3 Moheimani, S., 30 Mohsen, S., 18 Mokaya, F., 4 Monger, E., 4

158 Moons, B., 98, 124 Moore, S.A., 18, 20 Moradi, B., 146 Mora, J.C., 13 Mordonini, M., 147 Morgado-Dias, F., 13 Mortazavi, B.J., 149 Mostafa, S.S., 13 Mota, J., 13 Mottola, L., 35, 46, 58, 63, 111, 137 Mouapi, A., 34 Mouftah, H.T., 18 Mujadin, A., 33 Muncuk, U., 34 Mun, M., 16 Mursalin, M., 102, 126 Musolesi, M., 18 Mutlu, O.C., 14 Myers, A., 34

N Nakamura, Y., 54, 58, 63, 64, 75, 76, 78, 97, 121, 145, 148, 150 Nalepa, G.J., 13 Napoletano, P., 20 Nathan, V., 149 Nauroze, S.A., 33, 46 Naveed, H., 18 Neenu, V., 34 Neufelt, C., 6 Nguyen, D.T., 97, 121 Nguyen, M.N., 18 Nguyen, S.M., 6 Nguyen, T., 4, 6, 7, 9, 15, 20, 99, 120 Ning, C., 5 Niotaki, K., 29–31 Niu, L., 149 Noh, H.Y., 4 Nurmi, P., 40, 84, 96, 102, 126

O Oliveira, O.N.Jr., 13 Oneto, L., 20 Ordónez, F.J., 14 Otoum, S., 18 Oukhellou, L., 18

P Panda, S.K., 34 Pandy, M.G., 102, 126 Panero, E., 18

Author Index Pan, W., 16 Pardo, M., 56 Park, G., 73 Park, J.W., 97 Park, S., 33 Parra Perez, X., 20 Paskov, K., 14 Patel, R.A., 97 Paul, S., 5, 149 Paulovich, F.V., 13 Peng, S., 57 Penhaker, M., 34 Pesch, D., 6 Peter, C., 16 Peter, L., 34 Peterson, R., 4, 18 Petrich, W., 120 Phua, C., 18 Piechocki, R., 98, 124 Pirkl, G., 4 Pomares, H., 20 Pope, J., 98, 124 Poria, S., 64 Portmann, M., 3, 21 Pottie, G.J., 18 Pourhomayoun, M., 6 Pozo, B., 35 Pradhan, S., 59 Prentow, T.S., 102, 126 Prinsen, S.K., 6 Prioleau, T., 149 Priyantha, B., 4 Proto, A., 34 Puldon, K., 120 Putra, P.U., 14

Q Qadir, J., 15 Qirtas, M.M., 6 Qiu, L., 59

R Rafiq, J.I., 18 Raghunathan, V., 7, 8, 46, 121 Rahman, M., 149 Rahmati, A., 120 Ramadhan, A.J., 120 Ram, S.K., 47 Rangarajan, S., 59 Rao, A., 120 Rao, G.K., 33

Author Index Rasheed, M.B., 18 Ravelo-García, A.G., 13 Reddy, S., 16 Redondo-Tébar, A., 13 Redouté, J.M., 33 Reginster, J.Y., 5, 6 Rehman, S., 15 Reiss, A., 20 Revadigar, G., 149 Reyes Ortiz, J.L., 20 Rezaie, H., 98, 124 Riz, M., 6 Robinson, H., 149 Rochat, S., 4 Roggen, D., 20 Rojas, I., 20 Röning, J., 16 Rosati, S., 18 Roseway, A., 4 Ross, B.C., 102, 126, 127 Rowbottom, J.R., 3 Rowlands, D.D., 18 Ruan, T., 34, 113, 141 Ruiz-Hermosa, A., 13 Runge, R., 3 Russell, D.M., 102, 126 Ryan, C., 13

S Sabharwal, A., 149 Sablowski, C., 16 Saez, A., 20 Saez, Y., 146 Safaei, M., 58 Sagha, H., 20 Sahoo, S.R., 47 Sahu, A.K., 147 Samara, W., 18 Samijayani, O.N., 33 Sánchez-López, M., 13 Sánchez-Sinencio, E., 35 Sanchis, A., 14 Sandhu, M.M., 3, 21, 114 Sanhaji, F., 18 Saponas, T., 16 Saraereh, O.A., 18 Sarin, S., 6 Sarode, J.D., 34 Sarrafzadeh, M., 6, 55, 58, 64, 74, 96, 121 Satori, H., 18 Satori, K., 18 Sawchuk, A.A., 20

159 Sazonov, E., 4 Scalise, L., 4 Scharfschwerdt, P., 6 Schauer, T., 6 Schindhelm, C.K., 16 Schmandt, C., 4 Schmid, M., 34 Schmidtke, H.R., 16 Schmitter-Edgecombe, M., 18 Schnyer, D.M., 6 Scholten, H., 16, 20 Scholz, S.G., 18 Schöning, J., 84 Searson, P.C., 4 Seel, T., 6 Seelye, A.M., 18 Seneviratne, A., 4, 6, 7, 9, 15, 17, 19–21, 29, 36, 53–58, 58, 64, 75, 76, 78, 79, 81, 84, 95, 96, 99, 100, 102, 120, 121, 145, 148, 150 Seneviratne, S., 4, 6, 7, 9, 15, 20, 99, 120 Sen, S., 97, 121 Seshadri, D.R., 3 Shafique, M., 15 Shahbazi, B., 6 Shahzad, M., 97 Shakya, S.R., 146 Shan, S., 65 Sharma, H., 18, 29, 98, 123 Sheng, F., 5 Shima, K., 14 Shimatani, K., 14 Shirbandi, A., 146 Shoaib, M., 16, 20 Shumake, J.D, 6 Siddiqi, A., 18 Siddiqi, M.H., 18 Sigrist, L., 76 Siirtola, P., 16 Sinthamani, S., 47 Skinner, J., 4 Slomian, J., 5, 6 Smarr, B.L., 120 Smith, P.J., 30 Smith, V., 147 Snader, R., 149 Snyder, M., 3 Sodano, H.A., 73 Soleiman, A., 58, 63 Sommer, P., 114 Sonne, T., 102, 126 Sorber, J., 7, 149 Sousa, P.A., 18

160 Sparacino, G., 6 Srivastava, M., 7, 8, 16, 46, 54, 59, 121 Stisen, A., 102, 126 Stockham, N., 14 Stockx, T., 84 Stricker, D., 20 Stuckenschmidt, H., 120 Sudeendra, K., 47 Sudevalayam, S., 46 Sudharsanam, S., 47 Sundaresan, K., 59 Suri, R.E., 6 Suwa, H., 54, 58, 63, 64, 75, 76, 78, 97, 121, 145, 148, 150 Swibas, T., 4 Syed, A.A., 35, 46, 111, 137 Sztyler, T., 120

T Taati, B., 13 Taj-Eldin, M., 13 Talary, M.S., 6 Talwalkar, A., 147 Tang, L., 39, 63, 64, 73 Tan, Y.K., 34 Tapia, E.M., 14 Tarkoma, S., 40, 84, 96, 102, 126 Tentzeris, M.M., 29–31, 33, 46 Thiele, L., 76 Thiemjarus, S., 16, 18 Thilakarathna, K., 4, 6, 7, 9, 15, 20, 99, 120 Thomas, B.L., 16, 18 Thomaz, E., 6 Thompson, J.E., 6 Touceda, D.S., 149 Tröster, G., 6, 20 Tufail, A., 16 Tunçel, O., 120

U Uddin, A., 46, 58, 64, 97, 103, 121 Uddin, M.Z., 3 Ugulino, W., 4 Umay, I., 5 Umetsu, Y., 54, 58, 63, 64, 75, 76, 78, 97, 121, 145, 148, 150 Upama, M.B., 46, 58, 64, 97, 103, 121 Urban, B., 16 Ustev, Y.E., 18

V

Author Index Vahidnia, R., 120 Vaizman, Y., 20 Vala, D., 34 Valencia, W.M., 13 Vapnik, V., 84 Varshney, A., 58, 63 Veeraraghavan, A., 149 Velez, F.J., 18 Venkatasubramanian, K.K., 149 Venkatnarayan, R.H., 97 Verhelst, M., 98, 124 Villalonga, C., 20 Voigt, T., 58, 63 Voos, J.E., 3 Vo, Q.V., 16 Vyas, R., 29–31

W Wall, D., 14 Wang, A., 98, 124 Wang, C., 33 Wang, C.H., 57 Wang, H., 18 Wang, J., 59, 64 Wang, W., 73 Wang, W.H., 128 Wang, W.Y., 103 Wang, X., 13 Wang, Y., 15, 97 Wang, Z., 18 Wang, Z.L., 5 Ward, J., 6 Ware, A., 18 Washington, P., 14 Weber, S., 6 Wei, C., 5 Weiss, G.M., 18, 20 Wen, H., 15 Westcott, D., 114 Weston, J., 84 White, E.B., 6 Woo, W.L., 15 Wu, C., 6 Wu, J., 18 Wu, T., 33 Wu, Y., 56, 146

X Xiang, T., 39, 63, 64, 73 Xiong, K., 34 Xu, C., 4, 18

Author Index Xu, D., 4 Xu, R., 97, 121 Xu, W., 9, 56–58, 63, 64, 72–75, 77, 79, 95, 96, 98, 103, 121, 124, 145, 146, 148– 150 Xu, Z., 34 Y Yahampath, P., 18 Yahya, K., 18 Yamazaki, K., 16 Yan, Z., 16 Yanco, H., 149 Yang, D., 15 Yang, H., 33 Yang, J., 16, 18 Yang, X.D., 97 Yang, Y., 39, 63, 64, 73 Yasaratna, R., 18 Yeatman, E.M., 33 Ye, B., 13 Yetisen, A.K., 120, 147 Young, T., 65 Youssef, M., 46, 58, 64, 97, 103, 121 Yu, H., 15 Yuce, M.R., 30, 33 Z Zacharia, B.E., 5, 6

161 Zafeiridi, E., 6 Zanon, M., 6 Zarepour, E., 59 Zeadally, S., 149 Zelenika, S., 33 Zeng, J., 65 Zeng, Z., 35 Zhang, B., 5 Zhang, C., 146 Zhang, D., 97 Zhang, M., 20 Zhang, P., 4 Zhang, Q., 73 Zhang, S., 97, 121 Zhang, X., 8, 9, 18 Zhang, Y., 5, 15, 97, 102, 126 Zhao, J., 65 Zhao, K., 114 Zhao, L., 39, 63, 64, 73 Zhao, Y., 97, 121 Zheng, X., 18 Zhong, Z., 34 Zhou, H., 34 Zhou, J., 34 Zhou, X., 97 Zhou, Z., 146 Zhu, M., 34, 113, 141 Zhu, Y., 30 Zimmerling, M., 31, 71 Zorman, C.A., 3

Subject Index

A Accelerometer, 5 Accessories, 4 Accuracy, 19 Acquisition Power Ratio (APR), 90 AC voltage is enveloped, 79 Aggregated global model, 147 Alternating polarity, 76 Ambient sensors, 14 Analogto- Digital Converter (ADC), 35 Audio signal detection, 56 Authentication and key generation, 58 Authentication protocols, 149

B Backscattered phase, 59 Batteries, 7 BLE packet, 111 BLE wireless communication protocol, 103 Bluetooth communication, 33

C Capacitor charging curve, 57 Capacitor voltage envelops, 74 Capacitors, 7 Cattle monitoring, 150 Chest wearable transducer, 58 Classification, 18 Cloud-based classification, 97 Communication receiver, 146 Compatibility, 149 Complement, 102

Confidence interval, 84 Confidentiality and integrity, 149 Confusion matrix, 19 Controlled environment, 49 Converter-based design, 39 Converter-less design, 38 Correlation based feature selection, 103 Costs, 90

D Data acquisition, 15 Dataloggers, 40 Data/user privacy, 147 DC-DC boost converter, 31 Decoupling of the transducer, 39 Decrypt, 149 Deep learning models, 146 Deep sleep mode, 138 Depletable energy source, 35 Diode, 38 Discharge the capacitors manually, 73 Distinct signatures, 54 Distorts the generated AC signal, 72 Diverse environments, 134 Diversity of light sources, 104 Dominant feature, 83 Duty cycling, 21 Dynamically track, 42 Dynamic characteristics, 33 Dynamic load, 57

E Electrical loading, 37

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. M. Sandhu et al., Self-Powered Internet of Things, Green Energy and Technology, https://doi.org/10.1007/978-3-031-27685-9

163

164 Electromagnetic, 29 Electrostatic, 29 Encryption, 149 End-to-end activity recognition, 9 End-to-end HAR, 138 Energy burst, 73 Energy harvesters as information sources sources, 53 Energy negative sensors, 74 Energy Neutral Operation (ENO), 64 Energy Storage Unit (ESU), 31 Energy-positive, 10 Energy-positive sensors, 74 Envelope distortion, 81 Environment-agnostic, 109 Environmental sensors, 6 E-patches, 4 Equal sized windows, 84 E-tattooes, 4 E-textiles, 4 Excitation of the transducer, 79 External sensors, 14

F F1 score, 19 Feature extraction, 17 Feature-level fusion, 127 Feature selection techniques, 17 Federated learning, 147 Filtering algorithm, 73 Fine grained, 108 Fine grained variations, 20 Finger gesture recognition, 97 Fitbit, 6 Food intake and eating habits, 96 Full-wave bridge rectifier, 31

G Gait recognition, 57 Generalisable model, 142

Subject Index High fidelity signal, 15 High source impedance, 74 Hotwords, 56 Human daily energy expenditure, 58 Human knee-joints, 34 I Implantable, 5 Inductor, 39 Interference problem, 74 Intermittent execution, 38, 76 IV curves, 42 K KEH-based necklace, 58 L Leave-one-user-out CV, 103 Light-weight machine learning, 147 Liquid level detection, 97 Localization, 63 Long range communication, 146 Low duty cycle, 64 M Machine learning algorithm, 14 Mechanical tuning, 37 Metal Oxide Semiconductor Field Effect Transistor (MOSFET), 39 Metrics, 19 Microcontroller, 35 Microphone-based decoder, 146 MIMO communication, 146 Moving average filter, 15 MPP tracking, 33 Multi-source energy harvesters, 121, 148 Mutual information, 102 N Natural noise filter, 56

H Hand crafted features, 146 Hand gestures, 58 HAR pipeline, 98 HAR power ratio, 113 Harvest-and-store, 31 Harvest-and-use, 31 Heterogeneous data, 147 Higher resolution, 108 High-fidelity fused signal, 122

O Obstacles, 120 Offline, 64 Opportunistic communication, 21 Orientation of the solar surface, 97 Oscilloscope, 112 Overlap, 102 Overlapped, 17

Subject Index

165

P Peak-based features, 102 Personalised AI models, 147 Pervasive deployment, 7 Photocurrent, 46 Photodiode array, 97 Piezoelectric, 5 Piezoelectric bending transducer, 78 Piezo-resistive sensors, 57 PIR sensor, 6 Power-hungry sensors, 53 Power Management Unit (PMU), 30 Precision, 19 Preprocessed, 15 Principal component analysis, 102 Privacy-protection interfaces and tools, 149 Proliferation, 35 Proxy, 121 Publicly available datasets, 20

Simultaneous source, 9 Sine wave, 40 Sliding windows, 16 Smart watches, 6 Solar cells, 7 Solar module, 99 Source meter, 41 Specificity, 19 Stable 1-lb load shaker, 40 Standard error of estimate, 19 Static activities, 111 Static and sedentary activities, 34 Step count, 55 Stop periods, 102 Supervised, 18 Sustainable operation, 33 Switching losses, 45 Synthetic Minority Over-sampling Technique (SMOTE), 103

Q Quiescent current, 45

T 10-fold CV, 103 Thermoelectric Power Generator (TEG), 30

R Radio Frequency Identification (RFID), 29 Real-time activity monitoring, 148 Recall, 19 Reduced feature set, 103 Regression, 18 Regulate the voltage, 77 Remote detection of chemical reactions, 59 Resonance frequency, 36 Resonant frequency, 88 Resource-constrained edge IoT devices, 147 Robustness, 109 Room-level place recognition, 58 S Segmentation, 16 Self-powered IoT, 35 Sensing points, 75 Sensitive, 107 Sensitivity, 19 Sensor-related power saving, 55 Shadowing, 102 Shunt ampere-meter, 89 Signal acquisition circuit, 73 Signal fusion, 9 Signal sampling rate, 21 Simultaneous sensing and energy harvesting, 71

Thermopile, 34 Thin-film photovoltaic panel, 33 Time- and frequency-domain features, 17 Time windows, 16 Tip mass, 36 Touch sensing, 58 Train and test sets, 18 Transceiver, 35 Transport mode detection, 57 Tricycle, 78, 79 Turn-off threshold, 37 Turn-on threshold, 37

U Ubiquitous Sensing, 13 Uncontrolled delay, 73 Undersampling, 42 Uninterrupted operation, 9 Univariate, 103 Unreliable harvested energy, 64 Unsupervised, 18 Untuned KEH, 95

W Water flow, 59 Waveform generator, 40 Wireless link, 76