Workshops Proceedings of the 5th International Conference on Intelligent Environments [1 ed.] 9781607504641, 9781607500568

Citation preview

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

WORKSHOPS PROCEEDINGS OF THE 5TH INTERNATIONAL CONFERENCE ON INTELLIGENT ENVIRONMENTS

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Ambient Intelligence and Smart Environments The Ambient Intelligence and Smart Environments (AISE) book series presents the latest research results in the theory and practice, analysis and design, implementation, application and experience of Ambient Intelligence (AmI) and Smart Environments (SmE). Coordinating Series Editor: Juan Carlos Augusto Series Editors: Emile Aarts, Hamid Aghajan, Michael Berger, Vic Callaghan, Diane Cook, Sajal Das, Anind Dey, Sylvain Giroux, Pertti Huuskonen, Jadwiga Indulska, Achilles Kameas, Peter Mikulecký, Daniel Shapiro, Toshiyo Tamura, Michael Weber

Volume 4 Recently published in this series Vol. 3. Vol. 2.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Vol. 1.

B. Gottfried and H. Aghajan (Eds.), Behaviour Monitoring and Interpretation – BMI – Smart Environments V. Callaghan et al. (Eds.), Intelligent Environments 2009 – Proceedings of the 5th International Conference on Intelligent Environments: Barcelona 2009 P. Mikulecký et al. (Eds.), Ambient Intelligence Perspectives – Selected Papers from the First International Ambient Intelligence Forum 2008

ISSN 1875-4163

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments Edited by

Michael Schneider DFKI, Germany

Alexander Kröner DFKI, Germany

Julio C. Encinas Alvarado University of Castilla-La Mancha, Spain

Andres García Higuera University Castilla la Mancha, Spain

Juan C. Augusto University of Ulster, UK

Diane J. Cook Washington State University, USA

Veikko Ikonen VTT, Finland

Pavel Čech University of Hradec Králové, Czech Republic

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Peter Mikulecký University of Hradec Králové, Czech Republic

Achilles Kameas University of Patras, Greece

and

Vic Callaghan University of Essex, UK

Amsterdam • Berlin • Tokyo • Washington, DC

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

© 2009 The authors and IOS Press. All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher. ISBN 978-1-60750-056-8 Library of Congress Control Number: 2009936332 Publisher IOS Press BV Nieuwe Hemweg 6B 1013 BG Amsterdam Netherlands fax: +31 20 687 0019 e-mail: [email protected]

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Distributor in the USA and Canada IOS Press, Inc. 4502 Rachael Manor Drive Fairfax, VA 22032 USA fax: +1 703 323 3668 e-mail: [email protected]

LEGAL NOTICE The publisher is not responsible for the use which might be made of the following information. PRINTED IN THE NETHERLANDS

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved.

v

Preface

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Workshops Proceedings of the 5th Int. Conf. on Intelligent Environments Barcelona, Spain, 19th of July, 2009 We are witnessing an historic technological revolution as computing reaches maturity to become immersed in our daily life to an extent that some decades ago was considered science fiction. Advances in the engineering of sensing and acting capabilities distributed in wide range of specialized devices is providing at last an opportunity for the fundamental advances that computer science achieved in the past few decades to make an impact in our daily lives. This technical confluence is matched by a unique historical context where users are better informed (and more aware of the benefits that technology can provide) and production of more complex systems is becoming more affordable. Sensors/actuators deployed in an environment (in this context it can be any physical space like a house, an office, a classroom, a car, a street, etc.) facilitate a link between an automated decision making system connected to that technologically enriched space. This computing empowered environment enables the provision of an intelligent environment, i.e., “a digital environment that proactively, but sensibly, supports people in their daily lives”. This is a very active area of research which is attracting an increasing number of professionals (both in academy and industry) worldwide. The prestigious 5th International Conference on Intelligent Environments (IE’09) is focused in the development of advanced Intelligent Environments and stimulates the discussion on several specific topics which are crucial to the future of the area. As part of that effort to stimulate developing in critically important areas five workshops were supported as part of IE’09. This volume is the combined proceedings of those five workshops: The Workshop on Digital Object Memories (DOMe’09) aims to bring together technical experts, artists, designers, and potential end-users of Digital Object Memories to explore the technical, social, privacy, and legal implications of digital object memory systems, to establish a common view on the underpinning requirements to digital memories, and to leverage cooperation in future activities. Digital Object Memories comprise hardware and software components that physically and/or conceptually associate structured and unstructured digital information with real-world objects in an applicationindependent manner. If constantly updated, the over time Digital Object Memories provide a meaningful record of an object’s history and use. From a technical point of view, Digital Object Memories provide an open-loop infrastructure for the exchange of objectrelated information across application and environment boundaries, and as such allow for novel classes of applications in which rich object histories are created and exploited. From the user’s point of view, Digital Object Memories create a new design space for everyday interactions and our relationship with physical artefacts. Digital Objects could become sites for their owners’ personal stories, but also afford people the opportunity to

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

vi

explore an object’s provenance and connections to other elements of physical and digital life. The Workshop on RFID Technology: Concepts, Practices and Solutions (RFID’09). Bar codes technology has taken a leading role for more than 30 years in the identification field and RFID will soon replace it world wide in many applications. RFID offers promising opportunities for businesses (manufacturing, transport, logistics, pharmaceuticals, health, agriculture, security and more) and society, due to their power to report their location, identity and history. The unavoidable advance of RFID technology will have tremendous advantages and provide us with good applications in different areas. A few years ago, applications based on RFID systems were hardly used internally inside some companies. But, nowadays advances in electronics, wireless communication and networking architecture have facilitated the implementation of real deployments and commercial applications of systems based on RFID. RFID technology allows for the automation of data capturing along the supply chain. This makes product tracking and process control very dependable and accurate. Such reliability and accuracy is made possible because identification technology assigns a unique identification number to every single object (EPC: Electronic Product Code). Then an advanced data management system uses this number to allow access to the associated information in real time. A small radiofrequency tag is attached to the object to host its identification number. Thus, RF is used to read the tags (or write them in some cases) and the internet can be used to access data bases with the related information. This workshop should bring together researchers, engineers and practitioners interested in the advances and business applications of RFID. The Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’09) aims at stimulating the development of human-like effectiveness within the artificial systems that provides support to humans. The event is not focused on a specific application area, although it welcomes reports on applications given the value to inform the community with regards to solutions for specific cases and to extrapolate strategies across areas. The overall emphasis is providing a forum where to analyze the possibilities that Artificial Intelligence has to make smart environments smarter. Learning, reasoning, adaptation, user preferences and needs discovery, sensible interaction with users, and many other topics form the regular agenda of this event. The content of this section includes the abstract of one keynote speaker, and papers accepted for oral and poster presentations. This section has two focused subsections on: “Mobile Robots in Automated Building Systems”, organized by Dr. A. Sgorbissa and “Intelligent Environments Supporting Healthcare and Well-being”, organized by Dr. J. O’Donoghue. All these contributions come from recognized professionals in the area which are reporting on their latest reflections and achievements in the problem of improving the decision making capabilities of intelligent environments. The Workshop on Ethical Design of Ambient Intelligence (EDAmI’09) presents research and studies that have tackled with issues related to new technologies, ethics and user experience of ambient intelligence. One of the goals of the workshop is to focus on ethical guidelines that have been produced especially for the design of new technologies. Since Ambient Intelligence applications based on information and communication technology embedded in our environment and everyday objects will clearly raise several ethical issues it seems very timely to have a workshop dedicated to this issue. While the technical design of smart environments is still quite challenging, it seems that it could be even more challenging to do social design in this very broad area of research and

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

vii

development. The Workshop is organised by the Ethical Issues of Emerging Technologies (ETICA) and Micro-Nano integrated platform for transverse Ambient Intelligence applications (Minami) projects, which are looking at new technologies and subsequent arising ethical issues at both practical and theoretical levels. Co-chairs of the workshop come from both projects and include Veikko Ikonen, Catherine Flick, Philippe Goujon, Eija Kaasinen, Bernd Stahl and Kutoma Wakunuma who have all put lots of effort into reviewing all the contributions to ensure high quality papers to a broad audience of researchers, developers and designers. The papers of the workshop offer good state of the art view to the ethical design of Ambient Intelligence. Some of the papers are focused on practical issues of Ambience Intelligence while others look at the issue from a more theoretical perspective. The Workshop on Smart Offices and Other Workplaces (SOOW’09) concentrates on intelligently helpful working environment that ensures broad but focused and personalized access to relevant information and knowledge resources, supporting thus both learning needs of the managers as well as their decision making activities. The area of smart offices and other workplaces covers naturally technical point of view, however, the workshop extends to much broader but related topics. Thus the contributions to the workshop range from autonomous multimedia gathering for meetings summaries, through contextual filtering of social interactions to exercise in the smart workplace. All accepted contributions in this section present an interesting aspect of the smart office and intelligent workplace domain and open space for further discussion hereby fulfilling the intent of the workshop. As a result of the content focus of those events described above, this volume offers you a glance of the latest developments in key areas of the development of Intelligent Environments. It compiles the latest research done by active researchers in the area working to push ahead the boundaries of science and focused on achieving the deployment of intelligent environments in the real world. The effort of this professionals will influence the way we leave tomorrow’s world. We hope you enjoy as a reader the content of this volume as much as the attendees of these workshops enjoyed the live presentation of the papers and the thought provoking discussions emanating from them. The co-editors of this volume want to thank all the people that facilitated the realization of each one of these events: the remaining co-chairs of the workshops, the members of their Program Committees, which facilitated the review of papers, the external reviewers which also contributed to that task, and the conference organizers which provided a supportive environment for the realization of these events. July 2009 Michael Schneider, Alexander Kröner, Patrick Olivier, and Peter Stephan DOMe’09 Julio C. Encinas and Andrés García Higuera RFID’09 Juan Carlos Augusto, Diane Cook, John O’Donoghue, and Antonio Sgorbissa AITAmI’09 Veikko Ikonen, C. Flick, P. Gojoun, E. Kaasinen, B. Stahl, and K. Wakunuma EDAmI’09 ˇ Peter Mikulecký, Pavel Cech, and Carlos Ramos SOOW’09

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

ix

Contents Preface Michael Schneider, Alexander Kröner, Patrick Olivier, Peter Stephan, Julio C. Encinas, Andrés García Higuera, Juan Carlos Augusto, Diane Cook, John O’Donoghue, Antonio Sgorbissa, Veikko Ikonen, C. Flick, P. Gojoun, E. Kaasinen, B. Stahl, K. Wakunuma, Peter Mikulecký, Pavel Čech and Carlos Ramos

v

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1st International Workshop on Digital Object Memories (DOMe’09) Storytelling Memories: A Tangible Connection to Bomber Command Veterans Tanya Marriott

3

Imparting Digital Memory Content Through Game Form Candice Lau

9

Memory Baubles and History Tinsels Daniela Petrelli and Ann Light

15

Informing Customers by Means of Digital Product Memories Alexander Kröner, Patrick Gebhard, Lübomira Spassova, Gerrit Kahl and Michael Schmitz

21

An Object Memory Modeling Approach for Product Life Cycle Applications Joerg Neidig and Peter Stephan

27

Exploring the Design of a Memory Model for Smart Objects Daniel Fitton, Fahim Kawsar and Gerd Kortuem

33

A Trace-Based Framework for Supporting Digital Object Memories Lotfi S. Settouti, Yannick Prié, Damien Cram, Pierre-Antoine Champin and Alain Mille

39

Using Object Memory Patterns to Make Plan-Driven Help Systems More Flexible Damien Cram, Alexander Kröner and Alain Mille Querying Sensor Data for Semantic Product Memories Christian Seitz, Christoph Legat and Jörg Neidig

45 51

1st International Workshop on RFID Technology: Concepts, Practices & Solutions (RFID’09) Design of a RFID Based Traceability System in a Slaughterhause A.M. López, E. Pascual, A.M. Salinas, P. Ramos and G. Azuara

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

61

x

Development of a Customized RFID Reader for a Distribution Centre with Agent-Based Control Javier G.-Escribano, Roberto Zangróniz, Andrés García, Julio Cesar Encinas, Javier de las Morenas and José Manuel Pastor

69

Using RFID Information in Routing and Sequencing Warehouse Operations Francisco Ballestín, Pilar Lino, Ángeles Pérez, Sacramento Quintanilla and Vicente Valls

77

Improvements in Supply Chain Tracking Using a Three-Levels RFID System Antonio Abarca, Javier G.-Escribano, Juan J. de Dios and Andrés García

85

Empirical Analysis of a UHF RFID Multi Tag Pallet Using a Handheld Reader Rafael V. Martínez Catalá and Víctor Brito Megías

93

Internet Browsing Through RFID-Powered Gadgets Leire Muguira, Jonathan Ruiz-de-Garibay and Juan Ignacio Vazquez

101

On Building a Web Services-Based Prototype for RFID Applications David Sundaram, Wei Zhou, Selwyn Piramuthu and Schalk Pienaar

109

NFC/RFID Applications in Medicine: Security Challenges and Solutions Antonio J. Jara, Miguel A. Zamora and Antonio F.G. Skarmeta

117

RFID and Real-Time CRM Wei Zhou and Selwyn Piramuthu

125

4th International Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’09)

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Synthetic Training Data Generation for ADL Modeling Dorothy N. Monekosso and Paolo Remagnino An AI-Planning Based Service Composition Architecture for Ambient Intelligence Florian Marquardt and Adelinde M. Uhrmacher

137

145

Adaptive Neuro-Fuzzy Systems for Context Aware Offices Christian G. Quintero M. and Jorhabib Eljaik G.

153

Collaborating Context Reasoners as Basis for Affordable AAL Systems Jan Kresser, Michael Klein, Sebastian Rollwage and Peter Wolf

161

Caring of Old People: Features of Cognitive Technologies Development Amedeo Cesta, Gabriella Cortellessa and Lorenza Tiberio

169

Support for Context-Aware Monitoring in Home Healthcare Alessandra Mileo, Davide Merico and Roberto Bisiani

177

Robotic Furniture in a Smart Environment: The PEIS Table E. Di Lello, A. Loutfi, F. Pecora and A. Saffiotti

185

A Model for the Cooperation Between Robots and Smart Environments Fulvio Mastrogiovanni, Antonio Sgorbissa and Renato Zaccaria

193

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

xi

Genetic Programming Application for User Capabilities Determination Kseniya Zablotskaya

201

Why We Need a User Interface Resource Server for Intelligent Environments Gottfried Zimmermann and Benjamin Wassermann

209

Context-Aware Sensing in Pervasive Computing Environment: Based HMDC with MAS Salahideen M. Alhaj and Linda R. Alnaqeep Feeling Market System: A Rational Support for the Trader Javier Martínez, M. Esther Prada, Ralf Seepold and Natividad Martínez Madrid

217 225

1st International Workshop on Ethical Design of Ambient Intelligence (EDAmI’09) Ethics of Ambience Satinder P. Gill

239

Privacy, Security and Safety in e-Ticketing Michel Arnaud

247

Design and Ethics of Product Impact on User Behavior and Use Practices Steven Dorrestijn

253

Defining Ethical Guidelines for Ambient Intelligence Applications on a Mobile Phone Veikko Ikonen, Eija Kaasinen and Marketta Niemelä

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Moving Towards an Ethical Governance of New Emerging Technologies Alessia Santuccio, Marco Marabelli, Penny Duquenoy, Philippe Goujon, Sylvain Lavelle and Norberto Patrignani Understanding Ethical Issues of Emerging AmI Technologies in Europe (A Framework) Bernd C. Stahl, Simon Rogerson and Kutoma J. Wakunuma

261 269

277

1st International Workshop on Smart Offices and Other Workplaces (SOOW’09) IGAMSAI – Idea Generation with Autonomous Multimedia Gathering for Meetings Summaries in AmI Carlos Filipe Freitas, António Meireles, Lino Figueiredo and Carlos Ramos Workspace Awareness Without Overload: Contextual Filtering of Social Interactions Adrien Joly Notes on Smart Workplaces Peter Mikulecký

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

289

297 305

xii

Adapting Information Displays Contents in Smart Spaces According User Emotions and Personality Eugénia Vinagre, Goreti Marreiros, Carlos Ramos and Lino Figueiredo

313

The Proposal of Intelligent Assistant for Graduation Thesis Preparation Pavel Čech and Kamila Olševičová

321

Exercise in the Smart Workplace M.J. O’Grady, Jie Wan, R. Tynan, G.M.P. O’Hare and C. Muldoon

329

333

Subject Index

339

Author Index

341

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Time Awareness Tool for Enhancing Group Times’ Coordination in the Virtual Workspace Margarida Romero

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

1st International Workshop on Digital Object Memories (DOMe’09) Michael Schneider DFKI, Germany Alexander Kröner DFKI, Germany

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Peter Stephan DFKI, Germany Patrick Olivier Newcastle University, UK

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-3

3

Storytelling Memories: A Tangible Connection to Bomber Command Veterans Tanya Marriott Massey University Institute of Communication Design, New Zealand

Abstract. This paper reports the development of the Storytelling Memories project that formulates an interactive platform of memory immersion and experience within a museum environment. Specifically focusing on the memories of Bomber Command veterans the project utilises a touch sensitive surface as an interface between the viewer and the memories. A physical controller, when placed near the interface surface will “unlock” contained memories, enabling an open-ended storytelling experience. The design encourages the user to interact directly with the memories to create their own dialogue, with the intention of developing a more emotive, personal connection to the Veteran. Keywords. Memories, Storytelling navigation, Interactivity, Tangible Navigation, Human memory access,

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction Storytelling Memories endeavours to bridge the gap between the desire for knowledge and understanding for the museum visitor; and the need to record and make available, memory testimony and artefacts of Bomber Command Veterans. Through research of current museum display trends this paper observes that within a museum environment the technique of display is provided through transference [1]. Personal testimony memories are disregarded in favour of highly visual, calculated displays incorporating collections of contextually detached objects such as medals, aircraft and weapons. The methodology of memory and artefact display has created much debate within museum display scholars, who believe traditional displays marginalise the natural emotive qualities associated to the memory by detaching them from their physical environment and contextual meaning. A more immersive formulation of memory presentation is yet to be consistently integrated into the museum environment, as it requires access to large databases of archived memories in a digitised format, and a more visually and interactively adaptable mode of memory recording and collection in order to build a cohesive and stimulating system of display. Storytelling Memories provides the framework for the user to synthesise a physical relationship with the memories and artefacts, which informs an emotional connection. Memory testimony is a reflective narrative, visually and emotionally rich within the mind of the contributor. It is theorised that if this richness of memory could be provided within a visual context, which is substantiated by the unique situation of each memory, it could foster an engaging understanding and relationship between the user and the memory. [2]

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

4

T. Marriott / Storytelling Memories: A Tangible Connection to Bomber Command Veterans

1. Project Background 1.1. Bomber Command representation within a museum environment

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Storytelling Memories specifically focuses upon the representation of the memories of the men of Bomber Command. During the war they were regarded as heroes, yet as soon as the war ended public opinion started to question whether the aerial bombing of civilian targets was a justified act. As a consequence the portrayal of Bomber Command history within a museum context has often shied away from contentious issues for fear of offending both the veterans and the public, which has left the veterans feeling misunderstood and marginalised by society. The Canadian War museum upset Bomber Command veterans in 2006 with a controversial text panel “An Enduring Controversy” [3] which veterans felt branded them war criminals and has sparked much public debate. Canadian War museum director Dean. F Oliver describes the role of the museum. ‘The museum preserves, educates and remembers, interpreting the effect of military events on the country and its citizens.’ But in this interpretive role, the memory story has become distorted in order to create more impact in the museum display to the detriment of the truth behind the actual memory. Judy Attfield describes the importance of accurate memory portrayal, and it’s importance in the education of future generations “Within the public space of the museum, memories are triggered through people’s real or assumed relationship with the objects, events and images they are witnessing [4] War was a limited but very significant aspect in a veteran’s life, forever changing them. However their memory contribution is larger than just their war years. Their early memories give an insight into the pre-war way of life, and their reasons for joining up. Memories after the war offer a reflective narrative. The veterans, as old men decompartmentalize their thoughts and explain their war years in relation to their postwar experiences. It is important to represent the veterans’ life in a holistic view. Where the public has the opportunity to relate to the veteran on relative terms of human experience. 1.2. Transference of Memory and the method of Loci Objects within a traditional museum structure were displayed either grouped together with other similar objects, or in dioramas based on the objects purpose and use. With little or no appreciation or record of the objects original purpose and existence. Kavanagh [1] describes this mode of memory display as “object centred” which is a form of transference. We shield ourselves from acknowledging our part within history behind inanimate objects. Acknowledging its physicality but not its purpose as related to human reactions and responsibilities, sensitive to the individual the object belonged to. He describes the RAF bomber as not being responsible for aerial bombing but the hierarchy of people involved. From government to Airmen. The method of loci [5] is another system of memory hierarchy that argues an object or memory should not be divorced from environment representation. The object or memory can be recalled if placed within an environment. Memorising the environment layout and structure recalls the placement of the object or memory.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

T. Marriott / Storytelling Memories: A Tangible Connection to Bomber Command Veterans

5

1.3. Memory Collection The presentation of memories begins with the format of recording. From Memory in New Zealand [6] and The memory Project in Canada [7] are two organisations dedicated to the collection and digitisation of memory testimony and artefacts to be made available to the public. However the format of memory collection is often not suitable for museum presentation. Oral history must be kept in its entirety and not adapted or adjusted from its original length and format. One aspect of Storytelling Memories was to address testimony collection methods and propose a different scenario that would provide easier display integration. If an oral history interview drew focus from a location or an object and the interviewee was asked to describe their memories surrounding the location or object, then memories could be tied to the object, giving significance and a clear and concrete tangible existence. For example if the interviewer has been given photos of the Veteran’s home-town, the Veteran could be asked about where they grew up, so the interview would focus memories around tangible artefacts and locations. The objects the oral history references are preserved in the form of a memory box. Shoeboxes stored in Cupboards and under beds containing tangible reminders of the past. In the digitisation process these three-dimensional objects loose their physicality, their ability to be handled, arranged and explored in detail. Often referenced in exhibits as a supplement to dialogue the object plays a vital role in memory presentation.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1.4. Memory Display The memory display within Storytelling Memories focuses on a person-centric and environment-centric context as illustrated in the Churchill Museum [8] touch table and the City of Memory [9] website. Developed by Small Design the Churchill Room display contains a 17 metre long interactive table that documents the life of Winston Churchill. It contains thousands of articles about his remarkable life. Specific months in each year are catalogued in digital manila folders which when touch activated open to reveal information and memories associated to the date. The memories are located within a digital representation of their physical container- the filing cabinet, which it itself is a digital manifestation of the memory box. The mode of interaction draws upon the curious explorative nature of the user, who rifles through the files in the same manner as they would a physical filing cabinet. The memories as well as existing within the context of their relation as a filed object also relate to simulations of pivotal events of the time. This gives the files a placement within history alongside their localised placement as an object. The Churchill museum system is person-centric, in comparison to the environment-centric City of Memory project orchestrated by Jake Barton in New York. The City of Memory project documents the memories of citizens of New York. The web-site documents snippets of memory located within a map environment added as solitary editions or as sequential memories relating to personal journeys. The site lacks a visual display of the memories, the map being the only expression of their tangibility. The origins of the Memory project involved a physical map with locational opinions layered in a collage across the surface. The depth of note-paper indicated hubs of communal narrative, each explored through exploring the depths. This tangible connection is lost within the web-site where the memories become less personal and explorative in their placement.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

6

T. Marriott / Storytelling Memories: A Tangible Connection to Bomber Command Veterans

2. Interaction Techniques 2.1. The Cube The Storytelling memories project uses a tangible cube controller as a simple hierarchical activator to access the veteran’s memories. A separate memory box represents each veteran. The cube is the physical embodiment of the memory box, made of wood, warm and inviting; it simulates the feeling of life. Each of the six faces on the cube corresponds to a different turning point within the veteran’s life as illustrated in Figure 1. When one cube is replaced by the next another set of memories is accessed, specific to each person.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 1: The cube controller relationship to the interface.

2.2. The Table and peripheral screens The table is a physical window to memory presentation. The interface itself is segmented into two sections relating to a different hierarchy of memory. Memory display within the bottom half of the table contains the memory testimony of the veteran. The memories are placed at the closest proximity to the user so they are easily accessible. The second tier of information is contextual or environmental and relates to the locations that are discussed within the veterans’ memories. The third tier located in the peripheral projections; is the most distant from the user. It utilises collected memories which do not belong to any veteran in particular, but which contains information which substantiates veteran testimony and the context of the memories. This third tier might include technical or related stories. The sections that correspond to the icons illustrated on the side of the cube are Childhood and Growing up, Squadron life, Inside the Aircraft, On Operations and After the War. Turning the cube activates a new interface. A digital representation of an old drawer contains images and artefacts, reminiscent of mementos packaged away in an old shoebox under the bed. Each image utilizes a set of tools, a magnifying glass for closer inspection, and an old bakelite switch that controls sound. Set into the drawer is a digital window into

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

T. Marriott / Storytelling Memories: A Tangible Connection to Bomber Command Veterans

7

the past; a detailed visual with subtle movements that narrates the environment relating to the memories as illustrated in Figure 2. 2.3. Implementation

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Storytelling Memories utilizes a collection of interactive hardware and software product in an innovative new configuration. The touch table is contained seamlessly within a physical display table. The cube controller is an independent tangible object, and is the head of the navigation tree. The cube sits beside the table in a separate subtle docking system. Each face in the cube also contains a unique RFID tag (Radio Frequency Identification). When the cube is brought into proximity to the table, the RFID tag communicates with the RFID reader located in the tabletop, which tells the computer software to change the interface screen file. When the cube is placed in the dock the top-facing icon indicates which section of the interface is opened within the table. An Arduino [10] physical computing device, provides the link between the physical and the digital environment.

Figure 2: The final integrated system

3. Conclusion The change in museum display techniques, and the implementation of digital opportunities provides a platform of research for this project. It is envisioned that this

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

8

T. Marriott / Storytelling Memories: A Tangible Connection to Bomber Command Veterans

prototype can provide Veteran testimony to the public arena, while also staying true to Veteran sensitivities towards portraying their war memoirs in an accurate and honest context. The tangible connection to veteran memories is illustrated through the use of a physical memory box as the key access point to the interface. The head of the navigation tree each cube represents an individual veteran. Each cube face represents a differing aspect in his life, with equal importance. This paper also illustrates the depth of research conducted into new methodologies of memory collection, storage and presentation within a museum environment. Study has indicated that in order to give an immersive emotive experience of any historical account, the background architecture and context is as important for the memory navigation as the memories themselves. This architecture gives the memories placement and adds further justification and detail to back up the stories conveyed through the memories. The project differentiates between the personal memories and the background “facts” through separate presentation and navigation techniques. Storytelling Memories encourages museum users to explore the memory narrative in depth and understand the viewpoint of the veteran through a structured hierarchy of navigation and the tangible symbolism of the memory box.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. References [1]

Kavanagh, G. (2000) Dreamspaces. Continuum International Publishing Group

[2]

Ashplant, TG, Dawson, G, Roper, M, (2000) The Politics of War Memory and Commemoration, Routlege

[3]

Oliver, DF, (2007, March 22) The War Museum and the Strategic bombing campaign (published op-ed). Retrieved 14th February 2009 from www.communities.canada.com/ottawacitizen/forum

[4]

Attfield, J (2000) Wild things: The material culture of everyday life, Berg Publishers

[5]

Yates, F (1992) The Art of Memory/Frances A.Yates, London:Pimilico

[6]

New Zealand Ministy of Culture and Heritage. (2008) From History- War oral history programme. 2008 Retrieved 13th August 2008 from www.nzhistory.net.nz

[7]

The Dominion Institute, Canada (2008) The Memory Project.2008. Retrieved 10th September 2008 from www.thememoryproject.com

[8]

Kabat, J, The Audience takes over, The people will be heard: Interactive technology in public spaces. Retrieved November 30, 2008 from www.adobe.com/designcenter/thinktank/jenkabat/

[9]

Krygier, J: (2006) Jake Barton’s Performance Maps: An Essay, Cartographic Perspectives, Number 53, Winter 2006. Page 41-50

[10] Arduino (2008) Arduino physical computing and RFID system Retrieved 8th August 2008 from www.arduino.cc/en

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-9

9

Imparting Digital Memory Content Through Game Form Candice Lau Masters of Design by Research, University of Technology, Sydney, Australia Abstract. Taking on the role as an advocate for maintaining memories of displacement and migration for the Estonians, ‘Our New Home’, the Estonian Exhibition in 2007 and 2008 at the Powerhouse Museum of Design and Technology in Sydney, explored the narrative of the journey traversed by these individuals through a series of videos and a collection of artifacts. The objective of this paper is to present a concept of a follow-up exhibition. The focus will be the use of a single object from the Estonian community as a physical interface to open up digital content within a museum space that is augmented by technology. This concept aims to create a more experiential exhibition through visitors’ participation. Keywords. Estonia, oral history, digital memory, interaction, computer-augmented environment

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction In the midst of the historical social/political accounts of the Soviet Occupation, and World War II in Estonia, a Baltic State bordering Latvia and Russia, come some of the most fascinating personal recollections. However, these stories are often displaced within the realms of the Estonian peoples’ memories, and the social framework to which these memories are validated also decay as the living members who uphold them pass on [1]. With a didactic approach, the Estonian Exhibition not only attempted to educate and inform visitors of the lives of this specific cultural community in Australia, but also to capture and help maintain these memories before they disappear. Bearing in mind that this is one of the first public shows that explored this community in Australia, the two-corridor space dedicated to this exhibition provided a short overview of the lives and times from 1900 till today. This project aims to create a follow up exhibition as an extension to ‘Our New Home’ and provide a deeper insight into the Estonian people. This project is based on a larger framework that explores the notion of objects as a trigger for memories. It also aims to move away from the didactic approach, to a more experiential and participatory method through enabling interactive technology. With a practiced based approach, it draws from collective memory, interaction and game theories, and a series of empirical research activities to drive the design proposal.

Figure 1: ‘Our New Home’, Estonian Exhibition at the Powerhouse Museum, Sydney Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

10

C. Lau / Imparting Digital Memory Content Through Game Form

1. Theoretical and Conceptual Framework 1.1. Theoretical Framework Moving beyond the common perception that museums have left objects in the ‘process of dying’, as Misztal eloquently puts it, this work takes the object beyond the glass case and promotes physical interaction [2]. Therefore, the value of the object reverts back from the status of an object, as opposed to an artifact within a museum. Susan Pearce exclaims that ‘the meaning of the object lies not wholly in the piece itself, nor wholly in its realisation, but somewhere in between the two.’[3] As such, not only the physical and tangible traces that are scratched onto the object bring about its significance, but also the intangible, intimate and intrinsic process of realisation of each individual. This project examines this dichotomy of the object with its explicit values, and the intrinsic process of being a trigger of memories in order to create meaning.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1.2. Conceptual Framework ‘Our New Home’ addressed three key themes: ‘Estonian Culture and Identity’, ‘Invasion and Dispossession during WWII’, and the ‘Estonian Immigrants in Australia’. The themes were divided into three main parts of the exhibition space. Amongst the collection from the second theme, was a set of Estonian playing cards that were selected to compliment enabling interactive technology to create a computeraugmented environment for the follow up exhibition. These playing cards, emblazoned with graphics that connote the traditional dress of the Estonian people, were produced in the first Republic of Estonia between 1918 and 1940. Historically and culturally, they signify the nationalism of the Estonian people, a mark of their triumph over adversity in the War of Independence in 1918 to Figure 2: Estonian Playing Cards 1920, and a product of their rapid prosperity throughout the republic. This is the intangible signification of the object. Conversely, in a much wider context, they are more closely related to tangibility and tactility that is found in game form. In ‘Our New Home’, this unique and commemorative set of playing cards were glass cased and this gave the sense of distance and inaccessibility. This is quite contrary to the common association of play with playing cards.

2. Design Concept The design concept is framed around the issue of significance, (in)tangibility, and engagement. It explores the intrinsic significance of the Estonian history within the parameters of ‘Invasion and Dispossession during WWII’ as the content, and the explicit value of card game as a form of physical interaction. The physical components

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

C. Lau / Imparting Digital Memory Content Through Game Form

11

of the design concept use technology such as sensors and RFID tags to render a computer augmented environment. It includes: 1. A reproduction of the original Estonian cards with RFID tags, so that each card may prompt audio/visual (AV) playback to be projected onto the floor; 2. Estonian historical content in AV format that responds to RFID data. 2.1. Participatory Experience, Interaction and Anticipation This design concept, which focuses on technology to create a more participatory experience, is an alternative to the didactic approach of ‘Our New Home’. By drawing on museum visitors' prior knowledge of their interaction with playing cards, it invites and engages visitors to pick up the cards and begin to play. In doing so, it promotes physical interaction between visitor and object, and encourages interaction amongst the visitors. This interaction is expected to encourage visitors to not only share the experience, but also their possible knowledge and expertise within the context of the Estonian Exhibition. Visitors may anticipate the AV response as generated by their interaction with the cards, which may encourage visitors to engage more with the exhibition.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3. Content Development

Figure 3: Design concept

Firstly, observational research was conducted to study the dynamics of a card game. The aim was to consider the physical, and emotional aspects of human interaction with playing cards, to which the glass case method of display have severed. Players were observed and were found to focus highly on body language and power negotiations and struggles. More interestingly, they often use it as a prompt for social interaction. At times, the game became secondary to the experience, while conversations begin to generate in topics of common interests. Reflecting on this idea: could a game using this clearly evocative and culturally significant set of playing cards amongst Estonians trigger or provoke conversations within the parameters of their commonality? Such as their Estonian identity, memories of the first republic, WWII and subsequently the occupation. As such, the concept was further developed towards a focus on exploring personal Estonian memories throughout this period. 3.1. Audio Data Four Estonians, who had fled Estonia between 1940-1950, were interviewed and were prompted to recall their memories during that time. The choice to integrate these oral accounts as the audio content was a response to the significance of the object, initial observational research, and also the theoretical underpinning of German sociologist Maurice Halbwachs in his work on collective memory. Oral/biographical accounts form a part of the collective memory, which

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

12

C. Lau / Imparting Digital Memory Content Through Game Form

Halbwachs denotes as 'memory in (the) group, not memory of the group.’[1] They are 'currents of thoughts whose continuity is not at all artificial, for it retains from the past not only what still lives or is capable of living in the consciousness of the groups keeping the memory alive. By definition it does not exceed the boundaries of the group.'[1] 3.2. Visual Data An observation of the oral accounts illustrates that each Estonians’ recall is based on a collective repository of a shared history. Collective acceptance of the currents of events, or an historical context was pertinent in forming a structure for bringing coherence to their own memories. Interviews with visitors of ‘Our New Home’ also denoted that the historical context is also critical for a wider audience to understand the context of the Estonians’ memories. In December 2008, visual material (photo, video) was gathered in Estonia to be integrated with the oral Figure 4: Video from Estonia accounts as proposed in the concept to provide historical context to the narratives. A series of user testing were conducted to access the creative intervention. Images of the Estonian landscape were recorded to give a sense of the geography. Participants from the user testing were uncertain of the veracity of the visual landscapes and if it conveyed the historical context. An alternative to the current visual material is to be collected in Estonia in late 2009.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. Designing Interaction Possible forms of interaction were considered to impart the AV content. Drawing from interaction theories within the area of interactive and virtual art, theorist Mark Hansen inferred that ‘embodiment’ and ‘tactility’ is primary to all communication, thus inducing engagement. He posits ‘…the flesh of the body makes us understand the flesh of the world,’[4] and while the visual sense has always played the pertinent role in art, ‘the gap is larger between the seeing and the seen then between the touching and the touch.’[4] This notion inferring the sense of touch and embodiment have been explored by interactive artists such as Rafael Lozano-Hemmer [5] and Camille Utterback [6]. These artists position spectators as central to the experience, where the spectators’ actions and reactions become the art itself, and shifting the status from merely a spectator to a participant. Often the participants are generators of the content or even become a part of the content. The above approach of human-centered design has inspired this project. The visitors to this exhibition are expected to find their physical actions and reactions to handling the playing cards will impart AV content, which will ultimately augment the digital exhibition space.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

C. Lau / Imparting Digital Memory Content Through Game Form

13

4.1. Iteration of Design Concept A combination of AV content of the four Estonians’ memories on invasion and dispossession is edited and segmented into 52 digital files. Each one of the Estonian memories is represented by one of the suits. Visitors are encouraged to play, make choices and control the card game, in which the AV will respond to the RFID tagged playing cards. AV content will then be Figure 5: Conceptual design with Estonian imagery projected onto floor, therefore augmenting the environment. Their decisions may be driven from the notion of play, and may ignore the content altogether. Alternatively, they may let the audio (the memories) and the desire to listen to a linear narrative, or the visuals (the historical context) influence their decisions. To test this design concept, a series of paper-testing, interviews and focus group sessions were conducted.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4.2. Testing Stories and Card Game in a Focus Group Session Placed on the back of each playing card were the transcribed stories of the Estonian immigrants. Participants were invited to play a specifically designed game, whereby the winner collected a series of cards and read out the story behind them. What resulted from these focus group sessions were that the participants found the task of reading out the story to be laborious, contrary to the immediacy that is expected from a museum visit. Also, stories were much too segmented to give a clear linear progression. Participants longed for the progression of conventional story telling of beginning, middle and end. Focus groups were asked to discuss and redevelop a set of rules that may enable the linearity to come through more easily, while not restricting the game to simply a ordering of the cards. A majority decided that a game in the form of rummy or solitaire, which is based on a chronological structure, might better enforce the linearity needed for story telling. Responding to these findings, the design will require more direct instructions and rules on visitors’ interaction with the cards, and also how that may influence their interaction with multiple visitors in the exhibition. 4.3. Testing audio stories and visual content What evolved from the previous game was that the narratives were supposed to impart immediately after the opening of the cards. This was lagged and participants had to wait till the end of the game to hear the stories. Therefore, the test game had to be redeveloped so that stories will impart more readily to players’ actions. In this case, the game must then move beyond paper testing so that digital content can be designed to respond more readily and eliminate the laborious act of reading. Interviewees were instructed to listen to the four Estonian stories with previously filmed Estonian landscape as a backdrop. Many responded positively to the audio, as

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

14

C. Lau / Imparting Digital Memory Content Through Game Form

the voices were emotionally charged and gave a sense of the Estonian people. However, the snowy landscape were thought to represent northern Europe, and made no direct link to Estonia. It did not provide the historical and temporal context as imagined. Evidently, alternative visual content is to be captured before moving into a reiteration of the game test in digital form.

5. Project to Date This project is in its developmental stage under the conceptual framework of creating a follow-up Estonian Exhibition with a focus on user-centered design. Conceptually, it is expected to provide more insight into the Estonian history through personal memories, and engages visitors through dynamic and physical interaction. The physical space in the exhibition will be augmented through large-scale visual projections, where a central table will enable visitors to interact with the physical interface of the playing cards.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

6. Conclusion The combination of game form and narrative has been widely explored, especially in video and computer games. Taking games into the museum context requires disseminating it from the deep association with the notion of play, and consider the museum’s objective to inform. The video game experience is private and intimate, and choices may not be so easily scrutinised. Meanwhile visitors’ choices within a museum may not always be judged, but definitely observed. Where visitors' interaction with the installation generates a direct response, as the concept of this project proposes, visitors’ action may be affected under the public eye. The design concept for this project brings this particular set of Estonian playing cards to life. Displacing it from its artifact status, this object is utilised as a physical interface for imparting digital content to generate a computer augmented environment. In the context of a museum, knowledge and information takes precedent, and the playfulness of game becomes secondary. The game is then considered a strategy to engage visitors with the exhibition content.

References [1] [2] [3] [4] [5] [6]

Halbwachs, M. The Collective Memory. (trans.) Ditter, F. New York: Harper and Row, 1950, p 294 and p 80. Misztal, B.A. Theories of social remembering, USA: Open University Press; 2003, p 21. Pearce, S.M. Interpreting objects and collections, London; New York; Routledge, 1994, p 26. Hansen, M. Bodies In Code: Interfaces with digital Media. USA: Routledge, Taylor and Francis Group, 2006, p 68 and p 77. Lozano-Hemmer, R. Rafael Lozano-Hemmer. www.rafael-hemmer.com 2009. 15th March, 09 Utterback, C. Camille Utterback. www.camilleutterback.com 2009. 2nd February, 09

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-15

15

Memory Baubles and History Tinsels a

Daniela Petrelli a,1 and Ann Lightb University of Sheffield,bSheffield Hallam University

Abstract. This design proposal explores the concept of tangible digital mementos concealed inside baubles to be periodically revisited at special occasions. A photostory describes the concept and a technical description provides details. We outline how this concept could be extended to other objects, like tinsels, creating chains for stories. Keywords. Digital mementos, tangible interaction.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction Autobiographical memories have been the focus of studies in many different disciplines, e.g. physiology, psychology, sociology. Computer science has looked at autobiographical memories mainly with the lens of life-logging: digital technology to capture ‘everything’, every event we experience, conversation we participate in, and any piece of digital data we touch [1, 3, 4]; these accurate digital records can then be accessed to re-construct past events. However this stream of data bears little relation to what people actually do with their mementos, objects given or kept in memory of someone or something. Objects are carefully selected as meaningful and their display (or safekeeping) has a particular function [6, 7]. They are deliberate, often purposefully constructed, and they are looked after and revisited. Digital mementos do exist: people have many digital photographs and videos, they have emails and text messages they would never delete, they have audio recordings and digital artifacts [8]. The owner cherishes their digital belongings as much as they do physical ones, but the lack of easy access and of consolidated practice makes the appropriation of (and chance to build meaningupon) digital memories particularly difficult and frustrating: the current technology is ‘too much like work’ [8]. What prevents the appropriation of digital mementos is mainly the lack of immediacy (need to switch the computer on, navigate the file system, start the right application) and the burden of maintenance (e.g., sorting, grouping). ‘Digital’ is also perceived as ephemeral and the need to migrate to new technology is seen with trepidation; there is no trust it will last for future generations.Digital mementos should be fun to use, easy to preserve, personalizable and shareable [8]. Tangible interaction and ambient technology introduce the potential to make digital mementos easily accessible and enjoyable. The design concepts we present explore the intersection between individual and collective experience, and that between material and digital culture. We focus on 1

Corresponding Author: [email protected].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

16

D. Petrelli and A. Light / Memory Baubles and History Tinsels

the value of rituals as memory practice taking the case of Christmas (or any other cultural celebration like Diwali, Eid or Passover) as instantiation of such an event.

1. The Social Context of Autobiographical Memories The act of remembering generally occurs in a social context and the memory of an individual triggers the memory of others [2]. Celebrations create the ideal context for collective remembering: remote friends and family come together; events are made special by exceptional behaviours (e.g., preparing special food or dressing up) and the performance of rituals (e.g., home decoration or religious rites). Some celebrations recur year after year, creating a chain of reference points in people’s life. For many in the West, such a special time is Christmas: the home and the tree are decorated, greetings cards are sent, the family meets, carols are sung, presents are given. Setting our design around Christmas offers the possibility of exploring the concepts identified above: making digital mementos easily accessible and enjoyable to use; self contained and lasting for generations; rich and evocative; concealed and revealed; personal but shareable. We identify ways that digital mementos can extend the reach of our celebrations, by allowing moments to be shared across space and in different social formations (fig 1).

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 1. A map of how digital mementos can enrich personal and collective memory.

2. Memory Baubles 2.1. A Christmas Tale2 It is Sunday, the 8th of December and I take out the Christmas tree. My children are always very excited: we put on Christmas carols, take the boxes with the decorations from the cellar and munch spice biscuits.

2

This narrative is autobiographical, describing the first author’s ritual of decorating the Christmas tree, until it launches into possibility.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Petrelli and A. Light / Memory Baubles and History Tinsels

17

We take the decorations out: every single one is related to a memory, a travel or a gift. The black angels I bought in Boston in 1996, the wooden rocking horses are from 1997 when my son was born, the metal snowflakes Oslo 1998. Then there are the gifts: the ‘hand of God’ from a Jewish friend, the glass bell Elena sent me last year. Every object is a short memory trip.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The delicate baubles are individually wrapped. These two are precious to me, the only survivors from when I was born in December 1963. My Dad bought them for our first Christmas tree: a ‘welcome home’ for my Mum and me.They have been passed to me a few years back. I like their colour and feel the lightness of the thin glass. This one is from my daughter Martina. She decorated it and inside is a digital video clip of her singing “A chubby little snowman”. Recorded in last year’s snowfall, I’ve not seen it since. I call her and open the ball to play it: the video clip appearing inside. She has grown so much! I wonder how it will be when she shows it to her children in 30, maybe 40, years time. We hang it carefully on the tree, connecting it with the Christmas lights for the annual recharging. On Christmas Day, the whole family comes for lunch. I take out a new Memory Bauble and open it to start recording my son Andrea challenging Granddad at table football. We throw the image at the TV and it plays there, briefly cutting into the afternoon film. 2.2. How Memory Baubles work Memory baubles (fig. 2) come in packs of 25, so that they last a generation. Pretty enough to hang unfilled, as a group they acquire meaning when each year more balls may be used to store captured moments of Christmas activity. Fig 2 A Memory Bauble. Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

18

D. Petrelli and A. Light / Memory Baubles and History Tinsels

Baubles have two functions: they capture and show/throw. To record a short snippet of audiovisual material, you open the bauble in the direction of the scene you wish to capture (fig 3a). A camera is positioned so that even a small aperture will allow filming to proceed. This is also a small microphone below. To stop capture, close the bauble. To view, pull out the pin at the base and open. Record over/make changesby replacing the pin. To lock the memory, remove the pin and discard it (though other bauble pins, and even a chunky piece of wire can be used instead if later Fig 3a Capturing and 3b Transferring circumstances cause you to change your mind.) The chip inside can be removed for editing by those so motivated, but will otherwise exist to hold a moment in time within the bauble. Special greetings cards can be passed through the opening in the bauble to capture material from the bauble on a chip, which can then be sent to absent friends (fig 3b) for viewing or slotting into theirbaubles. Opening the bauble and ‘throwing’ the contents at a screen will display it there, so that memories may be added to TVs, laptops and large screens,both at home and possibly in the street, with the advent of public displays of community content [eg 5]. A special display bauble can scan chips in nearby baubles and play random moments from previous years while hanging on the tree, giving an almost ghostly presence to the past (fig 4). We note that in opening and shutting snugly back together, and especially as envisaged here with a degree of translucency, baubles offer the potential for storing small objects. This may, of course, interfere with showing the media contents of the ball, yet this kind of activity makesthe ornaments into repositories Fig 4. for things that are themselves mementos. Thus, baubles may have emergent properties invited through their role and form.

3. History Tinsels The memory bauble design concept explores the easy capture, storage and access of single memories. However memories are not isolated and stories can be composed out of a number of related episodes. As some African oral cultures use ‘quilts’ to encode the story of the village (the storyteller uses the sequence in the quilt to help with remembering and to tell the story), in a similar vein, tinsels could be created by composing pieces of cloth out of digitally enriched objects…Memory sticks containing photos of a year and incased in a beautifully sparkling silicon objectare threaded in and

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Petrelli and A. Light / Memory Baubles and History Tinsels

19

alternated with large beads containing sonic memories recorded similarlyto the baubles. We do not elaborate fully here, but finish by considering how to make linked narratives. 3.1. My Family History in a Tinsel This Christmas is special: my Granny is coming to my home for the first time. She has promised she takes the old family photos for my children to look at: she is always full of old family stories, you never know if they are true or not. We sit around the table with the old family photos. A few are very old, the late 19th century or the beginning of the 20th. We look at the picture of my GreatGrandfather, my Granny’s Dad, the one taken when he left for the 1st World War. She says a bomb exploded next to him: he was sent home in shock and he did not speak for many years.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

My son Andrea is studying the two world wars at school and wants to record this snippet of history. He picks up a Sonic Bead from the box and turn it right to start recording. When Granny has finished the story, he turns it left: one to stop, two to play. He has captured it in the Bead. I take the photo and scan it. I’m making a family history digital scrapbook with photos, hand written stories, audio and video clips. The albumis stored in a beautiful wooden USB memory stickthat plays a photo show when plugged in the TV. I also print a fewphotos to thread in the family history tinsel. I choose a wedding series: my mothergrandparents wedding, my parents and mine.

4. Conclusion In conclusion, Christmas is a time when personal, family and cultural spheres intersect. For instance, in one year alone, we might experience Dickens’ A Christmas Carol, Midnight Mass, carol singers collecting in the street or Mall, and the out-of-tune uncle that insists on performing. And we might capture all these in baubles and hang them on the tree. As these various encounters are collected, combined and played back in new contexts, so they become rich over the years in ways of which we did not,at outset, conceive. But we stress, we use Christmas and its trappings as an example of a repeated and ritualized activity which inspires strong emotional and nostalgic responses. The form of baubles is particular to Christmas, but the idea of supporting occasional and repeated activities is not. Memory is linked strongly to context and we present objects that gain much of their meaning in being attached to a repeated event and found again at a moment that links to other moments like it. These objects have the potential for ambiguity and mystery, for being put away and rediscovered at times of shared delight,

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

20

D. Petrelli and A. Light / Memory Baubles and History Tinsels

for dissolving the distance between remote friends and family, for building upon old stories with new memories and creating layers of association as well as holding simple aesthetic appeal. By capturing ghosts of Christmas past inside the objects, they develop a duality as both item and medium which forever changes their relation to the tree. But, unlike the ‘corny’Christmas card that plays the same tinkly carol in a thousand living rooms every time it is opened, these trinkets hold unique and chosen moments as keepsakes. References

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[1] Bell, G., and Gemmell, J A. Digital life. Scientific American, March 2007. [2] Halbwachs, M. On Collective Memory. The University of Chicago Press (1992). [3] Kern, N., Schiele, B., and Schmidt, A. Recognizing context for annotating a live life recording. Personal and Ubiquitous Computing, 11 (2007), 251-263. [4] Mann, S. Continuous lifelog capture of personal experience with EyeTap. Proc. CARPE 2004, ACM Press (2004), 1-21. [5] Müller, J., Cheverst, K.,Fitton, D., Taylor, N., Paczkowski, O., and Krüger, A. (2009) Experiences of Supporting Local and Remote Mobile Phone Interaction in Situated Public Display Deployments. International Journal of Mobile Human-Computer Interaction, 1 (2). pp. 1-21 [6] Petrelli, D., Whittaker, S. and Brockmeier, J. Autotopography: What Can Physical Mementos Tell Us about Digital Memories? Proc. CHI’08 (2008). [7] Petrelli, D., van den Hoven, E., and Whittaker, S. Making History: Intentional Capture of Future Memories, Proc. CHI’09 (2009). [8] Petrelli, D. and Whittaker, S. Family Memories in the Home: Contrasting Physical and Digital Mementos Journal of Personal and Ubiquitous Computing, in press.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-21

21

Informing Customers by Means of Digital Product Memories Alexander Kröner1, Patrick Gebhard, Lübomira Spassova, Gerrit Kahl, and Michael Schmitz German Research Center for Artificial Intelligence Stuhlsatzenhausweg 3, 66123 Saarbrücken, Germany

Abstract. The continuous collection of digital information via smart labels attached to physical objects is a promising way to support information availability across all stages of a product's lifecycle. Since such "digital product memories" may contain a vast amount of heterogeneous data, we expect a strong demand for user support in tasks related to information retrieval and discovery. In this article, we focus on the interaction between consumers and digital product memories in a retail scenario. On the basis of several prototype implementations, we summarize various ways of retrieving and presenting product-related information with the goal to shed some light upon aspects of relevance for the interaction between users and object memories in general. Keywords. smart items, digital memories, tangible user interfaces, retail experience

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction From a customer's point of view, product-related information has a broad range of applications. For instance, it helps in choosing a product based on its features, judging its quality, applying it, and learning about it. A retailer may comply with this interest by offering such information in order to increase customer satisfaction (cf. [1]). The basis of such a service is a rich pool of product-related information, whose completeness and soundness is a direct reflection of the product's integrity (cf. [2]). The building of such a pool requires the continuous collection of information across all stages of a product's lifecycle, ideally on the level of objects, in order to achieve a unique and detailed view on the particular situations a product individual is exposed to. This task can be supported by smart labels, which can be beneficial for all partners of a business process (cf. [3]). Further potential arises from the connection of the physical item with a dedicated storage for digital data – a digital product memory [4]. If such a memory is open to access for any party along the value chain, then it might become a key element of new business models on the basis of ambient intelligence (cf. [5]). The interaction between the user and such extended products may benefit not only from a rich resource of object-related data, but also from technical extensions attached to the product. In the context of this article, we will discuss selected product information systems, which are part of a complex shopping environment installed at the 1

Corresponding Author, email: [email protected].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

22

A. Kröner et al. / Informing Customers by Means of Digital Product Memories

Figure 1: A kiosk-like system enables the customer to retrieve and explore contents from a digital product memory. By placing the product on an RFID antenna, the respective memory is accessed (1). A diary-like view supports the exploration of the product’s history (2). A feature-oriented view combines memory contents with external information sources in order to explain the product’s features (3).

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Innovative Retail Lab (IRL, see http://www.dfki.de/irl/). The systems are connected to the Internet, the store's backend, and a so-called Object Memory Server (cf. [6]). Each product is equipped with a smart label providing a pointer (on the basis of a URL) to the respective product's memory. However, in contrast to these commonalities, the individual systems vary in the technical nature of the smart label (which ranges from RFID to sensor nodes) and put emphasis on different aspects of a digital memory in order to illustrate its diverse applications.

1. Browsing a Digital Product Memory The access to digital information via a tangible product enables a broad range of customer-oriented services. For instance, such information may be exploited to retrieve multimedia content that fits to a product in focus and to express recommendations based on product similarity and compatibility [7] – services which can be exploited to enrich the customer’s retail experience in shopping environments such as [8]. In order to research access to a digital product memory, we realized a prototype of a public access point, which enables a customer to explore a product memory without additional equipment. This product memory browser uses a kiosk metaphor: if the user places a product on a sensitive surface, then the system presents product information compiled from the respective memory, the store's backend, and the web. Designed for machine processing, the content of a product memory may be difficult to interpret by a user without additional support. Therefore, digital assistants are required which perform a translation and visualization of memory contents which meets the user’s abilities and goals. For instance, in the context of quality control, an ecologically-oriented customer might be interested in the product’s carbon footprint. Often the aggregated value of all events related to this particular value will be sufficient

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Kröner et al. / Informing Customers by Means of Digital Product Memories

23

to satisfy this interest. If the user is willing to learn about the rationale behind this value, then she may explore processes applied to the product via a list display where related events arranged in a timeline. Touching an event puts it into the focus of the browser, which in turn displays the status of the product as well as the process applied for the selected point of time. This approach emphasizes an explorative interaction with the historical aspect of memory contents. In contrast, the so-called "product lens" provides a feature-oriented perspective on the product. First of all, it enables an optical zoom of the product’s packing. On the basis of a digital model, the user may select different perspectives of the product's packing and magnify these in order to obtain easy access to information printed there. In parallel, the system displays the matching digital representation of that information from the product's memory. Here, the system exploits an OWL-based semantic encoding of the product features. First of all, it allows for translating memory contents into a terminology familiar to the customer. For instance, the system may extract a notion such as “E415” from the memory, exploit “same as” relationships in the underlying ontology to obtain the alternative notion “Xanthan”, and then exploit both notions to extract a matching description (e.g., “A food additive used to increase the viscosity of a liquid.”) from a public web service. In this way, the user may "zoom" into information exceeding the one printed on the packing on the basis of the meaning of product features, similar to the idea of a “semantic zoom” (cf. [9]). Similarly, the system exploits the semantic encoding in order to provide support on the basis of feature interpretations. For instance, in Figure 1 the system guides the user via "smileys" through the zooming process to the information that an ingredient is suspected to cause allergies. Thus, a customer interested in a quick overview may get an idea of a potential risk, while other customers may exploit the combined information sources of product memory and the web in order to learn about the rationale behind that hint.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. The Product as a Means of Interaction An aspect shared by these examples is that the physical object becomes part of the interaction between user and information service. This matches the idea of tangible interfaces, which aim at a direct manipulation and experience of digital information in the real world [10]. A particular advantage of this kind of interaction is the seamless integration of the communication between man and machine in procedures the user is familiar with. The Digital Sommelier [11] illustrates how this idea can be transferred to a product recommendation process (see Figure 2). Via RFID, a shopping assistant recognizes if a customer has just removed a product from a shelf. Without interrupting that action, it proactively responds with information regarding that particular product’s features. This content is retrieved from the product’s memory, and displayed on a mobile PC mounted to the shopping cart. The display combines static aspects (such as vineyard and type of grape) with dynamic data (such as current temperature) retrieved from a temperature sensor attached to the bottle. The assistant exploits the combined data to provide recommendations for the general kind of product (e.g., matching dishes) as well as for the particular instance (e.g., “This bottle should be cooled before serving.”).

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

24

A. Kröner et al. / Informing Customers by Means of Digital Product Memories

Figure 2: An accelerometer attached to the packing enables the shopping assistant to capture the customer’s interaction with the product. The assistant interprets a turning movement as an information search gesture, and responds with a presentation of additional information on a display attached to a shopping cart.

If the customer is turning the product package (e.g., in search of more information), the assistant switches the displayed content as well – e.g., to the homepage of the manufacturer. This support is enabled by an acceleration sensor attached to the product’s packing. Thus, the product becomes a means of interaction. The combination of this approach with additional output channels – e.g., speech output –, allows for creating products, which serve the customer as a communication partner. Such objects can be used within an intelligent environment for achieving a symmetric multimodal interaction, where physical objects mimic the user’s interaction behaviour (cf. [12]).

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3. Employing Digital Product Memories for Interactive Product Exploration Beside other methods for conveying product information, we also aim to present product information in a narrative style. This should enable customers to look into products interactively in a playful and enjoying manner. Since products provide different types of information (e.g., facts of the product memory or the narrative structure), customers have different interactive experiences that invite to explore products directly in the store. We explore this idea in a setting, which applies a narrative structure that comes with the product itself to control the interaction with virtual characters. Especially intuitive interfaces, like tangible objects, that are used to communicate with virtual characters help to attract users and engage them in interacting with the system. This approach seems to be appropriate for consumer service systems that (1) inform customers about individual product information and (2) advertise related products. Virtual characters (VC) have been used in many applications in various fields for more than ten years [13]. Their human-like communication skills can be exploited for the creation of powerful and engaging interfaces. In addition, VCs can easily take different roles, e.g. a guide, a companion, a trainer or an entertainer. Their appearances and their communication skills enrich interactive systems, such as kiosk systems, virtual training systems [14] and museum guides [15]. Especially systems with fullbody and human-sized VCs in public spaces, trade fairs, entrance halls and museums draw users’ attention and invite them to interact. In such settings, the use of digital product memories can improve the interaction experience substantially. Based on

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Kröner et al. / Informing Customers by Means of Digital Product Memories

25

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

individual product information, the interaction with customers is colored individually (e.g. verbal comments, special actions performed by virtual characters that represent individual facts, which are stored in the product’s memory). As a result, the interaction experience varies and gets livelier. For the realization, we rely on the SceneMaker approach for modeling interactive performances with virtual characters. It follows the concept of content separation and narrative structure, which we have introduced in [16] and successfully used for the creation of interactive systems. The content is organized in Scenes and a Sceneflow represents the narrative structure. Scenes are pieces of author-edited contiguous dialog. Additionally, they can contain commands for controlling the characters’ non-verbal behavior and for the presentation of media objects (e.g. showing pictures or videos). Authors usually refer to a scene as a coherent and closed unit regarding a message or a comment. They can define the narrative structure by linking the scenes in a graph called Sceneflow. Transitions in a Sceneflow are triggered by transition events. These events represent the user's actions. In the case of a customer service system, an event would be e.g. the placement of an object in a designated area as described in the section before. The content for interactive presentations (the Scenes) is created by the authors using standard text processing tools. The creation of the Sceneflow is done by means of a graphical user interface, which is part of the SceneMaker tool and can also be used by non-programmers. The use of digital product memories offers different ways for controlling an interactive presentation involving virtual characters. One approach is to store a Sceneflow and Scenes for each product in a product's memory. This would ensure that each product holds the information (and the interactive) structure for its presentation. When a customer brings a product to this system, the virtual characters are controlled by the product's individual Scenes and Sceneflow and the interaction is individually adapted to the product's specific knowledge. However, a general Sceneflow and additional Scenes are needed to embed those from a product seamlessly for an interactive presentation. Both can be individually tailored to customers and individually created by the manufacturers of products or influenced by retailers. In general, this approach would enable a new way for product information and advertisement.

4. Discussion Within the aforementioned examples, a basic function of the physical product item is the one of an information key. It grants access to a broad range of information about the object itself (from the product memory), the kind of object, and related information (taken from the web). This information pool will easily exceed what a user is willing to handle in a shopping situation, an issue which can be addressed in various ways. For instance, we presented a feature-oriented presentation as well as a diary metaphor. The former focuses on a quick summary of the product’s current status and thus it is more appropriate for immediate shopping decisions. In contrast to that, the diary metaphor enables the user to learn about the evolution of these features (and thus about cause and effect), which requires that the user is in a situation with little time pressure. The design of a digital product memory may support these processes through the employed hardware as well as the respective memory contents. For instance, attaching sensors to the product is not only a way to capture product-related data, but also to emphasize the physical item’s potential role as a means of communication or even as a

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

26

A. Kröner et al. / Informing Customers by Means of Digital Product Memories

communication partner during the presentation process. Furthermore, a presentation may benefit from the well-defined semantics of memory contents. In addition, we suggest deploying presentation knowledge (ranging from presentation style to dialog structure) within the memory, which enables parties contributing to the memory (e.g., manufacturer, retailer) to take influence on the way the product is presented – or is presenting itself. The discussed information systems could further benefit from information about the particular customer. Here, we see further potential in dedicated memories which save a (personal) dialog state in order to enable consistent and coherent dialogs across information points (in potentially varying contexts). The presented research was partially conducted in the project “Semantic Product Memory” (SemProM), which is funded by the German Ministry of Education and Research under grant 01 IA 08002A. In this project, we want to exploit the lessons learned so far for the general design of a digital product memory and for the realization of novel applications based on digital memories along the supply chain.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] R. Kaufman, A Well-Informed Customer is a, 2006. Retrieved April 30, 2009, from . [2] K.B. Clark, T. Fujimoto, The Power of Product Integrity, Harvard Business Review, 2009. [3] C. Decker, M. Berchtold, L.W.F. Chaves, M. Beigl, D. Roehr, T. Riedel, M. Beuster, T. Herzog, D. Herzig, Cost-Benefit Model for Smart Items in the Supply Chain, Proceedings of Internet of Things 2008, LNCS 4952 (2008), Springer,. [4] W. Wahlster, Digital Product Memory: Embedded Systems Keep a Diary, Harting tec.News 15 (2007), 7–9. [5] C. Schmitt, K. Fischbach, S. Muhle, D. Schoder, Towards Ambient Business: Enabling Open Innovation in a World of Ubiquitous Computing, Advances in Ubiquitous Computing: Future Paradigms and Directions, Idea Group Inc, Hershey (2008). [6] M. Schneider, Towards a General Object Memory, UbiComp 2007 Workshop Proceedings (2007), 307– 312. [7] W. Maass, A. Filler, Tip 'n Tell: Product-Centered Mobile Reasoning Support for Tangible Shopping, Proceedings of MSWFB 2007: Making Semantics Work For Business, part of 1st European Semantic Technology Conference, Vienna, Austria, 2007. [8] IDEO, Staff Devices and Dressing Rooms for Prada, 2000. Accessed June 9, 2009, from . [9] D. Modjeska, Navigation in Electronic Worlds: A Research Review, Technical Report. Computer Systems Research Group, University of Toronto, 1997. [10] H. Ishii: The tangible user interface and its evolution. Communications of the ACM 51(6) (2008), 32–36. [11] M. Schmitz, J. Baus and R. Dörr, The Digital Sommelier: Interacting with Intelligent Products, Proceedings of Internet of Things 2008, LNCS 4952 (2008), Springer, 247–262. [12] W. Wahlster, R. Wasinger, The Anthropomorphized Product Shelf: Symmetric Multimodal Interaction with Instrumented Environments, True Visions: The Emergence of Ambient Intelligence, Springer, Heidelberg, Berlin, New York (2006), 291–306. [13] J. Cassell, T. Bickmore, M. Billinghurst, L. Campbell, K. Chang, H. Vilhjálmsson, H. Yan, An Architecture for Embodied Conversational Characters, Proceedings of the First Workshop on Embodied Conversational Characters (1998), Springer, 109–120. [14] Z. Ruttkay, H. v. Welbergen, Elbows Higher! Performing, Observing and Correcting Exercises by a Virtual Trainer, Proceedings of the 8th International Conference on Intelligent Virtual Agents, LNAI 5208 (2008), Springer, 409–416. [15] S. Kopp, B. Jung, N. Leßmann, I. Wachsmuth, Max - A Multimodal Assistant in Virtual Reality Construction, KI 4(17) (2003), 17–23. [16] P. Gebhard, M. Kipp, M. Klesen, T. Rist, Authoring Scenes for Adaptive, Interactive Performances, Proceedings of the 2nd International Joint Conference on Autonomous Agents & Multiagent Systems (2003), ACM, 725–732.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-27

27

An Object Memory Modeling Approach for Product Life Cycle Applications Joerg Neidiga, 1 and Peter Stephan b, 2 a Siemens AG, Germany b German Research Center for Artificial Intelligence, Germany

Abstract. Today, industrial production and supply chains are facing increased demands regarding flexibility and transparency of processes, caused by a trend for mass-customization and increasingly tighter regulations for the traceability of goods. To fulfill such challenging market demands, auto-ID technologies and semantic product descriptions are becoming part of future value chains. In this paper a modelling approach for a digital object memory (DOM) allowing for the attachment of product life cycle (PLC) information to everyday objects is presented. After reporting on the design aims, memory architecture and data structure, potential benefits of the chosen approach are presented. Keywords. Digital Object Memory, Modeling, Internet of Things, Distributed Systems, Product Life Cycle, Radio Frequency Identification.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Introduction Today, the increasing trend of shortening product life cycles and the customer desire for highly individualized goods are asking for flexible industrial and supply chain processes which are able to adapt quickly to changing demands. When it comes to quality products, perishable goods or healthcare products, this is supplemented by the request for a transparent monitoring of events that happened during the product’s life [1]. In order to face such challenges competitively, the collection, storage and management of comprehensive product life cycle (PLC) information becomes a crucial factor for optimizing processes. In this context, auto-ID technologies like barcode and radio frequency identification (RFID) have already proven their potential to align the physical flow of goods with the digital flow of corresponding information [2]. Most of current solutions in operation use auto-ID technologies and object-related information only in the context of closed-loop applications in single domains or even a single company [3]. Although ID system solutions like the electronic product code (EPC) or the GS1 coding system already allow for a information exchange between certain stakeholders of a value chain [4, 5] these systems focus entirely on referencing information in dependence of the object’s ID. 1

Joerg Neidig: Siemens AG (Sector Industry), Gleiwitzerstr. 555, 90575 Nürnberg, Germany; E-mail [email protected] 2 Peter Stephan: German Research Center for Artificial Intelligence (DFKI GmbH) Trippstadter Str. 122, 67663 Kaiserslautern, Germany; E-mail: [email protected] Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

28

J. Neidig and P. Stephan / An Object Memory Modeling Approach for PLC Applications

In this paper, the modeling approach for a digital object memory (DOM) is presented, to associate object-related PLC information with physical products in an efficient way. The solution will be tailored to the needs of real life applications, and allow for the gathering and storage of information from different sources. Furthermore the DOM will represent its content in a way enabling a seamless exchange and use of PLC information in applications throughout the whole value chain.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. Related Work Going conceptually beyond current auto-ID applications, DOM describe an approach to flexibly associate digital information items with physical objects [6, 7]. They are created by gathering and storing information from information sources in the environment (e.g. sensor networks) or processes a physical object participates in. Exemplary research projects concerned with the creation of DOM are SPECTER [8] or SharedLife [9]. In SPECTER, a DOM is utilized to deliver ad-hoc assistance in a CD shopping scenario by triggering situation-aware mechanisms based on previously recorded user interactions. SharedLife drives this idea one step further by capturing, sharing and exploiting cooking experiences through DOM in a SmartKitchen environment. As other implementation examples show, DOM can vary in several dimensions like location of the physical data storage [3, 8], implementation approaches [10, 11] and potential application contexts [12, 13, 14]. By accompanying a smart pizza packaging through stages of its PLC, it has been proven in [15] that the concept of DOM also works for dealing with PLC information in broader application contexts like a complete value chain. Numerous standards for object descriptions have been created, all aiming at the definition of a comprehensive list of object properties. Well renowned approaches are the Electronic Device Description Language (EDDL) [16], the Field Device Tool (FDT) [17] the Physical Markup Language (PML) [18] or the Smart Description Object (SPDO) [19]. As all these models are created for a certain application or domain, they are deemed not be abstract and generic enough to describe all kinds of objects throughout an entire PLC.

3. Object Memory Design 3.1. Design Aims The aim regarding the DOM design presented is not to simply enrich the world with another model for describing a set of object-related information. Instead, the idea is to create a container format for object-related information which does not substitute the different device and object description languages, but unites them under one roof. The requirements for the approach are given below. 1. The DOM must contain a number of mandatory entries (e.g. object ID) 2. A set of information has to be intelligible to all stakeholders of the value chain, i.e. can be accessed and understood throughout the whole PLC 3. Besides common DOM information, an option to embed information given in proprietary formatting must be offered, in order to enable stakeholders to include already existing data structures

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. Neidig and P. Stephan / An Object Memory Modeling Approach for PLC Applications

4. 5. 6.

7.

29

The DOM has to protect itself, meaning that access should only be possible via an interface, which also allows to search for specific content To allow for a hardware independent DOM implementation, the model needs sufficient flexibility and scalability The DOM has the option to link to external information sources, i.e. incorporates arbitrary information entities like single parameters, entire information blocks or even the complete DOM The DOM has to be identifiable and allow for the versioning of it’s states

3.2. DOM Architecture The key to enable a DOM solution for dealing with information over a whole PLC lies in the flexibility of its architecture. To allow for a maximum of adaptability and openness, all DOM information is arranged in a modular block structure similar to a file system (see Figure 1). The structure consists of distinct blocks each with a clear function and determined content.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 1: Structure of the DOM and its information blocks.

In the presented architecture, the header is a mandatory part of the DOM including all information needed to identify this model as a valid DOM of a certain version. Optionally, the header can be extended to include a schema for the DOM model making it self-explanatory. The information part contains all the information related to the object itself. It consists of a collection of blocks filled with different types of information. These blocks can be coarsely divided into common and specific ones. The common blocks contain information which is not specific to a certain application, domain or section of the PLC, e.g. physical object properties, warranty information or hazard statements. To make this information commonly understandable the available keywords and the structure are fixed. The specific blocks contain information that is only of interest for a limited set of users. As this information is likely to be proprietary (e.g. production parameters or sensor readings) only very little restrictions are made, allowing to embed almost any kind of information. The modular structure of the information part offers high flexibility and scalability. The combination of a defined structure, which gives the DOM a unified appearance and the support for any kind of information formats make the DOM a convenient information pool. The chosen design allows for the retrieval of specific information by simply accessing specific blocks. Furthermore, with the presented structure the DOM can be extended easily by simply adding further blocks to the information part. 3.3. Information Block Description The structure of an information block is simple and independent of it’s content. It consists of a Block Info, Metadata, and the Data itself. Whereas the Block Info and

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

30

J. Neidig and P. Stephan / An Object Memory Modeling Approach for PLC Applications

Meta Data are mandatory using fixed keywords, the Data may be given in any format desired. The Block Info is a short section which contains the block characteristics Block Type, Block Format and its ID. By that, the Block Info includes the information needed to access and parse the block’s Data section. The Block Type categorizes the block itself. If it is a common information block the type must be taken from the list given in the DOM specifications (e.g. “owner”, “physical properties”, …). If the block is a specific one the Block Type is set to “none”. The Block Format specifies the block’s data format given as a mime-type. In all cases (with the exception of “none”) an XMLstream is expected in the Data section. The ID is unique to the block and is the key for accessing it’s information content. The Meta Data contain all information to classify and label the content of the Data section. It contains tags like the block’s name, search keywords, the block’s creator, a history log listing the changes made to the block, and so on. Consider the Meta Data section as the source of information needed for powerful search applications. The Data section contains the object-related information itself given in the format specified by the Block Info. In this section any kind of information can be stored without restrictions, but it is expected that the Meta Data are kept accurate. Optionally, it is possible to store the schema of the data format in the block header. It is important to keep in mind that for the user of a DOM it is of no interest how any of the data mentioned above is stored physically. Common understanding is that on RFIDtransponder or embedded devices a highly compressed bit-coding is needed whereas on pc-based devices a more elaborate encoding might be possible.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.4. Interface Description An application utilizing DOM information must not access memory content directly, but use a standardized interface instead (see Figure 2). In that context, the interface assumes several tasks. It works as an abstraction layer from the DOM hardware by translating the data from the format in which it is stored physically (e.g. bitcode on an RFID-tag) into the format expected by the user (e.g. a XML-stream) and vice versa. Furthermore, it has to act as a kind of administration software by creating, organizing and deleting DOM blocks and implementing rudimentary search capabilities. Information from a DOM is accessed via the ID of the respective information block, e.g. the data of Block 42 is accessed via the function GetUserData(42). As the IDs are generated by the interface automatically, they have to be retrieved via a search operation implemented in the interface. For example, the command GetBlockIds() will return the IDs of all blocks, GetBlockIds(Metadata) returns the IDs of blocks with certain Meta Data. More complex searches operations have to be implemented in external applications. The intended way to access the DOM is to retrieve a list of relevant IDs through the search option, then to (partially) retrieve the respective blocks and finally to filter the information in the application. This has the advantage of keeping the interface layer lightweight by limiting its complexity. The creation of a new block is initialized via the function CreateBlock()returning the ID of the new block. User and Meta Data can be added or altered with the appropriate Set functions.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. Neidig and P. Stephan / An Object Memory Modeling Approach for PLC Applications

31

Figure 2: Object Memory Interface.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. Benefits Section 2 showed that a large number of different object description languages and data formats specific to certain domains and optimized for limited phases of the PLC exist. The presented approach offers a container in which existing ways to describe objectrelated information can be united. This will allow for the gathering and storage of information over the whole PLC always in the most appropriate way. By that, potential users can stick with the object languages and data formats already in use. In order to migrate to the DOM framework, simply an interface plug-in is required. Regarding the presented DOM container itself, its flexible architecture sets it apart from other current approaches. Adopting a structure consisting of distinct information blocks allows to interact with small subsets of DOM information. i.e. initializing, searching and parsing DOM content can be limited to only relevant blocks without loading the DOM completely. By that, only little computing power and small communication bandwidth are sufficient to interact with the DOM, enabling it for lightweight smart item solutions attached directly to a product. To verify the viability of the presented approach, an initial DOM version has been implemented in a hardware demonstrator at this year’s CeBIT 2009 fair. Experiences and further benefits are reported in a publication currently under review [20].

5. Conclusion In this paper a modelling approach for a digital object memory (DOM) is presented, allowing for the effective attachment of product life cycle (PLC) information to everyday objects. Based on a number of design aims, the development of a container format is described, in which existing ways to describe object-related information can be united. The key to enable such functionality lies in the flexible DOM architecture and the information storage in a block structure which allows for a maximum of openness in information handling. The access to the DOM is managed by a standardized interface. Main benefits of the presented approach are its capabilities to integrate object-related information of different domains over a whole PLC, it’s easy

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

32

J. Neidig and P. Stephan / An Object Memory Modeling Approach for PLC Applications

expandability and the possibility to interact only with DOM information blocks of interest.

Acknowledgements The research described in this paper was conducted within the project Semantic Product Memory (SemProM). SemProM is funded by the German Federal Ministry of Education and Research under grant number 01 IA 08002. The responsibility for this publication lies with the authors.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] Garcia, A., McFarlane, D., Fletcher, M., and Thome, A. Auto-ID in Materials Handling, Cambridge, MA: White paper, Auto-ID Center, MIT (2003). [2] Fleisch, E. and Mattern, F. Das Internet der Dinge: Ubiquitous Computing und RFID in der Praxis, Berlin: Springer (2005). [3] Neidig, J., and Opgenoorth, B. RFID in der Automatisierung - ein Blick in die Zukunft, In: atp – Automatisierungstechnische Praxis, 7 (2008), 34-38. [4] Engels, D., Foley, J., Waldop, J., Sarma, S., and Brock, D. The Networked Physical World: An Automated Identification Architecture, 2nd IEEE Workshop on Internet Applications (2001). [5] Römer, K., Schoch, T., Mattern, F., and Dübendorfer, T. Smart Identification Frameworks for Ubiquitous Computing Applications, In: Wireless Networks, Special issue: Pervasive computing and communications, 10, 6 (2003), 689-700. [6] Mase, K., Sumi, Y., and Fels, S. Memory and Sharing of Experiences, In: Personal and Ubiquitous Computing, 11, 4 (2007). [7] Sumi, Y., Sakamoto, R., Nakao, K., and Mase, K. ComicDiary: Representing individual experiences in a comis style, In: Proc. of the 4th International Conference on Ubiquitous Computing (2002). [8] Plate, C., Basselin, N., Kröner, A., Schneider, M., Baldes, S., Dimitrova, V., and Jameson, A. Recomindation: New Functions for Augmented Memories, In: Proc. of the International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH2006), 141-150 (2006). [9] Schneider, M. The Semantic Cookbook: Sharing Cooking Experiences in the Smart Kitchen, In: Proc. of the 3rd International Conference on Intelligent Environments (IE'07), 416-423 (2007). [10] Wahlster, W., Kröner, A., Schneider, M., and Baus, J. Sharing Memories of Smart Products and Their Consumers in Instrumented Environments, In: it - Information Technology, 50, 1 (2008), 45-50. [11] Barbu, C., and Kröner, A. Designing a Study Concerning the Functions of Sharable Personal Memories, In: Proc. of the IADIS International Conference WWW/Internet 2008 (2008). [12] Pollack, M. E. Intelligent technology for an aging population: The use of AI to assist elders with cognitive impairment, In: AI Magazine, 26, 2 (2005), 9-24. [13] Kuwahara, N., Noma, H., Kogure, N., Tetsutani, N., and Iseki, H. Wearable auto-event-recording of medical nursing, In: Proc. of the 9th IFIP TC13 International Conference on Human-Computer Interaction (INTERACT’03), (2003). [14] Wasinger, R., and Wahlster, W. The antropomorphized product shelf: Symmetric multimodal interaction with instrumented environments, In: Aarts, E., and Encarnacao, J. (ed.): True Visions: The Emergence of Ambient Intelligence, Heidelberg, Berlin, New York, Springer (2006). [15] Schneider, M. and Kröner, A. The Smart Pizza Packing: An Application of Object Memories, In: Proc. of the 4th International Conference on Intelligent Environments (IE’08), 1-8 (2008). [16] Riedl, M., Simon, R., Thron, M. EDDL – Electronic Device Description Language, Oldenbourg, (2001) [17] Simon, R., et al. FDT Field Device Tool, Oldenbourg, (2005) [18] Brock, D.L., et al. The Physical Markup Language, White Paper, MIT Auto-ID Cener, (2001) [19] Janzen, S. and Maass, W. Smart Product Description Object (SPDO), In: Poster Proc. of the 5th International Conference on Formal Ontology in Information Systems (FOIS2008), (2008) [20] Stephan, P., et al. Product Mediated Communication Trough Digital Object Memories in Heterogeneous Value Chains, Under Review: 11th Intl. Conference on Ubiqitous Computing, (2009).

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-33

33

Exploring The Design of a Memory Model for Smart Objects Daniel Fitton, Fahim Kawsar and Gerd Kortuem Computing Department, Infolab21, Lancaster University, Lancaster, UK

Abstract. This paper presents an exploration of the design of a memory model to support the management of persistent historical memories recorded by smart work objects. Analysis of a range of potential application categories and scenarios involving a smart work object is used to highlight the requirements and different characteristics of object memories. The analysis is then used to identify a range of pertinent issues and trade-offs which are used to inform the design of a generic parameterized memory model. A case study involving a smart object prototype in a workplace application scenario is then presented. The case study then analyzes how the proposed memory model can be applied to memories collected by the prototype. Keywords. Smart Work Objects, Memory Model, Pay-Per-Use

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction The storage and management of the digital memories of humans (images, videos, emails, documents and so forth) is an ongoing research topic [1][2] and has been recognized as a ‘Grand Challenge’ for computer science [3]. This paper is concerned with the emerging area of digital memories produced by smart objects. Smart objects are everyday physical objects augmented with embedded technology that include sensing, processing, communication and persistent storage. The memories generated by a smart object are often related to sensing/context awareness and, through their analysis, enable a wide range of new and novel applications. For example, existing applications have involved health and safety monitoring [4] and the support of new business models [5] within the same domain. The focus of this paper is the design of a memory model to support persistent historical memories recorded by smart objects in the workplace. A memory model may have to manage the recording of a multitude of different memories to support different applications. For example, memories of its usage, its service history, its location etc. These memories may vary in level of detail, importance and granularity. A memory model must also take action when storage limits are reached. While some objects may remain indoors with plentiful network connectivity and access to backend infrastructure others, such as the smart work object discussed in this work, may spend long periods outdoors with no network connectivity or access to backend infrastructure. Therefore the memory model must take action when storage limits are reached and three main approaches, each with unique problems and trade-offs, are discussed in this work. This paper focuses on a memory model for smart work objects we also consider how our finding can be applied in a more generic smart object memory model.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

34

D. Fitton et al. / Exploring the Design of a Memory Model for Smart Objects

The following section discusses application scenarios for smart objects with persistent memories. Next requirements highlighted by the scenarios and characteristics of object memories are considered. The design of the memory model is then presented followed by a case study involving a prototype smart object, together with analysis of the memories recorded in the context of the memory model. Finally this paper discusses related work and presents concluding remarks.

Smart Objects Applications A wide range of potential applications exist for smart objects which maintain a persistent history of their memories. Memories include an object’s experiences (events or activities involving the object) and other important pieces of information. The focus in this work is smart work objects (specifically tools and associated equipment) and Figure 1 shows four different categories of memories and associated applications.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 1. Categories of Experiences Recorded by a Smart Work Object.

The category of ‘Life Cycle’ is related to the physical lifecycle of the object from design and manufacture through to disposal and recycling. Storing these memories enables possibilities such as identification of materials used in manufacture in order to inform recycling or re-use in the disposal process. Memories of an object’s origins may also be used to validate its authenticity. In the category of ‘User’ these memories are related to instructional information or legislation (user manuals, health and safety policy etc.) and memories of use of the object. For example, usage memories can be used to check associated risks to a user’s health [4]. The ‘Consumer’ category relates to issues affecting the value or desirability of the object such as a record of repair, maintenance or past owners. The ‘Organization’ category relates to memories such as object movement and use within an organization to enable analysis of business processes (and whether they need to be revised or redesigned etc.). Additionally, new business models are enabled through the use of smart objects such as pay-per-use equipment rental [5]. From this discussion two findings can be summarized: • Smart object memories are likely to emerge from a range of different sources. • Historical object memories enable a diverse range of novel applications.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Fitton et al. / Exploring the Design of a Memory Model for Smart Objects

35

Smart Object Memory Model In this section the key properties of memories stored by smart objects are identified, followed by a discussion of the design of a smart object memory model. Smart Object Memories In order to effectively manage memories it is first necessary to understand their origin and characteristics in order to make appropriate decisions about how they should be treated. On differentiation is between external sources where memories are imprinted and internal sources where memories are dynamically generated through interpretation of embedded sensor data (using predefined algorithms). For example: • Imprinted: Memories regarding design and manufacture. • Dynamically Generated: Memories regarding instances of use and misuse. It is probable that imprinted memories cannot be altered and must either be stored or deleted. However, in the case of dynamically generated memories it may be possible to change the parameters of the algorithm to produce fine-grained memories (high sampling interval, high resolution etc.) or course-grained memories (low sampling interval, low resolution etc.). These extremes represent a trade-offs in terms of detail in data recorded vs. storage space utilization. The actual requirements placed on memories are, of course, application dependent. While a memory model may be required to support multiple applications it is likely the underlying requirements must be known in advance in order to define, then potentially adapt, these algorithms. The temporal importance of memories may be short-term or long-term relative to the lifetime of the object. For example, imprinted memories about object manufacture have long-term significance while memories of individual periods of use may only be required for calculating hire cost based on pay-per-use billing in the short term. This is an additional consideration in the design of a memory model.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Managing Memories When recording memories to support multiple applications there is potential for commonalities or replication in the information stored and this must be considered in order to make efficient use of storage. For example, when requiring memories of each instance of use and the use total, the summation of the former provides the latter. In this example the memory model could potentially optimize use of storage by only recording memories of use. This is a trade-off between pre-processing (processing required during recording of memories) and post-processing (processing required during retrieval of memories) within the memory model. In more detail, the saving in terms of storage is at the cost of additional processing required to retrieve memories at a later stage. Application requirements would have to be known prior to implementation of the memory model to enable this example. In an ideal situation memories would be transferred to a backend infrastructure for periodic archival. However, objects such as tools may be in use outdoors for long periods (where the deployment of infrastructure is challenging). When no free storage is available and new memories need to be stored three possibilities exist: • Forgetting - Discarding new memories or overwriting existing ones. • Abstracting - Combing multiple memories or reducing level of detail. • Compressing – Compressing memories unlikely to be removed or changed.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

36

D. Fitton et al. / Exploring the Design of a Memory Model for Smart Objects

Each of these three possibilities offers a trade-off in terms of processing overhead vs. loss of data. Forgetting requires the least overhead and greatest loss and while compression does not incur loss there is significant processing and power consumption overhead. This trade-off is particularly important in a resource constrained embedded scenario. It is likely that long-lived memories would not be forgotten or abstracted, but may be compressed as a matter of course. The selection of a compression algorithm is likely to involve another trade-off in terms of complexity (and associated processing overhead) vs. size reduction. While the application requirements and memory characteristics dictate the applicability of these three techniques in a given scenario, we would argue that the notions of forgetting, abstracting and compressing provide the foundation of a more generic memory model. The applicability of each technique for a particular type of memory could potentially be represented as a set of three parameters.

Figure 2. Smart Work Object Prototype.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Case Study: Smart Drill Prototype This work considers memories in the context of a heavy-duty battery-powered drill, the prototype implementation of which is shown in Figure 2. The prototype contains an ARM7 processor, microSD card slot, 802.15.4 radio, Bluetooth radio and 2-axis accelerometer. The hardware and battery is contained in a small case (see Figure 2, insert). The prototype primarily records memories of its use in order to support a payper-use equipment hire model [5]. Each memory requires 20 bytes of storage and includes an identifier of the user (8 bytes), a timestamp (8 bytes) and a use duration (4 bytes). These are short-term memories and will be removed when the hire period ends and the drill is returned to the hire company. For the application of a pay-per-use model the most important memories are those that containing total usage time and other information, such as the time that usage occurred, can be used to refine the model. Several solutions for abstracting these memories exist, such as replacing the individual records with a single use total or removing either the user identifier or timestamp (each of the latter freeing 40% of the currently occupied space). Other possibilities include changing the granularity of the memories from single instances of usage to encompass more information. For example, the individual memories could be replaced with single usage total for each user for a specified time period such as a day. Conversely, the memories could be combined in terms of the times of day they occurred, giving a usage total for each time period. The savings gained by latter two possibilities are dependent on the data and the exact

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Fitton et al. / Exploring the Design of a Memory Model for Smart Objects

37

algorithms used but potentially could be as high as 80%. Another solution would be to actually execute the pay-per-use billing model on the object, storing the total cost and removing the usage memories instantly. However, in this case it becomes challenging verify that the billing model has been executed correctly or change the model.

Related Work Most of the existing works on attaching structured digital information to physical objects are limited to tag based approaches. For example, auto-ID[6] technologies like barcode [7], RFID [8][9], QRCode [10], etc. are successfully used in logistics, supply chain management, and healthcare applications. However, these tags are primarily used for object identification and unable to store more fine-grained dynamic contents due to their architectural limitations. Some researchers have investigated a more holistic approach of associating digital information by applying the notion of “Digital Object Memory”. Schneider and his colleagues designed memory models revolving around a variety of application scenarios (kitchen, shopping, etc.) and reasoned about the dimensions that influence the model e.g., on-board or off-board storage, software or hardware implementation [11]. On an application level, there are several works that have looked at capturing and sharing everyday experiences using a multitude of personal devices [1][2]. Furthermore, “Technology for Life Long Memories” has been identified as one of the grand challenges of computer science [3]. However, most of these works have taken an ad-hoc approach in terms of the memory organization. This work is concerned with systematizing this organization using a range of customizable parameters.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Discussion This paper has categorized memories in several different ways based on their characteristics (imprinted, generated, long-lived, short-lived) in order to consider how each should be treated within a memory model. It is also possible to assign memories to different physical or logical storage areas and use this as a basis for categorization. As storage limits were reached memories could then be moved between these areas forgetting, abstracting and compressing as appropriate. A key aspect of the memory model is the generation (interpretation) of memories from sensor data. Our goal is for a generic memory model that is device and application independent to allow, for example, the same pay-per-use billing model to be used on different devices in different scenarios. As sensing possibilities and capabilities change between different hardware implementations and scenarios ‘pluggable’ sensor interpretation algorithms are required. With a generic model in place such a pay-per-use billing model would be primarily based on memories of usage duration but flexible enough to include detailed memories of usage parameters (intensity, time of day, experience of user etc.) if available. The possibility of utilizing compression on smart object memories is an area for future work and requires careful consideration in a resource-constrained embedded scenario. However, existing work has shown that compressing data has the potential to save energy in terms of reducing the overhead of transmitting that data [12] and this reduced overhead may also apply to storage. Additional areas for future work include

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

38

D. Fitton et al. / Exploring the Design of a Memory Model for Smart Objects

support of external communication in the memory model to enable access to and imprinting of memories (addressing issues such as privacy, security and authentication). A key area is the mechanism for moving memories from a smart object onto a backend infrastructure. This includes issues such as verification that external memories have been transferred successfully (before removing them from the smart object) and how to maintain some form of ‘link’ to them.

Concluding Remarks This paper has considered the design of a generic memory model for smart objects, everyday objects augmented with embedded technology, with a focus on supporting smart work objects in the construction domain. The model is intended to support memories with a range of different characteristics and requirements in order to cater for multiple diverse application scenarios involving a single object. Smart objects typically operate independently and potentially spend large periods without network connectivity and backend infrastructure (especially in the work object scenario considered in this work). A key aspect in the design of the memory model is the action to take when storage space is running out. Three main techniques exist in this situation each with their own advantages, drawbacks and trade-offs. Additionally, the selection of an appropriate technique is related to individual memory characteristics and application scenarios. This can potentially be represented through a sets of parameters understood by the memory model.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] M. Czerwinski, D. Gage, J. Gemmel, C. Marshall, M. Pérez-Quinones, M. Skeels, and T. Catarci. “Digital memories in an era of ubiquitous computing and abundant storage”, Communication of the ACM, 49, 1, 44-50, 2006. [2] K. Mase,Y. Sumi, and S. Fels. “ Memory and Sharing of Experiences” Personal and Ubiquitous Computing, 11, 4, 2007. [3] A. Fitzgibbon, and E. Reiter. Memories for life: Managing information over a human lifetime. In T. Hoare & R. Milner (Eds.), Grand Challenges in Computing Research, (13-16). Swindon, 2004. [4] G. Kortuem, L. Ball, J. Busby, N. Davies, C. Efstratiou, M. Iszatt-White, J. Finney, and K. Kinder, Sensor Networks or Smart Artifacts? An Exploration of Organizational Issues of An Industrial Health and Safety Monitoring System. Proceedings of Ubicomp 2007, Inssbruck, Austria, 2007. [5] D. Fitton, V. Sundramoorthy, G. Kortuem, J. Brown, C. Efstratiou, J. Finney, and N. Davies, Exploring the Design of Pay-Per-Use Objects in the Construction Domain. European Conference on Smart Sensing and Context, Zurich, Switzerland, pp 192-205, 2008. [6] Auto-ID Labs http://www.autoidlabs.org/ [7] P. Ljungstrand, J. Redstrom, and L. E. Holmquist, Web-stickers: using physical tokens to access, manage and share bookmarks to the web. Designing augmented reality environments, pp 23-31., 2000 [8] S. Konomi, and G. Roussos, G. Ubiquitous computing in the real world: Lessons learnt from large scale rfid deployments. Personal and Ubiquitous Computing, 11(7), 507-521, 2007 [9] R. Want, K.O. Fishkin, A. Gujar and B. Harrison. Bridging physical and virtual worlds with electronic tags. In ACM Conference on Human Factors in Computing Systems, pp370 – 377, 1999. [10] J. Rekimoto, and Y. Ayatsuka, Y. Cybercode: Designing augmented reality environment with visual tags. In Designing Augmented Reality Environment, pp 1-10, 2000 [11] M. Schneider, M. Towards a General Object Memory, 1st International Workshop on Design and Integration Principles for Smart Objects, 2007 [12] S. Rein, F. Fitzek, "Compression of Short Text on Embedded Systems", Journal of Computers, 2006.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-39

39

A Trace-Based Framework for supporting Digital Object Memories Lotfi S. Settouti a , Yannick Prié a , Damien Cram a , Pierre-Antoine Champin a and Alain Mille a a LIRIS , University of Lyon, France, UMR CNRS 5205, Insa-Lyon, Université Lyon 1, Université Lyon 2, EC-Lyon Abstract. In this paper, we present a Trace Based framework for managing and transforming traces of observation and use of real life objects. Considering trace based systems as Digital Object Memories (DOMe), we describe how our framework can be used to manage DOMe, using trace models and transformations to raise the abstraction level of traces and infer useful knowledge. To demonstrate our approach, we present a simple fictional example of smart home where the use and the state of some of everyday objects are observed, and the resulting traces of such observations are exploited as DOMe providing useful services. Keywords. Trace, Digital Object Memories, Trace Transformation, Trace-Based Reasoning

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Introduction The idea of creating context aware environments has long interested the research community, especially since the pervasive computing technologies (sensors, handheld or wearable devices, RFID and wireless technologies) have become cheaper and common. The focus has been twofold. At the lower level, attention has focused on getting data from sensors and instrumented objects and creating architectures and frameworks that support their integration [1,2]. At the upper levels, efforts have essentially focused on inferring context models from sensed information, with applications ranging in complexity from simple notifing/reacting systems (e.g. if temperature > t then start air conditioner) to more complex systems that seek to build rich models and then reason over them (e.g. patient monitoring systems [3,4]). In particular, context aware real-world objects need mechanisms for reasoning about whole histories of sensed informations rather than reacting and reasoning only on recent information, requiring thus a real Digital Object Memory to provide intelligent services. This workshop’s call1 defines the Digital Object Memories (DOMe) as comprised of hardware and software components that physically and/or conceptually associate digital information with real-world objects in an application-independent manner. Such information can take many different forms (structured data and documents, pictures, audio/video streams, etc.) and originate from a variety of sources (automated processes, 1 http://www.dfki.de/dome-workshop/

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

40

L.S. Settouti et al. / A Trace-Based Framework for Supporting Digital Object Memories

sensors in the environment, users, etc.). If constantly updated, Digital Object Memories over time provide a meaningful record of an object’s history and use. As human memory, Digital Objects Memory is always changing; rarely does it stay the same over time and such changes are both quantitative, by adding continuously rough observations (at lower level), and qualitative, by aggregating and inferring new informations (often at more higher levels). Indeed, the management of DOMe needs a specific framework for dealing with observation at different levels, by providing the means to store and integrate informations emanating from several sources at lower level, together with mechanisms to abstract and infer interpretations about observed informations collected on real objects at several higher levels. In this paper, DOMe are considered as traces of observations captured and collected about real objects, associated with management and reasoning mechanisms for such traces. Under such consideration, we define a trace-based framework as means of supporting the processing and transformations of traces. We will describe our approach with a fictional example not yet implemented, in order to precise and explain the introduced concepts and demonstrate implicitly the use of our framework to support DOMe applications. The remainder of the paper is organized as follows. Section 2 gives an overview of th basic concepts of our approach and the architecture of our framework. Section 3 describes the fictional example that illustrates the models and transformations that can be applied. Our last section deals with discussion and future work.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. Traces and Trace-Based System The notion of trace as reification of the observation of the interaction of a human with a complex artefact (computer, vehicle) has long interested the SILEX2 team. We have been studying various issues such as the modelling of traces of interactions or their manipulation for knowledge discovery within several application domains (activity reflection and analysis [5,6], decision support and reuse of traced experience [7,8,9,6]). As a means to facilitate traces exploitation within our projects, we have defined the notion of Trace-Based System (TBS) as a kind of knowledge based system whose main source of knowledge is a set of continuously updated modelled traces describing userssystems interactions. We define the trace model as the schema or ontology describing the content of traces, and modelled traces as traces associated explicitly with their model. Assuming that trace-based systems use an abstract architecture as described in Figure 1, we will describe each component by specifying its use to support DOMe. 2.1. Trace-Based System Architecture At the top of the general architecture of a TBS (Figure 1) is the tracing system, which collects the observed data from different input sources (log files, streamed actions, video records, interface events, sensor data, RFID signal, etc.). The Tracing System elaborates so called primary traces (often low level) from tracing sources. A Transformation System allows to infer new trace elements by specifying transformation rules. Transformation rules provide a powerful mechanism to process traces, like 2 http://liris.cnrs.fr/silex

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

L.S. Settouti et al. / A Trace-Based Framework for Supporting Digital Object Memories

41

Figure 1. Trace-Based System Architecture

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

applying filters, rewriting and aggregating elements, computing elements attributes, etc. so as to produce so called transformed modelled traces that can be more easily reusable and exploitable in a given context than primary traces. Transformations can also be considered as semantic abstractions when associated with models (ontologies) of high level describing a specific task or activity. The Querying System enables the mining and extraction of episodes and patterns from the traces. All primary and transformed traces are stored in a persistant trace database. To define and implement interoperable TBS systems, we have formally defined the above framework, specifying languages to model, query and transform interaction traces. Due to space limitations, we cannot describe these languages, but the interested reader can refer to [10]. We rather provide in the next section an example to illustrate that framework and to demonstrate how TBS can support DOMe manipulation and management.

3. Trace-Based System for Digital Object Memories In this section we describe our framework through a fictional example in the context of real-world objects observation traces issued from common technologies such as sensors or Radio-Frequency IDentification (RFID). Both the collected traces and the mechanisms for their exploitation enact the role of digital memories for such objects. 3.1. Running Example Consider the case of a smart home where three of everyday objects are equipped with observation capabilities. As showed in figure 2, let us consider a milk bottle and a baby bottle having both 1/ sensors that transmit bottle temperature measurements and 2/ RFID tags. Several RFID antennas are also integrated into the refrigerator, the microwave oven, within each room (baby room, kitchen, etc.). We assume that all RFID antennas within the smart home are able to read and recognize both milk and baby bottle RFID tags and to send this information to a computer system. We also assume that each transmitted bottle temperature comes with the temperature of the room containing its.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

42

L.S. Settouti et al. / A Trace-Based Framework for Supporting Digital Object Memories

Figure 2. Smart home observation architecture with an example of resulted primary modelled trace

We assume that each object is associated to a trace or memory that stores information from the RFID antenna when an RFID tag activates it or the temperature sensor. Each piece of information about a bottle is associated to a time interval. In the case of temperature, the observation are instantaneous (no duration) delivered every 15 minutes, while RFID-obtained observations last as long as the antenna detects the RFID tag3 .

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.2. Modeling Traces We defined a trace model as a kind of ontology describing the temporality, vocabulary and content of a trace. By considering traces as a support for digital object memories, our idea is to model the observation of a real instrumented object and to constitute the whole history of its life cycle at several abstraction levels, using transformations to “raise” low level memories to a higher level. Each trace model is therefore designed represent useful information at an adequate level, integrating streamed informations emanating from several sources. For instance, in the upper right of Figure 2 the primary trace model for both bottles defines Obsel4 , BottleRFID, BottleTemperature as the basic components of the observation ontology. Each concept corresponds to observations obtained from a sensor or RFID antenna concerns about a specific bottle (baby bottle or milk bottle in our example). A trace corresponding to that model for the baby bottle is illustrated in the lower right of Figure 2. It contains two observed elements. They corresponds to a observation of the bottle being in kitchen during 4’20” at 10h35, and an observation of its temperature being 30◦ C at 19h45 while the temperature in the room is 21◦ C. 3 The preprocessing required to obtain this kind of information could be performed inside the TBS, but we ignore it for the sake of simplicity. 4 “Obsel” is contraction of "OBServed ELement"

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

L.S. Settouti et al. / A Trace-Based Framework for Supporting Digital Object Memories

43

Figure 3. Trace Transformations to improve DOMe

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.3. Querying and Transforming Traces to improve Digital Object Memories TBS allows the expression of pattern-extraction based queries and transfromation rules to infer and detect complex situations. TBS supports rules as an abstraction and reasoning mechanism for traces, with the same motivation and benefits of rules in deductive and knowledge based systems [10]. Based on the primary trace model and a defined model for the transformed trace, a transformation allows to populate a new transformed trace by specifying 1/ the pattern (rule body) describing which elements will be matched to be transformed and 2/ the template (rule head) to generate and infer the new observed elements from the ones retrieved by the patterns. For instance, Figure 3 illustrates how such a transformation, producing the transformed trace T1 from the primary trace T1 , can simply infer higher level information such as BottleLocationChange when two successive BottleRFIDs within T1 do not have the same location. More complex tranformation rules can infer a new observation such as BottleEmpty when two successive BottleTemperatures show a quick convergence of the bottle temperature towards the room temperature. This kind of information can be useful in the memory of baby bottle indicating that the baby bottle is empty and it is left in the baby room. Let us now consider the milk bottle memory: trace T2 in Figure 3 shows how the milk bottle is detected in the refrigerator, then the kitchen then the refrigerator again. Based on a transformation of trace T2 , the TBS can infer within T2 that, since the milk has been left out of the refrigerator for more than four hours, it has probably become unhealthy. Such information can be very useful for the person preparing the baby bottle, especially if s/he is not the one who put the milk back in the refrigerator at 15h29. The TBS can send an email or SMS warning message to home members, indicating that the milk bottle is possibly inappropriate for use in baby bottle preparation.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

44

L.S. Settouti et al. / A Trace-Based Framework for Supporting Digital Object Memories

4. Discussion and future work In this paper, we gave a brief overview of Trace Based Systems (TBS), a framework for managing and transforming modelled traces about observation and use of real life objects. We have demonstrated, with a simple fictional example, how that framework can be used to manage DOMe, using transformations to raise the abstraction level of traces and infer useful knowledge. Conversely to context-aware application frameworks (as [11]) that use context abstraction based on XML technologies, we use ontologies with precise semantics (see [10]) to express more complex context specification and reasoning mechanisms. That approach is actually being developed and experimented in several contexts: eLearning personalisation, personal and corporate knowledge management. Beyond these software-centred user-system interactions, we also used our framework in the study of vehicle driving behaviours, and are developing the use of other sources of observations (eye tracking, sensors, RFID, etc.), especially in the context of accessibility for the disabled. However, our framework has been designed and examplified to manage several DOMes within a centralised architecture. A more in-depth investigation to applying our framework to an open and distributed environment is currently on our agenda.

References [1]

[2]

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[3] [4]

[5]

[6] [7]

[8]

[9] [10]

[11]

Chalermek Intanagonwiwat, Ramesh Govindan, and Deborah Estrin, “Directed diffusion: A scalable and robust communication paradigm for sensor networks”, in ACM International Conference on Mobile Computing and Networking (MOBICOM’00, 2000, pp. 56–67. Levent Gurgen, Claudia Roncancio, Cyril Labbé, André Bottaro, and Vincent Olive, “Sstreamware: a service oriented middleware for heterogeneous sensor data management”, in ICPS ’08: Proceedings of the 5th international conference on Pervasive services, New York, NY, USA, 2008, pp. 121–130, ACM. Sheetal Agarwal, Anupam Joshi, Tim Finin, Yelena Yesha, and Tim Ganous, “A pervasive computing system for the operating room of the future”, Mob. Netw. Appl., vol. 12, no. 2-3, pp. 215–228, 2007. EMJ Koski, T. Sukuvaara, A. Mäkivirta, and A. Kari, “A knowledge-based alarm system for monitoring cardiac operated patients-assessment of clinical performance”, Journal of Clinical Monitoring and Computing, vol. 11, no. 2, pp. 79–83, 1995. Olivier Georgeon, Matthias J. Henning, Thierry Bellet, and Alain Mille, “Creating Cognitive Models from Activity Analysis: A Knowledge Engineering Approach to Car Driver Modeling”, in International Conference on Cognitive Modeling. July 2007, pp. 43–48, Taylor & Francis. Damien Cram, Béatrice Fuchs, Yannick Prié, and Alain Mille, “An approach to user-centric contextaware assistance based on interaction traces”, in MRC 2008, Modeling and Reasoning in Context, 2008. Pierre-Antoine Champin, Yannick Prié, and Alain Mille, “Musette: Modelling uses and tasks for tracing experience”, in workshop From structured cases to unstructured problem solving episodes - WS 5 of ICCBR’03, Béatrice Fuchs and Alain Mille, Eds., Trondheim (NO), 2003, pp. 279–286. Julien Laflaquière, Lotfi Sofiane Settouti, Yannick Prié, and Alain Mille, “A trace-based System Framework for Experience Management and Engineering”, in Second International Workshop on Experience Management and Engineering (EME 2006) in conjunction with KES2006, Oct. 2006. Alain Mille, “From case-based reasoning to traces-based reasoning”, Annual Reviews in Control, vol. 30, no. 2, pp. 223–232, Oct. 2006, Journal of IFAC. Lotfi Sofiane Settouti, Yannick Prié, Pierre-Antoine Champin, Jean-Charles Marty, and Alain Mille, “A Trace-Based Systems Framework : Models, Languages and Semantics”, technical report, LIRIS, University of Lyon, France, UMR CNRS 5205, Université Lyon 1, 2009, http://hal.inria.fr/inria-00363260/en/. Anind Kumar Dey, Providing architectural support for building context-aware applications, PhD thesis, Atlanta, GA, USA, 2000, Director-Abowd, Gregory D.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-45

45

Using object memory patterns to make plan-driven help systems more flexible Damien Cram a,b , Alexander Kr¨ oner a , and Alain Mille b a German Research Center for Artificial Intelligence (DFKI) Stuhlsatzenhausweg 3, 66123 Saarbruecken, Germany b Universit´e de Lyon, CNRS Universit´e Lyon 1,LIRIS, UMR5205, F-69622, France Abstract There are many strategies to assist the user in pervasive environments. In some cases, the user has to achieve a goal according to a plan known by the environment. We address the issue of enabling the environment to be aware of which task of the plan is currently on-going. We present a strategy that makes use of temporal patterns called memory chronicles and that watches events that occur in object memories to recognize if they match any memory chronicle of any task of the plan. We take the example of a smart kitchen where tools and ingredients are considered as objects with memories and where the cook reproduces a recipe according to a recipe plan known by the kitchen. We explain how the kitchen makes use of memory chronicles to applying cooking task recognition and guide the cook through the recipe plan.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Keywords. Memory Patterns, Task recognition, User assistance, Interaction Traces, Chronicles

1. Introduction Objects from the outside world that are equiped with sensors allow to keep track of object-centric interaction traces, and then to reuse these traces. In many cases, the reuse of interactions aims to enable intelligent behaviors and to help its owner [1]. In the context of pervasive environments, some other objects continually try to match events occurring in their environments to a task plan in order to infer in real-time where the user steps in that plan. Smart kitchens [2,3] are such pervasive environments. The smart kitchen of the SharedLife project [4] is equiped with a video screen and sensors (e.g. RFIDs, accelerometers) that are attached to each tool and ingredient of the environment. In that way, each object can store its event story, which makes it an object with a memory. The screen displays information and instructions to the cook. To provide the user with help, the smart kitchen is beforehand given a list of linear recipe plans that can be realized again by new cooks. When a cook enters the smart kitchen, he selects a recipe plan he wants to realize. Then the kitchen keeps up to date the knowledge of which tools enter and leave the

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

46

D. Cram et al. / Using Object Memory Patterns to Make Plan-Driven Help Systems More Flexible

cooking workbench and infers in real time which step of the recipe plan the cook has to do next. As showed in [5], this design brought notable help to users of the smart kitchen, but it also suffers from some inconvenient strictenesses. Flexibility needs to be brought into the system. The main strictness is that the cook has to stick rigorously with the given order of tasks in the recipe plan. Some tasks in cooking activities, as in most of other activities, are actually allowed to be done independently from some others. As a result, the cook was also forced to look both at the workbench and at the screen alternatively so as to ensure that he performs the task the right way. In this paper, we propose that the recipe plan is no longer a fixed sequence of tasks, but instead a more flexible structure allowing some recipe tasks to be done according to any order (cf. section 3). Section 4 explains how the smart kitchen is still given the ability of infering where in the recipe plan the cook steps thanks to memory chronicle recognition, despite that the smart kitchen can no longer rely on a unique order of tasks in the recipe plan.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. Definition of object memory and memory chronicle Object memory. In this paper, we define an object memory as follows. Given a finite set of event types E, an object memory is a sequence of events, where each event is a pair (ε, t) such that ε ∈ E and t is an integer called the date. For example, given an object bread-knife in the smart kitchen and E = {bk-appears, bk-disappears}, bread-knife could have the following memory in the smart kitchen: M0 = (bk-appears, 25)(bk-disappears, 210)(bk-appears, 220)(bk-disappears, 300). ( bk stands for bread-knife) The memory M0 can be interpreted as follows:“ bread-knife appeared into the workbench at date 25 (in the smart kitchen, each date refers to the number of seconds since the cooking session started) and disappered at date 210. Then it appeared again at date 220 and disappeared again at date 300”. An interaction trace merges memories from different objects that are used in the context of the same activity, e.g. in the context of the smart kitchen. In the smart kitchen, several digital objects are handled by the user, e.g. bread-knife, cutting-board, bread. . . For example, considering only two digital objects: bread-knife and bread, the interaction trace of the smart kitchen could be: T0 = (bk-appears, 25)(b-appears, 27)(b-disappears, 100)(b-appears, 110)(bk-disappears, 210) . . . ( b stands for bread) Event types about appearance into the workbench or disappearance are typically those collected from RFID sensors but several other event types, e.g. start-mixing or turn-oven-on, can be collected as well. Memory chronicles. We use chronicles [6] to model patterns that can occur in object memories, that is why we will refer to them as memory chronicles. Memory chronicles are structures that represent a typical or recurring episode in an object memory. In the case of the smart kitchen, each memory chronicle represents a cooking task. Defining memory chronicles rigourously would be too long and out of scope of this paper. Basically, a memory chronicle specifies which events must appear in the memory and gives time constraints on these events.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Cram et al. / Using Object Memory Patterns to Make Plan-Driven Help Systems More Flexible

47

For example, the figure opposite shows a memory chronicle that represents any subsequence that is composed of two bk-appear and one bk-disappear, where bk-disappear occurs whenever between 100 and 200 seconds after the first bk-appear and 10 to 20 before the second bk-appear. There is only one occurrence for this memory chronicle in the memory M0 : (bk-appear, 25)(bk-disappear, 210)(bk-appear, 220)

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3. The hyper recipe model of the smart kitchen A hyper recipe is a recipe in which each task is not only described textually, but also formally and with an attached video. A hyper recipe is composed of a recipe plan and a task map. (cf. Figure 1) The recipe plan is similar to a workflow that defines all tasks to be done and their inter-dependencies. Existing workflow models, e.g. Petri Nets and its extensions, are very powerful to express a wide range of task dependencies. However, such formalisms are too complex to be handled by end-users. For this reason, we define a recipe plan as a sequence of parallel tasks. No task composition is allowed. For example on Figure 1, the recipe plan in brackets is non-valid since we could define a compound task G as the sequence of subtasks task B and task D. Finally, the task map associates each task defined in the recipe plan with a set of resources that are used by the flexible assistant (cf. section 4): a textual explanation, a video stream that shows how the author of the recipe performed that task, and a memory chronicle. The memory chronicle describes the pattern of events in object memories that results from the realization of that task. We keep the global architecture of sharing hyper recipes over a community that was introduced in [4]. This architecture allows two roles in particular: contributor and reuser. The role contributor consists in adding a new recipe to the shared recipe database: the user can create and post an hyper recipe to the community. Figure 2 illustrates the role contributor. A cook member who wants to publish a new hyper recipe to the community starts with cooking that recipe in the smart kitchen under video recording (cf. step a). Then the cook starts a reflexive phase (cf. step

Figure 1. An example of a hyper recipe. The task map associates task A with Video A, Memory chronicle A and Textual description A, and so on with task B to task E. Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

48

D. Cram et al. / Using Object Memory Patterns to Make Plan-Driven Help Systems More Flexible

Figure 2. Role contributor : creating a hyper recipe and adding it to the shared recipe database. Role reuser : realizing a hyper recipe selected from the share recipe database with the help of an assistant.

b) where he reuses all materials (object memories and the video stream) that came out of the cooking session to make a hyper recipe ready to be published at step c. The role reuser consists in searching the shared recipe database for a hyper recipe to reproduce (cf. Figure 2). Once the reuser cook starts to reproduce the hyper recipe, the assistant uses memory chronicle recognition to infer which task is currently on going and helps the cook.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. The task recognition-based assistant of the smart kitchen The assistant tries to match current events occurring in object memories with one of the memory chronicles in the task map. If a memory chronicle matches the current events, then the assistant provides the reuser cook with help like displaying the video segment for that task. Recognizing all occurrences of a memory chronicle in a memory is a challenging issue that was adressed in [6] through the algorithm CRS (Chronicle Recognition System). CRS keeps for each memory chronicle a set of hypotheses up to date while time passes and new events occur. Each hypothesis is a partially recognized memory chronicle. Basically, an hypothesis is composed of a set of events that have been recognized and a set of event types that are still to be recognized. Figure 3 explains this concept with the example of the memory chronicle Ctomatoes . The flexible kitchen assistant must guess which task the cook started before that task comes to end. In our approach, it consists in choosing the best memory chronicle among a set of candidate memory chronicles. To do that, we define a very simple scoring function as follows: fscore (C) = “the ratio of recognized event types in the current hypothesis”. Figure 3 gives fscore (Ctomatoes ) = 4/8 = 50%. Thus, we address the partial recognition problem by setting a threshold recognition

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Cram et al. / Using Object Memory Patterns to Make Plan-Driven Help Systems More Flexible

49

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 3. Example of a hypothesis. This figure assumes that the cook is currently realizing his recipe and that the current time is 46. The sequence shows the events that occurred in the smart kitchen so far. The memory chronicle showed is the one that was defined for the task “cutting tomatoes”. It is composed of 8 events regarding 4 objects. It defines the task as adding the tomatoes ( t-appear) to the workbench and removing them 30 seconds to 200 seconds after, and so does it for objects cutting-board, bread-knife and bowl. So far at date 46, CRS has recognized 4 events of that memory chronicle and has formulated a hypothesis with respect to the temporal constraints of the memory chronicle, e.g. since G must occur from 30 to 200 after C, and since C occurred at date 4, G is then expected between now and date 204.

Figure 4. Screenshot of the flexible recipe assistant that is displayed on the screen of the smart kitchen while the user is cooking. The left part shows the recipe plan, which has one example of parallelism: tasks Tomatoes, Onions, and Red pepper can be realized according to any order. Tasks that have been fully realized are painted in red, those who have not been started are in green, and those who are currently on-going in orange. On that example, our partial recognition system recognized that the user is currently working on task Tomatoes and automatically displayed on the right part the instructions for that task and the video of how the task was performed by the author.

ratio, e.g. 40% and by assuming that the cook is realizing the highest rated task, if there are some over the threshold. (cf. Figure 4 for a screenshot of the help provided) Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

50

D. Cram et al. / Using Object Memory Patterns to Make Plan-Driven Help Systems More Flexible

5. Conclusion The way we address the issue of “making plan-driven help systems more flexible” implies that the plan on which the help is based allows that flexibility. Since the resulting plan does not have anymore imposed order of tasks, we introduced the notion of memory chronicle partial recognition to match actions currently occurring in object memories to tasks in the recipe plan. The system has been implemented and pre-tested with “fake cooks”, i.e. by replaying actions of past recorded cooking sessions. Our first observation reported that our chronicle recognition system can detect almost every time the right task that is currently ongoing, on condition that several memory chronicles are associated to each task rather than only one, otherwise it appears sometimes that some occurrences of the task cannot be covered by a single chronicle. User studies such as [5] need to be led to observe and measure if the new system actually relaxes constraints and eases the cooking process, to cooks’ opinions. In the case of the smart kitchen, we can think of even more flexible ways of reusing hyper recipes, like a free mode in which the reuser selects no recipe and cooks his own one. In such a mode, the smart kitchen would match current actions with tasks in all existing recipes. But our next research question concerns the design and implementation of an interface that supports the reflexive phase and gives the contributor cook the possibility to easily find pertinent memory chronicles. In particular, we are working on a design that uses the discovery of chronicles from interaction traces [7] to make the definition of pertinent chronicles more automatical and hence easier.

References

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[1]

[2]

[3] [4]

[5]

[6] [7]

Plate, C., Basselin, N., Kr¨ oner, A., Schneider, M., Baldes, S., Dimitrova, V., Jameson, A.: Recomindation: New functions for augmented memories. In Wade, V., Ashman, H., eds.: Adaptive hypermedia and adaptive web-based systems: Proceedings of AH 2006. Berlin, Berlin, Germany, Springer (2006) Chi, P.y., Chen, J.h., Chu, H.h., Chen, B.Y.: Enabling nutrition-aware cooking in a smart kitchen. In: CHI ’07: CHI ’07 extended abstracts on Human factors in computing systems, New York, NY, USA, ACM (2007) 2333–2338 Terrenghi, L., Hilliges, O., Butz, A.: Kitchen stories: sharing recipes with the living cookbook. Personal Ubiquitous Comput. 11(5) (2007) 409–414 Schneider, M.: The semantic cookbook: Sharing cooking experiences in the smart kitchen. In: Proceedings of the 3rd International Conference on Intelligent Environments, Ulm , Germany., IET (2007) 416–423 Jacobs, O., Kr¨ oner, A., Schneider, M.: Interaction with the digital memory of a smart kitchen. In Baehren, T., Czogala, D., Gehring, C., Klausing, H., Wahlster, W., eds.: Proceedings of the 2nd German Congress on Ambient Assisted Living (AAL 2009), Berlin, Germany, VDE-Verlag: Berlin, Offenbach (January 2009) Dousson, C., Gaborit, P., Ghallab, M.: Situation recognition: Representation and algorithms. In: IJCAI. (1993) 166–174 Cram, D., Cordier, A., Mille, A.: An Interactive Algorithm for the Complete Discovery of Chronicles. Technical Report RR-LIRIS-2009-011, LIRIS UMR 5205 CNRS, University of Lyon. Available on-line: http://liris.cnrs.fr/publis/?id=3897. (April 2009)

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-51

51

Querying Sensor Data for Semantic Product Memories Christian Seitz a,1 , Christoph Legat a , and Jörg Neidig b a Siemens Corporate Technology, Intelligent Autonomous Systems, Munich, Germany b Siemens Industry Sector, Advanced Technology and Standards, Nuremberg, Germany

Abstract. Today, RFID is used to identify a wide range of work pieces or individual products for tracking their movements through the logistics chain. For future purposes the idea of storing only a single ID must be extended to a Semantic Product Memory. This memory stores data of the complete product life cycle. This paper introduces a middleware for accumulating semantic product memories with sensor data. Our contribution encompasses software modules for a uniform sensor access, a sensor data ontology and a query interface for semantic product memory applications.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Keywords. Sensor middleware, Sensor data ontology, Query language

1. Introduction Today, RFID is used to identify a wide range of work pieces or individual products for tracking their movements through the logistics chain. For future purposes the idea of storing only a single ID must be extended to a Semantic Product Memory. This memory stores data of the complete product life cycle and is embedded in the product itself. But the product memory is not only a passive data storage, it is also able to control and manipulate its environment by communicating with other products. Integrating semantic memories to products result in shorter product and innovation cycles, more complex logistics chains, and a product driven production process is possible. Finally, the semantic product memory opens up new dimensions for protection from product piracy, consumer protection and product liability. A key problem of semantic product memories is their cross domain nature, i. e. new relevant data must be added to the memory from various stakeholders during the complete product life cycle. This paper introduces a middleware for accumulating semantic product memories with sensor data. Our contribution encompasses software modules for a uniform sensor access, a sensor data ontology and a query interface for semantic product memory applications. The paper is organized as follows. The next section presents existing approaches for sensor middleware architectures and sensor data modelling. Chapter 3 presents our architecture and describes its components in detail. The next section introduces the sensor meta model, which consists of several ontologies. The paper concludes with a summary and a future outlook. 2. Related Work Various middleware approaches for sensor networks, with the goal to provide a unified access to sensors, are existing. The authors in [11] discuss challenges for sensor middleware approaches, but their analysis is limited to sensor networks. Our approach must 1 Corresponding Author: Christian Seitz, Siemens AG, CT IC 6, Otto-Hahn-Ring 6, D-81739 Munich, Germany; E-mail: [email protected]

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

52

C. Seitz et al. / Querying Sensor Data for Semantic Product Memories

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

also include sensors which are part of existing systems, e. g. automation systems. Kumar et al. [8] present a sensor middleware with autonomous services. These services deal with basic sensor network functionalities like a status monitor or monitoring quality of services issues. Although a uniform access to heterogeneous systems is part of the system, no sensor modelling is included in their approach, which would have made their approach even more flexible. In addition to creating an interface that realizes a uniform access to get sensor values, the configuration of sensors has also taken into account. Grace et al.[3] consider reconfiguration of the network layer. We suggest that not only the communication infrastructure needs to be reconfigured, there must also be a possibility to change the sensor parameters (e. g. measuring range) as well. To achieve a ubiquitous access to relevant product data, it is not sufficient to create a uniform sensor interface, additionally a machine-comprehensible semantic description languages for product memory data is needed. One example is the Context Ontology Language (CoOL) [12] which enables a detailed description of observed information relating an entity to an observed symbol or value-range which in turn belongs to a specific aspect of the environment. Beside the support for metrological information, it enables an expressive notion of quality aspects but lacks considerably in the description of entities itself as well as in temporal and spatial aspects. The SWEET ontologies [10] developed by NASA provide an important knowledge summary of physical and environmental concepts. While the strength of metrological descriptions of environmental phenomena is unequaled, it lacks in the support for context information as environmental observations and provides neither support of quality descriptions nor exact definitions of entities. Another well known ontology for pervasive applications is SOUPA [1] designed for intelligent pervasive agents. It follows a human-centric approach causing weakness in the expressibility of metrological and qualitative aspects but provides a broad sum of concepts related to spatial and temporal aspects. It can be summarized that no existing ontology provides a conceptual model to describe sensor data to realize semantic product memory applications. 3. System Architecture Semantic product memories must be enriched with sensor data during the product life cycle. It should be possible to query sensor data in an abstract and uniform way, e. g. get the room temperature of the engine room. This functionality makes our solution more powerful than existing approaches. In this section the general architecture for collecting and querying sensor data for semantic product memory applications is presented. The architecture is depicted in figure 1, and in the following paragraphs each component is explained in detail. Sensors: The sensor layer is the point of interest of our work, because sensor data has to be made accessible in a uniform way. Currently, we integrate data from sensors, measuring e. g. temperature, acceleration, or pressure. Additionally, sensors from the automation domain are supported, e. g. gas sensors for detection of carbon monoxide, light barriers, RFID reader, or distance sensors. We also support wireless sensor and are able to integrate Bluetooth sensors and sensors which are compliant to the 802.15.4 standard. Sensor Proxy: The proxy concept is the foundation concept of the sensor middleware. It can be seen as an abstraction for wrapping heterogeneous sensing devices and providing unified access to sensor measurement and configuration, hiding the details of raw data acquisition and sensor configuration from the higher-level components. A proxy represents either a single sensor or a collection of sensors, e. g. from a sen-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

C. Seitz et al. / Querying Sensor Data for Semantic Product Memories

53

P r o d u c t M e m o r y A p p lic a tio n Q u e ry /R e s p o n s e

C o n fig u r a tio n R e g is te r

S e n s o r R e g is tr y

Q u e ry In te rfa c e R u le E n g in e

In fo r m a tio n B ro k e r

K n o w le d g e B a s e

R e a s o n in g E n g in e

S u b s c r ib e / U n s u b s c r ib e

S e n s o r d a ta

S e n s o r P ro x y

S e n s o r P ro x y

S e n s o r P ro x y

S e n s o r

S e n s o r

S e n s o r

1

2

N

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 1. Architecture of the sensor acquisition and querying system

sor network. The proxy integrates all devices, manages them internally and represents them to the outside as one logic sensor network. The proxy registers its sensors with the sensor registry. To enable a unified access to heterogeneous sensing devices, the sensor proxy provides methods for accessing raw sensor data, for configuring individual sensor devices and for monitoring their state. Access to sensor data is supported in two ways. First, data can be requested from the proxy. Second, other components can subscribe to changes of sensor measurements. Because the proxy should be usable by multiple applications or components at the same time, it has to coordinate competing data requests and subscriptions internally. In addition, potentially large amounts of raw data should be processed in an application independent and configurable way. Sensor Registry: Components that should be discoverable can register with the registry based on a unique identifier a component type and various attributes. Especially, sensor type, quality information of sensors, and the location of components are represented as attributes. The main task of the registry is to perform a mapping from the sensor type and other attributes to a unique identifier. The identifier must provide information about the sensor proxy and the specific sensor. Information Broker: This component is responsible for data dispatching in the proposed architecture. It receives configuration data, which consists of a specification of sensor data needed for an application. By using the sensor data ontology a transformation from sensor data to a specific sensor is achieved. In order to identify the corresponding proxy, the information broker needs information from the sensor registry. Furthermore, the information broker receives sensor data from all sensor proxies. This data is shifted to the knowledge base. The information broker consists of a reasoning subcomponent and a rule engine, which are both needed for reasoning and querying. Reasoning is necessary to analyze existing data and derive new, formerly hidden data. The rule engine additional executes the defined rules supporting functionality not realizable with description logic based inferencing. Configuration: The configuration component prepares the sensor infrastructure for a specific product memory application. A configuration is needed on two levels. At first, it is necessary to configure the sensor data which is needed for an application and the intervals, the data is provided. Additionally, it is necessary to specify which data a sensor proxy should return to the information broker. A sensor proxy can either returns raw sensor data, or transformed data. For this purpose the proxy consists of cascadable interpretation modules, that perform basic conversions on the raw data.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

54

C. Seitz et al. / Querying Sensor Data for Semantic Product Memories

Query Interface: The querying of sensor data is enabled by the SPARQL Query Language for RDF. It provides a standardized way to express queries across RDF data, stored in the knowledge base. The implementation of the sensor data as an ontology implemented in OWL enables to use SPARQL as query language. Sensor data can be expressed with the help of the sensor data ontology proposed in section 4. Knowledge Base: Accumulated sensor data is passed from the information broker to the knowledge base, where they are stored in a semantic data base or triple store. Unlike a relational database, a triple store is optimized for the storage and retrieval of many short statements called triples, in the form of subject-predicate-object, which is basically the format of RDF (Resource Descriptions Format). There are some open source tools available (e. g. Joseki or Sesame), but since such tools are rarely used in commercial applications, we use the semantic power of Oracle, which delivers an advanced semantic data management capability as part their data bases. With native support for RDF/OWL standards, this semantic data store provides an open, scalable, secure, efficient platform for RDF and OWL-based applications. It enables storing, loading, and access to RDF/OWL data and ontologies, inference using OWL semantics and user defined rules, querying of RDF/OWL data and ontologies. Since reasoning heavily depends on the number of entries in the knowledge base, for performance reasons the configuration contains a maximum age of sensor data, stored in the data base to control the deletion of old values or confines the amount of triples to an upper threshold. Application: On top of the architecture applications are built. At first it must be specified which sensor data is needed for the application. This configuration is read by the information broker, which performs a mapping from the sensor data to a corresponding sensor. For this step the sensor data ontology is needed, which is presented in the next section. Additionally, the information broker retrieves the necessary sensor proxy information from the sensor registry. After the configuration the sensors transfer their data to the associated proxies which forward the data to the information broker. The sensor data is stored in the knowledge base. Since the knowledge base has a unified view on all sensor data with reasoning methods new data can be derived. When the application needs to integrate sensor data into the semantic product memory, the query interface is used and a SPARQL query is formulated. This query is passed to the information broker, which uses the reasoning modules to answer the query with the data stored in the knowledge base. Finally, the response is returned to the application by the query interface and the corresponding data can be integrated in a semantic product memory. 4. Sensor Data Model The goal of much of the research in the field of context-awareness and ubiquitous computing is to provide higher level abstractions of complex low-level concepts easing the design and implementation of applications. The usage of sensor data within such applications is mostly achieved by a uniform interface on sensors abstracting their heterogeneous communication and hardware to ease the connection and usage of sensors. But in fact, the applications are interested in the information provided by the sensors and not in the sensors itself why most approaches lack in their benefit caused by the heterogeneity of sensor data. For this reason, we decided to shift the abstraction level to the layer of data integration in order to enable product memory application developers to focus on the key features of their applications by utilizing the information provided by a uniform data interface. Whereas the communication and interface heterogeneity of sensors can be overcome by abstracting sensors with a middleware-like architecture, the semantic and schematic heterogeneity of delivered data cannot be solved in this way. The research of information in-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

55

C. Seitz et al. / Querying Sensor Data for Semantic Product Memories  o b s : h a s P ro p e rty

F e a tu r e O n to lo g y F e a tu re

 o b s : h a s F e a tu re O fIn te re s t

(N a m e s p a c e : o b s )

P r o p e r ty O n to lo g y  o b s : h a s O b s e rv e d P ro p e rty

P ro p e rty

O b s e r v a tio n

F e a tu re O fIn te re s t ...

O b s e r v a tio n O n to lo g y

...

 o b s : h a s Q u a lity

 o b s : h a s S o u rc e  o b s : h a s T im e

L e g e n d : O n to lo g y

T im e O n to lo g y T e m p o r a lE n tity

C o n c e p t R e s tr ic tin g R o le

...

Q u a n tity O n to lo g ie s

U n it O n to lo g y

u s e s

M a th O n to lo g y

u s e s

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 2. Structure of the upper ontology layer and its main ontologies to describe observations

tegration has identified the use of ontologies as appropriate remedy to integrate different data of different sources and enables the handling of the latter two heterogeneity problems by the explication of implicit and hidden knowledge [4]. The precise and formalized nature of ontological knowledge encapsulation enables the generation of additional knowledge not explicitly given within the knowledge base which can be used directly by the product memories reducing the effort of their mostly limited computing resources. We use a description logic based two-layered sensor model architecture extended with DL-safe [9] rules consisting of a domain-independent upper ontology layer and subjacent domain ontology layer to balance interoperability between different domains and often highly domain-specific formalization of knowledge. The upper ontology layer encompasses a few ontologies of different concerns as shown in figure 2 to enable a flexible design and reducing the complexity for development and reasoning. The macro-structure of the upper ontology described in the following is embossed by a simplified but expressive view on the physical world. Entities like robots, machines or the product itself are faced to their imputable aspects like temperature or speed. The observation procedure to detect this relationship is done by sensors. This can be expressed by using the Observation Ontology which is inspired by [2]. It is the principal part of the upper ontology layer, which defines the Observation concept and its relationships. An observation assigns an entity, called feature of interest, to one of its aspects called properties with the roles hasFeatureOfInterest and hasObservedProperty. A feature of interest is a feature being the object of interest of an observation in accordance to the ISO standards [6,5]. The Feature Ontology encapsulates knowledge about features and enables time dependent and independent reasoning on features with standard reasoning engines to support the detection and generation of additional knowledge. A semantical exhaustive description of properties achieved by a generic meta-model is the concern of the Property Ontology. It is expanded by a summary of ontologies for quantifiable properties (quantities) which are separated by concern of science into different ontologies in a recursive hierarchical manner. The strong mathematical grounding of quantities assisted by the Math Ontology enables their automated computing and precise distinction e.g. between speed and velocity. The ontology for physical quantities is the most important one of them to describe low-level sensor information and is completed by an ontology for units of measurement which enables reasoning on the different dimensions of units. With a view to the effort of abstracting sensors by focusing on the information provided by them, some additional information beside the description of features of interest and their properties are necessary to represent all facets of an observation: the quality, the source and the time. The qualitative information is indispensable if sensors should be abstracted because e.g.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

56

C. Seitz et al. / Querying Sensor Data for Semantic Product Memories

correctness, granularity or resolution of observed data are highly dependable on the sensor hardware and has to be presented within the data. This enables reasoning about inconsistencies, the detection of faulty sources and provides a selection criterion for data requests. Quality aspects of an observation are integrated with the help of the hasQuality role linking to a property because it can be seen as the property of an observation and therefore it can be modelled in the same way as properties of a feature of interest. The annotation of an observation with a description of its source providing the respective observation enables on the one hand the detection of faulty sources. On the other hand it offers the possibility to describe processes used to compute an information e. g. by a transducer or a smart sensor node to reason relationships about information correlations and enable the detection of damaged sources. Sources, either physical or abstract, are entities within the physical world and can therefore be modelled in the same way as any other entity as feature of interest. This is expressed by the role hasSource. For a comprehensive and formal view on the structure of the upper ontology layer and its conceptualizations in detail see [7]. 5. Conclusion and Future Work This paper presents a middleware for accumulating semantic product memories with sensor data, which uses a sensor data ontology to achieve a high level querying functionality. The two key concepts are a unified interface and a sensor data ontology. The former is responsible for accessing and configuring sensors in a uniform way, the latter allows to perform high level queries on the sensor data. In the future we plan to integrate a semantic sensor and sensor data discovery in the system architecture. This allows a dynamic querying of sensor data without predefined subscription or storing data in the knowledge base. Acknowledgements This research was funded in part by the German Federal Ministry of Education and Research under grant number 01 IA 08002 G. The responsibility for this publication lies with the authors.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] H. Chen, F. Perich, T. Finin, and A. Joshi. Soupa: Standard ontology for ubiquitous and pervasive applications. In Proceedings of Mobiquitous 2004, 2004. [2] S. Cox. Observations and measurements – part 1 – observation schema. OpenGIS Implementation Standard, Open Geospatial Consortium Inc., December 2007. [3] P. Grace, D. Hughes, B. Porter, G. Coulson, and G. Blair. Middleware support for dynamic reconfiguration in sensor networks, 2007. [4] H. Wache et al. Ontology-based integration of information – a survey of existing approaches. In Proceedings of the IJCAI-01 Workshop on Ontologies and Information Sharing, 2001. [5] ISO - Int. Org. for Standardization. ISO 19109 - Geographic information – Rules for application schemas, 2005. [6] ISO - International Organization for Standardization. ISO 19101 - Geographic information – Reference model, 2002. [7] C. Legat. Sensor-driven context provisioning for smart environments. Diploma thesis, Ludwig-Maximilians-Universität München, Munich, Germany, December 2008. [8] K. Modukuri, S. Hariri, N. V. Chalfoun, and M. Yousif. Autonomous middleware framework for sensor networks. In Proceedings of the International Conference on Pervasive Services, 2005. [9] B. Motik, U. Sattler, and R. Studer. Query answering for owl-dl with rules. In The Semantic Web - Proceedings of the third International Semantic Web Conference (ISWC 2004), 2004. [10] R. Raskin. Semantic web for earth and environmental terminology. In Proceedings of the Earth Science Technology Conference (ESTC 2003), Maryland, USA, 2003.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

C. Seitz et al. / Querying Sensor Data for Semantic Product Memories

57

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[11] K. Römer, O. Kasten, and F. Mattern. Middleware challenges for wireless sensor networks. ACM SIGMOBILE Mobile Computing and Communication Review, 6, 2002. [12] T. Strang, C. Linnhoff-Popien, and K. Frank. Cool: A context ontology language to enable contextual interoperability. In Proceedings of the International Conference DAIS, 2003.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

1st International Workshop on RFID Technology: Concepts, Practices & Solutions (RFID’09) Andrés García Higueras University Castilla la Mancha, Spain

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Julio C. Encinas Alvarado University of Castilla-La Mancha, Spain

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-61

61

Design of a RFID based traceability system in a slaughterhause A.M. López, E. Pascual, A.M. Salinas, P. Ramos and G. Azuara Electronic and Communications Engineering Department. Polytechnic University School of Teruel. University of Zaragoza. Spain

Abstract. The preliminary design of a RFID (radiofrequency identification) based traceability system that will work in a slaughterhouse is presented. One of the aims of this system is to guarantee the quality requirements established by the CRDO (Consejo Regulador de la Denominación de Origen or the governing board of the protected designation of origin) “Jamón de Teruel” in its regulation. The information will be uniquely attached to the meat by means of RFID tags. The special adverse environmental conditions and the economic restrictions will strongly influence our design. Keywords. Radiofrequency identification, traceability, food chain.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction Traceability has been defined as the ability to trace the history, application or location of that which is under consideration[1], or as the ability of the retrieval of the history and use or location of an article or an activity through the registered identification [2]. Traceability systems are mandatory in the European Union since January 1, 2005 for all activities related to food production and distribution. The aim is to guarantee the quality, safety and security. Every element that takes part in the food chain must be perfectly identified and its history must be perfectly known at all stages [3]. Traceability is one of the main applications of the RFID technology that uses a wireless link to transfer data between a reader and a transponder by means of a modulated radio frequency carrier signal. Each of these transponders or tags contains a unique identification number that will unambiguously identify any element to which is attached. All the information related to this item will be saved in a database associated to this number. Any new information can be physically stored in the traced element inside the memory tag and data can also be retrieved at any point where a RFID reader is placed. This process of reading and writing information requires no contact or perfect alignment between tag and reader, so, just placing the tag inside the communication range, the new information can be written or read in a fast and easy way. The RFID identification of pigs is not mandatory in the EU as in the case of goats, sheep[5] or cows[6], but several examples of RFID based traceability systems applied to the production of by-products from pigs can be found in the case of living animals [7-8] and also in the production of meat and ham[9-10]. In this paper, the traceability system will control the parameters of the slaughter process whose value will determine if the ham produced deserves the quality brand “Jamón de Teruel”. The system should be

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

62

A.M. López et al. / Design of a RFID Based Traceability System in a Slaughterhause

completed including the previous information (birth and feeding parameters) and the information related to the consequent salting and drying process. In the following, the slaughter process and the information required by the CRDO at every phase will be described. Afterwards, the architecture of the proposed system and the memory organization of the tag that have been designed from these data, are presented. This work ends with the description of the future steps that will conclude the project.

1. Problem description The main aim of this project is to automate the information´s collection that the CRDO (Consejo Regulador de la Denominación de Origen / governing board of the protected designation of origin) “Jamón de Teruel” requires to guarantee the quality of the ham (pork leg) identified by this denomination. The animals, whose meat will be sold under this quality brand, must fulfill a number of specifications from the moment they are born till their entrance into the slaughterhouse. Further requirements are checked at different positions in the abattoir and at the same time extra information about the piece of meat is collected. Once the pork leg is taken apart in the slaughterhouse, it must suffer a drying process under specific conditions of temperature, humidity and salt content. Our work is to implement the part of the traceability process corresponding to the slaughterhouse. In the following, this process is briefly described as well as the information that must be checked and collected according to the CRDO internal regulations[11].

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1.1. Implementation scenario The traceability system is going to be developed in the slaughterhouse of the company Pelbor SA in Calamocha, province of Teruel, Spain. This company is associated with the CRDO “Jamón de Teruel”. For this reason, there is always an employee belonging to the CRDO controlling the slaughter process. This person will be named “watcher” and will be the one in charge of the traceability system. When the pigs arrive at the slaughterhouse, the group of animals coming from the same feeding farm, and so belonging to the same batch, are enclosed inside a yard and set aside in this way from the pigs belonging to any other different batch. The watcher at this input point checks the documentation associated to every batch. Afterwards, the animals are conducted from the yards to the slaughter chain. The steps of this process are: •

Step 1: Slaughter after being stunned with CO2.

•

Step 2: Scalding process.

•

Step 3: Peeling.

•

Step 4: Viscera are taken away.

•

Step 5: The dressed carcass is slit open.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A.M. López et al. / Design of a RFID Based Traceability System in a Slaughterhause

•

Step 6: Watcher’s check point.

•

Step 7: Airing chamber.

•

Step 8: Cooling chamber.

•

Step 9: Quartering room.

•

Step 10: Cooling chamber again.

•

Step 11: Delivery/ Distribution.

63

1.2. Information to be collected

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The watcher must confirm at the yards, that the batch information is correct (feeding farm of origin and number of animals of this batch). Every batch is uniquely identified by a batch number[12]. Afterwards, when this operator works at his check point (Step 6) he already knows how many pigs are going to be slaughtered and from which farms they come from. At the check point, the watcher has a PC connected to the CRDO database. It must be said that an internal communication network is already working at the slaughterhouse. The watcher looks at the two back legs from the animal and must collect the following information that will determine if the pork leg fulfills the CRDO’s requirements: •

Weight of every animal. The value must be above 86 Kg. This value is automatically stored from a scale connected to the PC.

•

Fat thickness. The value must be inside the range from 4 to 7 cm. So far, the watcher must estimate if this requirement is fulfilled just by visual inspection.

•

Correct genre: castrated males or females not on heat.

•

Lack of bruises, breakages or other defects in the meat.

If the watcher proves that the animal passes all these conditions, he marks it, using ink valid for food applications, with a specific number composed of ten digits: Digit 1: Identifies the slaughterhouse. Digits 2-3: Week of slaughter. Digits 4-6: Identify the feeding farm. Digits 7-10: Ordering (rol) number. This number uniquely identifies the animal among all the pigs coming from the same breeding farm and slaughtered the same week. This number, easy to notice, indicates that that piece of meat is suitable for the CRDO so far. The watcher also introduces this information to the database through the computer, using the screen as an input interface. Finally, at the quartering room, every leg of pork is controlled to verify: •

Weight. The value must be above 11.6 Kg.

•

Lack of bruises, breakages or other defects in the meat.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

64

A.M. López et al. / Design of a RFID Based Traceability System in a Slaughterhause

2. Design of the traceability system 2.1. Block diagram Four PC terminals are going to be displayed in the abattoir. The first one at the yards, the second one will be the one already placed at the watcher’s check point, the third one at the quartering room, and the last one at the exit of the building (at the point where the pork legs are delivered). All of them will be connected to the internal communication network. These are the points where the information is going to be collected. To guarantee the traceability, some of this information is going to be attached to every leg of pork by means of a RFID tag. This tag, whose characteristics are going to be detailed below, is going to be read/written with RFID readers connected to PCs 2, 3 and 4. Two scales, one connected to the PC at position 2 (already in use) and other connected to the PC at the quartering chamber, will complete the necessary hardware.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2.2. RFID hardware The presence of a high humidity concentration, recommends the use of the HF band, 13.56 MHz, for the RFID elements (ISO 15693). One tag is attached to each half of the animal at the watcher’s check point if the piece fulfills the CRDO requirements. This step must be done by the watcher, because is the only employee of the CRDO present at the slaughter. The process must be fast because of the speed of the slaughter chain. The tag cannot affect the quality of the meat. For all these conditionings, adhesive HF tags have been chosen. The adhesive material must be suitable for being used with meat products. To generate this tag, a RFID printer will be connected to the PC. In this way, the information collected by the watcher will be stored on the tag and the specific number, marked with ink, can be also printed on the tag surface. A peeling tool will ease the watcher’s work. The printer is an AD Monarch 9855 HF and the tags NPX icode SLI with a memory size of 1024 bits. At the quartering room, a second RFID reader will be connected to the PC. The weight of the pork leg will be also stored on the previously attached tag. In this case, a reader SkyeModule M2 has been selected. Finally, a third reader similar to the previous one will be available at the output point. The information to be stored in this case will relate the ham with the drying installation. 2.3. Information storage In this part, the information collected at each terminal is detailed. PC 1: yards. The watcher must provide the feeding farms identification numbers and the number of animals coming for each of them.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A.M. López et al. / Design of a RFID Based Traceability System in a Slaughterhause

65

PC 2: Watcher’s check point. The tag is generated and attached at this point. The steps of scalding and peeling could destroy this element. The watcher selects on the PC which batch is being slaughtered at that moment. The number of animals corresponding to every batch is used by the application to detect possible errors. The watcher can change at any moment the batch. When the animal, divided into two parts, arrives to this point: • • • • •

The scale weights the pig and sends this value to the computer. If the weight is out of the valid range, the application will not generate a new tag. The watcher, using the screen, indicates if the rest of requirements are fulfilled. If one faults, the tag will not be generated. The RFID printer sends to the computer the identification number of the two tags that are going to be written with the information collected. In this way, in the database every piece is identified in a unique way. The program calculates the ordering (rol) number. The RFID reader creates two identical tags. The information stored is: o Feeding farm identification number. o Slaughterhouse identification number. o Slaughter week. o Rol number. o Weight of the animal. This information is also saved in the database together with the identification numbers of the tags.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

PC 3: Quartering room. Once taken away from the rest of the pig, each leg identified with a RFID tag is weighed again. This weight is available at the PC connected to the scale. Also, using the screen as input interface, the presence of defects, previously not noticed, can be added to the database. With this new information, the application decides if the ham is still valid for the D.O. If not, a message is sent demanding that this piece must follow a different path. The RFID reader acts following the steps below: • •

Reads the tag identification number and sends it to the PC. Writes the new information, including a reject value, if this condition happens.

PC 4: Output. The RFID reader connected to the PC terminal at this point sends to the tag the information about the exit date and the drying place identification number. Likewise, the database stores the pieces (tag identification numbers) that are sent to each drying place. There are some drying places that demand hams with special conditions, for example, a weight greater than a specific value. The attached tag will simplify this selection.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

66

A.M. López et al. / Design of a RFID Based Traceability System in a Slaughterhause

2.4. Final memory content The data present at each tag after leaving the slaughterhouse are shown on table 1 together with their bit size. Every farm, abattoir and drying place associated to the DO “Jamon de Teruel” is identified by an internal number consisting of three decimal digits. Table 1. Information stored inside the tag after leaving the slaughterhouse. Columns 2 and 3 show where this information is collected and its bit size.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Information Feeding farm Slaughterhouse Slaughter week Weight of the animal Ordering number Weight of the leg Reject at the quartering room Exit date Destination drying place

Collecting point PC 1 PC 2 PC 2 PC 2 PC 2 PC 3 PC 3 PC 4 PC 4

Bit size 10 bits 10 bits 6 bits 10 bits 14 bits 5 bits 1 bit 9 bits 10 bits

The four initial fields contain the same information that the number that is being used at this moment by the CRDO to identify each animal. The only difference is that also a three digit number is used to code the slaughterhouse information. In addition to these data, a code of 163 bits is stored to guarantee the watcher supervision by the use of public-key cryptography and at an affordable cost without the need for substantial investment in infrastructure. Aggregate signatures are used so that all the steps can be signed in a reduced memory space (every step the same). This type of signature is a cryptographic primitive that “consolidates” several signatures into one in such a way that if n users sign n messages, all the signatures can be grouped into one single signature. Using this method it is possible to use PKI without increasing the computational requirements of the RFID card given that the computation will be carried out by a computer connected to the reader. The memory size of the i-code tags is big enough to store the data shown, and there is space left for keeping the parameters of the drying process. The structure of the database is going to resemble as much as possible the current structure of the CRDO database, though new tables have been created. Modules for the data exchange are being developed using technology XML (eXtensible Markup Language).

3. Improvements of the system 3.1. Use of LF tags for the traceability of living animals A complete traceability system should begin with the animal’s birth. At this moment, the information of the birth week and the birth farm, that could be different from the breeding farm where the animal is moved later, is marked on the animal’s ear with ink. The CRDO is interested in storing also this information in the tag attached to each pork leg, but it would require reading the pig’s ear at the watcher´s check point. The speed of the slaughter chain does not make possible this action.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A.M. López et al. / Design of a RFID Based Traceability System in a Slaughterhause

67

The collection of this information could be also automated if it was previously stored in an RFID tag. Living animals are uniquely identified by means of RFID elements working in the range of low frequency, LF, centered at 134.5 KHz. RFID chips are being currently used in pets and are compulsory in the case of sheep, goats and cows[4-5]. RFID ear tags are used in the case of cows and RFID bolus for goats and sheep. The problem of attaching and recovering RFID tags in pigs hasn’t been solved. Solutions like crystal capsules inserted to the hoof or between the teats have been proposed[7]. Our group has developed an application that writes on this kind of RFID tags the birth information. Sharing the same database, this application is able to communicate with the program developed for the HF readers, so a PC connected to a LF reader and a HF reader, will transfer the birth information to the adhesive HF tag. Our LF system is able to read/write tags at a distance up to 1 meter with the tag in air. We use a RI-K2A001A module of Texas Instrument with the radiofrequency module S2000 High performance and a Gesant antenna of the same brand. The system is completed with an autotunning element designed by Rumitag. Two RFID tags have been studied. The basic one has a memory of 80 bits, 64 for storing data, where the information is placed following the ISO 114874 norm. In the second case, a memory of 1024 bits is available. This fact permits store data like breeding parameters or the type or medicines and vaccines given to the animals.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.2. Change the adhesive tags for more robust ones The resistance of the adhesive tags to environmental conditions must still be checked. It is desirable that the tags will support the process until they leave the slaughterhouse, but it is less likely that the material keeps its integrity during the drying process because of the direct application of salt for a long period of time. RFID tags more specially designed for ham production exist in form of bridle or nail. These options have not been chosen, because it would be necessary a second person at the watcher’s check point to place them on the meat. However, at the quartering room or at the exit point, the available time is not limited by the slaughter chain, and the original adhesive tag could be replaced. This would increment the costs of the traceability system, but at the same time makes it possible to replace the HF tag by a tag working in the range of ultra high frequency, UHF, more advisable to logistic and product distribution applications.

4. Following steps To move forward to a real implementation of the system, some measurements must be done in the scenario. First, the presence of electromagnetic interferences in the working frequency range of the HF RFID systems must be detected in order to select the exact position of the different readers. Afterwards, the read/write distance will be measured and the problem of multiple readings analyzed. With this information, the appropriate antennas for every position will be selected. The behavior of the adhesive tags inside the slaughterhouse must be tested too, where the worst environmental conditions are found, namely the airing and cooling chambers. Not only must the information stored inside the chip be preserved, but also the printed text. The problem of the physical conditions at the drying place, though it

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

68

A.M. López et al. / Design of a RFID Based Traceability System in a Slaughterhause

will be kept in mind, is expected to be deeply considered (together with the traceability parameters to be studied at these installations) in future projects. Efficiency analysis will be done, once a first version of the system is installed in the slaughterhouse.

5. Conclusions The preliminary design of a RFID based traceability system that will work in one slaughterhouse has been shown. Apart from the common requirements associated to every food chain, in this case the fulfillment of the regulations of the CRDO “Jamón de Teruel” is aimed. So the internal normative of this institution has determined which parameters are going to be traced. The initial physical implementation of the RFID hardware is adapted to the working conditions of the slaughter chain. This scenario is very demanding and the future success of the system proposed must be checked in real conditions. The possible improvements are limited by economic reasons.

6. Acknowledgements This work is supported by Instituto Nacional de Investigación y Tecnología Agraria y Agroalimentaria (INIA) funds under grant PET2007-08-C11-06.

References

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[1] [2] [3]

ISO 9001:2000 quality assurance standard. ISO 8402:1994 quality assurance standard. Foodtrace Concerted Action Programme Generic Framework for Traceability. www.eufoodtrace.org. 2004 [4] Council Regulation (EC) N o 21/2004 of 17 December 2003 establishing a system for the identification and registration of ovine and caprine animals and amending Regulation (EC) No 1782/2003 and Directives 92/102/EEC and 64/432/EEC [5] Regulation (EC) No 1760/2000 of the European Parliament and of the Council of 17 July 2000 establishing a system for the identification and registration of bovine animals and regarding the labelling of beef and beef products and repealing Council Regulation (EC) No 820/97 [6] Babot D., Hernández-Jover M., Caja G., Santamaría C., Ghirardi J.J. 2004. Tracing pigs by using conventional and electronic identification devices. J. Anim. Sci. 82, Suppl 1. P. 142 [7] Chiesa F.Marchi E., Zecchini M. Barbieri S. and Ferri N. 2003. Electronic identification of pigs: injectable transponders in abdominal cavity. 54th Congress of the European Association for Animal Production, Rome p. 190 [8] De la Fuente M., Abarca A., García A. y Abril J. Sistema de identificación automática mediante tecnología RFID en el proceso de elaboración de jamones. IX Congreso de Ingeniería de Organización. Gijón 2005. [9] Hernández E. Gestión automatizada de la trazabilidad con RFID para un sector muy tradicional. RFID magazine. Año 01, revista 03 Julio-Agosto. [10] ORDEN de 29 de julio de 1993, del Departamento de Agricultura, Ganadería y Montes, por la que se aprueba el Reglamento de la Denominación de Origen "Jamón de Teruel" y su Consejo Regulador. Boletín Oficial de Aragón. [11] Orden APA/3164/2002 de 11 de Diciembre, por la que se establece y regula la base de datos informatizada Sistema Nacional de Identificación y Registro de los Movimientos de los Porcinos SIMOPORC. [12] G. Azuara, J.J Piles, J.L. Salazar. Securización de un sistema de trazabilidad RFID mediante firmas agregadas. JITEL 08 (Jornadas de Ingeniería Telemática) 2008. Alcalá de Henares (Madrid)

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-69

69

Development of a customized RFID reader for a distribution centre with agent-based control Javier G.-Escribano a,1, Roberto Zangróniz b , Andrés García a, Julio Cesar Encinas a , Javier de las Morenas a , and José Manuel Pastor b a School of Industrial Engineering, University of Castilla-La Mancha, Ciudad Real, Spain b Polytechnic School of Cuenca, University of Castilla-La Mancha, Cuenca, Spain

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Abstract. Distribution centers management is often a drawback for many companies in which delivery time is a critical factor. Especially for companies which have to send their products in small quantities to their customers, orders configuring and retrieving is a tedious and time consuming task. The use of mixed pallets formed with different classes of products is also a common practice for retailers. In order to improve performance of all this, this work proposes an experimental platform which represents the whole distribution center, and in which agents and RFID elements are used. Use of both RFID and agents is what is called here an RFID-Integrated Manufacturing System II. In addition to one virtual simulation, a physical model is being used in laboratory, composed of physical elements, which represents inputs and outputs of the warehouse in a distribution center. In this part is where physical RFID readers are included. They are in charge of taking information of loads getting in and out from the distribution center. The main target of this article is to show how these physical readers have been developed and what are the results of using them in the described physical platform. Keywords. RFID Agents, UHF, PoE, MAS, RFID-IMS II, Distribution Centers.

Introduction In a highly competitive environment, companies are forced to constantly improve their production and delivery systems. The task of managing people and resources in an efficient way is the key to achieving the required productivity demands [2]. Growing in manufacturing flexibility, products’ complexity and customers’ specifications make difficult orders configuring and managing of the whole distribution center. Given the high number of competitors from other areas, European Companies have to take advantage from flexibility and proximity to customers. These added features allow time-to-market reduction for new products. However, all this largely increases

Corresponding autor: Javier G.-ESCRIBANO SÁNCHEZ-P. E.T.S. Ingenieros Industriales. Universidad de Castilla-La Mancha. Avda. Camilo José Cela s/n. Edificio Politécnico. 13071 Ciudad Real (Spain). E-mail to: [email protected] 1

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

70

J. G.-Escribano et al. / Development of a Customized RFID Reader for a Distribution Centre

the number of references in the production lines, without margin for significant changes in production costs. This research intends to show the development of new management methods with the application of RFID and Multi-Agent System technologies (RFID-IMS II). The development a customized RFID reader connected to an agent’s platform is the target of this work. The proposed solution can be applied to specific areas in real factory environments with high production levels and large distribution centers. Additionally, in the application of a control system, an appropriate representation of the distribution process is necessary to be able to evaluate the different control schemes [5]. In order to experiment with these new methodologies it has been necessary to build up a physical model connected to a distribution center simulation. With this experimental platform it has been possible to prove new control, planning and decision techniques, without obstructing a real enterprise operation. The structure of this article is as follows: section 2 presents the used RFID-IMS II methodology, and section 3 describes the experimental platform with its virtual and physical part. Section 4 explains the process of agentification by which every part is controlled by its own agent. After that in section 5, RFID Agents are described as virtual and physical ones. In the last part of the article, a description of the developed RFID System, connected to the RFID Agent, is given. This RFID System is composed of an RFID reader, compatible with ISO 18000-6C and EPC Class 1 Gen 1/2 tags, which has been connected to a microcontroller with Ethernet features. Finally some conclusions drawn from the experiments made in laboratory are included in the last section.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. RFID-IMS II Methodology When a RFID technology is used in an Information Management System, the whole can be called Radio Frequency Identification – Information Management System (RFID-IMS). This methodology was created with the aim of supporting all the data processing that is generated in the highly distributed control systems and to providing that information to management systems [4]. In systems where MAS models are used, there is an excessive dependency on the information available from the elements that comprise the system. The MAS controls the system in an instantaneous form; operators only have to determine the design characteristics and implementation. This new solution, RFID-IMS/MAS is called: Intelligent Manufacturing Systems enhanced with Radiofrequency Identification (RFID-IMS II) [4]. RFID Agents, which are those in charge of managing RFID systems, are described in this article. The huge information management capacity that can be achieved with RFID justifies its use. Moreover, thanks to its properties it is possible to detect, identify, and verify the nature of the incoming products, which allows, for example, storing them according to different rotation levels [1]. With RFID technology, products can be marked and inventoried while on conveyor lines or during the loading/unloading from trucks at docks, or while handling the loads in warehouses or distribution centers [10].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. G.-Escribano et al. / Development of a Customized RFID Reader for a Distribution Centre

71

2. Experimental Platform Description The experimental platform that has been developed is a very powerful tool for designing, experimentation, and testing the new approaches. This integrates the RFIDIMS II technology in a storage and distribution system based on a real business located in Spain. In this experimental platform physical and simulated parts are combined. The simulation represents the processes of storage and distribution, whereas in the physical model the loading/unloading process is carried out. 2.1. Virtual Simulation

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

A way to reduce costs in the design phase is to use simulation, as it provides an insight of the performance of the system in different situations that can happen in future [3]. In this work, the simulation of one of the biggest bottling companies’ facility in Spain has been made by means of the software Grasp 10. The aim of this simulation is to represent the movement of pallets and the processes of picking in the warehouse. RFID readers are responsible of managing pallets’ tag’s data, control their position in the facility and send this information to the RFID Agent.

Figure 1. Physical model picture

On the other hand, a computer connected to Ethernet acts as a control interface for the simulation in Grasp 10. This interface, programmed in java, serves as a bridge between the decisions taken by a PLC network, which represents a network of agents, and the simulation. 2.2. Physical Model The physical parts that make up the physical model of the experimental platform appear in the picture of Figure 1. The platform, controlled by other PLC, represents the inlet/outlet of products in the distribution center. Colored and tagged (with RFID tags) pallets arrive to the merchandise loading/unloading zone (loading docks: red square in the bottom). They

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

72

J. G.-Escribano et al. / Development of a Customized RFID Reader for a Distribution Centre

use different colors to indicate the rotation grade of the products: red (high rotation), yellow (medium rotation), green (low rotation) and blue (from picking). Found in the loading and unloading dock are two conveyor belts (letter A) for the entrance and exit of products. On the entrance conveyor belt an RFID reader (letter C, right) has been installed to read the tags that pallets are carrying. It generates the necessary information so that the system identifies the entering pallet and indicates the automatic warehouse (letter W) the position it must take. Pallets with higher rotation occupy the lowest and closest part to the exit of the automatic warehouse. Once pallet has been allocated it appears in the simulation, where depending on product’s grade of rotation and destination, the pallet will follow different processes. In the same way, when a pallet leaves the automatic warehouse it connects with the platform and robot puts it on the exit conveyor belt. Later the fork-lift takes it to the loading docks. Another RFID reader (letter C. left), located in the exit door, verifies the contents of the products leaving, checking that products going out match with customers requests. 3. RFID Agent Structure RFID agents are those in charge of the autonomous functioning of RFID systems and their communication with the rest of the agents of the platform. Two RFID Agents has been included in this research: the virtual and the physical one. 3.1. Virtual RFID Agent

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The virtual RFID Agent is located within the simulation, and does not represent any physical part in this case. The simulation is now representing a real distribution center, thereby if it was a real installation, virtual RFID readers would be real ones. In the simulation, like in a real facility it is important to know where a pallet is going thought every moment. This information can be retrieved from simulation parameters in running phase. Despite of this, it is important to know in which points the system would use real RFID readers. 3.2. Physical RFID Agent Description The RFID System developed uses passive RFID tags and only one reader which are installed in the physical model of the experimental platform. The tags used have been selected due to their reach, isotropy, and price. The reader uses a multiplexer to split its signal into two lines, to connecting to two antennas. As mentioned before, the two antennas are located one on top of a reading gate at the entering conveyor belt, and the other at the exit door. In Figure 2 there is a picture of one of the antennas used in the model. They have been chosen due to their small size an appropriate reach for this application.

Figure 2. Dipolar Antennas used in the model

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. G.-Escribano et al. / Development of a Customized RFID Reader for a Distribution Centre

73

Several tests have been made in order to choose the most appropriate RFID tag to use. Several were the characteristics it had to fit in: size, reach, flexibility, reading/writing capabilities. The tags had to be introduced in slots practiced inside each wood model pallet. Tags had to be folded in order of being introduced in pallet’s holes. The RFID reader developed consists of a Skyetek M9 module that has been enabled with Ethernet capabilities. For doing this, an Ethernet featured microcontroller is used in the reader. Some other components have been included to providing the reader with serial and USB connection. The microcontroller, the reader module and the rest of elements in the circuit are powered via the Ethernet using the PoE technology. A block diagram of the whole circuit components is presented in Figure 3. In the following sections descriptions are included of the developed reader’s components.  Reader Module

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The reader module used is the Skyetek M9 SkyeModule [9] . It is an 862-900 MHz UHF reader/writer that is enabled with the main host communication interfaces: UART (TTL), SPI, USB and I2C. The reader is also provided with a 7 pin GPIO port that is used here to command the multiplexer’s operation. The design of the complete reader device is based on the features given by this module. In this way a PCB has been designed in which the module is plugged. This board has a microcontroller (8-bit Microchip PIC18F67J60 [8]) to which all the information taken from the reader is sent via SPI. Other connection lines are being used within the board like UART and USB. In Figure 3 it can be seen the USB connector with D+/D- lines to M9 module and also the MAX232 transceiver that is in charge of adapting TTL to RS-232 levels. The reason of having added these connections is that the system could be operated from an external host (maybe a computer) while in the first stages.

Figure 3. Reader's components block diagram.

The power needed to feed the whole reader device and components is retrieved from a Power Over Ethernet (PoE) supply. This was a prerequisite to have no other lines for powering the system distinct from the Ethernet line. Hence, the need of a PoE Power supply has been satisfied by means of a Step-Down Converter (LM2596, datasheet in [7], Figure 3). This converter is connected to two pair lines in the Ethernet Base-T cable which contain the data and are spare respectively. In order to get regulated 3.3 V current to feed the microcontroller, the KF33 regulator has been added.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

74

J. G.-Escribano et al. / Development of a Customized RFID Reader for a Distribution Centre

4. Reader Firmware and Application Program The reader used in this approach has been especially designed to fit with the characteristics first required. The use of the Skyetek M9 Reader Module provides the system the reliability and standardization level in terms of global compatibility with every tag protocol. Other feature required was access via Ethernet in the way the reader can be commanded in a ubiquitous way. Also the addition of an on-road power system, like PoE, makes an easy installation and integration within the existent industrial facilities.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 4. Main program operation blocks diagram.

Reader firmware consists of a program within the main controller which is in charge of running execution of the main routine, sending M9 module and Ethernet commands, and managing interruptions from them. The main program execution is summarized in the blocks diagram of Figure 4. The main operations and states in the microcontroller firmware and their descriptions are: • Initialization and configuration: initializes clock frequency, serial ports data transmission rate (SPI), TCP/IP settings and M9 power transmitted, gain, antennas type used, protocol used, etc. • Run Mode: microcontroller enters a loop of sending reading requests to M9 module and waits for interruptions. • µC TCP packet processing: after an interruption from TCP buffer, microcontroller takes data and reads the message. Then if it has to send a request to M9 it puts the suitable command into the SPI port to be sent to the reader module. • µC M9 packet processing: when a packet is received through the SPI port, an interruption is launched and microcontroller takes data and reads the message. Usually it has to send data from tag to Ethernet using TCP protocol. The selected microcontroller has a branch of Ethernet features, like the possibility of being used as a HTTP or FTP server. These features are all included in the TCP/IP

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. G.-Escribano et al. / Development of a Customized RFID Reader for a Distribution Centre

75

free stack provided by Microchip for all of its Ethernet products, and that is why PIC18F67J60 microcontroller was mainly selected. In this project the TCP protocol contained in the stack has been implemented. This protocol has been used due to its secure and reliable data transmissions, leaded by the sending of acknowledge packages and the resending of frames when data is not well received. This way, transmission is assured between two points. In addition, this protocol uses a flow control which rule out messages coming too fast and advises receiver that some are going to be missed. Moreover, DHCP protocol has also been implemented in this project to provide the system with an automatic IP dynamic address assignment at system’s start-up. 4.1. Application Software

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

For the testing of the new developed reader and to begin using Ethernet capabilities, an application program has been developed. The program developed launches an application whose main window appears in Figure 5. In this software, application basic functionalities has been implemented for setting reader parameters and running reading and writing commands. Thereby it is possible to see in the picture the main white area where all the read tags’ EPCs appear. This program also shows the number of antenna that collected data and the acquisition time and efficiency in readings. In the center of the image it appears another window related with connection configuration, where reader’s IP must be inserted, and where the type of connection (Ethernet, USB o serial) is also selected. Other available options are the protocol selection, the power and gain of the outcoming wave from reader, the number of antennas connected and which of them have to be used in readings.

Figure 5. Developed application program for testing reader’s operation.

5. Conclusions With the use of an experimental platform it has been possible to represent in a reliable manner and evaluate different distribution processes. Some advantages have been proved with the implementation of RFID-IMS II in a distribution centre. The proposed methodology endows the system with an inherent capability to reacting to unexpected

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

76

J. G.-Escribano et al. / Development of a Customized RFID Reader for a Distribution Centre

changes in a flexible manner. This is achieved by the use of negotiation through agentbased control. An information management system is able to feed accurate information about products to all the agents. The use of this methodology does not only include agents but also RFID systems. These systems are connected to the network of agents forming what is called the RFID Agents. The last agents are in charge of identifying and managing the information coming from RFID readers and tags in a reliable way. With the use of this system, a good unallocated connection has been achieved. One of the included RFID Agents is presented here as a physical agent. The design of the hardware elements (readers, antenna and tag selection) was the target of this work. Antenna and tags have been selected according to given requirements. The performance obtained with them is exactly what expected for this application. However they could not achieve the features of reach and reliability needed in real distribution centre facilities. The reader proposed here has been demonstrated to have the required characteristics in terms of connectivity, energy consumption, price and ease of integration. The connexion with the RFID Agent, which is able to control the operation and also the powering of the reader via Ethernet, has been proved in laboratory with great success.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] Abarca, A., Encinas, J.C. & García, A. Optimización de gestión y manipulación de stock usando RFIDIMS II en la cadena de producción-distribución. Proceedings of the 6th International Workshop on Practical Applications of Agents and Multiagent Systems. (2007), 309-318. [2] Cheeseman, M., Swann, P., Hesketh, G. & Barnes, S. Adaptive manufacturing scheduling: a flexible and configurable agent-based prototype. Production, Planning & Control, Taylor & Francis, Vol. 16, No. 5 (2005), 479-487. [3] García, A. Célula de Empaquetado con Arquitectura de Control basada en Holones y Tecnología de Identificación por RF. XXIV Jornadas de Automática FAE. León. (2003). [4] García, A., Cenjor, A., Chang, Y. & de la Fuente, M. An Agent-Oriented Design Methodology for RFID Improved Manufacturing Control. 11th IEEE International Conference on Emerging Technologies and Factory Automation. Prague, Czech Republic (2006), 20-22. [5] Gradišar D. & Mušič G. Production-process modelling based on production-management data: a Petri-net approach. International Journal of Computer Integrated Manufacturing, Taylor & Francis, Vol. 20, No. 8, (2007) 794-810. [6] IEEE Standards Association. IEEE 802.3TM. http://standards.ieee.org/getieee802/download/802.32002.pdf. (2002). [7] National Semiconductors. LM2596: Step-Down Switching regulator. http://www.cheertech.com.tw/HTC/Data/23-1-2596.pdf. (2004). [8] Microchip Corp. PIC18F87J60 Family Datasheet. http://ww1.microchip.com/downloads/en/DeviceDoc/39762d.pdf [9] Skyeteck, Inc.. SkyeModule M9. http://www.skyetek.com/Portals/0/Documents/Products/SkyeModule_M9_DataSheet.pdf. (2007). [10] Xiao, 2007. Xiao, Y., Yu, S., Wu, K., Ni, Q., Janecek, C. & Nordstad, J. (2007). Radio frequency identification: technologies, applications, and research issues. Wireless Communication and Mobile Computing, Vol. 7, (2007), 457–472..

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-77

77

Using RFID information in routing and sequencing warehouse operations Francisco Ballestín a,1, Pilar Lino b, Ángeles Pérez b, Sacramento Quintanillab and Vicente Valls c a Dpto. de Estadística e Investigación Operativa, Universidad Pública de Navarra, Spain b Dpto. de Matemáticas para la Economía y la Empresa, Universitat de València, Spain c Dpto. de de Estadística e Investigación Operativa, Universitat de València, Spain

Abstract. Warehouses are essential components of logistics and supply chains. The performance of warehouse operations significantly affects the efficiency of the whole chain it belongs to. Radio frequency identification (RFID) is an emerging technology capable of providing real-time information about the location and properties of tagged object(s), such as people, equipment or products. The main objective of this article is to provide some insights into the practical benefits that can be drawn from the RFID technology in the context of warehouse operations management. To this end, we have developed a set of heuristic routing and sequencing procedures that take, and, alternatively, do not take into account, real time RFID information and compared their performance via simulation on a set of randomly generated although realistic warehouse scenarios. Keywords. RFID technology, warehouse operations management

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction Warehouse management systems (WMSs) have been developed for handling warehouse resources and monitoring warehouse operations in order to increase their efficiency and effectiveness. However, the current WMSs are incapable of providing timely and accurate warehouse operations information because they contain no feature of real-time and automatic data retrieval. Radio frequency identification (RFID) is an emerging technology that is increasingly being used in business and industry, particularly in logistics and supply chain management (see for example [1], [2], [3] and [4]). RFID is intended to replace traditional barcodes in many ways. It makes possible a range of applications related to the tracking and intelligent management of any entity tagged with an RFID chip in real-time for which the barcode appears to be completely unsuitable. For warehouses that keep inventory, the basic warehouse operations are to receive Stock Keeping Units (SKUs) from suppliers, store the SKUs in storage locations, receive orders from customers, retrieve SKUs from storage locations and assemble them for shipment and ship the completed orders to customers ( surveys can be found at [5], [6] and [7]). 1

Corresponding Author: Francisco Ballestín, Dpto. de Estadística e Investigación Operativa, Edificio de los Magnolios, Universidad Pública de Navarra, Campus de Arrosadía., 31006 - Pamplona, Spain; E-mail: [email protected]

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

78

F. Ballestín et al. / Using RFID Information in Routing and Sequencing Warehouse Operations

The main objective of this article is to provide some insights into the practical benefits that can be drawn from the RFID technology in the context of warehouse operations management. More specifically we focus on routing and sequencing operations. To address this issue, we develop a stylised model that captures and generalises the main characteristics of the structure and routing and sequencing operations of a given real warehouse. Furthermore, we develop a set of heuristic routing and sequencing procedures that take, and, alternatively, do not take into account, real time RFID information and compare their performance via simulation.

1. Warehouse description In this paper we consider five types of warehouse layouts. Each of them is formed by either one or two storage location zones of the same or different types. The warehouse contains a common shipping area. The SKUs are pallets. We consider two types of zones. Zone layouts are illustrated in Figure 1(a) and Figure 2(a) (real dimensions are not shown). Similar layouts have been previously described in the literature; see for example [8]. Both zones have a rectangular layout with a certain number of parallel pick aisles and parallel cross aisles. Storage locations are at both sides of the pick aisles in multiple levels. Cross aisles do not contain storage locations. Storage locations are double-depth so each of them can store up to 4 pallets two on top of the other two. Each zone has a single entrance and exit point. We have considered these types of warehouse layouts because the layout of the real warehouse that has been taken as a source of inspiration is formed by two storage location zones, one of each type. We have introduced the other four warehouse layouts with the aim of being more general in our experiments.

cross aisle 2

cross aisle 1

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

pick aisle 2

pick aisle 1

Entrance / Exit (a)

(b)

Figure 1. First zone layout (plan view) and its corresponding graph representation

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

F. Ballestín et al. / Using RFID Information in Routing and Sequencing Warehouse Operations

79

cross aisle 3 pick aisle3

pick aisle 2

pick aisle 1

cross aisle 2

cross aisle 1 Entrance / Exit (a)

(b)

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 2. Second zone layout (plan view) and its corresponding graph representation

Order pickers receive the orders in the shipping area one at a time. To fulfill a given picking (storing) order an order picker driving a forklift retrieves (stores) a pallet from (to) the corresponding storage location located in one of the two zones and transports it (returns) to the shipping area. Order pickers can traverse cross and pick aisles in both directions. A forklift can change direction in a picking aisle after retrieving or storing the pallet. Two forklifts cannot go in parallel in any aisle although they can go one after the other through the same aisle. Two forklifts can work at the same time in two locations of the same aisle provided that there are at least a given number of locations between them. The warehouse is of the chaotic type where a pallet can generally be stored in any location although some restrictions are applied to certain pallets. The most important restrictions are that certain pallets should be stored below a maximum level and that there are pallets that cannot have another one on top of them. At a given time instant there is a set of orders to be fulfilled, O ={o 1,…,on_o}. Each order is determined by three elements, oi = (p(oi), l(oi), ll(oi)), where: p(oi) is the pallet that must be stored (retrieved), l(oi) is the storage location where p(oi) has to be stored (retrieved from) and ll(oi) (ll(oi) = 1,2,3,4) is the position of p(i) in the storage location l(oi). The value ll(oi) will be 1 and 2 for the first depth (down and top, respectively) and 3 and 4 for the second depth (down and top, respectively). There is a set F = {f1,...,fn_f} of available forklifts that can work simultaneously. There are four types of forklifts: • • • •

Type 1: forklifts that can manage pallets only at the first level and at the first depth. Type 2: forklifts that can manage pallets up to the third level but only at the first depth. Type 3: forklifts that can work at any level but only at the first depth. Type 4: forklifts that can work at any level and any depth.

Forklift types are hierarchical in the sense that the locations reachable by a forklift of a type include the locations reachable by a forklift of any inferior type. All forklifts work at the same speed. Each order should be assigned to a forklift capable of executing it.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

80

F. Ballestín et al. / Using RFID Information in Routing and Sequencing Warehouse Operations

The warehouse layout can be represented by a graph G = (V, A) where V and A are the sets of vertices and arcs, respectively (see Figures 1(b) and 2(b). The set of vertices V contains: • • • •

A vertex for every zone entrance point. A vertex for every two columns of storage locations placed one in front of the other in the same aisle. A vertex for each intersection of aisles. A vertex representing the shipping area.

The set of arcs contains: • • • •

An arc for every pair of vertices representing adjacent columns of storage locations. An arc for every pair of adjacent aisle intersections. An arc for every pair formed by an aisle intersection and a storage location column located first in any aisle leaving the intersection. Two arcs connecting the shipping area vertex with the vertices representing the entrance/exit points of the zones.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

In this context, to execute an order oi it is necessary to traverse a route starting at the shipping area vertex, visiting the location l(oi) vertex and coming back to the shipping area vertex. The length of an arc is the time (in minutes) a forklift spends in traversing it. Due to the confined and narrow travel paths in a warehouse, another relevant issue in the routing and sequencing problem is congestion that occurs when multiple picking orders are intended to be executed at the same time in the same area. The time to fulfil an order, t(o i), is calculated as the sum of several times: the travel time, the waiting time spent due to congestion and the time a forklift spends either storing or retrieving a pallet from the storage location l(oi). The averages of these times are assumed to be given. To sum up, the problem we study in this paper is the following: given a set of orders to be fulfilled find a sequence of order fulfillment, a route and a forklift for each order with the objective of minimizing the total flow time of the set of orders (min T = n_o

min ∑ t (o i ) ). All orders are available at time zero and there is not a pre-established i =1

priority between them. In this paper we explore the impact of RFID technology on T. In our experiments, we assume that the storing/retrieving and travel times are random variables whose probability distributions have been approximated through historical data. It is also assumed that they are not affected by the use of RFID technology. However, the congestion time can be reduced by using RFID technology. If RFID active tags are attached on the floor at aisles crosses, at zone entrances and at the shipping area and RFID readers are integrated into forklifts then a forklift position will be determined by reading the active tag the forklift has just gone through. The movement direction of the forklift will be determined by the order in which the two last visited tags have been visited. This relatively low cost system allows obtaining real-time information on the position and movement of the forklifts that are currently fulfilling orders. We can make use of this information for designing next order selection policies aiming at reducing the congestion time as will be apparent in the next section.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

F. Ballestín et al. / Using RFID Information in Routing and Sequencing Warehouse Operations

81

2. Solution Approach A solution for our problem consists of a policy that indicates how to calculate the sequence in which the orders are going to be executed. The actual order sequence is decided on line and depends on the events (e.g. interferences among forklifts) which happen during the execution of previous orders. These events cannot be predicted in advance because they depend on randomness. Although we have an estimation for the times needed to traverse an arc and to store/retrieve a pallet, there is an intrinsic variability in these actions; the workers can go faster or slower than average, for example. In this paper we consider that the duration of each action follows a probability distribution. To evaluate a policy we will simulate the duration of each action and make the decisions at certain decision times according to the warehouse situation and the fixed policy. A realization will be the simulation of every action until every order has been completed. The value of the objective function associated to a policy and an instance will be the average total flow time over 100 realizations. Several steps should be followed to perform a realization. The outline of these steps is as follows.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. 2. 3.

Calculate a shortest route for each storage location vertex. Next decision time t = 0. For the next decision time t: a. Calculate Elig(t). b. Calculate AO(t). c. While(Elig(t) ≠ ∅) i. Assign a priority to each order o ∈ Elig(t). ii. Select the order o with highest priority. iii. Start order o: S(o) = t. iv. AO(t) = AO(t) ∪ {o}. v. Recalculate Elig(t). d. If AO(t) = ∅ then STOP. T = t. e. Calculate next decision time t. Return to 3.

AO(t) is the set of orders which are active at time t – have begun but have not ended yet – and T is the total flow time. S(o) represents the start time of order o. The priority assigned to an order in step 3.c.i is calculated via a priority rule. Let us revise the different elements of the approach. 2.1. Calculate shortest routes On the graph representing the warehouse we calculate the shortest routes to access each storage vertex from the shipping area vertex. It is only necessary to make these calculations once for a warehouse and consequently time is not a key issue. 2.2. Decision times A decision time is a time when we decide whether to begin an order. Theoretically, we could start the process of an order at any time. To reduce the number of decision times we have taken into consideration that reducing congestion goes in the direction of total flow

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

82

F. Ballestín et al. / Using RFID Information in Routing and Sequencing Warehouse Operations

time minimization. While the active forklifts are traversing arcs, we have approximately the same congestion. Therefore, we match decision times with the moments when a forklift reaches a node or finishes the storage/retrieval of a pallet. The next decision time t will be chosen as the minimum of these values. 2.3. Selection of orders to start We will say that an order is eligible for a decision time t if, assuming that every forklift spends the estimated time to perform each action, the forklift assigned to the new order will not enter when following its associated shortest route an arc occupied at that moment by another forklift performing an active order. Note that the cross of routes depends on the position of the forklifts at a certain time. However, we do not know their exact situation unless a real-time location technology is used. RFID technology allows us to know the forklifts locations with enough precision; otherwise we can only guess where they are according to when they had started their trip and their average speed. We denote Elig(t) as the set of eligible orders. Note that the definition of an eligible order for a decision time t assumes that an available forklift capable of performing the order has been assigned to it at time t. The policy of starting only eligible orders greatly limits the decision times when new orders might start, as well as the number of orders that can begin at the same time.

3. Computational Experiments We have created a set of instances with different characteristics. We have run our program and evaluated the following priority rules:

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

a) Maximum time: we select the eligible order which takes the highest time in being executed, if we consider the estimations of the different elements. b) Best resource: we select the eligible order for which its assigned forklift has the maximum type number. c) Minimum number of common arcs: we select the eligible order whose route has the minimum number of arcs in common with the active routes. d) Random: we select the eligible order randomly. The forklift assigned to an order is one of those that belong to the inferior forklift type that contains available forklifts capable of executing the order. We have randomly generated, based on company characteristics, a total of 330 instances of different complexity. The number of cross aisles and pick aisles have been stated respectively to 2 and 8 in first type zones and to 5 and 4 and 3 and 4 in second type zones. Regarding the rack size: the number of levels has been stated equal to 9 and the number of columns vary in [4,10]. The total number of forklifts in an instance varies between 6 and 20. The forklift types are also randomly stated although at least 30% of them must be of type 4. The total number of orders in an instance varies between 80 and 290. For each instance, we have calculated the best average time among all algorithms using RFID information. We have calculated the deviation with respect to this minimum. The average of these deviations is included in Table 1, along with the number of minimum values (best solutions) an algorithm has obtained.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

F. Ballestín et al. / Using RFID Information in Routing and Sequencing Warehouse Operations

83

Table 1. Results of the different priority rules Priority rule Maximum time Best resource Minimum number of common arcs Random

Average Deviation 5.65% 5.80% 18.10% 9.21%

Number of best solutions 31.21% 51.82% 1.21% 14.55%

As we can see in the table, the best results have been given by priority rules “best resource” and “maximum time”. They provide better results than the random rule, being the difference around 4.5%. To conclude, we have applied the “maximum time” priority rule, but in this case without the help of the information given by the RFID. The average deviation obtained by this version is 15.54%, with only a 1.52% of best solutions. Therefore, these results seem to indicate that warehouse operations efficiency can be significantly improved by using the RFID technology.

Acknowledgements This research was partially supported by the Ministerio de Educación y Ciencia under contract DPI2007-63100.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] Chow, H.K.H., Choy, K.L., Lee, W.B., Lau, K.C., Design of a RFID case-based resource management system for warehouse operations, Expert Systems with Applications 30 (2006), 4, 561-576. [2] García, A., Chang, Y.S., Valverde, R. Impact of new identification and tracking technologies on a distribution center, Computers & Industrial Engineering, 51 (2006), 3, 542-552. [3] Chow, H.K.H., Choy, K.L., Lee, W.B., A dynamic logistics process knowledge-based system –An RFID multi-agent approach, Knowledge-Based Systems, 20 (2007), 4, 357-372. [4] Poon, T.C., Choy, K.L., Chow, H.K.H., Lau, H.C.W., Chan, F.T.S., Ho, K.C., A RFID case-based logistics resource management system for managing order-picking operations in warehouses, Expert Systems with Applications, 36 (2009), 4, 8277-8301. [5] Gu, J., Goetschalckx, M., McGinnis, L. F., Research on warehouse operation: A comprehensive review, European Journal of Operational Research, 177 (2007), 1-27. [6] De Koster, R., Le-Duc, T., Roodbergen, K.J., Design and control of warehouse order picking: A literature review, European Journal of Operational Research, Volume 182 (2007), 2, 481-501. [7] Roodbergen, K.J. and De Koster, R., Routing methods for warehouses with multiple cross aisles, International Journal of Production Research, 39 (2001), 9, 1865-1883. [8] Caron, F. Marchet, G., Perego, A., Layout design in manual picking systems: a simulation approach, Integrated Manufacturing Systems, 11 (2000), 94-104.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-85

85

Improvements in Supply Chain tracking using a three-levels RFID System Antonio Abarca a, Javier G.-Escribano b,1, Juan J. De Dios c, Andrés Garcíab , a University of Jaen, Jaen, Spain b School of Industrial Engineering, University of Castilla-La Mancha, Ciudad Real, Spain c Polytechnic School of Cuenca, University of Castilla-La Mancha, Cuenca, Spain

Abstract. Monitoring of environmental conditions of products in Supply Chains is commonly demanded by users in distribution and manufacturing processes. This information is also required by customers and authorities at final stages. Sometimes is difficult to collect this information, but especially when products are mixed within the same pallet. The process of mixing products in pallets is called “picking” and is a usual practice when customers require small quantities of each item. The number of resources used and the amount of mistakes involving this process suppose and important drawback for distribution companies. To tackle this lack of competiveness created, this work proposes a three-levels based RFID System. This system tries to improve the process of identification and monitoring of condition of products by means of passive and active tags, and the necessary RFID readers. The three different RFID components are connected to three different levels which Supply Chain has been divided in. RFID readers are actually in the third level, the highest one, are wirelessly connected using ZigBee networks. GPRS modules are also used here to provided real-time global localization of products (RTLS) in the moment of being accessed by readers.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Keywords: Picking, MAS, ZigBee, GPRS, RFID Agents, RFID-IMS.

Introduction Decreasing delivering time of products is critical for companies to achieve competitive advantages. Further, the products’ information provided to customers is starting to be used as another competitive advantage by companies. The first factor is directly related with eventual problems in production and distribution phases. In addition, when picking processes are included, high investment and an effective verification system are required in order to detect and resolve errors. Tracking of each unit of product should be performed during the manufacturing and distribution phases. The system must deter-

1

Corresponding autor: Javier G.-Escribano Sánchez-P. E.T.S. Ingenieros Industriales. Universidad de Castilla-La Mancha. Avda. Camilo José Cela s/n. Edificio Politécnico. 13071 Ciudad Real (Spain). E-mail to: [email protected] Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

86

A. Abarca et al. / Improvements in Supply Chain Tracking Using a Three-Levels RFID System

mine the stage and the exact location in the world where a product is at any time, from manufacturing to reception by customers. The system proposed here is an RFID system, which is able to support active and passive tags, and is managed by a middleware capable of correctly process all the generated data [6]. RFID passive tags are a good solution for tracking of products along the distribution process; however, they lack the advantages of the active tags, in important aspects such as the reach and memory. Comparing both types of tags, the active ones have the advantage of reach, whilst the lasting of batteries is an important disadvantage. From a technical view, it is necessary to analyze this issue in order to minimize its importance: improving the life of the batteries and reducing the power consumption of the tag for an optimal operation. The open issue of the traceability and the tracking of products can be solved in a simple and economic way by means of the RFID technology [6]. The structure of this article is as follows. In the first section, a State of the Art reviews the current state of RFID technology and its application in distribution and manufacturing. In section 2 the description of the used system and its connection with the Multi-Agent Systems (MAS) is presented. In the following section, number 3, the physical implementation is included. Finally, some conclusions and information about the future work still to be done are added.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. State of the Art The unavoidable advance of RFID technology will have tremendous advantages and provide us with good applications in distribution and manufacturing.Therefore, there is a global need for the normalization of the RFID standards, by means of different ISO recommendations, so that they aid the implantation of the RFID technology all over the world. A few years ago, applications based on RFID systems were hardly used internally inside some companies. A normalization procedure is required for sharing information among different areas of several companies, so that a global spreading of the RFID is performed. Multi-agent systems (MAS) provide with different solutions both in the manufacturing environment and also in the logistics field. In the logistics environment, the involved agents negotiate orders to suppliers, considering aspects such as prices, delivering times, availability, etc. [10] The application of both the RFID technology and the MAS for the management of the information related to each product applied to the production and distribution processes is known as RFID-IMSII (intelligent manufacturing system enhanced and based on RFID) [3]. 2. System Description RFID-IMSII was first devised for manufacturing applications. However, today, due to the RFID’s globalization, RFID-IMSII can be also applied to logistics processes. In these schemes, MAS manage the production and distribution lines, establishing a negotiation of the delivering times and prices of the products.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Abarca et al. / Improvements in Supply Chain Tracking Using a Three-Levels RFID System

87

In the RFID-IMSII, some RFID devices must be included to collect the information regarding the products by means of tags and readers. The devices will be grouped together forming one or several RFID agents. The RFID devices will send these data to the agents, so that it is available for the rest of the agents in the system. Passive and active RFID tags are integrated on the products that go through the manufacturing/distribution lines; and also readers to take the information from them. During distribution processes, products are controlled when passing under readers to identify them. To managing this information a middleware that communicates the RFID system with the computer that conforms the RFID Agent, has been designed. This software mainly consists of databases and the capacity to process this data [5]. Supply chain has been divided into three levels here. The first level is formed by the products themselves or the boxes they are inside. The second level corresponds to the pallets of the product boxes that can be non-homogeneous. And, finally, the containers capable of transporting a number of pallets inside them constitute the third level. The passive tags on each unit of the product are part of the first level. The active tags, which are part of the second level, include the whole information of the pallet, and they have also sensors to get the data related to the environment where the pallets are going by. In the last level, the readers for the active tags collect all the information regarding the container by means of reading the whole set of the pallets’ tags inside it. These readers communicate among them using a ZigBee wireless network and they are connected to a GPRS network, too.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

In the picking tasks, each pallet can be made of units of the same product, layers of the same product, or different options for mixing the products in order to complete a pallet (a quarter, or a half of it, etc.). The system collects all this information, and stores it on an active tag on each pallet. The information referred to all the products inside the pallet is recorded on this tag. The content of a pallet can be verified in each phase along the logistics chain, wherever there is an RFID reader. Active tags must be always ready to be read and written in order to perform the tracking and verification of the transported products. The information that the system is recording is used to have a control of time intervals, processes, etc., so that the visibility and traceability of each unit of product is achieved. The third level of the tracking system can be managed with the creation of a wireless network implementing one or both of IEEE802.15.4/ZigBee [13] standards. Every element of this network can be located inside a container-type load unit. These control units have a double mission: first, they must control the active tags which are stored inside them (pallet level); and, on the other hand, they are able to communicate to other similar units, establishing a radio network with an estimated span of about fifty meters (like a warehouse or a boat, etc.) [7].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

88

A. Abarca et al. / Improvements in Supply Chain Tracking Using a Three-Levels RFID System

3. System Implementation Once the theoretical model to implement has been presented, this section will describe how it is applied to the real system. The system starts from the basis that every product is labelled with a passive tag at the first level. The devices used for reading these tags provide the user with the information of the products inside the pallets at this first level. Using these identification codes it is possible to access the system databases, where it is recorded the data related to the manufacturing process and also to inherent data of the products. The features of the physical system used for each level are described in detail next. 3.1. Level 2: Active Tags As previously mentioned, active tags are used in the second level. The block diagram of the active tags specifically designed for this application is shown in Figure 1. The electronic components used in these active tags have been especially selected due to its low cost and reduced power consumption. Besides, the program which controls the microcontroller for the tag is always in a low-power mode and switches to the high-power mode only for a short period of time. Therefore, the life of the batteries can be enlarged.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Basically, the active tags are made of two main components: a microcontroller and a transceiver for the RFID communication. The PIC16F819 microcontroller has been specifically selected due to its low power consumption and its simplicity. The NRF905 RFID transceiver made by Nordic was selected because it is easy to configure, and due to its high reach compared to its power consumption.

Figure 1. Active tag's block diagram

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Abarca et al. / Improvements in Supply Chain Tracking Using a Three-Levels RFID System

89

In this level, each pallet has an active tag as the one shown in supported by the middleware which is going to store, together with the steps made in the distribution process, the units and classes of product that contains. Going nearby each passive reader, this reads the product ID from the passive tags and sends all this information to update the memory of the active tags. 3.2. Level 3: Active Readers and ZigBee Network In the third level, pallets are grouped together and an active tags reader can read all the pallet tags at the same time. This pallet grouping is usually done in containers, so an RFID-like reader will be available for each one. These reads can also connect to each other, so, on one hand, the RFID reader checks the containers contents reading the pallets active tags and, on the other hand, it connects to similar units which are detected within its operational range in other containers. The block diagram of the active tags reader specifically designed for this application is shown in Figure 2. Basically, the active tags reader, as in the case of the active tags, is made of a microcontroller and an RFID transceiver. However, in this case the lower consumption is not a must, as the important issue is the reach that should be as long as possible, both for the RFID transmission and for the creation of the ZigBee network.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Due to its limited range, a network connecting all the readers should be established in order to let all the data recorded by the reader using a ZigBee connection reach the center of the RFID agent. These readers will operate as repeaters for the signals coming from the remote ones to make contact with the so-called “PAN Coordinator”.

Figure 2. Block diagram of the RFID reader.

Thereby, all the wireless communicated units can send the data required by the system through the “ZigBee PAN Coordinator”. This module is equipped with an addi-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

90

A. Abarca et al. / Improvements in Supply Chain Tracking Using a Three-Levels RFID System

tional line, for connecting to the computer through an USB bus, and it is able to manage the communication among all the modules. Sometimes, the ZigBee network is not connected to a computer or if it is, it does not have an internet connection to access the systems servers. In these cases, the ZigBee coordinator is connected via USB to a GPRS module to send the products’ data that are part of the load. So, the information about the load can be received in real time. By using this module, it is also possible to indicate the GPS coordinates in which the load is located, so the traceability data are added to the products.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 3. Final aspect of the RFID reader with ZigBee connection used in this reader

Figure 4. GE863-GPS module. Courtesy by Telit

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Abarca et al. / Improvements in Supply Chain Tracking Using a Three-Levels RFID System

91

The GSM/GPRS communication module selected is the GE863-GPS model by Telit [11], because, as well as its low power consumption, small size and weight (9 g.) The GE863-GPS module is represented in Figure 4, where its small dimensions (41.4 x 31.4 x 3.6 mm) are shown. 4. Conclusions The proposed RFID system is an effective and economic solution to the tracking of products not only over the production and distribution lines, but also during the service life of the product. Besides, though there are not direct applications, those with a special social interest, such as recycling can be implanted. If RFID is applicable to the productive and logistics processes, each unit of product will have an associated TAG code, from its manufacturing to the end of its life, with the pros and cons it implies. Pros are mentioned widely along the text: monitoring and traceability, batches’ control, stock management, etc. Cons are related with security and intrusion in personal data [1]. To prevent for this situations, it is considered to disable tags after being used or when they are given to customers. An absolute traceability and visibility, that is location of the product in real time, can be managed during logistics. In short, it means a modification of the concept of logistics as it is understood at present. Besides, the automation of the ordering process can be made much easier. The application of the proposed system provides a reliability and quality control close to 100% in a so complex and necessary process, such as picking is. It is even able to detect errors in any point of the process. In addition, with the presented system, flexibility in companies is well aided and therefore, is also improved.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The multi-agent system manages the relations among processes which influence the operation of the system, trying to coordinate its phases, and optimizing the available resources. It is open to improvements and changes in its implementation, that will achieve a greater functionality as new features are added. 5. Future Work The described work shows the great potential of a technology which opens a new world of means in applications such as the identification of products, its tracking, and the processing of the information generated using multi-agent systems. The application of the different levels of identification and wireless communications aids the performance of a extensive tracking of the products. This work is not finished yet, as the whole system for achieving the real-time localization of the products and determination of their status has not been completed. At present, the different modules have been separately tested, but the global test has not been yet concluded. The system test including all the operational parts will be completed soon, and allow the real-time evolution of the products around the world.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

92

A. Abarca et al. / Improvements in Supply Chain Tracking Using a Three-Levels RFID System

The GSM/GPRS communications system has not been still integrated in coordinator modules, as the testing period for the management of the connection from its microcontroller to the module using AT commands via a serial communication is not finished yet. However, tests have been made of its use with good results. This module is intended to send data such as the load status and its position in real time. For the use of it and of the whole system, an application program is intended to be built. With this program, the location and real-time tracking (RTLS) can be presented in a graphical way using an application like Google Maps.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] Albrecht, K. & McIntyre, L. (2006). Spychips: How Major Corporations and Government Plan to Track Your Every Move with RFID. Nelson (Thomas) Publishers. [2] Baronti, P., Prashant, P., Chooki, V., Chessa, S., Gotta, A. & Fun Hu, Y. (2007). Wireless Sensor Networks: a Survey on the State of the Art. Computer Communications. Vol. 30(7). [3] García, A. & Abarca, A. (2006). RFID Enhanced Multi Agent System For Stock Control At Group Lo Monaco, 13th EurOMA Annual Conference - Moving Up the Value Chain. Glasgow, UK. June 2006. pp. 663-670. [4] García, A., Chang, Y. & Valverde, R. (2006). Impact of new identification and tracking technologies in a distribution center. Computers & Industrial Engineering, ScienceDirect, Vol. 51(3). pp. 542-552. [5] García, A., Chang, Y., Abarca, A. & Oh, C. (2007). RFID Enhanced Multi Agent System For Warehouse Management. International Journal of Logistics: Research and Applications. Vol. 10(2). June 2007. pp. 97–107. Taylor & Francis. [6] Glidden, R., Bockorick, C., Cooper, S., Diorio, C., Dressler, D., Gutnik, V., Hagen, C., Hara, D., Hass, T., Humes, T., Hyde, J., Oliver, R., Onen, O., Pesavento, A., Sundstrom, K. & Thomas, M. (2004). Design of ultra low cost UHF RFID tags for Supply Chain Applications. IEEE Communications Magazine. No. 8. pp. 140-151. [7] Javed, K. ZigBee suitability for Wireless Sensor Networks in Logistic Telemetry Applications. (2006). Technical report, IDE0612. Halmstad University. Suecia. 2006. [8] Luh, P.B., Ming, N., Haoxun, C. & Thakur, L.S. (2003). Price-based approach for activity coordination in a supply network. IEEE Transactions on Robotics and Automation. Vol. 19(2). pp. 335-346. April 2003. [9] Rao, A.S. & Georgeff, M.P. (1998). Decision Procedures for BDI Logics. Journal of Logic and Computation 1998. Vol. 8(3). pp. 293-343. Oxford University Press. [10] Tapia, Dante I., Bajo, Javier, Corchado, Juan M., Rodríguez, Sara & Manzano, Juan M. (2007). Hybrid Agents Based Architecture on Automated Dynamic Environments. B. Apolloni et al. (Eds.): KES 2007/ WIRN 2007. Part II. LNAI 4693. pp. 453–460. 2007. Springer-Verlag. Berlin Heidelberg. [11] Telit (2009). Datasheet Modem GPRS/GSM GE863.http://www.telit.com/en/products.php?p_id=3&p_ac=show&p=8 April 2009. [12] Wong, C. Y. & McFarlane, D. (2007). Radio frequency identification data capture and its impact on shelf replenishment, International Journal of Logistics: Research and Applications, Vol. 10(1). March 2007. pp.71–93. [13] ZigBee Alliance. (2008). ZigBee specification 1.0. http://www.zigbee.org. January 2008.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-93

93

Empirical analysis of a UHF RFID multi tag pallet using a handheld reader Rafael V. Martínez Catalá a,1 and Víctor Brito Megías a Department of Information and Communications Technology, Packaging Transport and Logistics Research Center, ITENE, Spain a

Abstract. This paper presents the analysis of a particular case of study consisting of a full pallet with multiple UHF tags readings using a single handheld reader and following the EPC Class1 Generation 2 standard [1]. To perform the analysis, an empirical experiment to obtain the RSSI (Received Strength Signal Intensity) transmitted from a handheld reader and the number of readings for each tag over a fixed period of time was designed. A pallet consisting of 15 boxes filled with plastic objects was employed and measures were taken at different reading angles. To experiment with different materials the configuration of the test pallet was modified including some metallic objects in two random boxes. Finally, some conclusions regarding correlation between RSSI, tag location and number of tag readings are presented for both cases. Keywords. UHF tag location, multi tag analysis.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction Recent advances in low power microelectronics, wireless communication technologies and networking architecture have facilitated the implementation of real deployments and commercial applications of systems based on RFID (Radio Frequency Identification) technology. RFID is not a new technology, its origins date from the 40s, during the Second World War, and was used in combination with radar technology for Identification of Friend or Foe (IFF) systems [2], [3]. Since then, the technology has evolved until the point of becoming a real solution for many problems ranging from track asset, logistics to animal tracking, real time location systems among many others. In a passive UHF RFID system, the reader or interrogator transmits a modulated RF signal to the tag, consisting of an antenna and an integrated circuit chip powered by the RF energy transmitted by the reader. After powering up, the tag responds to the reader by modulating the backscattered signal with the unique identification code [1]. While RFID is a technology implanted in many companies, the installation of RFID systems posses many engineering challenges, most of them related with the optimum hardware selection (tag, reader, antennas) for a specific environment and application needs. This study presents an empirical analysis of a full pallet with multiple UHF tags using a single handheld reader and following the EPC Class1 Generation 2 standard. The paper is structured as follows; section 1 introduces the 1

Corresponding Author.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

94

R.V. Martínez Catalá and V. Brito Megías / Empirical Analysis of a UHF RFID Multi Tag Pallet

driving forces and motivation that leaded us to perform the multi tag analysis. Section 2 describes in detail, the measurement procedure and setup. In section 3 the results are presented and finally section 4 is dedicated to analyze some interesting conclusions extracted from the measurements.

1. Motivation

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Placement of RFID tagging on single pallet items is a difficult and challenging task and care should be taken when deciding on tags selection and location. While experience and a deep understanding of the technology are key factors and might help to rapidly determine optimum scenarios and successfully integrate the technology, without previous evaluation of the environment and optimization of equipment setup, the “must achieve” goal of 100% tags reading ratio cannot be ensured. The task of designing a successful RFID system includes selecting the operating frequency, hardware components such as readers, tags, and antennas. The operating frequency is normally determined by the application type, required read range, and operating conditions. Some main factors considered in selecting readers and tags are also read performance, ruggedness, compliance with standards, operating conditions, and ability to upgrade. The final goal in all cases is to maximize read performance. After the selection of the operating frequency of the RFID system, potential sources of interferences around the operating frequency must be identified. Therefore, the design is not final until a site analysis is performed. Since the operating environment for the experiment is a controlled laboratory with no interferences in the working frequency band, in this work we have focused on determining an empirical methodology to collect data in order to tune the system for optimal operation conditions prior evaluation of the environment. We have restricted to the study and characterization of a system composed of a single pallet, boxes, tags and handheld reader following the standard EPC Class 1 Generation 2.

2. Experimental Setup With the aim of characterizing the system we have setup an experiment to register the RSSI and the ratio of tags read and tag readings. The elements of the experiment are a wooden pallet of 120x100cm loaded with 15 cardboard boxes measuring 39x58x18cm each, and filled up with empty plastic bottles, as shown in Figure 1. We have included one single identified tag ALN-9540 Squiggle Inlay [4] per box trying to cover as much as possible all the space in the pallet and the possible location of the tags for a real application. Then, we have stacked the cardboard boxes in the configuration shown in Figure 2. There are three levels or stacks of five cardboard boxes per level. The bottom level contains tags 1 to 5. The orientation of the tags is always perpendicular to the ground in two different directions. The tags are located along one of the longer sides of the box.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

R.V. Martínez Catalá and V. Brito Megías / Empirical Analysis of a UHF RFID Multi Tag Pallet

95

Figure 1. Example of one of the fifteen cardboard boxes filled with empty plastic bottles and the pallet with tags and sensors

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The middle level, on the other hand, contains tags 6 to 10 and their orientation is parallel to the ground. Tags have been placed in both the top face and the bottom face of the cardboard boxes. Finally, the top level holds tags 11 to 15 with the same orientation as the tags from the bottom level. As opposite to the bottom level, the tags on the top level have been placed in one of the shorter sides of the boxes.

Figure 2. Pallet stack up of the analysis

With the aim of registering the RSSI on each tag location, we have used an RF Field Recorder [5], which has the capability of measuring the RSSI at 16 different locations with the use of their sensor modules. We have placed the modules as close as possible of the tags, but sufficiently far away to avoid interaction with the near field of the antenna. For systems operating according to the EPC Class1 Generation 2, at around 868MHz, the wavelength on free space is approximately 35cm, hence, we have distanced the tag with the sensor module around 8cm which is close to one fourth of the wavelength. The radiation pattern of a dipole antenna like the ones used in both RSSI sensors and UHF tags presents a notch on the radiation pattern in the direction of the antenna, for that reason we have placed the sensors and tags in the same direction where the interaction between the tags and the sensor is less noticeable. In order to register the number of readings per tag, a portable RFID reader ATID670 [6] is used. The device is essentially an industrial PDA with RFID capabilities. We set our measuring reading point at a reasonable distance of 2m respect the center of the pallet and at 55cm high from the ground. Then, we change the reading angle at steps of 30º until a full turn is completed, performing a total of 12 measurements per round. At

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

96

R.V. Martínez Catalá and V. Brito Megías / Empirical Analysis of a UHF RFID Multi Tag Pallet

each measuring point, both RSSI and number of readings are recorded during 20s using the multi tag read anti collision capabilities offered by the EPC Class1 Gen2 standard. A total of three rounds of measurements are performed. In order to quickly evaluate how small contents of metallic objects can affect the tag readings and RSSI inside the pallet we have emptied the boxes number 7 and 11 and filled them up with metallic objects as shown in Figure 3. The rest of the boxes remain filled with the same contents as they were in the first experiment. Then, we repeat the measurements performed in the previous assay.

Figure 3. Cardboard boxes 7 (left) and 11 (right) respectively filled with metallic objects.

3. Results

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.1. Tag readings The results for the total number of tag readings conducted in the two scenarios, with and without metallic objects, are represented in Figure 4. At glance, the first thing to notice from the graph is that there are some cases in which having metallic objects in the pallet improve the readability of the tags, as it happens for tags 1, 5, 8, 10 and 11. On the other hand, in the case of tags 3, 4 9, 12, 13 and 14 the readability is worse when metallic objects are present in the pallet. This situation is even more significant for the case of tag 6, since there is not even one reading registration in presence of metallic material. With regards of the total number of tags read or registered by the handheld interrogator, as can be seen in Figure 5, a maximum number of 9 detected tags at 210º for no metallic objects and a maximum of 8 tags at 330º in the presence of metallic objects were measured. It is also noticeable the lack of detections in the zones 240º and 270º, in addition, is worth to mention that changing the measuring angle by just 30º is translated in a change on the tag detection from 2 to 9 tags.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

R.V. Martínez Catalá and V. Brito Megías / Empirical Analysis of a UHF RFID Multi Tag Pallet

97

Figure 4. Total number of tags readings during the length of the experiment with and without metallic objects. There are 17526 readings when no metallic objects are present and 16711 readings in presence of metallic objects in the pallet.

The augment of readings of some tags, such as tag 1, in presence of metallic materials can be explained by the fact that there are fewer detected tags and therefore the anti-collision algorithm deals with fewer tags producing a higher reading rate for the same period of time.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 5. Total number of tags read during the length of the experiment with and without metallic objects

3.2. Measurement of RSSI Using the RF Field Recorder the RSSI for each sensor placed close to its respective tag is measured. Figure 6 illustrates what we call tag reception pattern; this is nothing else but the amount of received power per rotation angle. To better understand the concept, in an ideal free space scenario, the tag reception pattern would be the same as the wellknown radiation pattern of the tag antenna, as states the reciprocity principle. Since the tag is not in free space, and is, in most cases surrounded by different materials with diverse characteristics, that not only may modify the near filed of the antenna, but also absorb or reflect the incoming electromagnetic waves, the tag reception pattern is far from similar to the ideal radiation pattern. To have evidence of the effect on RSSI, Figure 6 shows the effective received power or the tag reception pattern for tags 3 and 13, that are located just opposite to the metallic objects and, therefore, the change on the RSSI is more noticeable. It can be seen that the zone comprised between 270º and 360º corresponds with the location of boxes 7 and 11.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

98

R.V. Martínez Catalá and V. Brito Megías / Empirical Analysis of a UHF RFID Multi Tag Pallet

ϭϮϬ ϭϱϬ

ϭϴϬ

ϱ Ϭ Ͳϱ ͲϭϬ Ͳϭϱ ͲϮϬ ͲϮϱ ͲϯϬ Ͳϯϱ

ϵϬ ϲϬ

ϭϮϬ

Ϭ

ϲϬ

ͲϭϬ

ϯϬ

ϭϱϬ

ϯϬ

Ͳϭϱ ͲϮϬ

Ϭ

ϮϭϬ

ϯϯϬ ϮϰϬ

ϯϬϬ

ͲϮϱ ϭϴϬ

Ϭ

ͲϯϬ

ϮϭϬ

ϯϯϬ ϮϰϬ

ϮϳϬ

EŽŵĞƚĂů

ϵϬ

Ͳϱ

ϯϬϬ ϮϳϬ

tŝƚŚŵĞƚĂů

EŽŵĞƚĂů

tŝƚŚŵĞƚĂů

Figure 6. Reception pattern of tags 3 (left) and 13 (right).

4. Conclusions 4.1. “Blind” zones From the analysis of the measurements, some areas with a significant reduction of reader-tag interaction were identified. We call this area blind zone, and for our case, is the area comprised between 240º and 270º. This area experiments a drop in both, the total number of readings and the total detected tags, as depicted from Figure 7. Apparently, and with any further analysis we have not found any other reason rather than the change of orientation of the tags to explain this issue. Despite the drop in activity, tags 1, 7 and 9 are still read within this area. Investigating the reception pattern shown in Figure 7 for tags 1 and 7, the RSSI levels at angles 240º and 270º is consider to be high enough explaining the registered tag lectures.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4.2. “Invisible” tags Tag 6 is placed on the top face of a box in the middle level. For measures with no metallic objects involved the reading ratio of tag 6 is quite high, registering a total of 1656 readings. However, in presence of metallic objects the readability of tag 6 diminishes to 0 readings. The existence of metallic objects in boxes 7 and 11 shield completely the tag 6 making it “invisible” for the reader. 4.3. Reading angle As shown in Figure 5, it is important to remark the importance of finding the best angle of reading, since, in our case, as much as 7 out of 9 tags can be either missed or found just by changing 30º the reading angle. 4.4. Tag “shading” In relation to the correct angle of reading, shielding between tags must be avoided. Figure 8 displays the effect of tag shielding. Tag number 3, 4 and 5 are in line and placed in the same position on the box. When a tag in placed and aligned between the reader and other tag, such as happens when tag 5 is interposed between the reader and

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

R.V. Martínez Catalá and V. Brito Megías / Empirical Analysis of a UHF RFID Multi Tag Pallet

99

tags 3 and 4, the communication process of the tags is disturbed. Figure 8, illustrates de number of readings per tag at reading angles of 0º and 30º. When the reader is perfectly aligned with the tags, only the closest tag (tag 5) is read. However, a slightly change of 30º in the reading angle produces a misalignment between tags and, therefore, the reader is able to register tags 3, 4 and 5. ϲϬϬ

ϭϰϬϬ

ϱϬϬ

ϭϮϬϬ

ϰϬϬ ŐƐ ŝŶ ϯϬϬ Ě Ă Ğ Z ϮϬϬ

ŐƐ ϴϬϬ ŝŶ Ě Ă Ğ ϲϬϬ Z

ϭϬϬ

ϮϬϬ

ϭϬϬϬ

ϰϬϬ

Ϭ

Ϭ

ϭ

Ϯ

ϯ

ϰ

ϱ

ϲ

ϳ

ŶŽŵĞƚĂů

ϭϮϬ ϭϱϬ

ϭϴϬ

Ϭ Ͳϱ ͲϭϬ Ͳϭϱ ͲϮϬ ͲϮϱ ͲϯϬ Ͳϯϱ ͲϰϬ

ϴ

ϵ ϭϬ ϭϭ ϭϮ ϭϯ ϭϰ ϭϱ

ϭ

Ϯ

ϯ

ϰ

ǁŝƚŚŵĞƚĂů ϵϬ ϲϬ

ϭϮϬ

ϯϬ

ϭϱϬ

Ϭ

ϮϭϬ

ϯϯϬ ϮϰϬ

ϱ

ϲ

ϯϬϬ

ϭϴϬ

Ϭ Ͳϱ ͲϭϬ Ͳϭϱ ͲϮϬ ͲϮϱ ͲϯϬ Ͳϯϱ ͲϰϬ

ϴ

ϵ ϭϬ ϭϭ ϭϮ ϭϯ ϭϰ ϭϱ

ǁŝƚŚŵĞƚĂů ϵϬ ϲϬ ϯϬ

Ϭ

ϮϭϬ

ϯϯϬ ϮϰϬ

ϮϳϬ

EŽŵĞƚĂů

ϳ

ŶŽŵĞƚĂů

ϯϬϬ ϮϳϬ

tŝƚŚŵĞƚĂů

EŽŵĞƚĂů

tŝƚŚŵĞƚĂů

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 7. Total number of lectures per tag at reading angles of 240º (left) and 270º (right) respectively and reception pattern for tags 1 (left) and 7 (right).

4.5. Tag position, RSSI and tags reading With the information of RSSI and tags readings is, in theory, possible to correlate measurements with tag position. As a rule of thumb, the tags that register higher RSSI are more susceptible to be read, nevertheless, this is not always true and it might be possible to find tags that are read under lower level of RSSI and tags that are not read exposed at higher RSSI. From this simple analysis we were not able to guarantee a minimum level RSSI required to read a tag. In order to determine this threshold level a formal statistic analysis of the scenario might be required. Without getting into deep detail, we identified that inlays located in the bottom level of the pallet experiment more lectures than the middle level and top level respectively, as seen in Figure 4. The handheld reader antenna was always placed perpendicular to the ground; we believe this effect could be explained by either reflections in the ground or directivity of the reader antenna.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

100

R.V. Martínez Catalá and V. Brito Megías / Empirical Analysis of a UHF RFID Multi Tag Pallet

4.6. General conclusions Indoor electromagnetic propagation is a topic difficult to predict and strongly depends of many external and dynamic factors such as presence of people, reflection in metallic objects in the near surroundings, interferences from other electromagnetic sources etc. Metallic objects reflect RF signals, and that reflection can cause interference with the incident signal. It is a good practice to keep metallic objects such as casings outside the interrogation zones. In some cases it might also be needed to adjust the distance and reading angle between a reader and a tag to avoid the reflection effect and shading. ϵϬϬ

ϭϮϬϬ ϭϬϬϬ

ŶŽŵĞƚĂů

ϴϬϬ

ŶŽŵĞƚĂů

ǁŝƚŚŵĞƚĂů

ϳϬϬ

ǁŝƚŚŵĞƚĂů

ϲϬϬ

ϴϬϬ

ϱϬϬ

ϲϬϬ

ϰϬϬ ϯϬϬ

ϰϬϬ

ϮϬϬ

ϮϬϬ

ϭϬϬ Ϭ

Ϭ ϭ

Ϯ

ϯ

ϰ

ϱ

ϲ

ϳ

ϴ

ϵ ϭϬ ϭϭ ϭϮ ϭϯ ϭϰ ϭϱ

ϭ

Ϯ

ϯ

ϰ

ϱ

ϲ

ϳ

ϴ

ϵ ϭϬ ϭϭ ϭϮ ϭϯ ϭϰ ϭϱ

Figure 8. Number of readings for each tag at reading angles of 0º and 30º.

This study presents the starting point of a simple methodology for characterizing multi tag RFID systems with a handheld device. Nevertheless, the dispersion of the measures taken for both RSSI and number of tag reading indicates that a formal statistic analysis is required. At ITENE we plan on continuing this analysis by further experimenting with readers, tags, materials and pallet configurations.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] EPCGlobal, EPC Radio-Frequency Identity Protocols Class-1 Generation-2 UHF RFID Protocol for Communications at 860MHz-960MHz Version 1.2.0., 2008 [2] S.B. Miles, S.E. Sarma and J.R. Williams, RFID Technology and Applications, Cambridge University Press, New York, 2008. [3] J. Landt, “The history of RFID”, IEEE Potentials, vol 24, no. 4 (2005), 8-11. [4] Alien Technology Corporation, ALN-9540 Product Overview, available online at: http://www.alientechnology.com/tags/index.php [5] Cisc Semiconductor Design+Consulting GmbH, available online at https://www.cisc.at/rfidrfcomm/field-recorder.html [6] All That Identification, ATID, website: http://www.atid1.com

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-101

101

Internet browsing through RFID-powered gadgets Leire Muguira1, Jonathan Ruiz-de-Garibay and Juan Ignacio Vazquez DeustoTech, Tecnológico Fundación Deusto

Abstract. The vision of the Internet of Things is challenging researchers from different fields with the common goal of designing innovative Internet-connected experiences or augmenting existing objects’ capabilities with Internet-related features. During these first stages, researchers must explore new ways and concepts, and share the results with the scientific community. In this paper, we present our work on creating RFID-based smart gadgets in order to take Internet experience to the next level, merging the on-line and the tangible worlds into a single continuous space. Keywords. Internet of Things, RFID, Interaction

Introduction

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The term “ubiquitous computing” was adopted in 1991 by Mark Weiser. He explored enhanced computer users through the increasing “availability” and decreasing “visibility” of processing power. In other words, “the most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it” [1]. The Internet of Things can integrate perfectly with Weiser’s vision. This new Internet will be able to detect and monitor changes in the physical status of connected things (through sensors and RFID) in real-time hiding computation from users. The Internet of Things will create innovate applications and services, and Internet will be more pervasive, interactive and intelligent [2]. In this paper we present several prototypes that realize the vision of the Internet of Things by combining several techniques, predominantly RFID and wireless communications. Section 1 analyses involved technologies from an Internet of Things point of view. Sections 2, 3 and 4 illustrate different example applications we have designed in the form of interactive prototypes, describing their goal, architecture and evaluation results. Finally, section 5 provides some conclusions of our work and future directions.

1

Email: [email protected].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

102

L. Muguira et al. / Internet Browsing Through RFID-Powered Gadgets

1. State-of-the-art technologies for IoT RFID has been defined as the main technology of the Internet of Things, however, other aspects such as user interaction must be taken into account. 1.1. RFID – Radio Frequency Identification – and NFC – Near Field Communication Radio frequency identification is a term that is used to describe a system that transmits a unique identification wirelessly, using radio waves [3] [4]. This method can be grouped under the category of Automatic Identification Technologies. Auto-ID technologies include bar codes, optical or biometric readers, such as retinal scanners. The RFID characteristics provide the following benefits (in comparison with other identification system): non-line of sight communication, read moving elements, more physically robust, larger range of reading, use with bar codes (as an mechanism), easy integration with software platform and real-time systems. In the context of the Internet of Things, it is necessary to collect data about objects, their location and status. Identifying each object is fundamental to associate data, and RFID provides just this capability. Also, using RFID is possible to integrate sensors directly with the tags. NFC is a protocol based on a wireless interface. The connection is point-to-point between two devices, operated at 13.56 MHz (it is not necessary any license for its use) and data transfer up to 424 Kbits/second. NFC is both a “read” and “write” technology. The maximum distance between devices must be less than ten centimetres [5]. A simple wave or touch can establish an NFC connection where the device, which starts the communication, is responsible for monitoring the transference. This first device sends a message; the second one receives it and generates a reply.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1.2. User interaction Other important aspects related to IoT are those who describe the user interaction. The existence of embedded computers in the environment, into objects and in the background is suggested by the concepts of Ubiquitous and Tangible Computing [6]. An important component of Ubiquitous Computing is Tangible User Interfaces (TUI) [7] [8]. Physical Browsing is an interaction method for Ubiquitous Computing [9]. Besides, Multitouch and Surface Computing are natural user interfaces that generate different ways to interact with the digital world [10] [11]. Another important concept is Social Computing. A novel game space has been created in [6] using together the theories of Ubiquitous, Tangible and Social Computing.

2. RFIDGlove: a wearable RFID reader There are different experiences in the use of RFID technology for inventory in warehouses. Reader devices have appeared with different form factors. The first experience integrating an RFID reader into a glove was [12]. Other RFID technology integration examples in clothes are [13] and [14]. In all these projects, there are some

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

L. Muguira et al. / Internet Browsing Through RFID-Powered Gadgets

103

lacks, such as: portability restriction, limited information about the tag that has been read, energy inefficiency, multimedia feedback into a small form factor and natural interaction problems. So, we decided to design an RFID reader that overcomes them. The user’s hand will be completely free for doing any task without inconveniences due to the presence of the reader. The tag reading has to be as transparent as possible: the process of moving the hand next to the tag makes it natural and the system connects with the Internet to obtain updated information about the products. As mentioned above, the main goal of the project was to design a natural interaction system, able to obtain updated information about the products. It is based on RFID tags for executing inventory works. The system should be made up of: a wearable element with a glove form. In this way, after short training, the user will not perceive consciously the act of wearing the device, being transparent; an energy efficient wireless communications system to send data about the read tags to a near information server and tolerant to mistakes to recover from coverage loss; an advanced visual feedback system, based on the display integrated in the glove, which empowers the user showing images about the handled products and allowing the information server to send instructions to the user indicating how he/she can do the possible activities. Fig. 1 shows the complete architecture of the RFIDGlove system.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 1. RFIDGlove and System Architecture.

As it can be seen in the diagram, the first step is to read a tag (1). Almost immediately, the RFIDGlove provides a feedback through the visual and audio interfaces, and sends the information got through the mesh network to the base station (2). The communication between the RFIDGlove device and the base station was developed with the MicaZ wireless module. The information server processes the message with the code of the read tag (3) and replies to the glove sending information to be displayed on the display (5). Furthermore, the server can send, automatically or manually, a text message to the glove to notify the user about performing a specific action. We have developed and deployed a sample inventory application to prove the functionality of our device. While the user moves different boxes to its corresponding shelves, the RFIDGlove communicates the movements through the network to the base station. It is able to connect to the Internet in order to analyze the existing stock, and depending on the result ordering more products on line. Additionally, the base station can connect to the Internet informing the user about extended product information downloaded from the website.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

104

L. Muguira et al. / Internet Browsing Through RFID-Powered Gadgets

Figure 2. Inventory Application.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2.1. Evaluation The distance range in which the RFIDGlove is able to make readings depends on the tag type used. By placing the RFID reader on the back of the glove the signal is reduced when it goes through the glove and the hand, the reading distance is reduced, but the reader is more transparent for the user. In addition, it is very important to extend the battery life, to perform as many readings as possible without changing the battery. Although the battery used for tests was a rechargeable battery of 9V and 260 mAh, the information is generalized for a 1500 mAh battery, similar to those available in cellular phones and digital cameras. The working frequency selected for the calculations is a read operation every 2 minutes and the average will be calculated with the corresponding percentages. The battery is supposed to have an energetic efficiency of 90%. Therefore the battery will provide practically 24 hours of continuous usage of the glove and around 720 readings. It can be extended with small optimizations depending on the usage pattern of the RFIDGlove.

3. iCompass: an Internet activity indicator Nowadays, there are several applications and gadgets that implement the concept of the Internet of Things. Some of them can be small systems that provide services to users, such us alerting when an important mail is received, telling the news or playing the radio. Nabaztag [15] is a special toy-rabbit, designed by Violet, that uses Wi-Fi technology to connect to the Internet. It can tell the weather in the user’s location or wake them up with their favorite MP3. It uses lights, sounds and the movement of its ears to communicate with the user. This approach generates a new way of keeping users informed, without screens or computers. The next step is that any object at home could be improved with these new technologies. For instance, an alarm clock that tells the daily planning, a photo frame that shows the emails or a clock that informs about the weather. In this way, an

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

L. Muguira et al. / Internet Browsing Through RFID-Powered Gadgets

105

inexperienced person could use these gadgets intuitively, and get information quickly and easily. The main goal of the project iCompass is to generate a gadget that it is easily integrated in home and provides different services. In order to manage the services, it is fundamental that their information is clearly identifiable and easily differentiated. The object selected is a compass that everybody understands how to use it. The iCompass does not point to the North but it is capable of indicating information about different services of the Internet: the weather, information about Facebook or popularity of searches in Digg. Using this gadget at home will resemble having a clock on the wall. The iCompass project has implemented two services but its flexible architecture allows integrating new ones easily. Additionally, the iCompass can be used by different users that will obtain their own responses for the same service. In order to differentiate services we use several dials that contains RFID tags. When a dial is placed over the iCompass, it reads the tag and identifies the service to make the request and show the result. Fig. 3 shows the main components of the iCompass system. The Propeller chip is the brain of the gadget and interacts with the other elements. The RFID reader gets the tag’s identifier and sends it to the Propeller chip. This element is disabled while the iCompass does not have a dial. The iWifi module allows the Propeller chip to get access to the Internet. Through this component, the Propeller chip generates the requests of the different services. Finally, the stepper motor moves the pointer appropriately with the information obtained from the Internet.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The iCompass has also some LED diodes to show the state of the system (iWifi connected, an error occurred, and so on), and other elements to initialize the system or detect the dial.

Figure 3. iCompass architecture.

As shown in the Fig. 3, the first step is to read a dial tag (1). The service is identified and iCompass generates the request (2 and 3). It receives the reply and the motor moves the pointer (4). The two services that have been implemented at the moment are weather information and popularity of searches in Digg: •

Weather.com: The user selects the city where he lives or works. When he places their weather dial in the iCompass, it accesses weather.com and

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

106

L. Muguira et al. / Internet Browsing Through RFID-Powered Gadgets

requests the forecast for the following hours. With the reply, the iCompass moves the pointer to the adequate position. To show the information, the dial contains very intuitive and easy to understand weather’s symbols (sunny, cloudy, snow, etc). In the Fig. 4, several dials corresponding to various towns are shown.

Figure 4. iCompass and example dials.

•

Digg.com: The iCompass gets the number of entries that have the word “Linux” in the latest Digg entries, thus showing the popularity and trends of a term. Here, we used the number of Tux (the mascot of Linux) to indicate the relative number of entries, because is not important the exact number. The dial can be seen in Fig. 3.

3.1. Evaluation

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The global consumption is an important aspect because the iCompass is a wireless device. The iWifi module and the RFID reader are the two elements that consume more energy (260mA and 30mA, respectively). The RFID reader is disabled most of the time and it is only active when a tag has to be read, therefore, its consumption is reduced considerably. In the case of the iWifi module, it is more difficult to disable the device because it requires some seconds to initialize and be fully operational. Generally, the use of the prototype has been satisfactory in many aspects: •

Understandable: the iCompass operation is very easy to explain using an analogy with the traditional compass.

•

Friendly service: dials have been designed with graphical elements and do not have technical or specific information, so anybody can use the system very quickly.

The response times of the services are not adequate in some cases, as shown in the Table 1: Table 1. Service times evaluation. Service Wheater Digg

Max. time 4.23s. 13.45s.

Min. time 3.11s. 7.67s.

Avg. time 3.32s. 9.56s.

The weather service has an acceptable average time, but the Digg service is a bit slow time. These values depend directly on response times of weather and digg.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

L. Muguira et al. / Internet Browsing Through RFID-Powered Gadgets

107

4. Souvenir-aware Google Earth: browsing the Internet through object manipulation Simple experiences on Tangible User Interfaces (TUI) [16] can be easily implemented using RFID technology. One of the most popular and challenging applications for TUI is browsing the Internet without using the traditional input devices, such as keyboard or mouse, but through direct object manipulation. The main advantage of activating a Web page as a result of manipulating a real-world object is that for users is more natural to associate the displayed information to a related object rather than using generic input mechanisms for displaying the web page. For example, for elderly or non-computer literate people is easy to display a weather forecast report just by placing the umbrella near the computer or TV set, which detects the proximity and automatically opens the weather.com page for their town. The traditional mechanism of switching on the computer, logging in, opening the browser and typing the URL is not only more cumbersome but also less natural.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The goal in the Souvenir-aware Google Earth prototype was to design an easy-touse Google Earth interaction technique where computer-related activities were completely transparent to the user, just focusing on the interaction itself. The idea was to provide a set of souvenirs or items related to different places in the world. In the waiting room of a travel agency, as the user moved them over to a large display showing a Google Earth view, the world globe will fly and zoom to the related location. The experience from a user’s perspective is that the world can be easily browsed just by dragging and moving objects near the display, thus, creating other ways of surfing the Internet.

Figure 5. Souvenir-aware Google Earth.

The technological infrastructure for this prototype was composed of an RFID reader embedded into the display, and hidden from the user’s point of view, and a file containing the associations of concrete RFID tags (attached to the souvenirs) to places in the world. For coding this information we used semantic technologies such as RDF [17], RDF Schema and OWL [18]. In particular the association of the tag to the concrete location was carried out by means of the rdfs:seeAlso predicate. One of the most rewarding aspects of this prototype was the feedback provided by the users about how easy and funny was to interact with Google Earth in this way. While users still had the possibility of using the normal Google Earth controls in order to move around and check concrete details, browsing to a different part of the world was much faster, and cognitively simpler, just by bringing a souvenir near the display.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

108

L. Muguira et al. / Internet Browsing Through RFID-Powered Gadgets

5. CONCLUSIONS In this paper we presented some experiences and prototypes for bridging the Internet and the real-world, based primarily on the use of RFID as a mechanism for providing “touch computing” interactions. The three prototypes described in the paper illustrate different ways of browsing the Internet with RFID, from wearable computing to direct object manipulation. It is especially interesting to study the connections between the most popular paradigm of the Internet during the last years, which is the Social Web or Web 2.0 that has boosted people collaboration, and the real world objects. Creating mappings between both worlds is at the same time exciting and problematic, since they have been isolated from each other from the very beginning. Further research in this field must address not only this challenge but also how to overcome existing difficulties in creating consumer-ready Internet-of-Things products taking into account complex user interactions and energy consumption.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] M. Weiser. The computer for the 21st century, ACM SIGMOBILE Mobile Computing and Communications Review, Vol. 3, pp. 3-11, 1991. [2] J. I. Vazquez and D. López-de-Ipiña. Social devices: autonomous artifacts that communicate on the Internet, In Proceedings of Internet of Things 2008. First International Conference, IOT 2008, Springer Lecture Notes in Computer Science, Vol. 4952. Zurich, Switzerland, March 26-28, 2008. [3] www.aimglobal.org/technologies/rfid/what_is_rfid.asp, Association for Automatic Identification and Mobility - RFID Technology. [4] R. Want. An Introduction to RFID Technology, IEEE Pervasive Computing, vol. 5, pp.25-33, 2006. [5] www.nfc-forum.org/aboutnfc/, NFC forum - About NFC. [6] A. D. Cheok, et al. Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing, Personal Ubiquitous Comput., vol. 6, pp. 430-442, 2002. [7] A. F. Blackwell, et al. Tangible user interfaces in context and theory, CHI '07 extended abstracts on Human factors in computing systems, San Jose, CA, USA, 2007. [8] T. Doering, et al. Towards a sensible integration of paper-based tangible user interfaces into creative work processes, Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, Boston, MA, USA, 2009. [9] P. Välkkynen and T. Tuomisto. Physical browsing research, 2005. [10] P. Peltonen, et al. It's Mine, Don't Touch!: interactions at a large multi-touch display in a city centre, 2008. [11] J.-B. d. l. Rivi\, et al. CubTile: a multi-touch cubic interface, Proceedings of the 2008 ACM symposium on Virtual reality software and technology, Bordeaux, France, 2008. [12] A. Schmidt, H. Gellersen, and C. Merz. Enabling Implicit Human Computer Interaction: A Wearable RFID-Tag Reader, Proceedings of the 4th IEEE international Symposium on Wearable Computers. ISWC. IEEE Computer Society, Washington, DC, 193, October 18 - 21, 2000. [13] K. Fishkin, M. Philipose and A. Rea. Hands-On RFID: Wireless Wearables for Detecting Use of Objects, Ninth IEEE International Symposium on Wearable Computers. ISWC. 38-43, 2005. [14] T.Stiefmeier, D. Roggen, G. Ogris, P. Lukowicz and G. Tröster. Wearable activity tracking in car manufacturing, IEEE Pervasive Computing. Mobile and Ubiquitous Systems, Volume 7, Number 2: 4250, 2008. [15] www.nabaztag.com/en/index.html, Nabaztag web page. [16] H. Ishii , B. Ullmer. Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 2227, 1997, Atlanta, Georgia, United States. [17] World WideWeb Consortium. RDF Primer, W3C Recommendation. World Wide Web Consortium, February 2004. [18] World Wide Web Consortium. OWL Web Ontology Language Guide, W3C Recommendation. World Wide Web Consortium, February 2004.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-109

109

On Building a Web Services-based Prototype for RFID Applications David Sundarama, Wei Zhoub, Selwyn Piramuthub1, and Schalk Pienaar a Information Systems & Operations Management, University of Auckland, Private Bag 92019, Auckland, New Zealand b Information Systems & Operations Management, University of Florida, Gainesville, Florida 32611-7169, USA a

Abstract. RFID tags are frequently being incorporated in supply chain systems that communicate with one another. Among other things, these systems communicate individually with RFID tags that pass through their field of presence as well as share data thus collected with other systems. There is a need to link these elements to develop e-infrastructures that enable an organization to learn and evolve, be agile and flexible, and adapt to the changing requirements of the highly inter-connected environment. While providing flexibility and evolvability we argue that the characteristics of information generated through Web Services and RFID tags enable us to provide services that are also stable and available. The stabilisation and convergence of standards for the execution of business processes, Web Services, RFID data enable us to design cyber-infrastructures that resemble the supply chain services they support. We develop a Web Services-based prototype for such applications. Keywords. RFID, Web Services, knowledge-based system

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Introduction Radio Frequency Identification (RFID) tags are primarily used to track and trace objects to which they are attached. These tags have been used to provide item-level information on objects as they move through a supply chain (e.g., Zhou, 2009). RFID tags can be broadly classified as active, semi-passive and passive. Active tags are characterized by their internal power source, whereas passive tags do not have their own internal power and operate by harvesting power from signals transmitted by RFID reader(s). The semi-passive tags are in-between with sufficient internal power for critical purposes such as recording sensor data on ambient (e.g., temperature, humidity) conditions. RFID tags also have limited memory and minimal processing power. They can, therefore, be used to store information about the objects to which they are attached and also for basic computations. RFID tags are primarily used to supplant barcodes and also in other applications where barcodes have traditionally not been used. Although RFID tags are generally associated with many beneficial properties compared to barcodes, several constraints preclude their widespread use. The more common among these constraints are high unit cost of RFID tags, privacy/security issues associated with their use, the general lack of infrastructure to handle their presence, and need for 1

Corresponding Author. E-mail: [email protected]

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

110

D. Sundaram et al. / On Building a Web Services-Based Prototype for RFID Applications

appropriate decision support tools for extracting useable knowledge from huge amounts of data generated in RFID-tagged systems. Unit cost of these tags is bound to come down as production volume increases and improvement in technology occurs over time. Security/Privacy issues are tricky since there is a general mistrust among the population with regard to technology (e.g., www.boycottbenetton.com) that can be used to track objects. There is a vast amount of literature on security/privacy issues associated with RFID tags and related systems (e.g., Piramuthu 2007, Piramuthu 2008 and the references therein). We consider the infrastructure used to communicate with RFID tags as they move through a service chain in order to locate and also to process the enormous amounts of data generated by these tags. Specifically, we develop a Web Services-based component prototype using peer to peer service chains incorporating RFID tags and associated systems. By using components that are based on Web Service architecture in a peer to peer context, we develop a prototype that seamlessly integrates RFID tags and their related systems including RFID tag readers, databases, and analysis tools, among others with the rest of the system that existed before RFID-related system components were implemented. The remainder of the paper is organized as follows: We provide an overview of some of the relevant characteristics of the proposed system prototype in Section 2. We provide a brief discussion and conclude the paper in Section 3.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. A Working Prototype

Figure 1. Key Features of the Prototype We developed a prototype with promising results. A series of steps are performed when creating a service chain of Web Services using the prototype application. Many of the services that we use in the context of value chain creation have been well established in traditional literature dealing with decision support. Data from RFID tags as well as

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

D. Sundaram et al. / On Building a Web Services-Based Prototype for RFID Applications

111

associated systems are gathered and maintained in all organizations involved in the supply/service chain. The data could be from traditional data sources or from RFID data sources. Models allow us to specify rules by which data can be manipulated. Such manipulation may be simple (e.g. re-arranging the data with a customer focus) or quite complex (e.g. generating growth forecasts for family of products). Solvers are implementation specific components that can integrate data and model components to generate results. The first step to using the system is to add services to the system. To add a service to the canvas (Figure 1), the user simply double clicks on the appropriate service icon in the toolbox on the left of the screen. Once the services have been added to the canvas, they are (at this point) blank representations of real-world services that will be ‘filled in’ or instantiated later. They may be positioned about the Canvas as required, and operated on using the Editor Panel. Using icon representations of services in this fashion shields users from the complexity, location, and implementation of the Web Service details that they represent. The second step is to find and access the Web Services our icons will represent. Accessing a Web Service is achieved by querying its WSDL interface via HTTP. The user provides a URL (Uniform Resource Locator) for the WSDL file and the application uses that URL to read its description. The third step is to analyze the Web Service via its description to determine the methods it offers. The XML Document object created by the application is analyzed to determine the implementation of SOAP the Web Service uses. For example, a Web Service may be implemented in SOAPLite, ApacheSOAP, MS.NET SOAP, Delphi, and so forth. Once the implementation type has been determined, the application reads the WSDL to extract the information about the Web Service methods and their parameters. The fourth step is to choose among the methods of the Web Service for use. The application generates a form that shows the user what methods are implemented by the Web Service and the parameters of those methods. The user may then select one of those methods as an input and another as the output for that input. Generally, for each Web Service, an input procedure and its equivalent output procedure will need to be selected. Selecting the desired methods of the Web Service populates the proxy service with the parameters of those procedures. Thus far, we have three blank services: a data service, a model service, and a solver service. However, our data service contains no data, our model service contains no model parameters, and our solver service does not have any active functionality available to it as of yet. To move forward, we begin by populating the data service, then the model, and then the solver service. To populate the data service, we link it to a data source. The data source could be a traditional data source such as a database or an RFID-enabled data stream or warehouse. Data captured from the RFID tags are passed onto operational data stores (ODS) using RFID readers, appropriate middleware and backend systems. These ODS could be accessed directly or the data captured could be centralized through a data warehouse. Access to such data sources are done by clicking on data service on the Canvas, going to its Connection tab, and then either typing in a connection string for the database we are looking for or clicking on the Wizard button to automatically build the connection string. Figure 2 shows the data connection wizard that appears after the Wizard button has been clicked. This form prompts the user to select the OLEDB (Object Linking and Embedding) Provider.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

112

D. Sundaram et al. / On Building a Web Services-Based Prototype for RFID Applications

Once the user has chosen a database to connect to, the Tables listbox will become populated with a list of the tables in the chosen database. The user sets up the data service by selecting a table from this list. Once the table has been selected, the application initializes the proxy service for the database. At this point, the data service is fully populated. The next step is to populate the model service. Populating the model service requires the user to query a Web Service to view its description and decide which parts of its functionality they wish to use in their model. The application lets a user query a Web Service description by typing the URL of the Web Service’s WSDL description file into the Web Service textbox, and clicking the Analyse button. As shown in Figure 3, clicking the Analyse button pops up the Web Services window.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 2. Using the Data Connection Wizard

Figure 3. Analyzing a Web Service Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Sundaram et al. / On Building a Web Services-Based Prototype for RFID Applications

113

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

At this point, the application sends a request to the URL entered by the user. The request returns a description of the Web Service in a WSDL file format. The application uses the XML Interpreter to parse the file, determine which implementation of SOAP the Web Service uses (e.g., Microsoft.NET, VelocigenX, etc.), and present the usable procedures of the Web Service in a treeview. The user is able to select from this treeview. Once the user has selected the inputs and outputs, the application transforms the description of the Web Service into a proxy service that the user can manipulate locally. To populate the solver, the user follows the same steps that were used to populate the model, because the model and the solver are in fact derived from the same Web Service. However, potentially they could be derived from different services. Once we have populated the services, the fifth step is to bind them together through a process called mapping. Mapping services together is the way in which we orchestrate them. To map the populated services together, the user selects two of them on the Canvas with the mouse, and then clicks on the mapping button on the toolbar. A blue line representing the mapping between these two services will then appear on the Canvas, linking the two icons together graphically. Figure 4 depicts the three services after they have been mapped together. Mapping the services together is an important first step towards integration. The real test of the integration capabilities of the application, however, is the ability to map the parameters of the services together, thereby fusing them into an integrated process chain. Mappings are between parameters that represent the same data item.

Figure 4. Mapping the Services Together The seventh step is to run the finished service chain. When all of the necessary services have been mapped together, populated, and their parameters mapped together, the user is able to click on the solver they wish to execute and enable the Run button in the toolbar above the Canvas. When this button is clicked, the application instantiates

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

114

D. Sundaram et al. / On Building a Web Services-Based Prototype for RFID Applications

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

each of the parameters that are mapped to other parameters, retrieves values for the parameters, and then sends those values via a SOAP message to the Web Service. Figure 5 shows the service chain after the Run button has been pressed. At this point, the solver is busy executing. A progress bar shows the proportion the solver has executed. A record of the XML messages being sent between the application and the Web Service server is written to a SOAP Log window. The application executes the solver by creating a SOAP request that contains an envelope, a header, and a body containing all of the parameters of the Web Service and their current values that are pulled from the data service. This message is then sent to the Web Service, which interprets it, executes its own methods accordingly, and then sends a SOAP response back to the application. When the Run button is pressed in the user interface, the following sequence of actions occurs. We instantiate the data sources in the service chain, get the data, execute the solver, set the data, and synchronize the data with the database.

Figure 5. Executing a Service chain First, the application finds all of the data services in the service chain, and instantiates all of the mapped parameters of those data services. This prepares them as candidates to provide data for when the solver service is looking for data to use. Next, the application retrieves the data from the instantiated parameters, and runs the solver, sending the data from the parameters to the Web Service. Once the solver has been run, the application updates the client data service with the data returned by the Web Service.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D. Sundaram et al. / On Building a Web Services-Based Prototype for RFID Applications

115

Finally, the application updates the database that the data service is linked to with the data returned by the Web Services. This entire process amounts to a series of push and pull operations, where data are pulled from one service and pushed to another. The order of this pushing and pulling defines the business process. The next step in using the system is getting and analysing the results of executing the service chain. The application allows the user to see a preview of the updated data in the Preview tab of the Editor Panel on the main window. This allows the user to analyse the results of the solver they have executed, albeit to a very primitive degree. True support for flexible analysis of data could easily be achieved through plugging in visualisation services, either in the form of other Web Services, or a local application. Once the solver is executed, the data returned by the Web Service is “pushed” back to the data service. The final step is to make the service chain persistent. It is possible to save it to an XML file by clicking on the Save command in the file menu. This allows the user to save service chains that they have created, and then load them again whenever they are needed, thereby rendering them persistent. The XML file stores the information required to regenerate distributed services within a service chain. Each service in the application has the ability to describe itself in an XML format (also known as serialization). Once it is saved, the service chain can be loaded again. The ability to retrieve stored service chains allows users to share and incrementally develop them as they require new functionality or changes to existing functionality.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3. Discussion We develop a Web Services-based prototype for use in supply chain systems incorporating RFID tags. In order for organizations to evolve, they will need to be able to reap the benefits of the new service model. Quickly and dynamically creating peer to peer interactions will allow them to build inter-organizational value chains that save time and money and increase the performance efficiency of the entire system. Our prototype provides users with these tools, while providing a set of functionality they can use to quickly and easily knit together diverse services. Web Services continue to be an important emerging technology. They have important implications for e-business in particular. We have developed a prototype that supports the creation and management of service chains with RFID tags. Being able to create and orchestrate service chains will provide many benefits to users, including increased understanding of processes and higher productivity. Service chains will also increase the ease of inter-organizational integration in the future. This seamless integration is especially critical when RFID tags that pass through various systems and organizations during their lifetime are incorporated.

References [1] S. Piramuthu, Protocols for RFID Tag/Reader Authentication, Decision Support Systems, 43(3) (2007) 897–914. [2] S. Piramuthu, Lightweight Cryptographic Authentication in Passive RFID-Tagged Systems, IEEE Transactions on Systems, Man, and Cybernetics – Part C, 38(3), (2008) 360–376. [3] W. Zhou, RFID and Item-Level Visibility, European Journal of Operational Research, 198(1) October (2009) 252–258.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-117

117

NFC/RFID applications in medicine: security challenges and solutions Antonio J. Jara a, Miguel A. Zamora a and Antonio F. G. Skarmeta a a Department of Information and Communications Engineering Computer Science Faculty, University of Murcia, Murcia, Spain {jara, mzamora, skarmeta}@um.es

Abstract. NFC technology is being included in the new generation of cellular phones and with this, a new generation of services and applications. These new applications can be defined in any scope, e.g. shopping centre, transport, education, medicine, among others. All of them share the security problems that NFC inherits from RFID. Particularly in this paper are presented the deployment of services in medical environments that can be carried out with NFC and how to apply security measures in them. We must make a distinction between these defined services into two categories. On one hand, it will be based on cellular phone, where hospital staff (doctors, nurses, etc.) can manage patient’s information, capture data from medical devices and manage of Electronic Health Record (EHR). On the other hand we find a set of services where NFC is used like communication channel to transfer information and to set up devices implanted in the human body (implants) and other solutions where we take the properties of electromagnetic induction of passive RFID to use as power source to recharge batteries of those implants (e.g. pacemakers, glucometers, etc.).

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Keywords. NFC, RFID, Security, hospital.

Introduction Cellular phone has become an indispensable device in digital lifestyle, nowadays, it has a high performance memory, high processor’s clock speed, high bandwidth and with a presence of more than 3 American billion around the world [1]. In the nowadays generation of cellular phones have been included a set of new services to improve our quality of life as well as provide a range of useful tools for our professional and social tasks, that were previously delegated to other separate devices. Therefore, the goal of the actual cellular phones is to embrace the functionality from separate devices into just one [2], some examples of these services can be found into nowadays cellular phone market are: camera, music player, GPS navigator, internet browser, email client etc. To these services is added a new generation of services from NFC technology (Near-Field Communications) and cryptographic SIM cards [3-5]. This paper presents the deployment of services in medical environments that can be carried out with NFC. We must make a distinction between two categories of services. On one hand, it will be based on cellular phone where hospital staff (doctors, nurses, etc.) can manage patient’s information, capture data from medical devices, management of Electronic Health Record (EHR), identification of patients, biosanitary material and medicines, taking patient’s vital signs and checking that diets and

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

118

A.J. Jara et al. / NFC/RFID Applications in Medicine: Security Challenges and Solutions

medicines are provided as it is indicated in the patient’s HER [6-9]. On the other hand, we find a set of services where NFC is used like a communication channel to transfer information and to set up devices implanted in the human body (e.g. pacemakers, glucometers, etc.) and furthermore to exploit the features of electromagnetic induction used in passive Radio Frequency Identification (RFID) to use as power to recharge batteries of those implants to avoid the maintenance of power sources in implants [10]. On first place in section 1 is reviewed the characteristics of NFC technology and the advantages and weaknesses which adds to cellular phones. Deficiencies from NFC in security issues lead us to incorporate a new element to get security support. This element is a cryptographic SIM Card, which has the capabilities of a normal SIM card more asymmetric cipher, digital certificate management and safe storage of data and applications; it is analyzed in section 2. Finally in section 3 is carried out an analysis of the medical solutions where can be applied RFID/NFC, defining the desirable characteristics of safety that must be accomplished, and how to satisfy it, to do NFC a safe and suitable technology for the applications in the medical world.

1. Near Field Communication

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

NFC is a short-range wireless technology that enables an interconnection between electronic devices in an intuitive, easy and simple way. It is based on passive High Frequency (HF) RFID, which operates in the 13’56 MHz frequency band, where no restrictions are applied and no license is required to use it. It has a reach between 0 and 20 cm (an average of 10 cm) and transmission speeds of 106 kbits/s, 212 kbits/s and 424 kbits/s. Remark that one of the main goals of NFC is standardization in order to achieve interoperability among solutions from different vendors. NFC is an extension of ISO/IEC-14443 standard for contactless proximity cards, which combines the interface of a smartcard and a reader in the same device, making it compatible with the current infrastructure of payment and transport based on contactless cards. 1.1. NFC device architecture We can distinguish three elements in the architecture of a NFC device: application processor, NFC controller and cryptographic SIM card as secure element (features of secure element are similar to a smartcard), as shown in figure 1. NFC device has three communication interfaces: NFC contactless interface, over the air (OTA) interface, which is the interface to communicate with the network operator (GSM, GPRS, UMTS) and human-machine interface (HMI). Cryptographic SIM card has a memory capacity of around one Gigabyte, therefore it can store our personal applications. For the correct operation and to ensure security of those applications, in the SIM card is defined a security domain for each application, hence each one is to be isolated from another one. Those applications will be loaded via Bluetooth, WiFi or network operator (OTA), this last communication interface is very important to be able to install applications for providing services via the network operator.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A.J. Jara et al. / NFC/RFID Applications in Medicine: Security Challenges and Solutions

119

Figure 1. NFC device architecture [3].

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1.2. NFC operation modes NFC supports three operation modes (see figure 2): 1. Read/Write Mode: The read/write mode allows a NFC device to read and/or write a NFC tag (e.g. smart poster). 2. Peer to Peer Mode (P2P): It is the interaction between two NFC devices, (e.g. either to exchange contact information or to make a payment). 3. Card Emulation Mode: It allows that NFC device can be read and/or written as a RFID tag. This mode is available without battery. It can be used to fill any function usually covered by any standard contactless card. 1.3. Security problems in NFC and possible solutions The security problem in NFC is inherited from RFID, which is not completely safe in most versions. Nowadays, DesFare is the only safe version after being proved Mifare vulnerability. DESFare’s encryption algorithm is based on symmetric keys (shared key), exactly DES algorithm is applied, which despite of high constraints in processing power, memory and energy consumption, that finds in RFID tags, is working. The problem is that they have not achieved response time suitable for communications with

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

120

A.J. Jara et al. / NFC/RFID Applications in Medicine: Security Challenges and Solutions

a high number of messages exchanged, together with the rest of problems that arise to manage and to deploy a shared key in an environment where a lot of business and different organizations need to work together and exchange information. Therefore, NFC device needs to be complemented by a new element to cover deficiencies in security field. In our solution, this is element is a cryptographic SIM card, which works with asymmetric encryption, hence, keys deployment is facilitated. Furthermore, it includes a large number of new features that they are very useful to improve quality of offered services. Security problems in NFC and possible solutions are [4]:

Figure 2 NFC operation modes [3].

1.3.1. Only one ID

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Each tag has only one ID; it is used for identification and in the anti-collision algorithm. Hence, it can be read and used by other card to supplant the owner’s ID card. Solution: A random generation algorithm could be used to generate a different ID. This random ID can be used in the anti-collision algorithm. Hence, real ID is just given when reader or tag has been authenticated. 1.3.2. Denegation of Service Because NFC technology activates when is approached a card, reader is going to work with wrong cards, sending error messages and even with white cards (empty cards). Hence, NFC device is busy and furthermore in case of reader is using a battery as happens in cellular phones, it is going to wear out, and reader will stop of working. Solution: We can use a button to activate the device under demand. Hence, this problem could be solved. 1.3.3. Eavesdropping in card emulation Data of the NFC card can be read even when the device is turned off, it is because card emulation mode does not need battery power to work. This implies a security problem because an unsafe device can access information that we do not want it to be accessed Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A.J. Jara et al. / NFC/RFID Applications in Medicine: Security Challenges and Solutions

121

by any user or device. Furthermore we find other problems, because we can be traced or identified without our consent when we go through close to some NFC/RFID reader. Solution: Similarly, a button could be used to activate the card emulation mode, avoiding the possible reading of the card when the user does not wish it. 1.3.4. Eavesdropping in peer to peer (P2P) mode The communications are not ciphered, therefore they could be intercepted. Solution: The solution to this requires a cipher. Two kinds of ciphers can be defined: 1.

2.

Symmetric ciphers: it needs that tag and reader shared a key, where data is ciphered with the shared key. It is a suitable solution for environments where all NFC devices are under the control of either the same domain or organizational unit. Hence, a shared key can be defined before of communications take place. Therefore, it is not suitable for large deployments where we need to obtain a safe communication between devices from multiple organizational units. We can find this solution in DESFare tags. Asymmetric ciphers: it is suitable for large deployments where we need to obtain a safe communication among devices from multiple organizational units, because it allows safe communications between two devices without a shared key. It is not available in RFID/NFC solutions.

As we have seen between these two options, asymmetric cipher is more interesting, because it could interact with a lot of different devices which have not any shared key. But it is not defined in NFC/RFID solutions. Hence, an element to carry out asymmetric ciphers need to be included. This element is a cryptographic SIM card.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1.3.5. Privacy of the device’s contents Malicious applications in our mobile could sniff the NFC index of applications existent in some cards (e.g. NXP in Mifare, JCOP) and access to resources of the device. Solution: We just allow access to application index to applications with a digital signature (for authentication that it is not a malicious application). Hence, we need to add digital certificates management to NFC devices. One more time, it is not available in NFC/RFID solutions; therefore we are going to use an element to digital certificates management. This element is a cryptographic SIM card too. 1.3.6. Social engineering In a communication medium where other devices can interact with the NFC device, even contactless is easy to use either a malignant card or a malignant reader to carry out unwanted operations. Solution: Only allow our NFC device interact with digitally signed cards and readers. Hence, we are going to use an element to digital certificates management. This is element is a cryptographic SIM card too.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

122

A.J. Jara et al. / NFC/RFID Applications in Medicine: Security Challenges and Solutions

2. Cryptographic SIM Card We realized that it needs a secure element to cover the needs of asymmetric cipher and digital certificate management. The best secure element for a cellular phone is a cryptographic SIM card [5], with the capabilities of a normal SIM card plus asymmetric cipher, digital certificate management and safe storage of data and applications. Furthermore, cellular phones are identified as a personal object that it is always with us, in consequence of this, SIM card is the ideal place to define new services. The changes in cryptographic SIM card with regard to SIM card that can be found right now in our cellular phones are: an increase in capacity from a current limited capacity of up to 256 KB, where only is stored services from the network operator, new SIM card has a capacity up to 1 GB. Moreover, it adds cryptographic functionality, which allows us to use our cellular phone to digitally sign. Finally, it also has memory space for personal files and applications. Hence, SIM is an ideal place to store critical business and personal information. It works with an asymmetric encryption and can carry out hashing operations with algorithms as either MD5 or SHA-1. Hence, it can be used to check integrity of received information and in the digital signature process. 2.1. Services 1.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2.

3.

Identification: Identity of the issuer of a query or message, it is based on digital certificates, e.g. an equivalent of either digital ID (DNI-e) for the cellular phone or business certificates (Camerfirma) [3]. Applications of this service are: user identification to access restricted zones (access to intranets, classified information, servers, specific applications etc.) and user validation and authentication (to check payrolls, absences, banking operations, etc.) [5]. Information integrity: it is applied when cellular phone receives either an application or information through any of the available networks (GSM, GPRS, UMTS, WiFi, Bluetooth or NFC), it checks that it has not been modified. It is carried out with the hashing algorithms MD5 and SHA-1. Digital signature: it is based on the union of the two previous services, because digital signature is a hash sum from the data that we are signing ciphered with the private key of the user certificate. Digital signature can be applied to all type of digital documents (i.e. signing contracts, insurance policy, special offers, etc.). In addition, digital signature guarantees the non-repudiation; thereby avoiding the issuer denies the issuance of certain information after he has sent it signed.

3. NFC services in medical environments Two kinds of applications of NFC in medical environments can be distinguished: 1.

2.

External applications to the human body: cellular phone is a place to manage the EHR (electronic health record), taking data from medical devices, consult dose of medicines, diets etc [6-9]. Internal applications to the human body (implants): NFC as passive RFID to communicate with implanted devices wireless, and in addition either to avoid the use of internal battery power or to recharge batteries without surgery [10].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A.J. Jara et al. / NFC/RFID Applications in Medicine: Security Challenges and Solutions

123

3.1. External applications to the human body We can define external applications to the human body, where NFC device is used to management of EHR (Electronic Health Record), to set up medical devices, to query the medicines that patient needs, taking patient’s vital signs and checking that diets and medicines are provided as indicated in the patient’s EHR. Hence, hospital staff’s tasks can be speeded up. Moreover, we can use that patient’s vital signs are took with the cellular phone to add this information directly to the patient’s EHR. Another interesting use in nurse’s rounds is checking that provided medication is the same that indicated in EHR. This is important because according to [6] in Finland has led to 40 million euros of additional costs, and similar figure is in United States.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.1.1. Security in external applications to the human body We know that not always we need a safe communication, e.g. typical solutions like accessing to medicines identification, where we read the medicine ID. The medicine ID is available in a simple tag without security layer. Here there is not available confidential information. Hence, it is suitable without security. The same happens for taking patient’s vital signs from a medical device, e.g. get blood pressure measures of a patient with a NFC device. It does not need security, because it is just a value without personal identification in the transaction. But when we use NFC to get or transmit information to the EHR system, we need a safe communications because in this case appears confidential information. Here can be used an asymmetric cipher, certificate managements and digital signature. Hence, that only authorized personal (nurses and doctors) can read and write information in EHR. For example this kind of communication occurs when data are transferred from the NFC device that leads hospital staff to EHR server (assuming that this system is offline) or in transmission of patient’s information from one hospital staff to another one using peer to peer (P2P) communication. Hence, in this scenario must be used asymmetric encryption with digital signatures, therefore that hospital staff can authenticate with EHR system, to set or get any classified information. 3.2. Internal applications to the human body We can define internal applications to the human body i.e. implants, where NFC like passive RFID can be used to communicate with implanted devices wireless, and in addition to avoid the use of internal battery power and can be used the properties of electromagnetic induction of passive RFID to recharge batteries without surgery. Furthermore, NFC is very interesting to communicate with devices implanted, because it is works in high frequency band (HF), exactly to 13.56 MHz, which is not considered harmful to health. In addition, this value is common in all countries. NFC communication with implanted devices can be for collecting data, setting up parameters and power supply (recharge batteries). 3.2.1. Security in internal applications to the human body Here we need to define a security layer, because we cannot allow for example that anybody can modify a pacemaker’s configuration. We have defined two solutions, asymmetric and symmetric cipher. Here an asymmetric cipher is not available because Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

124

A.J. Jara et al. / NFC/RFID Applications in Medicine: Security Challenges and Solutions

an internal device cannot provide the elements that are needed for that solution. Hence, here the best option is symmetric cipher. Furthermore the devices (readers and implants devices) are going to be very well known, therefore that a shared key can be assigned without deployment problems. 4. Conclusion Integration of NFC in cellular phones improves usability of systems, a better access to services and capacity to interact with the context, what leads us to improve user experience. With regard to the medical environments, inclusion of NFC solutions in these places bring us the next services, on one hand, from the external applications to the human body, this improves the offered services, reduce errors, costs and facilitate digitalization of patient’s records (EHR). Hence, this speeds up medical process, which is required because the lack of hospital’s staff. On the other hand, from the internal applications to the human body, it is interesting to communicate with devices implanted, because the use of batteries can be avoided with the properties of RFID passive. Finally, this paper has shown that NFC is not a safe technology. But it can be managed to do safe the communications, adding a SIM cryptographic card. The feature of security in NFC for medical solutions is necessary because, it works with protected data, i.e. personal and medical information. To sum up, this paper defines a set of security measures for medical services, where RFID/NFC technology can be applied. 5. Acknowledgment This work has been carried out in frames of the Spanish Program: Programa de Ayuda a los Grupos de Excelencia de la Fundación Séneca 04552/GERM/06 and the FOM/2454/2007 project from the Spanish Government.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] Abuin, Rubén, de Benito, Raúl (2007). Cátedra Movistar de la Universidad de Deusto, Near Field Communication. http://www.morelab.deusto.es/images/talks/NFC.ppt [2] Cossio, Carlos. RFID Magazine (2008), Tecnología NFC para aplicaciones de proximidad seguras. [3] TID (2009), Telefónica I+D, http://www.tid.es/ [4] Madlmayr, Gerald et al (2008), NFC Devices: Security and Privacy, The Third International Conference on Availability, Reliability [5] Calman, György et al (2007), SIM as Secure Key Storage in Communication Networks, Proceedings of the third international conference on wireless and mobile communications (ICWMC’07). [6] Lahtela, Antti et al (2008), RFID and NFC in Healthcare: Safety of Hospitals Medication Care. Pervasive Computing Technologies for Healthcare, 2008. PervasiveHealth 2008. Second International Conference on Jan. 30 2008-Feb. 1 2008 Pages: 241 - 244 [7] Bravo, J et al (2008), Tagging for Nursing Care. Pervasive Computing Technologies for Healthcare, 2008. PervasiveHealth 2008. Second International Conference on Jan.30 2008 - Feb. 1 2008 pp: 305 307 [8] Strömmer, Esko et al (2006), Application of Near Field Communication for Health Monitoring in Daily Life. Proceedings of the 28th IEEE EMBS Annual International Conference. [9] Bravo, José et al (2007), NFC: Una nueva forma de concebir la RFID. Aplicación para grandes superficies. 1as jornadas de RFID in Ciudad Real (Spain). [10] Freudenthal, Eric et al (2007), Suitability of NFC for Medical Device Communication and Power Delivery. Engineering in Medicine and Biology Workshop, 2007 IEEE Dallas 11-12 Nov. 2007 pp:51-54 [11] Jara, Antonio J., Zamora, Miguel A., Skarmeta, Antonio F. G (2008). Modelo de negocio en centros comerciales para servicios NFC. 2as Jornadas científicas sobre RFID in Cuenca (Spain). [12] Das, Raghu (2008), NFC – enabled phones and contactless smart cards 2008-2018, Card Technology Today, Volume 20, Topics 7-8, July-Ausgust 2008, Pages 11-13.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-125

125

RFID and Real-Time CRM Wei Zhou and Selwyn Piramuthu 1 Information Systems and Operations Management, University of Florida, Gainesville, Florida 32611-7169, USA; E-mail: {selwyn, wzhou}@ufl.edu Abstract. The importance of Customer Relationship Management(CRM) to firms in exceedingly competitive environments where customers have an increasing array of access to information cannot be overstated. There has been a resurgence in interest in CRM, specifically real-time CRM, as more data about customer behavior and means to processing these data to generate actionable policies become available. Recent years have also seen the emergence of Radio-Frequency Identification (RFID) tags in a wide variety of applications where item-level information can be beneficially leveraged to provide competitive advantage. We provide an overview of existing literature in this are and then propose a knowledge-based framework for real-time CRM incorporating RFID-generated data. We discuss possible scenarios where such a framework could be utilized. Keywords. RFID, real-time CRM, knowledge-based system

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction There has been a massive power shift from vendors to customers in recent years due in part to the seemingly limitless selling channel brought about by the Internet. Customers today have more information about where they can look for what they want, including alternate sources of providers. They expect immediate gratification. And if a vendor is unable to deliver, the customers are more than willing to move on to competing vendors. Unlike in the corner market case, it is becoming rather difficult for vendors to keep track of their customers, understanding and responding to their needs and wants. In the days long gone, the vendors were able to remember almost every one of their regular customers, along with their other relevant information. The means as well as the media used to interact with customers have changed dramatically over the past decade. Continuing business with a customer is no longer a guarantee. Today’s customers have more complex demands and much higher expectations than ever before. Added to this is the criticality of the ability of the vendors to pro-actively assess the needs and wants of customers to be able to quickly respond to these with the appropriate products and services. In order to be successful under these conditions, a vendor can no longer afford to wait until the signs of customer dissatisfaction appear before taking any action. Several factors have contributed to the change in the extent and mode of interaction among vendors and customers. This include (1) compressed marketing cycle times (if the customers’ needs/wants are not met by a vendor, there are several competing vendors who are both willing and able to meet those wants/needs - this has led to shrinkage of 1 Corresponding

Author

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

126

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

marketing cycle times), (2) improvement in knowledge of customers’ wants and needs, in general, thanks to advances in data collection, analyses, and interpretation techniques as well as the availability of relatively cheaper data storage and computing power, (3) increased marketing cost (given the speed of knowing about customers as well as their needs/wants, the extent of competition, and the low margin due to strong competition, the cost to market and value-add to distinguish one’s products/services also increases), (4) streams of new product offerings (customers want their needs/wants satisfied exactly and not approximately - thus the increase in product/service variety and the trend toward mass-customization), (5) niche competitors (a vendor’s best customer is also on the radar screen of competitors who are niche players), (6) changing customer expectations due to the extensive availability of a wealth of information, and (7) the difference between loyalty and captivity of customers, where the former is favored by customers who choose a vendor based on a superior value proposition in the product and/or service and the latter happens when customers are locked-in for lack of a choice. Therefore, to be competitive, a vendor needs to make the right offer to the right customer at the right time through the right channel. The right offer and customer here simply refer to making the most appropriate offers while keeping the irrelevant ones to a minimum. The appropriate ones are those that satisfy specific customer’s wants/needs. It also refers to (mass) customization since all customers most likely do not have the same wants/needs under all circumstances. The right time is achieved through continual gauging from observations and interactions with customers. Given the availability of several channels, their differing effectiveness, interactive-ability, as well as intrusiveness, the choice of medium for the interaction is important. Customer Relationship Management (CRM) has been touted as a partial answer to the question of being able to offer and deliver the right goods and services at the right price to the right customer at the right time. CRM has been defined as a way to identify, acquire, and retain customers; a way of automating the front office functions of sales, marketing, and customer service; as a way to leverage the internet as a channel for customer care and prevent customer churn through the use of web site personalization; bringing the call center to the desktop and provide real-time response to customer inquiries. Research firm The Gartner Group defines it as consisting of a selling platform, a communications infrastructure, and a suite of applications. Infrastructure can include email, online chat, among others. And the application can be geared toward self-service, customer service, or data analyses. The diversity of definitions is a result of differences in perspectives. CRM evolved from the Sales Force Automation (SFA) market, which in turn evolved from contact management. Contact management helped sales personnel manage their prospects’ and customers’ names and contact information. This helped them to keep track of their past contacts with prospects and customers to be used for follow-up purposes. This process helped them manage their time and customer interactions more effectively. From the level of individual sales personnel, this process further evolved to encompass the entire sales organization. Sales personnel entered their opportunities into the SFA application, and these opportunities are evaluated to determine their status in the sales pipeline. This was also used to generate sales forecasts. This then evolved to CRM to extend the same technology to field service groups, marketing group, billing department, help desk, and so on. Recently, AMR Research considered all customer management applications in its study including the historical trio of sales, marketing and service and

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

127

predicted that the market for CRM products will continue to climb, reaching $18 billion worldwide in 2010. CRM is not a panacea. CRM has been around for around at least three decades, and hundreds of packaged software applications are specifically targeted for this purpose. However, several surveys including those by Gartner in 2002 indicate that 50 to 70 percent of CRM implementations do not deliver on their original promise. Moreover, it has been reported by CRM-Forum that the abandonment rate for CRM projects are around 5 to 11 percent. Gartner, in 2005, also found that about 75 percent of CRM projects that do not deliver measurable ROI failed due to poor executive decisions. Radio-Frequency Identification (RFID) tags are increasingly being used throughout the supply chain from raw material providers to retail shop floors to generate item-level information about tagged objects (Zhou, 2009). Researchers in this area are beginning to appreciate the potential associated with RFID tag-generated data for CRM. We provide a brief overview of existing literature in this area in Section 1. We then present our proposed knowledge-based framework for CRM incorporating RFID-generated data in Section 2. We conclude the paper with a brief discussion in Section 3.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Related Literature During the past few years, several publications have considered the beneficial aspects of RFID-generated data in CRM applications. We consider a few of these publications in this section. Schloter and Aghajan (2005) propose a system for real-time CRM. This system comprises RFID-embedded store cards that are distributed to the customers who carry these cards when visiting the store, wireless sensors and aggregator motes that are distributed throughout the store to collect RFID information, aggregation server, database, means to mine data in database, and a Web server for input and output of information. They The sensors are manually moved to various locations within the store to gather relevant information from shelves in different aisles. A Web interface is used to let the customers sign into the system before the process begins, and the authors mention that this could be automated through use of RFID readers at the store front. Information from the card is linked with the customer’s demographic information including ZIP code, gender, age group, level of education, purchase history, product booth visited, among others. These data are then visualized using “heat maps" to identify problem areas and address them as they occur. They propose addressing customer privacy issues by assuring the customers that the data thus collected will be used solely for statistical analysis. Being among the earliest publications in this area, this paper does an excellent job of identifying opportunities for RFID-enabled systems for CRM. Lin, Lo, and Chiang (2006) provide a generic overview of customer service issues associated with supply chain management in the presence of RFID. Unlike Schloter and Aghajan (2005), they do not identify specific issues that could be address nor propose a system that incorporates customer service issues. The objectives of Bayraktar and Yilmaz (2007) is similar to Lin, Lo, and Chiang (2006) in that they provide a generic overview of customer loyalty issues but do not provide any specifics to develop a system that incorporates these issues. Inaba (2009) addresses issues related to reducing costs through appropriate supply chain management and improve profit through improving CRM. Inaba proposes al-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

128

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

gorithms that address issues related to customer loyalty status and inventory turnover through RFID-generated data to provide customized one-to-one promotions. Inaba also develops a prototype and present results using this prototype. The proposed system includes loyalty cards with RFID tag, RFID reader with display, a dynamic pricing engine that computes instantaneous price information based on the customer’s loyalty status and the item’s inventory level, a customer database, an inventory management system, and a recommendation engine that provides recommendations on other comparable products. He proposes providing customers with RFID tag-embedded loyalty cards and placing RFID tag readers with display in the store for customers to scan their cards to receive the item price on their items of interest. His main idea is to present different prices to different customers based on their loyalty levels. While RFID-embedded loyalty cards already exist, and has been shown to be effective in generating data about a customer’s purchasing habits, etc., RFID tag readers with display does not seem like a feasible idea since it expects customers to pick up, walk to the nearest tag reader, and scan every item that they are even remotely interested in purchasing. On the other hand, these readers could, in principle, be placed on all shopping carts. This may be an expensive option depending on the number of carts at the store and the security of these carts themselves since the customer’s privacy and security could be violated when any of these RFID readers are compromised. Inaba also provides a mechanism to control inventory levels, but given the extensive literature in inventory control the proposed mechanism ignores the complexities associated with this process and is therefore neither novel nor effective in practice. To address issues related to customer privacy and security, he mentions that measures that prevent other customers from viewing the display screen need to be in place and as to the legality of charging different customers different prices for the same item he does not have a solution. We considered a few related publications that address some of the issues related to gathering information using RFID tags and utilizing this information for CRM purposes. Given the recency of this phenomenon (using RFID tags to aid in CRM), there are no standards nor time-tested frameworks that address several issues that are specific to the CRM domain. In the next Section, we attempt to address some of these issues through a proposed knowledge-based framework.

2. Proposed CRM Framework with RFID-generated Data 2.1. Consumer Preference A traditional view of CRM involves the use of technology to attract new and profitable customers as well as forming tighter bonds with existing ones. The capability to observe and understand customer preferences is of paramount importance for a firm to develop an effective CRM strategy. The existence of consumer preference has been widely acknowledged in academic literature from different disciplines. For instance, Carpenter and Nakamoto (1989) show that consumers form preference not only in a matured market but also on new products that have “pioneering advantages." Dynan (2000) evaluates household data on food expenditures and finds evidence for habit formation. Consumers not only form preference on items sold but also on shopping behavior. Bell and Lattin (1998) show that brick and mortar shoppers with “large basket" are more

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

129

likely to benefit overall from everyday low price than other marketing strategies. On the other hand, shoppers with a relatively short shopping list may prefer occasional deep promotions. Assessing consumer preference is critically important for practitioners to fully understand and utilize this consumer preference information in the market. Consumer preferences on price and other easily quantifiable product attributes have been studied extensively. Other preferences, such as environmental attributes, related to a product’s production are often difficult to assess, but nevertheless may have important welfare implications for certain consumers. Wessells et al. (1999) evaluate consumers’ possible acceptance of an eco-labeling program that integrates environmental attributes for seafood products based on a contingent choice survey in which respondents chose between a variety of certified and uncertified seafood products. Overall, despite extensive importance assigned to it in related business disciplines, CRM with fine granular information enabled by modern identification technology such as RFID has not been studied until recently. We consider RFID technology, consumer preference assessment, and computerized adaptive learning mechanism to develop an effective customer relationship management framework for business adopters. In what follows, we incorporate and present a framework utilizing fine granular information by modern tracing and tracking technology and information systems, and strategies to improve existing CRM initiatives. 2.2. CRM Workflow with Fine-grained Information Figure 1 demonstrates the work flow of CRM with RFID data acquisition and consumer preference learning. Consumers in the market form preferences on items which have a diRetailing Consum er Preferenc e

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Consumers

M erch an d ize

Market Selec tion

learning

Pric ing

Co n sum er p referen ce K n o wledge

learning

Optim ization

CRM

Figure 1. Workflow of CRM with RFID data and consumer preference analysis

rect influence on their shopping behavior. In the meanwhile, marketers carefully estimate the value of the market and adjust as well as target their products to appropriate groups of consumers. Transactions are made during some of these interactions between customers Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

130

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

and marketers. Information from transactions as well as non-transaction information are collected and learned to form a knowledge base on consumer preference that is further utilized to design an effective CRM strategy. While the objective here is to retain maximum customer patronage, fine granular information acquired through RFID-generated data provides more accurate and useful suggestions for the marketers than those obtained through traditional (relatively coarse) information collection methods. 2.3. Proposed Adaptive System

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The proposed knowledge-based framework for CRM using RFID-generated data is given in Figure 2. Traditionally, stores have obtained information about customers through several means including non-transaction data (surveys, phone/fax/mail/email inquiries, complaint logs, cookie logs when the customer visits the store’s Web site) and transaction data (order information tracked when a customer uses a loyalty card). We assume all these to be present. In addition, we introduce RFID-generated data to aid in decisionmaking with the ultimate goal to improve CRM.

Figure 2. RFID investment decisions across time

Similar to related systems in extant literature, we assume the existence of RFIDembedded store-issued loyalty cards that the customers carry with them when visiting the store. A reader at the store entrance reads these cards as customers enter and leave the store. This provides data on the time that any given customer spends inside the store during any given visit. The store is also assumed to contain RFID readers in all its carts and shelves so that a customer is always in the field of a reader at all times when inside the store. Issues related to collision due to multiple readers simultaneously attempting to read a card are not considered, and are assumed to be taken care of through other means (e.g., Piramuthu, 2008). We also do not consider security/privacy issues here since this is an extensively studied area (e.g., Kapoor et al., 2008). These readers are assumed to

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

131

be able to accomplish at least the following when the customer is inside the store: (1) identify the (approximate, to a reasonable degree of accuracy) location of the customer at all times, (2) identify the items that are present in the cart at all times including those that were placed and later removed from the cart, (3) identify the items (on shelves) that are in the proximity of the customer at all times, and (4) be able to compute the time duration that the customer spends at each location of the store using the above three. In general, data about customers are created whenever customers interact with the store through the store’s Web site, a call center, a retail outlet, a response to a promotion, etc. These data are usually stored in operational databases that are designed for speed rather than for purposes of data analysis. In most cases, the focus of these databases is on fast response to queries and reduction in delays while interacting with customers. Therefore, historical data are usually archived elsewhere or destroyed. Padmanabhan et al. (2001) identify differences in results using data from user’s side vs. log data obtained from customer’s visit to the store Web site. The most critical feature of any CRM solution is the ability to transform customer data, collected from a wide variety of sources, into the type of detailed customer information around which a company can organize its enterprise and build its customer relationships. In the proposed framework, (transaction, non-transaction, and RFID-generated) data collected in the system are first cleaned and pre-processed to remove noisy data as well as to format the data in a form (e.g., feature selection, feature construction) that is conducive to extracting useable patterns in the data. The cleaned and pre-processed data are then used as input for data mining and knowledge discovery, including the process of association rule mining. The useable knowledge extracted from this process is then stored in the system knowledge-base. The knowledge-base also contains domain knowledge on the products in the store as well as the store layout. The knowledge-base can then be used to generate ideas for service and product development and customization. The requirements matching engine matches recent transaction data with related knowledge in the knowledge-base to generate customized promotional offers including coupons, discounts, additional store hours for premier customers, etc. Several advantages can be realized using RFID-generated data in CRM applications. For example, the following exemplifies potential improvements in information content of knowledge used for CRM decision-making in a retail store setting. • Departure/arrival time information: It is widely accepted that the longer a customer stays in a store, the higher the likelihood for purchasing (more) items. The store personnel could, therefore, study the purchase, behavior, and demographic characteristics of customers who spend more time at the store and try to accentuate store characteristics that would entice the customers to stay longer in store. The actual time of day (e.g., 8 A.M. - 8:10 A.M. on a Wednesday) and duration could also be used to appropriately target customers for promotional offerings. • Instantaneous location of customer information: For example, if the customer stood next to some item for a long time but left the store without buying that item, this customer may be provided with necessary coupon(s) for the item of interest. Knowing this information, the customers could be provided appropriate instantaneous dynamic pricing as well as coupons for their next visit to this store. • Instantaneous shopping cart information: same as above. • Information on items considered but not purchased: When a customer picks an item from a shelf but puts it back without purchasing it, it is highly likely that the

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

132

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

customer was unsure of his/her decision to purchase this item. The store could offer some promotion (e.g., discount, coupon) to facilitate the customer’s decision to purchase this item either right away or during the next visit to this store. The store can also, maybe, target a few carefully chosen items that are in the cart for additional discount. • Information on customer response to promotions: These promotions can be instantaneous based on RFID-generated data. The response from the customer to such a promotion can be monitored and learned to fine-tune the promotional offerings custom-tailored to each customer.

3. Conclusion RFID has garnered much attention from both industry practitioners and academic researchers recently thanks to its advantages in being able to render fine-grained item-level information. Its additional value and potential applications in CRM are still being studied. We consider some possible potentials in utilizing RFID tracking/tracing identification technology to gain refined and better information about consumer preference, which in turn can be beneficially utilized as input for effective CRM strategy design. With refined information on (1) timings of consumers’ shopping activities (2) shopping cart details and (3) customized promotions, we show that marketers can develop more effective CRM strategy based on the proposed adaptive learning mechanism. We are currently in the process of implementing and studying the dynamics of utilizing RFID-generated information vis-à-vis traditional CRM approaches.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

References [1] A. Bayraktar and E. Yilmaz, Implementation of RFID Technology for the Differentiation of Loyalty Programs, Proceedings of the First RFID Eurasia (2007), IEEE Society Press, 1–6. [2] D. R. Bell and J. M. Lattin, Shopping Behavior and Consumer Preference for Store Price Format: Why "Large Basket" Shoppers Prefer EDLP Source: Marketing Science 17(1) (1998) 66-88 [3] G. S. Carpenter and K. Nakamoto, Consumer Preference Formation and Pioneering Advantage, Journal of Marketing Research 26(3) (Aug., 1989) 285-298. [4] K.E. Dynan, Habit Formation in Consumer Preferences: Evidence from Panel Data, The American Economic Review 90(2) (Jun., 2000) 391-406. [5] T. Inaba, Realization of SCM and CRM by Using RFIC-Captured Consumer Behavior Information, Journal of Networks 4(2) (April 2009) 92–99. [6] G. Kapoor, W. Zhou, S. Piramuthu. RFID and Information Security in Supply Chains, Proceedings of the 4th International Conference on Mobile Ad-hoc and Sensor Networks (MSN08), (2008) 59–62, IEEE Computer Society. [7] H.-T. Lin, W.-S. Lo, and C.-L. Chiang, Using RFID in Supply Chain Management for Customer Service, IEEE International Conference on Systems, Man, and Cyberbetics (2009) 1377–1381. [8] B. Padmanabhan, Z. Zheng, and Steven O. Kimbrough, Personalization from Incomplete Data: What you don’t know can hurt, ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), (2001), San Francisco, CA. [9] S. Piramuthu, Adaptive Framework for Collisions in RFID Tag Identification, Journal of Information & Knowledge Management 7(1) (March 2008) 9–14.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

W. Zhou and S. Piramuthu / RFID and Real-Time CRM

133

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[10] P. Schloter and H. Aghajan, Wireless RFID Networks for Real-Time Customer Relationship Management, Proceedings of the EUC Workshops (2005). LNCS 3823, 1069–1077. [11] C. R. Wessells, R. J. Johnston and H. Donath, Assessing Consumer Preferences for Ecolabeled Seafood: The Influence of Species, Certifier, and Household Attributes American Journal of Agricultural Economics 81(5) (Dec., 1999) 1084-1089 [12] W. Zhou, RFID and Item-Level Visibility, European Journal of Operational Research, 198(1) (October 2009) 252–258.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

4th International Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’09) Juan C. Augusto University of Ulster, UK Diane J. Cook Washington State University, USA

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

John O’Donoghue University College Cork, Republic of Ireland Antonio Sgorbissa University of Genova, Italy

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-137

137

Synthetic Training Data Generation for ADL Modeling Dorothy N. Monekosso a,1 and Paolo Remagnino b a CAWS, CIT, Ireland b CISM, Kingston University, London, UK

Abstract. This paper describes a data generator to synthesize sensor

observations in the context of environment monitoring. The overall goal of our work is to monitor the well-being of occupant(s) in a home. Sensors are embedded in a smart home to unobtrusively record environmental parameters. Based on the sensor observations, behavior analysis and modeling are performed. Behavior modeling and analysis require large data sets to be collected over long periods of time to achieve the level of accuracy expected. A data generator - was developed based on initial data i.e. data collected over periods lasting weeks to facilitate concurrent data collection, and development of modeling algorithms. The data generator is based on statistical inference techniques to produce models. Variation is introduced into the data by perturbing the models. Keywords. Synthetic data generation, perturbation model, statistical analysis, HMM modeling.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction This paper describes an Activity of daily Living (ADL) data generator to assist with the development of methodologies in the context of assisted living. The simulator produces observations consistent with activities of daily living in a smart home. The goal of our work is to monitor the well-being of occupant(s) in a home. This is achieved by observing activity and analyzing behavior. In analyzing behavior, we aim to detect gradual changes and atypical (sudden changes in) behavior. The smart home is equipped with sensors that monitor directly environmental conditions and indirectly the occupants. Activity is inferred from observations and models of behavior are built. However data analysis and modeling requires large data sets collected over long periods of time to achieve the level of accuracy needed. To allow development of algorithms to proceed concurrently with data collection, synthetic data is generated based on perturbation methods. The distribution parameters of a limited sample are the initial values. In the next Section, the need for extensive datasets is explained. In Section 2, we describe the architecture followed by a discussion of the probabilistic techniques used

1

CAWS, CIT, Cork, Ireland. [email protected]

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

138

D.N. Monekosso and P. Remagnino / Synthetic Training Data Generation for ADL Modeling

for generative modeling. In Section 3, we present results followed by discussions and conclusions in Section 4.

1. Related Work Intelligent Environments (IE) employ embedded sensors to record activity. Dependent on the application goal, various machine learning and statistical techniques are used to analyze and infer from the data generated by the sensors. A number of research projects addressing issues of intelligent environments have been published; including Microsoft's EasyLiving project [1], the Intelligent Dormitory iDorm [2], the Interactive Room iRoom [3], the HyperMedia Studio [4], and The MavHome project [5]. Examples of IE can be found in health applications [15] to [17]. The sensors range from arrays of relatively simple devices that record on/off status, temperature and lighting level [13], [7], [25] to more complex sensors to record sound and images [9], [10]. Supervised ([7], [8], and [9]) and unsupervised learning ([11] to [14]) algorithms have been applied to learning a model of activity. In the above mentioned works, the problem is one of pattern classification solved with machine learning techniques. These techniques generally require a large dataset for training; the lack of data hinders progress in algorithm development in many applications such as sign language, speech and handwriting recognition, and human face classification. Improvement of pattern classification with sample size was investigated in ([18], [22]). Methods for generating synthetic data include Ber resampling of face images [19], genetic algorithms to produce new face images [20], mean shift algorithms for sign language [21] and perturbation models for handwriting recognition [23]. The method presented in this paper is akin to the latter.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. Methodology The requirements of the data generator are easily understood by describing the system that it simulates. The typical home comprises a number of rooms (locations); each location contains embedded sensors and a sensor processing board to monitor environmental conditions. The sensors are of two basic types: continuous output sensors such as temperature, and discrete output sensors such as the on/off status of appliances. Communication between sensor boards and the base station is wireless. The data is noisy; comprising intermittent failure of IE equipment to measure and/or record activity, RF interference or appliance failure. Added to these noise sources is the variability of the occupant’s daily routines. The data generator accounts for noise in observations using statistical models (Figure 1). 2.1. Synthetic Data Models for Events Two types of events are defined; the sensor state when sampled referred to as a primitive event and events that are defined by the start and end of an activity referred to as compound events. The days are divided into periods; during each period the events of interested are counted. Event counts are modeled as probability distributions. The parameters for a suitable distribution i.e. the shape, location and scale parameters are

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D.N. Monekosso and P. Remagnino / Synthetic Training Data Generation for ADL Modeling

139

estimated using a graphical technique based on probability plot and probability plot correlation coefficient (PPCC). To generate synthetic data, the model parameters are perturbed. Consider a base normal distribution, N0—0, ı0), for an event type with ORFDWLRQ—0DQGVFDOHı07KHSDUDPHWHUVDUHFRQVWUDLQHGZLWKLQERXQGV>—min—max] DQG >ımin  ımax] determined experimentally. Synthetic data is generated from the SHUWXUEHG GLVWULEXWLRQ ZLWK SDUDPHWHUV —1 —0ǻ— DQGRU ı1=ı0ǻı ZKHUH ǻ— DQG ǻı are selected randomly. To generate a desired superimposed trend, at each step in time ti, the distribution becomes Ni+1 —i+1 ıi+1) = Ni —i ıi)+ Ni—i ıi)×İ ZKHUH İLV D VPDOO FKDQJHDSSOLHGWR—DQGı

Figure 1. Model-based data generator.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2.2. Synthetic Data Models for Behavior Profiles A behavior profile is a detectable pattern in the sequence of observations that make up the ADL. The data-generator must be capable of generating varied daily, monthly or even yearly profiles. Variations in a profile might be a result of seasonal variations or a trend with some diagnostic value.

Figure 2. Profile synthesis. Behavior profiles are built using Hidden Markov Models (HMM), a statistical technique for modeling based on the assumption that the process is a Markov process with hidden parameters. The states are not directly observable but the process has observable parameters and the hidden parameters can be determined. A Markov model is a stochastic state automaton in which a state has associated a prior probability and a set of transition probabilities. The prior probability of a process describes the probability of starting in a given state, and the transition probability describes the likelihood of a process moving into a new state. Evaluation of the HMM parameters requires calculating 1) WKH SULRU SUREDELOLW\ ʌi for each state Si , representing the probability that a particular state is the starting state,

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

140

D.N. Monekosso and P. Remagnino / Synthetic Training Data Generation for ADL Modeling

2) the transitional probabilities aij between two states Si and Sj and 3) the probability distribution function bj(O) of an observation vector O for a state, Sj, i.e. the conditional probability of a particular position O given the state Sj . 0RGHOVHOHFWLRQFDQEHSHUIRUPHGE\ILQGLQJWKHPRGHOȜZKLFK\LHOGVWKHKLJKHVWD posteriori likHOLKRRG 3Ȝ_2  JLYHQWKH VHTXHQFH RI 1 REVHUYDWLRQV 2  21, … , ON ) associated with the time series. Reproducing the work of Rabiner [22], the probability RIWKHREVHUYDWLRQVHW2JLYHQWKHPRGHOȜLV

P (O | O )

¦S

b (O1 )D q1q 2 D qT 1 qT bqT (OT )

q1 q1

(1)

q1qT

At each moment in time, the likelihood of a model given the current set of observations LV FDOFXODWHG 7KH PRGHO Ȝ WKDW \LHOGV WKH KLJKHVW D SRVWHULRUL SUREDELOLW\ LV WKH RQH FXUUHQWO\SURYLGLQJWKHPRVWOLNHO\LQWHUSUHWDWLRQLHȜ  DUJPD[Ȝ‫א‬a P(O_Ȝ  The most likely model is calculated using the classical forward iterative procedure. The process LVUHSHDWHGXQWLOWKHWHUPLQDWLRQVWDJHLQZKLFKWKHDSRVWHULRULSUREDELOLW\RIDPRGHOȜ LV FRPSXWHG E\ VXPPLQJ RYHU DOO ILQDO YDOXHV RI WKH Į YDULDEOHV computed for the PRGHOȜ Thus with an HMM model a profile can be generated.

3. Experimental Results

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The data was obtained from the experiments, described in [16], recorded over several weeks. Observations were recorded as time stamped events (Figure 3). 15/02/2007 06:20:55 22855.7 0 BathMotionDetect 1 15/02/2007 06:20:55 22855.7 1 BathroomLight 1 Figure 3. Raw Data from Sensors.

3.1. Simulated Observations The first step in generating synthetic training data is to produce the base-model from the experimental data. Continuous output sensors such as temperature are modeled using regression; the regression function is updated daily for cumulative daily mean (Figure 4). For discrete output sensors, the model is a probability distribution for events in a given period; the parameters (location, scale and shape) of the distribution are estimated using a probability plot and the probability plot correlation coefficient for the selected distributions. The distribution with the highest PPCC is selected as the most representative of our data set and its parameters recorded. The dataset corresponds to a family of distributions and so a shape parameter is estimated using a PPCC plot. The values for parameters indicated a reasonable fit since the probability plot is close to linear for the Weibull distribution. The parameters are listed in Table 1.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D.N. Monekosso and P. Remagnino / Synthetic Training Data Generation for ADL Modeling

141

Table 1. Estimated parameters for a Weibull distribution to model observations of kitchen sensors; corresponding to (a), (b), and (c) above. Parameter

PPCC Location Scale Shape

D1 (a)

D2 (b)

D3 (b)

0.9848 26.1057 15.8846 3

0.9936 7.6167 34.6466 4

0.9961 26.9300 19.5253 4

Figure 4. Cumulative Temperature Models.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

The diagrams in Figure 5 show the base distribution and the resulting distributions after altering distribution parameters as described above for the kitchen light events (a) under normal conditions, (b) simulating greater variability in kitchen activity, and (c) simulating longer kitchen activities.

Figure 5. Histograms - kitchen light events.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

142

D.N. Monekosso and P. Remagnino / Synthetic Training Data Generation for ADL Modeling

3.2. Simulating Behavior Profile Hidden Markov Models (HMM) are employed to generate profiles. The observed states are the events; one event/state per period. The duration of a period is selectable and can be coarse lasting 4 hours or as fine as the data processing sampling time. The event/state associated value is the distribution location parameter. The model is assessed by cross-validation; the measure is the (Log) probability of the model generating the profile. The test is repeated for an increasing number of hidden states and profile length. Figure 6 shows the results of modeling a profile; increasing the number of hidden states improves the model performance, up to 8 hidden states after which no further improvements are achieved. Model performance is poorer with increasing validation profile length as witnessed by the increasingly large and negative log (probability).

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 6. Log P(O|model) versus hidden states. 3.3. Simulating Deviations from Norm Figure 7 (a) and (b) show the PMF for a normal sensor reading and anomalous sensor observations respectively for the same sensors [16]. These suggests that deviation from norm can be produced by altering the distribution parameters.

(a)

(b)

Figure 7. Modeling Sensors – normal (a) and anomaly at 16h (b). In a similar manner, altering sequences of events can alter the behavior profile. The diagrams in Figure 8 compare profiles - a generated normal profile (blue) against generated spurious profile (red). The measure used is the probability of observing a

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

D.N. Monekosso and P. Remagnino / Synthetic Training Data Generation for ADL Modeling

143

specific sequence given the model. The increasingly negative log (probability) indicates a higher probability that change has occurred in the profile.

Figure 8. Log P(O|model) for unmodified and modified profiles.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. Discussion and Conclusions The validity of the data generator (simulator) can only be fully assessed against real experimental data. However by combining real sensor and actuator data with the physical characteristics, individual models for sensors can be assessed. From initial results, the synthetic data is sufficiently representative to allow development of learning algorithms. The profiles are more complex in nature. However we are less concerned with the accuracy of the individual datum (sensor observation) and more with the type of distribution. The profile is a time-series for which a predictive model is required. The paper describes a simulator – training data generator – to support a project which has for aim to develop a system to support independent living. The system monitors and analyses behavior to detect trends and deviations from what might be considered normal behavior of the inhabitant. Behaviour analysis is perrformed by gathering information such as lights status, motion detectors status; creating a model of daily activity. The nature of problem requires a solution based on statistcal techniques. Statistical techniques for leaning models require a large amount of training data which would take years to obtained. In order to speed up the development process, a training data generator (simulator) was built. Data representative of months and years can thus be generated and variations injected to simulate change in behavior of any type.

References [1]

[2] [3]

B. Brumitt, B. Meyers, J. Krumm, M. Hale, S. Harris, and S. Shafer, 2000, EasyLiving: Technologies for Intelligent Environments, Proc. of the 2nd international symposium on Handheld and Ubiquitous Computing, Lecture Notes In Computer Science; Vol. 1927, pp. 12-29, ISBN:3-540-41093-7 The iDorm project home page, Essex University, UK, http://iieg.essex.ac.uk/idorm.htm The iRoom project home page, Stanford, http://iwork.stanford.edu/

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

144 [4] [5] [6] [7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15] [16] [17] [18]

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[19] [20] [21] [22] [23]

[24] [25]

D.N. Monekosso and P. Remagnino / Synthetic Training Data Generation for ADL Modeling The HyperMedia studio project home page, UCLA, http://hypermedia.ucla.edu/ The MavHome project home page, 2005, University of Texas, Arlington, http://cygnus.uta.edu/mavhome/ M. E. Pollack, (2005), Intelligent technology for an aging population: The use of AI to assist elders with cognitive impairment. AI Magazine, 26(2) (2005) 9–24. E. Tapia, Munguia, S. S. Intille, and K. Larson, 2004, "Activity recognition in the home setting using simple and ubiquitous sensors," in Proceedings of PERVASIVE 2004, LNCS 3001, A. Ferscha and F. Mattern, Eds. Berlin Heidelberg: Springer-Verlag, (2004) 158-175. M. Mühlenbrock, O. Brdiczka, D. Snowdon and J.-L. Meunier, 2004, Learning to detect user activity and availability from a variety of sensor data. In Proceedings of the Second IEEE Conference on Pervasive Computing and Communications, Orlando, FL, March. O. Brdiczka, D. Vaufreydaz, J., Maisonnasse, P. Reignier, 2006, Unsupervised Segmentation of Meeting Configurations and Activities using Speech Activity Detection.3rd IFIP Conference on Artificial Intelligence Applications and Innovations (AIAI), June 7-9, 2006, Athens, Greece , 195-203 O. Brdiczka et al, 2007, Detecting Individual Activities from Video in a Smart Home, 11th International Conference on Knowledge-Based and Intelligent Information & Engineering System KES07, Special Session on, Intelligent Environments: Algorithms, Techniques and Applications, Vietri sul Mare, Italy, September 2007 F. Rivera-Illingworth, V. Callaghan and H.A. Hagras, Neural Network Agent Based Approach to Activity Detection, 2005, in AmI Environments, IEE International Workshop, Intelligent Environments (IE05), Colchester, UK, 28-29th June 2005 F. Doctor, H.A. Hagras, and V. Callaghan., 2004, An Intelligent Fuzzy Agent Approach for Realising Ambient Intelligence in Intelligent Inhabited Environments, IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 35, No.1, pp.55-65 M. C. Mozer, (2005). Lessons from an adaptive house. In D. Cook & R. Das (Eds.), Smart environments: Technologies, protocols, and applications (pp. 273-294). Hoboken, NJ: J. Wiley & Sons S. Rao, and D. J. Cook, 2004, Predicting Inhabitant Actions Using Action and Task Models with Application to Smart Homes, International Journal of Artificial Intelligence Tools, 13(1) (2004), 81100. Hori, T., Nishida, Y. And Murakami, S., " A Pervasive Sensor System for Nursing Care Support" in 'Ambient Intelligence Techniques and Applications', Computer Science Springer Verlag, (2008) Monekosso, N., and Remagnino, P., "Ambient Intelligence for Assisted Living" in 'Ambient Intelligence Techniques and Applications', Computer Science Springer Verlag, (2008) Cesta, A. et al., "Robotic, Sensory and Problem-Solving Ingredients for the Future Home" in 'Ambient Intelligence Techniques and Applications', Computer Science Springer Verlag, (2008) Baird, H., 2000. State of the art of document image degradation modeling. In: Proc.4th IAPR Workshop on Document Analysis Systems, 1–16 (2000). Lu, X., Jain, A.K., 2003. Ber. resampling for face recognition chem. In: 4th Internat. Conf. on Audio and Video based Biometric Person Authentication, pp. 869–877 (2003) Chen, J., Chen, X.L., Gao, W., 2004. Resampling for face detection by self-adaptive genetic algorithm. In: Proc. Internat. Conf. on Pattern Recognition, 822–825 (2004) Jiang, F., Gao, W., Yao, H., Zhao, D., and Chen, X. 2009. Synthetic data generation technique in Signer-independent sign language recognition. Pattern Recogn. Lett. 30, 5 (2009) Varga, T., Bunke, H., 2003. Effects of Training Set Expansion in Handwriting Recognition Using Synthetic Data, 200–203 (2003) T. Varga and H. Bunke (2008), Perturbation models for generating synthetic training data in handwriting recognition In: Machine Learning in Document Analysis and Recognition, ed. by Marinai, S. and Fujisawa, H., pp. 333-360, Springer. Rabiner, L. R., "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition," Proceedings of the IEEE, 77 (2) (1989), 257–286 Millonig, A., and Gartner, G., Shadowing - Tracking - Interviewing: How to Explore Human SpatioTemporal Behaviour Patterns, in Proceedings of the 2nd Workshop on Behaviour Monitoring and Interpretation, BMI'08, Kaiserslautern, Germany, September 23, 2008 1-14

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-145

145

An AI-Planning based Service Composition Architecture for Ambient Intelligence Florian Marquardt and Adelinde M. Uhrmacher Albert-Einstein-Strasse 21 18059 Rostock, Germany

Abstract. Ambient Intelligence (AmI) is characterized by dynamic ensembles of devices that offer individual services to the user in an unobtrusive manner. For more advanced services often the cooperation of devices is required which can be accomplished by automatic composition of services. AI planning is one possibility to realize service composition. We introduce an architecture containing a service composer that uses AI planning to compose services in AmI environments. The composer can be seen as an enhanced meta planner that uses service technologies. It makes use of three strategies for selecting planners and a heuristic to cancel planners. This design is based on extensive experimental evaluation in which we analyzed the behavior of planners. The overall goal of the design is to balance between runtime of planning processes, used resources, and erroneously not returned planning results.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Keywords. Service Composition, AI planning, Ambient Intelligence

Introduction Ambient Intelligence (AmI) aims at assisting users in environments which are enriched with smart devices. The cooperation of these devices relies on goal oriented service composition to provide higher level services. This assistance should be provided ad hoc and ubiquitous. AmI environments pose some general requirements to service composition. They are dynamic, heterogeneous, and should be able to react to human behavior without explicit interaction of the user. In addition, the reaction should happen as transparent as possible for the user. E.g. a user should neither initialize nor notice that a service composition is executed. Reacting to the user in such an unobtrusive manner requires fast and reliable composition strategies. Already today AmI environments offer many basic services for composition [1,2,3], and the amount of services is likely to increase [4]. Therefore, powerful composition techniques and architectures are needed. A variety of methods is used for composing services in AmI environments [5]. However, a common composi-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

146

F. Marquardt and A.M. Uhrmacher / An AI-Planning Based Service Composition Architecture

tion architecture is still an open challenge [6]. In contrast, in the semantic web service community, where most composition approaches stem from, architectural proposals exist. Here, the task of composing services is often considered as an AI planning problem, e.g. [7,8]. This solution to the problem of service composition is seldom found in AmI environments. A point of scepticism is the dynamic of AmI environments and the fact that AI planning was initially developed for static environments. Classical planning is restricted to off-line planning [9], hence can not react to changes in the environment during planning time. However, if planning processes are sufficiently fast, off-line planning should pose no problem. For this reason the runtime performance and predictability of the planning process is a vital aspect of service composition in AmI environments. Unfortunately the runtime behavior of AI planners is irregular [10,11]. In previous research we investigated this behavior with respect to the requirements of AmI environments [12,13]. Exploiting these empirical results we propose a planning component embedded in a general architecture of an AmI environment that is able to reduce the computational needs of compositions significantly and furthermore helps in making composition runtimes more predictable. The remainder of the paper is structured as follows. We will first review literature concerning current architectures for service composition in ambient intelligence environments. Afterward the special runtime behaviors of AI planners in AmI environments will be investigated. An architecture will be developed, which realizes three possibly strategies to incorporate planners and uses a specific threshold strategy to cope with the runtime behavior of AI planners.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Related Work Amigoni et al. [6] characterize the planning problem within AmI. They propose a composition architecture that combines centralized and distributed features based on HTN planning. Each device is annotated with decomposition methods and thus with knowledge about this device and its possible role in different compositions. Based on the available devices and knowledge, the composition will take place as a planning process centrally. However, the most severe drawback of HTN-based composition is the need for additional information in terms of decomposition methods. Already the definition of preconditions and effects for each service a device might offer, which is required for classical AI planning, implies additional effort, as this kind of description is not included in ordinary service descriptions of devices. Even worse, if a variety of additional decomposition methods has to be defined for all foreseeable circumstances. In [8] a framework called Haley for hierarchical web service composition based on first order semi-Markov decision processes is introduced. Haley is able to handle uncertainties in the planning process. This approach is able to react to devices that fail or services returning failures. As their work is concerned with the composition of web services, Zhao et al. assume planning run times and service response times that are significant longer than corresponding times in AmI environments. In AmI environments, a replanning might disarm the problem of handling failures and other uncertainties. As Amigoni et al. do, Zhao et al. utilize additional structural

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

F. Marquardt and A.M. Uhrmacher / An AI-Planning Based Service Composition Architecture

147

knowledge about hierarchical service dependencies which is comparable to the methods of HTN planning. One aspect of our approach is based on the combination of different planners. Howe et al. [14] introduced a meta planner called BUS. It combines six different planners. In beforehand, all of these planners were evaluated and ranked. Probabilities for success and expected runtime were derived. Based on this an ordered list of planners annotated with allowable execution time is created. In case a planner is not able to solve a given problem within the defined time span, the next planner in row is started. Even though BUS was able to solve more problems than a single of the tested planners could, the ranking was based on a specific test suite and thus, its generalization and application in AmI environments is questionable. Furthermore, it is not intended that planners join or leave the meta planner. Roberts et al. [15] provide an improved ranking.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. Runtime Behavior of AI Planners Making precise predictions of planning runtimes is not feasible as it is not even possible to make clear and deductive assertions about the complexity of a given planning problem [11]. In case more precise assertions are needed only an experimental approach is practicable. In [13] we investigated the behavior of different planners in more detail. We conducted several millions of experiment runs to get an impression on how planners react to changes in the AmI environment. Based on these experiments it can be stated that the runtime of recent planner implementations is suitable for service composition in current AmI environments. For the generated AmI environments that reflects the complexity of currently installed environments we can find planners whose average runtimes are in the scale of ms. Assuming that state changes occur less often, a composer based on these planners can cope with an AmI environment. In rare cases, i.e. the proximity of phase shifts [11], planners tend to get stuck in runtime peaks, which are characterized by runtimes that are several times above the average in the same problem region. These peaks hamper the aspired imperceptibility of ambient intelligence. In addition, the longer the time that is needed for execution, the higher the probability of a state change and with this the invalidity of the produced plan will be. Thus, an architecture that employs AI planning for service composition in AmI environments has to take this behavior into account. The results of Howe et al. [14] suggest that in more complex environments different planners run into different peaks due to increasing specialization of the planners. Thus, a dynamic switching of planners at runtime might improve the overall performance. Current implementations of planners possess internal timeouts or limits to confine the search space and thus the runtime. To this end it is possible that planning results are false negatives, i.e. although a planner returned no plan, a solution exist and an other planner may be able to find it. Although this event is very rare, it should be taken into account.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

148

F. Marquardt and A.M. Uhrmacher / An AI-Planning Based Service Composition Architecture

3. Architecture with Smart Composer AI planning requires three inputs, the current state of the world, the available operators, and the goal description. In AmI environments the current state of the world is given by the information from sensors and other information sources, which we subsume as context. The available operators are given by the descriptions of services. Goal descriptions are provided by an intention analysis component that is able to infer intentions of the user [16]. AmI Environment

AI Planning

Context

Worldstate S0

Services

Operators O

Intentions

Goals G

Compositions

Plans P

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 1. Mapping Service compostion in AmI environments to AI planning.

Planning for service composition is performed based on states. The states of each domain are expressed with literals, which reflect aspects that can be accessed and/or changed by the services. Examples can be as diverse as ”light on“, ”have document A“, or ”canvas down“. The according mapping between AmI environments and AI planning is illustrated in Figure 1. The proposed architecture is designed to use several planners for composition. In contrast to the meta planner BUS introduced in [14], we do not assume a fixed set of planners. We allow that different planners may be shipped with according devices or brought into the environment otherwise. Thus, the set of available planners in an environment may change during runtime. We make use of a service oriented architecture pattern to allow planners to join and leave the system. To handle this dynamics we introduce a central component called composer which manages the dynamic set of planners as well as the planning process. Based on the mapping in Figure 1 and the mentioned components of AmI environments we developed an architecture as shown in figure 2. It consists of the central composer, an intention analysis, several planners, and a middleware. The middleware provides access to the services offered by the devices offer, and to the sensor data. The composer forms the core of the architecture. It generates domain and problem descriptions using the intentions of the user, the current state of the world, and the service descriptions. It furthermore includes a list of currently available planners and incorporates a strategy that chooses the best suitable planner, starts it, collects the planning results, and cancels planners occassionaly.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

F. Marquardt and A.M. Uhrmacher / An AI-Planning Based Service Composition Architecture

149

Figure 2. Concept of the composer

Although all planners claim to be sound, in our experiments we found that plans returned by planners can be wrong in rare cases. Hence, after receiving a planning result the composer is responsible for verifying whether the returned plan is complete and consistent. The runtime complexity of this procedure is linear in the length of the resulting plan and, thus, negligible compared to the antecedent planning run. According to the functionality of the composer two challenges arise. First a general interface to the planners is needed, second a strategy how to incorporate the planners have to be determined.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.1. Web Service Wrapper for AI Planners To be able to use different and possibly unknown planners, we have to implement them as services.1 For this, we need a language for data exchange and a common service interface. Due to harmonization efforts of the planning community the Planning Domain Description Language (PDDL) [17] as common language for planning domains and problems is widely accepted and the de-facto standard in this area. Service description languages that are able to encode AI planning domains and problems such as OWL-S or WSMO are still premature [18]. Therefore, relying on PDDL appears as a good choice for now. Hence, a mapping between the service descriptions at hand (e.g. in WSDL) and PDDL is needed. We used the Java-PDDL library from Zeyn Saigol2 as basis for our PDDL-parser, that encoded the generated domains and problems. As mentioned before it is a vital point that the service description contains its preconditions and effects, which we assume to be defined in PDDL. For the definition of the interface WSDL is used. Each plan wrapper implements the following three functions: findPlan(Domain d, Problem p), stop() and 1 Fortunately

AI planning algorithms are available as stand alone implementations that are comparably small. Their integration and execution on even small devices as services is therefore reasonable. 2 http://www.cs.bham.ac.uk/˜zas/software/graphplanner.html Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

150

F. Marquardt and A.M. Uhrmacher / An AI-Planning Based Service Composition Architecture

sendResult(Result r). The planning process is started by findPlan. The planner service calls back the requesting composer with the result of the planning process. The service cancels the actual planning process if the function stop is called. Therefore, the planner wrappers are implemented multi threaded. Due to the dynamic nature of AmI environments, a document based communication style between the components of the architecture has been implemented. This enables an asynchronous communication. It has been realized by using Apache Axis2 as web service API.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.2. Strategies for choosing a planner Three different strategies for choosing a planner are considered: Single, Chain and Race. Among them the latter is the most trivial. Race causes all available planners to start simultaneously. As soon as the first planner returns a result, all other planners are stopped by the composer. This approach guarantees fastest results at the cost of computational resources. This approach works well if sufficient computation power is available. There is no need for a priori knowledge about the planners. The second approach, Chain, is comparable to the approach described in [14]. It requires a ranking of the planners. According to this ranking each planner is given a time slot to solve the given problem. When time is up the next planner in line is given a try. The results of Howe et al. indicates that in more complex environments specific rankings may be possible and general outperformers become rare. In comparison to the first, a significant amount of runtime may be saved with this strategy. However detailed a priori knowledge about the planners is required to provide the ranking. In addition to be efficient, planners are needed that behave differently in the phase shift regions. As we identified the planner blackbox [19] to be an outperformer in most of our experiments, the Single approach, i.e. to select one and the same planner, appears as a suitable strategy because vast computational power cannot be assumed in most AmI environments (and this is a requirement for the Race strategy). All of the approaches suffer from the problem of false negative results. One possibility to address this problem to compare multiple planning results, which implies a higher computation effort. Whereas in the Chain approach each planner is stopped after a certain time, in the Single strategy and in many cases also the Race strategy3 runtime peaks might threaten the unobtrusiveness of the service composition. Thus, thresholds are introduced, which determine when a planning run will be cancelled latest. Every abort of a planning run saves runtime, but includes the risk of loosing a valid solution. To get an idea of the loss of solutions and the saved runtime we evaluated our experiments [13] against different threshold levels. The results are shown in Figure 3. It is evident from Figure 3 that the amount of saved runtime correlates to the threshold level. Already for low thresholds (e.g. 100ms) the quota of found solutions increases to a reasonable amount of about 98%. At this point more than 45% of runtime is saved. Hence canceling planning runs saves a lot of runtime while the amount of lost solutions is reasonable. Currently developed heuristics will 3 In

our experiments planners shared most of the runtime peaks.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

F. Marquardt and A.M. Uhrmacher / An AI-Planning Based Service Composition Architecture

151

Figure 3. Quota of found solutions and saved runtime depending on the threshold level (in ms)

take into account physical constraints of the devices where planners are deployed, e.g. a switch between planners might be induced due to battery limitations in the Single strategy and the runtime behavior of the planners also depends on the CPU power and the current load.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. Conclusion Based on a mapping between components of AmI environments and AI planning we described an architecture for service composition in AmI environments. The heart of this architecture is a composer, which acts as a meta planner on a dynamic set of loosely coupled planners. Based on the special behavior of AI planning algorithms we introduced three different strategies to incorporate the planners and outlined their respective field of application. We evaluated the use of thresholds to restrict planning runs and embedded this functionality in our composer, too. Using a threshold approach enables to guarantee upper limits for response times. The architecture and the strategies are based on extensive experiments where about 55000 problems were explored in several millions of experiments. These experiments allowed us to deduce certain knowledge about the behavior of planning algorithms in AmI environments and induced errors due to premature cancelling of planning processes [13]. The architecture has been implemented and is currently tested in a smart environment, i.e. a smart meeting room [16]. Acknowledgment This research is supported by the German Research Foundation (DFG) within the context of the project MuSAMA (Multimodal Smart Appliance Ensembles for Mobile Applications). Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

152

F. Marquardt and A.M. Uhrmacher / An AI-Planning Based Service Composition Architecture

References [1]

[2]

[3]

[4]

[5]

[6]

[7] [8]

[9] [10] [11]

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[12]

[13]

[14]

[15]

[16] [17]

[18]

[19]

Remi Dupuis, Jerome Pierson, and Mathieu Vallee. Flexible & dynamic device usage based on software agents and semantic web technologies in an ambient home environment. Demonstration at Ubicomp2007, 2007. Barry Brumitt, Brian Meyers, John Krumm, Amanda Kern, and Steven Shafer. Easyliving: Technologies for intelligent environments. In Peter J. Thomas and Hans-Werner Gellersen, editors, Handheld and Ubiquitous Computing, volume 1927 of LNCS, pages 12–29. Springer-Verlag London, UK, September 2000. Sheetal Agarwal, Anupam Joshi, Tim Finin, Yelena Yesha, and Tim Ganous. A pervasive computing system for the operating room of the future. Mob. Netw. Appl., 12(2-3):215– 228, 2007. Michael P. Papazoglou, Paolo Traverso, Schahram Dustdar, Frank Leymann, and Bernd J. Kr¨ amer. Service-oriented computing: A research roadmap. In Francisco Cubera, Bernd J. Kr¨ amer, and Michael P. Papazoglou, editors, Service Oriented Computing (SOC), number 05462 in Dagstuhl Seminar Proceedings. Internationales Begegnungs- und Forschungszentrum fuer Informatik (IBFI), Schloss Dagstuhl, Germany, 2006. Jeppe Bronsted, Klaus Marius Hansen, and Mads Ingstrup. A survey of service composition mechanisms in ubiquitous computing. In Proc UbiComp RSPSI, pages 87–92, 2007. F. Amigoni, N. Gatti, C. Pinciroli, and M. Roveri. What planner for ambient intelligence applications? Systems, Man and Cybernetics, Part A, IEEE Transactions on, 35(1):7–21, 2005. Mark Carman, Luciano Serafini, and Paolo Traverso. Web service composition as planning. In Workshop on Planning for Web Services, Trento, Italy, June 2003. Haibo Zhao and P. Doshi. Haley: A hierarchical framework for logical composition of web services. In IEEE International Conference on Web Services, ICWS 2007, pages 312–319, 2007. Dana S. Nau. Current trends in automated planning. AI Magazine, 28(4):43–58, 2007. Adele E. Howe and Eric Dahlman. A critical assessment of benchmark comparison in planning. Journal of Artificial Intelligence Research, 17:1–33, 2002. Jussi Rintanen. Phase transitions in classical planning: an experimental study. In In Principles of Knowledge Representation and Reasoning, pages 710–719. AAAI Press, 2004. Florian Marquardt and Adelinde Uhrmacher. Evaluating AI Planning for Service Composition in Smart Environments. In Proc. MuM, pages 48–55, Umea, Sweden, December 2008. ACM. Florian Marquardt and Adelinde M. Uhrmacher. Evaluating the behavior of ai planners for service composition in smart environments. Submitted to ICAPS 2009 Conference, April 2009. Adele E. Howe, Eric Dahlman, Christopher Hansen, Michael Scheetz, and Anneliese Von Mayrhauser. Exploiting competitive planner performance. In Proceedings of the Fifth European Conference on Planning, pages 62–72. Springer, 1999. Mark Roberts and Adele Howe. Learned models of performance for many planners. In Proc. of Workshop on Artificial Intelligence Planning and Learning, Providence, Rhode Island, USA, September 2007. Thomas Heider and Thomas Kirste. Supporting goal based interaction with dynamic intelligent environments. In ECAI, pages 596–600, 2002. Malik Ghallab, Adele Howe, Craig Knoblock, Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. PDDL - the planning domain definition language, July 1998. Christiane Reisse, Christoph Burghardt, Florian Marquardt, Thomas Kirste, and Adelinde M. Uhrmacher. Smart environments meet the semantic web. In Proc. MuM, pages 88–91, Umea, Sweden, December 2008. ACM. Henry Kautz and Bart Selman. Blackbox: A new approach to the application of theorem proving to problem solving. pages 58–60, 1998.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-153

153

Adaptive Neuro-Fuzzy Systems for Context Aware Offices Quintero M., Christian G. a,1 and Eljaik G., Jorhabib b,2 Department of Electrical and Electronics Engineering, Universidad del Norte, Colombia

a, b

Abstract. - Our proposal is aimed at achieving a reliable control architecture for intelligent environments. This approach is implemented in a professor’s simulated office. To that end, adaptive Neuro-Fuzzy training of Sugeno-type fuzzy inference systems are used. Experimental results and conclusions are shown, stressing the relevance of this approach inspired by the ambient intelligence metaphor in the adaptation of the environment to the professor’s physical and emotional conditions. Keywords. Ambient Intelligence, Computational Intelligence, Neuro-Fuzzy Systems, Control Architecture

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction Nowadays, high quality universities around the world perform great efforts to provide a suitable physical infrastructure for their staff. For professors, this infrastructure could be represented by means of their workspace, their offices. Therefore, a suitable space is a key factor to obtain good conditions for intellectual productivity and performance of each one. In this sense, this paper presents an architecture inspired by the Ambient Intelligence (AmI) metaphor to control electronics devices typically installed in the professor’s workspace and adapt this environment to the current user’s physical and emotional conditions. To that end, a simulated “intelligent office” is used. Current approaches mainly take into account the hardware implementation of this matter, focusing on the production of electronic devices capable of interacting with the user in a natural and non-aggressive way, as well as sensors to provide emotional and ambient measures, as it can be seen in [9][10][11]. However, an approach based on the intelligent architecture behind these technological immersed environments, has not yet been completely carried out, and is object of the current work.

1

Christian G. Quintero M.: Assistant Professor, Department of Electrical and Electronics Engineering, Universidad del Norte, Km 5 Via Puerto Colombia, Barranquilla, Colombia; E-mail: [email protected]. 2 Jorhabib Eljaik G.: Electronics Engineer, Department of Electrical and Electronics Engineering, Universidad del Norte, Km 5 Via Puerto Colombia, Barranquilla, Colombia; E-mail: [email protected]. Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

154

C.G. Quintero M. and J. Eljaik G. / Adaptive Neuro-Fuzzy Systems for Context Aware Offices

1. Related Work Home-Lab [1] is at first sight a regular house, but it is not. It has been provided with all kind of electronic devices (speech and movement recognizers, digital displays in restrooms mirrors, etc.). This place is the tests platform used by Phillips to develop and test all prototypes of electronic devices that can be considered ambientally intelligents, putting the user in the center of its functionality. Such prototypes can act on their own and without direction, anticipating habitants’ needs [1]. Another advance in our competent field, is the iDorm [2], which is a project consisting of a room, provided with a great amount of sensors and actuators. Such electronic devices give the room the capability to adapt the environment to user’s needs, improving therefore his quality of life. Parameters such as, temperature, humidity and light intensity are controlled by the system. Learning techniques based on fuzzy logic [3][4][5] and artificial neural networks have been used, to determine user’s mood with the goal of verifying and demonstrate its feasibility a utility, with very positive results. Project AMIGO (Ambient Intelligence To Go) has been also on this way, framed in AmI metaphor, developing applications with the objective of providing to the enduser services that allow him to communicate and socialize with people in other locations [6].

2. Proposed Approach

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2.1. User Model An intelligent environment must take into account user’s needs, preferences and/or habits [7]. To obtain this information, questions were made to the target user to be modeled, regarding the variables that have the strongest influence on his behavior, mood, physical and emotional states. Therefore, for this particular user, temperature desired by the user depends on his physical state, stress level and time. Exterior light intensity has a bigger influence on the preference for light intensity inside the office. Preferences on music genre, smell and type of information depend on time, mood and stress level. Finally, stress level is measured by the number of tasks that the user has along the day. Based on previous information, four models of the user have been used, as described in Table 1. Table 1. Models for user. Model 1 2 3 4

Physical State Hot Hot Cold Cold

Mood Happy Sad Sad Happy

2.2. Office model The model of the office was made, based on electronic devices susceptible to be controlled, such as: Blinds and light dimmer, air conditioner, electronic air-freshener,

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

C.G. Quintero M. and J. Eljaik G. / Adaptive Neuro-Fuzzy Systems for Context Aware Offices

155

stereo, electronic agenda and television. Each one of these devices has associated a variable to control, which are respectively: light intensity, office temperature, smell, music genre and information. Table 2 summarizes this information containing corresponding levels for each variable as well. Table 2. Office model. Electronic Device Blinds Light dimmer

Controlled variable

Levels

Light intensity

0 – 100%

Stereo

Music Genre

Air conditioner

Temperature

Electronic Air-freshener

Smell

Electronic Agenda

Tasks

Television

Type of information

Rock Balad Salsa Vallenato 10 – 25ºC Orange Lemon Cherry Apple News Weather Sports Academic

There are also external variables to the office, also taken into account for the design of the architecture. These are: external light intensity, temperature and noise. These variables have been modeled through time-varying noisy Gaussian functions. Time has been also included, and ranges from 6am to 6pm, due to the regular working schedule of a professor.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2.3. Control Architecture The architecture is willing to control six variables: Dimmer light intensity, opening percentage of blinds, smell, music genre, type of information and temperature. In order to achieve this, six Adaptive Neuro-Fuzzy Inference Systems (ANFIS) were used to control each variable. Since these need to be specific for each user model, there is a total of 24 ANFIS. ANFIS work in a similar way to Takagi-Sugeno Fuzzy Inference Systems (FIS), with the additional capability to be retrained, using either Backpropagation gradient descent or a hybrid technique between backpropagation and minimum least squares [8]. Figure 1 shows the proposed control architecture, showing the direct interaction among office and user’s model with the Sugeno Fuzzy Inference Systems. After making Gaussian membership functions for each input variable to the ANFIS and defining the corresponding descriptors, membership functions for output variables are also set, depending on every model and combinations of these. For the latest, an example can be seen in Table 3.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

156

C.G. Quintero M. and J. Eljaik G. / Adaptive Neuro-Fuzzy Systems for Context Aware Offices

Figure 1. Proposed control architecture.

Table 3. Example of membership functions for output variables.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Membership functions descriptors

Value

Information

Music Genre

Smell

Models 1, 2, 3, 4

News Weather Sports Academic

Rock Balad Salsa Vallenato

Orange Lemon Cherry Apple

0,25 0,5 0,75 1

The following step in the design of the intelligent control architecture consisted of setting the rules that governed each ANFIS. Such rules are based on a subjective interpretation of the user models obtained, as of the levels previously set for the current electronic devices to control. General syntax of these rules is as follows: If (in_var1) and (in_var2) and … and (in_varN) then (out_var) Where N is the number of input variables to a specific FIS; in_var is an input variable and out_var an output one. The application of these rules to every FIS yields fuzzy output surfaces, which as a whole will represent the preferred state of the office given certain input conditions. Figure 2 shows the output surfaces for opening percentage of blinds.

(a)

(b)

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

C.G. Quintero M. and J. Eljaik G. / Adaptive Neuro-Fuzzy Systems for Context Aware Offices

(c)

157

(d)

Figure 2. Fuzzy output surfaces for opening percentage of blinds. (a) Output surface for inputs: Number of tasks and time, and (b) Output surface for inputs: External light intensity and external noise, for models 1 and 4; (c) Output surface for inputs: Number of tasks and time, and (d) Output surface for inputs: External light intensity and external noise, for models 2 and 3.

3. Implementation The implementation of the formerly described architecture was achieved using the ANFIS Editor in The Mathworks Fuzzy Logic ToolboxTM. To visualize the outputs of the implemented architecture in a suitable and dynamic way, a GUI was made as it can be seen in Figure 3. Electronic devices are put on the stage, and labels used to indicate current state of the different variables. User can also indicate a specific input to the system, and this is intended to represent in a graphical way, the outcomes of the architecture behind. The error measure has been set to the square sum of differences between the system and expected answers after training, based on datasets used. The ideal error will be therefore zero.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. Experimental Results First test to be performed on the system was a Functional Test. Through this, it can be verified that the system is actually responding to time-varying inputs. The simulator (Figure 3) has served for this purpose. In this manner, examples of inputs used are presented in Figure 4, as well as dynamic response is also shown in Figure 5. System is therefore being altered due to the variations in the inputs and operation of the architecture is verified. Every Fuzzy Inference System is tested using two different learning techniques: Hybrid and Backpropagation. Results are analyzed based on the differences between training data and FIS outputs, before and after training. Changes in membership functions are also put on the spot, to find out possible advantages in the use of any particular technique. Table 4 summarizes training parameters information for SMELL FIS.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

158

C.G. Quintero M. and J. Eljaik G. / Adaptive Neuro-Fuzzy Systems for Context Aware Offices

(a)

(b)

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Figure 3. Professor’s simulated office.

Figure 4. Time-varying inputs for one day simulation, ranging from 6am to 6pm. (a) Model Vs. Time (b) Tasks Vs. Time (a)

Figure 5. Response of the system to time-varying input variables.

Table 4. Training parameters for SMELL FIS using Model 1 and Hybrid learning technique. Trained FIS: SMELL

Model: 1 Learning technique used: Hybrid

Number of epochs: 50 Training data pairs: 99

Error tolerance: 0

Figure 6 shows the FIS outputs to different scenarios put in contrast with training data. Horizontal axis stands for number of trials and vertical axis to corresponding

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

C.G. Quintero M. and J. Eljaik G. / Adaptive Neuro-Fuzzy Systems for Context Aware Offices

159

outputs. A perfect match between ideal outputs and those from the FIS before 80 trials is remarkable. Subsequent mismatch is evidenced due to a change in user’s preferences. Hence, a retraining of the system is required.

Figure 6. Training data and outputs before training.

Figure 7. Training data and outputs after training.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Training of the system is carried out using a hybrid learning technique and parameters formerly presented (see Table 4). The results of this training can be perceived in Figure 7. The preceding mismatch between training data and actual outputs of the system, is now diminished at the expense of less certainty in the overall system. As expected, changes on membership functions occurred trying to adapt the system to the changes in user’s behavior. Likewise, subtle changes in the output surface were evident as it can be seen in Figure 8 in contrast with Figure 9 (look at marked zone) in the region comprised by the morning time and few tasks, given the new trend in user’s behavior.

Figure 8. Fuzzy output surface for inputs Time and Tasks before training using Hybrid technique.

Figure 9. Fuzzy output surface for inputs Time and Tasks after training using Hybrid technique.

The same Fuzzy Inference System was later trained with a Backpropagation learning technique with the same parameters shown in Table 4. Results of training with this techinque are shown in Figure 10, evidencing significant differences between this kind of training and using Hybrid technique (See Figure 7).

Figure 10. Training data and post-training outputs.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

160

C.G. Quintero M. and J. Eljaik G. / Adaptive Neuro-Fuzzy Systems for Context Aware Offices

However changes in membership function are not significant. Moreover, output surfaces did not experiment relevant changes.

5. Conclusions The use of a Sugeno-type Fuzzy Inference System turned out to be an acceptable approximation to ANFIS, not being of course, the only and/or more suitable way to do this approach. User modelling is a radical task that became an outstanding piece in the learning process of the inference systems. Achieving an appropriate representativeness of the input variables makes part of this process. It is noteworthy the fact that this adaptive architecture is focused at adapting to behaviour trends instead of isolated events. Thus, a particular change on user’s behaviour as a singular event on a regular day will not necessarily require a retraining of the system. But if this singular event shows up as a consequence of a noted behaviour trend, then it becomes relevant, and a retraining of the system, to adjust its response needs to be done.

References:

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[1] Ruyter

B., 365 days’ Ambient Intelligence research in HomeLab. Available in: http://www.research.philips.com/technologies/misc/homelab/downloads/homelab_365.pdf [2] Pounds, A., Holmes, A., The iDorm – a Practical Deployment of Grid Technology. Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGRID’02). [3] H. Hagras, V. Callaghan, G. Clarke, M. Colley, A. Pounds-Cornish, A. Holmes, H. Duman, Incremental Synchronous Learning for Embedded-Agents Operating in Ubiquitous Computing Environments, The International Series "Frontiers in Artificial Intelligence and Application" IOS Press, pp.25-54. [4] Hagras H, Colley M, Callaghan V, Clarke G, Duman H, Holmes A, A Fuzzy Incremental Synchronous Learning Technique for Embedded-Agents Learning and Control in Intelligent Inhabited Environments, IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). [5] Doctor, F., Hagras, H.A.K., Callaghan, V., Lopez, A., An Adaptive Fuzzy Learning Mechanism for Intelligent Agents in Ubiquitous Computing Environments, Proceedings of the 2004 World Automation Conference, Seville, Spain. [6] Georgantas, N., Mokhtar, S., Bromgerg, Y., Issamy V., Kalaoja, J., Kantarovitch, J., Gérodolle, A., Mevissen, R., The Amigo Service Architecture for the Open Networked Home Environment. Proceedings of the 5th Working IEEE/IFIP Conference on Software Architecture (WICSA’05). [7] Ahora J., Ambient Intelligence: Plenty of Challenges by 2010. Advances in Database Technology, Springer Berlin/Heidelberg, 2002. [8] Jang, R., Shing, J., ANFIS: Adaptive-Network-Based Fuzzy Inference System, Jyh-Shing Roger Jang. Department of Electrical Engineering and Computer Science University of California. [9] Norbert A. Streitz, Carsten R?cker, Thorsten Prante, Daniel van Alphen, Richard Stenzel, Carsten Magerkurth, Designing Smart Artifacts for Smart Environments, Computer, vol. 38, no. 3, pp. 41-49, Mar. 2005, doi:10.1109/MC.2005.92 [10] Stanford V, Garofolo J, Galibert O, Michel M, Laprun C, The (nist) Smart Space and Meeting Room Projects: Signals, Acquisition, Annotation, and Metrics. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03), Vol. 4, pp 6–10 [11] D. Moore, The IDIAP smart meeting room, IDIAP-COM 07, IDIAP, 2002.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-161

161

Collaborating context reasoners as basis for affordable AAL Systems1 Jan Kresser a , Michael Klein a , Sebastian Rollwage a , and Peter Wolf b a CAS Software AG, Karlsruhe, Germany b Forschungszentrum Informatik, Karlsruhe, Germany Abstract. One of the goals of Ambient Assisted Living solutions is to enable elderly people to live in their established environment longer before being in need of care. To be able to support daily life sufficiently and do context reasoning, such systems need extensive amounts of context information, of which most is not immediately detectable by the non-invasive sensors commonly used in this area. Additionally, to guarantee the usability of those systems, they cannot have a static layout but have to be adaptive to the personal needs of the assisted person (AP). Hence, common Artificial Intelligence(AI) models relying on training such as Bayesian Networks, Neural Networks or Hidden Markov Models cannot easily be applied to such systems. The approach presented in this paper was developed for the SOPRANO project and solves the problem of reasoning about complex problems by distributing the AI to independent components and by using a blackboard and a light-weight ontology for communication between these components. Keywords. Ambient Intelligence, Dynamic System, Context Reasoning

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Introduction The EU-funded project SOPRANO aims at enabling older Europeans to lead a more independent life in their well-known environment, to integrate people with functional impairments into social life and to help them keep their autonomy. To do so, SOPRANO will rely on smart ICT-based technologies and will develop the next generation of a platform for telecare systems. Special about SOPRANO is its stringent market-orientation: the resulting system shall be affordable for the mass of people. This is achieved by using standard off-the-shelf sensors on the one hand and a flexible system architecture on the other hand that allows to adapt the system to the needs of a individual elderly person without having to reprogram or re-implement its algorithms. This combination of high scalability, market-orientation and the potential for personalisation characterises the SOPRANO approach. The technical core of the SOPRANO system is SAM, the SOPRANO Ambient Middleware, which provides the intelligence of the system. Data coming from sensors (in SOPRANO e.g. smoke detectors, fall sensors, door status sensors, device status sensors detecting electricity consumption, in-room radar systems for breathing detection and RFID readers) is stored and used as a basis for deriving new context facts. This process is commonly called context reasoning or semantic uplifting. N URMI defines it as: 1 This work was partially supported by the European Commission under the Sixth Framework Programme of IST (IST Priority 6th Call on Ambient Assisted Living - AAL) within the project SOPRANO. http://www.soprano-ip.org Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

162

J. Kresser et al. / Collaborating Context Reasoners as Basis for Affordable AAL Systems

Context reasoning is deducing new and relevant information to the use of application(s) and user(s) from the various sources of context data. However,[. . . ] context is by nature hierarchical in the sense that raw context data can be further mapped into higher level categories [14].

In practice, AAL systems are often interested in high-level context facts that cannot be easily detected by one single sensor. Examples for that are the AP’s current activity (e.g. sleeping, cooking, leaving the house) or his/her long-term mood (e.g. being bored, becoming isolated). Creating special recognisers for each of them and training them with data from the specific person is a huge effort and would lead to a non-scalable and unaffordable solution. With SOPRANO, we have developed a solution that overcomes this problem. It is based on the following ideas: (a) flexible uplifting components that do not need training, (b) the possibility of the system to work with a dynamical set of sensors in the house that might be changed to the actual needs of the AP, (c) an ontology as common vocabulary that enables uplifters and sensors to be developed independently from each other. The paper is structured as follows: in Section 2, we analyse the current state of the art and show why existing approaches of context reasoning cannot be used to build affordable systems. In Section 3 and Section 4, we show how the problem of a scalable and affordable context reasoning is solved in SOPRANO and evaluate our approach in Section 5. The paper closes with a summary and outlook to further work in Section 6.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. Related Work Currently, much research is done in the area of ambient assisted living and many European research project deal with the topic: the INHOME project [3] which focuses on the personalised management of audio-vision content, the PERSONA project [5] which will be based on a bus architecture to combine a set of intelligent devices, the AMIGO project [1] that has a service oriented architecture with service composition strategies. Further projects in this area are the SENSACTION-AAL [6], ENABLE [2], SHARE-IT [7] and the OLDES [4] project. A lot of work has been done in the area of Activity Recognition. Most of the approaches use Hidden Markov Models. S ÁNCHEZ ET AL . [17] use RFID tags on a large amount of objects to detect activities of APs. Their experiments also show the superiority of Hidden Markov Models to Neural Networks in this domain. D UONG ET AL . [9] use a special adaptation of Hierarchical Hidden Semi Markov Models to detect activities from camera recordings by exploiting the hierarchical structure of activities of daily living. Another example is described by TAPIA ET AL . [18], who use Naive Bayesian Classifiers to detect activities from the input from a large number of different ubiquitous sensors. Another task in Ambient Intelligence, especially concerning care of the elderly, is Situation Classification. Ambient Assisted Living systems designed to support elderly people in their daily lives not only support people regarding comfort, but also regarding safety. This requirement results in a need for reliable detection of critical situations, which are sometimes not detectable by a single sensor. This problem can be approached by common Artificial Intelligence Models, like Bayesian Networks, Neural Networks, or in cases with almost no uncertainty, Rule Based Systems. Research in this area has been done for quite a long time, resulting in well documented models and algorithms [8,13,16]. These models are part of the AI components of many existing systems and some SOPRANO components are based on them.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. Kresser et al. / Collaborating Context Reasoners as Basis for Affordable AAL Systems

163

The algorithms for activity recognition and most of the classification algorithms have been designed and trained to work with a static sensor layout provided by a laboratory or laboratory-like environment. The resulting models can only detect the events they have been trained on, and only in the single static environment in which the training data has been recorded. Due to the strong dependence on such a static environment, these systems are expensive, hardly adaptable and non-expandable. Unlike SOPRANO, previously developed systems have been designed disregarding economic aspects. Monolithic systems are commonly inflexible, considering the variance and amount of installed components. On the other hand, agent-based system designs do not fulfil the intelligence related requirements of complex systems. Detection of highlevel events is very hard if the input from several sensors cannot easily be combined, especially if these sensors are non-invasive and as such can only detect very “primitive” events. Furthermore, their overall behaviour can be unpredictable, which is not acceptable in some domains. SOPRANO’s demand to be a strongly market-oriented, but nonetheless reliable solution makes previous approaches of context reasoning infeasible. On the one hand, the monolithic systems’ highly customized algorithms have to be adapted to any changes of the system, and on the other hand, flexible approaches with autonomously acting components in almost all cases lack a central intelligence component ([19]). In these systems, much effort is needed to combine the information from different parts of the system to derive more complex information whenever a new component is added to the system. SOPRANO’s context model captures two layers of abstraction between sensors and services and is thereby loosely based on the work of Z HANG and G U ET AL . who proposed a layered approach to context modelling called the context stack(see [20]). But unlike Z HANG ET AL . the SOPRANO approach does not neglect imperfection and uncertainty of sensor based context data. In chapter 3, we will introduce a system design that solves the problem of integrating complex Artificial Intelligence into a dynamic, adaptable and open system.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3. SOPRANO’s blackboard-centred Approach To ensure affordability, SOPRANO has to be easily adaptable, extendable and updateable without having to put too much effort into adapting and training algorithms. To meet these requirements, a novel design principle has been developed. The SOPRANO Ambient Middleware (SAM) is not one fixed piece of software, but consisting of various collaborating software bundles. The AmI component itself is not static bundle as well. Instead, it is divided into different units, each unit solving a single problem. These units (semantical uplifters or context reasoners), just like the sensors, are technically completely independent from each other, making their use flexible and the behaviour of the system adaptable by just adding or removing them when necessary. To create this flexibility, different technologies have been combined to form the flexible system shown in Figure 1. The centre of all AI components is a blackboard (see Section 3.1), collecting and providing all available context information. Components contributing to context reasoning can either be sensors or semantical uplifters which only communicate with the blackboard. Communication is done using ontology-compliant messages (see Section 3.2). By sharing information via the blackboard, context uplifters can collaborate transparently (see Section 4). The technical basis for this approach is a service-oriented architecture, which will be explained in Section 3.3.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

164

J. Kresser et al. / Collaborating Context Reasoners as Basis for Affordable AAL Systems

Figure 1. Design Schema of the SOPRANO Context Reasoning Mechanisms

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.1. The Blackboard The Blackboard is a model of a structured approach for managing context data and for providing context information to other components of a context-aware application and was first introduced by H AYES -ROTH [10]. It has a data-centric point of view and consists of three components: knowledge sources, a blackboard and a control shell. It was originally designed to cope with complex, ill-defined problems, which can easily occur if there is no kind of conflict resolution. The conflict handling is based on work done by S CHMIDT (see [15]), who focuses on imperfection handling in “high-level" context data. The emerging system is based on a data management paradigm concentrating on efficient query processing and conflict-resolution by means of varying heuristics, extended by schema of semantic abstraction. The blackboard represents a common knowledge base that is updated by all “output-generating” components (in this scenario uplifters and sensors) of the system. The control shell has the responsibility to control the “flow of problem solving” by receiving events from the blackboard about the current actions made by a knowledge source and by calling the appropriate knowledge source. The control shell is comparable to a moderator that controls human specialists working collaboratively on a problem. In the case of context management and reasoning, the blackboard itself is a central repository which stores all context data. 3.2. Ontology as a common language As various components can be added and removed from the system at any time, there is a need for some kind of global communication interface. The most flexible way to enable components to communicate all thinkable kinds of information is to create a “common language” as a mediator between all components. In the SOPRANO case, we chose to solve this problem by introducing an ontology2 . All components which are contributing to the context reasoning process do so by inserting subject-predicate-object triplets (“statements”) according to the ontology into the blackboard (e.g. front-door-1 has-open-status open). This ensures that technically every component is capable of inserting or requesting any possible kind of information, without any dependence on where it came from or where it is used, and each other component is able to understand it. Additionally, each of these statements contains a set of meta-literals which is holding information about e.g. a confidence value, and a timestamp. All logic is implemented in 2 H EFLIN: “Ontologies are models that represent an abstraction of a domain in a formal way, such that several

parties are able to agree on the abstraction and reuse the model in their own application.” [11] Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. Kresser et al. / Collaborating Context Reasoners as Basis for Affordable AAL Systems

165

other components, and so the ontology is in fact a light-weight ontology with rare use of expressive attributes and no inherent reasoning mechanisms, just providing a vocabulary to the other components. It has different levels of abstraction to provide comfortable handling by different kinds of components with different kinds of requirements (cf. [20]). 3.3. Service-oriented Architecture Based on OSGi OSGi specifies a Java-based service platform and implements a dynamic component model. It provides an environment for the modularization of applications into smaller bundles, which can each separately be installed, uninstalled, started, stopped and updated during runtime. The gateway for communication between the bundles are services, which are each defined by an interface. All bundles can provide or request services, which are managed by the framework’s service registry. These characteristics meet the SOPRANO requirements pretty well, which is why OSGi has been chosen to be the base for the SOPRANO Ambient Middleware. Every function and every device which is part of SOPRANO or connected to the system will be represented by one or more services. This way, it can be ensured that SOPRANO will always be open for new components (software or hardware) to be integrated into the system. All sensors integrated into the system will have to be delivered by the manufacturer as OSGi bundle, which defines the sensor’s basic functionality via emitted, ontology-compliant messages, equivalent to a driver. (Additionally, a semantic service description has to be delivered for each actuator to enable SOPRANO to use it’s services. However, this will not be part of this paper).

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

4. Distributed AI – Collaborating Uplifters In the previous section it was mentioned that the AI of the system is divided into separate units. Each of these units solves one special problem and each of them on the base of ontology statements instead of sensor input. From this point of view, the Ontology does not just offer an interface for communication, but also acts as an abstraction layer, which, in combination with the mediating Blackboard, leads to the independence of the uplifters and sensors from other components (except for the blackboard) in the system. This enables the implementation of complex context reasoning mechanisms like Hidden Markov Models, which can rely on a static ontology model as basis instead of a static sensor layout. Information can be injected into the system either by sensors, if available, or by other uplifters, inferring the information from sensor input of other sensors. The location of the AP, for example, can be detected by motion detectors, by RFID scanners or by uplifters which might derive this information from other information such as electric devices being turned on or off or doors being opened or closed. Components which are interested in such a statement can register for it at the blackboard or actively query it. This way, the system is always open for new, better or just more (software or hardware) components, and existing uplifters will automatically work with the input of new components. An exemplary collaboration of several of uplifters is outlined in Figure 2. The left box on the bottom shows a semantical uplifter inferring information about the current location of the AP. The right box on the bottom shows a semantical uplifter inferring information about a certain door. The box on top represents a semantical uplifter to detect when the AP is about to leave. It is interested in openings of a front door. If the door from the bottom right uplifter is a front door, it is informed by the blackboard about the

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

166

J. Kresser et al. / Collaborating Context Reasoners as Basis for Affordable AAL Systems

Figure 2. Collaboration of Several Uplifters

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

opening, queries further information and writes a statement if it is decided that the situation is classified as Leaving. This can, of course, trigger further reactions by interested components. SOPRANO will be delivered with a set of uplifters, which in combination will be able to detect all events which are of “global interest" with more or less confidence, depending on the sensor layout. Furthermore, there are optional uplifters which just have to be activated by the administrator to adapt the system to personal needs. Weaknesses detected once the system is being evaluated by a test person can easily be straightened out by adding uplifters. This way, software developers as well as sensor developers merely need to know the OSGi specifications and the Ontology to be able to independently contribute components which can be integrated into existing systems. 4.1. Different Types of Uplifters The distribution of the AI automatically leads to another advantage concerning the complexity of the Intelligence Algorithms. As there is not one monolithic AmI-Component, but (at least) one for every problem, each can be solved using the Artificial Intelligence model that suits best. In SOPRANO, Bayesian Networks, Hidden Markov Models, Decision Trees and Boolean Rules are used. As several uplifters are built upon others, there is implicitly a hierarchy evolving, with simple uplifters at the bottom and more complex ones on the higher levels. A rough overview about what kind of uplifters suit which problem is given in Table 1. 5. Evaluation As SOPRANO’s development is not finished yet and sensors are developed separately from SAM, there have not been many possibilities to test the whole system under “real life” conditions, which is the reason why we cannot present qualitative evaluation data here. Most of the testing has been done by “simulating” the expected sensor input and the generated results seem promising. Furthermore, not all planned kinds of uplifters are

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

J. Kresser et al. / Collaborating Context Reasoners as Basis for Affordable AAL Systems

167

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Table 1. A rough overview: what kind of uplifter suits which kind of problem AI Model

Domain

Example

Boolean Rules

commonly used for events which include almost no uncertainty, such as mappings from sensors to events

Mapping from sensor to device, e.g. IF sensor-a has-status state-a AND sensor-a is-connected-to device-b THEN device-b has-status state-a

Bayesian Networks

classify situations or make decisions when uncertainty is included

the decision if the opening of the front door is indicating that someone is leaving the house or not, indicators might be the location of the AP, the time, the activity currently being performed, . . .

Decision Trees

similar to Rule Based Uplifters or as part of the Bayesian Networks, affecting their input

converting queried statements from the blackboard to input for a Bayesian Network

Hidden Markov Models

activity recognition

listening for types of devices being used and infering the activity currently being performed

Neural Networks

not applicable without training data

implemented yet, but the existing ones already produce acceptable results. Currently implemented uplifters include location detection (motion detectors, electrical consumption sensors, window sensors, RFID readers), location tracking (door sensors), detection of “leaving-situations”(based on location, door bell sensors, door sensors, activity) and activity recognition (based on location, types of devices being used, brightness) including activities like cooking, eating, watching TV, sleeping, boredom or going to the toilet. As expected, the more high-level the detected problems are, the more sensors are needed for reliable detection, and the accuracy of monolithic, customized systems can probably never be reached. As missing input will always be assumed to be “worst case” by the corresponding uplifters, the systems can reliably detect critical situations, but the more sparse the sensor layout is, the higher is the rate of wrong detections. Anyhow, this is not such a big problem as unnecessary uplifters can easily be removed from the system, avoiding false alarms, and necessary ones should be supported by an adequate set of sensors. These first results show that it is possible to customize an Ambient Assisted Living System by just adding or removing general hardware or software components which are capable of dynamically combining information. 6. Summary and Outlook SOPRANO offers a highly flexible and open solution for an Ambient Assisted Living system. The distribution of the AI and the communication via Ontology/Blackboard are a comfortable and reliable way of handling the dynamic sensor layout. Technically, the system is only limited by the Ontology, which can easily extended. This way, it is even possible to integrate new kinds of sensors into the system, even sensors which have not been thought of during the initial development of the system. Weaknesses can be resolved by simply adding another uplifter, and the existing ones will automatically collaborate

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

168

J. Kresser et al. / Collaborating Context Reasoners as Basis for Affordable AAL Systems

with it via the blackboard as long as it offers interesting information. Of course, the system’s ability to recognize certain events always depends on the amount of sensor data available, but scaling can always be done by just adding appropriate sensors. This way, SOPRANO manages to be an affordable, practical and flexible system and still has the ability to reason about complex problems. References [1] [2] [3] [4] [5] [6] [7]

[8] [9]

[10] [11]

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[12]

[13] [14] [15]

[16] [17]

[18]

[19]

[20]

AMIGO Project. Ambient intelligence for the networked home environment. STReP in the 6th Framework Programme of the European Union, 2004. http://www.amigo-project.org. ENABLE Project. A wearable system supporting services to enable elderly people to live well, independently and at ease. STReP in the 6th Framework Programme of the European Union, 2007. INHOME Project. An intelligent interactive services environment for assisted living STReP in the 6th Framework Programme of the European Union, 2007. http://www.ist-inhome.eu. OLDES Project. Older people’s e-services at home. STReP in the 6th Framework Programme of the European Union, 2007. PERSONA Project. Perceptive spaces promoting independent aging. IP in the 6th Framework Programme of the European Union, 2007. http://www.aal-persona.org. SENSACTION-AAL Project. Sensing and action to support mobility in ambient assisted living. STReP in the 6th Framework Programme of the European Union, 2007. http://www.sensaction-aal.eu. SHARE-IT Project. Supported human autonomy for recovery and enhancement of cognitive and motor abilities using information technologies. STReP in the 6th Framework Programme of the European Union, 2007. Eugene Charniak: Bayesian Networks without Tears, AI Magazine Volume 12 Number 4 (1991) (Copyright AAAI) Thi V. Duong, Hung H. Bui, Dinh Q. Phung, Svetha Venkatesh: Activity Recognition and Abnormality Detection with the Switching Hidden Semi-Markov Model, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR’05) 1063-6919/05. Hayes-Roth, B.: A blackboard architecture for control, Artificial Intelligence 26, S. 251-312, 1985. Heflin, J., R. Volz, J. Dale: Requirements for a Web Ontology Language. W3C Working Draft, March 2002. http://www.w3.org/TR/2002/WD-webont-req-20020307/. Michael Klein, Andreas Schmidt, Rolf Lauer. Ontology-Centred Design of an Ambient Middleware for Assisted Living: The Case of SOPRANO. Workshop "‘Towards Ambient Intelligence: Methods for Cooperating Ensembles in Ubiquitous Environments"’ (AIM-CU) at the 30th Annual German Conference on Artificial Intelligence (KI 2007), September 10-13, 2007, Osnabrück, Germany. David J.C. MacKay: Information Theory, Inference, and Learning Algorithms,Copyright Cambridge University Press 2003. http://www.cambridge.org/0521642981 Petteri Nurmi, Patrik Floréen: Reasoning in Context-Aware Systems, Position Paper, http://www.cs.helsinki.fi/u/ptnurmi/papers/positionpaper.pdf Schmidt, A.: Situationsbewusste Informationsdienste für das arbeitsbegleitende Lernen,(Dissertation in German), Chapter 14, Intitute for Program Structures and Data Organization, University of Karlsruhe (TH), 2009. Stuart Russel, Peter Norvig: Artificial Intelligence: A Modern Approach, Pearson Education, Inc, publishing as Prenctice Hall, 2003. Dairazalia Sánchez, Mónica Tentori, Jesús Favela: Hidden Markov Models for Activity Recognition in Ambient Intelligence Environments, Eighth Mexican International Conference on Current Trends in Computer Science, 0-7695-2899-6/07 2007 IEEE Emmanuel Munguia Tapia, Stephen S. Intille, Kent Larson: Activity Recognition in the Home Using Simple and Ubiquitous Sensors, A. Ferscha and F. Mattern (Eds.): PERVASIVE 2004, LNCS 3001, pp. 158-175, 2004. Peter Wolf, Michael Klein. SOPRANO - An extensible AAL system for elderly people based on an open platform. Proceedings of the 3rd Workshop on "‘Artificial Intelligence Techniques for Ambient Intelligence"’ (AITAmI’08). 21-22 July 2008. Patras, Greece. Daqing Zhang, Tao Gu, Xiaohang Wang:Enabling Context-aware Smart Home with Semantic Web Technologies, International Journal of Human-friendly Welfare Robotic Systems, Vol. 6, no. 4, December 2005

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-169

169

Caring of Old People: Features of Cognitive Technologies Development1 Amedeo Cesta a , Gabriella Cortellessa a and Lorenza Tiberio a,2 a ISTC-CNR, Italian National Research Council Abstract. Aging has a negative effects on cognitive abilities and can often make independent living difficult. It has been shown that cognitive training programs can significantly mitigate the effects of age-related cognitive decline. Cognitive Technologies that incorporate intelligent techniques can synthesize now devices able to enhance memory functioning of older person through cognitive exercises based on neuropsychological aspects of cognitive aging. In designing new technology is very important to maintain a strict contact with the problems of the actual users. These paper presents a brief review of two research programs both concerning the synthesis of intelligent systems for old people and underscores aspect connected to evaluation criteria related to acceptability, usefulness and other cognitive features of such technology. Keywords. cognitive aging, intelligent & cognitive technology, cognitive training, users’ acceptability, old people

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Introduction The increase in life expectancy has led to a change in the structure of industrialized countries society where elderly people represent a significant percentage. Older adults are living longer and they desire to maintain functionality and independent lifestyle. Mobility and cognitive impairments associated with aging can have negative effects on the basic and instrumental daily-life activities and, in general on quality of life of elderly people. Maintaining a physical functional performance and participating in cognitively stimulating activities can help an older person remain independently for as long as possible. In recent years, an increasing amount of research supports the protective effects of intellectual stimulation and suggests that risk for cognitive impairment decreases in subjects involved in cognitive training interventions[1]. Innovative AI-based (Artificial Intelligence) technologies can become a useful tool in eldercare for monitoring and detectioning cognitive decline. Assistive Technology for Cognition (ATC)[2] is envisage to allow the elderly to live independently in their own homes through the use of sensors, robotic devices and remote control devices supporting individuals with compromised cognitive abilities. To obtain such an ambitious goal it is very important to keep into account the requirement connected to the particular class of targeted users. Purpose of this article is to shortly present an overview of our recent and current efforts in both synthesizing cog1 Work partially supported by a project from Italian Ministry of Education and Research (R OBO C ARE ) and a grant from Microsoft Research (C OG G YM). 2 Corresponding Author: Lorenza Tiberio, ISTC-CNR, Via S. Martino della Battaglia 44, I-00185 Rome, Italy; E-mail: [email protected].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

170

A. Cesta et al. / Caring of Old People: Features of Cognitive Technologies Development

nitive technology for old population and integrating heterogeneous AI technologies to obtain innovative interactive systems. During our work we have paid particular attention to understanding features remarkable for user involvement and acceptance. Such features are the particular focus of the current analysis. The first section of this paper offers a brief explanation of common cognitive disorders related to aging and describes some psychological and technological interventions to ameliorate cognitive impairments. A second section presents a sketch of two of our projects aimed at underscoring the role of both the acceptability issues and the very features of an intelligent cognitive system for elderly people living in their home. The paper aims at providing a discussion which can be useful for the design of future cognitive technology systems.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Cognitive decline in healthy older adults: problems and solutions Researches indicates that the most common difficulties older people have with memory are connected both with the difficulty in paying attention to more than one thing at a time and with learning new information and retrieving previous information. It has generally been found that fluid intelligence (ability to think and reason abstractly and solve problems) tends to decline with advanced age, whereas crystallized intelligence (deposit of acquired knowledge) tends to remain stable and can even continue to increase throughout adulthood improving into very old age [3]. Verbal and acquired abilities, such as vocabulary, are included in the crystallized intelligence while cognitive function as attention, memory, learning, executive functioning and processing speed are included in fluid intelligence. These difficulties lead to changes in the performance of normal daily activities and also have a considerable impact on the psychological aspects. But aging is not just “to lose” and some cognitive, motor and sensory functions can be preserved, maintained and improved by an active use day by day. In this respect, a growing body of research has focused on the strategies to enhance intellectual functioning in old age. A good cognitive performance can be maintained through constant mental stimulation contributing to preventing cognitive losses, thus fostering prolonged independent life style for older people [4,5]. Memory training programs produce improvements in the person’s subjective evaluations of memory as well as in actual memory performance. In a metaanalysis of thirty-two studies, Verhaeghen et al. [6] described the benefits of cognitive training for improving memory abilities in elderly people pointing out that the effects of memory training appear to be durable over six-month periods or longer after training. Traditionally training cognitive interventions consist in simple exercises “paper and pencil” or computerized form including different tasks (e.g., reasoning, recognition, perception, speed of information processing, attention and memory). Training is particularly effective if associated to ecological tasks [7] (activities regarding every-day situations and environments): high familiarity produces a facilitation in carrying out of the task. Traditional cognitive training programs typically do not have an ecological connotation. As a consequence, a common problem in cognitive training is the lack of transfer of the strategies learned through exercise to real life. Conversely, setting cognitive stimulation in a meaningful context provides a strong motivation to transfer learned strategies to new every-day situations. Another limitation that emerges from an analysis of state of art of memory training concerns the fact that these training interventions are didactic and their primary purpose is to tech memory strategies, but there is no evidence regarding strategy maintenance. In addition, these training tend to be task-specific and not personalized with little or no transfer to task that were not incorporated in the training regimen and belonging to management of everyday life. Majority of these interventions have focused on remediating kinds of memory loss, as verbal and numeric material or face-name associations, towards a “stereotyped” regimen. Hence, the transfer of improvement of cognitive interventions to everyday problem-solving is still controversial [8].

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Cesta et al. / Caring of Old People: Features of Cognitive Technologies Development

171

2. Assistive Technology for Cognition ATCs are increasingly producing results [9] aiming at alleviating cognitive decline of elderly people . ATCs are aimed to stimulate the brain and learning of people and in particular to compensate age-related cognitive deficits through cognitive supports software or external stimulation and support of mental abilities. This kind of technology support should provide a compensatory strategy for people with cognitive impairment increasing the possibility to live independently [10,11]. Simultaneously a host of new products are arriving in stores and online to help people “exercise” their brains. Some companies that provide brain training exercises have a scientific base. For example, Cogmed [12] is an innovative and service-oriented company that develops software-based working memory training products for children and adults who need to improve their working memory, attention, impulse control and problem solving skills. The interactive individualized training system tests each user and adapts to the users’ profile to give each user a particularized training regimen. In all these products, training occurs through a set of pre-defined interactive leisure activities that may improve specific skills involved in a game. Overall, there is a little evidence that playing computer games improves other skills or reduce the risk of memory loss, so the effectiveness of these systems has not been established, and none of them are contextualized to the personal life of the user. When dealing with technological applications the acceptability by elderly users is a relevant issue. Research on how and why individuals adopt new technologies has had much attention in the last two decades. For example, Demiris et al. [13] analyzed older persons’ perceptions of home-based technology using a participatory evaluation approach with the purpose of understanding end-users’ needs and expetactions. User acceptance does not only depend on the practical benefits they can provide, but on complex relationship between the cognitive and affective aspects and expectations of old users. Usually elderly are not familiar with the use of new technologies, as a consequence, they can be fearful at the possibility of having a technological device at home, by experiencing negative feelings and general anxiety [14].

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3. Two Support Systems for Old People We now focus on authors’ direct experience within two projects aiming at producing future generation ACTs. We will stress in particular the studies related to user perception of the synthesized artifacts. 3.1. ROBO C ARE The ROBO C ARE Domestic Environment (RDE) is the results of three years multidisciplinary research and efforts, combining robotics, automated scheduling and distributed constraint reasoning and social and cognitive psychology. The project was aimed at developing AI-based technology for domestic cognitive support system for elderly people [15]. The motivation for this research derives from the need to provide alternative forms of care and assistance services through the maintenance of the elderly in their home environment. This solution is clearly preferred by older persons [16] and it has the advantage of supporting some psychological processes that play a fundamental role in maintaining an adequate level of perceived competence, self-efficacy and control on everyday life. These factors motivate AI-based technological solutions to human caregiving. The project has involved research groups with different backgrounds with the goal of investigating how state of the art AI (Artificial Intelligence), ambient intelligence, and robotics techniques can be combined to enhance the quality of life of elderly people. In this respect, ROBO C ARE has led to a prototype of intelligent system in which robots, sensors and intelligent software agents coordinate to provide support in the daily activ-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

172

A. Cesta et al. / Caring of Old People: Features of Cognitive Technologies Development

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

ities of an older person [17,18]. The key feature of ROBO C ARE Domestic Environment is the capability to acquire continuously updated data from the life context and to analyze the status of every activity being performed within the monitored space through a set of heterogeneous hardware and software agents, such as vision-based sensors and a schedule management software agent. The main interface between the environment and the person is intelligent robotic mediator able to move in the domestic environment and to interact verbally with the user. The ultimate goal of the overall system is to provide on-demand as well as proactive cognitive support in the management of an elderly person’s daily activities. An intelligent observer and a proactive assistant constitute the ROBO C ARE Domestic Environment. The interaction between the assisted person and the system is mediated by a robot with interactive capabilities. The information coming from environmental sensors is used for maintaining an updated representation of what is really happening in the environment (the “intelligent observer” modality). Through the Activity Monitor (a schedule execution monitoring system), the environment infers when and how to take initiative (the “proactive interactor” modality). Communication between the user and the robotic mediator occurs in simplified natural language. For the purposes of our current argumentation, we distinguish between two forms of interaction based on who takes the initiative to start a dialogue: User-initiative in which the user takes the initiative first; System-initiative in which the intelligent system starts interaction guided by its internal reasoning. Since the beginning, the ROBO C ARE project was dedicated to exploring the end-user perspective by studies aimed at exploring end-users’ expectations/requirements of domestic robots [19]. Afterwards, an evaluation methodology was defined and used to understand the requirements of users towards cognitive intelligent technology developed in ROBO C ARE. Our approach to user evaluation. The aim of analysis was to evaluate users’ attitudes and psychological variables affecting acceptability of our intelligent system by videobased real life interactions in which old person may be involved. For this purpose eight different scenarios representative of daily situations were developed. Specifically, the study focused on different aspects. First of all, the focal point of this study was provided a precious indication as to whether we are employing intelligent technology to solve real needs in terms of likelihood of each scenario, the usefulness and acceptability of the cognitive support system. Scenarios were arranged according to the different typologies of interactive conditions: User-initiative situations (explicit request for the system activity by the final user); System-initiative situations (the robot autonomously intervenes in the domestic environment). Moreover, user’s attitudes towards the robotic system were evaluated referring to evaluation criteria related to potential influence of psychological characteristics of individuals (levels in perceived health and worry about future cognitive losses). The evaluation. Eight short movies were showed to 100 elderly (average age of 70 years) recruited through the support of and an university for the elderly in Rome and a snowball sampling procedure. They were interviewed by a questionnaire focused on: (a) likelihood of the situations presented, the usefulness and acceptability of the system; (b )users’ attitude towards intelligent system; (c) emotional reaction to possible implementation of the system in users’ home. The answers to the questions were given on an agreement/disagreement five point Likert-scale (ranging from 0 to 4). The participants watched the movies either on a notebook monitor, in a face-to-face administration, or on a larger screen, in a small-group administration. After showing each scenario participants were invited to fill the paper referring to it. At the end of the whole presentation, subjects were asked to give general evaluations of the robot and to fill the final part of the questionnaire, referring to socio-demographic (gender, age, etc.) and psychological variables (perceived health, worry about future cognitive impairment).

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Cesta et al. / Caring of Old People: Features of Cognitive Technologies Development

173

3.1.1. Outcomes of ROBO C ARE evaluation The results of ROBO C ARE offer some remarks on contribution that Cognitive Technology may offer in assisting elderly people with everyday life activities and in promoting the independence that is so important to many older persons. Likelihood, usefulness, acceptability. When the situation depicted refers to an important activities related to health and safety the intelligent robotic system is perceived as a useful and acceptable support. These results confirm the central role of health and safety issues in elderly people’s life. In fact, with respect to activities which are considered as uncritical old people show a tendency to assign a low score both on likelihood of occurrence and on usefulness and acceptability. Outcomes are in accordance with the model of Selection, Optimization with Compensation (SOC; [20]) suggesting the strategies for a successful aging stressing the role of selection and optimization of activities with increasing age, and the importance of compensation strategies to manage the loss of personal resources.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Interactive situations: User-initiative vs. System-initiative. Results on the distinction between User-initiative/System-initiative based interactions showed the key role played by the relative importance attributed to the specific activity to be performed. Systeminitiative referring to the autonomy of the robot in the management of the home environment and in taking decisions is appreciated in those activities which are perceived of greatest relevance in elderly people’s experience. In addition, elderly users evaluate the system as a very useful resource when the activity performed concerns its capability in responding to a specific need expressed by the user, such as cognitive impairment associated with ageing. On the contrary, in the case of System-initiative based interaction the system’s ability in giving suggestion related to unimportant activities is perceived both as less useful and less acceptable. Influence of psychological aspects. The influence of better or worst health conditions perceived have an effect of the features’ assessment of the intelligent system. Elderly people complaining for worst health didn’t show some practical advantages of having a cognitive support device at home and they are more scared by the robotic system than individuals perceiving better health state. Beyond the aspects related to personal health, another psychological variable influence significantly users’ attitude toward the robot. Elderly persons showing a great apprehension for age-related cognitive impairments perceived an higher utility and acceptability of the system in terms of a cognitive resource for daily activities demanding use of memory. They valued much more physical features of the robot and its interactive behavior and communication modalities. In addition, users considered the system as a device that can make them safer at home helping in the management of the tasks of daily life. 3.2. C OG G YM A second project still on-going addressed an important connected issue: namely the possibility to enhance with an ACT the memory performance of people suffering of agerelated cognitive disorders. In this second study an important concern is the effectiveness of the training program incorporated in the ATC. A factor to be studied concerns how to transfer tasks and learned strategies to the cognitive support system. The C OG G YM project starts from the ROBO C ARE experience. Underlying is a research line that attempts to answer the question by offering cognitive help at home through an intelligent environment which fosters cognitive well-being of older users in their own homes by batteries of ecological memory exercises concerned aspects of everyday memory. Authors have also explored both the problem of a device like C OG G YM can be realized to inter-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

174

A. Cesta et al. / Caring of Old People: Features of Cognitive Technologies Development

act with elderly users’ life and the issue of system acceptability. It is worth underscoring again how users involvement is necessary in order to realize a customized, personalized and appropriated intelligent system to alleviate cognitive decline. A first prototype of C OG G YM is under development using constraint-based reasoning technology, including planning and scheduling techniques, as well as mixed initiative reasoning paradigms and automatic explanation generation. The use of this technology, even in a preliminary system version, enables the development of an underlying framework which may provide: (1) contextualized response to what the system perceives; (2) sophisticated interaction capabilities through which the system can stimulate the assisted person to exercise his/her cognitive skills. Current outcome of C OG G YM is a prototypical software framework which retains a good level of portability and generality: within the C OG G YM software architecture, the major effort was to integrate off-the-shelf components for cognitive support. The added value of orchestrating such components through the software framework is in the overall service-providing environment that is achieved as a result: the key idea which distinguishes the C OG G YM software effort is to employ recommendations drawn from current state-of-the-art in cognitive training to provide an overall set of service that foster the uptake of cognitive compensation strategies through ecological training. The development of the software infrastructure has followed an approach based on case study. Each case study development has been implementated in two phases: first, the necessary support technology was integrated into a common infrastructure; second, the services provided by the support technology were orchestrated in order to provide cognitive training programs. Some specific cognitive training schemas were selected and implemented in the framework. C OG G YM software was not focused on component technology development, rather it was employed state-of-the-art, off-the-shelf technology for acquiring the context of the environment and the of the assisted person.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.2.1. An architectural design for C OG G YM C OG G YM framework provides the user with training exercises that are contextualized to his/her domestic environment by a pervasive and non-obtrusive system that may relieve age-related decline of the elderly cognitive capabilities, while at the same time assisting users where necessary. C OG G YM aims at leveraging the results obtained in ROBO C ARE by identifying a generic infrastructure to cognitive training applications. Figure 1 shows the general sketch of the functional architecture we have realized for C OG G YM. The exercises are based on standard cognitive training knowledge (Training Knowledge Repository) and are administered to the user through an Interaction Manager after a personalization process that takes into account environmental and user models which are kept continuously updated. Role of the Problem Solver is to decide the most beneficial proactive acFigure 1.C OG G YM architecture tion (type, frequency and difficulty of exercises, etc.) according to the user’s status and on the currently detected environmental factors. The software architecture enables autonomous and continuous assessment of: (a) User Status: ecological training is ensured by a continuous monitoring and storage

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Cesta et al. / Caring of Old People: Features of Cognitive Technologies Development

175

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

of user performance data (Performance DB). Such information is filtered (Data Filtering) and reasoned upon (User Modeler) to the aim of maintaining the user model consistent with his/her assessed cognitive status; (b) Environmental Status: exercise contextualization is also guaranteed through the continuous assessment of a model of the environment (Scenario Modeler), obtained with a proper analysis (Data Filtering) of environmental data coming from a set of active Environmental Sensors (e.g., intelligent refrigerator, domotic devices, etc.). It is worth saying that current approach to the C OG G YM is to create a system which can be adapted to stimulate areas of cognition which are commonly affected by normal aging. As a starting testbed we are focusing the previously mentioned areas of cognition (prospective memory). The test case is used to drive the initial development. Usually, old people have difficulties in managing plans, retain and recollect them, and to carry them out plans, to retain and recollect them, and to carry them out at the appropriate time or in the appropriate context. This ability is based on the use of Prospective Memory (PM). In C OG G YM PM is employed as a case study in order to drive the development of framework by a testbed. In one possible scenario within the Prospective Memory area the assisted person uses a smart blackboard to maintain a list of grocery items that need to be bought at the supermarket. The blackboard is both written by the assisted person and read by the system, so the system can keep track of what the user plans to buy. Based on this information, the system can propose a cognitive exercise pertaining the list of shopping items. The traditional prospective memory training exercises consist of presenting the user with a list of items that have to be memorized and recalled at determined intervals. In the C OG G YM perspective the exercise contents are concretely related to real aspects of the user needs (i.e., the need to prepare the shopping list), their difficulty and frequency are decided on the basis of previous users performances. Contextualizing and personalizing the exercises enhances the training efficacy and increases the probability that the memorization strategies are employed in real-life situations because of their immediate practical implications. C OG G YM status. In C OG G YM we have realized a useful first software prototype. Currently we are furtherly identifying the user’s needs as well as their preferences about the cognitive technology. It will be necessary to organize a participatory design because it is important involving users in the design and development process. User involvement can: (a) help to avoid the application of technology that creates more problems that it solves; (b) promote the view that elderly are not “a problem” for which technology may contribute an answer. Objectives of potential focus group discussions about this project are different. First of all it is important to explore the extent to which people can identify and describe the nature of memory loss problem. Moreover, gathering users’ ideas for resolution, obtaining responses to this proposal and using these responses to development an ecological and personalized memory training software. 4. Conclusions Intelligent assistive technology for people cognitive abilities can be an innovative approach to cope with age-related cognitive disorders. In this paper, we described our approach to the development and testing of AI-based cognitive devices. ROBO C ARE and C OG G YM projects have directly addressed one of the open challenges in AI for eldercare: integrating diversified intelligent capabilities to create an intelligent system for everyday memory tasks. In particular after describing cognitive decline in healthy older adults and giving a short introduction to the assistive technology for cognition, we described our proto-

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

176

A. Cesta et al. / Caring of Old People: Features of Cognitive Technologies Development

types of the two intelligent cognitive system. Our research stressed the importance of employing experimental procedures involving real users and referring to everyday domestic situations in order to get helpful guidelines for future developments in assistive home technology to alleviate cognitive impairments. A long term goal of this research is to demonstrate how such technology can facilitate human coping with a successful aging by avoiding diseases and maintaining cognitive functionalities over time. Acknowledgments. Interactions with several colleagues have helped us to shape our views on the topics of this paper. We acknowledge in particular joint work with Maria Vittoria Giuliani, Federico Pecora, Riccardo Rasconi and Massimiliano Scopelliti. References [1]

[2] [3] [4] [5]

[6] [7] [8] [9] [10]

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

[11] [12] [13] [14] [15]

[16] [17]

[18] [19] [20]

Ball K., Berch D.B., Helmers K.F., Jobe J.B., Leveck M.D., Marsiske M., Morris J.N., Rebok G.W., Smith D.M., Tennstedt S.L., Unverzagt F.W., Willis S.L. (2002). Effects of cognitive training interventions with older adults. Journal of the American Association, 288, 18/2271-2281. LoPresti, E.F.; Bodine, C.; Lewis, C. (2008) Assistive technology for cognition [Understanding the Needs of Persons with Disabilities]. Engineering in Medicine and Biology Magazine, IEEE,27, 2/29-39. Horn, J. L. and Cattell, R. B. (1966). Refinement and test of the theory of fluid and crystallized intelligence. Journal of Educational Psychology, 51/253-270. Park D.C., Gutchess A.H., Meade M.L., Stine-Morrow E.A.(2007) Improving cognitive function in older adults: non traditional approaches. Journals of Gerontology Series B: Psychological Sciences and Social Science, 62:45-52. Wolinski, F.D., Unverzagt, F.W., Smith, D.M., Jones, R., Stoddard, A. and Tennstedt, S.L.. (2006). The ACTIVE cognitive training trial and health-related quality of life: Protection that lasts for 5 years. Journals of Gerontology: Medical Sciences, 61A, 1324-1329. Verhaeghen P., Marcoen A., Goossesn L. (1992). Improving memory performance in the aged through mnemonic training: a meta-analytic study. Psychology of Aging, 7:242-51. Cavallini, E., Pagnin, A., Vecchi, T., (2002). The rehabilitation of memory in old age: effects of mnemonics and metacognition in strategic training. Clinical Gerontology,26,2/125-141. Bottiroli S., Cavallini E., Vecchi T.(2008). Long-term effects of memory training in the elderly: A longitudinal study. Archives of Gerontology and Geriatrics,47,2/277-289. Pollack M. E. (2005) Intelligent Technology for an Aging Population: The Use of AI to Assist Elders with Cognitive Impairment. AI Magazine,26,2/9-24. LoPresti, E.F. Mihailidis, A., Kirsch, N. (2004). Assistive technology for cognitive rehabilitation: State of the art. Neuropsychological Rehabilitation, 14,1/2, 5- 39. Kapur, N., Glisky, E., Wilson, B. (2004). Technological memory aids for people with memory deficits. Neuropsychological Rehabilitation, 14,1/2, 41-60. http://www.cogmed.com/ Demiris G., Parker Oliver D., Dickey G., Skubic M., Rantz M. (2008) Findings from a Participatory Evaluation of a Smart Home Application for Older Adults. Technology and Health Care, 16, 111-118. Venkatesh V., Morris M. G., DavisG. B., and DavisF. D.(2003). User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly, 27/425-478. Cesta, A., and Pecora, F. (2006). Integrating Intelligent Systems for Elder Care in RoboCare. In W. C. Mann and A. Helal (Eds.), Promoting Independence for Older Persons with Disabilities, 65-73. Amsterdam: IOS Press Lawton, M. P. (1985). The elderly in context : Perspectives from environmental psychology and gerontology. Environment and Behavior. 17, 501-519. Cesta, A., Cortellessa, G., Pecora, F., Rasconi, R (2007). Supporting Interaction in the RoboCare Intelligent Assistive Environment. In Proceedings of AAAI Spring Symposium on Interaction Challenges for Intelligent Assistants. Pecora, F., Rasconi, R., Cortellessa, G., and Cesta, A. (2006). User-Oriented Problem Abstractions in Scheduling, Customization and Reuse in Scheduling Software Architectures. Innovations in Systems and Software Engineering, 2(1), 1-16. Scopelliti, M., Giuliani, M. V., Fornara, F. (2005). Robots in a domestic setting : A psychological approach. Universal Access in the Information Society. 4(2), 146-155. Freund, A. M., Baltes, P. B. (1998). Selection, optimization, and compensation as strategies of life management: Correlations with subjective indicators of successful aging. Psychology and Aging, 13, 531-543.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

Workshops Proceedings of the 5th International Conference on Intelligent Environments M. Schneider et al. (Eds.) IOS Press, 2009 © 2009 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-056-8-177

177

Support for Context-aware Monitoring in Home Healthcare Alessandra Mileo a , Davide Merico a and Roberto Bisiani a a Department of Informatics, Systems and Communication, University of Milan-Bicocca, Italy Abstract. This paper tackles the problem of supporting independent living and well-being for people that live in their homes and have no critical chronic condition. The paper assumes the presence of a monitoring system equipped with a pervasive sensor network and a nonmonotonic reasoning engine. The rich set of sensors that can be used for monitoring in home environments and their sheer number make it quite complex to provide a correct interpretation of collected data for a particular patient. For this reason, we introduce a logic-based context model and use logic programming techniques to reason about different pieces of knowledge. Keywords. Independent Living, Nonmonotonic Reasoning, Knowledge Representation, Wireless Sensor Networks

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

1. Introduction An Independent-Living System (ILS) should be able to i) gather information about the world through sensors, ii) translate sensor data to map them into a consistent assessment of the real situation, iii) reason about available knowledge to support the patient’s well being, iv) perform actions and give feedback to the patient according to the results of the reasoning process, v) capture reactions to feedback in order to adapt the behavior of the system. With this perspective, the capability of identifying meaningful information about the context in which the user lives is a critical issue for home healthcare systems. The use of a reasoning component that does not only rely on static user-specific needs, but that continuously analyzes the evolving state of the patient and of the environment, simplifies the situation assessment process. The aggregation and the interpretation of different kinds of information from heterogeneous sources (such as light, position, movement, localization, load cells) enhances reliability and accuracy of context interpretation because considering heterogeneous sources of information helps in compensating errors and incompleteness of data. In order to address these concerns, we have designed and developed a system (called SINDI) that has the following capabilities [1]: • gathering data about the user and his or her environment through a Wireless Sensor Network (WSN); • combining different data sources to interpret the evolution of the patient’s health state and predict changes into risky states according to medical knowledge and the clinical profile of the person monitored;

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

178

A. Mileo et al. / Support for Context-Aware Monitoring in Home Healthcare

• storing histories of data and making them easily available to caregivers; • identifying risky situations and providing feedback for prevention. Other important requirements like: (i) technological and medical soundness, (ii) adaptivity, (iii) unobtrusiveness, (iv) user-friendliness, (v) reactivity and (vi) affordability have also been investigated in the specification of our system, but in this paper we focus on our context representation model and reasoning about the context.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

2. The Context Model A well designed model is crucial for any context-aware system. In the literature there is a rich variety of context models discussed and proposed for different purposes [2,3]. In pervasive environments, context-dependent data can arise from different sources; for example data may be gathered by sensors or collected from several knowledge-bases. The incompleteness and heterogeneous nature of such data stress the need for expressive reasoning techniques in order to implement effective, context-dependent reasoning. While most of the implemented context-representation models are domain-dependent and do not support powerful inference, declarative logic-based models fail to provide a representation of context-dependent data that is both general and with good computational properties. The model we describe in this paper aims at being generic and computationally rich at the same time. Other requirements we take into account are simplicity, flexibility, extensibility and expressivity. To fulfill these requirements, we utilize a high level description of home environments in terms of rooms, areas, objects, properties, relations and observations. The resulting context specification is then mapped into a set of logic predicates in the Answer Set Programming (ASP) framework (see Section 3 for details). In addition to the description of the model, a reduced set of consistency constraints can be specified to make sure that observations and context interpretation are coherent. These constraints are also mapped into ASP, so that reasoning under uncertainty is possible and incompleteness of data can be taken into account. Relations and properties used in the model do not take spatial relations into account. This results in greater generality because we do not need a physical description of the environment. In addition, while data gathered by the sensors are processed and aggregated according to specific algorithms for feature analysis, the information available at upper levels is filtered by the abstraction. This enables us to represent meaningful information as properties of objects, rooms or areas, keeping the model independent from sensors’ characteristics and positioning. Our modelling approach is similar to what we would obtain by using an ontology, with the difference that the ASP reasoning enhances the expressivity and computational efficiency of the model. We are aware of the fact that research efforts are converging toward the combination of nonmonotonic reasoning and ontology-based knowledge representation, but available implementations are still domain dependent and formal issues need to be further explored. For this reason we decided to encode our contextual information directly into logic predicates, that can be easily mapped into an (existing or new) ontology if needed. Previous investigation of context models has indicated that there are certain entities in a context that, in practice, are more important than others. These are location, identity,

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Mileo et al. / Support for Context-Aware Monitoring in Home Healthcare

179

Figure 1. Example: modelling a bedroom

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Table 1. Generic Spatial Relations Among Entities. Relation Name

Object

Reference Object

Relation Type

locate in

Person Area

{Room,Area} Room

generic directional relation generic directional relation

under on near

Object Object Object Object

{Room, Area} Object Object Area

generic directional relation generic directional relation generic distance relation

activity and time [4,5]. In fact, in the context of home monitoring, the more intuitively relevant aspects of a context are: where you are, who you are (clinical profile), which resources you are using, what you are doing and when. In order to represent this information in our model we identify the following entities: • • • •

Person entity to identify the person, her clinical profile and her movements Room entity to identify rooms in the environment Area entity to identify disjoint areas of interest in a room Object entity to identify objects or resources the person can interact with

We also define a small subset of generic spatial relations among entities, summarized in Table 11 . All the other pieces of information are available at a higher level of detail and can be indexed as attributes of the context entities. Attributes value may come from i) external knowledge (observed values for attributes of the Person entity), ii) opportunely aggregated sensor data (all other attributes) or iii) results of the inference process (inferred values for attributes of the Person entity, when observed values are not available). Values of both attributes and spatial relations (except the inclusion of an area in a room which is static) are dynamic and need to be associated to an interval of time. In this way, the reasoning system can take into account their evolution during context interpretation. Lack of specificity w.r.t. the spatial relations is compensated by the inn 1 Note that the spatial inclusion of areas A1 , . . . , An in a room R is such that

i=1

Ai ⊂ R.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

180

A. Mileo et al. / Support for Context-Aware Monitoring in Home Healthcare

Table 2. Attributes representing information about the Person entity. Signature

Domain Values

Description

Func={gait, balance, vision, cognition, bmi, sleep} Val ={absent, mild, moderate, severe}

Functional disability

Adl={mobility, dress, eat}

ADL dependency

Val={ok, needy, dependent} Risk={fall, depression, fragility}

Risk assessment

Val ={absent, mild, moderate, severe}

Test={amtest, minimental, nutritionTest, gds, visual,. . . }

Test results

Val={ok, mild, moderate, severe} Drug={benzodiazepine, antidepressant, diuretic, ssri,. . . }

Medications

Val ={yes, no}

Disease={visual_impairment, artrosis, depression,. . . } Val={ok, mild, modearte, severe}

Diseases



Val={1..300} Val={walk, still, zeroSignal2 }, P={0..100}

Weight in kilograms Motion activity

Val={sit, lay, stand, zeroSignal2 }, P={0..100}

Posture of the person

Val={turn, straight, zeroSignal2 }, P={0..100}

Direction of motion

1 Value “zeroSignal” is related to the fact that no signal is received from sensors detecting movement. 2 Parameter “P” represents data reliability, and it is computed by the algorithms used for feature extraction.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

Table 3. Attributes representing information about the Room and Area entities. Attribute Name

Domain Values

Description

ambientLight ambientLightType ambientHumidity ambientTemperature ambientSound presence smoke

{dark, ..., bright} {natural, artificial} {dry, medium, wet, superWet} {cold, chilly, warm, hot, burning} {mute, mild, medium, noisy} {yes, no} {yes, no}

brightness of the environment nature of the light humidity level temperature noise level presence of a moving entity presence of smoke

ference process: reasoning about relations and attribute values may help inferring new information as detailed in Section 3. Attributes associated with entities included in our model and their values are detailed in Tables 2, 3 and 4. Most of the values for attributes associated to rooms, areas and objects are the result of a process that maps numerical sensor data into meaningful thresholds. The thresholds are derived both from objective considerations, e.g. a given temperature might be too hot for the human body to survive, and from patient-profile and environment related considerations, a southern Italian and a British person might have a very different idea of what is comfortably hot or cold.

3. Intelligent Monitoring and Assessment of Well-Being We refer to an intelligent monitoring system as a monitoring system that is able to i) reason about gathered data providing a context-aware interpretation of their meaning and i) support understanding and decision.

Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

A. Mileo et al. / Support for Context-Aware Monitoring in Home Healthcare

181

Table 4. Attributes representing information about the Object entity. Signature

Domain Values

Description

objectLight objectLightType

{dark, ..., bright} {natural, artificial}

light produced by the object nature of the light

objectTemperature

{hot, cold}

meaning depends on object

objectSound switch

{noSound, regularSound, loudSound} {open, closed}

meaning depends on object state of doors and windows objects

state

{on, off}

state of on/off devices

filteredLoad

{0..300}

weight measurement from load-cells

loadVolatility waterflow

{stable, mildlyUnstable, veryUnstable} {yes, no}

volatility of the load measurement water flowing through the object

presence

{yes, no}

moving entity detected by the object

In the SINDI system, results of context-aware interpretation of gathered data are used to predict and explain possible evolutions of the person’s health state in terms of functional disabilities, dependency in performing daily activities and risk assessment, as well as to identify correct interaction patterns [1,6]. In this section we want to focus on how the system reasons about incomplete and potentially inconsistent sensor data to contextualize them and use them in supporting intelligent monitoring.

Copyright © 2009. IOS Press, Incorporated. All rights reserved.

3.1. Data collection and aggregation First of all, reasoning about gathered data is used to understand what the person is doing in terms of movements, and to localize the person. Some data aggregation (fusion) is already performed at the feature extraction level with statistics-based algorithms, e.g. particle filters. Data can still be imprecise, even after this aggregation process.The expressive power of ASP is used at this stage to disambiguate unclear situations. If we consider SINDI’s localization component (based on the intensity variations of the radio signals exchanged between nodes), it is not always true that the higher the measured intensity of a signal from a node, the closest the person is to that node. Given proximity values with a certain accuracy P and defined over (possibly overlapping) time intervals Ti , Tj , the ASP program takes all available sensor data as input and identifies all possible consistent sequences of moves across rooms and areas. Logical rules for disambiguation state that, by default, proximity to an area A of a room R in a temporal segment T1,T2 is given by the fact that a signal has been received from the corresponding node in that temporal segment. This holds unless there is a more reliable signal received in the same interval from another node. This other signal determines proximity unless additional contextual data make it invalid (e.g. a mat sensor indicating pressure in a different area A1 of room R1) and in case several of these data are available, a measure of reliability can be used to identify the best solution. The high level representation of the collected data is automatically mapped into logic predicates resulting in a set of facts which is combined with the ASP logic program. This high level representation complies with the context model described in Section 2. 3.2. Context Interpretation to Support Health Assessment Besides disambiguation, context interpretation is also a crucial phase in understanding basic behaviours that may be important for health assessment. Workshops Proceedings of the 5th International Conference on Intelligent Environments, IOS Press, Incorporated, 2009. ProQuest

182

A. Mileo et al. / Support for Context-Aware Monitoring in Home Healthcare

As an example of context interpretation, consider night activity. Data aggregation and logical disambiguation produce the following information: • localization details, represented by predicate person_in(Room, Area, T 1, T 2); • state of the wearable device, represented by predicate state(w_device, V, T ), describing the state attribute of object w_device; • values returned by mat sensors, represented by predicate f iltered_load(Obj,V, T ), describing the correspondent attribute of object Obj; The context-aware interpretation of night activity make it possible to infer the following additional information: • • • •

beginning/end of the night period (predicates nightstart(T 1)/nightend(T 0)); the fact that the person exits bed at time T , indicated by predicate f ar(Obj, T ); the fact that there is a sleep break at time T , indicated by predicate break(T ); the fact that the person gets out of bedroom between time T a and T b, indicated by predicate out(Room, Area, T a, T b).

We report a simplified version of the logical encoding, to be parsed by the Gringo grounder and evaluated by the Clasp solver: :- person_in(bedroom,bed,T1,T2), state(w_device,off,T1). :- far(bed,T0), state(w_device,on,T), T0